Bug #2771

503 response in showtime.httpReq(url) causes error

Added by Michal Pisarski over 6 years ago. Updated about 6 years ago.

Status:FixedStart date:08/07/2015
Priority:NormalDue date:
Assignee:Andreas Smas% Done:

100%

Category:API
Target version:-
Found in version:4.9.524.g52f7d Platform:Linux

Description

In movian when showtime.httpReq(url) gets 503 response it causes the plugin to exit.
Dump:

kisscartoon     [ERROR]: Error (HTTP request failed http://kisscartoon.me/CartoonList/LatestUpdate -- HTTP error: 503) at /project/repo/checkout/src/ecmascript/es_io.c:412
kisscartoon     [ERROR]: STACK DUMP: Error: HTTP request failed http://kisscartoon.me/CartoonList/LatestUpdate -- HTTP error: 503
                                /project/repo/checkout/src/ecmascript/es_io.c:412
                                anon  native strict preventsyield
                                anon showtime/http:101 tailcalled
                                browseItems file:///home/pisek/workspace/showtime-plugin-kisscartoon/kisscartoon/kisscartoon.js:47
                                anon file:///home/pisek/workspace/showtime-plugin-kisscartoon/kisscartoon/kisscartoon.js:60 preventsyield
                                apply  native strict preventsyield
                                anon showtime/page:366 preventsyield

It should not work like that as there are many pages that respond with 503 (with custom error-page) together with Refresh header url which reloads content from the dynamically created, proper site (eg. CloudFlare).

Associated revisions

Revision 59277a6b
Added by Andreas Smas over 6 years ago

ecmascript: Add 'noFail' option

Fixes #2771

History

#1 Updated by Leonid Protasov over 6 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

Please use at least 4.10 and if the same is there - use following code:

var ok = false;
try {
  httpReq here
  ok = true;
} catch(err) {

}
if (!ok) ...

#2 Updated by Michal Pisarski over 6 years ago

The error is still there.
The response headers are not attached to any field in 

err
...

#3 Updated by Michal Pisarski over 6 years ago

The error is still there.
The response headers are not attached to any field in err...

#4 Updated by Michal Pisarski over 6 years ago

Object.getOwnPropertyNames(err) shows only message field which shows:

HTTP request failed http://192.168.0.11:9091/transmission/rpc -- HTTP error: 409

#5 Updated by Andreas Smas over 6 years ago

  • Status changed from Fixed to Accepted

This could be improved.

#6 Updated by Andreas Smas over 6 years ago

  • Status changed from Accepted to Fixed

#7 Updated by Andreas Smas over 6 years ago

Added noFail option

example:

var http =require('showtime/http');
var doc = http.request('https://movian.tv/foobarfoo', {
  noFail: true
});

print("Response object:");
print('Status: ' + doc.statuscode);
print(doc.toString());
for (var hdr in doc.headers) {
  print(hdr + ": " + doc.headers[hdr]);
}

yields:

Response object:
Status: 404
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <title>Redmine 404 error</title>
  <style>
    body {font-family: Trebuchet MS, Georgia, "Times New Roman", serif; color: #303030; margin: 10px;}
    h1 {font-size:1.5em;}
    p {font-size:0.8em;}
  </style>
</head>
<body>
  <h1>Page not found</h1>
  <p>The page you were trying to access doesn't exist or has been removed.</p>
  <p><a href="javascript:history.back()">Back</a></p>
</body>
</html>

CF-RAY: 2385bb0218b216a6-ARN
X-Rack-Cache: miss
Cache-Control: private
X-Runtime: 0.002521
X-Request-Id: ad6d315c910b2c92be58cdd11c7efe1b
Status: 404 Not Found
Connection: keep-alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Oct 2015 15:24:35 GMT
Server: cloudflare-nginx

#8 Updated by Andreas Smas over 6 years ago

I also added the statuscode property of the response object which might come in handy.

There's also a question if Movian should transparently handle Refresh: 0,URL -tag similar to a 3xx redirect?

#9 Updated by Michal Pisarski over 6 years ago

Great! Thanks!:)

#10 Updated by Michal Pisarski over 6 years ago

And about this:

<meta http-equiv="refresh" content="0;URL='http://dupa.com/'" />

IMHO yes, but there should be an option to forceNoRedirect or sth so that one can read this redirection page - there might be some important stuff there like session id or some important header etc.

#11 Updated by Buksa ... about 6 years ago

cf antibot
source https://github.com/Anorov/cloudflare-scrape/blob/master/cfscrape/__init__.py
@respond = http.request(BASE_URL, {
method: 'GET',
headers: {
'User-Agent': USER_AGENT
},
noFail: true
})
if (respond.statuscode === 503) {

match = /name="jschl_vc" value="(\w+)"/.exec(respond);
            jschl_vc = match != null ? match[1] : "";
            match = /name="pass" value="([^\n]+?)"/.exec(respond);
            pass = match != null ? match[1] : "";
var match = /setTimeout\(function\(\)\{\s+(var t,r,a,f[^\n]+?\r?\n[\s\S]+?a\.value =[^\n]+?)\r?\n/.exec(respond);
            result = match != null ? match[1] : "";
            result = result.replace(/a\.value =(.+?) \+ .+?;/g, "$1");
            result = result.replace(/\s{3,}[a-z](?: = |\.).+/g, "");
jschl_answer = (+(eval(result)) + +BASE_URL.split("/")[2].length)
setTimeout(function() {
                respond = http.request('http://online.anidub.com/cdn-cgi/l/chk_jschl?jschl_vc=' + jschl_vc + '&pass=' + pass + '&jschl_answer=' + jschl_answer, {
                    //method: 'GET',
                    headers: {
                        'Referer': BASE_URL,
                        'User-Agent': USER_AGENT
                    },
                    noFail: true
                })
            }, 4000);
}
        print(respond)@

Also available in: Atom PDF