Bug #2447

ST won't check that announce http kind of URL inside the torrent file already has some params (ST adds one more '?')

Added by Wain . almost 5 years ago. Updated almost 5 years ago.

Status:FixedStart date:01/01/2015
Priority:NormalDue date:
Assignee:Andreas Smas% Done:

100%

Category:General
Target version:4.10
Found in version:4.8 Platform:Linux

Description

I'm currently developing plugins for several torrent sites and ran into a strange problem. After some struggling, I was able to download .torrent files and show its contents, but it still fails to play when probing/downloading. It seems like the download actually never starts (shows 0 kB/s). Nothing in the logs. As it affects ALL the torrents of the site (not dependent of the content), I assume it's a bug and you should take a look at it. This exact torrent is downloading fine otherwise (tried with uTorrent, KTorrent, Transmission).
I've included sample .torrent file with the movie, and the site plugin itself. You can use the following test account I've created:
Username: rutracker_showtime
Password: showtime

the_interview.torrent - sample torrent file (15.2 KB) Wain ., 01/01/2015 11:25 PM

rutracker.zip (21.5 KB) Wain ., 01/01/2015 11:25 PM

sample_movie.torrent - This torrent file contains announce URL, but Showtime doesn't use it. (18.6 KB) Wain ., 01/03/2015 12:06 PM

Associated revisions

Revision 4d0fae43
Added by Andreas Smas almost 5 years ago

httpclient: When appending args to path, don't use '?' if path already contains it

Fixes #2447

Change included in version 4.9.16

Revision f5d7508c
Added by Andreas Smas almost 5 years ago

bittorrent: Support 'compact' representation of peers in HTTP tracker reponses

refs #2447

Change included in version 4.9.17

History

#1 Updated by Leonid Protasov almost 5 years ago

  • Related to Feature #2259: Add uTP support (Many seeders use that proto only) added

#2 Updated by Wain . almost 5 years ago

Okay, this might NOT be a problem with UTP. I've disabled UTP in my torrent client (Transmission), and still torrent downloads fine.
I've done some wiresharking and may assume that it can be related to peers announcing. For the given .torrent file and rutracker.org host, Showtime makes the following request:
/ann?uk=0kpqEk18rE?info_hash=%E7%8B%40%F8%E0%AA%A9W%97H%E9%DA%8FL%83r%E74%FA%01&peer_id=5w.i9kzCd87WhYHelMfy&port=7898&uploaded=0&downloaded=0&left=0&event=started
And 403 Forbidden is returned from the tracker, so it can't get any peers (DHT not used, right?).
Transmission CAN get the peers using the following request:
/ann?uk=0kpqEk18rE&info_hash=%e7%8b%40%f8%e0%aa%a9W%97H%e9%da%8fL%83r%e74%fa%01&peer_id=-TR2820-pu4dqn9bc3qy&port=51413&uploaded=0&downloaded=0&left=1573025792&numwant=80&key=3e9ce2d&compact=1&supportcrypto=1&event=started
As we can see, there's much more properties used, and info_hash is NOT uppercase.

#3 Updated by Leonid Protasov almost 5 years ago

I've done some more research on this:

I the request ST does there are missing following params, which Transmission sends:

&numwant=80&key=3e9ce2d&compact=1&supportcrypto=1

6 years ago Transmission had similar bug and they resolved it:
https://trac.transmissionbt.com/ticket/2225

#4 Updated by Leonid Protasov almost 5 years ago

rutracker.org looks at the User-Agent in the http header and bans Showtime. Looking for the proper User-Agent string for ST now...

#5 Updated by Leonid Protasov almost 5 years ago

  • Subject changed from Some torrent files never start downloading with Showtime to Some http-based torrent trackers 403 Showtime by checking it's User-Agent string when requesting announce

Well, now I exactly know what the problem is - rutracker.org tracker sends 403 to the clients which User-Agent are not in white list (ie only known clients are allowed).

I managed to get announce file via Chrome with User-Agent switcher extension and link:

http://bt.rutracker.org/ann?uk=0kpqEk18rE?info_hash=%E7%8B%40%F8%E0%AA%A9W%97H%E9%DA%8FL%83r%E74%FA%01&peer_id=5w.i9kzCd87WhYHelMfy&port=7898&uploaded=0&downloaded=0&left=0&event=started

The User-Agent string I set was: uTorrent/2210

So we can try:
plugin.addHTTPAuth('http://bt.rutracker.org.*', function(req) {
req.setHeader('User-Agent', 'uTorrent/2210');
});

And after that you can check that ann file is downloading by this:

showtime.httpReq('http://bt.rutracker.org/ann?uk=0kpqEk18rE&info_hash=%e7%8b%40%f8%e0%aa%a9W%97H%e9%da%8fL%83r%e74%fa%01&peer_id=-TR2820-pu4dqn9bc3qy&port=51413&uploaded=0&downloaded=0&left=1573025792&numwant=80&key=3e9ce2d&compact=1&supportcrypto=1&event=started');

I dunno if addHTTPAuth affects torrent client of ST - it's up to you to check now :)

#6 Updated by Leonid Protasov almost 5 years ago

  • Related to deleted (Feature #2259: Add uTP support (Many seeders use that proto only))

#7 Updated by Leonid Protasov almost 5 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from Some http-based torrent trackers 403 Showtime by checking it's User-Agent string when requesting announce to Add ability to change User-Agent of torrent client of ST (Some http-based torrent trackers 403 Showtime by checking it's User-Agent string when requesting announce)
  • Target version set to 4.10

#8 Updated by Andreas Smas almost 5 years ago

Leonid Protasov wrote:

I dunno if addHTTPAuth affects torrent client of ST - it's up to you to check now :)

It will affect the HTTP (both HTTP used to download .torrent and communicate with HTTP trackers)

#9 Updated by Wain . almost 5 years ago

Fine, we can use addHTTPAuth to change User-Agent. However, peer_id should also match User-Agent, according to bitTorrent specs. E.g., for uTorrent -UT... peer_id must be used. What's that '5w.i9kzCd87WhYHelMfy' peer_id for? Just a random string? Some trackers may ban Showtime for using this value.

#10 Updated by Wain . almost 5 years ago

My idea is to pretend that Showtime is some commonly used torrent client, such as uTorrent, and use its User-Agent and peer_id for maximizing compatibility with most trackers.

#11 Updated by Leonid Protasov almost 5 years ago

Maybe it's good idea to make st torrenter - copy peerid from user-agent set by httpauth then.

Wain - does your plugin work with httpAuth trick?

#12 Updated by Leonid Protasov almost 5 years ago

Leonid Protasov wrote:

Maybe it's good idea to make st torrenter - copy peerid from user-agent set by httpauth then.

Wain - does your plugin work with httpAuth trick?Ah, regarding the specs wec need a way to set peer_id from plugin http://bittorrent.org/beps/bep_0020.html

#13 Updated by Leonid Protasov almost 5 years ago

Leonid Protasov wrote:

Leonid Protasov wrote:

Maybe it's good idea to make st torrenter - copy peerid from user-agent set by httpauth then.

Wain - does your plugin work with httpAuth trick?Ah, regarding the specs wec need a way to set peer_id from plugin http://bittorrent.org/beps/bep_0020.html

But as I posted above - I was able to get announce list with changed user-agent only.

#14 Updated by Wain . almost 5 years ago

Nope, changing user-agent doesn't help. This is from the logs with HTTP debug enabled:

HTTP            [DEBUG]: Connected to bt4.rutracker.org:80 (id=10)
HTTP            [DEBUG]: > GET /ann?uk=k0WwL02fyu?info_hash=%0E%10%CC%1A%9C%B2%81%B4ZM%04%EA%E4%E9%F2%B1%9B%B9%1C%BA&peer_id=JEDqNsbKRgL2V4jmZUTG&port=7898&uploaded=0&downloaded=0&left=0&event=started HTTP/1.1                                                                                                                                                                                 
HTTP            [DEBUG]: > User-Agent: uTorrent/2210
HTTP            [DEBUG]: > Connection: keep-alive
HTTP            [DEBUG]: > Accept: */*
HTTP            [DEBUG]: > Accept-Encoding: identity
HTTP            [DEBUG]: > Host: bt4.rutracker.org
HTTP            [DEBUG]: http://bt4.rutracker.org/ann?uk=k0WwL02fyu: Response:
HTTP            [DEBUG]: < HTTP/1.1 403 Forbidden
HTTP            [DEBUG]: < Content-Type: text/html
HTTP            [DEBUG]: < Content-Length: 345
HTTP            [DEBUG]: < Date: Fri, 02 Jan 2015 15:34:57 GMT
HTTP            [DEBUG]: < Server: lighttpd/1.4.31
HTTP            [DEBUG]: < 
drain           [DEBUG]: 0x000000: 3c 3f 78 6d 6c 20 76 65  72 73 69 6f 6e 3d 22 31    <?xml version="1
drain           [DEBUG]: 0x000010: 2e 30 22 20 65 6e 63 6f  64 69 6e 67 3d 22 69 73    .0" encoding="is
drain           [DEBUG]: 0x000020: 6f 2d 38 38 35 39 2d 31  22 3f 3e 0a 3c 21 44 4f    o-8859-1"?>.<!DO
drain           [DEBUG]: 0x000030: 43 54 59 50 45 20 68 74  6d 6c 20 50 55 42 4c 49    CTYPE html PUBLI
drain           [DEBUG]: 0x000040: 43 20 22 2d 2f 2f 57 33  43 2f 2f 44 54 44 20 58    C "-//W3C//DTD X
drain           [DEBUG]: 0x000050: 48 54 4d 4c 20 31 2e 30  20 54 72 61 6e 73 69 74    HTML 1.0 Transit
drain           [DEBUG]: 0x000060: 69 6f 6e 61 6c 2f 2f 45  4e 22 0a 20 20 20 20 20    ional//EN".     
drain           [DEBUG]: 0x000070: 20 20 20 20 22 68 74 74  70 3a 2f 2f 77 77 77 2e        "http://www.
drain           [DEBUG]: 0x000080: 77 33 2e 6f 72 67 2f 54  52 2f 78 68 74 6d 6c 31    w3.org/TR/xhtml1
drain           [DEBUG]: 0x000090: 2f 44 54 44 2f 78 68 74  6d 6c 31 2d 74 72 61 6e    /DTD/xhtml1-tran
drain           [DEBUG]: 0x0000a0: 73 69 74 69 6f 6e 61 6c  2e 64 74 64 22 3e 0a 3c    sitional.dtd">.<
drain           [DEBUG]: 0x0000b0: 68 74 6d 6c 20 78 6d 6c  6e 73 3d 22 68 74 74 70    html xmlns="http
drain           [DEBUG]: 0x0000c0: 3a 2f 2f 77 77 77 2e 77  33 2e 6f 72 67 2f 31 39    ://www.w3.org/19
drain           [DEBUG]: 0x0000d0: 39 39 2f 78 68 74 6d 6c  22 20 78 6d 6c 3a 6c 61    99/xhtml" xml:la
drain           [DEBUG]: 0x0000e0: 6e 67 3d 22 65 6e 22 20  6c 61 6e 67 3d 22 65 6e    ng="en" lang="en
drain           [DEBUG]: 0x0000f0: 22 3e 0a 20 3c 68 65 61  64 3e 0a 20 20 3c 74 69    ">. <head>.  <ti
drain           [DEBUG]: 0x000100: 74 6c 65 3e 34 30 33 20  2d 20 46 6f 72 62 69 64    tle>403 - Forbid
drain           [DEBUG]: 0x000110: 64 65 6e 3c 2f 74 69 74  6c 65 3e 0a 20 3c 2f 68    den</title>. </h
drain           [DEBUG]: 0x000120: 65 61 64 3e 0a 20 3c 62  6f 64 79 3e 0a 20 20 3c    ead>. <body>.  <
drain           [DEBUG]: 0x000130: 68 31 3e 34 30 33 20 2d  20 46 6f 72 62 69 64 64    h1>403 - Forbidd
drain           [DEBUG]: 0x000140: 65 6e 3c 2f 68 31 3e 0a  20 3c 2f 62 6f 64 79 3e    en</h1>. </body>
drain           [DEBUG]: 0x000150: 0a 3c 2f 68 74 6d 6c 3e  0a                         .</html>.
HTTP            [DEBUG]: Disconnected from bt4.rutracker.org:80 (id=10) Request destroyed

This certainly needs more investigation. I'll try work it out now.

#15 Updated by Leonid Protasov almost 5 years ago

That is something new. Old link won't work too. Maybe your user key is banned.
About peer_id - st way of generating it is good as: http://www.ghacks.net/2012/09/22/utorrent-3-3-alpha-randomizes-peer-id-during-sessions/
So ST should follow that practice.

#16 Updated by Wain . almost 5 years ago

Leonid Protasov wrote:

That is something new. Old link won't work too. Maybe your user key is banned.
About peer_id - st way of generating it is good as: http://www.ghacks.net/2012/09/22/utorrent-3-3-alpha-randomizes-peer-id-during-sessions/
So ST should follow that practice.

My user key is definitely working - I'm trying the same torrent with Transmission, and it's working there (announce returns 200 OK).
About the peer_id - it's rather unclear from the article whether it randomizes full peer_id, or just the its random part (after client ID and version).

#17 Updated by Leonid Protasov almost 5 years ago

Wain . wrote:

My user key is definitely working - I'm trying the same torrent with Transmission, and it's working there (announce returns 200 OK).
About the peer_id - it's rather unclear from the article whether it randomizes full peer_id, or just the its random part (after client ID and version).

Yeah, but interesting thing is - 3.3 is banned on that tracker (i suppose because of empty User-Agent string)
http://wiki.rutracker.org/%D0%9D%D0%B5%D1%81%D0%BE%D0%B2%D0%BC%D0%B5%D1%81%D1%82%D0%B8%D0%BC%D1%8B%D0%B5_%D0%B8_%D0%BD%D0%B5%D1%80%D0%B5%D0%BA%D0%BE%D0%BC%D0%B5%D0%BD%D0%B4%D1%83%D0%B5%D0%BC%D1%8B%D0%B5_%D1%82%D0%BE%D1%80%D1%80%D0%B5%D0%BD%D1%82-%D0%BA%D0%BB%D0%B8%D0%B5%D0%BD%D1%82%D1%8B_%D0%BD%D0%B0_rutracker.org

#18 Updated by Leonid Protasov almost 5 years ago

Not banned but not recommended. And if I got it right - User-Agent is empty on 3.3 and some peer_id is random (or as you noted maybe only some part of it).

#19 Updated by Wain . almost 5 years ago

Yea, finally I've found a bug! Aside of empty User-Agents and peer_id's, that one was extremely simple but yet hard to observe. Look at the following URL Showtime produces:
/ann*?*uk=k0WwL02fyu*?*info_hash=%0E%10%CC%1A%9C%B2%81%B4ZM%04%EA%E4%E9%F2%B1%9B%B9%1C%BA&peer_id=JEDqNsbKRgL2V4jmZUTG&port=7898&uploaded=0&downloaded=0&left=0&event=started
See the TWO '?' signs? Second one must be '&'. When I change it manually, the request is OK even with Showtime's default User-Agent.

#20 Updated by Leonid Protasov almost 5 years ago

  • Tracker changed from Feature to Bug
  • Subject changed from Add ability to change User-Agent of torrent client of ST (Some http-based torrent trackers 403 Showtime by checking it's User-Agent string when requesting announce) to ST won't check that announce URL inside torrent already has some params (double '?' problem)
  • Found in version set to 4.8
  • Platform set to Linux

#21 Updated by Leonid Protasov almost 5 years ago

  • Subject changed from ST won't check that announce URL inside torrent already has some params (double '?' problem) to ST won't check that announce http kind of URL inside the torrent file already has some params (ST adds one more '?')

#22 Updated by Andreas Smas almost 5 years ago

Seems the URL double '?' is not the only problem with this torrent but it also uses a compact formating of peers. So i'm gonna fix that as well.

Still for this interview torrent I only get one peer which refuses to respond so it's hard to know if it works 100%

#23 Updated by Andreas Smas almost 5 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

#24 Updated by Wain . almost 5 years ago

I can confirm that with the latest build 4.9.21 rutracker.org is finally working. Thanks a lot Andreas!

#25 Updated by Wain . almost 5 years ago

Okay, this one is fixed, but I have more =) Please take a look at the other torrent file (another domain, but the tracker engine is the same). This one downloads fine using Transmission but Showtime never makes announce request, according to Wireshark (.torrent definitely contains the URL, Transmission makes it OK). Thanks in advance!

#26 Updated by Wain . almost 5 years ago

Sorry, I was mistaken with the last one, but the problem remains. The announce request is being made, but you have to wait A LOT for it. I just left Showtime opened for about 20 minutes before it actually sent announce. This is not ok I guess?

#27 Updated by Leonid Protasov almost 5 years ago

Wain . wrote:

Sorry, I was mistaken with the last one, but the problem remains. The announce request is being made, but you have to wait A LOT for it. I just left Showtime opened for about 20 minutes before it actually sent announce. This is not ok I guess?

In settings:dev at the bottom there are bittorent debugs. Turn them on and see...

#28 Updated by Leonid Protasov almost 5 years ago

Wain . wrote:

Sorry, I was mistaken with the last one, but the problem remains. The announce request is being made, but you have to wait A LOT for it. I just left Showtime opened for about 20 minutes before it actually sent announce. This is not ok I guess?

Try latest bleeding edge - it's fixed there :) Hope to see your plugin alive soon ;)

Also available in: Atom PDF