uTorrent, DHT and private torrents

[denney]

Hey guys. First post here so go easy on me! :P

With all the commotion about DHT, PeX and the like, I decided to go against what my membership to some private trackers told me and enabled DHT and Peer Exchange in uTorrent v1.8.5.

Now, at the time I was ONLY downloading PRIVATE torrents. Nothing public yet. It wasn't long before I noticed there was something wrong. Looking at the tracker list, DHT, Peer Exchange and Local Peer Discovery were DISABLED for all my private torrents. This was fine.

The problem came when I looked at the number of seeds/peers on the torrent. One torrent had 32 seeds and 18 peers in the trackers tab. In the torrent list however, the seeds and peers columns were: 5 (38) and 24 (31). Where did the extra 6 seeds and the extra 13 peers come from if DHT was disabled on those torrents? I was even connected to 6 more peers then there were on the private tracker!

Down the bottom right of the screen, it said something along the lines of "DHT: 121 nodes" (can't remember exactly).

I disabled DHT and PeX straight away and reset the peer lists on the torrents and all went back to normal. So now, WITHOUT STARTING A FLAME WAR, why was uTorrent connecting to seeds/peers that is shouldn't on a PRIVATE torrent?!?!

[riley]

the figures on the trackers tab (32 and 18) show you what the tracker reported last time you announced to it, which may have been half an hour ago. in the meantime new people may have started the torrent, obtained your ip address from the tracker, and connected to you. the (38 and 31) figures are all the peers your client has heard of since you started the torrent. Some of those may have stopped the torrent in the meantime, and told the tracker about that, but clients don't inform each other when they leave a swarm so your client will still remember those peers for a while i'm not sure quite how long). and the (5 and 24) no.s are the actual count of how many people you are currently connected to. which for hopefully obvious reasons, is less than either of the other figures.

in other words, there is nothing mysterious or untoward happening and you are safe to leave dht and pex enabled. as you saw for yourself, even when you leave it globally enabled it will be automatically disabled for private tracker torrents.

[denney]

Why is it that I can disable DHT and reset the peer list and the torrent list never shows numbers higher than what the tracker tab says then?

For some reason the seeds/peers on the torrent list only show more seeds/peers when DHT is ENABLED and not when it's disabled. That's why I'm so confused as to where it's getting these numbers from.

Also, if I force an update of the tracker, the numbers still don't match up (which I assume they should if you only announced to the track 10 seconds ago).

[riley]

i think i've explained that. but anyway, numbers are obviously changing constantly as people join and leave the swarm. perhaps if you could post a screenshot or two we could review those. but one thing i can guarantee 100% is that utorrent is not using dht on private tracker torrrents. it's the most widely used torrent client there is an people watch it like a hawk. once, years ago, a single version of bitcomet available for a couple of weeks did do that, and people still haven't stopped talking about it. nobody has ever suggested that utorrent does that.

and the force updating won't work because every other member of the swarm won't have force updated mere seconds before you. and because when people leave the swarm they don't, as i said, notify other swarm members and they don't always notify the tracker if, for example, their pc crashes or their internet craps out on them.

[denney]

I'll try and grab a screenshot tonight if I can. I don't think I'm explaining how the numbers come about properly so I'll try to be more specific:

1. DHT disabled, private torrent = tracker tab and torrent list numbers match.
2. DHT enabled, private torrent = tracker tab and torrent list numbers DON'T match.
3. DHT disabled, private torrent, force update on torrent = tracker tab and torrent list numbers match.
4. DHT enabled, private torrent, force update on torrent = tracker tab and torrent list numbers DON'T match.
5. DHT was enabled, now disabled, private torrent, clear peer list = tracker tab and torrent list numbers match and STAY MATCHED.
6. DHT enabled, private torrent, clear peer list = tracker tab and torrent list numbers match for maybe 2 or 3 seconds then more seeds/peers appear.

Can you understand my confusion now? Under all circumstances, the numbers stay matched with DHT disabled. Only when DHT is enabled do the numbers get out of sync.

On a related note, if I'm ONLY seeding PRIVATE torrents, why does the DHT nodes list keep growing with DHT enabled? Where is it getting these nodes from? Remember, there are NO public torrents being downloaded or even in the torrent list.

[riley]

lol, probably i'm just tired but i'm not really following you. it's not really valid to expect no's to align perfectly in an ever changing environment where the majority of variables are beyond your control. i mean in terms of who joins or leaves when, and whether they tell you about it or whether they tell the third party (ie. the tracker).

but i say again, although its hard to wrap your head around what's going on, if it was anything even remotely dodgy then people would be screaming about it.

the thing with the nodes list is that it is completely unrelated to what torrents you have running. with peer exchange, you ask other members of swarms you are in for the addresses of peers they are connected to. but dht is totally different. when you start dht your client randomly generates an id for itself that looks like a torrent hash. then it takes on the responsibility of being a tracker for torrents with hashes that are mathmatically similar to the random id it generated for itself. so the nodes you are connected to will be people downloading those torrents, but you won't be downloading or seeding those torrents, you will simply be keeping track of the addresses of the people who are.

[denney]

Ah OK. I understand the DHT nodes thing now. So the client keeps a list of peers that are downloading torrents that are "remotely" similar to the hash it generated? My question now becomes how do the other clients know what my hash is and how do they know to connect to me?

Does uTorrent connect to some central server to announce it's ID? I'm trying to understand how, without seeding a single public torrent (ie. NO connection to anyone except people inside the private torrents), it knows these other clients. In my mind, without another public torrent to connect to other peers, it shouldn't be able to connect to anyone else. How do they find my client then?

Back to my first question, the reason I'm so hesitant and so full of questions is because I know at least 8 people who were banned from private trackers because they had DHT enabled in uTorrent and somehow, the tracker owners knew it.

[riley]

YES, you get it!!! (sorry but dht is tricky to explain and most times i fail to get it explained properly). ok, so, next lesson. say you're connected to nodes a, e, and s, and you want to download torrent l (remember that the nodes have picked for themselves node id's that look like torrent hashes). so you client asks a, e and s whether they know of anyone up/downloading l. a doesn't reply because his pc just crashed and he didn't even recieve your message. e replies that he doesn't BUT, e is connected to b, f and h and he knows that h is closer to l than he is, so he sends you h's address and you fire of a query to him. h doesn't know whos up/downloading that torrent but he happens to be connected to m and he knows that m is closer to l than he is so he sends you m's address and you contact m. and m DOES know whos downloading that torrent and sends you their addresses.

theres a lot of traffic in dht, that's why its slower than a tracker at finding peers BUT, theres also a lot of redundancy which is why it's so bulletproof. and another advantage is that because it's so cumbersome compared to tracker the mafiaa doesn't use it to harverst ip addresss. remove all the trackers from your public torrents and it might take you a few more minutes to ramp up your connections and speeds but you will be much safer.

when you start dht and "login" you try to reconnect to all the nodes you were connected to last time, to tell them your node id and to reconfirm your ip address. if it's the very first time you've ever used dht or if you've deleted your dht.dat file then you won't know anyone and you will try to connect to the dht bootstrap server. but, lets assume you didn't know anyone and you couldn't get through to the bootstrap server for some readon. then your client would just sit and wait and each and every time you connected to a new peer (because you had gotten their ip address from a tracker) or a new peer connected to you (because they had gotten your ip address from a tracker) then your client would ask them 'do you have dht enabled? and when they say yes then that would be your entry point. so you see there is a bootstrap server in there for convenience, but its a decentralised system because its not a necessity to have a bootstrap server.

from then, you kindof "register" yourself within the dht network in the same way as the i described above, you repeatedly ask nodes you're connected to whether they know of anyone else more similar to your node id than they are, then you contact them and ask again until you find the closest people. then y'all wait for people to come looking for you.

"Back to my first question, the reason I'm so hesitant and so full of questions is because I know at least 8 people who were banned from private trackers because they had DHT enabled in uTorrent and somehow, the tracker owners knew it."

well, look at it this way, you don't HAVE to use dht. if you're really worried then just don't use it. but i wouldn't believe everything you hear (the irony of me telling you not to believe what people tell you is not lost on me ) but really there is not even any reason for private tracker owners to ban people for using dht, because they can easily prevent people using dht on "their" private torrents.

[denney]

Thanks for that explaination! I actually understand it. I do have a couple of more questions though if you don't mind.

1. How does the client know where this "boostrap" server is to begin with? Is it hard coded into a client or do you get it from the first DHT enabled client you connect to?

2. I've never run a public torrent with DHT enabled before so the nodes I get when I first enabled DHT, where would they come from?

[riley]

you're welcome

utorrent has two bootstrap servers hard coded in router.utorrent.com and router.bittorrent.com other clients will have other servers (i know there is a router.bitcomet.com)

remember that the nodes have nothing to do with the torrents you are running, you would have picked them up from the bootstrap server the first time you enabled dht

i'm working on a little walkthrough for you that will let you actually see some of this happening but one of the logging options doesn't seem to be working properly anymore so i'm going to see if i can find an oldversion of utorrent because last time i did this was a few versions back. it might take me a day or two. (feel free to keep asking questions in the meantime but i wouldn't want you lose interest and leave before i got a chance to show this to you)

[denney]

That's fine. I'm understanding how this works a little better now. Would like to see the log so I'll be happy to wait for it.

[riley]

update, its working (actually, it seems i was just being a little too impatient and not giving it enough time hee hee). i'll get the writeup done tomorrow

[riley]

ok, this is gonna be long sorry but hopefully educational. close your current utorrent, we're gonna do this in a test installation. create a new folder on your desktop and go to utorrent.com and download the latest utorrent.exe and save it in that folder. then go to the folder and create a new and empty text document and name it settings.dat. now open your hosts file in notepad and add 127.0.0.1 router.utorrent.com and 127.0.0.1 router.bittorrent.com. then double click on the utorrent.exe. say no you don't want it to be the default and use the selected settings. next to dht in the status bar it should say waiting to log in. go to the logger tab and right click and select dump dht buckets and it should show

[2009-11-30 12:07:36] Num buckets: 1. My DHT ID: F3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (i've changed that for privacy thanks to a mods advice)
[2009-11-30 12:07:36] Bucket 0: 0000000000000000000000000000000000000000 (replacement cache: 0)
[2009-11-30 12:07:36] Total peers: 0 (in replacement cache 0)
[2009-11-30 12:07:36] Outstanding add nodes: 0
[2009-11-30 12:07:36] Received: 0 requests (0 B), 0 replies (0 B), 0 invalid (0 B)
[2009-11-30 12:07:36] Sent: 2 requests (134 B), 0 replies (0 B)

see your dht id, that's the thing i told you it creates that looks like a torrent hash. and see that there are no peers, and you're still waiting to log in. thats because you've blocked the bootstrap servers and nothing can happen. now, right-click again and select dump dht tracked and it will show

[2009-11-30 12:08:04] List of tracked torrents:
[2009-11-30 12:08:04] Total peers: 0
[2009-11-30 12:08:04] Total torrents: 0

which should be pretty obvious

now exit utorrent and you will find a dht.dat file created in your directory. that's where utorrent stores all the ip addresses of the nodes it has been connected to between sessions. if you open it in notepad you can see at the end "nodes0" because obviously, we haven't connected to anyone yet.

now, go back to your hosts file and remove those two entries and start up again. this time it should connect to one or two nodes pretty quickly and if you go to logger and dump buckets and tracked you should see

[2009-11-30 12:19:48] Num buckets: 1. My DHT ID: F3xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[2009-11-30 12:19:48] Bucket 0: 0000000000000000000000000000000000000000 (replacement cache: 0)
[2009-11-30 12:19:48] 496892C88F34A9BFD24BAED1D2A850233AD27C08 67.215.242.139:6881 fail:0 seen:1 age:8s ver:UT:999
[2009-11-30 12:19:48] D7C23BF32D946DF36280D31ACDEA461E704F77F2 67.215.242.138:6881 fail:0 seen:1 age:2m 47s ver:UT:999
[2009-11-30 12:19:48] Total peers: 2 (in replacement cache 0)
[2009-11-30 12:19:48] Outstanding add nodes: 0
[2009-11-30 12:19:48] Received: 0 requests (0 B), 7 replies (1.25 kB), 0 invalid (0 B)
[2009-11-30 12:19:48] Sent: 40 requests (3.88 kB), 0 replies (0 B)
[2009-11-30 12:17:09] List of tracked torrents:
[2009-11-30 12:17:09] Total peers: 0
[2009-11-30 12:17:09] Total torrents: 0

so you see that it has connected to one or other or both of the bootstraps. and leave it for longer, maybe up to half an hour and it should connect to more nodes. once it has, dump buckets and tracked again and note that there are now lots of node ids and ip addresses, and that most of the node ids are similar to yours. but you're probably still not tracking any actualy torrents or peers (this is the part i was impatient with and though wasn't working). leave it running overnight or even a whole day like it took me and when you come back and dump traked it shoud show you something like

[2009-11-30 08:25:51] List of tracked torrents:
[2009-11-30 08:25:51] 0: F3AA3BCEDB39160860F0FD564C6326E0F9913FB5: 3 peers
[2009-11-30 08:25:51] 1: F3B7EF292FB02AE37010E7522F9E98D7ED3A76C3: 1 peers
[2009-11-30 08:25:51] 2: F3C0139B33BA371DF055D5A2C71DF24AF5965147: 1 peers
[2009-11-30 08:25:51] Total peers: 5
[2009-11-30 08:25:51] Total torrents: 3

and so you see that the torrents you are tracking are ones with torrent hashes similar to your node id. and also pretty obviously you are tracking torrents that you aren't running

now go to the pirate bay top 100 and pick one of the torrents (it doesn't really matter but we want things to go quickly) and open it. when the box with the list of files comes up click on advanced and delete all the trackers and uncheck peer exchange and local peer discovery and then start the torrent. it should start downloading. obviously it has got the peers from dht because theres nowhere else. go to the general tab and note the hash. unless there has been a ultra incredibly coincidence it will be nothing like your node id, then go back to the logger tab and dump tracked. that hash won't be there of course because you only track torrents like your node id. you've found the peers by finding the person with the node id like that torrent hash and they've been tracking the ips of the peers downloading that torrent

wwwwhhhhheeeeeewwwwww!!!!!! let me know how you get on...

[denney]

Wow. Thank you for the ultra-informative post! I've tried it out and I'm understanding DHT much better now. I still have some reservations about enabling it when downloading private torrents but, and I never thought of this, would it be possible to run 2 instances of uTorrent with different ports? One for private torrents with DHT disabled and one for public torrents with DHT enabled?

Thank you for all the help and information though! I greatly appreciate it.

[riley]

you're very welcome yes you could run two instances (though like i said before i don't think that's necessary but, whatever). but to run them both at the same time you have to create a shortcut to the utorrent.exe and add the /RECOVER parameter to it. otherwise utorrent only lets you run one at a time. and you would probably also want to cap the upload and download bandwidths lower than normal otherwise they will be fighting each other to each use the full share. and of course if you're using a router you will need to forward both of the ports