Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints.
Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice.
Unlike HTTP, you get reasonable retry behavior on network hiccups.
Also, more robust data integrity guarantees, though a manual hash test is probably a good idea either way.
At least among people I'm throwing TBs of data around with, torrent infra is common and it's nice to not have to deal with some special-purpose tool that, in practice, is probably a pain in the ass to get compatible versions deployed across a range of OSes. Basically every platform known to man can run a torrent client of some sort.
And obviously, no dependency on an intermediary. This is good if you're trying to avoid Google et al. That does, however, bring a potential con: if my side is fast and your side is slow, I'm seeing until you're done. If I'm uploading something to gdrive or whatever, I can disconnect one r the upload is done. If you control an intermediary like a seedbox, that's less of a problem.
In general, though, torrents are pretty great for this sort of thing. Just encrypt your stuff beforehand.
I am working on a tool that will help with this. It is built with AlpineJS, Nim, Aria2 and Webview resulting in a 5MB download which doesn't include the torrent file as that varies. The idea is that it is a one click solution as the torrent is embedded with the binary.
The inspiration for this tool is for assist with LAN parties. The one thing I have against the private flag is that it also disables LAN peer discovery, which would be okay if you use a tracker behind the LAN (though the torrent would need to be modified if the hostname/IP changes). Since you can't configure other people's clients I found it simpler to use Aria2 that is preconfigured to disable PEX and DHT.
Combine this with Metalink or Web Seeds, so that you can have a initial seeder based on HTTP. I think using IPFS would be a great web seed as long as the their gateway and Cloudfare's continues to stay up. IPFS creates a permanent URL if you will, so no need to worry about dynamic IPs or domains. It would be great if IPFS would have a smaller binary (right now it is around 20MB compressed) and had a way to get a file and "seed" it like a torrent. But for now I think torrent is mature enough.
I have considered it. The one thing about Syncthing is that there is no true readonly way to sync. For example, someone may accidentally extract an archive in the Syncthing folder and it will sync everywhere. If Syncthing gets this feature I think I would be 100% onboard with it for LAN parties.
Resilio I believe supports it but I would prefer an opensource alternative.
Sure, I can have for example my computer be send only. But unless everyone else configures it properly as receive only, they can start pushing data to other clients. It would be far easier to have Syncthing create a readonly share similar to Librevault and Resilio.
You could also gpg encrypt the file first which is a little more work but bulletproof. Maybe someone should make a tool that creates a torrent and encrypts it for you and presents a password box on the other end.
That's why you encrypt it or at least make it a password protected archive. That way no one will see the file names in the content list. And without seeders, it's highly unlikely anyone will download it. Torrents work well with 1 seeder, 1 downloader.
I'm not an expert on this, but as far as I'm aware, the main difference is that your client won't go out and announce the infohash to the DHT (used for trackerless torrents, magnet links, that kind of thing) and won't provide it to other clients for download. It'll only ever send requests relating to it to the trackers specifically listed within the torrent itself. This means that anyone who wants to download it basically needs to posses the torrent file. Trackers don't really crawl in this way, however, some searching sites like the one I mentioned there do, they'll go out and look at the entire DHT, trying to find entries that they don't know yet and when they find one - go download the torrent and post it to their index.
For some reason I haven’t understood, syncthing would often miss transferring files but mark the entire transfer/sync as complete. No errors in the logs either. I tried it for a few months earlier this year, then gave up and switched to plain rsync in a shell. Based on that sample of one personal experience, I wouldn’t recommend syncthing to anyone.
Note: I was using syncthing for a one way sync all the time, with send-only and receive-only settings on the endpoints.
I've been using Syncthing extensively for years and never observed this behavior - it always either completes the transfer or fails to complete the transfer (which is obvious in the UI). A silent failure is certainly not typical.
This sounds like a pretty severe bug, so if you still have the systems you were able to produce the behavior on it might be helpful to reproduce it and submit an issue.
Yeah, I gave up an Syncthing when using it between my Android device and Linux server. Too many hours were lost trying to debug why the server no longer showed as connected on the phone. At some point I went back to emailing myself files, because that actually worked, and I realized that I didn't actually need an entire program with a synchronization algorithm to send a few large files on the occasion.
I don't see how it was made impossible. It's definitely possible, you can do it yourself there's many examples on github (even btdig has an old version on it)
Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers).
I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something right.
Does Firefox have an extension to download torernts?
I know it used to have one, but that now that I'm aware of the v57+ extension issue, I notice that the one linked to first in a search seems to be dead.
Probably, but it is quite a neat technology. As people in this industry, we know what else it is being used for, and it serves those purposes pretty well. I love torrents. I love the technologies that it uses. Many other projects use it, for example take a look at https://docs.ipfs.io/concepts/dht and https://github.com/savoirfairelinux/opendht just for DHT alone. There is also https://en.wikipedia.org/wiki/Magnet_URI_scheme. There are many more, just too lazy to find them at the moment, but believe me, the things you will encounter once you go down this road, is truly amazing. Worth it!
> the bad connotation due to piracy made this protocol not standard
Back when Resilio Sync was known as Bittorrent Sync, I proposed it as a file-sharing solution between multiple locations for the company I was working for. Immediately returned with an emphatic refusal to use any kind of software that had "Bittorrent" in the name, regardless of what it was for.
This is the same company where they picked up I had "Bittorrent Software" installed on my work laptop, and the people who ran the auditing software freaked out because, you know, it's Bittorrent! I told them it was a single icon in my PortableApps folder but they just kept saying "There's a Bittorrent client on your PC, you could be fired!" until I deleted the icon.
So yes, the fact it's associated with piracy has made a huge difference in some places.
WebTorrent [1] can be used to run BitTorrent on the web.
It's actually integrated into the Brave browser which is a nice feature. Other browsers might catch on soon.
Not quite there yet. Latest release of libtorrent was 1.2.9, seems it's scheduled for "some branch" after 2.0 (https://github.com/arvidn/libtorrent/pull/4123#issuecomment-...). Add in some time for clients to actually start using it as well, which is not guaranteed either.
It was probably phones and datacaps that really did it. Torrents are really only ok on laptops/desktops and the unexpected data usage for customers is much worse than the costs of distributing data.
Steam could probably benefit loads from torrents but they do have extremely good distribution servers already.
BitTorrent didn't catch on simply because there aren't very many actual use cases for it. The majority of P2P content distribution will always be illegal (due to the nature of copyright), and so the client-server model is much more representative of the real world. It is technologically pretty neat though, and some publishers do actually use the protocol for things like software updates.
Except for:
- updates of software.
- distribution of public video files.
- distribution of install images.
- distribution of raw public measurements (ea science data).
Basically BitTorrent is the protocol for any usecase with 1:n (or n:n) with public data (ea. No data access controle is needed).
Copyright has nothing to do with it and the nature of BitTorrent does make it also a good fit for some pirates.
Never forget that the following is shared over BitTorrent:
- World of Warcraft updates.
- Linux distribution install images.
- haveibeenpowned sha lists.
And that’s just what I personally know. I know there is more.
You need to notice the connection died and do that manually, or think ahead and have your script do it in a loop. A bt client already has the reconnect logic. And it will hash the contents to make sure it worked correctly.
I had to transfer 2tb of data over a wireless connection that kept dropping packets. BitTorrent was the only method I found that worked; everything else either broke down when it disconnected, or had too low of a retry limit to be useful, or wouldn't resume properly.
Almost all BT clients can create torrents. You can use a public tracker if you want, or you can use a magnet link (DHT magic will find the peer eventually), or worst-case scenario you can manually add the peer IP on either end.
With transmission-create you can create a torrent from the commandline. Load it into your bittorrent client, find its hash in the UI, and make it into a magnet link, like:
BitTorrent is the best for sending large files. I wish there existed a tool to make torrent creation easier, and I also wish my computer could seed. Every time I create a torrent, nobody can discover my server, even though the port is open and even if I add the server IP as a peer directly onto the client.
I have no trouble seeding in swarms, only when I create torrents. Other than that, BT is ideal.
This is such a facepalm moment. "This service is awesome, it does everything you need, it's reliable, it's safe, it's practical, and all you have to do is encrypt your files, share encryption keys, validate identities, and establish trust with the person you're sharing with. Also if you screw it up in any way, your file is public forever. It's so rad!"
No, "here's a magnet link, and here's the matching 50-char password; don't share the password with anyone you don't want have file". Just because you can make security more complicated doesn't mean you're obligated to.
> Also if you screw it up in any way, your file is public forever.
You can literally say that for ANYTHING out there. Not sure what point you are trying to make. Let me guess, "Encryption is hard?". Well there's no free lunch.
Well there was free lunch when Mozilla's Send thing was running. Sharing the password in the URL hash was brilliant. But then it ran into all the same Mega.nz problems where people immediately use it to host malware.
Encryption is hard, which is why it's sad to see that when someone gets it right and nails the UX, it still gets ruined by the real world somehow.
Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice.
Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea either way.
At least among people I'm throwing TBs of data around with, torrent infra is common and it's nice to not have to deal with some special-purpose tool that, in practice, is probably a pain in the ass to get compatible versions deployed across a range of OSes. Basically every platform known to man can run a torrent client of some sort.
And obviously, no dependency on an intermediary. This is good if you're trying to avoid Google et al. That does, however, bring a potential con: if my side is fast and your side is slow, I'm seeing until you're done. If I'm uploading something to gdrive or whatever, I can disconnect one r the upload is done. If you control an intermediary like a seedbox, that's less of a problem.
In general, though, torrents are pretty great for this sort of thing. Just encrypt your stuff beforehand.