This is effectively what IPFS is, but IPFS takes it to the next level by using the equivalent of torrent files for subdirectories within the mounted torrent files, so you can "traverse" entire networks of the pretty-much-a-torrent files without first having to download and mount them. And like, you could implement similar functionality here by having torrent files that contain files named .torrent have those files be treated as directories, but at that point IPFS is going to be much more feature complete; even like, if you have some giant file, on IPFS that might be represented by trees of hashes as opposed to merely a giant array of hashes so you can incrementally and efficiently access the parts you need. So like, while this is awesome work, if you find it inspiring, maybe channel that inspiration into learning about IPFS and extending that ecosystem, rather than doubling down on trying to build on torrent files.
How is it "doubling down", they could exist side by side. Your comment kind-of shows the mentality I've seen with the IPFS project before. The IPFS project has doubled down on it's effort of being more "advanced", "innovative" and "efficient" than BitTorrent, losing what actually makes people like torrents and distributed systems.
Unfortunately right now the IPFS project is totally ignoring that it's effectively nothing and takes nothing to the next level. I think what I mentioned before, is the root cause. Zero ounce of compatibility with the existing BT swarm. It is simply a massive amount of wasted potential to ignore the thousands of terabytes of content the torrent system hosts, the work and services provided and spent for the ecosystem. So much could be done that would strengthen both systems at the same time. Just from the IPFS point of view we could right now have embedded resources from torrents, tens of different clients available that could just be CDNs, existing seedboxes strengthening both swarms and probably much more.
That would be the mix of torrents and web people want, not what IPFS is currently. IPFS is basically reinventing the wheel, but throwing out the round part.
Maybe the IPFS project should channel more work on building next to giants rather than doubling down and throwing away everything BT has to offer?
To be clear, I am not involved in the IPFS project and in fact don't like key aspects of the project and think it is doomed to long-term failure when it gets replaced by something better.
That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem?
As such, I don't quite disagree with what you'd like to see done, but that work should go in IPFS... like, trying to extend the way BitTorrent works "up" to something that works for this use case is going to involve a much more fundamental set of low-level changes than even the WebTorrent shift (which is where I'd argue "if only BitTorrent understood the value of working with existing browser-based systems and opening up powerful use cases they would understand that not doing this is like throwing away the round part of a wheel... etc." ;P WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed?...) than the other direction, which is really just "go into IPFS and add a little adapter that lets it work with torrent files (at the fundamental loss of efficiency and awkward lack of resource sharing that such usage implies).
The primary difference I see here is that IPFS is rather centralized in terms of development, it is much much easier for them to write a BT compat layer of some sort than it is for the BT community. A core set of BT is really rather ossified thanks to the amount of implementations out there. Just as an example there are quite a few BEPs out there that implement some features that IPFS has, unfortunately those features haven't gotten any mainstream adoption so far.
> That said, the core and pretty fundamental limitation with BitTorrent, however, is that the swarms exist "per torrent", which kind of makes a non-starter for a general purpose filesystem?
Well, ipfs nodes also tend to pin only certain content. I've seen IPFS links stop working mentioned here on HN even. I see no functional difference in that sense. Content is kept available by those willing to spend resources storing and uploading the specific content.
To address the claimed performance/efficiency problems, IPFS peers could share the same files using IPFS itself, but still also seed them for BT clients.
I totally get though that it wouldn't be super easy to implement, but IMHO the increase in IPFS usefulness, size and resiliency thanks to the integration seems to outweigh the downsides I can see.
Though a while ago I did try sharing the same set of files on two systems by running the two in parallel. It kind-of worked but manually updating the file locations in the torrent client and having an overview became too annoying, it should be built-in. E.g. IPFS sees `x.torrent` and `x`, then assumes `x` is downloadable from BT using that torrent file.
> WebTorrent is the future of BitTorrent as far as I'm concerned, and anything failing to embrace that is already kind of doomed?
What are the current best practices for hosting dynamically updated content on IPFS? Something like a blog that gets updated once a day or a Hacker News clone that gets multiple updates per minute. Last time I checked it was DNS, has something changed sice then?
HN is too frequently updated, but for blogs or personal web pages it's okay. You can either use IPNS or update a DNS record each time you update your content. I chose the latter as years of experiences proved it more robust than IPNS (also, my registrar provides a very simple API that I can use to automatically update the appropriate DNS record, so this is done by the same Python script that update my IPFSite).
The downside is that hosting an IPFS node is quite heavy on the network. I have a reasonably good connection at home (a bit less than 1MB/s up, a bit more than 10MB/s down) and if I run the IPFS daemon locally I can't really use my internet connection for anything else. I used to rent a little VPS for that, but now I use the free-tier of temporal.cloud and it is sufficient for my usage.
Something that's as dynamic as HN seems to me like a bad fit for IPFS. A blog seems fine for it; I make my own blog accessible through IPFS.
DNS seems like the best and currently most popular option for letting you have an address with an update-able IPFS link. The nice thing about using DNS is that you can use the same address for people accessing via IPFS and people accessing via standard browsers.
I've had a breif look at IPFS, but not enough to say anything with any amount of confidence. That said, the thing that immediately put me off was that everything seemed to want to be browser based, like starting some service that makes a local web server that acts as a GUI - there was even a chrome extension that tried to make this easy for you.
A feature that might exist, but I haven't discovered, that would make IPFS more appealing to me, would be a way to mount the IPFS like it were an FTP server, and navigate it with whatever file manager you prefer and not have to worry about managing the backend stuff.
Given the other reply to your comment was someone who seriously thinks IPFS's core feature--being able to be used as a filesystem--doesn't exist, I guess I'll have to go with "better onboarding documentation" ;P.
The source code is far smaller that I would've expected, tiny in fact - no seriously, its probably <2000 lines of actual code and looks extremely simple to audit and fix bugs. It seems the only real external dependency is libtorrent.
Honestly, if your not already amazed if not at the simplicity of this project, than you should be by the simplicity of fuse filesystems and the capabilities it exposes.
If you think that's excellent and are interested in fuse, you're probably intereted in 9front which takes the concept to another level. Ftpfs does what it says, but webfs handles http and forms and the like, upas/fs can show your imap inbox, it's splendid and simple and allows your tooling to be the familiar nix-like utilities like awk and sed. Even capture your screenshot by catting /dev/screen to a file
SSH for job running and management, NFS for filesystem access. Have remote systems mount the local file system. If NFS isn’t available, have the remote job pull files over rsync from your local system, then rsync back the resulting output.
I wouldn't recommend NFS over the open internet but there's a few options based around SFTP (rsync, scp, sshfs) and being based on SFTP means they can run without granting that user full SSH login access while still taking advantage of the security benefits that SSH brings.
For job execution you could write your own agent but doing likely wouldn't be any more secure than SSH. Just make sure you have disabled password logins (use keys instead) and fail2ban or equivalent running to auto blacklist attacks. You could probably use Chef or SaltStack if really wanted to avoid a remote shell but if you're not already running config management then you have to ask yourself if you're over-engineering a solution.
An alternative solution would be to run an OpenVPN tunnel and then you can SSH to your hearts content. But even here, unless you have multiple machines you want to connect to, I can't help thinking you're just making life harder for yourself without getting any realistic gains.
This is all based on the very high level spec provided so I accept there might be some currently undisclosed detail that renders the above suggestions moot.
NFS works great over the open Internet, as long as you do it through a secure tunnel. I've been doing this for years as a way of increasing the size of the available storage in a VPS.
I've used both vtun and WireGuard for remote NFS. On a good day, I get 100MB/s to the NFS filesystem on my San Jose AWS instance (from Vegas) via WireGuard. Note: That was before CoVid-19. CenturyLink/Quest has since (stealthily) throttled my bandwidth down from 1Gbps to ~750Mbps.
It's only 2000 lines because it's in C++. I wrote a simple Fuse system to mount zip and rar files (read-only like this) in <300 lines of Python. Fuse is pretty easy to make useful things with.
This is cool. I can mount most of the Internet Archive as a directory with this. Would love this to grow into what your consumer cloud storage services are but for torrents (a global file system you can mount locally, using torrent data as inodes, with visual representations for availability status similar to the Dropbox green check boxes). Perhaps even with support for web seeding from a storage system of last resort (Backblaze, S3, etc).
I know it's not good spirit, but mounting torrents into the filesystem has been done a few times already, there are many projects that do that already.
The cool part is you can have S3 serve a torrent, and then disable access to the object with permissions once it reaches an adequate seed level. You’re no longer charged for outbound data, but S3 will continue to act as a tracker for the swarm.
While you're here I have to thank you for your go torrent library, it's really nice !
I have never used S3's torrent capability, but at this point bittorrent is pretty much "done" -- there's not much to add to make the protocol better without breaking compatibility, which means there's not much support to maintain a working software.
But I'm not sure about it, maybe they don't participate in the DHT or don't use PEX, so that could be something they can improve.
I wrote a device driver for Windows 8 that would mount a .torrent file as a virtual disk way back in 2011. We pitched the intellectual property around to several large companies. The only interest we had in the technology was video game companies... where we could mark file pieces as 'high priority' and allow the player to play the game even when the game disk was partially downloaded.
The IP was sold to a company developing their next-generation console video game system as part of a 24 million dollar package and never saw the light of day. I suspect what they really wanted was the associated patents.
I wonder how intelligently this handles piece download order. Torrent clients usually use rarest-first download algorithm, but if you've mounted a movie and you're playing it with VLC, you'd probably want it to download sequentially so there's as little time spent buffering as possible. You could download the pieces on demand (whatever piece the application is trying to read), but I suspect that will provide a suboptimal experience, especially if your download speed is barely above the bitrate of the movie.
It can handle videos and other streamable formats if there are enough seeds: the connection is not usually a problem. The pieces are not downloaded strictly sequentially but using a sliding window[1], so if a piece is not immediately available btfs will not waste time and bandwidth waiting for it but just keep downloading other pieces nearby.
The problem is until enough pieces have been assembled, applications reading the file may literally hang: I guess not much software was made with slow I/O in mind.
There are all kinds of reasons a filesystem could be slower to serve certain bytes: striping across multiple heterogeneous physical disks, network filesystems like nfs/cifs/btfs, even IOp starvation.
Video players can gracefully handle streaming media despite spotty connections; is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long? They could default to wanting all the bytes, but have a setting that instructs it to treat the filesystem as if it were a network source.
> is there a reason that they can't pre-read a sliding window of bytes from a file on disk and plow on in case a given range of bytes is taking too long?
AFAIK vlc has that functionality. It allows you to buffer the video n seconds in advance.
Setting a modest buffer size helps but does not solve this entirely. For example, turning on the subtitles or switching audio track will usually hang the player for a while.
I think this should be the default behavior: as soon as you mount the torrent btfs start fetching it like any other torrent client.
I use it mainly for streaming videos: btfs comes with a small python script "btplay <magnet>" that will mount the torrent and start your player, very handy. Another cool thing you can do is downloading and extracting a file simultaneously, just mount a torrent and call "unzip".
I wrote https://github.com/anacrolix/torrent specifically to address the on-demand and streaming requirements of some products I created. It also doesn't make assumptions about how you store the data as you stream, so there's no requirement that the files exist on disk as they're described in the metainfo, like other clients.
libtorrent has repeatedly ignored requests for sequential download because of the silly spec. There's a patch out there that quite easily enables it. It's a bit absurd to me because there's a very good use case for sequential download and it's not clear there are any actual side effects.
Maybe, but probably less relevant for seeds that are not new. Also if people are streaming via torrents it's a net benefit if all peers have early bits not just random data. In short I do not have empirical evicidence but I do believe the decision to do so should not be made by a library and I also believe the intial assumption that random is better holds true today.
Libtorrent-rasterbar has supported sequential downloading for a long time. Are you sure you're not thinking of the other libtorrent (libtorrent-rakshasa)?
> Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers.
However clients can certainly choose to download in whatever order they wish. Like Popcorn Time.
Can someone explain why I might use this instead of downloading a select group of files using a torrent file/magnet link? Is the advantage that it selectively downloads files on an as-needed basis? Or, is the product niche somehow related to the fact that the torrent file/magnet link is mounted as a filesystem, instead of only as a collection of files and folders?
The latter - it lets you navigate the file structure of a torrent as well as read (stream) file content using any of your usual applications, without having to download the files first.
Okay. I'm afraid to ask. What happens in the case of a) a torrent with no seeders or b) Has seeders with enough data to get a directory listing but either the seeders "drop off" afterwards or are incomplete
Other experiences with network filesystems (CIFS/NFS) on multiple platforms sometimes have things getting hairy if the connection to the server drops while the filesystem is mounted or in use
The author could mitigate this by automatically failing any reads where there are zero seeders and/or if the peer piece map does not contain the block.
I would be way more interested in this project if it downloaded normally in the background immediately after the torrent is added, and then downloaded specific bits as programs requested them. I haven't looked into the code base for this exactly but it says nothing about it in the Readme.
Otherwise, like you're describing, you could add a file and then go to read it a week later and be met with no seeders and thus no file when it could have been grabbed somewhere in between.
But maybe that extends beyond the capabilities of a file system?
This is a brilliant concept of a tool. I’ve never worked with monorepos before, but something like this with rw capacity would be an absolutely amazing pipeline, if every developer in the office and beyond are full nodes in the system.
Check out hyperdrive (https://github.com/hypercore-protocol/hyperdrive), specifically the daemon. Torrents are notorious for being read-only, but hyperdrive builds on bittorrent ideas and is making read/write shared spaces a breeze
Every torrent client I've used recently has an "enable sequential download" option. The only torrents this would realistically affect are maybe ones where the seeder may go offline when a bunch of clients have been doing sequential downloads and no one has the final parts of the content.
"download the most recent version signed by this pubkey" is literally how hyperdrive works.
The append-only log is hypercore, one of the building blocks of hyperdrive. Hyperdrive is a layer above and gives you a view of how the archive is at any point in time, with preferential access to the latest version.
TBF what I understood from the request of "mutable torrents" wasn't the literal feature, but a way to have decentralized nodes exchanging updatable content. Bittorent with the Mutable Torrents BEP does work, but it's nowhere near as developed as what hyperdrive is today. And as sad as it may sound I don't think it will ever be.
Isn't that just IPFS? And forget piracy; it's great for ready access to any data that you want mirrored to a lot of servers at once - imagine if you mounted /var/cache/yum over BTFS:) (...maybe with an overlay filesystem in the middle; I forget the internals of yum's cache...)
>At Alibaba, the system transfers 2 billion times and distributes 3.4PB of data every month, it has become one of the most important piece of infrastructure at Alibaba. The reliability is up to 99.9999%.
If you make your own torrent files, how to prevent it accidentally share with the world? Is there anything in the torrent files that can prevent this? (Program and configuration can make mistake.)
Yes, there are BEPs that describe not announcing or accepting connections in certain circumstances. There are also peer discovery mechanisms that are designed to work on local networks. Of course none of this matters if you mess up (and someone externally cares enough to take your data), or if your client doesn't support any of this stuff.
edit: more constructively, this seems to just use libtorrent in the backend, which supports the 'private' flag https://www.libtorrent.org/features.html . Though even if you didn't, it would still 'work' in the sense of being able to download chunks.
Well that is neat, albeit an unfortunate name (1-char off from BTRFS). Wonder what performance is like - does it try to preemptively download? Cache? I would expect it to take a while to run `cp ~/mnt_btfs/big_file ~/Downloads/`, but how about `find ~/mnt_btfs`? If it's performant, it could be really nifty:)
I dont know whether the maintainer is a supporter of crypto (as in tokens) but having a common name with a Tron product (or cryptocoins in general) can backfire.
It looks like the BTFS in the OP was the "original", with its first commit in July of 2015. Could be wrong, because the GitHub interface doesn't make it easy to get to the beginning of the 13k commits for go-btfs.
Yes, OP's btfs was definitely first - in Tron's git history you'll also see that it's a fork of go-ipfs, so the vast majority of commits is from IPFS people.
However it's good to see that Tron are no longer trying to hide the fact it's a fork. For a long time they didn't provide the source code of their btfs.
The TRON Project is also a 1980-1990s project in Japan to develop IoT infrastructure.
They even made their own CPUs and developed 3 different flavours of Operating Systems:
- an embedded real-time kernel ITRON
- an soft real time single user desktop OS called BTRON
- CTRON a server operating systems with real time guarantees for packet/phone line switching
In addition they also developed an true international character encoding before Unicode. It wasn't ASCII backwards compatible which is why it wasn't picked by Unicode committee.
There's also an amazing full length Daft Punk music video that sometimes is considered a sequel to the 80s film, but you'll likely want to watch the spin-off cartoon for a lot of the necessary plot.
In the plot of the cartoon the titular character (program) Tron is assumed dead, and then corrupted by a corrupt system. I can only assume this is why his name is being bad mouthed by some sort of crypto-currency scheme, as Tron was a good program, a security program built to defend the users and would never stand silent on for-profit exploitation of users.
If you are only using torrents for "stolen archives", then you don't know about OS distribution via torrents and archive.org? And we use it internally to distribute large files.
At first I wasn't sure if you were being too serious or just literal when that wasn't needed, but then I realized that you didn't read my full comment.
> Awesome! People can now use standard tools for adding malware into stolen archives! /s
But the most charitable reading I can think of is that you could use BTFS to pirate something, and very slightly reduce the number of steps when modifying it? I guess? But that's no different than just using a normal torrent client and then modifying the files once downloaded. It's just a different interface.
