I think BitTorrent has all the pieces needed for a fully distributed version of your idea. My initial thought is that you could publish a magnet link that points to a mutable DHT item, which in turn points to a torrent that has a JSON file with some metadata and a list of infohashes the publisher cares about. The client could then scrape the "leaf" torrents from multiple lists to get the peer counts and use that for local prioritization of what to store. By reusing existing torrents you could then share resources with standard torrent clients that are unaware of your system.
The list idea could be extended to nested lists (stavros recommends Internet Archive) for discoverability and composition.
If you go with v2 or hybrid torrents from the beginning you could deduplicate and cross seed files from different collections.
The lists could also be modified to have torrents to exclude, possibly using some salt + rehash idea to make it hard to reverse into a list of e.g. CSAM you don't want to publish as is.
Feels like a neat project that could interoperate nicely with existing torrents.
I see things like this, and I wonder why the following software doesn't exist:
I want a piece of software to which I can add a collection of files, say multiple TB. The software will then behave a bit like a BitTorrent tracker, and know which peer has which files. A peer joining this swarm will be able to say "I want to donate X GB of space", and the tracker would tell it "OK, then download and seed these files, which are the least seeded".
The peer would download the files from the rest of the swarm and make them available to it. Then, a request layer on top of the swarm could be used to request a file from the peer which had it. Adding/removing files to this collection would also need to be a feature.
Does anyone know if anything like this exists? If not, how easy would it be to make something like it out of BitTorrent? I might give it a go.
The list idea could be extended to nested lists (stavros recommends Internet Archive) for discoverability and composition.
If you go with v2 or hybrid torrents from the beginning you could deduplicate and cross seed files from different collections.
The lists could also be modified to have torrents to exclude, possibly using some salt + rehash idea to make it hard to reverse into a list of e.g. CSAM you don't want to publish as is.
Feels like a neat project that could interoperate nicely with existing torrents.