I'm fairly certain this has been a feature of S3 since its launch in 2006. Here's as far back as I could find on archive.org[0] it's May 24th 2006 and the URL indicates the documentation was published on March 2006.
Anyone downloading software or any content over a slow and flaky network would question why people don't use torrents for distribution with some "permanent" seeder.
I really had a horrible experience sometime back, while downloading a software of around 2 GB. Where a network would die down and chrome would discard the partial download.
Some distributors like Humble Bundle do on fact do this; the "permanent seeders" are called "web seeds", which are implemented by just having a standard HTTP server implementing the RANGE request. This let's a client fetch arbitrary chunks of a file, so a BitTorrent client can fetch ranges corresponding to torrent chunks.
Could you elaborate on this? I'm not familiar with Bittorrent's protocol. Can a regular HTTP server (say, nginx) be used as a web seed without any software in front of it translating the Bitorrent protocol requests into a regular HTTP request?
The trick is that the client has been modified to read the URLs in the .torrent file and to construct the appropriate range requests for getting pieces from the web seed.
Downloader tools can be quite unpleasent. It is often a choice between overcomplicated adware, or you have to use the vendors software that may not be very good. Downloading large files should be as simple and reliable as small files for the user. Perhaps Metalink[1] support would help?
There's plenty of uncomplicated, adfree software. For Windows, I'm partial to Miniget (which supports both HTTP and Bittorrent): http://www.miniget001.com/
They do. It's used in games very often for example. However, you have to cover this in your EULA (or whatever) and have to have fallback (either optional or automatic) because many ISPs would block such traffic.
AT&T does. My internet would become unusably unstable every time I torrented, so I called them about it. They said they blocked torrents to "comply with the law" or somesuch BS, nevermind that I was attempting to download a Linux distro.
Can you elaborate on this? My current ISP definitely shapes torrent traffic to the point where it is impractical to download anything larger than a few KB.
> I really had a horrible experience sometime back, while downloading a software of around 2 GB. Where a network would die down and chrome would discard the partial download.
Me too. And it was _Apple_ providing the download, for crissakes.
Just moved a bunch of our company's large downloads over to S3 recently (mostly BSP releases and drivers), and this would be awesome for that 100MB - 4GB package size range that they have!
Tried it with a few clients: Transmission does not seem to work, though (both web client, and GTK): "Tracker gave HTTP response code 404..."; rtorrent seems to be able to download (but looks like no upload?); deluge downloads well and also kickstarts the transmission clients.
I wonder what's unacceptable in the Amazon infrastructure for transmission... And the 80kb/s max seed rate mentioned in the comments might be a showstopper already.
A couple of months ago I tested with Transmission (on a Mac) and it worked fine. The problem from my point of view was that the S3 side only seeded at 50-100 KB/sec, but if I downloaded directly from S3 I could pull 2MB/sec.
Unless you have a bunch of seeds, that are essentially permanent, it's pretty useless.
I'm currently using this to build a decentralized distribution system to ship relatively big database files to iOS clients without having to go through S3 all the time. It works quite well but some providers throttle the bittorrent traffic (or worse, throw in RST packets) so I'm not sure how well this will work in practice. Also note that the S3 default seeders only upload at around 80kb/s, so you'll always need at least one external seed to get good performance.
May I ask how you're doing torrent on iOS? The reason for my interest is that I recently ported ipfs (http://ipfs.io) to iOS and am interested in any experience you've had with running these styles of protocols on iOS clients in the field ..
It's just libtorrent packaged up as iOS8 framework + a convenience API, available here: https://bitbucket.org/jberkel/torrada. Still WIP and not used in a production app, but hopefully soon.
It works well if you're on a non-throttled network and has support for local peer discovery which is great for testing (install from a local torrent seed, or another device). Apart from bandwidth saving it's also very convenient to let libtorrent handle consistency checks and resuming of transfers.
ipfs looks interesting, have you published your port?
Thanks for the details about torrada, I'll check it out!
Seems that for peer apps on mobile, we need to make it easy for the user to understand their bandwidth-usage - indeed in my testing I never run the app where I don't have solid, unlimited Wifi. For these classes of apps, I think the presentation of this problem to the user is going to be important.
>ipfs port
Its more of a harness than a port - basically, its possible to use Go on iOS, and so I have a project set up to include Go, cross the bridge between Obj-C and Go runtimes, and so on .. and, then I've built a Go app with ipfs, running on iOS. There aren't any features yet - but the harness is done, so, TODO: write Go, use ipfs framework, interface with iOS-GUI normatives, etc.
When I get actual ipfs functionality coded, I'll put it up on a repo .. too much futzing around is required right now to see anything actually functioning.
Ah, I had a look at goios, I can imagine that it is a lot of work, the integration looks quite complicated and far from being finished, good luck.
About sharing bandwidth, was thinking about only enabling when the user is on wifi, but I think I'll use torrent more as a distributed mirror network than a "real" P2P system, which is difficult in a mobile context. People can easily "donate" bandwidth by seeding the data files needed for the app (using a separate torrent client though, not the app itself).
Actually, I got everything working, and can build and include an ipfs project in the iOS bundle .. I just don't have a UI to wire it up to the Go-side functionality. But in case you're interested you can have a look here:
Might have to change a few paths in the build-script, but it'll give you an idea (if you're interested) how to go about building Go apps on iOS. Turns out its not that hard! :)
>Sharing bandwidth
Yes, I see the same sort of conclusion from my perspective of wanting to put ipfs on iOS - its really only as a way of committing content to the mesh when the User wants to serve - i.e. idle times connected/charging on local Wifi .. I think my next step will be to wire up the camera and a few fields to be used to create a publishable set of content ..
RST should be easy to avoid by prefering UDP where possible, but in general i understand why ISPs would try to hamstring bittorrent with dramatic upstream overprovisioning that they've got.
As someone who prizes his high quality movie torrents, I can testify that files over 5GB in size don't stay adequately seeded for very long after their initial release.
It seems like an arbitrary limit, but an adequate one for the time being.
I wouldn't think it's hugely limiting. In practice I'd speculate that the majority of torrents out there are either around 180MB, 370MB, or 1800MB
If (for whatever reason) you choose to use S3 as your tracker, then you just need to commit to breaking your >5GB content lumps up into multiple pieces.
You're probably right that the majority of torrents on the internet line up with the common TV show sizes, but it doesn't at all follow that users of this service will follow that distribution.
I have 60gb files in S3, and saturating available bandwidth to download them in a reasonable time frame if they're not in the same country is actually a bit of a challenge.
Bit torrent would be one of the fastest and fault tolerant ways to retrieve the files, so it not being available stings a bit.
While I'm trying to download large files on consumer internet, I'd imagine that huge files between geographic locations and server grade connections would face a similar problem.
Javascript is disabled or is unavailable in your browser.
To use the AWS Documentation, Javascript must be enabled.
...really? Their documentation is perfectly readable without it. All the links are real, bookmarkable links; even the buttons for the PDF, forums, and Kindle version work.
Are there easy to use multiplartform BT libs around that would let you use this just for high bandwidth, nearby peers? Eg filter peers by latency as a first pass.
S3 will normally provide ample bandwidth, but in some situations there's a local shared downlink that is the bottleneck. For example in asset heavy multiuser VR or game apps used by many people in same location. By only including the nearby peers I get all the benefit but minimize the risk of people accusing me of clogging / mooching off their uplink.
Wow! This may lower the entry barrier for a new movie/TV streaming company. They can lower their costs by offsetting the bandwidth to users. Legal torrent streaming might have just got the boost it requires.
Spotify already did that early on - the client was also a P2P node that would transfer music files from and to other clients. They shut that part down last year.
When I looked into it a few years ago, Spotify used a custom protocol which was similar to BitTorrent but was not actually using BitTorrent. It had some clever features to optimize for the chunk of the song to be played and it had encryption for the DRM.
Honestly, I'm surprised this is built into S3, rather than CloudFront. Ideally, you want your object replicated to a bunch of edge nodes, which then all simultaneously join into a single swarm as seeds.
I once read an article or email about why web browsers can't/won't implement BitTorrent clients -- one of the biggest barriers IMO to making BitTorrent more ubiquitous for file download/upload is the fact that I can't just click on a link and have my browser handle the rest, like I can with an HTTP/FTP link.
Does anyone have a link that explains why browsers won't do this, or at least a brief explanation?
Opera used to do this, so there's no particular reason specific to BitTorrent that the others do not. Probably the usual questions of priority and bloat.
Apart from myself, anyone else hoping to see downloading straight into an S3 bucket? (Seeding from a bucket has been around for some time, you could imagine my disappointment when clicking the link)
[0] http://web.archive.org/web/20060421112025/http://docs.amazon...