The edited title was a bit confusing to read for me. I would've found it easier to see it reversed:
The Four Most Expensive Words in the English Language: "This time is different"
Anyway, on to the actual topic... I need to sit down to give this more thought, but I still think there's value in P2P networks, and it's worth pursuing. You should be able to easily replicate and share content that you care about. The big incentive is being able to keep your own copy and being able to do whatever you want with it. How many times have you tried searching for an old YouTube video, only to find it got taken down?
Another example: package managers! You want to have a machine-local cache, a remote cache for reproducible builds and deployments, and whatever other remotes you get the original packages from. Maybe different registries have varying content policies, and you can pick your favorites. If you have a few developers in an office it's a lot faster to grab dependencies from the laptop of the person sitting next to you than a CDN.
Aren't you just replicating the same mistake he's describing as being made over and over again? Where people express a reasonable desire but don't think through the issues before deciding it's worth pursuing?
In tech we have this habit of going from, "Wouldn't it be cool if X..." to "Let's build X right now!" I think that's a fine instinct if it hasn't been tried and it's something that runs on the author's machines. Something will be learned.
But when we're talking real money and real work from a lot of people, I think it's worth stopping and asking some questions. Yes, sure, X would be cool. But are there perhaps subtle reasons why X has not happened yet? As with jetpacks and 3D video, sometimes cool things are impractical or don't pay off as much as you'd first think.
Subtle reasons why P2P technologies are squashed likely include the difficulty of subversion by authorities and the threat that private communications represent to the copyright lobby.
What you're talking about is the difference between an ecology and a monoculture.
The reason why we don't see monocultures in nature is because they put every single one of their members at risk of sudden death. A single pathogen can wipe them out. The exhaustion of resources required to sustain them can starve them. The build up of by products can kill them. Think of a medieval town drowning in excrement, running out of top soil and about to get hit by the plague.
An ecology on the other hand puts none of their members at risk by virtue of their existence because they are diluted. Think of a band of hunter gatherers, they might individually die at 40, be exposed to ebola and worse, but the fact they are so widely separated and so few means that they have few if any vulnerabilities that can kill the whole tribe.
The reason why we see monocultures now is because they are more efficient and our time horizons are tiny. Amazon will fail catastrophically. Billions or trillions will be lost. No one will learn anything because it's cheaper not to than to build a really resilient network.
This is a greatly simplified and probably unproductive view of both ecologies and monocultures. Certainly nature as a whole is an ecology, but you are vague in defining what a monoculture even is, or how both even apply to human culture.
To wit, there are massive redwood forests that occur in nature that are just one cloned genome. These are reproductively successful on timespans many times a human life.
Is Amazon itself a monoculture? Is that relevant? Is enforcement of law by the state a monoculture? There's a lot of evidence to suggest that centralized information management entities are actually much more robust in the long term than a loose confederation of peers.
>To wit, there are massive redwood forests that occur in nature that are just one cloned genome. These are reproductively successful on timespans many times a human life.
That there is a massive monoculture in an area that was an extreme desert until as little as 5000 years ago makes my point that a monoculture is more efficient in the short term. A human life is a terrible stick to measure geological time, one might as well measure success in the stock market using an atomic clock.
I have not defined what a monoculture is because it should be obvious by the name. Likewise for ecology.
>Is Amazon itself a monoculture? Is that relevant?
Yes. And because monocultures are prone to sudden collapses.
>Is enforcement of law by the state a monoculture?
Depends on the number of states. Depends on the diversity of laws between the states.
>There's a lot of evidence to suggest that centralized information management entities are actually much more robust in the long term than a loose confederation of peers.
The fact that we don't observe this in nature, or that the few times we do observe it is either before or after a mass extinction event should tell you that evidence has too narrow a time horizon.
I think your central claim is sort of refuted by history, and I think there's two parts to it.
1. How relevant "natural" ecology is to understanding longevity of modern culture: unclear, probably not very.
2. Monocultures by nature are less robust, even in the context of human society: well, you need only examine how quickly distributed hunter gatherers were exterminated by less distributed agrarian societies, who were in turn more swiftly replaced by more centralized industrialized societies...
My personal hypothesis: It's probably more reasonable to say that humans were the first aspect of natural ecology to get really deep into economies of scale, and by virtue of that, retained a monopoly on that incredible power relative to the rest of natural ecology.
Industrial society is itself rapidly approaching overshoot of the habitability of this planet. Without being able to fully, artificially generate everything from food to atmosphere without it being subsidized by natural resources that are not renewable at the rate at which we're exhausting them, industrial civilization will collapse in all its complexity. To take a natural ecological example: just because a predator figures out a very efficient way to eat all of the available prey does not protect it from when the food is exhausted. Just because an invasive herbivore is better at stripping the landscape bare of vegetation does not protect it from the consequences of its success. Just because agricultural and industrial civilizations supplanted the hunter gatherers does not mean that they are sustainable long term. If society manages to completely replace its habitat and resource inputs with technological prowess, then it will survive. We are not even close to that point yet given our current dependencies. We depend on a finite amount of energy, sunlight stored in an efficient chemical form millions of years ago, and predictable weather in order to grow crops in the ground outdoors. Oceans with a limited regenerative capacity needed to feed the world that are being overexploited. Soil phosphate and other minerals that we already must add back artificially, also using non-renewable inputs. We are not separate or dominant over natural ecology until we are no longer dependent on it for survival, and until that point we're like college students living on loans and our parents' money and calling ourselves wealthy and independent.
This is just moving the goalpost. What you're saying is that modern humans are too good for their own good :).
Anyway, I don't understand the religion of distribution (especially in context of finite resources). Yes, distributed systems are more robust. But they pay for this by being massively less efficient than equivalent centralized ones. It's a fair tradeoff, but a tradeoff. It's like 5 people in a (single) car vs. 5 people on bicycles. Which setup is better depends on where those people want to go.
What we have is either ten people walking or 10,000 people strapped in the space shuttle cargo bay. That it takes the ten people 30 years to walk from Lisbon to Singapore vs 45 minutes with the shuttle isn't the right comparison. It's that the shuttle explodes 2% of the time.
That you are forcing me to be with you on the shuttle because you don't understand the risks should be all the reasons we need for walking instead.
The glib rebuttal would be that the internet was once distributed, and it was unimaginable that the content would become as centralized as it has today.
> I need to sit down to give this more thought, but I still think there's value in P2P networks, and it's worth pursuing.
My argument against this is just because it's purely peer-to-peer, it doesn't always directly mean it's an overall improvement in the user experience.
> You should be able to easily replicate and share content that you care about.
Yes there is definitely value in that, I even agree with there! But I personally think turning the whole thing into a completely peer-to-peer approach isn't the answer. It's the raw shit problems about bandwidth capability and network latency which really matters - which, if you've tried streaming from utorrent/bittorrent before, it's very obvious that the experience is still greatly affected by the lack of a little centralization that YouTube has.
> The big incentive is being able to keep your own copy and being able to do whatever you want with it. How many times have you tried searching for an old YouTube video, only to find it got taken down?
Neutral on this, very nuanced since most of the time it involves legal issues and opinionated perspectives, lol.
The lack of upstream bandwidth offered by cable providers almost seems deliberate. (I.e. a disinterest in improving the upstream capabilities of docsis standards, as well as offering pitiful upstream bandwidth in their plans.)
I think replicated personal data is great, but the author's point is you are probably going to get a lot more storage and bandwidth per dollar by replicating across centralized storage providers.
There may be some wins for the specific case of loading something from someone on the same LAN as you, but that's probably not something that can sustain a P2P storage market.
The obvious benefit of something like Filecoin is that you don't have to trust people to host your data. However vile it may seem to some, if they want to get paid they have to provably host it. And it's fragmented enough that you can't go after everyone who has only part of a series of forbidden bits.
I think there's a market for impossible to take down information. The competitors aren't the Amazons of the world, more like whomever is hosting Scihub.
Yeah, I think we can agree that there will probably be lots of interesting innovation in the distribution of illicit material. However, for an information technology to truly shape our culture, it needs a plausible path to go mainstream. None of the champions of current P2P crypto storage systems are claiming their focus will be on the distribution of said illicit materials. They all claim it will revolutionize transmission and storage of totally ordinary data, too.
I think the core assumption blockchain proponents make is that a lot of problems can be better solved if they rely on trustless sytems, and that this includes mainstream things.
I'm not sure if i agree with that, but there once was a time when the only people who would ever need a decentralized network were the military, and the benefits did generalize to everyone.
I don't think the author is arguing against the entire concept of P2P networks (though if I read correctly he does make an off hand comment saying they can be difficuly, which is probably true of anything), just the idea that cryptocurrency backed distributed storage is likely to be widely adopted and profitable.
I think the main thing that this misses is the potential benefit of an open protocol.
We still don't have a standard open equivalent of Dropbox.js (deprecated).
That is, a user controlled account which a browser or phone knows about that can store files, and that every app that needs to store files can read and write to.
Proprietary solutions and B2B solutions will never meet that need - it needs to be something that all the big tech companies can get behind, and which consumers have visibility and control of.
I can't see a route to such a protocol right now, in terms of either technology or finance, other than something open source and based on a cryptocurrency. As such, I hope one of them succeeds.
Even if all the files end up stored in Amazon, at least the protocol will be open and everything will be encrypted. And you will be able to trivially as a consumer store the data on a competitor (Microsoft, Google, Baidu), or on your own server if you want or need (countries and companies and people at risk of human rights violations may care).
git is popular because of many external factors, some of which (github gitlab, etc) are centralized closed systems. I don't think git by itself would have become popular. I'd say that in today's world, you need someone somewhere making mad cash for a protocol to become that popular. Could an open storage protocol achieve that?
I honestly don't see the need for it to be distributed. A zeroconf home file server that handled giving you access when you were out and about would solve the 80% problem.
It's not exactly what youre describing, but isnt e-Mail a decentralized file store? Sure, most mail is google, microsoft, or verizon by now, but still, you can spin up your own compatible instance and participate with others.
Almost any client you can dream of can access IMAP mailboxes and send through SMTP.
I don’t think it’s missing so much as being realistic about the costs of developing a production grade service.
Cryptocurrencies aren’t really up to that since it’s unlikely to lead to the kind of lottery returns VCs are chasing — the most people are going to pay is capped at Dropbox/iCloud/etc. rates — and the volatility makes it hard to sustain years of development to make something stable enough to trust and easy to use.
> But that's not the worst of it. Suppose P2 storage became profitable and started to take business from S3. Amazon's slow AI has an obvious response, it can run P2 peers for itself on the same infrastructure as it runs S3. With its vast economies of scale and extremely low cost of capital, P2-on-S3 would easily capture the bulk of the P2 market. It isn't just that, if successful, the P2 network would become centralized, it is that it would become centralized at Amazon!
That is exactly what I said about a year ago, although I predicted centralization on BackBlaze instead of S3 because they had lower costs: https://news.ycombinator.com/item?id=14988800
>No P2P storage networks have succeeded? What about torrents?
You have to look at the specific context his essay is about.
Torrents are popular for data that many participants desire such as "StarWars_1080p.mp4" and "LadyGaga_album.mp3".
Torrents do not work for a 500GB encrypted backup of "JohnDoe_personal_vacation_photos_20180621.zip". There's no financial incentive for others to download and store it so the number of seeeds/peers will be 0.
His essay is not about a p2p version of Netflix or Spotify (e.g. bittorrent or Napster). He's talking about a p2p version of "cloud storage" such as BackBlaze/AmazonS3/DropBox for storing data that may only be important to that user. That's why he's dissecting Filecoin and its economics.
Because torrents are not a storage network. Even ignoring the read-only aspect of it, there is no way to ensure that someone will download your torrent.
These assertions are descriptions of the current set of crypto P2P storage systems.
Peers only store one copy of your data, per peer. Of course, multiple peers will store copies of your data or the system could never work, but there isn't too much incentive for any one peer to store two copies of your data.
Price volatility for these systems is already a fact, and it's unlikely that any crypto could match AWS' historical price predictability of "monotonically decreasing".
The points made in this article remind me of how CD-ROM was adopted in the early nineties.
Once upon a time CD-ROM was new and with Windows for Workgroups you could share a given CD-ROM across a network. If you needed to install $SOFTWARE on machine B you could theoretically put the CD-ROM in machine A's drive and install from there. The person on machine A might even put the right disk in for you.
Given that CD-ROM drives cost a reasonable amount of money at the time and probably went at 3x speed, you would have a financial incentive to specify machine B without the CD-ROM, same with machine C and machine D. In so doing you could save enough money to buy machine E (or at least the monitor for it).
Nobody went down this route, in offices or in education. In education settings there were financial incentives to save money plus tech savvy staff on hand to make it work smoothly. There was no 'status' value in having the CD-ROM attached.
Everyone was okay about sharing the same printer but not the same CD-ROM drive.
You would not even need the CD-ROM as often as a printer in those days. The publishing industry was in its infancy and you only really had install media. Writeable media was a long way off. Surely it would have made sense to have your MS-Office install media or 'Encarta' available to all, on the network which was faster than the CD-ROM drives of the era?
Yet this never happened.
Instead every machine was specified with the CD-ROM. Disks were passed around much like floppies were. Nobody network-shared their CD-ROM drives. Yet all that was needed was a 'right click' and 'share'. The financial aspect in doing so was not a micro-payment in some alt-coin it was proper $$$ saved from not having to purchase another CD-ROM drive.
In time the $999 CD-ROM drives went to 4x and cost nearer $200, then they became 52x and cost nearer $100 to eventually become the $20 things bundled with every machine.
When writeable CD-ROMS and DVDs came along the storage space was vast compared to HDDs of the time yet still there was no network sharing of the things, everyone had their own disk in their own machine. Sure there were exceptions - network machines of the time - but these were the exceptions.
For the same reasons of capitalism and human nature I suspect that file-sharing-by-altcoin will fail. Moore's Law makes 'common sense' arguments for file sharing in a P2P way where you rent out your disk not make sense. Layer on top the fact that every alt-coin is a scam-coin and you know that P2P file-coins are only for the true believers.
I remember being in grade school, and we networked (shared) the drive on the one computer - only because the cd-rom drives on the other ones were broken and we wanted to play/ install Warcraft.
I disagree with his argument that for anyone to use P2P storage, it has to be less expensive than S3. It's true that this will be hard to do, for all the reasons he listed. But some amount of people will be willing to pay a premium for P2P storage, because it can't be censored or ordered to be deleted as easily as data stored on S3. Pirates, journalists, and activists will appreciate this use case.
There's also another economic reason for why people would pay extra to use P2P storage. I'm sure there's some sort of term for this, but many people have stacks of cryptocurrencies that they can't or don't want to exchange for their day-to-day currency. Maybe their government restricts exchanges, the funds came from a questionable source, they have some philosophical issue with "fiat" currency, or they're just avoiding taxes. If you need to store 50TB of data, and have $900 in your checking account and 400 ETH in your wallet, you'd probably choose P2P storage over S3.
"It's true that this will be hard to do, for all the reasons he listed. But some amount of people will be willing to pay a premium for P2P storage, because it can't be censored or ordered to be deleted as easily as data stored on S3."
Some will, but most won't.
"Pirates, journalists, and activists will appreciate this use case."
Journalists are probably the only group among those that would have the money to spend on it.
The Four Most Expensive Words in the English Language: "This time is different"
Anyway, on to the actual topic... I need to sit down to give this more thought, but I still think there's value in P2P networks, and it's worth pursuing. You should be able to easily replicate and share content that you care about. The big incentive is being able to keep your own copy and being able to do whatever you want with it. How many times have you tried searching for an old YouTube video, only to find it got taken down?
Another example: package managers! You want to have a machine-local cache, a remote cache for reproducible builds and deployments, and whatever other remotes you get the original packages from. Maybe different registries have varying content policies, and you can pick your favorites. If you have a few developers in an office it's a lot faster to grab dependencies from the laptop of the person sitting next to you than a CDN.