Make no mistake: Docker Inc has a lot of excellent engineers.
But there's also a landrush going on. Everyone has worked out that owning building blocks isn't where the money is. The money is in the platform. Businesses don't want to pay people to assemble a snowflake. They want a turnkey.
CoreOS, Docker and Red Hat are in the mix. So too my employers, Pivotal, who are (disclosure) the majority donors of engineering for Cloud Foundry. IBM is also betting on Cloud Foundry with BlueMix, GE with Predix, HPE with Helion and SAP with HANA Cloud Platform.
You're probably sick of me turning up in threads like these, resembling one of the beardier desert prophets, constantly muttering "Cloud Foundry, Cloud Foundry".
It's because we've already built the platform. I feel like Mugatu pointing this out over and over. We're there! No need to wait!
A distributed, in-place upgradeable, 12-factor oriented, containerising, log-draining, routing platform. The intellectual lovechild of Google and Heroku. Basically, it's like someone installed all the cool things (Docker, Kubernetes, some sort of distributed log system, a router, a service injection framework) for you. Done. Dusted. You can just focus on the apps and services you're building, because that's usually what end users actually care about.
And we know it works really well. We know of stable 10k app instance scale production instances that are running right now. That's real app instances, by the way: fully loaded, fully staged, fully logging, fully routed, fully service injected, fully distributed across execution cells. Real, critical-path business workloads. Front-page-of-the-WSJ-if-it-dies workloads.
Our next stretch goal is to benchmark 250k real app instances. If you need more than 250,000 copies of your apps and services running, then you probably have more engineers than we do. Though I guess you could probably stretch to running two copies of Cloud Foundry, if you really had to.
OK, a big downside: BOSH is the deployment and upgrade system. It's not very approachable, in the way that an Abrams main battle tank is less approachable than a Honda Civic (and for a similar reason). We're working on that.
The other downside: it's not sexy front-page-of-HN tech. We didn't use Docker, it didn't exist. Or Kubernetes, it didn't exist. We didn't use Terraform or CloudFormation ... they didn't exist.
Docker will get all this stuff right. I mean that sincerely. They've got too many smart engineers to fail to hammer it down. More to the point, Docker have an unassailable brand position. Not to be discounted. Microsoft regularly pulled did this kind of thing for decades and made mad, crazy cash money the whole way along.
@jacques_chester - after your last comment, I did check CF out. It is orders of magnitude harder than k8s for example.
To deploy CF on AWS - i need 20 instances. k8s can happen with a master and slave. I know that you probably do much much more.... but it will be nice to gradually scale out the components.
Second .. well BOSH. I see that there are 13K commits to the BOSH repo, so its probably set in stone anyway... but it would be nice to have something more standard (like Chef, Salt or Ansible. Ansible would be the closest thing to BOSH). These are extremely flexible tools... but far more accessible.
Third - the big one : Docker. I know that you have the Garden repository to run Docker.... but I dont think its production ready.
Cloud Foundry is an incredible piece of technology. But the problem is not that its unsexy... its inaccessible.
@sandGordon - Hey, I work for Pivotal and am the anchor of the Garden team.
Is there anything in particular that has you concerned about Garden? I see jacques mentioned we're moving (very nearly have moved!) everything over to runC backend which is certainly a big step in the right direction but I'd love to get a feel for what else you feel we need to get right before you would consider it "production ready".
Feel free to reach me here, in the #garden channel on the Cloud Foundry slack, or any other medium that works for you. Thanks.
hi @wlamartin - thanks for replying. It will be great if you can follow the (rather long) parallel thread on this topic. You should have a fairly good idea of how I personally look at this.
I wouldnt generalize that this is the problem that a small startup would always see ... but I think it is fairly close.
I did read it, and believe I understood, I just wasn't sure whether to apply your criticisms of CF as a whole to Garden individually, or whether there were additional distinct technical issues that you had concerns about.
The good news from a small startup/lone developer/tinkerer is that we have some work coming up in our backlog to make that much easier, including a single downloadable binary with sensible pre-configured defaults so people who don't want to wrestle with BOSH don't have to. (https://www.pivotaltracker.com/story/show/119174859)
While we grew to meet the containerization needs of CF first and foremost, I'd definitely love to see more projects like Concourse (https://concourse.ci) make use of Garden. Seeing first hand the dance they do to consume it, we definitely need to up our game if this is to be an attainable goal.
> I know that you probably do much much more.... but it will be nice to gradually scale out the components.
Agreed. There are intermediate options.
To try a full Cloud Foundry that fits into a laptop, you can install PCFDev[0].
As an intermediate position, you can install BOSH-lite in the cloud. You lose a major advantage of Cloud Foundry, though, which is that it's a distributed system which BOSH can keep alive.
Similarly, the default installation for Cloud Foundry uses 20 VMs. You can, if you wish, consolidate these to fewer. It's a matter of copying and pasting some YAML. But not recommended in the interests of robustness.
> but it would be nice to have something more standard (like Chef, Salt or Ansible. Ansible would be the closest thing to BOSH). These are extremely flexible tools... but far more accessible.
I have not used Ansible or Salt. I did spend time with cfengine, Puppet and Chef a few years back.
BOSH's concept of operations is a complete inversion of those tools. cfengine, Puppet or Chef take a system as it is and transform it, through declared goals, into the system you want it to be.
This actually means they have to do with a great deal of incidental complexity. Different distros, different versions of distros, different packaging tools and so on and so forth.
The BOSH paradigm is: wait, why are you trying to automate the construction of pets? You want cattle. So it starts with a fixed OS base image and installs packages in a fixed, non-negotiable way. At the low level, very inflexible. But it allows BOSH to focus on the business of installing, upgrading, repairing and reconfiguring distributed systems as single systems. The single distributed system is where BOSH starts, not where it was extended to. It's an important distinction.
> Docker. I know that you have the Garden repository to run Docker.... but I dont think its production ready.
Strictly, Garden can run Docker images. It's an API with driver backends. The original Garden-Linux backend is being replaced by one which uses runC instead. No point replicating engineering effort if we can share it with Docker.
But the problem is not that its unsexy... its inaccessible.
You're right. We're in the classic position of other heavy, robust tech. Hard to tinker with. I've been jawboning anyone inside Pivotal about this for years. We're making headway -- PCFDev is a start, bosh-bootloader is under active development and BOSH itself is going to get major love.
But compared to docker swarm or k8s, it's harder to set up an experimental instance.
You consistently respond to this style of CF critique by enumerating alternatives and workarounds to the individual points. But you're missing the forest for the trees. CF is by its nature too complex! No amount of justification of the complexity will fix that; no number of afterthought scripts will make up for it. The future of this space is scaling 1-dev, 1-node systems up to production, not in asking devs to wrap their minds and laptops around intrinsically huge platforms like CF or OpenStack.
I'm not saying CF is dead, because I'm sure you've got lots of paying customers, and lots of interest in larger enterprises. But I think it's pretty obvious the way the winds are blowing, and I think the smart money is on Kubernetes.
By design, developers are not meant to have to care about how Cloud Foundry works. It's meant to be a magic carpet. You send your code and pow, it works.
You mention afterthought scripts -- how many blog posts do you read about people who've rolled their own 1/3rd of a platform on top of Docker + various other components? It's fun, it's easy to start, and pretty quickly you're married to a snowflake for which you carry all the engineering load.
It's not that I don't see the forest for the trees -- I do, internally I am constantly saying that developer mindshare predicts the 5-10 year performance of a technology and we suck at it.
But, at some level, we're talking about different forests.
> By design, developers are not meant to have to care about how Cloud Foundry works.
Yes, that's exactly the problem. At small scale -- which is where the vast majority of the market is, to be clear -- there is no difference between "the person who is responsible for the platform" and "the person who deploys code onto the platform". Platforms need to optimize for making both of these tasks easy. Docker understood this from day 1 and are backfilling technical competence. Kubernetes figured this out a bit late but are now rushing to make the adminstrata easier. CF and OS seem stuck in maintaining the separation, which keeps them out of the running.
> how many blog posts do you read about people who've rolled their own 1/3rd of a platform on top of Docker + various other components? It's fun, it's easy to start, and pretty quickly you're married to a snowflake for which you carry all the engineering load.
And doing so is stilleasier than bearing the cognitive burden of learning CF!
I agree that being swallowed up from the bottom is our main risk. Which is why, above, I said I talk about this a lot internally.
> And doing so is still easier than bearing the cognitive burden of learning CF!
Right, the incremental cost seems small. A bit here, a bit there. And actually, it's a lot of fun to roll your own. You know it, you can push it in any direction and so on.
The flipside, and this is where Pivotal and other CF distributors are making money, is that it turns into a nightmare pretty quickly.
A lot of big companies are perfectly happy to pay someone else to build their platform.
Before I transferred to Cloud R&D, I was in Pivotal Labs. I got to see several hand-rolled platforms. Some of them were terrible. Some were ingenious. And all of them were millstones. That experience is pretty much what turned me into the annoying pest I am now.
We're at an interesting moment. Right now, writing your own platform seems reasonable -- even attractive -- to lots of people. In maybe 5-10 years, it won't be a thing that many people do any more. These days, for most projects, most engineers don't consider first writing an OS, or a database, or a web server, or a programming language, or an ORM, or a graphics library and on and on. Those are all well-served. Once upon a time it was a competitive advantage to roll your own; these days it just doesn't enter anyone's thinking.
There will be between 1 and 3 platforms that everyone has settled on, with a handful of outliers that people tinker with. My expectation is that Kubernetes will slowly morph from an orchestrator and be progressively extended into a full platform. Docker are clearly doing likewise. I think Cloud Foundry will be one of the three, simply because it's the first and most mature full platform already available for big companies to deploy.
Doesn't mean we can't have some of our lunch eaten.
> The BOSH paradigm is: wait, why are you trying to automate the construction of pets? You want cattle. So it starts with a fixed OS base image and installs packages in a fixed, non-negotiable way. At the low level, very inflexible. But it allows BOSH to focus on the business of installing, upgrading, repairing and reconfiguring distributed systems as single systems. The single distributed system is where BOSH starts, not where it was extended to. It's an important distinction.
This is a problem compared to how we are reasoning about systems in the Docker world.
You either use an orchestration tool - which doesnt care what the base system is, and only cares whether it is discoverable/alive/dead/etc - or you use a construction tool . The construction tool could be a Dockerfile .. which you massage in shape for the particular OS/config you have... or you use ansible/salt to "transform" the system.
For example, k8s uses Salt for orchestration and Docker for construction.
BOSH is a merger of both.. and is therefore extremely, extremely opinionated and incestuous. If I may overstep the limits of my intelligence - I would say, it is a product of times when Docker was not invented yet.
It is also very hard to work with as a consequence. Just wanted to let you know that.
> If I may overstep the limits of my intelligence - I would say, it is a product of times when Docker was not invented yet.
Well, it was, so that makes sense.
I actually said ... a year ago now, I think? ... that tools like Puppet and Chef would morph from "discipline my pets" into straight up image builders. They're so much nicer than Dockerfiles -- the latter is severely constrained by the nature stackable filesystems.
> It is also very hard to work with as a consequence. Just wanted to let you know that.
Out of curiousity, where do you see the seams to cut it open?
> tools like Puppet and Chef would morph from "discipline my pets" into straight up image builders
Maybe, but if you're running something like Kubernetes, which I genuinely think is the future, "images" stop having any sort of relevance to this problem space.
If you've ever written a dockerfile, as I'm sure you have, you'll know that the recipes for individual applications' setup become so simple that problems that Puppet, Chef etc. exist to solve just melt away. Most of my dockerfiles are just a RUN line or two. A tool like Puppet, much of which is architected around the issue of mutable state, would just get in the way when your build is immutable.
As for the host, containers don't care what kind of host environment they're being scheduled in, and so the recipes for building Kubernetes hosts also become a lot simpler.
I think Docker or runC is pretty much mandatory. Not because it is great tech or anything.. but because of https://hub.docker.com/explore/
So BOSH needs to work with my Dockerfile. That's the first step.
Second, build a minimal cluster (1 node?) for me with my Dockerfile. All that the cluster is doing is restarting my application if it dies and helping me deploy. And let me scale out gradually from there.
FWIW, I totally bailed on writing a provisioning system once you clued me into BOSH. It's the right answer, as near as I can tell.
I don't currently use it, because I am in a single-cloud environment and CloudFormation/Chef make more sense (plus, as you note, it's not currently super friendly), but I fully expect, should the progress that I've been seeing continue, I'll be using it in the next couple years.
One of the things I like about K8S is that is serves as the building block for creating a PAAS. I built a small tool on top of K8S for a production run. Deis is doing something similar (and far more ambitious), though they are also creating packages of abstractions called Helm. K8S is flexible and extensible because it doesn't try to be a Heroku, more of a framework for building your own Heroku.
Is there value-added in someone providing the higher level pieces? I bet there is. The flip side is, why did Kubernetes gain so much traction so fast, in a way that Cloud Foundery hasn't?
And if Deis and Helm overtakes Cloud Foundry? (They dropped what they had in order to build everything back from K8S) Maybe there is something CF folks are not seeing about this.
> K8S is flexible and extensible because it doesn't try to be a Heroku, more of a framework for building your own Heroku.
Agreed. Cloud Foundry is not a "competitor" to Kubernetes, in that sense. Diego is the Kubernetes analogy. The Cloud Foundry internals have become progressively more finely abstracted and teased apart to make it possible to swap components, so you never know.
That said, for those who want Docker + Kubernetes bundled into a turnkey PaaS, Red Hat went all-in on those two techs for OpenShift 3.
CF is a complete PaaS, isn't it? So it's not directly comparable to Docker nor Kubernetes.
From my perspective, CF seems far too large, monolithic (as in all-or-nothing) and opinionated, and honestly a bit heavy-handed and antiquated (the last two partly due to BOSH).
What's attractive about K8s in particular is the elegant, layered, modular, unopinionated design. K8s itself isn't simple, but it's quite small and lightweight, and its core principles are easy to understand; more importantly, it offers primitives without imposing a particular framework on anything (for example, it doesn't care about how the underlying host is configured or provisioned), which makes it a superb foundation for a higher-level PaaS, once someone designs that (maybe Deis).
We've already released a PaaS built ontop of Kubernetes at Red Hat, with the upstream named OpenShift Origin and the supported version named OpenShift Container Platform (https://www.openshift.com/container-platform/).
Our developers contribute a ton of work to the Kubernetes project, and pretty much everything that works on Kubernetes will also work on OpenShift. We do have some security settings by default that are more restrictive (like no containers running as root) but that can be easily modified to be more permissive.
But there's also a landrush going on. Everyone has worked out that owning building blocks isn't where the money is. The money is in the platform. Businesses don't want to pay people to assemble a snowflake. They want a turnkey.
CoreOS, Docker and Red Hat are in the mix. So too my employers, Pivotal, who are (disclosure) the majority donors of engineering for Cloud Foundry. IBM is also betting on Cloud Foundry with BlueMix, GE with Predix, HPE with Helion and SAP with HANA Cloud Platform.
You're probably sick of me turning up in threads like these, resembling one of the beardier desert prophets, constantly muttering "Cloud Foundry, Cloud Foundry".
It's because we've already built the platform. I feel like Mugatu pointing this out over and over. We're there! No need to wait!
A distributed, in-place upgradeable, 12-factor oriented, containerising, log-draining, routing platform. The intellectual lovechild of Google and Heroku. Basically, it's like someone installed all the cool things (Docker, Kubernetes, some sort of distributed log system, a router, a service injection framework) for you. Done. Dusted. You can just focus on the apps and services you're building, because that's usually what end users actually care about.
And we know it works really well. We know of stable 10k app instance scale production instances that are running right now. That's real app instances, by the way: fully loaded, fully staged, fully logging, fully routed, fully service injected, fully distributed across execution cells. Real, critical-path business workloads. Front-page-of-the-WSJ-if-it-dies workloads.
Our next stretch goal is to benchmark 250k real app instances. If you need more than 250,000 copies of your apps and services running, then you probably have more engineers than we do. Though I guess you could probably stretch to running two copies of Cloud Foundry, if you really had to.
OK, a big downside: BOSH is the deployment and upgrade system. It's not very approachable, in the way that an Abrams main battle tank is less approachable than a Honda Civic (and for a similar reason). We're working on that.
The other downside: it's not sexy front-page-of-HN tech. We didn't use Docker, it didn't exist. Or Kubernetes, it didn't exist. We didn't use Terraform or CloudFormation ... they didn't exist.
Docker will get all this stuff right. I mean that sincerely. They've got too many smart engineers to fail to hammer it down. More to the point, Docker have an unassailable brand position. Not to be discounted. Microsoft regularly pulled did this kind of thing for decades and made mad, crazy cash money the whole way along.