I chose Docker for my project with hope that it would help me create an easily reproducible development environment... And I'm not really sure if it was a good idea. There's so many moving parts and I can't name one that actually works well. Configuring storage is a hell (running out of disk space can lead to errors that are really difficult to debug), the dance with bridges makes configuring firewall a terrible experience. I definitely wouldn't recommend it as a stable container solution.
I totally agree. Docker feels really tidy conceptually. I ended up somehow convincing myself that adding this extra layer of technology into my (relatively tiny) infrastructure will decrease it's complexity. Unsurprisingly, the effect was the exact opposite.
I've been enjoying Nixos[0] and hope to try Nixops[1] soon. The difference between Nixos and docker is that nixos makes it possible to have a truly reproducible environment. See the surprisingly readable paper[2] for more details.
I feel the exact opposite! The Dockerfile is a simple way to define my environment and it's trivial to run the app in either development, QA, or production. Throw in some of the other functionality such as being able to deploy from an image rather than having to rebuild each time, and it's been a productive addition to my toolset. I'm guessing it depends on how complex your application is, but I've found it fantastic for running Python (Flask), PostgreSQL/PostGIS, MapServer, and some other smaller applications.
It's not an image in the sense that you can run updates/installs and strip out the crap associated with that but it's so trivial to run "images" that the benefits far outweigh the additional disk space for our use cases.
You might want to try out SmartOS: one .json file, vmadm(1M) and imgadm(1M) commands later, you'll have a fully isolated UNIX server, running at the speed of bare metal and residing on ZFS, protecting your data. If you need a more complex network setup, dladm(1M) will enable you to configure virtual switches and routers. The manual pages are really good, too. And there is nothing to install, as SmartOS runs from memory, so that all storage in the system can be used for virtual servers. Upgrades are a breeze by just booting into a new image, either from USB or PXE booting from the network.
I also didn't get the point of Docker until a few months ago when I got acquainted with Kubernetes (K8s) and Rancher.
It doesn't really make sense to use Docker without some sort of distributed scheduling/orchestration system like Kubernetes, Docker Swarm or Mesos.
I think the first few years of Docker were just pointless but I genuinely feel that it is all starting to come together now.
Docker 1.12 looks promising on the orchestration front - Hopefully they will back all these new features with thorough documentation (which I found lacking in Swarm when compared to K8s).
It looks like Swarm is just following in K8s' footsteps and maybe even improving things along the way.
Based on what I just read, it looks like Docker Swarm might finally be a serious contender to K8s.
From my experience with Docker one should just avoid docker-supplied volumes and manage the storage oneself using host bind mounts for containers and consistently using --read-only for containers themselves.
With that it is really weired to read all complains and issues about volumes management. It seems Docker just picked up wrong model for their storage. Then they try forever to fix impedance mismatch between that model and how people want to use Docker.
Care to clarify issues you have with volume management?
I think the biggest issue is around the `-v` syntax (and the corrosponding API).
In the UX we call things a volume which are actually a mount (which happen to use a volume).
I like the way we have implemented this in the new services API.... `docker run --mount type=volume,source=important_data,volume-driver=foo,volume-options=...`
For me the biggest issue with docker volumes is lack of any notion of subvolumes or volume grouping. Consider an application consisting of few containers communicating over unix sockets each with its own state. Surely I can create a bunch of volumes for the containers, but those would not have any notion of grouping. That complicates administration and development.
For example, it is very useful to have a shell view of all volumes for the application to poke inside or debug issues. Surely I can do docker run ubuntu and mount all relevant volumes under /mnt there, but that is not user friendly.
Or consider that during development or debugging I want to transfer the state to another machine. Why it is not possible to tar all application volumes with single command and restore them in a new place again with single command? I.e, something similar for docker save/load, but for volumes?
All those issues are trivially solvable if I use some directory on a file system for the whole application state and use subdirectories there for individual volumes for the containers and then pass those with -v to relevant containers.
check out weave for managing networking, IMO it's much easier and I'm not even talking about the docker plugin. just use weave without the plugin and it works amazingly well.
for the configuration aspect, just use dockerfiles and a system like mesos and marathon for deploying production.
you can then write simple restful API commands to launch different environments with marathon.
build all your docker containers to act as single server instances, i.e., spark on one container, SQL on another.
tie in a default install of cadvisor on all mesos clients and you can have easy monitoring as well.
I've been experimenting with Docker for reproducible dev environments as well for a while now, and my biggest issue has been with the app build step, which often requires a specific non-Linux platform (OS X for iOS and Windows for UWP).
Back in my previous project using Cordova, I made the Docker workflow work adequately by offloading builds to web-based build services like PhoneGap Build and Ionic Package (building only a thin live-reloading wrapper app that points to a configurable IP:port for development). But my new project uses React Native, and I haven't found any similar build services for React Native apps (or even just regular native apps) yet. Anyone have any suggestions?
Bitrise and Circle CI both support OS X environments for building projects. Combined with fastlane, you'd probably have a good setup for continuous delivery. I've only got experience with running fastlane locally though.
My experience with Docker is kind of ambivalent. We are using docker with docker compose on a legacy vmware stack in a corporate environment. No swarm and no docker overlay network as it does undermine firewalls. It is perfect to apply gitlow and semver to each build of each docker image, so each outcoming application stack is fully determenistic and auditable. Docker storage and network are really simple and docker is the perfect fit.
On the other side devs were asking for jenkins for that process. We run also Jenkins on Docker but it's cumbersome as it doesn't allow declarative configuration. There are dozen of plugins out there but it doesn't change the fact that the outcome is ressource hungry and slow. For now we are actually optimizing using Job-DSL plugin. We reached our target, but it was extremly time consuming and pain. For a new project I'm in, maybe we go to replace jenkins with concourseci or drone.
Imho a PaaS doesn't solve processual versioning and testing requirements. For that you still need further tools - with or without docker.
Why is storage an issue? With databases, for example, you store data upstream (on the host) and keep the dockerised app/thing mostly stateless, or at least capped with regards to storage. More details about your environment would be appreciated.
Keeping data on the host is exactly the reason that it's an issue. Swarm offers not guarantees that containers will be rescheduled on the same host that it was initially scheduled on and assumes that it can reschedule containers as failures arise... this doesn't work when your database gets rescheduled and emptied along the way.
In docker 1.12 you can 'tag' a node and tell a 'service' to only run on the tagged nodes.
Not saying that's a good idea, but it is getting closer.
You could, for example, have a node that's only for DBs that has volumes on it. You could then use DRBD on the host to clone that data to a secondary node. then in the event that node 1 dies swarm would bring the DB up on node2.
With the mesh network stuff they've added the endpoint would remain the same, so all your apps would need to do would be re-connect.
> Not saying that's a good idea, but it is getting closer.
No, that isn't getting closer it's getting farther away. The whole point of containers is that they make the host machine completely fungible. If I can only schedule my DB containers on a specific machine then I might as well just run my DB on that machine and be done with it.
My personal opinion is, Docker's good when you need to be sure you deploy exactly the same stuff on multiple hosts. This includes ensuring that the code you try on local machine would be the same on the production hosts. Docker is completely worthless for deploying singleton services that you already have packaging and deployment recipes for - it just doesn't add any value for such use cases.
That is, Docker is just a packaging system + a tool to spawn the deployed stuff in an isolated environment. Now, 3 years after, they've improved dependencies (previously done by docker-compose + hacks for using it in production).
That is, I use Docker only because its images are somewhat easier to build and deploy, compared to .deb packages or plain tarballs (and it's more OS-agnostic, since Docker Machine's available on Windows and OS X, so I can just build an image with dev. environment and don't care what OS is used). Doubt it's something more than this.
> My personal opinion is, Docker's good when you need to be sure you deploy exactly the same stuff on multiple hosts.
There is a technology for that already: it's called OS packaging. Docker does not maintain enough metadata to allow for in-place upgrades of individual components in an image (software and configuration lifecycle management). The best you can do with Docker is install lumps, you can forget upgrades. Docker is not a replacement for configuration management, and specifically, Docker is not a replacement for the operating system's software management subsystem.
Yes, you're right. I've mentioned that technology in the comment above. ;)
Docker is just a quick-and-dirty way to have a full system image. It's somewhat simpler than building a proper .deb (.rpm, .tar.xz, whatever one distro eats) package, especially when there's a ton of dependencies from other packaging systems (PyPI, Ruby gems, npm packages, etc.)
Oh, and unlike with many popular OS packaging systems, with Docker can actually have multiple versions of the same "package" "installed" at the same time (that's my biggest issue with dpkg), but IIRC there are no built-in provision for the migration scripts (prerm, postinst - this sort of stuff).
Containers are actually resource constraints, and were first introduced in Solaris 10, along with zones, which is what you appear to understand under containers. Even on GNU/Linux, containers are implemented via cgroups, which are resource controls and not a virtualization solution.
On SmartOS, when you provision a zone, you get a fully virtualized UNIX server, and you can apply a container to it by defining resource constraints, but that is both pointless and unnecessary there. Once you have a fully virtualized server provisioned from an image with the imgadm(1M) and vmadm(1M), it is only logical that you will want to service individual components via pkg_rm and pkg_add, rather than baking an entire image all over again, and redeploying it, all over again. It's the rule of modularity: "write simple parts connected by clean interfaces" [Kernighan-Plaguer], and it applies particularly well to lightweight virtualization.
I see how that was confusing. I meant that the fact that Swarm doesn't offer such a guarantee is the reason that storing the data on the host isn't a solution. My real complaint is that none of these give you a way to do persistence that doesn't destroy some other really nice properties of containers.
There's lots of people trying to fix this problem the right way, Kubernetes has made a lot of progress on having volumes follow the containers around from host to host. Torus is a slightly different approach, Flocker is another one.
> I don't think it's docker's responsibility to solve database clustering.
I don't know if I'd call it a responsibility. But Docker is obviously trying to expand their platform into more and more aspects of containerization. If they figured out persistence that would really set them apart, something that this product doesn't really do. It mostly just keeps them even with Kubernetes at best. And an imitation of it at worst.
Disagree...
Also disclaimer, I work at Docker, and particularly spend a large amount of time working on storage for the docker engine.
Volumes are exactly what you should be using to have data that lives beyond the life of the container.
Essentially, if you are writing to disk in your container, you should probably be writing it to a volume.
Volumes can be backed by any number of storage solutions, including from every cloud provider, various distributed block, gluster, ceph, etc...
Docker should _not_ be changing the way you run a database. If you need a highly available database, you don't use 1 database process and hope for the best, you use the built-in replication features available in just about every database product in existence today... database goes down, oh well you have a hot-standye ready.
> My real complaint is that none of these give you a way to do persistence that doesn't destroy some other really nice properties of containers.
100% agree with you there.
> It mostly just keeps them even with Kubernetes at best. And an imitation of it at worst.
Personally, when I looked at kubernetes, mesos, etc and saw the xml hell that i'd be living in and said "no way".
If i have to run something as complex as those to have docker then it's not worth it to me.
When I look at the new docker swarm stuff. For the first time I think that docker is a viable thing (assuming it all works). because I'm not adding additional complexity to get my docker simplicity.
Not sure why you are being downvoted. My sentiment is similar... Docker's historical advantage over other solutions is easy-to-use interface ( including Dockerfile). Can't wait to try swarm out.
This is very much the wrong way to approach storage (and sorry, for being so blunt, not sure how else to say it!).
Docker has support for a wide-array of storage solutions, everything from the various providers solutions (ebs, s3, gce storage, azure storage), to vsphere, netapp, scaleio, various distributed block solutions, nfs etc...
You should _not_ be changing the way you handle storage just because you are in a container. Use the tools you already have.
If you need HA on a database, use the built-in replication service available on just about every database product in existence.
If you really want distributed/shared storage, see the list above.
> If you need HA on a database, use the built-in replication service available on just about every database product in existence.
There are often valid reasons to not do this. For example, MySQL does not guarantee consistency between a source and its destination. mysqlrplsync can do this as a bandage, but it's something extra you need to set up and configure.
I do not think this is a reason to not use replication. In this sense, no distributed system guarantees consistency. If a node goes down (particularly a leader!) while trying to distribute a new update, you are very likely to run into issues.
> You should _not_ be changing the way you handle storage just because you are in a container.
You mean exactly like zones handle it inside of SmartOS (;-))
The ZFS storage pool is abstracted away from the zone the virtual server runs on, and since it is persistent, it's completely out of the way, JustWorks(SM).
> your database gets rescheduled and emptied along the way
That's bundling an CP application into seemingly AP infrastructure (from what I gleaned from their docs). Completely the wrong tool for the job. If the application was designed for that it would have recovered. In your defense, they have nothing to warn about that in their documentation.
This isn't really a CAP theorem problem and it's certainly not right to describe Docker as "AP infrastructure." What would that mean exactly? Would you expect this problem to go away if I used a database that sacrificed consistency for availability such as Riak?
Swarm (not the whole of Docker) seems like something designed for databases like Riak, yes. A SQL failover cluster might work, but certainly not a single database container on infrastructure designed for highly available apps.
why not run the database directly in the container as well? we're doing that with https://crate.io - just expose your local instance store as volume, crate will take care of the rest
Just based on the details mentioned in the keynote, the services API and swarm features look very similar to what's offered by Kubernetes... service discovery, rolling updates, load balancing, health checks, auto-healing, advanced scheduling.
I guess this was to be expected, but it's also kind of sad. I think it would have been a more strategic move to embrace Kubernetes instead of trying to compete with it.
The problem for docker is that just by building docker engine, they don't really have a way to make money – so stuff like this is probably all an attempt to actually have differentiating products that can generate revenue.
I don't think this is an attempt by Docker to sell tools to make money. They are improving their ecosystem to make their company more relevant which may or may not snowball into other companies actually paying for their services (Docker Cloud, etc).
Few companies are investing in making money by selling tools. Instead, you see Google/Amazon/Microsoft improving their cloud offerings and hoping you will your money there. They are giving away their tools (Kubernetes, Visual Studio, etc) hoping their environments become more attractive to you.
I hope we are not repeating history with the "DAB" the same way the container spec in the past. The industry is tired of these format fights and we need standardization to move things forward. I would put more faith in the "DAB" format if it was announced as something coming from the CNCF or similar. During the keynote the Atomic Nulecule [1] format came to mind... I don't know how it compares but it looks similar.
I should clarify: CoreOS + Kubernetes using rocket by default. From what I understand, CoreOS is driving the effort to get rocket supported by Kubernetes. K8S looks like it is going for agnostic implementation.
There is a log of emphasis going on in Kube right now in making sure that there is lots of room for experimentation around container formats, tools, and architectures - while at the same time, trying to keep Docker working well at scale and introducing Rktnetes. There's also work being done around runc support and VMs (from the hyper guys). Ultimately containers are just processes (some of them kernels), images are just another archive format, and the glue is what makes them interesting.
> I think it would have been a more strategic move to embrace Kubernetes instead of trying to compete with it.
Here's my theory.
Kubernetes is a product given away to entice you to buy a service from Google. (Edit: a better way to put this is that Google have nothing to lose and everything to gain from giving away Kubernetes, insofar as GCP has the best hosted Kubernetes offering).
Docker can't make money from a service, so they need to sell a product.
Disclaimer: I work for Pivotal, we donate the majority of engineering to the Cloud Foundry project. Insofar as Docker are moving back into PaaSes, we're competitors.
If people start running their apps on Kubernetes, even on AWS- Google has a shot at getting them over to GCE/GKE. On the other hand, apps that are tied tightly to AWS services are very difficult to move.
Have you ever tried running Kubernetes yourself ? Kubernetes is cool yes, but only on GCE, and managed by Google people. Apart from that, its nowhere near usable.
Docker 1.12 does just what Docker does best: Making complex stuff simple to use and give control back to people.
Or this nasty one[1] that has been around just over a year now. It didn't take me more work than using docker as a build server to run into this. There are some udev hacks that make it better, but the underlying issue is ultimately docker here.
Doesn't seem that docker inc upstream takes stability seriously from my standpoint.
Hard problems are hard. We are looking at all kinds of issues related to this, and in reality reports from that issue are likely caused by different things (some things are fixed, others obviously not).
There are several people looking how we can resolve issues like this.
This seems like a really exciting, albeit natural, evolution of the Docker "platform". It's really true that no one cares about containers, everyone just wants their apps to work.
That being said the one giant omission from Docker is still seems to be management of volumes/data. Great, we can run 100 nodes on AWS in a matter of minutes, but if your system has data storage requirements (ex: Almost every database ever) ... You're kinda left up to your own still. How does Docker Orchestration migrate the data volumes?
They really tried to sell this as "You don't need to do anything but the AWS beta, the DAB and a few commands" which would be wonderful. However with the need for reliable data storage... you're still stuck doing everything "the old fashion way".
(Edit: No, I don't mean store data IN the container, I presume no one is that silly. I meant the attached volumes. No volume mgmt = less greatly less helpful).
Docker relies on the already existing storage providers (ebs, gce, azure, netapp, emc, etc) to manage multi-host data.
Ultimately how do you manage data when it's not in a container? This is how you should be managing data inside the container as well, and docker provides an abstraction for doing this.
P.S. if you are at DockerCon, come to my talk on storage in Docker tomorrow @ 2:25
I guess the question is what do you do if you don't have one of those "existing storage providers" at your disposal. For example, I have a "traditional" collection of VPS instances hosting a variety of services. I want to containerize those services and run them on some kind of a cluster. My VPS provider doesn't offer anything besides local volumes. How do I manage persistent volumes such that they are created/destroyed as needed (say for scaling up/down database read-only slaves) and follow their containers around?
The answer to, "Ultimately how do you manage data when it's not in a container?" is, "Well, today, I spin up a new VPS and clone the database from the master (or a designated read-only slave) on first boot." Obviously that is not the pattern we want to pursue here.
It seems like the answer to the GP's question is that you have to build out the storage container cluster infrastructure in addition to your compute container cluster infrastructure, or move to a provider that offers both aaS. I don't know if that's GlusterFS or something else. This is where people like me need guidance, and on how to orchestrate storage containers with compute containers.
Yes, Docker does not provide storage infrastructure services. If all you are looking for is advice on how to handle storage, this is not a problem of the platform but rather education.
All someone can do is educate on the pros and cons of different solutions.
Docker containers are not meant to be persistent. Things like storage and state should be kept elsewhere and linked into your containers. You should be able to dynamically tear your containers down or spin up 20 of them and load balance requests between different ones.
If you are trying to store a MySQL database inside of a Docker container, you are missing the whole point.
> docker containers are not meant to be persistent. Things like storage and state should be kept elsewhere and linked into your containers.
This just means you need two systems to handle job creation, migration, replication, and load balancing. Everything you listed for docker containers is also needed for databases.
You need to remember: At scale, nothing is persistent. You need to notice a failure in your database systems,tear down the broken one, spin up a new instance, migrate the data from a replica, and keep on going.
It seems to me they are more focused on NOsql databases with multiple nodes and shared data like cassandra. So that you don't have to worry too much about volume/data mgmt because you will have multiple copies of the data on other containers when you lose a docker host or a container.
You also should conside that docker might not be the best option as opposed to just some vms on openstack for your much needed databases.
I'm stuck trying convince my co-workers that docker is not the answer to everything.
In no way did I mean to convey "store data IN the container" as I had simply assumed no one was that silly.
What I actually meant was as "there is no way to deal with volumes". So spinning up 100 instances and hot failover doesn't help if it doesn't deal with the attached volumes. Without that functionality, the orchestration is less magic and more "eh".
Don't use OpenStack. It's an appropriate solution never. I've been at too many shops where OpenStack is just a bottomless cesspool of the company hemorrhaging money to keep a shitty broken platform barely stable.
I'm not surprised the Kubernetes primitives got copied into this. I had stated before that Kubernetes was what Docker wants to be when it grows up -- and maybe that time is here. Having used Kubernetes in production, I don't know robust the Docker orchestration primitives are in comparison. But I'll probably find out soon.
The big advantage having a built-in Docker orchestration is that Kubernetes is painful to install from scratch. (Yes, there are scripts to help mitigate this; yes, GKE is effectively hosted Kubernetes). I'm involved in another Dockerization projection, but we don't know if we want to invest the time into setting up Kubernetes (though GKE is an option). This would be a good time to check things out.
Just to show how quickly things change. In 2015, I tried the docker compose, AWS ECS, and Kubernetes for production deployment, and of all of them, Kubernetes addressed the pain points in a practical way. The Kubernetes project ended in late 2015, and now six months later, things changed again ...
Having set up Kubernetes to test, I'd say it's a bit less complicated than many people claim, though still obviously complex. I had a two-node cluster up and running in an hour using Flannel and Kismatic's prepackaged binaries for Debian/Ubuntu [1].
The biggest hurdle, ironically, is the official documentation, which seems to have been written by someone with no understanding of explaining things from first principles, or any talent for organizing ifnroamtion. You'd think the "setup from scratch" guide [2] would be the most detailed and useful for someone who wants to see how each piece fits into the next one, but it's actually terrible. It skips a lot of implicit information and makes some odd assumptions/suggestions. Almost every part of Kubernetes is multiple choice, which affects later steps. For example, if you use Flannel and you're not on GCloud, much of what Google's docs say about networking is moot.
Everything is also made harder by the fact that Kubernetes is changing awfully fast in non-backwards-compatible ways, and chances are that if you google some config problem, the answer is long out of date — the necessary kubectl option no longer exists or whatever. A lot of config recipes are useless now.
I set up Kubernetes on AWS, a staging and a production cluster. Staging had 1 master, 3 worker nodes, Production has 3 masters, 5 worker nodes, with two compute classes (fixed and burst, so pods could be scheduled to what was needed). All of this was on CoreOS, and included independent etcd clusters. So all total, we're talking 18 AWS EC2 instances. I put live users on it, with real traffic and tinkered with it as I learned things from a production workflow.
This was back in mid-2015, which meant figuring out things like how to get overlay networking with AWS's routing, fiddling with AWS's IAM settings, VPC settings, figuring out how to split the nodes across availability zones. This included integrating with AWS AutoScalingGroups to bring up new CoreOS/K8S nodes whenever something goes down. I never got EBS working as a Kubernetes volume type (at the time, you have to add hackish tools into CoreOS to detect unformatted EBS blocks and format them when they get mounted; forget it), but fortunately, used AWS RDS for persistent stores. Persistent storage is something I look forward to trying on GKE.
I could have tried creating CloudFormation scripts but I was making my way through documentation, blog posts, and Github issues to pull it all together. It was worth it when I was done.
I didn't say it was complicated. I said it was a pain in the ass, and it's something that Kubernetes developers have acknowledged and are working on it.
I'd be interested to see what you think of the CoreOS "step by step" guide that we've honed over the past few months, trying to strike a balance of explaining the background while keeping it as brief as possible:
There are efforts underway to simplify cluster bring up, but this is not an easy problem because Kubernetes has to abstract away the underlying differences in O/S and cloud providers (example: load balancers on AWS vs. GCE).
I think they are getting there, but if you want dead simple "it just works", GKE is a great option.
I would love to see equivalent for Docker Toolkit, Docker for Mac, and Docker for Windows. It would make it easier to convince my team. I'll check out Minikube.
But yes, I know it is not an easy problem. I had looked at the automation involved as well as deployed Kubernetes by hand to AWS, having to translate what works on AWS.
And as I have mentioned in my comment, GKE is an option.
Having said that, on the desktop, I think all I really care about would be the equivalent of a docker-machine setup for Mac and Windows. Something that brings up a single-node CoreOS running stable, with networking working, and kubelet running via Docker. (Back in 2015, I had to rely on kubelet being baked into the CoreOS release). Probably not a big deal on Mac if I put a vagrant script, or use the existing ones that CoreOS has. But my teammates that are using Windows will want something that works well.
I'm curious -- what's the underlying distro being used for minikube?
I'll seed what will quickly be an enthusiastic crowd of spruikers for various platforms: have you looked at Cloud Foundry?
It's mature, has been scaled to 10,000 real app containers in a single install with a 250,000-container milestone coming soon, belongs to an independent foundation instead of a single company, runs on AWS, Azure, GCP, OpenStack, vSphere with more coming.
It handles routing, service injection, logging, container orchestration and a whole bunch of other bits and bobs. You can get started by just pushing your sourcecode, or you can provide container images, whichever works better for you.
We even made a version you can test drive on your laptop[0] and we've been working on a "bootloader" for spinning up a full AWS version quickly[1].
Disclaimer: by "we" I mean Pivotal, my employer, which donates the majority of engineering effort to Cloud Foundry.
Back when I looked at CloudFoundry (years ago now), it didn't work out because there was no middle ground deployment between a single PC and a full blown production deployment. You either ran it on one machine or you spun up what would have costed ~$1200 in servers at the time.
I can see your point, insofar at CF is biased to provide HA for all its foundational components.
It's actually possible to consolidate it down to fewer machines if you want. But it requires BOSH knowledge and the willingness to surrender the HA guarantees.
If you only need to deploy a handful of apps, it'd be easier to start with a hosted service and go from there. We have one in Pivotal Web Services[0], pricing is based on RAM usage. IBM have BlueMix, there are also service installations managed by CenturyLink, AnyNines and Swisscom.
BOSH knowledge isn't easy to come by unfortunately and I don't think the problem is HA for the foundational components so much as there being so many foundational components.
Look at a HA Kubernetes cluster. You've got kubelets and 2-3 masters and you're done. Look at CF and you need 22 servers total just to get started.
It may be easier/cheaper to use a hosted service compared to setting up your own CloudFoundry cluster but it's still extremely expensive and doesn't compete with alternatives like hosted Kubernetes. Say I want 32GB total of apps with 8 CPUs for a few development environments. For CF I pay ~$1600, for Pivotal I pay ~$700, for self hosted Kubernetes I pay ~$400 (4x8GB DigitalOcean instances + 3x2GB for masters) and for managed Kubernetes I pay ~$205 (4x n1-standard-2 Google Compute Engine instances + free cluster management since under 6 nodes).
I'm not sure it's apples to oranges, though. Out of the box Kubernetes provides some, but not all, of what Cloud Foundry provides.
When you say "so many foundational components", you're right: CF is featuresome and opinionated. That's the whole idea. With Kubernetes you can absolutely assemble all the other parts yourself and wire them together to taste, but that will chew up some of your 32Gb. And our calculations should expand to include engineering time, which is, last time I checked, quite expensive.
I'll definitely give Google due credit for competitive pricing though. For that price, and given their expertise, it'd be silly to self-host Kubernetes if you could use GCP.
One of the things I liked about CF was how it encoded application dependencies into the app package. Chef has just recently announced Habitat, which apparently brings that kind of an idea into play.
The thing is though, Kubernetes is a set of lego blocks from which you can compose into CF, but you couldn't create Kubernetes from CF. That makes Kubernetes more interesting to me.
I think that's a fair call. CF is an already assembled kit and we're constantly looking for ways to have better and better fit-and-finish. The whole point is to allow operators and developers to have smooth "Day 0", "Day 1" and "Day 2" experiences.
For those who really want a prebuilt Kubernete PaaS, Red Hat built OpenShift 3.
That said, Diego (our scheduler tech) was deliberately architected to be ignorant of Cloud Foundry. Right now it's pretty much serving the historical bias of Cloud Foundry's initial architecture -- applications -- but as the v3 API arrives we're breaking it down into finer grained concepts, some familiar to Kubernetes folk.
At one point, Diego was extracted out into a Kubernetes-like package with the Lattice project, which Pivotal supported for a while. But once we went and talked to developers, they said "well I don't really trust it because it's not a real Cloud Foundry". Hence we pivoted to PCFDev.
If Docker and Kubernetes or Mesos had emerged earlier, it's likely we'd have used them as our building blocks (as we've adopted other components). But when Cloud Foundry was born, none of these components really existed outside of private systems owned by Google or Facebook, or in some respects at Heroku.
Disclaimer: I was not responsible for any of these decisions, or even near to them, so take it all of the above with a grain of salt.
I am not the people who would make that decision, but nothing is impossible.
Like I said, it's a twist of fate. Red Hat decided to abandon their own effort when Kubernetes first emerged and I think they've done Kubernetes a lot of good in bringing their inhouse systems integration know-how to it.
Diego has a lot of properties that we really like, though. And it's already been proved at larger production scale than Kubernetes. Our next milestone is 250,000 real app containers per install, which is not far off.
Edit: Also I'm not sure how much re-engineering would be required. I kind of expect we'd just teach Diego to drive Kubernetes.
That mindset - being able to build all types of application on top of the "platform" - is why OpenShift rehosted itself on top of Kubernetes. Being able to compose building blocks and run higher level services (batch, application, clustered pets, 12-factor, legacy, etc) means you're much closer to the Goldilocks zone of Datacenter infra. I.e. avoiding needing to deploy multiple application stack platforms.
We saw the same gaps with Cloud Foundry -- the CF v2 API is resolutely about apps, the whole app and nothing but the app.
That's one of the reasons that the Diego effort was launched and it's why the v3 API is under development.
Red Hat has rebased on Kubernetes, but at the time we launched Diego, there was no alternative to building it ourselves.
I note that the workloads you mentioned are all supported or supportable by Diego and we're building out the missing ones very rapidly.
Disclaimer: I work for Pivotal, we donate the majority of engineering on Cloud Foundry. I am not near the Diego or core runtime teams, so my version of events may be hilariously inaccurate.
You should check out Rancher - they fully automate/orchestrate the installation of Kubernetes (along with Mesos, Swarm, or Cattle) and support some advanced features like blue/green deployments by integrating with the Kubernetes ingress feature: http://rancher.com/rancher-controller-for-the-kubernetes-ing...
Rancher Labs definitely seems to be further along the path of a production-ready container hosting platform than any other one I've tried so far.
I'm surprised I don't read more about Rancher qurg everyone focusing on k8s.
Not only can Rancher use k8s for scheduling, but mixing and matching environments with different scheduling technologies is extremely powerful instead of pigeon holing yourself on a one-size-fits all approach for you applications.
Kubernetes really stole Flynn's thunder. I wonder, if the Flynn team could travel back in time to 2013, what they would do differently knowing how things turned out.
We view the work of the Kubernetes (and Mesos, Docker Swarm, and other scheduler projects) team as incredibly valuable to Flynn.
We try to solve problems and create value at a the platform layer. Schedulers are one technology we use, but aren't at the core of the value that we (or most other platforms) deliver. We want to make development and management of production apps simple and invisible. For most users that should mean never needing to know about or interact with a scheduler directly.
We closely monitor progress on the major scheduler projects and look forward to a day when we can utilize one of them in Flynn instead of maintaining our own. As these technologies mature we'll be able to deliver even more value on top of them, while saving users from interacting with lower level components.
I'm interested in different schedulers. Is the Flynn scheduler documented anywhere? Anything you want to say about it and how it differs from the other offerings out there?
We've basically implemented the minimum viable system for our use case, and only add features as needed. The main thing that's missing right now is awareness of resource constraints and usage and then balancing based on that. There is a patch in progress that adds awareness of host-specific persistent volumes for our database appliances.
As an aside, when we started Flynn, the only open source scheduler that was available was Mesos and it didn't meet our criteria for integration so we ended up writing our own code. That code has since been rewritten approximately twice.
I have, wasn't interested. I'm more interested in Dies since it was rewritten against Kubernetes, and the same with Helm. Thanks for the suggestion though, I had not thought of Flynn in a while.
Cool! Looking forward to seeing what you guys got. We seem to have a Cambrian explosion of tools and approaches. It'll also be interesting to see who survives after the die-off.
At first I was worried that this is yet another Raft implementation, but it seems they're building on etcd's version [1] which should be a win for everyone trying to prove correctness for the various implementations.
I like it that etcd is self-contained and easy to troubleshoot, as opposed to a blackbox with tons of magic.
The demo for `docker swarm` was great but as an infrastructure engineer, it worries me. First because of the blackbox effect I mentioned above and secondly because I'm not sure how much flexibility is compromised to achiveve that.
Sure, it's awesome to quickly have a cluster up for your app and the user experience is phenomenal... but what about storage, network, firewalls, integrating with my linux distro, etc. Things are never this simple when they reach a certain scale and/or number of requirements deviate from those of a simple app.
From a quick look at the docs and watching the demo, Kubernetes still feels like a safer option, today. That being said, it's nice to see competition in this space. I guess this will put pressure on Kubernetes and others to improve the user experience further.
I wish docker released lite version of docker. Only containerisation, no networking configurations, weird cache mechanisms - just something small and easy to use locally.
Yes, if you don't look at rkt's for its minimal approach, then at least look at it for its ability to run containers without requiring a root service that runs all your containers under one process.
Oh gosh, no. Whatever faults Docker may have, they're outweighed by its very approachable UI compared to the discombobulation that is libvirt's interface.
The new orchestration looks based on Swarm and possibly Compose; have they resolved the issue where the lack of lifecycle semantics means that your services don't work if they have any sort of cluster semantics, like Riak, Cassandra or Zookeeper? These previously always required outside hacks to work, and work flawlessly with Kubernetes. It looks like there's some service discovery support; which may or may not fit the bill.
Due to caching implementations which change between OSes, programming languages, and routers, DNS is actually a really terrible discovery mechanism at scale.
For example, I'd never want to implement service discovery via DNS in Java, which ignores expiration times and caches queries according to its own settings. I've also seen some server configurations which result in every gethostbyname call to a fully recursive query against DNS.
> Due to caching implementations which change between OSes, programming languages, and routers, DNS is actually a really terrible discovery mechanism at scale.
Only if you rely on the client's resolver caching implementation, which we don't. Docker networking guarantees that a given hostname lookup by a given container in a given service discovery scope will always return the same IP. In that configuration, using DNS is a robust fit for production.
Believe it or not, Docker is built by people who have actual operational experience :)
> Docker networking guarantees that a given hostname lookup by a given container in a given service discovery scope will always return the same IP. In that configuration, using DNS is a robust fit for production.
I really wish there was more technical documentation on this "feature", which would make actual discussions about this much more productive, as opposed to what amounts to "trust me" with an appeal to authority.
As it stands now, there's not enough technical documentation about this feature available to make intelligent decisions about whether this really is a "robust fit for production".
> Believe it or not, Docker is built by people who have actual operational experience :)
I believe in track records, and Docker's hasn't been too great so far. From "not our problem" responses to issues like devicemapper version differences (on major OSes, no less), to the fact that I had to automate the cleanup of orphaned mounts after stopping containers, to non-backwards compatible releases with major core re-writes every other month, to containers which are quite literally unkillable without a host restart; Docker's track record is pretty poor.
I've used it in production, and the benefits barely outweighed the pain of attempting to manage it.
> a given hostname lookup by a given container in a given service discovery scope will always return the same IP
Did you mean this to cover the situation where N containers have registered the same name? I thought Docker DNS returned all such results in random order.
Given that, if one of those containers goes offline and Java has cached its address, you have the problem the parent was alluding to.
That's incorrect. You are describing DNS-based load-balancing, which would indeed rely on the container's resolver implementation. But Docker doesn't do that. Instead it always resolves the service name to the same IP, which is load-balanced by IPVS. That way even the world's crappiest dns caching implementation will still be handled properly.
So when I said that Docker doesn't rely on the container's DNS resolver, I really meant it. We have seen in past lives the consequences of "DNS abuse" and have been careful to avoid it.
Docker 1.12 built-in load-balancing supports both VIP based LB using IPVS and also DNS-RR and it is configurable per-service. VIP based LB is the default though. All of these will be fully documented shortly.
Not all use of DNS is the same. If you're attempting service discovery over DNS and need a timely response on changes, any form of caching behavior that does not conform to the specs is going to cause you a headaches when you need it to work most in production.
I find it super amusing that GCE isn't mentioned, that ostensibly Kubernetes is the source of lock-in (despite being open-source and likely having more cycles on EC2 than on GCE/GKE), and that this proudly uses gRPC for http/2 and protobuf goodness. So which is it, is Google an evil vendor trying to lock you in, or are we actually doing work in the open and just hoping you'll choose us when you want infrastructure?
Disclaimer: I work on Compute Engine, and think of the Kubernetes folks as friends.
You are reading way too much into this. We're introducing orchestration in Docker because it solves a problem for our users. We haven't called your employer or anyone else "evil", and frankly this announcement is not about Google or any other company. It's about improving Docker for its users.
Speaking personally, I think GCE is a great product, Docker wouldn't exist without Go, and although grpc is not my personal favorite, it gets the job done and it's quite popular with Docker engineers. The libcontainer project (now runc/containerd) started with a very fruitful collaboration between Docker and Google engineers. The current Docker networking model is also heavily based on early feedback from the Kubernetes team (one IP per container, remove nat between containers, etc.)
Yes, some features of Docker overlap with other products. One of those products is Kubernetes but there are dozens of others. That kind of healthy competition is normal and good for users!
I have two code bases that are microservices. I thought using Docker would be easy to setup and share a redis instance between these services. After spending a day trying various thing I just gave up and did 'brew install redis' and got back to real app dev. Docker isn't streamlined enough, does have great examples and is still changing lots. Sometimes doing it the old fashioned way is what lets you ship features faster.
I actually think this is a valid point, even if somewhat colloquially expressed. I had a lot of trouble getting past that menu bar on Opera Mini and even on an iPad, it's entirely too much.
The article is lots of blah blah plus "lock-in" for good measure, with exactly zero code or configuration examples, and it reads like a snake oil salesman brochure. I'm still skeptical about the purported advantage of Docker / Linux over zones / SmartOS.
I was using osx beta but uninstalled it for now because docker was eating all my space without recovering it after deleting containers.
Anybody using rkt for dev wants to share their experiences, I am thinking about trying it but would not invest time if it is not ready yet.
Sorry about that -- this was a bug in one of the betas where log rotation was broken. It's been fixed about 3 or 4 betas ago via a mid-week hotfix, so the latest open beta should be just fine. Hope you get a chance to try the latest series again soon!
