I never understood the appeal of service meshes. Half of their reason to exist is covered by vanilla kubernetes, the rest is inter-node VPN (e.g. wireguard) and tracing (cilium hubble). Unless I’m missing something encrypting intra-node traffic is pretty silly.
K8S has service routing rules, network policies, access policies, and can be extended up the wazoo with whatever CNI you choose.
It’s similar to Helm, in that Helm puts a DSL (values.yaml) on top of a DSL (go templates) on top of a DSL (k8s yaml), just that it is routing, authentication, and encryption on top.. well, routing (service route keys), authentication (netpols), and encryption.
I've worked on several k8s clusters professionally but only a few that used a service mesh, Istio mainly. I'll give you the promise first, then the reality.
The promise is that all of the inter-app communication channels are fully instrumented for you. Four things mainly 1) mTLS between the pods 2) Network resilience machinery (rate limiting, timeouts, retries, circuit breakers). 3) Fine grained traffic routing/splitting/shifting. And 4) telemetry with a huge ecosystem of integrated visualization apps.
Arguably, in any reasonably large application, you're going to need all of these eventually. The core idea behind the service mesh is that you don't need to implement any of this yourself. And you certainly don't want to duplicate all of this in each of your dozens of microservices! The service mesh can do all the non-differentiated work. Your services can focus on their core competency. Nice story, right?
In reality, it's a little different. Istio is a resource hog (I've evaluated Linkerd which is slightly less heavy weight but still). Rule of thumb: For every node with 8 CPUs, expect your service mesh to consume at least a CPU. If you're using smaller nodes on smaller clusters, the overhead is absurd. After setting up your k8s cluster + service mesh, you might not have room for your app.
Second, as you mention, k8s has evolved. And much of this can be done, or even done better, in k8s directly. Or by using a thinner proxy layer to only do a handful of service-mesh-like tasks.
Third, do you really need all that? Like I said, eventually you probably do if you get huge. But a service mesh seems like buying a gigantic family bus just in case you happen to have a few dozen kids.
One major usage of services meshes that I’ve come across is for the transparent L7 ALB. gRPC, which is now very common, uses long-running connections to multiplex messages. This breaks load-balancing because new gRPC calls, within a single connection, will not be distributed automatically to new endpoints. There is the option of DNS polling, but DNS caching can interfere. So, the L7 service mesh proxy is used to load balance the gRPC calls without modification of services.
Look, back in the day, things weren't encrypted, so you could listen in on your neighbor's phone calls, read their email, hack their bank accounts. Wireshark and etherdump and the most fun of all, driftnet. So, since then, everything has to be encrypted, lest someone hack there way to the family jewels. Never mind that the number of breaks to get there means there are usually bigger fish to fry. The important thing is to sprinkle magic encryption dust on everything because then we know it's Very Secure. (That's not to deride the fact that encryption is important, because it is, but sometimes it goes a bit far when there are other gaping holes that should be patched first.)
Usually, unless someone is really doing naive things, you will need to have access to a lot of almost physical things to sniff traffic. You almost need to physically have access to room where either the server or the client is, even with unencrypted traffic. People say; 'but they can sniff it at level3'; they sure can, IF they have actual access to level3 on a higher level than just using them for normal traffic. Hacked switch or router or so. Probably state actors can and do pull that off, but outside that, it's really not so easy to get to unencrypted traffic of just a random target. You still should encrypt things of course when you can, but you don't have to get quite that paranoid about it.
All major hacks are 0-days (well, not updated Wordpress is not necessarily 0-day; a lot of 0-days are exploited months or years later), stolen credentials (social engineering usually), brute force password hacks or applications that are left open (root/root for mysql with 3306 open to the world). Those have nothing to do with (un)encrypted traffic.
if you have the ability to execute code on a CPU, and that CPU is connected to a bus, and that bus is connected to a network card, you can sniff traffic. If you have data and business processes that include at least one entity A that lacks absolute trust at least one other entity B in your cluster, then the visible traffic of A by B is bad.
Yes, but if you know that I run unencrypted traffic on my network and if I tell you that, you still won't be able to get to any of that if you cannot get into our network. Even if I tell you that I host at provider X and the traffic is unencrypted until it hits our webserver, you still won't be able to sniff any of it without getting very intimate with someone who has deeper access. Just hiring a machine at the same provider and putting the card in promiscuous mode is not going to get you anything from us.
Service Meshes are something necessary for a small portion of Fortune 500s which have 1000s of microservices. Sure you could use load balancers but it becomes cost efficient to move towards a client-side load balancer.
If you aren't a Google, Apple, Microsoft, ...etc scale company than a service mesh might be a tad overkill
You're close, but it's really when you have thousands of microservices using either shitty languages or shitty client RPC libraries where you can't easily perform client-side load balancing.
There are plenty of languages and RPC frameworks where you can solve this without resorting to a service mesh.
Practically, and to your point, service meshes solve an organizational problem, not a technical one.
I don't get this either. Doesn't the mesh become an scalability bottleneck just like load balancers?
On that scale I'd expect people to use client-selected replicated services (like SMTP), and never something that centralizes connections (no matter where it's close to).
You can always add observability at the endpoints. Unless your infrastructure is very unusual (like some random part of it costing millions of times more for no good reason, as on the cloud), this is not a big challenge; you add it to the frameworks your people use. (I mean, people don't go and design an entire service with whatever random components they pick, or do they?)
With Istio (envoy) you run a "sidecar" container in your pods which handles the "mesh" traffic, so it scales with the number of instances of your pods.
kube-proxy operates at L3/L4 while service meshes generally operate at L7 so it can load balance on a per HTTP request basis. Particularly useful for long lived connections commonly used in gRPC and others.
I actually never understood the appeal of Kubernetes in the first place. I have production apps running on bare bones VMs serving millions of customers. Is this sort of complexity really necessary? At this point I would just consider serverless options. Sure, they would be a little more expensive, but that's a huge savings if we account for engineering teams' time.
Counter-take: I never understood the appeal of virtualizing the hardware. Is that complexity really necessary?
Of course there's tradeoffs, but I think it's a specific perspective that says that Kubernetes is any more complex than virtualizing the hardware and scheduling multiple VMs across real hardware.
I personally am of the opinion that you only really need one of these. Containers don't really need VMs and vice versa to get all of the benefits of abstracting away the fact that your hardware might break. It's just a choice of which abstraction you prefer to operate at (of course, clouds will put your containers in VMs anyway because they need that abstraction). It sort of sounds like the parent prefers the VM abstraction, and you prefer the container abstraction.
You didn't respond with any benefits of k8s though. What's the true value-add? Surely, coding an entire infrastructure in YAML is not it because that's horrific for anyone who wrote actually working software (one that didn't need 20+ commits of "try again" for a single feature to start working anyway).
I find it hard to believe that you've considered this for more than than two seconds and can't think of a single reason why k8s might be a good fit for someone's requirements. But here's one:
It's a curated, extensible API that provides a decent abstraction over a heterogeneous collection of hardware. Nobody's done that before, and it's extraordinarily useful for being able to define intent.
No-one forces you to use YAML, it's just a serialisation format.
Once you have a bunch of components that implement this API, it becomes trivial to deploy pretty much any level of complexity of containerised application, without having to care too much about the actual exact details of how scheduling, networking, storage etc. is implemented. Even better, I can hit two different clusters configured in two completely different ways with the same manifest and get roughly the same result.
> I find it hard to believe that you've considered this for more than than two seconds and can't think of a single reason why k8s might be a good fit for someone's requirements.
That could also tell you that you're in a bubble of converts and have forgotten what it's like to live without k8s. ;) There are always at least two sides of the coin.
> No-one forces you to use YAML, it's just a serialisation format.
Really? So when do I get to describe my singular app that needs Postgres and Kafka in 7-10 lines in a .txt file and not {checks our company's ArgoCD backend repo} ... not 8 YAML files? I ain't got all day or a week, where is that? Nice declarative MINIMAL syntax with all the BS inferred for me (like names and stuff, why should I think of 5-10 names of stuff? Figure it out, automated tool!) that only concentrates on what is being deployed. It can generate everything else just fine, or at least it should, in a more sane world anyway.
Excuse the slightly combative tone, I ain't trolling you here but you also come across as a bit blind about things outside the k8s holy land.
> Once you have a bunch of components that implement this API, it becomes trivial to deploy pretty much any level of complexity
Nope, nothing ever becomes trivial with k8s. I was on call with two senior platform engineers and we needed 3-4 hours to describe a single small app that needs Postgres and Kafka and to listen on a single port while needing a single env var and a few super small config files. These guys provisioned an entire network of 500+ pods working perfectly for years. They made 10+ mistakes while trying to help me deploy this small app on Argo. (And they've done the same dozens of times at this point.)
Do with that info what you will but I'd strongly disagree if your takeaway is "they are not that good" -- because they have done quite a lot very successfully (with and on k8s).
> It's the abstraction.
My 22+ years of programming have taught me that people enjoy abstractions too much and make huge messes with them. I am not convinced that having an "abstraction" at all is even a good selling point anymore.
You originally said "I don't see how it's useful", and I observed that some people find it useful.
I'm sorry you've had a bad experience, but it's a bit short-sighted to extrapolate from that to "this is universally useless".
> you also come across as a bit blind about things outside the k8s holy land.
You're not the only one with multiple decades of experience in writing code and managing infrastructure. I've used a lot of the tools in the toolbox, I know when each is likely to be more or less appropriate.
Ah, sorry if I sounded like saying it's universally useless. I know it helps people and more seriously speaking I'm aware of most of what it abstracts. But from where I'm standing it didn't do it well. Top much YAML, and I have to hand-hold it too much. It absolutely can do better at inferring or generating names / IDs, for example.
> Really? So when do I get to describe my singular app that needs Postgres and Kafka in 7-10 lines in a .txt file
Could you post those 7-10 lines needed to fully manage a Postgres AND Kafka deployment? I'm by no means a master, but I have a decent amount of experience outside the "k8s holy land" and I have no idea how to accomplish that.
We didn't need anything fancy at all, 99% can just be the defaults. That's what I meant. The other 1% is basically: user, password, port, topic name, replicas, partitions. Could have been two lines, namely two URLs.
No backups? Never do major version upgrades? Neither of these things are covered by the Postgres defaults. And even without them, I don't think I can get Ansible to install Postgres in < 10 lines. Yet all of these are covered in my ~30 line YAML running Postgres on Kubernetes. Plus of course recovering from pod and node failure, load balancing and more.
I think for most places, most of the time Kubernetes is overkill. Cloud in general isn't great bang for the buck. But speaking to your anecdote, having actually deployed these things in the past 3-4 hours to deploy a custom application, AND Postgres AND Kafka is a pretty compelling use case FOR Kubernetes. It would certainly take me a lot longer doing a proper job of it managing the system directly or using something like Ansible.
I as a dev should not waste days on this -- something I have no experience in. I can add the basic service / ingress / whatever and move on with life.
The infra / platform guys can add HA and backups later, right?
> I think for most places, most of the time Kubernetes is overkill.
Then we agree. I am not sure this app in particular was a good fit for k8s though; having it on a DO droplet + managed Postgres + managed Kafka would have taken me 1 hour, 40 minutes of which would be me cursing because I forgot to put a host name and port combo somewhere. :D
> I think for most places, most of the time Kubernetes is overkill.
True, but as you yourself said, backups and high availability are much harder on bare metal -- for devs anyway. I can do an amazing job at it because I do it at home for my stuff buuuuuuuut, it's going to take me days, maybe even two work weeks. Not a good time investment.
That's why I am resenting it when I have to do it: it's wasting my time with something that I never learn properly so I have to relearn it every time from scratch because I can't engrave it in my memory and that is because I never do it for long enough stretches of time to indeed remember it.
I already complained to my manager about it and he took it seriously, by the way. I am not raging at you or others about a deficient process in my employer's company (that I plan to help fixing, or at least mitigate somewhat). I am displeased with the fact how everyone just settled on a very, VERY low maxima -- and somehow nobody wants to rock the boat.
We can do better. We should do better. But alas, guys like myself have worked 22+ years on stuff they don't love -- and I already have burnout and hate working programming for money, which is another tragedy entirely -- and will NEVER EVER get the chance to change anything at all because I have no choice but work for the man... for now. Though at 40+ I am not sure I have the strength to manage to somehow command huge commissions for consulting or w/e else. Anyhow, off-topic.
Not the poster you're replying to, but when it comes to deploying your own applications: generating Kubernetes manifests with whatever language you're already using and feeding JSON to `kubectl apply -f -` can accomplish the same outcome with less effort.
Helm is still useful for consuming 3rd party charts, but IMO it's status as the "default" is more due to inertia more than good design.
In my own experience I started off doing this because I wasn't ready to learn helm. However, after using helm once I didn't see the reason to do it in my own code any more.
> Infrastructure teams can usually implement a feature faster than every app team in a company, so this tends to get solved by them.
Well, that's comparing apples to oranges. Product teams have completely different goals, e.g. adoption/retention/engagement, so naturally internal cluster encryption is so far out of scope that in fact only the platform team can reasonably implement it. I don't see how that statement is relevant. You don't send an electrician to build a brick wall
Application security should be everyone's responsibility. Architects, developers, and operations.
Too many times have I seen architects and developers completely ignore it to make their jobs easier, leaving it to operations/infrastructure to implement. It's easy to twist the arm of business people with a "I can't ship feature X if you want me to look at security Y".
If everyone took this seriously perhaps we would have fewer issues.
I agree, I was just making a point that different teams have different priorities and thus different scope. Saying "PodA can only talk to PodB over mTLS" is very different to "Users need to login using oauth". Who is going to build the product if product team is working on the service mesh?
you can implement mtls (and almost all the other service mesh features) without service meshes, and it's usually better because of lower overhead, less total complexity (see for example the fat client libraries in use by google, netflix, etc.). but people don't want to think about this so leave it to infra teams to plaster a service mesh over everything.
You can, but it's absolutely a pain in the neck. Services need to load the certs from the filesystem on boot-up and trust the certs provided by other services. To manage trust, you need a certificate authority. Now you need to load the certificate authority's cert, and you need to manage rotation of certs. You need to help developers set up laptop-local certificate authorities and get them to issue certs so that you have Dev/Prod parity. You need to ensure that developers are enforcing modern ciphersuites, not doing bullshit "insecure-skip-verify" kind of toggles that make their jobs easier (because remember, their job isn't security, it's shipping features), not accepting self-signed certs or other certs not signed by the certificate authority. You need to make sure all this stuff is put in the testing suite to make sure it keeps getting maintained, and you need the files for these tests marked in CODEOWNERS to be under InfoSec control to ensure nobody rips them out just because they're inconvenient. And you need to copy this for every single service you run in production and every single development team.
You know what else you can do? Write your own web server (/sarcasm). I mean, who needs nginx? Probably writing your own will have lower overhead and less total complexity, not running a bunch of features that you don't use. And probably it will not be anywhere close to as good as a battle-hardened web server used by millions of engineers that gets regular support.
Personally I think it's debatable whether services really need mTLS within a private network. It's mostly a question of what scale you're running at; probably there's higher benefit-to-effort-ratio InfoSec projects to tackle. But if you do decide you need it, unless you can prove that the overhead is unworkable for your requirements, really you need to bite the bullet and put in a service mesh.
Yes, traffic between generic service and the mesh entrypoint is clear text BUT since the proxy is in a sidecar of the generic service pod, it shares the same "localhost" by mean of Linux network namespaces, so it's virtually isolated (if there isn't a bug) from other code running on the same node. When it exits the pod localhost, traffic is already encrypted.
I've attended a talk on Kubecon last year on how one company adopted Istio service mesh. I've lost the guy in the first 10 minutes of the talk as it was so complicated, and decided that service mesh is 100% not going into our k8s clusters.
Recently an overly confident security engineer came to us and demanded that we get service mesh because thats a SOX requirement. I have no idea from where these people get pipe dreams like that.
I think Istio docs are great! I do agree that it's complicated, and I think their API is more confusing that it needs to be. The ontology of DestinationRules, VirtualServices, ServiceEntrys, Gateways (as in the K8s resource), gateways (as in the istio gateway Helm chart) is not the best.
Not having worked with K8s, it seems to me a number of things that service meshes are capable of can be done by SDN (e.g. Tailscale, ZeroTier). As far as I'm aware, SDN can do encryption and service discovery (via things like custom DNS) just fine. Can someone explain to me the differences and tradeoffs involved?
That's been my thinking for a while, too.
I work extensively on Kubernetes-hosted apps, but our org has (wisely, probably) eschewed service meshes in favour of ingress-based solutions. However, the simplicity of those solutions make things creaky and error-prone.
Rather than injecting sidecar containers that set up networking and so on, having pods join an existing SDN that just works with no app-side config would be a much more elegant solution.
Other than Cilium, I'm not aware of an SDN like ZeroTier or Wireguard that works seamlessly with Kubernetes this way (and which also works on managed Kubernetes like GKE and EKS).
As great as reducing complexity is, I just don't see how it's possible to avoid implementing a service mesh in a FedRAMP moderate or high impact level environment. You essentially need to implement mTLS to meet SC-8(1), and to implement mTLS at scale, you need something like a service mesh.
Are there other ways of going about this for FedRAMP moderate or high IL?
I think people often don't realize that depending on the language runtime, micro-services can easily be a must.
Most service boundaries at organizations are "I need a different version of a pinned package we can't upgrade.". This is common in languages where there is support for only using one version of a given package, and it's worse if there isn't a culture of preserving function APIs. E.g. any python company with pandas/numpy in the stack will need to split the environments at some point in the future, no ifs ands or buts!
I have heard that the reason Docker (and containers in general) took off was that they solved the problem of Python's awful package management. I didn't believe it until I saw people put Python in production and have to deal with this. At this point, I would rather have a physical snake in my server racks than any Python code.
I guess so. Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it. Try running any of the program which is >3 years old. I remember in one of the programs I needed to pin 10 dependency versions manually to make it run.
It wasn't my code. I pin all my dependencies, but lot of python code has been written by people without software engineering experience like university students and to an extent ML engineers.
> Virtual environments doesn't solve any of the problems as python authors either don't specify the version or their dependency author doesn't specify it.
This is a problem of lack of training or willful poor practice, not an issue with venvs themselves which absolutely do solve this issue.
Agreed, the problem would not exist at all if an approach like that of say, node development, existed, but here we are.
But only in what you deploy. For the love of God don’t pin versions in libraries unless you’re really sure it has a completely different api. That’s how you get a dependency graph with no solutions.
Agreed, or if you dont, you should run pip-compile during CI/merge in order to pin every dependency in the entire tree.
These are very solveable (and have been solved for a while) problems. It really pains me that most people are either unaware or do not dive deep enough below the surface to find them.
Dependency pinning is one of those things where I do see valid use cases, but "making it run" isn't one of them. It should primarily be use to deal with incompatible version until you can make the necessary changes.
If you depend on dependency pinning due to unmaintained code then you should go deal with the problem directly. Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to?
That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year.
> Say you pin 10 external libraries to three year old versions, how many security holes does that expose you to?
> That's really my issue with dependency pinning, you end up with software that are just allowed to rot, making upgrades more difficult with every passing year
You seem to be suggesting not pinning and just pulling in the latest (minor, I presume?) versions every time you redeploy.
A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis, especially for publically available services. A proper CI system should allow you to do this with confidence.
Occasionally you will be forced into major version upgrades, but in my experience this is rare.
This should be a regular and essential part of software maintenance and security vigilance.
> A better way is to pin _all_ your direct dependencies and check for minor version upgrades on a regular basis
I'd agree with that, I just don't believe that to be happening on any significant scale. Reasonably I think you should be able to do pinning like:
library>=3.0,<4
Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages.
Pinning is fine is you manage updates, but if you're then pulling in three year old libraries, which may come with their own pinned third party dependencies, I still think you're doing it wrong. In the example from GP it appears that they are pinning versions as a way to avoid forking and patching a deprecated and unmaintained package.
I'm not oppose to version pinning as such, but if you don't have a plan to stay on top of security updates, then you're better off pulling in any minor update. It's not just at redeploy, you may reasonably need to pull dependency updates more frequently than you redeploy.
If you deploy using Docker e.g. you're CI system needs to be constantly be pulling in updates, rebuilding and testing container images. People just don't do that. Most developers I worked with don't even care to update their container images if they aren't updating their own code. I've on multiple occasions had to deal with developers who absolutely lost their mind because we didn't automatically pull in OS updates, yet they themselves shipped outdated Java libraries or at one point even relied on an out date version of an alpha release of Tomcat that hadn't been updated for three years. The only different was that their dependencies where in containers and therefor, in their mind, "safe".
> Depending on the versioning scheme of the library, but yes, pulling in minor release. We currently do that using Debian packages and just rely on the OS to provide security updates to underlying Python packages.
Why not use a venv or Docker? It's a world of difference.
There are multiple issues with relying on the OS package manager for Python dependency management.
Overall it sounds like you know good practice and you know the value of venvs, pinning and Docker but you've had to deal with some workplaces with really shoddy practices. Still, that's no reason to say "venvs don't work" or to give up altogether on the notion of good practice by saying "People just don't do that".
I've worked in shoddy shops, I've worked in shops with excellent practices. In some places I've been the one that made them transition from the former to the latter.
In this case your accidentally upgrading your dependencies only when rebuilt. Especially with micro services some things can be ran for years without being rebuilt.
I have python service with does some AI job but it just doesn't build anymore. I have "golden image" which I carefully back up, because losing it would mean the catastrophe.
Had the same problem with node 16->18 recently. No recent OS version would build some extension. Had to rearchitect a good bit of an app to get back to something working.
Python has a real problem with version incompatibilities for the interpreter, and a few packages require C libraries of specific versions. But outside of that, vitualenvs solve all of the issues of "how do I run those two programs together".
After the Py2 vs. Py3 thing settled down, almost all of the operations issues got away.
That said, Python has a really bad situation about dependencies upgrade on the development side. But Docker won't help you there anyway. Personally, at this point I just assume any old Python program won't work anymore.
This was not my experience building infra for a startup heavily leveraging a Python monolith. It was painful AF (both when developing locally and deploying to VMs) and Docker made the deployment story palatable (build, push to hundreds of VMs, run).
Have you tried deploying to hundreds or thousands of VMs? In my experience, managing container deployment and state is much easier, vs wrestling with inconsistent env state on compute for whatever reason.
For the life of me I don't understand how deploying to 10 machines is functionally different from deploying to a billion machines as the process is exactly the same. Unless you sabotaged your deployment machines with some manual meddling, it's the same with Docker images and Ansible.
Can you please explain how this dependency hell manifests itself if you are using a dependency resolver such as poetry or pipenv which locates the appropriate maximum version? Once you've locked the appropriate versions, you just do pipenv sync in prod like I said.
Yes - you pull in two third party libraries that need two incompatible versions of a mutual dependency. That library (the common dependency) is not backwards compatible due to deprecated functions or whatnot. That dependency could also be Python itself.
This is an incredibly common occurrence, especially with ML systems which are not designed by people with an engineering mindset.
Not getting the package versions you want is a local dev problem and not specific for python at all. I will remind you that this thread is about deploying to prod and dispelling vague assertions of nebulous problems that are supposedly solved by docker.
That only works until you have a conflicting dependency (same codebase: a.py imports libraryA which needs dep>3.0.2, and b.py imports libraryB that needs dep==1.8.3). Then you're screwed.
It will use the max version, but that version will likely have deprecated things that the other dependency relies on in its chosen version. Specifying that you must use an old version of a library is how a lot of Python maintainers resolve deprecations in their dependencies.
Microservices (or really services in general) solve some of these packaging issues. If I have my application that depends on package A that pulls in dependency C version 1.x, and also on package B that pulls in dependency C version 2.x, this just doesn't work in Python, and many other languages. The only way to make it work is either rectify my dependencies so all my versions match (by running one of your dependencies out of date) or to split them up so my application is composed of one service that pulls in package A and another that pulls in package B, and have them talk over some IPC.
The article is about service meshes and the tradeoffs amongst them. Going back several years, teams at companies ask about feature X - mtls a big one. The discussion goes to - should we use a service mesh, often the answer was no.
K8s is a great platform with many options, but many decision makers have little knowledge (or don’t research) the implication of their choices.
> decision makers have little knowledge (or don’t research) the implication of their choices.
I hate how true this statement is within the industry. To many of these C-level executives base decisions off whatever CEO summit they recently attended.
“Every app to use microservices!111!”
“Hybrid cloud. We are doing it”
“Serverless, let’s start using this”
“We are fully going to the cloud!11!!”
Then when the results come in, the complaints start rolling in:
1) wHy iS aPp SlOwEr (after MSA)
2) gUys, iNfRaStruCtUrE cost is SoArInG (shifting to “cloud”)
3) the ApP is ToO cOmPlEx (after MSA, and “serverless”)
Some of these aging dinosaurs need to be put out to pasteur
> To many of these C-level executives base decisions off whatever CEO summit they recently attended. [...] Some of these aging dinosaurs need to be put out to pasteur
AFAIK, the virus of C-suite IT bad ideas doesn't discriminate on the basis of age.
It is worse than that. Many decision makers are making their decisions based on advice from people who fired up k8s and all its gadgets for their pet project or google.
I read the article as being about service meshes now being a cost item whereas they were free or low cost. I’m not sure debating the technical merits speaks to that.
I take what he said as admitting that he is one of the "many decision makers [who] have little knowledge (or don’t research) the implication of their choices", and that objective fact stings him personally.
Ideally you would be using immutable events over a message bus as a default pattern.
Just as anti corruption layers have a use case. So do service meshes.
But service meshes they are synchronous communication, with indirection typically through a sidecar, they will have the very real costs of synchronous communication in distributed systems.
IMHO the problem isn't that they are problematic when applied to the correct use case, it is that as a default pattern they inherently will cause problems with little to no benefit for systems that can use events.
It is simply the same as any tight coupling, it should be avoided unless a specific use case justifies the costs.
Cargo culting it in because it is superficially 'easier' will typically result in code that doesn't have the scaffolding in place to support events and is very expensive to pivot away from if you don't intentionally write your code to support looser coupling in the future.
Istio, Envoy, Linkerd, etc... currently cater to the synchronous request/response communication between microservices.
It doesn't matter how decetralized their implementation is, it is still essentially sync operations over a network.
We have known about the costs of that decision for a long time irrespective of the implementation complexity.
This was a good read, as someone using K8s a lot in the last two years but not service meshes yet, it gave me a lot to think about.
I understand there are various advantages like metrics, etc, but the encrypted traffic between pods and services is the one that I can see many orgs demanding. If not a service mesh what other options are there?
I am sort of surprised that the solution to encrypting your traffic isn't using HTTPS internally, and people rolled these wild systems of proxies. It makes sense thinking about it now because you have no idea whether you need the encryption or not (two containers may be on the same host), but this seems to be a pyrrhic victory.
Once again, the bad implications of a seemingly good idea come to bite us.
The solution _is_ running HTTPS internally. But as an application developer, do you want to manage certificates? Do you want to make sure you're using the "correct" TLS versions and that your app can talk to other TLS versions? What happens when team Z deploys their legacy Java app that doesn't? Where are you getting your certificates from?
All of these go away with a service mesh and sidecar model where the everything is encrypted for you and certificates are completely managed by the service mesh. No need to roll your own PKI. Your app only needs to speak plain HTTP and from it's perspective every other app is also just speaking plain HTTP, no TLS in sight. It makes developers' jobs a lot easier, only concentrating on shipping features vs. wrangling with TLS.
Saying all of that, you could just roll your own fine. Depending on the competency of your company.
To set up an open-source service mesh, the infra team anyway has to configure a private certificate authority and cert-manager to create k8s secrets for the service mesh components. From there, it's straightforward to extend the common deployment template (hopefully there is one) to mount a volume with an auto-rotated certificate. All an application developer has to do is to use that certificate, which is much less effort than what you are implying.
It’s not less effort. I’ve done both ways in production for large teams. What you described is literally entirely automated by the mesh in a more secure and maintainable way than a bespoke hand rolled solution.
> But as an application developer, do you want to manage certificates?
It might take years of trying to avoid TLS to learn that all the alternatives are far worse. So—yes, just bite the bullet, it's not that bad once you internalize the model, and you really only need to solve this once per organization.
It does have some advantages in that you don't have key material on every single server. How much this risk actually matters will vary. I suspect most enterprises would actually be relatively safe with their frontend certificates being compromised, though the same cannot be said about the actual compromise of the backend application.
I assume you would either use self-signed certs or run a private CA to do this. You wouldn't get the attestation benefits, but there's no material of any value in each container if you do this. The encryption is the only benefit.
Reading that comment, I sort of understand why people don't do this because it takes some understanding of HTTPS and cryptography to do this properly.
What I get is that the proxies are a really complete "not my problem" solution, that puts all the burden of setting-up the network encryption on the hands of the people with direct access to set-up and debug your network.
If this doesn't look like a real thing to you, congratulations, you are in a well run organization that doesn't have this problem.
Anyway, whether the costs of this thing are worth the gain, I have no idea.
It boils down to the complexity of running an internal DNS and of issuing certificates. If you don't do that, HTTPS is useful only for encryption against passive eavesdroppers.
As someone that got a lot of value out of client side load balancers as a language neutral network library (e.g. we ran envoy under synapse for a while) From the start the Istio design choices tended to address problems we didn't have / felt like architectural dream states while making problems we did have harder to address. The messaging and overall focus seems to have improved and focused more on meat and potatoes features but there's still a number of things that are hard to tweak via the standard istio apis that we have to backdoor in there.
Just like you probably shouldn't be using K8s, you probably shouldn't be using a service mesh. Only add it when you actually need it. You'll know when that happens.
Assuming that this is yet another DevOps tech I'll be expected to be an expert in, I looked up Istio [0]:
As organizations accelerate their moves to the cloud, they are, by necessity, modernizing their applications as well. But shifting from monolithic legacy apps to cloud-native ones can raise challenges for DevOps teams.
Developers must learn to assemble apps using loosely coupled microservices to ensure portability in the cloud.
This just isn't true. You don't need microservices to use the Cloud.
I think they mean ‘you must be able to run your monolith in a way which only starts a small subset of services’ to leverage the cloud cost model, otherwise just rent a colo.
Seems to be it's an ebb and flow of complexity; sometimes it increases like with sidecar proxies and their issues, like how they might listen on NICs differently, or how they cause grey failures, and sometimes it decreases like when the sidecar moves to the kernel and gets rewritten in eBPF and we can remove another ip filtering-firewall-chain. So right now it seems to be a bit of an ebb, as these things are becoming moduralised.
> When you start using Kubernetes one of the first suggestions you'll get is to install a service mesh.
Who on earth is giving you this advice? Service-messages are squarely in the “you’ll know when you need it, and it’s not day 1” bucket.
Is this the experience people are having with K8s? Welcome to Kubernetes, here’s a service mesh gl;hf. If this advice is common, no wonder some people think K8s is over complicated.
To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far.
I'm sad this is the top comment on an incredibly detailed and well researched article.
The context of the content is obviously for Kubernetes enterprise platform teams and not deploying your first nginx pod. The author also gives specific examples of why a service mesh is useful and in what scenarios it shines.
> The author also gives specific examples of why a service mesh is useful and in what scenarios it shines.
The author's entire point was money which is weird as it is peanuts compared to the DevOps salary of people needed to work with this complex setup. Unless you are in a startup and good in tech, in which case service mesh doesn't make a lot of sense. Why do you need encryption or metrics collection above ingress. Nginx does a good enough job for metrics.
It might come as a shocker for you but many medium-sized orgs that prefer to pay salaries for people who can do N things and just augment their capabilities with some Salas/paid service. But if the service is too costly, they will not allocate budget for it.
Also, unlike many Salas/software vendors would try to make you believe, those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium.
Also, not all salaries are SV-level, while these softwares usually have the same cost wherever you are in the world.
Thanks for the unneeded tone. But the solution that author likely uses is linkerd which is free for 50 users. And even then very likely it would be less than $1000/month. If it is too much for the org to pay, I don't think they need it.
Service mesh is not something a small or medium sized company needs.
> those solutions usually need tweaking & knowledge to be operated. And if any issues happen, swift support comes at a hefty premium.
Exactly my point. Base price is not something that is gonna cost them the most when using advanced tools like this. Even if it was free it would costed pretty much the same for any medium sized org.
Man, big corporations are often weird and way more complex than first (or tenth) look can tell.
You can have a guy banking 200k a year in base salary begging his superiors to have a fast laptop for work and compilation, and request is not even rejected, just hangs indefinitely. It would pay for itself within few days with increased efficiency. Or say Java dev fruitlessly begging for intellij idea license that cost less than 1 MD.
Been there, done that (in some way), now I just accept whatever boundaries are given, and let everybody know that due to this my work will take X amount of time. If they don't like it, escalate this so that management structure is aware of unreasonable expectations within given situation. If corp is so rotten that this all doesn't matter and its a long term situation, just leave such toxic place, you may feel scared of uncertainty but later you will thank yourself.
Health is a finite resource and leaking fast, you often don't notice it since warning signals come way too late.
> But the solution that author likely uses is linkerd which is free for 50 users. And even then very likely it would be less than $1000/month. If it is too much for the org to pay, I don't think they need it.
Didn't check the Linkerd plans in detail but the author says 2 grands per cluster per month. Now, even a medium org (like where I work), can have easily 10 clusters between different environments or different workloads. It's not that weird. And that would make the price already going to 20k a month which needs to be justified. Especially if you were using the same thing but not paying anything before.
K8s, service meshes, all this infrastructure complexity to me seems like more of a
band-aid to bad backend practices and lack of standardization.
This kind of world arises if an org has, say 1000 engineers, or ~100-200 teams, and every team wants to do their own thing with accountability improperly mapped etc.
And infrastructure engineers turn into police with an obvious overemphasis on observability.
I guess a large amount of complexity in infra and wastage is done to hide human inefficiencies.
The flip side is that some of this standardisation lets smaller teams very easily get this functionality without the man-hours required to implement it in your application layer.
< Usual caveats apply, don’t bother until you need it, etc etc. Personally though, I’ve found dropping in linkerd, or Cilium CNI about an afternoon’s work, with no application code changes necessary>
Or there is a company mandate to encrypt everything due to the customer and legal needs. Then instead of rolling your own solutions you use an established mesh and move to other problems. The complexity is not there but your lack of understanding why someone may need a mesh. The last sentence is because you pre-judge the need and the platform it needs.
Yes I do this for living snd know what I'm talking about.
To answer your question, yes this is the majority of the experience and is almost a given decision in the initial setup, in part because of the security association. Especially in enterprises it's easier to just say service mesh than have to think the decision through.
Regardless of service mesh though, although it's a part of it, the complicated reputation is not undeserved and not solely due to meshes.
I have small cluster and the only reason I’d want service mesh is to inspect req/resp between services for better debug. And I’m not even sure they can do that.
Retries for free are good I guess, but not something essential. MTLS is something I don’t want at all.
Couldn't you accomplish that with tracing? You may not even need any changes in your application as eBPF maybe able to automatically instrument the application.
Alternatively, If you use Cilium as a CNI, this functionality can come out the box. https://docs.cilium.io/en/latest/observability/visibility/
Exactly if you don't need the functionality provided don't use a mesh, but consequently if you then use it. Honestly not very complicated, but mTLS is very complicated done correctly.
If you don't hear this advice, you are simply not in the audience that routinely uses service meshes. They are quite useful in enterprise environments.
> If this advice is common, no wonder some people think K8s is over complicated
I mean, if even Google who were behind it in the first place, are saying it's complicated and have 3 levels of managed Kubernetes services, I'd say it's pretty clear to everyone it is indeed complicated. Some of the complexity is simply due to the complex problems it solves, some is footguns, some is layers upon layers of abstractions to glue around design deficiencies.
> To anyone not aware: it _does not_ have to be this complicated. Default ingress controller and a normal Deployment will carry you really, really far
You're absolutely right. However, Kubernetes is rarely chosen after a careful evaluation of requirements; it's more often than not because it's considered necessary or for resume driven development. In that case, service mesh is easy to tack on, regardless of need.
its yet another tradeoff, it adds more security for the apps too, now your apps dont need to have tls stuff, you let the sidecar envoy-proxy handle that ect
yes a bit overkill for first use; deff want to tinker around with it without the extra moving parts at first; but i'd say if somebody is venturing into that space they should also consider security as well and should certainly try to get the added layers. a lot of companies miss out on security and implement it as an afterthought and run into more pain for everybody involved
This article gives an overview of the 5 or so most popular services mesh options for Kubernetes for Enterprise installations.
So if you are reading it and not the target audience, keep that in mind.
Also, people in the comments arguing against service mesh with “it seems”, “i assume”, “i think”, etc. Please read the list of concerns that a service mesh is good at resolving, also in this article. If you don’t have these concerns, it doesn’t mean they dont exist elsewhere, right?
good lord is this what modern microservices are like?
How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service?
Actually, to the end — companies that embrace microservices, which see the value in them. How do they manage network traffic for these kinds of K8s setups at scale? Surely they’re not using REST and HTTP. Is it as simple as protobufs over HTTP? Quic? Something else?
Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :)
I really hate this pattern in principle, but it isn't as bad as you're making it out to be. Most of the service meshes operate as sidecars, which is to say that if service A is calling service B through a mesh, there are two proxies in between the services, but proxy A is on the same machine as A, and proxy B is on the same machine as proxy B. So it is kind of offensive to have 3 network hops involved, but actually only one of those is over the wire, and in most cases I don't think the proxies are actually causing any measurable latency. (At least, whatever latency it's causing is smaller than the latency caused by the TLS encryption, which is necessary and the whole point.)
I find that microservices and this type of architecture have become a religion - you do it this way because you do it this way. You add another layer of complication because that's what you do now. You add this product because that's what you do now. Now you do it this way. Now you stop doing this thing and do this thing instead. It's all proclamations and a truly insane level of complexity and often a truly stunningly low level of performance achieved from some very powerful hardware because everything is behind at least twenty layers of abstraction and you're like, encrypting traffic which is just being passed between VMs which are on the same hardware, but because you can't guarantee that they're always on the same hardware you have to encrypt and use a proxy and... oh wow
Watching it from the outside is a bit exhausting, it just seems to be so much churn and overhead.
> How is it better have service A request to a proxy, which requests to another proxy, which requests to service B? I get the security benefits of that, but the network architecture is boggling. How many PBs of data are sent each day for what could be a monolithic service?
The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source.
Ultimately engineering organizations need to decide the model that works best for them. Neither is inherently better, they’re solving different problems.
> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other. Monoliths remove the overhead of network requests but they present their own challenges. You have a lot of implicit dependencies and feature development becomes complicated with changes having unexpected effects very far from the source.
I have seen, developed, designed, managed, deployed, operated, fondled, and otherwise been around thousands of large systems that are anything between trivial importance to “must always be running, in the national interest”. I was around when SOAs were a hot new thing, and SOAP was being rumoured as the thing that was going to save us from everything. A fondly recall an overpaid Compaq consultant talking about “token passing systems” when they were describing message queues.
I have seen exactly two systems that really had to be designed and built along a microservice architecture. Both of these had requirements that introduces a scale, scope, and complexity you simply don’t see very often. All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego.
When discussing this topic, proponents of microservice proponents talk about many of the advantages these architectures bring, and they are often not completely wrong. What is lacking from these discussions is often a sense of perspective. “Is this solving real problems we have?”, “What is the compound lifecycle cost of this approach?”, and, of course, “How much work is involved in displaying the users’ birthday date in the settings page?”[1], and “when will Omega Star get their fucking shit together?!”[2].
Don’t start with microservices as the default. I will typically work out the monolithic approach as a point of departure. Want microservices? I’m open to that, just demonstrate how that will be better.
> All other microservice architectures didn’t solve for requirements, they solved for organisational inefficiencies, misalignments, mismanagement, and - in no small part - ego.
Be developed, maintained, and operated by the real organization that exists and not an ideal organization which doesn't is, in fact, usually a practical if not a theoretical requirement, and its usually easier to adapt architecture than to adapt organization.
Absolutely. But in many cases you'll find companies have far more than one microservice per team (or squad). At that point ypu start to wonder. Some devs seem to use microservices for what modules used to do.
> The tradeoff here is to decouple services in order to allow them to be developed somewhat independently of each other.
You don't need a service mesh for that, though? Heck, you don't even need an ingress for service to service traffic. ingress-nginx does the job well, without being overly complex, and most importantly to me, logs when something is wrong, which I cannot say the same for Istio which I was fighting earlier this week where it was just happily RST'ing a connection and saying nothing about why it was deciding to do that.
There are a lot of good and bad reasons to adopt a mesh. Some of which might relate to your concerns. The things I like most about them, working in infrastructur:
1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps.
2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates.
3. I can implement automation around canary releases without much lift from dev teams. Other projects leverage these capabilities and do it for you as well.
4. I can get the equivalent of tcpdump for a pod pretty easily which I can use to help app teams debug issues.
5. I can improve app reliability with automatic retries and timeouts.
Probably some other things as well... That said, it can be a big increase in complexity to your system the pains of which aren't always distributed to the folks getting the benefits.
> 1. I can have a unified set of metrics for all services, regardless of language/platform or how diligent the team is at instrumenting their apps.
But they're only going to be coarse metrics, like what requests/second. You're still going to be needing application-specific metrics.
> 2. I can guarantee zero trust with mTLS, without having to rely on application teams dealing with HTTPs or certificates.
I do like the idea of this feature of service meshes. It is a slog to get teams to do this responsibly. But, like I said: fighting Istio to understand why it was RST-ing a connection, for no apparent reason. Not logging errors is worse. Perhaps the idea is sound, but the implementation leaves one desiring more.
I should mention the same Istio service mesh above is a SPoF in the cluster it runs in, on account of being a single pod. I can't tell if the people who set it up were clueless, or if that's the default. I suspect probably the latter.
> 3., 4., and 5., as well as actually using mTLS in 2.
TBH, these are just benefits I've never been able to realize. I'm stuck slogging through the swamp of service mesh marketing and the people who want to bring the light of their savior the service mesh but without actually getting their hands dirty doing the work of deploying it.
The fact that you need other metrics does not substract from OPs original point. It's still good and much better overall to handle a set of comprehensive metrics at infra level than to orchestrate every app.
About the coarsness, i think it's not really true. Proxies are freaking powerful and they do a lot of stuff at l7, too much in fact (look at envoy, jesus christ). That's one of the reasons why despite the insane complexity of service meshes, they are paramount for observability.
1. I agree with the sibling comment, that's usually not the depth I need for application level info. So the entire skill and infrastructure cost for metrics still exists.
But it's nice to have as basic data for triage.
2. I always wonder whether that's a timing thing vs. NetworkPolicies and encrypted inter-node traffic.
Are the realistic attack scenarios where it's possible to read out intra-cluster traffic but not mess with the cluster, or even read the intra-pod traffic?
3. I've been quite disappointed with how little k8s provides here. I wish it was easier to move traffic off of an old version and only shut the pod down once the last connection was done :/
Maybe I need to look into a service mesh for that?
4. What's the difference to e.g. kubeshark, or just attaching a tcpdump debugcontainer to the pod? Another instance of first to market / potentially nicer ecosystem?
5. I get squeemish with infra-level activities like this.
Yes, technically the http method and some headers should make it obvious whether that's save or might break at-least/at-most once or similar semantics.
But that requires well behaved applications. While the premise here is infra imposing behaviour to allow applications to be looser around these kind of things.
Can someone enlighten me on this: if Authorization policies (which pods can communicate with which services) was built in kube-proxy, wouldn't it solve the use case for a large percent of service mesh deployments?
> Edit: lots of great discussion below but I really meant how do they manage network TRAFFIC, not microservices in general :)
I don't really understand the question you're asking, but I think maybe the answer is that network pipes are just bigger than the scale most people are operating at? I don't think anything I've ever done has really had that many qps, and if it has, it is more likely to raise an eyebrow that says "who's spamming requests" more that "I guess we've made it to the big time".
REST & protobufs are orthogonal. Empirically, literally nobody is doing REST, and most things are just ad hoc, poorly to not-at-all defined JSON/HTTP with a few HTTP verbs sprinkled in to make everyone feel good. It could be protobuf, too, if you like, but unless you have some truly gargantuan JSON, it really won't matter in the end. Compression will make up enough of the difference in size on the network. Some languages don't have to allocate the keys a billion times, too, though even in Python, it's a while before it starts to hurt.
What I see more of is processes just inexplicably using gigabytes upon gigabytes of RAM, burning through whole years of CPU time for no particular reason before just dropping back to nominal levels like nothing happened, and dev teams that can't coherently understand the disconnect between just how much power a modern machine has, and what their design doc says their process is supposed to do (hint: something that shouldn't take that many resources).
I like microservices, but there should be strong areas of responsibility to them. For most companies, I think that's ~2–3 services. At my current company, it's ~2 services + a database, with the rest being things like cron jobs or really small services that are just various glue or infra tooling.
> How do they manage these kinds of K8s setups at scale?
Our DevOps team starts off the monthly all hands meeting by leading a ritual during which they ceremoniously sacrifice an animal from the Fish and Wildlife Service's Threatened & Endangered Species list while the rest of the company chants:
Exorcizamus te, omnis immundus spiritus
omnis satanica potestas, omnis incursio
infernalis adversarii, omnis legio,
omnis congregatio et secta diabolica.
"Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles."
-- Anthony DeBoer
"SCSI is *not* magic. There are *fundamental* *technical* *reasons* why you have to sacrifice a young goat to your SCSI chain every now and then."
> How many PBs of data are sent each day for what could be a monolithic service?
The companies that need service meshes couldn't possibly run everything in a single monolith. They already have several if not dozens of monoliths, each one typically coming into the architecture when a large enterprise acquires another company and its monolith, or simply different business units / product lines that don't talk to each other because of sheer organizational scale.
It's called a service mesh, not a microservice mesh. First you get the benefits wrapping, securing, and monitoring each of your monoliths, then you reap more benefit when you start to break up each of those monoliths so common concerns can be addressed by common services and provide a unified experience to the enterprise's customers/users.
The data over the wire is the effect of having micro services and has little to do with service meshes. If anything, a service mesh could help by transparently enabling compression or upgrading to h3 without having to bake that in every app. Also, from a networking perspective, the proxies are hosted next to the instances so there is no difference there.
We run 1800 services in our mesh. Most of them rest/http, a select few gRPC and graphql. We even have some soap/http services in it.
with 1800 services, do you feel like you are programming/architecting code in the same way a complex monolithic codebase might, working across them all?
Or are the services just cogs in the machine, managed by individual devs / cogs in the machine? I imagine the latter and presume that's the main benefit of so many services, but genuinely curious as I've never experience that many services in an architecture
I would say it's right about the middle of those two extremes.
A well functioning service mesh (or even just a well maintained and discoverable ingress controller) is essentially invisible to the individual dev teams. Just think how modern stacks work from a frontend dev's perspective: team wants to use an additional feature, so they find the budgeted credit card, sign up to a random third-party provider, get their access token, and go. From the codebase standpoint, they merely added a new roundtrip to a random service and process the responses in their code.
From the dev team's point of view, having the same feature available internally, behind "just another URL", makes no big difference. Maybe less politics around vendor spend and, with luck, easier integration with the remote service auth. Almost certainly less wrangling with compliance and legal.
Whether that URL is provided by an ingress with a proper FQDN, or a service mesh entry with otherwise unresolvable name, is (and should be) irrelevant.
Modern distributed systems have long since become too large for any single person to fully comprehend them through and through. There is no Grand Design[tm], they are all results of organic changes and evolution. Service discovery and routing can be architected. Individual services within the system can be architected. The complete system where hundreds or even thousands of services interact can not.
Right but there are all those layers so that HNs one server can service your request and send a response.
How much traffic would be generated if, for every request to HN, six other requests fire? And every time you comment a cascade of requests fire in HNs imaginary K8s cluster?
It’s not that there is anything wrong with this, or that the tradeoff isn’t worth it… it’s just so much data flying back and forth over the wire.
Ultimately, in the age of massive cloud compute, the constraining resource in an organization is engineering hours, not CPU/memory/bandwidth. And even then, I've yet to encounter a system (outside of massively parallel MPI-based HPC) where saturating network pipes became the bottleneck for a system before CPU/memory utilization.
Microservice architecture means there's lot of data flying around, but it keeps local resource utilization predictable.
I thought that the constraining resource was money in all but the top 1000 or so businesses in the world with effectively 99%+ margins, mostly because they control their markets absolutely and can raise prices or change demand like the big three algorithmic advertising companies. Everybody else in the cloud goes to the wall without very careful cost control, which is by no means automated or low cost, either.
> How do they manage these kinds of K8s setups at scale?
You check in configs into the monorepo and there is tooling to continuously sync the configs with the actual state of the infrastructure.
The advantage of microservices at scale is that team X breaking the build doesn’t affect team Y. This scale is probably not until you have 1000+ engineers however.
You do deployments as one used to do with big C++ programs also, you can delete everything and deploy again to make sure everything is new. The latter is preferred. Doing continuous deployments is very hard and not sure possible, at least I haven't seen it work well even though it's touted as fundamental k8s feature. K8s groups resources and presents a platform, how you use it it's up to you, and the management.
Traffic within a data center is relatively cheap, is it not? Of course, I think managed network proxies tend to be expensive, but I think that mostly comes from the machine costs, not the network itself. You pay a bit of a tax for network serialization, but it's unlikely to be the bottleneck in most things ime.
Plus, applications built on an external database (e.g., postgres not sqlite) will already have 1 hop. Going from 1 to 2 hops is less dramatic than going from no hops to 1. And I guess implicitly any service sending lots of data to the client already has 1 hop.
Imo the only case where a network proxy would be egregious is an in memory database kind of workload where the response size is small relative to the size of the data accessed (maybe something like a custom analytics engine), but that's pretty niche.
a) It is sometimes impossible to run the entire application on a laptop. And so having a micro-service means you can quickly iterate and test before embedding it into the wider system.
b) You will commonly run into conflicting transitive dependencies which you simply can't work around. Classic example being Spark on the JVM which brings in years old Hadoop libraries.
c) The inter-relationships between component can become so complex that you really want to be able to use canary or green/blue deployment techniques to reduce risk.
The trick is to know which code you want to maintain yourself and which stuff you want to get off the shelf. Going the off the shelf way, also means being stateless and using distributed transactions with all their pitfalls (eventual consistency, fire and forget, ...).
Good architects will know how and when to do what E.g. start with a modulith instead of a monolith, to ease refactoring into micro-services once user count goes through the roof and vertical scaling won't do it any more.
They manage traffic with “service meshes” which includes anything from relatively” basic” distributed iptables/ipvs VIPs (kube-proxy) + dns to full-blown intercepting proxies a la istio/linkerd
Heh wait until you hear how the cell signalling and any biological process in your body works :) It's an absolute clusterfuck of dependencies and side-effects. It's like space bar heating functionality[1] all the way down...!
In my thinking, we might as well get used to the levels of indirection and complexity that AIs will be comfortable with. I suspect it will be more akin to what biological computation is "comfortable" with, and less like what our minds happen to prefer
But I digress :) yes, microservices strike me as wiiild
To me every of these sounds like "why don't we put 10ys(tm) inside ZFWs(tm) inside Funnylettes(tm) proxy-edge-borderline stuff so stuff is so messhy and uncomprehensible that you need an IA to explain your application deployment"
Same. I've never worked in infrastructure teams because I simply don't care about this stuff, and I always find it funny how the infrastructure details tend to leak out to us application developers.
Come on, I just want to deploy my app somewhere. I don't want to know about yamls and kubernetes, I just want a button where I deploy my branch.
It often feels like people working on infrastructure and platform development do it for it's own sake, forgetting that they were supposed to be enablers for other teams building on top of it.
I work on an infrastructure team, and I always find it funny how app developers think it's possible to abstract everything away. All abstractions leak! your code will reach a physical machine at some point in the process. We platform people do our best to abstract things for you, but we can't magically bypass the law of leaky abstractions (https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...)
I grew up in the 1980s, when you had to know your platform well to get nice software written.
I find it annoying that there are a lot of young (and not-so-young) developers who don't take the time to properly understand where their apps are running in order to make the best of it and also not to write crappy and slow software.
I see people climb mountains in Java when they had all they needed in the database or in the OS.
I also suspect people balk at infrastructure because it means learning a new and complicated thing to solve their problem, but the shape of the solution is the same whether they build it in their application as a library or whether they avail themselves of off-the-shelf infrastructure offerings (although there are some problems that can only be solved by infrastructure).
Either way, you have to understand the solution at a high level, and I think much of the balking at infrastructure is really just balking at needing to understand the fundamental nature of their own problem.
I'm an application developer, and I'm also an infra guy, because you can't get away from the fact that your code will run on a machine, and you need to know what happens. I don't have much respect for application developers who "don't want to know anything about infra", sadly.
I generally agree with you. In my experience, the infrastructure engineers are very often the best software engineers. Like there have been half a dozen times in the last year that an infra team at my company has essentially done the architecture design work and high level implementation for a complex system that the dev teams technically owned, but they kept stalling out on.
I think there are at least two reasons for this: the first is that infrastructure engineers (e.g., SREs) just think about reliability and architecture a lot more than SWEs and the second is that the infra eng position generally selects for people who have a penchant for learning new things (virtually every SRE was an SWE who raised their hand to dive into a complex and rapidly evolving infrastructure space).
Also, the best SWEs have been the ones who accepted the reality of infrastructure and learned about it so they could leverage its capabilities in their own systems. And because our org allows infra to leak to SWEs, the corollary is that SWEs are empowered to a high degree to leverage infrastructure features to improve their systems.
The problem is that our field has a deep seated problem with differentiating good engineering with over-engineering. The latter is taken as the former. In my experience SREs/devops will just as often over-engineer and leak their abstractions to everyone else due to paranoia and the current fad in the "cloud native computing" field. Suddenly people need to know about a dozen concepts to debug the travel of a http request. This was not needed before and mostly just a consequence of holes we've dug ourselves (like micro-services) and are now using to justify our over-engineering - all within the part that was previously seen the absolutely simplest one to manage: stateless app servers.
I mean, maybe? I won’t doubt that some people are prone to overkill or hype-driven engineering, but also most of the simplest solutions still look like Kubernetes (e.g., a cloud provider or a platform-as-a-service). The most severe, prevalent danger is people thinking these things are too complex and then trying to roll their own alternative on bare VMs by cobbling together disparate technologies which only the very best sysadmins can do passably (of course every other sysadmin thinks they are in this elite group) and even then the thing they build still leaks implementation details except now there is no market of engineers who are experienced with this bespoke alternative to Kubernetes nor is there a wealth of documentation available online.
> The most severe, prevalent danger is people thinking these things are too complex and then trying to roll their own alternative on bare VMs
No it's not. This was simply just not the case before k8s. Running a stateless app behind a (cloud) LB talking to a (cloud) DB has never been the hard part, and it's even easier today when something like Go is essentially just starting a single binary, or using docker for other languages. People seem to have forgotten how far so few components gets you. But incentives for too many people involved aligns towards increasing complexity.
I mean, yes, if you have a small org with one CRUD app on a handful of VMs, but if you want nice things like "lower environments that mirror production environments", "autoscaling", etc then yeah you pretty quickly run into the kind of complexity that justifies Kubernetes, a cloud provider, a PaaS, etc. Essentially, the problems start coming when you have to manage more than a handful of hosts, and you pretty quickly find yourself reinventing Kubernetes, but you're cobbling it together from a bunch of different tools developed by different organizations with different philosophies using different configuration schemes and your team (including everyone you hope to hire) has to understand them all.
> if you have a small org with one CRUD app on a handful of VMs
I think most would be surprised how big chunk of modern apps are within that space without active intervention by stuff like microservices. No need to stop at a handful of VMs though, although I imagine that most companies could easily be covered by a few chunky VMs today.
And yes, if you're a PaaS, multi-tenancy something something, then sure, that sounds more like a suitable target audience of a generic platform factory.
I don't know. If you have more than a few hosts, it already seems like you already need some configuration management setup so you can reproducibly set up your boxes the same way every time--I certainly wouldn't want to work anywhere with more than a couple of production systems that have humans SSH-ing into them to make changes.
And if that's the territory you're in, you need probably need to set up log aggregation, monitoring, certificate management, secrets management, process management, disk backups, network firewall rules, load balancing, dns, reverse proxy, and probably a dozen other things, all of which are either readily available in popular Kubernetes distributions or else added by applying a manifest or installing a helm chart.
I don't doubt that there are a lot of systems that are running monoliths on a handful of machines, but I doubt they have many development teams which are decoupled from their ops team such that the former can deploy frequently (read: daily or semiweekly) and if they are, I'm guessing it's because their ops team built something of comparable complexity to k8s.
No one changed VMs manually over SSH beyond perhaps deep debugging. Yes, creating a VM image might be needed, it might not, depending on approach [1]. Most of those things you listed are primarily an issue after you've already dug a hole with microservices that you then need to fill. VMs are still a managed solution. I'm not sure where people have gotten the idea that k8s is somehow easier and not requiring significant continuous investment in training and expertise by both devs and ops. It's also a big assumption that your use-case fits within all of these off-the-shelf components rather than having to adapt these sightly, or account for various caveats, that then instantly requires additional k8s expertise. Not to even mention the knowledge required to actually debug a deployment with a lot of moving parts and generic abstraction layers.
One downside I also see compared to the "old-school" approach, albeit maybe an indirect one, is that it's also a very leaky abstraction that makes the environment setup phase stick around seemingly in perpetuity for everyone rather than being encapsulated away by a subset of people with that expertise. No normal backend/frontend dev needed to know what particular linux distro or whatever the VMs were running or similar infra details, just focus on code, the env was set up months ago and is none of your concern now (and I know there's some devops idea that devs should be aware of this, but in practice it usually just results in placeholder stuff until actual full-system load testing can be done anyway). So a dev team working on a particular module of a monolith should be just as decoupled as with microservices. Finally, for stateless app servers, the maintenance required was much rarer than people seem to believe today.
I realize that a lot of this is still subjective and includes trade-offs but I really think the myth building that things was maintenance ridden and fragile earlier is far overblown.
I found it refreshing when a developer said to me point blank, "I don't care where this runs". She didn't say that she didn't care that it runs, but she was being upfront that she had other concerns more relevant to her, so she left the operations work for my team, but crucially, she followed our advice/requests on how to build things to work with our infra. Not everyone can do everything; it's why there are specialties, but it means that collaboration is vital.
I don't care where my stuff runs either, but, when it breaks, I care that I have a good enough mental model of what's going on that I can debug it. Whenever I see web developers not knowing how an HTTP server works, a part of me dies a little.
I remember learning about keyboard matrix encoders because I got into building MAME arcade projects when it was still a fairly young idea and consumer products that integrate arcade controls with computers were not commonly available. When you learn about them, you'll understand why older and/or cheaper keyboards have artifacts known as blocking and ghosting. It's something that I at least found interesting, but I doubt most people can be bothered to care. And as far as how that keyboard encoder is creating the ones and zeroes and then sending them over the USB bus - that might as well be magic to me.
I suppose we all live at the abstraction level that suits us. I'm of the opinion that a lot of the modern infrastructure we work with is over-abstracted. I'm not sure that most developers even know that an HTTP server exists thanks to the work of modern devops where you push a commit and it gets automatically deployed to some sort of container orchestration system. At some level it has to be this way; we stand on the work that was done before us; "the shoulders of giants" it's often said with more poetic flair. So I wonder if it's just me getting older and not understanding the new, or if we have genuinely gone astray in some ways. And those need to be mutually exclusive either.
I don't know, I'm 41 and I understand the application, the WSGI layer, HTTP, TLS, TCP, containers, OSes, CI, etc. Less so lately, as I've transitioned to leadership, but how would I deploy and debug my applications if I didn't know all this stuff?
I guess I've traditionally built and deployed all my apps myself, without the luxury of an infra person, so I had to learn all this, including Linux administration.
Similar age and skillset here though much more comfortable on the operations end of things rather than the development side though I have built small apps that did see some production use which was cool to do.
It's a wide-ranging skillset no doubt, but still built on the work that precedes it. Did you build the servers that ran your code? Rack them and set up the networking? Build redundant power systems for them? HVAC? I'm sure you've done some of that, but no one person can really do it all. I think the saying is something like, "fish don't know they're wet"; meaning that they're so adapted to the ocean that they don't really have any idea that there are other ways to live. I get that feeling when I hear older folks talk about working on mainframes - it just sounds like an alien world. I came of age in the era of distributed systems - Microsoft vs Open Source, but it was always on commodity x86 hardware and TCP networks. There's no law of the universe that said it had to evolve that way. It seems like the new way is "cloud-native" which I'm not sure I understand or like (as I've said, it may just be me getting old); my biggest problem is that it seems to give ever more power to Amazon and other members of the tech oligolopoly rather than any technical shortcoming.
I agree with you on the cloud-native stuff, I'm sure it's great after a certain scale, but I think people switch to it way too early, and now we're running our three apps on a huge stack not many people can know unless they spend a lot of time understanding the configs.
Maybe it's the same as before, maybe I'm just less familiar and it seems huge to me.
In a good infrastructure setup, any average dev should be able to get a new hello world crud service deployed to a test environment with the db and aws service backing of his choice and have it reachable by other services within an hour, without ANY additional config. Take the subdomain name from the repo name and be done with it!
If that is not possible, you have failed your job as masters of the infrastructure.
At my company, for this I need to copy-paste some terraform in repo 1, write some scaling config in repo 2 and put in new, encrypted credentials and other configs in repo 3, then finally I can deploy my docker app from repo 4.
I can all do it myself in two hours but it constitutes total failure on the infrastructure teams side.
While you're writing your 'hello world' app, the infrastructure you deploy to has to support every single permutation of everything every developer in the company writes, multi-tenanted, multi-environmental, multi-region, highly-available, backed-up, in-flight and at-rest encryption, permissioned appropriately, ISO27001 compliant, and that's not even getting to observability and metrics. Infrastructure people tend to have a much harder job, especially with ignorant developers who just want a one-button deploy. Too many developers only care about 1/3 of the SDLC.
The fact that you ended with "I can do it all myself in two hours" indicates to me you're horribly ignorant of all the complexity you want to hand wave away.
I'm not in devops-land, but I pretty much hear the same from devops guys: "Instead of running pip/poetry/npm/yarn/whatever-is-popular-this-month, wait for it to download gigs of dependencies, setup nginx in a specific way, start 10 services for a web shop, use exact python or node version and pray to infra gods that everything holds the water, why can't we copy a single jar/binary, start it and forget about it" ;)
I'm getting the impression that the whole industry shifted focus from solving actual problems in the most simple way to draining money and keeping things afloat as much as possible.
You say you don't want to understand infrastructure and just want a button that magically deploys your app, yet in the same breath you complain that devops folks are "working on infrastructure and platform development for it's own sake". These are contradictory sentiments.
"Leaking" infrastructure details to developers is actually less work for them because they are deferring the finer details of the app deployment to the developers who made it. Your request for a magic button that doesn't require you to understand anything about infrastructure is exactly why we have so much complexity.
I agree that you never escape the requirement to have knowledge of how your application is deployed. Things like autoscaling, security, performance etc, all intrinsically effect how you actually write your application.
I think there's a broader point though, in that very often, infra teams will pursue solutions that solve problems in the perspective of their own lens and interests without good oversight from the broader organisation, for developer experience, and economies of scale.
I've seen and worked in environments at both ends of that scale and the gap in dev-ex, and agility as a result can be absolutely staggering.
So often its to avoid 'vendor lock-in', only for infra teams to become the 'vendor', its complex, so they grow by necessity to be expensive, but then still lack the resources to be able provide a clean experience that can be easily migrated off of, resulting in lock-in.
As a dev, the cost isn't my concern, but whats frustrating is knowing that its possible to deploy a new service, with all the bells and whistles, in an hour, and being unable to.
The problem in complex, large infrastructures is I know exactly what I need to do and have no idea how to even get started. There are no obvious entry points, there are layers upon layers of abstractions to make developing IaC easy for those who maintain it but they reduce comprehensibility to almost zero.
My most recent example: deploy a lambda function. Building the image: 1 day. Deploying it to prod: 2 weeks.
Those pesky details like cache line misses and inter node latencies should just go away. I am far too important to know about such realities. Why should I care about cores or processor architecture, I should just be able to write my code and something should figure out the details.
If you agree with the above statement, please go into management.
Wow, this is pretty hard to read and understand someone who has this mindset. Best developers understand the platform that their apps run on and the infrastructure needed for it. There is no artificial separation between platform and apps. You cannot run your beautiful app without infrastructure needed. This is coming from someone who has done both and still does. Many times over, in new teams especially, you must create the infrastructure and then apps. Guess what if you can't you don't move forward. I think you want to write some code and then throw it over a wall and make it someone else's responsibility. Not very useful ;(. How about testing is that for someone else to do? You must be new to this game.
I don't agree that what we're doing stops at enabling other teams to build. That's part of the role but reducing it to that siloes infrastructure away from other domains and adds unnecessary friction.
You shouldn't have to know about the yamls and the k8s, but you should know about infrastructure concerns and how they relate to you and your design choices, just as infra guys should be aware of what's going on on the front side. Having to work with close-minded counterparts who refuse to elaborate further than "make it work now, or else" is about the worse thing there is. Being willing to participate in good faith in back and forth discussions around those subjects is important and pays dividends.
Honestly I’ve never worked anywhere that made deployment easier for application teams than with Kubernetes. The alternative to infrastructure details leaking everywhere has historically been a tedious coordination process with an Ops team whose incentive is to say “no” as often as possible. The deployment frequency drops precipitously and consequently the size of changes goes up significantly so when something inevitably breaks you have to sift through tons of changes to find the culprit and you have to coordinate even more to fix the release. And since this one ops team is supporting every team at the company, “coordination” hurts. I’m sure it doesn’t have to be that way—that there are some all-star ops teams who can make a more traditional VM-based infrastructure run as smooth as the median SRE team can build out a Kubernetes platform, but I’ve never seen it.
It’s also worth noting that sometimes the details leak because app teams want them to leak—they want to build some feature that requires some particular infrastructure solution (e.g., we want our app to run some background task in response to requests, and we don’t want to do it in the VM lest the autoscaler—which is background-task-agnostic—kill it while the background task is running).
This type of prima donna attitude ultimately negatively impacts customer experience. If your code is really that amazing and such gospel can only come from your hands, consider hiring a real software engineer to do the grunt work of making it work robustly, securely and in a performant manner on the substrate to which it is deployed.
K8S has service routing rules, network policies, access policies, and can be extended up the wazoo with whatever CNI you choose.
It’s similar to Helm, in that Helm puts a DSL (values.yaml) on top of a DSL (go templates) on top of a DSL (k8s yaml), just that it is routing, authentication, and encryption on top.. well, routing (service route keys), authentication (netpols), and encryption.
It boggles the mind!