A note to those who might dismiss this without looking at it carefully because you assume it's just some SPA that mocks the Twitter UI: it's actually a NixOps file (with everything inlined) that deploys the entire stack, including a full backend and database.
FWIW, the link posted here is just to the Haskell part. This is the fully-inlined NixOps file: https://github.com/Gabriel439/simple-twitter/blob/master/sim... (which is no less impressive -- fewer than 500 lines in a single file to specify a full web app!)
I mean thats not doing much... You can docker run anything in a line. A few lines of terraform or any other deployment tool can run a binary on EC2 (and roll forward, and have state management, and so on). The complexity comes when you want to do something more interesting.
from a programming perspecting it looks really un-readable if you don't use FP.
I don't mean this as a slight, and I myself am slacking in the same regard, but it's depressing to me that this attitude is so prevalent among professionals.
There might be some good engineering reasons that tools like Docker and Kubernetes have much more traction than Nix, but "it's unfamiliar if you don't use FP" is a terrible one. It doesn't require that much investment to become familiar, and there seems to be many fantastic advantages to functional approaches.
but "it's unfamiliar if you don't use FP" is a terrible one
I get the sense that there is a large contingent of programmers who feel disdain for mathematics and FP gets caught in the crossfire. It’s unfair toward both mathematics and functional programming.
I would love it if North American culture could adopt a better attitude toward mathematics in general. I think far too many people decide they’re bad at mathematics at some point in high school and never look back. This attitude is far less prevalent in many Asian cultures, where the emphasis is put on hard work instead of talent.
I know no good math teachers. Most math teachers pre-college don't know the material themselves. I still don't know why something is "efficient with" (a coefficient). And every time I try to get into any math there's 79 unspecified prerequisites I supposedly need to understand all the jargon. 3blue1brown is preaching to the choir AFAIK. I get lost trying to watch his videos as they also expect you already know the material.
> I still don't know why something is "efficient with" (a coefficient).
That's just a name. You don't need to know the etymology to understand the topic at hand. But if you want to know, you can think of co-efficient as "together-effector". In 3x, 3 is a coefficient of x, as it's something that "acts together" with x. There isn't really much more to it. You don't understand concepts by studying their names in much detail.
Another thing to get used to is that the same word may be used for different concepts in different contexts (or different words for the same thing). Often there is very little connection between them. You have to accept this as a fact of history. You can of course try to look for connections, but don't get frustrated because of not finding any. And some concepts are just badly named. Naming things is hard. Don't try to deduce things based on the names of things. Learning to understand names as just labels is an important step in acquiring good abstraction skills necessary for math.
I think this may be an issue for certain people that they approach math from just the wrong angle. The jargon and terminology is of course important for understanding of the content, but the prerequisites are not really lingo-understanding, but concept-understanding. Unlike in other fields, you don't learn math by pouring over the text and reading page after page. You read something and try to understand it. You play with it, you stand up, take a walk and try to work out why something is the way it is, think of examples, try to clear up what you don't quite get. You cannot use the same learning tactics that you'd use for, say, accounting or humanities. The pages read per hour is pretty low in math.
> you don't learn math by pouring over the text and reading page after page. You read something and try to understand it. You play with it, you stand up, take a walk and try to work out why something is the way it is, think of examples, try to clear up what you don't quite get.
Absolutely this. Math is a not a spectator sport, you can't acquire the skill and knowledge by reading. You must engage with it, fence with it, wrestle with it, make it your friend and spend time with it.
I remember not so long ago playing with some equations and accidentally proving Pythagoras' Theorem. It wasn't new, it wasn't deep, but it was satisfying, in much the same way as getting code to perform as you want, accomplishing some task in an elegant way.
But if you don't spend the time, you'll never get the insight(s).
Yes, I think some people falsely learn early on that stumbling upon something they don't quite understand is a sign that they are failing.
I sometimes even think (though I'm not so sure) that having too good textbooks can be an impediment for above-average students as they have less to wrestle with and things are laid out and pre-disgested too much. I may be wrong though, because you can then just get to the next levels until you hit the wall.
But I feel like anecdotally I got good value from deriving some basic things and reorganizing the somewhat badly presented materials in some of my classes. Later on I found very good American textbooks (although widely out of the reach of my wallet at the time), from MIT etc. which were crystal clear in some of the intuition that I had to sweat out for myself. Maybe I could have spent that energy at higher levels, though.
In my country (Hungary, but I saw similar attitude in Germany as well) its not unusual to learn entirely from the lecture material (own notes or provided script and assignments) without any textbook. Some people do get a book from the library, but it's not really expected from you.
my original comment before editing said book/teacher. I use them interchangeably, but I hope the message is still there. Namely: Is it a difficult subject or do you need a better teacher?
More people should play with math like they play with new programming languages or tools. I've learned a lot by just trying to describe in geometric terms the relationships among shapes I see around me or use in my design work.
I learned more about trig working out what angle I needed to input in some design software to make a line cross two exact points than I ever did in high school.
I'm very much a visual learner so I find geometry in particular very satisfying since I work out relationships visually (and then mathematically to prove it).
I find that I learn immensely more in even tiny little tasks where I care about finding the result, compared to any assignment or exercise. Suddenly you look at the resources in a totally different light, critically evaluating what they are saying. Like, I'd pull some lecture slides or papers from the internet and read them in a very different way, compared to actual lectures. The roles are changed, I suddenly see the math (or whatever the topic is), and evaluate what it can do for me in the particular case. Not like what I need to do for the math to conquer it and learn it to the satisfaction of a teacher.
This also applies to computers. Just get your hands dirty, try to build something and you'll find yourself learning about so many things and tools incidentally while trying to fix something.
I can learn much better this way than going chapter by chapter in a book.
But interestingly, even graded, project-style assignments don't accomplish the same effect, only if I'm really onboard with the topic. I seem to need a spark of excitement of "this is not exactly the way you're supposed to be doing things" and just breaking things and playing.
Couldn’t have said it any better. Whenever someone is asking me if I have been introduced to a tool needed to do a job, I tell them “no, but only because I never had a reason to learn it. I’ll pick it up.”
I can pick something up very quickly when I have an actual reason to know it, and getting a good grade in a course is not a strong enough reason compared to “Learning this will help me save the world”. I’m not going to be passionate about it or inquisitive or asking myself the extra questions I normally ask myself when I’m learning something in order to apply it to a problem. And real problems are better than made up problems.
I know no good math teachers. Most math teachers pre-college don't know the material themselves
I’m hoping to change this, not for all high school students, but for those I meet. I’m majoring in pure mathematics with a teaching option right now. My current plan is to teach high school algebra and calculus, as well as physics.
The other comments are right about terminology. I have no idea where the word coefficient comes from, yet I use them every day and understand them pretty well. At the moment, I’m studying ring theory, ideals, quotient rings, and ring homomorphisms and isomorphisms. Trying to figure out where these abstract terms come from may be interesting but it doesn’t help me in my proofs at all.
If you want to understand anything in math, the best approach is to read the definition, read some basic theorems, look at a few examples, and then start playing around with these things to get a feel for them. Try to prove some of the basic properties of the object you’re working with in terms of the axioms. Keep going, proving larger and more complicated theorems, and try some exercises that apply this knowledge to solve counting problems or measurement problems. If you’re learning about isomorphisms then look at some examples and try to understand how classes of object relate to one another, for example by having the same structure in their operations.
If you’re not able to understand 3blue1brown’s videos then you might benefit from using Khan academy to build up your understanding of the fundamentals. It has material stretching from grade 1 to college. Solving problems feels like playing a game, so it can be fun to sit there for hours just doing the problem sets.
Math is a deep, deep subject but it doesn’t require memorization. It just requires lots of effort and struggle in order to develop mastery.
You've probably seen this lecture but I'll pass it along anyway as I found it to be very insightful on the topic of teaching math and physics by using computation.
As someone with a mechanical engineering degree with minors in math and physics I would have loved to have been exposed to this while I was learning the stuff.
> I still don't know why something is "efficient with" (a coefficient).
I can't parse this - would you care to expand on it?
> 3blue1brown is preaching to the choir AFAIK. I get lost trying to watch his videos as they also expect you already know the material.
If you care, you can always track back to earlier material and build your knowledge. Most people I know in a similar situation to yours don't actually care, and won't actually put in any effort.
And that's a reasonable choice. There are places where math is hard, needs work, and most people have come to a point where they feel that the effort won't be worth it, that the ROI isn't there. But if that's your decision, then own it.
If that's not your decision, and you actually want to know more, there are things you can do about it.
FWIW, I know a lot of really good math teachers, but I agree that there are also some really poor teachers. If yours were bad then I feel bad about that, although there's nothing I can do about your experience in the education sector. What you do about it now, if anything, is up to you.
This is the realisation I had (20 years late). Especially beyond high school, maths is hard. You have to put the effort in, and you have to do the example problems.
I think its expressiveness and conciseness makes it more dispiriting to study: one page of a maths textbook can contain so much information, and it feels like you're just stupid when it takes an hour to understand that page fully.
> I think its expressiveness and conciseness makes it more dispiriting to study: one page of a maths textbook can contain so much information, and it feels like you're just stupid when it takes an hour to understand that page fully.
This is very important. Apparently some people get frustrated by this because they are used to studying lower-information-density textbooks. Maybe there would be value in teaching these meta-skills, like how to approach learning each subject. People may apply techniques, like rote learning or brute-force cramming, reading the same page for hours on end, expecting to magically grok it, where these are ineffective of even counterproductive. Not sure how well this is teachable though.
- illustration — whether drawings, gestures, animations, using objects, whatever works, but have a visual equivalent to 'translate' notation; the typical example would be fractions which are as simple as it gets with drawings and hard as hell using only numbers, for most people discovering them. The same is true for most common functions and transformations (which, when you reduce it to the maximum, are the entire body of applied math).
- real-world application (typically the best candidates are simple physics or statistics in high school). The point here is to give a point of reference, a "feeling" of the mechanism or logic or paradigm. The deeper problem is that this application must feel 'interesting', like a puzzle, and not everyone is interested by the same things. That's where, imho, narrow AI could help craft customized courses with ad hoc examples that fit people's "world view" and "approach" to problems, and show animations for every variable, equation, let people play with those and see the results.
Both probably fit in the "faster feedback" approach, i.e. have the teacher/teaching material give feedback as fast and as often as possible, to guide the learning mind on the right path. This is an extremely important discovery of UX in the last 50 years and I believe education has much to learn from these very applied insights into human cognition.
illustration — whether drawings, gestures, animations, using objects, whatever works, but have a visual equivalent to 'translate' notation
This approach breaks down when you go into higher math. You end up working with these abstract objects that have no visual representation. You may be able to find an equivalent object from geometry or graph theory, but it may be so complicated you could never draw it. Heck, even basic shapes like the Platonic solids are difficult to visualize, let alone draw, correctly.
I solved a problem recently requiring me to count the number of vertex colourings of an icosahedron, up to rotational symmetry. Trying to draw one of these things and visualize all of the possible rotations was basically impossible for me. I ended up loading Blender and playing around with one in 3D, using the coordinate planes to see all the axes of symmetry.
If I had been given a higher dimensional object like a tesseract then all bets would have been off. I would have had to find another approach entirely, likely algebraic.
> This approach breaks down when you go into higher math.
It doesn't have to be literally visualization. Just some "feeling" for the objects. Some conception of them as actors, actees, things interacting, or having emotions or whatever. Again, whatever works. Maybe some people can work with concepts just based on strictly memorizing the definitions and drilling through proofs, but others need to conceptualize it all into a narrative. I don't mean dumbing it down, but making it easier to "grab" mentally. Hard to describe.
> There are places where math is hard, needs work, and most people have come to a point where they feel that the effort won't be worth it
The problem with math is that it's seems much harder than it actually is. Every time I have to look into some specific math I haven't worked with before I spend a lot of time decoding what they actually mean. Usually what is happening is not that difficult at all, mathematicians just insist on writing it down in the most convoluted and incomprehensible way imaginable.
It's like they took every bad coding practice and applied it all to math. Why have descriptive variable names if you can just use random greek letters ? Why give a function or operation a name at all if you could make up some weird symbol instead ? And of course you want those symbols to have different meanings depending on context. Imagine if programmers worked like that and wrote everything as obfuscated C++ code, because fuck you.
> Why have descriptive variable names if you can just use random greek letters?
This is a common complaint I see from programmers all the time. However, I've seen people trying to do real maths with descriptive names, and it fails very, very quickly.
This is by no means comprehensive, but as a (probably thought) experiment, try writing reasonable complex code without variable name completion. You write the same variable name over and over and over, and after a while you realise that since this variable is only use in this screenful or two of code, it's easy enough just to use a short name and know what it is, because you're only thinking about it here, in this context.
I'm not going to try to defend mathematical notation in general because there are enormous inconsistencies, many of which are historical, accidental, and indefensible.
But when you're doing the maths it becomes tolerable, then usable, then actively helpful. It's like the parentheses in Lisp - all newbies complain about them, and those versed in the art know that after a while they not only don't matter, they are a genuine positive.
But unless you take the time to do the math, that won't happen for you. That makes it sound like a deliberate barrier, but it's not, it really isn't.
> This is by no means comprehensive, but as a (probably thought) experiment, try writing reasonable complex code without variable name completion.
So why don't math tools have variable name completion ? and/or why insist on still doing things on paper in 2019 ?
The other things about variable names is that they force you to think about what a variable actually contains. This in itself is very helpful not only for others reading your work, but also for yourself when trying to grasp a problem.
The research mathematicians I work with just stare in incomprehension at the idea of doing their work on any computer-mediated system. Yes, there are things that can be done, and yes, computer proof-assistants have made huge strides, and yes, there are always people at the cutting edge doing amazing work.
But your everyday research mathematician will just stare in disbelief.
I don't know your background, your profile is empty, but it sounds like you are someone who genuinely has no idea of how research in math works, and therefore feel that you really must have a better way of doing things. And maybe you have. But speaking as someone who has a PhD in pure math, and who has worked in safety critical software, I can only say that so far everything you're suggesting just really doesn't make sense.
The reason that for centuries mathematicians use single letter glyphs to represent the things they're dealing with is because it is, for the purpose of doing the work, the most effective thing to use.
> The research mathematicians I work with just stare in incomprehension at the idea of doing their work on any computer-mediated system. Yes, there are things that can be done, and yes, computer proof-assistants have made huge strides, and yes, there are always people at the cutting edge doing amazing work.
I'm not talking about computer assisted proofs or anything like that. Just using a readable syntax and the mathematical equivalent of a word processor would be an enormous step forward. No one is writing books in cursive with a fountain pen anymore either, which is basically analogue to what mathematicians are still doing.
> I'm not talking about computer assisted proofs or anything like that.
No, I'm not talking about proof assistants either, I'm talking about actually doing math using any kind of computer system.
> Just using a readable syntax and the mathematical equivalent of a word processor would be an enormous step forward.
Do you have any idea of how to do that? I've done research in math, and I've written software for safety critical systems. I don't know how to create a system like a word processor for math that would let me actually do the math.
Do you know how to do that? If so, please, let me know.
Why have descriptive variable names if you can just use random greek letters
Because math is abstract. The variables and constants don’t mean anything in most cases and are only given different symbols to distinguish one from another. Descriptive variable names or even descriptive subscripts get OLD very quickly when you’re working through a page of calculations to solve a problem.
The beauty of math, if you’re using it to solve some real world problem, is that you can forget about what the symbols mean and focus on solving the equations, evaluating the integral or derivative, or whatever other calculation you’re doing.
Once you’ve completed all of the calculations and gotten your answer simplified as much as possible, then you can go back to the problem you were trying to solve and interpret your result in context. Since you’ve done all of your calculations purely symbolically, you can see the relationships between the quantities you’re working with. In many cases, some quantities you thought were important actually cancel out and don’t appear in your final answer at all. This tells you something fundamental about the independence of your quantities. Perhaps the most famous case of this comes from physics: the acceleration of an object in free fall is independent of its mass. You would not see this so clearly if you substituted numerical quantities as soon as possible.
> The variables and constants don’t mean anything in most cases
Don't they really mean anything or are mathematicians just unable to grasp what they mean ? To me this seems like a symptom of an underlying bigger problem in our understanding of math rather than those variables not having a meaning.
True. Many times the notation gets in the way, but math is very conservative in this regard. Often I'd just prefer seeing a formula in SymPy (symbolic math library for Python) code.
I've never connected FP with math than any other paradigm. FP is easier to grok than OO, it's just that OO was the popular paradigm when most working programmers today started out.
> It doesn't require that much investment to become familiar,
It requires a lot of investment to become familiar.
I've found this mischaracterization very common among people who are proficient with FP: they forget where they came from, how long it took them to get familiar with all these concepts (the Haskell syntax first, then the FP mindset, then advanced concepts like monoids, functors, applicatives, monads), etc...
I haven't touched FP languages in years, but my first year undergraduate intro programming course was taught in Haskell... FP principles aren't so difficult that the average programmer can't learn them to at least a basic level.
I do a lot of I/O and there are arguments against FP in those scenarios.
I think imperative programming languages with functional addons often result in better applications from what I have seen. They are also just simpler to write and may net faster code.
The assertion that not having in depth knowledge about functional concepts is lazy is lazy.
What are the arguments against FP for IO? In my opinion when working with IO in Haskell the explicit tagging and quarantining of IO is super valuable. There is a clear, compiler enforced demarcation so you know when to be careful not to launch those missiles.
This actually actually looks surprisingly readable to me. What I find really funny though is how they use Haskell as a strict language mostly working around its lazy nature with all those do, let etc. constructs. It really makes you wonder if they wouldn't have preferred a strict version of Haskell to begin with. Which is funny of course, given Haskell's propaganda around laziness & pure functional programming.
These days Haskell proponents aren't as proud anymore on laziness. Simon Peyton Jones has actually admitted it was a mistake for it to be the default :
https://news.ycombinator.com/item?id=1924061
Kube is one layer that you use to get to a completely platform independent version of this. Look at kubecfg for an example of this being applied. When Cuelang, the next step after jsonnet, is finished and integrated with kubecfg it will be basically this with:
1. Run anything anywhere (no platform lockin)
2. The ability to talk about your arch using a shared dialect
3. Tooling that integrated with the API server for everything storage and DB (things like Rook), networking and security (CNIs), observability (Service Mesh), and many more abstractions.
Kube is the low level dialect. The tooling around it makes it work like you want.
You can build controllers and CRDs to completely integrate any platform features you need all while maintaining a workflow with kubecfg and using kube to manage state recovery, roll out, and networking.
My problem with Kubernetes is people adopt it without understanding it. They adopt it because everyone else is without ever asking, "Why?"
I've installed and managed a simple K8s cluster before now and it was a nightmare (this was only a year ago.) It wasn't until we moved the management of the cluster to a managed service that it became worth it. It was at THAT point it was worth containerising the application(s) (another complexity to overcome) and letting K8s handle the deployment and scaling.
I'm worried that as an engineering culture we're not pushing back enough and saying, "Yeah K8s is cool but look: do you really need it? It's very likely you don't".
For me to automatically recommend K8s to my clients I'd need to see a (near) zero friction deployment options that's fully feature compliant and truely fully managed. It's at that point I would be happy to say, "Sure! Let's go with K8s and just write YAML files to deploy our containers!"
(And then there's the whole debate as to whether or not you even need a container around your monolith... probably not. Then the next debate about people starting out with microservices when we all know that's not a good idea.)
> For me to automatically recommend K8s to my clients I'd need to see a (near) zero friction deployment options that's fully feature compliant and truely fully managed. It's at that point I would be happy to say, "Sure! Let's go with K8s and just write YAML files to deploy our containers!"
Isn't that what the managed kube offering you switched to gave you?
Managing your own kube deployment is very easy if you are only worried about that component.
> I'm worried that as an engineering culture we're not pushing back enough and saying, "Yeah K8s is cool but look: do you really need it? It's very likely you don't"
The benift in kube isn't the cluster management. It's the abstractions. Let's say "I don't need kube." Now we need to have this series of conversations...
- "The client said we need to make sure the site doesn't go down during deployments."
- "Can we deploy something that needs to be rolled out, restarted, upgraded, and use persistent disk" (like MySQL)
- "We need to monitor some metrics from the machines"
- "We need to scale the number of machines based on system utilization."
- "We need to hire an engineer to manage our infrastructure, so we need to bring them up to speed with our tooling"
- "We need to switch from cloud A to cloud B to save x/year"
- "We need to run on two clouds to fulfil some legal requirements"
There are all documented, standardized, and hireable (engineers already know it) tooling to do. Also it's all declarative so it's just YAML and no magic needed or manual management.
That's the magic of kube. That, the tooling, the shared vocabulary (no ramp up time for engineers), the platform Independence, the automatic scaling, the network security features, and the ability to define your own internal constructs (CRDs).
I set up auto scaling for a kube cluster and it didn’t feel like switching on a feature that was included, it felt more like building a semi-custom solution out of some publicly available parts.
Yes, it is not included. It is a publicly available part that you plugged in.
There are a limited number of people who actually have a use case where they want to manage their own kube cluster. Most people would be happy if they tried a managed kube cluster like GKE [0] and DO [1] for example.
If you are managing your own cluster from scratch you're likely:
- At a company with huge sets of bare metal machines
- A massive organization that can afford to hire infrastructure experts to optimize every penny out of their cluster ops by vertically integrating their cluster management (rather than outsourcing to something like DO).
- Hobbyist who wants to learn how kube fits together.
If you're not one of those three then all you need is to setup Rook, setup some autoscaling component, pick out a few DB operators, setup monitoring, and setup a logging infrastructure. Compared to older solutions for similar things the "kube way" is a pre-planned roadmaps with minor choices along the way that are mostly personal preferences that weigh: "how much to I care about security", "does this need to be more or less debuggable", and more.
If you need a cluster, want a "managed-like" experience, then just go with Kops. If you just need a cluster to run your applications on just go with GKE, AKS, DO, or another offering.
I don't think declarative architecture and kubernetes are mutually exclusive. Seems like you could write a declarative file (terraform, cloudformation, etc.) that deploys a dockerized app to a kubernetes cluster in AWS.
Thing is; nixos has been architected for being declarative from the start. Whilst Kubernetes is a stateful REST API.
Sure you can emulate a declarative view on top of a mutable thing (Think React and virtualdom), but having a declarative core makes declarative deployments easier to manage and more elegant.
E.g. in Kubernetes automatically deleting a resource is still hard; It's hard to figure out when something is not "used" anymore.
In nix, you simply run `nix garbage-collect` and all the things that are not referenced by your declaration are automatically deleted.
Network effects and money mostly (hype as you say).
NixOps is a great tool..when it works. When it doesn't, it's not inherent. It's just that it is clearly not heavily invested in by any entity with a bunch of money.
I think is referring to abstractions in general. Levels and levels of abstractions. Not always a bad idea but sometimes it ends up complicating things too much and burning new comers brains.
The levels of abstraction seem to be a hazing ritual that seems to be a point of pride sometimes. I also imagine it's good job security for those who spent years on these tools.
I went from using Singularity to bundle scientific tools for HPC cluster useage at my university straight into admining OpenShift at my current employer and I have to say it was not a smooth transition. Felt like a complete idiot for a good amount of time. But after spending a few months with it I’ve really gotten to see the power and use for the k8s infrastructure and its value as a tool/platform.
And honestly, I’m digging it. It’s really cool stuff and once you get developers writing production services and applications that would otherwise have been virtualized on dedicated systems on it, they love it to for their speed of delivery and development. But we’re also managing hundreds of projects with multiple applications/services with high availability, so OCP/Kube makes sense for us.
Repeating my reply to this question asked by someone else above. I genuinely want to engage people on this question.
---
My problem with Kubernetes is people adopt it without understanding it. They adopt it because everyone else is without ever asking, "Why?"
I've installed and managed a simple K8s cluster before now and it was a nightmare (this was only a year ago.) It wasn't until we moved the management of the cluster to a managed service that it became worth it. It was at THAT point it was worth containerising the application(s) (another complexity to overcome) and letting K8s handle the deployment and scaling.
I'm worried that as an engineering culture we're not pushing back enough and saying, "Yeah K8s is cool but look: do you really need it? It's very likely you don't".
For me to automatically recommend K8s to my clients I'd need to see a (near) zero friction deployment options that's fully feature compliant and truely fully managed. It's at that point I would be happy to say, "Sure! Let's go with K8s and just write YAML files to deploy our containers!"
(And then there's the whole debate as to whether or not you even need a container around your monolith... probably not. Then the next debate about people starting out with microservices when we all know that's not a good idea.)
Repeating my reply to this question asked by someone else above. I genuinely want to engage people on this question.
---
My problem with Kubernetes is people adopt it without understanding it. They adopt it because everyone else is without ever asking, "Why?"
I've installed and managed a simple K8s cluster before now and it was a nightmare (this was only a year ago.) It wasn't until we moved the management of the cluster to a managed service that it became worth it. It was at THAT point it was worth containerising the application(s) (another complexity to overcome) and letting K8s handle the deployment and scaling.
I'm worried that as an engineering culture we're not pushing back enough and saying, "Yeah K8s is cool but look: do you really need it? It's very likely you don't".
For me to automatically recommend K8s to my clients I'd need to see a (near) zero friction deployment options that's fully feature compliant and truely fully managed. It's at that point I would be happy to say, "Sure! Let's go with K8s and just write YAML files to deploy our containers!"
(And then there's the whole debate as to whether or not you even need a container around your monolith... probably not. Then the next debate about people starting out with microservices when we all know that's not a good idea.)
I work with Kubernetes quite a bit, so I'm reasonably knowledgable on the pro's and con's.
I definitely think there is a misunderstanding about what Kubernetes solves. I think Kubernetes is very good at helping structure very large applications where a split of micro-services makes sense.
Most start-ups don't need it - it's likely overkill, and unless you go with a vendor you're likely to set it up incorrectly. But it is an advantage in much larger, spread out teams if the culture surrounding all the teams makes sense.
If sub-teams can handle their applications really well, which many companies have achieved, then it does provide significant benefits over monoliths.
I think the problem you're noticing is that everyone wants to jump onto Kubernetes without realizing why it's even there - the same with micro-services. I shudder when I hear that from my friends with no more than 10 people on their development team, but I think it makes a lot more sense when I see teams of >200 complaining that their huge monolith is what causes the most friction.
I'm referring to "algorithmic" in its negative usage as short-hand for what you can see as of late with regard to Twitter, Youtube, and Facebook's suggestion engines where more complex sorting and filtering is happening than a simple ORDER_BY.
Sure, not the best short-hand, but it seems most people understood what I meant.
Only if you subscribe to everything - if you keep your follower list small, the individual experience doesn't change regardless of the overall size of the platform.
I'm not an PG user but there does not appear to be an index on tweet.time - and even if there was wouldn't it better to do ORDER BY tweet.id DESC? I assume ordering by primary key is going to be the fastest, and would (ideally) prevent maintaining an index on tweet.time.
Sure, but you're making some assumptions in doing so. For example, that the time and id columns will remain constant. It might be a decent assumption in the beginning, but once you start doing updates on the table, all bets are off.
Yup, as soon as you need to change ID schemes, you risk breaking query logic. Sure things like v1 UUIDs and Snowflakes (and ULIDs and so forth) try to maintain temporal ordering in their ID formats, but what if you need v4 UUIDs for better clustering in some hash-table or SHA-256 hashes for some sort of content addressing scheme?
Also, it's just a very premature optimization in a world where CLUSTERED INDEXes exist. The database engine doesn't have to cluster by primary key, it can cluster directly by time (or any other index) if you ask it to. The power of doing it that way is that you can flexibly change it based on real performance issues (how do your execution plans look?) and characteristics (are you read heavy or write heavy? which ones are your bottle necks?), whereas if your application makes assumptions about ID format it's a lot harder to on-the-fly tune queries that all need to be rewritten.
I am not sure IDs are exactly incremental in Twitter at this point. I think they are issued by blocks or something of that nature. Someone here could answer that far better.
Twitter calls them "snowflakes". It's a 64-bit ID of that consists of a tuple of sorts of a timestamp, machine code, sequence ID. They start with timestamps to mostly give them monoticity in the direction of time (ie, sort order them), but they definitely aren't simple sequence IDs.
With the query given, the optimiser can immediately figure out to get the latest record from the index and scan back through the index. If the index has included columns, it could scan the data straight off the index. Without the index you need to scan the entire table, sort it in memory, and then read off the top columns. If you were doing a top X query it would be more markedly faster by fetching less data from disk. But I think that query is getting all the records, but still it will be quicker by avoiding the in memory sort.
interesting, thanks for the insight. I haven't touched DB setup in many years, and even then was novice. The best person I knew told me to index if a column wasn't unique, but also wasn't something with only two or three choices. Sounds like I have more reading to do...
I'm certainly not an expert - I did a great DB course about 15 years ago and then used the skills every now and then since. I might not be up on the latest. And I am more of a SQL Server person. BUT... the main thing I learn is view actual execution plans, and see what is actually happening before adding indexes (unless it's an 'obvious' index).
> The best person I knew told me to index if a column wasn't unique, but also wasn't something with only two or three choices.
Yeah I think this is too broad advice, and you need to understand what you want to achieve. Mentally, choosing indexes is like choosing whether to use a hashtable vs. for loop vs. binary tree etc in an algorithm in code. There is not golden rule or "always use a hashtable if there are 100 entries" type thing. You just need to figure it out on a case by case basis. And usually there are only 2-3 tables in your DB worth a lot of effort in figuring it out!
So, in fairness I didn't dig through the code to see the column, but assumed some granularity like milliseconds? It was a genuine question on whether it would speed up the query at all if indexed since it is generally grabbing everything then sorting.
There are good faith arguments for applying smarter sorting / filtering to the stream.
For example, if someone follows two people and one person simply tweets at 1000x the rate of the other, I would argue that it would be better UX if the rare tweet from the other person gets more weight.
Another thing Twitter does is show you popular things that have been liked by the people you follow. It might be way too much to show you every thing that every person you follow has liked every time. But it can introduce some interesting content to your stream to see such second-tier content.
I don't see how promoted content plays into it since you can interpolate it anywhere.
If only they weren't inane. I think Twitter is probably the worst in terms of 'promoted content' because it always seems to serve up the same ad several days in a row and it's always something surprisingly off-topic. I'd be browsing my feed full of political news, movie trailers, and animal photos... and get an ad for ketchup. How on Earth could I confuse that for a real tweet or something my friends would retweet? A ketchup ad? Mental.
This doesn't make much sense to me. You don't need complicated algorithmic sorting and filtering to inject an ad space after every fifth tweet. Even the client can do that.
You see many more outlier tweets with thousands of likes when your feed is interspersed with "most liked by follows and follows of follows", as opposed to just "tweeted by follows". I think that improves the signal to noise ratio of the feed.
But yeah, I wish a simple timeline was the default, and twitter stopped switching away from it.
For anyone interested in this, google “hacker news gravity algorithm” to see an interesting example of how sorting was done here at one time (as I understand it the algorithm has evolved).
I was working on creating an rss reader with a reddit-like interface that allowed follows, voting, comments, etc. The computational expense of a fancy sort algorithm based on factors other than just one non-computed column is immense once you get to a decent number of users. I ended up removing the algo just to lower my hosting bills. Of course, there were other potential solutions, but for a side project, this was easiest.
There are a lot of very strong opinions in this thread on whether this code would be "production ready". I don't think that is the point here. This is awesome because of its brevity, it is like writing a ray tracer in 100 lines of C. It demonstrates that a task which might be perceived to highly complicated can be reduced to a file, of which the size many of us would just define a single class in such a system.
It's a nice demo. But there seems to be a surprising lack of validation? Seems like anyone can delete any other user, or post as anyone else, etc. Then again, I can't read Haskell, so I don't know if there's something preventing that at a higher level than the endpoint logic.
Doesn't look like there is. I think the idea is for this to be a nontrivial example of a full-stack Haskell+Nix(Ops) project. The Twitter part is meant to be more familiar that robust.
It's fair to argue auth should be included in a nontrivial example though. I honestly don't think it would add that much clutter either. Although unlike the other components, I don't know if there's a simple off-the-shelf Haskell solution for auth.
Servant (the typed routing system being used here) has pretty awesome and robust authentication support. I bet it could be added without too many additional lines.
Small, easy to write/maintain approaches are at an evolutionary disadvantage vs classically "enterprise" styles.
Easier to work with -> fewer people needed -> fewer people hired -> fewer people learn -> fewer people know.
Now there's fewer people in the market who know Simple, vs many people who've learned OverEngineered (TM). Leadership wants to pick a language/style they'll be able to hire for as-needed. OverEngineered has a much larger talent pool.
import Database.PostgreSQL.Simple.SqlQQ (sql)
...
let index :: Handler Markup
index = do
tweets <- query_ [sql|
SELECT "user".name, tweet.contents
FROM "user"
INNER JOIN user_tweet ON "user".name = user_tweet."user"
INNER JOIN tweet ON user_tweet.tweet = tweet.id
ORDER BY tweet.time DESC
|]
Yeah, another comment said this is “unreadable” if you don’t do FP... I have done a little lisp + Elixir, but 99% of my coding has been in C-descendant languages and this was definitely readable.
It's readable because we know what we're expecting. Had the logic been just a tad more complex (e.g validation of input, login flows etc.), it would quickly become unreadable and unmaintainable to anyone not familiar with FP, or specifically Haskell:
- there are fifteen LANGUAGE directives (what do they all bring?)
- there's an ungodly amount of ASCII-art: What do any of these do, and in which contexts: ! @ :> :<|>
And so on: just three things that caught my eyes looking at the code on my phone
As for validation of input, what do you mean? Haskell tends to excel there as well, with the availability of parser combinator libraries like Attoparsec.
> LANGUAGE directives (what do they all bring?)
In a real project you hide these in a config file for your project. They're usually well documented, and you can usually guess what they're doing from the name once you have some context.
> What do any of these do, and in which contexts: ! @ :> :<|>
All of those except @ are just functions or types. You can look up what they do in the same way you would look up what any other function does. @ is the only one that's a syntactic operator, and it has 2 meanings depending on context. It's either specifying what a type variable should be, or it's pattern matching and binding a name at the same time.
You're not helping the cause of "this is 99% readable" ;) If you add proper authentication and parser combinators [1], and configs, and.... It becomes anything but readable to a person without specific Haskell and very specific library knowledge.
And yes, I do have exposure to FP having worked with Erlang for a number of years.
I also seriously detest the unexplained love for ASCII art among many lib authors in the FP space. Servant is one such example: :> :. :<|> ! At this point, why not use J :)
[1] By the way, unless I'm mistaken, there's next to zero useful documentation on Attoparsec, there's only barebones package documentation. And no, you don't validate data using parser combinators unless your data is only strings that resolve to some DSL.
I hope this is only the beginning of a series of popular web apps and social networks all recreated as pithy single file documents. Next should be Facebook.
This is cool and reminds me of the Twitter clone[1] I included as a coding exercise in my book[2]. It's a very good way to learn as a basic Twitter clone is relatively simple to implement.
I don't know that I agree, or at least that's not always the goal. I've found that the more "wheels" already in the framework, the more opinions the framework has, and sometimes I just need something dead simple.
Laravel, for instance, has so many really nice wheels, but it's organization can feel restrictive at times.
web.py, on the other hand, has basically no wheels but I can drop into a project really quick.
Maturity is a spectrum, and sometimes you don't need a wagon when a bike will do.
There are tons of Flask/Sinatra level abstraction frameworks. They bring nothing new to the table. Stuff like auth and migrations are difficult problems, which is exactly why you need a framework. Any halfway decent web dev can cobble together yet another expressjs lookalike, but there are few that can reliably build Devise type authentication solutions. Ever wondered why most build-a-blog web framework demo tutorials tend to conveniently skip user registration/auth?
Slightly related - If you want a pre-categorised tweets feed with ability to add new categories, I have written one - http://trysensible.com/ (Open source https://github.com/kgthegreat/sensible). It comes with some standard categories such as politics, sports etc. but you can add whatever categories you want such as - music, academia and start adding your own curated list of keywords to support this category.
This is written in golang and is open source https://github.com/kgthegreat/sensible
A lot of people complaining that this wouldn't be able to hold the load of the original thing like that. I'd argue that all platforms attempting to be all-encompassing is a big problem right now. Maybe a single instance of a platform should really just accept as much load as this could handle in this form. They could federate with eachother to replicate the massive connected structure.
Might be nice if this title was changed. This is a toy app which is guaranteed to blow up at scale, but if the point is to show off a particular stack then maybe it should say so.
The fact that it's in a single file implies this. I don't think anyone would think they are trying to claim they can replace all of Twitter's production code with this single file.
I don't think that's true. You could make an effort to show that you'd solved some of the more well-known technical problems, e.g., different messaging architecture to handle users with millions of async subscribers. As it stands this title need not mention Twitter at all. I'm trying to be helpful. If you make the assumption that it's just a simple clone that makes the title even less interesting. The target audience for this post will likely just skip over it.
