- She made three passes to fix the existing code base.

- After that didn't work, with the CTO's involvement, this Go solution was devised.

- It took only 2 weeks to ramp up with Go and build a replacement that solved the problem.

- The number of instances needed dropped from 4 to 2

This sounds like a resounding success.

So congratulations Alexandra and Digg. And thank you for sharing. The article was well written and I learned some things about how the Node.js event system works under the hood.

I totally agree that it's impressive that it only took 2 weeks to build the Go version, it just seems like it would have taken 2 days to try the worker pool implementation with Node.js.

I've moved from writing code (including golang) to managing large projects. My biggest concern now is meeting the three metrics of project ecstacy: 1) correct solution, 2) on time, 3) within budget. If one language appears to get me better performance on these metrics over another, then I'm interested, whether that language has generics or not.

Another concern from the business perspective is whether or not a language is easy to hire for, and gets devs more productive in less amount of time without creating a lot of technical debt in the process. I have a gruesome time dealing with this very issue, and if golang was the basis for my toolchain, my guess is my hiring concerns would probably be minimized to enough degree that it would have a positive impact on my business -- looping back around to metrics I mentioned.

Maybe the solution should have been kept in js, but I guess it wouldn't surprise me if the golang effort these folks just went through will probably continue to pay off in a substantive way.

Something is definitely blocking or resource constrained and causing "thrashing": the (uncontrolled) number of requests allowed to spin up at a time (which creates resource contention) combined with the fan out (1 request = 100+ S3 requests/callbacks) seem like a likely causal factor. As you said, a worker approach (with limits on the number of concurrent requests) is going to be similar to the golang approach used.

The golang approach makes the average execution time of a given request more consistent but the overall wait time may still increase (dramatically) if the arrival rate grows too high. (Classic problem).

Say "easily" fixable by adding servers? Partly true. What happens if the S3 calls slows down dramatically?

The project has only 9 stars on GitHub. While it might be a perfectly fine solution that fixes OP's problem, it certainly doesn't inspire confidence as a battle-tested production-ready library. On the other hand, worker pools and Golang go together very naturally.

The author just doesn't know how to approach the problem and copied a solution, which happened to be in go.

Nothing wrong with that. Just don't try to pass it a success story or as an excuse to bash a language.

This got me thinking - why did the single-threaded "green" threading programming model like how the Java JVM was initially implemented all those years ago fall out of favor? It would be as performant as Node presently is but without the difficult async logic for the end user programmer. Async programming and green threads are duals - both use event queues behind the scenes to drive the scheduling. The complexity of the actual I/O and timer event callbacks is hidden from the programmer in green threads. The pseudo "thread" context switches would only be done upon blocking I/O. No need for actual mutexes as the program is still actually single threaded. Stack traces of green threads within a program would be easy to decipher. Computation on a green thread would also block all I/O as it would in Node, but we'd know that going in - no different from Node in that regard. Just the programming model would be simpler.

Because of the way tasks are scheduled in node.js you can have head of line blocking; i.e. a task that is scheduled to run first can block all the subsequent tasks, the issue that the OP suffered from. You can sometimes reduce this by calling process.nextTick(), but you need to remember that it is only IO is non-blocking in node.js.[1]

[0] https://stackoverflow.com/questions/14409609/does-the-v8-jav...

[1] http://greenash.net.au/thoughts/2012/11/nodejs-itself-is-blo...

    let result = await call();
    nextCall(result)

"when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later)."

And in Go,

"the service has a worker pool and a delegator which passes off incoming jobs to idle workers. Each worker runs on it’s own goroutine, and returns to the pool once the job is done."

Since our transition to Golang we haven’t looked back. ... we’ve begun the process of modularizing our code base and spinning up microservices to handle specific roles in our system.

If you have more links, I'll be happy to add them.

I have a much longer list of companies that use Go (https://quicknotes.io/n/1XB0-companies-using-go) but it's harder to find descriptions of rewrites than find out who's using Go.

https://www.youtube.com/watch?v=bAQ9ShmXYLY

Nothing says "I'm keeping up with my professional development" than relearning IO libraries and build tools, and rewriting in-house code, over and over again, every time in the new langue-du-jour.

Instead of flamegraphs I use nodegrind, which I can use with KCachegrind/QCachegrind.

There are actually a whole host of languages that would let you approach the problem with this technique and get there, so the only thing special about Go is that it is syntactically simple (although it's conceptually quite hard to build reusable code because of poor generic support the resulting bad error handling), so it makes for an easy migration.

We could see similar stories about going from Node.js to Erlang, Nim, Rust, F#, C#, Haskell, or even the newer Python & Ocaml runtimes. So it's not like Go is special here. But it's timing is special, because it's a harbinger of the industry's slow and reluctant admission that pretending everything is single threaded is okay. Even I/O multiplexing with coroutines simply cannot keep pace with even casual demands on modern infrastructure.

A single Node.js instance runs its business logic in a single process. A single Go instance can run its business logic in multiple processes. Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core.

But the thing the author fails to mention is that there are a lot of Node.js modules which help you easily scale across multiple CPU cores. E.g. PM2 https://www.npmjs.com/package/pm2

Had the author of the article actually understood the real cause of the problem, he would have saved himself and his company a lot of time.

Nodejs is still outpacing Golang according to Google Trends and yet I have never seen a single article about switching from X language to Node.js. My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself.

If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines. Heck, my grandma could scale this thing across multiple CPU cores.

Personally I think Node.js is great because it encourages you to think about these problems sooner rather than later.

If Node.js used language X,Y,Z instead of Javascript, it would certainly not have been the success it was. "Javascript on the server" was the marketing/propaganda . Nodejs came at the right moment, Js was exploding, websockets were exploding, coffeescript and co were exploding in popularity.

What Go does better than Nodejs is a better standard library, true concurrency, static typing. With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code. Writing callbacks is tedious, promises are tedious, co routines with yield need plumbing and async isn't in the spec.

> you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines

With Node.js any heavy computation needs to be launched in a separate process, which is not the case with Go. sure you can have one/2/4/8 process dedicated to heavy computation, but it will still be the bottle neck. In the end you'll have to use a different language in order to solve the single threaded problem.

Finally there is no mystery as to why some businesses are moving from Node.js to Go: They didn't need anything Node.js provides at first place. They didn't need to use javascript, they didn't need to write "universal" applications with the same logic running in the client and in the server, they didn't need single threaded async programming.

Go is not a silver-bullet, but I was a bit sick of having to write a callback every time I wanted to make an http request, just to prototype something.

> With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code.

No, it can't, and pretending that it can is misleading in a way that allows large teams of developers to cause themselves real problems.

The only sense in which this is true is that you can write a Go function with a simple signature like this:

    func DoSomeStuff (error) {}

But, make no bones about it: this function blocks until it completes. This means that while this function may do concurrent work it is absolutely a blocking function. This is fine: sometimes blocking functions are good. But you cannot write a non-blocking Go function in the same way.

To make a function non-blocking you can return a channel out of it for the return value to appear on, like this:

    func DoSomeStuff (chan error) {}

Except...that return value just there? That's a Future. It's a terrible, half-implemented version of a Future, but that's exactly what it is. It's a promise to return some kind of result at some point when the underlying process has returned.

And if you don't want to block your current goroutine, you cannot block on that channel receive either. That means that you need a callback. There are two patterns for doing that: you could have some kind of central loop that selects over all channels like this and calls the callback functions (boy that looks a lot like Node's event loop, doesn't it!), or you can manually spawn your own callback functions in their own goroutines. Either way, you have callbacks and futures here: you're just building them yourself and calling them something different.

There are lots of good reasons to switch to Go: it's a language that makes lots of developers remarkably productive, it has an ingrained philosophy of building concurrent programs, it's pretty damn fast, and it runs on all kinds of awesome platforms. But claiming that Go has learned something magic and new about how to write concurrent software in such a way that you don't have to care whether your code is async or not is just not true: you always have to care.

The caller can make function "non-blocking" by wrapping the call in a goroutine themselves. (There's some subtle differences, but they are mostly irrelevant here). For this reason, I'd say there is (almost) no reason to introduce asynchrony in your API in the way you suggest. The rest of your post built on this example seems shaky to me, since it seems built on an example API that doesn't need to exist.

I'd say that "you don't have to care whether your code is async or not" is a overstating the case. I would append the qualifier "unless you're introducing concurrency". Considering that almost no low-level APIs are asynchronous, this usually happens rarely (or happens in low-level code like the HTTP server). Examples that have come up for me: making N parallel RPCs, writing a TCP server. In those situations, you care about async vs not.

In event-loop based systems, it seems like async is in my face all the time, even when doing things that are entirely sequential.

Sure, but if they want the return value then either they need to construct the Future-y wrapper I just described or they need to assemble it together in a collection of other function calls wrapped inside a function that itself is either Future-y or uses a long-lived channel to communicate results.

It is not novel to build up a non-blocking system from purely blocking method invocations. We've been doing that for years: it's called threading. Doing things this way has many advantages when written with appropriate diligence, and I'm not pretending otherwise. However, if you actually care about communicating between these arbitrary threads of execution than you either need Futures or queues (both of which are essentially just channels in Go), and at this point you've got the exact same problems as you get in NodeJS or any other asynchronous programming environment.

> The rest of your post built on this example seems shaky to me, since it seems built on an example API that doesn't need to exist.

I don't think that's fair: as I mentioned above, the fact that you as library author would not write the Future-y extension doesn't mean that the Future-y extension isn't built: you just force your caller to build it. That's fine, it's a perfectly good architectural decision (probably you should't be making those decisions for your user), but it doesn't remove the problem.

> I'd say that "you don't have to care whether your code is async or not" is a overstating the case. I would append the qualifier "unless you're introducing concurrency".

Sure. The thing that matters here is that Node is always introducing concurrency, because Node is concurrent. This is why all Node programs have to care about concurrency: they are all concurrent because their system is concurrent.

This is desperately inconvenient for many one-off programs, which is why I personally don't use Node for anything like that: I'd much rather use Python or Rust or Go. But that was never my argument. My argument was about OP's assertion that "with Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code. Writing callbacks is tedious, promises are tedious, co routines with yield need plumbing and async isn't in the spec."

The first sentence is dangerously misleading (while technically true, any system that does that is usable only in that one context), and the second one misses the point, which is that those things get effectively built anyway in any moderate-scale concurrent system in Go.

But my biggest point is this: Go isn't magic in regard to concurrency, and there is a weird amount of magical thinking around Go. Go is a very good language with a lot to like, and I like it quite a lot. But when boiled down to it, Go's concurrency model is threads with a couple of really useful primitives. And that's great, and it works really well. But it's not new or novel.

The sentence "with Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code" is equally true if you replace "Go" with "C", or "Python", or "Java", or any language with a threaded concurrency model. There's no magic here. It's the same building blocks everyone else is using.

As I said above, green threading has advantages over OS threading, but they behave exactly the same in terms of design patterns and potential bugs.

This is what I was getting at when I said "not that different": compared to the difference between event-loop concurrency and threaded concurrency, M:N green threading is basically just a subcategory of threading.

In normal Go, nothing is blocking in the Node sense. (Oh, if you put your mind to it you can manage it, but I've never once encountered this as an practical problem, either in Go or the equivalents you can do in Erlang if you put your mind to it.)

This has profound changes on how you write code.

It's true, Go is not magic. It's just another threaded language in most ways, with the "real" magic in the community best practices around sharing by communicating instead of communicating by sharing. In theory, you could write an equivalent set of C libraries and get most of the same things, but you'd have a lot of library to write. (This is why things like porting C goroutines to C have a hard time getting traction. It can be done, but it's actually the easy part. Also, you'd still be in C, which is its own discussion. But you can get the concurrency.)

The real issue here isn't that Go is necessarily exceptionally strong at concurrency, the real issue is that Node is exceptionally weak. It introduces this new concept of "blocking" that only exists in the first place because it is weak, and then makes you worry about it continuously, to the point that many people seem to internalize the concept as what concurrency is, when it isn't. It's really just something Node laid on you. So when you step out of Node, and you see a community that isn't visibly as worried about "blocking" as the Node community, someone trained by Node thinks they are seeing a community that "isn't good at concurrency". My gosh! Look how cavalier they are about "blocking"! Look how they tell people not to worry about it, and how casual they are about having users wrapping library code in goroutines and explicitly telling library writers not to do the concurrency themselves. But what you're seeing is what happens when you simply no longer have the problems Node and "event-based code" brings to the table. Go is not magic in the general case, but, honestly, when someone coming from the Node world picks up Go, I can see why they might go through a period where they sort of think it is. There are really differences in code style, and how easy it is to write correct code.

You have to make sure you're not letting the limitations of one connotation of "blocking" spill over into the other, or you will have problems. (True in both directions.)

To speak to someone else's point, "futures" in Go don't "suck", they basically don't exist. If you're writing in a recognizably "futures" fashion, you are not writing idiomatic or even particularly good Go. You don't need futures, because (what are today called) futures are basically an embedding of a concurrency-aware language into a non-concurrency-aware language, and you don't need them when the language you're working in is already concurrency-aware. That's why you don't see futures in Haskell or Erlang either. (I have to qualify with "what are today called" because the term has drifted; for instance, Haskell does have explicit support for an older academic definition of the term with MVars, but modern software engineers are not using the term that way.)

Lots & lots of idiomatic go code exists in that form (anytime you wrap a select that times out in a function you have a future).

Channels, what we are really discussing here, have 2 problems: the first is in abstraction, they don't provide basic primitives that other similar structures provide, like timeouts & cancellation. The second is in implementation. As futures you have to worry about all the edge cases around nil & closed channels. As queues they are highly contended.

All I'm trying to do is to make sure that people who make bold claims about abstracting away blocking code aren't misleading others: when calling other code you should always be aware of how it interacts with the flow control of your program.

Are you sure you're not just explaining how to write a Node program in Go? Write Node programs in Node.

To have an abstraction where you really don't care about blocking or not you need promises/futures. Go's futures are bad. Real bad.

If you don't want the function to be non-blocking then you are fine with either method signature.

  function isValidFoo(val) {
      ...return boolean
  }

To avoid a ripple effect, all functions have to be designed to be asynchronous from the beginning, but this increases program complexity significantly. It means avoiding built-in language features such as direct return values, and often means substituting async library functions for built-in looping constructs.

Other languages including Go aren't like that. You don't need to wrap a simple return value in a promise or callback just because retrieving it might someday involve I/O. Execution of the thread/goroutine simply resumes whenever the value is ready, and the function returns normally. In those languages, Node's distinction between sync and async is artificial and unnecessary.

You can literally say that about every labguafe that has concurrency abstractions.

The whole point of the argument is that golangs abstractions just push the callbacks (or blocking sync) to the application layer.

To me, idiomatic Go suggests that libraries, data structures, business logic, &c is encapsulated in blocking functions, and that concurrency is expressed in glue code.

The argument to me is that Go advertises itself as being easy to write that concurrent glue code & then fails to provide basic abstractions around it.

Everything you mention about things being expressed in glue code would be easier if Golang provided a modern promises library & holds just as reasonably for other modern languages that do.

The "node requires callback hell" is a red herring because so does Golang (that or blocking code), just it does it at the app layer instead of the framework layer.

What‽ It provides channels, which are a really nice abstraction for concurrency.

> Everything you mention about things being expressed in glue code would be easier if Golang provided a modern promises library & holds just as reasonably for other modern languages that do.

Have you actually written any Go? Using channels in Go is much nicer than using callbacks. Callbacks basically force one to write one's code in continuation-passing style, while callbacks let one work about sync points only where you need.

Channels are a concurrent safe queue, nothing more. They are not even particularly well implemented concurrent queues as they are highly contended.

What go provides for abstractions around those queues are range and select. Range is a nice way to make simple concurrency cases look like traditional for loops (and is very rarely used in practice as simple concurrency cases are rare, at least for me).

Select is a nice addition to the language, yet the edge cases around channels make it very difficult to reason about without keeping http://dave.cheney.net/2013/04/30/curious-channels open.

> Have you actually written any Go? Using channels in Go is much nicer than using callbacks.

I have, for 2 years, full time. I have not used node. Have you ever used a language that supports concurrency besides Go? Go does not provide, as part of its concurrency abstractions, things that are deemed bare necessities by other languages such as timeouts, cancellation, supervision, lock free structures, etc.

The entirety of the Golang concurrency story can be wrapped up with 3 things a) the culture of the language prefers message passing concurrency b) the scheduler is a very good example of when simplicity gets you great performance in the main cases c) select as a language keyword is interesting.

By leaving the basics out you push all of that work to the application level, which is why I think the callbacks argument is a red herring. At the application level you are either going to encounter callbacks (as in the stdlib http handler) or blocking code (subject to races, deadlocks, etc).

What I was respond to was this specific sentiment: "With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code." My argument was that that is simply not true. Client code must know the difference between blocking and non-blocking code because it affects the flow of information around the program.

Please explain.

As for the go futures being bad, channels generally are a bad concurrent queue (contended, lack basic abstractions, etc) and using a single item queue as a future isn't in and of itself bad, but does mean you can't optimize for different usages that futures might have over message passing queues.

This appears not to be true for lodash.

So upgrades are optional and not pulling the rug out from underneath the feet of devs.

Major releases allows the project to make corrections and avoid carrying around years of baggage.

But a standard library is a little different due to the sheer number of dependencies and level of coordination needed to upgrade them all. Go got it mostly right the first time, and the lack of churn is a major advantage.

    async function() {
      const {data} = axis.get('foo'),
            arr = ['foo', 'bar', 'baz'];
      return arr.map(ele => await axis.get(ele));
  }

    return arr.map(ele => await axis.get(ele));

    return arr.map(async ele => await axis.get(ele));

    return arr.map(ele => axis.get(ele));

That is, it returns an array of promises of values, not a promise of an array of values like erlich's function.

erlich's example is correct, with the replacement of for-in with for-of.

Edit:

Also, a solution using Array.map() is not impossible:

    return Promise.all(arr.map(ele => axis.get(ele)));

This is not the endorsement you seem to think it is.

> My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself.

Your guess? This is pure emotion arguing here. I have rolled my eyes at the golang converts for literally years and recently I decided to give it a try and it truly sold itself, sans marketing. I also really like node.js, for what that's worth. This comparison sounds like the hurt feelings of the True Believer.

Different tools have different applications. Don't ever forget it.

> Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core.

I'm not intimately familiar with Amazon, so to me it's unclear what "2 medium ec2 instances" mean. It might mean, 2 m3.medium general purpose instances, in which case they actually expose only one cpu core. They also seem a terrible fit for an IO/CPU-bound work load?

While I can sympathise with the author that it'd be nice to spend as little as possible on a (micro) service, it strikes me as odd that they first tried going 2x medium -> 4x medium, rather than 2x medium -> 2x x.large (possibly a different intstance type, like c4 -- and run 8 (or whatever) node instances per vm.

It does sound like this is a typical area which golang would be a very good fit (Reminds me of: https://talks.golang.org/2013/oscon-dl.slide ) -- But it also strikes me as odd that rewriting the (apparently) simple service in Go ended up looking cheaper than a) scaling up (rather than horizontally), and b) rewriting the service in nodejs (but with an architecture more suited to the problem).

Node is not a speed demon. It's not as bad as Python, which can be insanely slow. But if you use Node, your server costs are always going to be some multiple of what they could be. (Also applies to RoR.)

Of course the usual argument is that servers are cheap and devs are expensive, so you'll be quicker to market with Node and that's worth the extra cost.

Which is fine until you need to run at speed and/or scale. If you start having to pile on the instances, you will be wasting money. If you're big enough to need hundreds of instances, you will be wasting a lot of money.

IME it's debatable whether Go can't also be faster for development. There's a learning bump at the start, but I found it easier to write code that just works, easier to write good tests, and easier to deal usefully with errors.

I am so tired of reading hacker news articles with comments that are so pushed down because of subjective opinionated voting that is so obviously skewed towards one personality type.

I have so many thoughts and opinions I want to contribute to these discussions but I have zero confidence it will fit within this type of environment that's been created.

It really saddens me to see some of the comments that get downvoted here.

At its worst, I've seen people get down voted for asking beginner level questions. The calibre of folk who post here can, by its nature, make commenting a nervous activity. Getting down voted for trying to expand your knowledge is both harmful and wrong.

If I see a comment down voted without the above properties, I up vote to neutralise whether I agree or not. If the comment contributes to debate or discussion, it should be valued. I know I value alternate perspectives.

The fact that down votes are silent makes things worse. Great, you've been down voted. Why? Ask why, down vote again. It's almost a recipe for gradual shaping to conformity of group opinion for anyone who wants to participate here. It can then come across as say these things, don't say these things and you'll be sailing.

I don't know, perhaps an explanatory comment should be encouraged so that real education happens. Otherwise its almost Pavlovian in nature, especially for the less confident and highly anxious.

Given the way HN is designed (comments are ranked, low scores get grayed out, user karma scores are visible) if people downvote to express disagreement it would worsen the experience of reading comments, and, I suspect, the overall conversation.

she

There are lots of bad things to say about Golang, but it's pretty wrong to suggest that PM2 makes it as easy to write performant systems as Go does. Go even has a really good race detector to help catch the few cases where you can get around channels.

The idea that a language cannot directly and transparently support scheduling across CPU cores is an antiquated one. Java is abandoning it. C# abandoned it long ago. Erlang showed long ago that we can do it reliably and with great performance.

Heck, even browser-level Javascript is moving away from this idea with async support, although everyone seems rather reluctant to make the plunge towards a stackless architecture.

> If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale

I've got 13 years experience, 10 of which are as a distributed systems engineer. I've definitely hit the absolute edges of what off the shelf opensource software can manage, and more than once butted up against bugs and limitations of language runtimes. I'm extremely qualified to say that this is wrong.

Sometimes re-architecture and rewriting is the only way forward. That isn't necessarily a failure of the thing to be replaced. It's not even a failure of the architect if it is anticipated and managed correctly.

This is one of the most absurd and myopic statements I've seen in a while. Go has a vibrant community because many people chose it, for reasons that are entirely valid. Just as people chose Node.

You're posting with an enormously superior tone for someone that didn't even read the original post (newsflash: they were using a 4 host service cluster).

But there are times when it can be a pain. For instance, a node app I run at scale maintains an in-memory read cache that rarely gets updated. When we scale with processes, we end up having to duplicate the cache across every process. We could set up a helper process and have it manage the cache over a socket, or possibly write a c++ module that shares some memory but at that point it's no longer simple and/or there are more parts to potentially fail.

Yes, Redis can be the right tool. But, note how quickly we went from "I just want to run on multiple cores" to "I need to share data between my cluster workers so let's install Redis". It's just a total blowup of moving parts.

Great-grandparent argument reeks of elitism. I know how to deal with concurrency, parallelism, avoiding data races. Doesn't mean I need to waste my time implementing the primitives every project. To say that the language forcing you into multi-process architecture is a benefit, is a pitifully desperate remark. I wonder what single digit percentage of projects actually go into that territory? Maybe even less than 1%? Maybe with a runtime suitable for the task (e.g. Go or Java or hand-woven C++), less than 0.1%, because you do not run into pointless runtime bottlenecks to begin with?

I have implemented such projects, because we needed the workers to be geographically distributed, not because the language runtime fucked us over. This was a decision made in DESIGN not in DEFECT REMEDIATION.

I can think of a handful of "single-purposed" reasons to use Node on the server, such as DOM rendering (and other stuff that is only available in Javascript-land). But a serious project, no way. Not sure how people can stomach the inherent lack of safety in the language, the huge (relative to comparable substitutes) number of limitations in the runtime and the ecosystem churn.

Ummm, both of them run in exactly one process, this is how operating systems work. Go can do it's n-m scheduling across multiple _threads_ sure, but threads are not processes (unless as an implementation detail of an OS). If you are running lots of instances of Node, that is a different thing.

Process to process communication has tremendous cost, although shared-memory does make that faster, but at the cost of complicating the interaction. Message-queue (ZeroMQ) based interaction also an option but, by definition, it cannot be as fast as languages that have concurrency support built-in.

Reasoning about concurrency is hard, and languages like Go (via CSP), Erlang and Scala (via Actors), Java (via Threads), Rust, and Clojure (via STM) take different approaches to make this less of a pain.

CSP and Actors are what seem the most 'natural' of solutions. And that's a major reason why Go and Erlang are frontrunners in the concurrency race.

Actors are pretty nice when it comes to distributed services, and Erlang/OTP is pretty much uncontested here.

Threads with manual locking are only useful when speed is really important, but this is somewhat of a moot point when it comes to concurrency. Both STM and Actors perform slower (starvation, message copying overhead).

Here [0] is a nice, in-depth discussion about some of the different approaches to concurrency.

[0] https://plus.google.com/109566665911385859313/posts/FAmNTExS...

Akka.io on JVM could pretty much do what Erlang can on BEAM, functionality wise. Although some people have raised concerns over it being a "pseudo" solution and that it is a no match for Erlang.

I'm really curious as to why this is neccessary - what are the 10-100 keys you are fetching per request ? Just reading the high level article it sounds like there is some issue with what this service is doing, irrespective of the language it's doing it in.

Might have been a day-long workaround rather than a weeks-long rewrite.

Or, coders are getting worse at master the more powerful language...

Tricks are just that: tricks. No one sets up a business based on how you pull a rabbit out of hat, nor should they.

I do not know much of JS, but people write C++ with loads of patterns are prominent examples of people who do not master the language. That's not trick. That's real engineering with (often) bitter experience.

FTFY

I understand the eventloop/nexttick/etc architecture of node, but i dont understand how in this case he was blocking the loop? Shouldnt all the operations to S3 be async (and thus non-blocking, even if waiting for a timeout)? What was the specific part in this scenario that was causing the loop to stall?

I wonder if using something like async.eachLimit would have helped; it might prevent the s3 batches from flooding the loop & give a chance for web reqs to interleave, but probably at a cost to the median resp time.

2013 - Making switch from Ruby to Node

2016 - Making switch from Node to Go

2019 - Making switch from Go to {hype}

Of course, it's good to be skeptical of the current hyped flavor of the moment. But some languages really are better than others for specific tasks, and new languages do sometimes improve on their predecessors.

I make it a rule to only jump on the new bandwagon when I know it's not going to explode in 3 months and currently isn't on fire.

2019 - switch to PL/1 and raw disk sector addressing because bigdata smoothie silicon googley disrupt the world

I've seen this very recently personally, where I was accused of adopting hype for switching to Swift for a major rewrite of my app, where I'd pretty much start over in Objective-c anyways so I may just as well do it in a safer, much nicer language that's going to be the future of the platform eventually anyway...

It turns out that you don't have to switch - if your current stack works well for you stick with it. It doesn't make you "less cool". But it also doesn't mean that somebody else doesn't have a legitimate reason to adopt the new technology, and this is a pretty textbook example of what I'd call a justified switch.

If Unix weren't so committed to doing I/O the stupid way around, maybe Node wouldn't have to provide fragile, ersatz asynchronicity the way that it does?

* All data has a type

* All I/O is asynchronous and interrupt (event) driven

At its inception Unix doubled down on stupid, first by encouraging data to be stored and migrated in the form of untyped text streams, which require ad-hoc parsing and unparsing logic at the endpoints; and secondly by failing to provide asynchronous I/O primitives to user space. POSIX later provided an AIO standard, but it was cumbersome to use, bolted on rather than integrated into the OS, and virtually nobody used it. What you want to be able to do is submit I/O requests to the kernel, and have the kernel notify you when they are completed while you wait or do other processing; you could even put the process to sleep until pending I/O is completed and have it take up no CPU during this time. Instead what people do is burn CPU cycles in select/poll loops, which are the I/O equivalent of asking "Are we there yet? Are we there yet? Are we there yet?" over and over. You can fake asynchronicity by doing a bit of processing before asking "are we there yet?" again, but you have to write your processing code in such a manner as to be done piecewise in a loop, and the lower latency you want the more frequently you have to poll (and the more CPU you have to burn polling). This is how Node does "asynchronous" I/O; the VM stops every few instructions to poll for ready FDs, dispatches to callbacks as necessary, and performs parts of pending large I/O operations which can be done non-blocking. (That's the other sucky bit: you can't just call read(2) and write(2) for large chunks of data on an fd that's been opened O_NONBLOCK and expect it all to work; you have to keep reading or writing in a loop when the fd is ready, subtracting the number of bytes successfully written until the entire buffer is processed.)

But if something -- a call into a C library for instance -- stops the VM from doing the poll and I/O bits of its main loop, all I/O simply... stops. And your throughput goes into the toilet. Whereas if the runtime had been based on an OS that natively supports interrupt-driven AIO -- like VMS, Windows, or AmigaOS -- it would be extremely difficult to stall the I/O pipeline completely this way. You might be able to stall further I/O calls with a really long operation, but any pending calls already initiated would complete, and the runtime would be properly notified of their completion.

And all this stems from the fact that Unix was the Node.js of its day -- an environment designed to make things easy for casual programmers, not to do things correctly.

I tend to agree with you here: it'd have been better had Unix offered a typed abstraction over byte streams, but OTOH its approach really was a Worse is Better deal. Building a large set of tools which can deal with typed streams (or even trees, or graphs …) takes a long time, and is error-prone: building a few simple tools which treat everything as text is quick & easy. Of course, it does mean that the next four and a half decades are spent dealing with a lack of types, but it means one can ship quickly.

> Instead what people do is burn CPU cycles in select/poll loops

I thought that select/poll blocked, but it's been a long time since I did anything low-level like that, so I could be wrong.

> And all this stems from the fact that Unix was the Node.js of its day -- an environment designed to make things easy for casual programmers, not to do things correctly.

Well, Worse is Better.

Well, yes. That's just the problem. ALL I/O ops in Unix block. Even the ones on O_NONBLOCK fds, which simply read or write -- blockingly -- until the underlying kernel buffer is full/empty, then return.

You don't have the option to request I/O, do some processing, and then wait until the I/O completes. If you want to interleave processing with I/O you have to buzz in a select loop, doing your I/O and processing in tiny explicit chunks. Or spawn subprocesses to do I/O, leadibg to more complexity and confusion.

The facilities in the Windows kernel for I/O are strictly more robust and flexible than what Unix (eve Linux and BSD with kqueue/epoll) provides. The designer of the Windows kernel, Dave Cutler, also designed VMS, knew what real-world I/O requirements were, and famously mocked the way Unix handles I/O.

That's what select() does though: you give it a timeout, and it goes to sleep until either an fd becomes ready or the timeout expires. As an example, the X server on you typical linux laptop uses select() on a whole bunch of sockets, one for each client, and it both has good enough latency to be interactive, and doesn't burn a lot of CPU doing so. Is this the most convenient programming model for this? Probably not. And of course it still has the issue that a long-running function can delay your getting back into the select loop.

Select is still not nearly as flexible as Windows I/O completion ports, for instance. You cannot kick off an I/O operation, do some other processing, and then wait for the I/O to complete before you use its results with a select loop. You have to do a little bit of processing, pump and poll, do a little bit more processing, pump and poll, etc. You are wasting cpu time compared to the interrupt-driven way, and if you have many processes going at once this can add up in terms of battery power used and latency suffered. Remember that select(2) and poll(2) are system calls -- and all the overhead that goes with that.

The X server, in its effort to implement a GUI which is by nature event driven, historically contained all sorts of hacks in order to simulate the asynchronicity which should come from the OS. And it has always been slow and laggy compared to its counterparts on Windows and Mac OS -- let alone the Amiga where interrupt-driven programming is the norm and the GUI has always been flawlessly responsive even when other tasks are going on.

There's also the fact that nonblocking I/O IS NOT ASYNCHRONOUS I/O. If the kernel needs time to perform an I/O operation -- for example, reads and writes from/to disk -- it WILL block your process doing so. All "nonblocking" really means is "read until read buffer is drained, write until write buffer is full, then return". Your process is still stopped during the read/write! With better primitives such as Windows I/O completion ports that map more closely to the asynchronous I/O subsystems of the underlying architecture -- DMA, interrupts, etc. -- that becomes a non-issue. I/O happens transparently in background from an application (and maybe even kernel) perspective, and nothing is blocked.

In short, nearly anything at all is better than the Unix model in terms of throughput, latency, and more natural programming style. Unix is actively regressive.

One feature that was implemented in the Amiga OS, BeOS, and just about nowhere else is the concept of datatypes. Third-party software could register OS-wide readers and writers for their file formats, allowing files of that type to be consumed or produced by any application.

PowerShell lets you construct pipelines of typed objects with fields and methods, not just untyped dumb text.

What's the answer for that in Erlang or Elixir?

Why wouldn't one use python for this job? Since the company already does a lot of python, and this job is actually easy to do in python (I've done something similar).

400 processes with python3.5 is way less than the 4gig on one medium instance (less than 2.5MB each). Just farm the work out to a ProcessPoolExecutor, and have a timeout on the S3 get requests. That would let you have enough resources to match the spike of 20000 requests per minute (334 per second).

A lot of an S3 GET request could be the SSL, AWS auth and such. All quite CPU intensive. So using an async framework that doesn't do SSL+AWS auth async is obviously not going to work well once the requests go up.

There's even an example in the concurrent.futures of downloading urls [3].

Made with the beautiful python3.5

  import concurrent.futures
  import requests
  from awsauth import S3Auth
  ACCESS_KEY = 'ACCESSKEYXXXXXXXXXXXX'
  SECRET_KEY = 'AWSSECRETKEYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
  
  URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/',
          'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/']
  
  def load_url(url, timeout):
      return requests.get(url, timeout=timeout).data
      # return requests.get(url, timeout=timeout, auth=S3Auth(ACCESS_KEY, SECRET_KEY)).data
  
  with concurrent.futures.ProcessPoolExecutor(max_workers=400) as executor:
      # Start the load operations and mark each future with its URL
      future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
      for future in concurrent.futures.as_completed(future_to_url):
          url = future_to_url[future]
          try:
              data = future.result()
          except Exception as exc:
              print('%r generated an exception: %s' % (url, exc))
          else:
              print('%r page is %d bytes' % (url, len(data)))

[1] https://coderwall.com/p/rlguog/nginx-as-proxy-for-amazon-s3-...

[2] https://nodejs.org/api/cluster.html#cluster_cluster

[3] https://docs.python.org/dev/library/concurrent.futures.html

out of curiosity, what tool are you referring to here?

Here's an upload example: https://github.com/jamesmarlowe/lua-resty-s3

https://www.npmjs.com/package/is-positive-integer

  > And those 4 load-balanced instances we were running Octo
  > on initially? We’re now doing it with 2.

As everyone has mentioned this seems to be a simple case where the rewrite is a completely different architecture, which could have been done in Node.js.

I'm interested in why this continues to happen.

Seems like Node.js is a victim of its own simplicity. The baked-in non-blocking concepts of Node.js makes everyone feel like single-threaded is the only way to go, and think that Node.js is inherently flawed when it comes to parallelization.

Because Go doesn't give you a simple and relatively scalable solution without even thinking about it like Node.js does, instead you have to choose what to parallelize and think about the problem more. With PM2 scaling Node.js over CPUs becomes incredibly trivial.

Also, I think programmers naturally like rewrites/starting from scratch, and a new programming language provides an opportunity to rewrite everything (which the OP from this article and every other "switcher" article seem to do).

The psychology is easy to explain. The nodejs.org home page says "Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient". And the about page goes on to say "As an asynchronous event driven framework, Node.js is designed to build scalable network applications".

But for that supposed efficiency and scalability you pay a price: You must structure your code according to a particular optimization technique instead of structuring it according to semantic coherence. You lose modularity and clarity to gain scalability. That's the deal.

And then one day you realize that the whole thing doesn't scale as well as it could on a given hardware and it isn't very efficient unless you bring back some of the supposedly heavyweight architectural principles that you left behind.

At that point doubts start to creep in: Did we use the right tool for the job? Did we turn our code base into callback spaghetti for nothing?

And it's not that surprising that sometimes people opt not to keep paying the price in terms of code quality when they lose the supposed benefits they were paying that price for in the first place.

Since the application relies heavily on sharing resources between sockets I found out the hard way that concurrent programming in Go is not much easier than in Java. Sure, it has a better syntax and channels but compared to Node where you never have to think about things like these the complexity is way higher.

Long live Node.js!

Which resources?

I'm not bashing Go here, the memory usage is about 3x lower (which was the most important for me). And for a static language it's pretty cool.

NodeJS is a victim of basing itself on a terrible scripting language that was thrown together to wire up basic event handlers to HTML buttons.

I just felt like I was switching rafts midstream if I went that route so I had to say no. When it comes to deciding which is better, being able to find Node.js developers is a huge plus. Also, there is a lot more resources for Node.js which might make development faster in an industry where the first out the gate is often the winner.