Observables are actually terrible at backpressure as they're one of the most eager models around - once you have subscribed to an observable _it_ decides when to push and how much and you can't handle backpressure very well.
Quoting Erik Meijer who invented Rx Observables:
> My opinion about back pressure is that you either have a truly reactive system where the producer is fully in charge and the onus is on the consumer to keep up; or a truly interactive the consumer is fully in charge and the onus is on the producer to keep up. This is one of the cases where having different types of collections makes sense (just like we have different collections like lists, maps, sets, ... with different performance characteristics and different operations)
You've got certain misunderstandings there. Observables aren't eager, what you want to say is that they are push-based.
But that doesn't mean they are terrible at back-pressure, in fact they can be better at back-pressure than any pull based model you can think of, because the communication is more efficient. Back-pressure is the scope of the http://reactive-streams.org protocol and I've also built a library inspired by Rx that has back-pressured observables, started before RxJava went this route as well: https://monix.io
Funny enough back-pressure doesn't help with just "slow consumers". In Rx.NET flatMap is an alias for mergeMap because it is safer than concatMap, as concatMap would require buffering without back-pressure. Introduce back-pressure in the protocol, and concatMap becomes safer and much more efficient if done wisely. Which is cool because it is concatMap that's the monadic bind. And many other operations become more efficient and easier to implement because you no longer need to do buffering all over the place.
In terms of those AsyncIterators, or whatever they are called, I tried that route but the model is terrible as you're forced to introduce asynchronous boundaries on each pull. This means the model is very inefficient.
And I'm going to make another claim. Erik Meijer doesn't know what he's talking about in this case, in spite of how awesome he is and this opinion is probably the reason for why the Rx.NET library is trailing behind most Rx implementations. If I'd have to use .NET for a project, I'd probably end up reimplementing Monix, as Rx.NET isn't suitable for my use-cases at all.
No, they're eager - I had this mixed up too until Erik Meijer set me straight:
> Now back to your question. I do not understand what you mean by “observables being fundamentally lazy”. I’d need to understand what you mean by that before I can give a coherent answer. Iterables are essentially lazy, because nothing happens before you call “moveNext”, but I’d categorize observables as (hyper) strict since “onNext” gets called wether or not the consumer wants it (so to speak).
I think I understand the eager confusion. You mean eager because it is a push model and not a pull model like iterator which is lazy.
They are eager once Subscribed. The lazy part that I and the previous poster was alluding to is that most observables do nothing till they are subscribed to unless they are hot where as a Promise and Future are executed/scheduled immediately and the value is pushed/pulled (respectively).
I totally agree that Observables are not that great for backpressure but really there isn't a silver bullet on that one regardless (many smart people have tried to come up with solutions that will work across domains but have failed). As you mentioned you just have to pick the right tool for the job/requirements. I for one prefer a broker ACK model like RabbitMQ but many do not like queues as it just shoves the problem some where else.
The addons in the newer RxJava version where the the consumer can request a number of items and the producer might not overflow the window makes that better. The same model is also used in reactive streams.
But I would agree Rx is not the best possible model for streams with backpressure. Besides signaling for backpressure I also have the concern that many operators don't have support for backpressure, automatic backpressure to multiple consumers is hairy and that the push based system is also a lot more prone to resource (or information) leaks - because the producer might push something into a buffer which is never read by anyone if the consumers cancel the stream.
I still think Rx is a great tool for some use cases (e.g. inside UIs for reacting to button presses, state changes, etc.) and I'm currently using it a lot in angular2 to connect between information providers and widgets.
But for lower level data streams with backpressure between single sources and sinks, distributing resources and data exchange between multiple threads I'm now favoring other approaches.
I've experimented with a lot of designs over the last years and in the meantime I favor pull based approaches (like Future<T> getNextItem() or in Go simply read(buffer) methods there. I get backpressure naturally if I only invoke a new read after the last one has finished, I can even reuse a single buffer for all reads and composition also works through normal functions. Drawback might be throughput, but on bottlenecks one can still introduce a buffer.
Node seems to favor it's own stream implementation (ReadableStream and WriteableStream instead of Rx, which is especially designed for this use-cases, but it's dual nature (switch between push and pull based) can make it hard to understand in some situations. The whatwg streams approach could be promising, but doesn't seem to be fully specified yet.
Quoting Erik Meijer who invented Rx Observables:
> My opinion about back pressure is that you either have a truly reactive system where the producer is fully in charge and the onus is on the consumer to keep up; or a truly interactive the consumer is fully in charge and the onus is on the producer to keep up. This is one of the cases where having different types of collections makes sense (just like we have different collections like lists, maps, sets, ... with different performance characteristics and different operations)
You probably want an async iterator for that