Interesting self-hosted alternative to Firebase. I have some questions (as someone seriously evaluating this for use)!
* How mature would you say your code is?
* Is anyone notable using this in production?
* What kinds of bugs have you seen so far, and what do you think the
biggest obstacles will be for you in the future?
* Is anyone working on this full-time? Do you plan to make money off of this?
Yes, there are quite a few production use-cases. The latest app going into production using deepstream io for both web and mobile is https://briteback.com/
* What kinds of bugs have you seen so far, and what do you think the biggest obstacles will be for you in the future?
Mobile devices with flaky connections are always a challenge. So are many users simultaneously working on very small datasets
* Is anyone working on this full-time? Do you plan to make money off of this?
Yes. We're a full time company, based in Berlin. We'll eventually add a PaaS offering based on deepstream (deepstreamhub.com), but that is still quite a bit down the line.
The "Under the hood" visualization is awesome. I think more projects need to take the time to communicate details like this (without just big blobs of text or links to big piles of source code without a high-level overview).
Personally, I think the time spent on such visualizations is better spent on proper documentation, which is often inadequate. And by proper documentation, I don't mean one sentence per function. Also more useful, I think, is to enhance the documentation with comments for feedback or general user discussions. In that case, if the documentation is lacking, at least the user base can educate itself.
> If anything, most opensource projects fail to do proper marketing.
Can you explain why opensource projects need such visual marketing? Shouldn't we choose the tools which are simply better in a technical sense, rather than the tools which have the flashiest pictures?
But I can't count the number of times I've gone to an open-source project and couldn't even figure out what it did.
And I don't consider that to be the lowest bar – any project that wants to get decent adoption should not only tell me what it does, but tell me why I should use it and not it competition. And do both of these things without me having to read the source code.
Just this week - well perhaps last - I was at the homepage of some new language – and I couldn't find any of this nor a code example. This might be a technically fantastic language but if they don't communicate that adoption is going to suffer - and when it comes to communicating pictures have a place.
It had word of mouth and was the only free implementation of an OS that ran ok. Now there are 50,000 tools all free and no one hears what anyone says with all the noise.
I think you're doing a disservice to marketing. Whilst I think a lot of it is just made-up hocus pocus as much as the next engineer =), how you present something is important. It's like being a genius inside your head, not not being able to express yourself.
Ultimately, if you combine a good technical foundation with solid communication, you have a winning combination.
Like it it not, that's how the world works. Nobody cares if you have great code and awesome tests, if you have poorly written documentation and you can't tell somebody what you do in a few sentences. Don't underestimate how difficult it is to get these last two right.
But in this case, communication is not to random people but to fellow engineers.
I've never heard anybody complain about the graphical communication of gnu.org, yet everybody seems to have been using the gnu C compiler for the last decades. Just an example.
Finally, have a good look at that picture: http://www.deepstream.io/#segment-diagram
And be honest, what does that tell you as an engineer? And what would you take from it as a "random person"?
That's a challenging question: It's almost exclusively bought by large enterprise customers with a multi step request for quote acquisition flow - so in order to stay economically viable it's either ramping the price up or go for a consultancy model. What do you think?
I guess it depends on the cost of acquisition, I've not dealt with large companies for a while but they can be an epic pain to sell to, on the other hand the consultancy model is a hard thing to get right, the better your product is documented and easier to use the less likely you are to get the consulting work which is a kinda perverse incentive (cynically some of the the enterprise stuff I've run across seems to have lots of value and horrible documentation).
I guess it comes down to what percentage of the larger companies would want to pay the consulting/support rates, We don't pay support for anything since we largely use stuff that is well documented or very popular largely because we've been burnt on the consulting stuff before (no issue with paying a consultant on a product but when they are less technically competent than the person they are supposed to be helping that stings). Since you are the author that is extremely unlikely to be an issue though.
I guess the upside of going the pure consultancy route is in adoption, if a company can adopt your product commercially and then it becomes a core part of their offering they are more likely to pay since it becomes a critical part of their business, that's always a nice place to be - ask Microsoft/IBM :).
Currently if a conflict occurs it’s reported back to the client that triggered it and its up to it to resolve it. We are currently working on configurable merge strategies on a per record level
Great question. The granularity of your synchronization impacts the user experience, and sometimes the product offering itself. It can go from "replace the whole object" all the way to "string operations are merged without conflicts to match user intention", with "centralized resource locking" somewhere along the way.
The goal product-wise is to ensure intention preservation, but that is hard to achieve in general as intention is tied to what the product does.
For instance, if you have an integer and an increment operation, say, in a HN-like app to track upvotes. If your synchronization simply resets modified values with a newest-win approach, the following situation would lose an upvote: you have 5 upvotes, A and B both upvote you simultaneously, each setting the number of upvotes to 6. The server tells A that the number of upvotes has been set to 6, then tells B the same thing, and they are both happy. However, product-wise, it should have been 7 instead.
That common approach is both elegant and tricky. It requires very precise clocks and time synchronization, as some non-commutative operations are order-dependent (such as list insertion).
That is why Google's TrueTime API (introduced in Spanner[0]) is such a big deal.
Actually, for most use cases, newest update wins is sufficient if it can be done on a fine granular basis (one property of an object/document).
It's what web applications have been doing for forever and being a 'realtime' framework doesn't change this if your use case isn't something like google docs.
Is there some way to get this working with Safari's terrible audio (don't even get me started on video) streaming api?
I know this isn't your fault, and I'm not holding my breath. But the landing page is so beautiful, it felt like there was a chance you had answered my prayers haha!
Most important tip for any WebRTC call: Mute your own audio output, that completely ruins it. Other than that, here's a list of tips we've found useful when building the WebRTC feature: http://deepstream.io/tutorials/webrtc-tips.html
I guess I was wondering if it was possible to request an audio stream from a safari user (rather than play one to them).
My understanding is that the html5 api calls for this return null (i.e. they are unimplemented).
I guess I was hoping that you guys had polyfilled in a (ew) flash plugin which is the only way to do this at the moment, giving google hangouts a monopoly.
Maybe I have to write a framework ha! Even better, expose an api for me to build this into via a community package and make wrappers for meteor! (lol)
@wolframhempel or whoever is making this. Thanks for the making it. However, your webrtc demo doesn't work: both of the links are broken on github: https://deepstream.io/tutorials/webrtc.html
Can we see some live working demo.
Also the example on that page is also not working because adaptor.js is not found, but it says that your browser is not webrtc compatible.
It's a new project, but it's made up of very established modules and libraries. It's an aggregation of a bunch of tools which work well together but without locking you in to any of them.
- Deepstream.io is a free open source server, not a PaaS offering
- Deepstream offers pub-sub, request-response and web-rtc call management in addition to data-sync
- Firebase’s data-sync approach is based on one large chunk of JSON data that allows you to observe and manipulate sub-paths. Deepstream does the same, but breaks the data down into individual units, called records
- Deepstream uses a functional permissioning model, allowing you to interface with other systems (data-bases / active directory) for user-management, as opposed to firebase’ configuration based permissioning approach
Question, how can I expand Deepstream.io? For instance I want it to become an install-able API server for blockchain (like Ethereum). How do I do it? Is there a plugin architecture in place? Would I have to modify the main source code?
Funny to see so many pubsub things on the home page today. I just finished making a NodeJS CLI app that syncs the song you are playing on spotify to others (https://github.com/jonovono/spotluck)
I wanted a pubsub service so I could send the songs around. I wanted to be able to use it in a open source app and ideally client to client.
Although I could only look at the surface up to now, this looks really interesting!
I always wondered why there are dozens of RPC-only and Pub/Sub-only protocols out there and nothing that is really suitable for record/property synchronization. Because this one important feature in my-domain I have implemented something quite similar to deepstream for my needs, but currently a little bit more limited (pure client-server model, there are no client-hosted RPC methods, no auth, etc.). I chose to make synchronization unidirectional, which avoids the need for merging and conflict resolution and works good for my use cases. There I would have a model where I say the client must call an RPC method on the server, that method would change a property and this would then get synced back to client. I guess that would also be possible with deepstream, but I woul need to configure it in order to let only a single client manipulate the property.
Some tech questions:
- Are there any ordering guarantees? E.g. between different RPC calls as well as between RPC calls and messages and property updates? Giving those guarantees can increase the difficulty in implementation a lot, but reduces complexity in API design.
- Have you also thought on meteor like optimistic updates? I thought it can probably be implemented by putting some additional information in the results of RPC calls - some information about which properties where updates as a side effect of the call and in which version of these properties the updates are applied.
Records use a version number for every update to keep track of missing messages. Events are currently unversioned and won't be queued etc. while a client is offline.
Really nice to see Deepstream being on the front page of HN!
I'm currently using Deepstream in production for one of our clients in-house applications, so unfortunately I can't share a link. I'd like to give a few words on why I chose Deepstream.
TL;DR
Deepstream is really good!
When I first found Deepstream and scimmed through the docs, I instantly noticed how thoroughly planned and well-thought it was. The features Deepstream offers ended up making us able to drop REST HTTP completely. Normally when you build a back-end, you have all your application logic in one place (authentication, CRUD API, sending emails, etc) but the way RPC's and Providers in Deepstream are implemented, you want and can easily write your back-end as microservices. For example, you build a client that recieves RPC calls for hashing passwords, and if that client starts getting under heavy load, you can instantly spin up another instance of that client and connect to the Deepstream server, register for the RPC and Deepstream server will then distribute the RPC requests evenly among the two instances. Normally when you scale, you scale the entire back-end. Deepstream allow you to scale parts of your back-end really easily (e.g the part that sends emails, the part that hashes passwords or the part that crops & scales images). This allows for some really efficient fine-tuning of your back-end.
Another awesome feature is Providers. Providers can do a lot of things, for example transform a standard third-party HTTP REST API (e.g Facebook) into an integrated record list in Deepstream that is synchronized in real-time. Another example is writing an efficient real-time provider that integrates with literally anything - filesystem, databases, logs, hardware utilization, nginx requests, third-party API:s, w/e. The capabilities of Providers are endless.
Deepstream is insanely fast - I've worked with Meteor & other real-time solutions before and Deepstream simply outperforms them by a long shot. This is mainly thanks to the Cache Connectors that are available, for example the Redis Cache Connector. Writing Storage- or Cache Connectors is really simple and straight-forward and seamlessly integrates with any kind of storage you want. Heck you can even write a Storage Connector that stores its data in Firebase if you want to be really crazy (and idiotic). Or maybe in a GitHub repository? These are just stupid examples to illustrate how extendable Deepstream really is.
I've just scratched the surface of Deepstream, so I really suggest that you go explore it yourself!
I see a lot of questions regarding how Deepstream differs from Socket.io, SocketCluster, Meteor, etc. I'll go through each respectively a bit:
Socket.io is nothing more than a WebSocket server. Both Deepstream and Socket.io uses Engine.io (can be configured in Deepstream) behind the scenes, but comparing the two would is not really optimal as Deepstream is Socket.io plus a ton of additional features and performance improvements. You can technically re-build Deepstream by using Socket.io if you want.
Meteor is quite similar, but it has additional features (like being a full-stack framework) and their implementation of real-time sync is really bad. Meteor is by default locked-down to MongoDB and it's a pain to write support for any other database as they've also locked you down to the fibers coroutines and you need a client-side implementation of it as well for caching purposes. The number of connected clients the two supports is massively in Deepstream's favor, and Meteor doesn't even come close to being as distributable as Deepstream is. Believe it or not, but I used to work with Meteor until I found Deepstream.
SocketCluster and Deepstream are very similar, and unfortunately I haven't tried SocketCluster apart from reading the docs but to me, Deepstream is more developer friendly and straightforward and looks to have a better implementation. Since SocketCluster is on the front-page of HN as well, just looking at the live example they have on their website shows that it's not able to keep up. The WebSocket connection is dropping a lot.
For those of you who use front-end frameworks like Angular/Aurelia/React/Vue/etc you'll find it super simple to integrate Deepstream's client directly into the bindings of the framework. For example (Aurelia): `<input type="text" value.stream="pets.husky.favoriteToy">` would setup three-way bi-directional databinding to `pets/husky` on the property `favoriteToy`. (I'm currently in the process of doing exactly this for Aurelia)
Do you have any suggestions how to get started with deepstream + polymer (webcomponent library by google www.polymer-project.org/1.0/ )?
I have been doing some research on your website and I found toutorials for every major frontend framework except polymer, I would really appreciate a toutorial as a starting point for further projects, deepstream looks very promising to me!
deepstream.io (in difference to Meteor) isn't a full framework. The server can easily connect to a wide range of databases, caches or messaging systems. Similarly the client library can be used with React, Angular, Knockout or whatever else your heart desires.
API-wise deepstream.io's pub-sub mechanism is more comparable to javascript event emitters. Events in deepstream are a one-off messaging concept, for persistent data, just use data-sync with records
After a streak of HN articles loading slow or crashing my old mobile browser, your site loaded instantly!
I have a heuristic bias that whether a web-tech company's web site crash my mobile browser or not, reflects the quality of their technology.
deepstream is a realtime server for data-sync, pub-sub, request-response and WebRTC. That's a very different thing from Redis (although deepstream can use Redis for caching and message distribution)
- deepstream supports encryption via https / wss/ ssl
- once a connection is established each client has to login. The server then decides whether to accept or reject the connection based on the authentication and connection data
- from thereon, every incoming message is authenticated independently
all this happens in a "permissionHandler". please find more information here
It can. The domain name is not sent in the clear in an HTTPS request. Instead, the connection is opened against the IP address and the domain name is moved to a Host: header which is encrypted.
Correct (I considered explaining SNI but ultimately didn't bother), and a determined attacker could probably figure out a large number of the sites you visit by the IP as well.
- express, koa and hapi are http server frameworks, that's just a very different thing
- strongloop offers pub/sub capabilities as well, but is also first and foremost a service for rest apis
- socket.io is a transport layer with a low level concept of "rooms" to establish pub/sub behaviour. Deepstream and Socket.io actually both use engine.io for browser communication
The deepstream client can be written in any language due to it's messaging spec. There actually happens to be someone writing the client in Java now as well! You can find more info here: https://github.com/deepstreamIO/deepstream.io/issues/66
