The way that this post describes porting and existing interface from server-side/ajax rendering to client-side rendering is particularly interesting -- I imagine that there aren't many apps that have made that jump after first being built around server-side templates.
The speed benefits that Diaspora is seeing from distributing their HTML rendering is probably only the first step. There are so many more interesting interactions you can accomplish once your data is being modeled in the browser.
Personally, I've rewritten a bunch of code at work to go from server-side rendered templates (in Jinja) to client-side rendered templates (using a Jinja-to-JavaScript compiler I wrote, https://bitbucket.org/djc/jasinja). Add some WebSockets magic and we now have a very fluid real-time application page.
The model is obviously very powerful. The one problem I have with it is JavaScript-the-language. Using it makes me love Python so much... I know it's not that bad and there are good parts, but it's still nowhere near Python.
IIRC, foursquare.com's client side was recently rebuilt in Backbone and now they retrieve all data using their API. It's an interesting trend that I think more and more large websites will follow.
It's a very pure model which is extremely appealing from a last-mile-developer standpoint. Actually building everything that you need to get there is the challenge, but it's a very worthy goal.
Live re-rendering of data -- imagine if the (now) "24 minutes ago" timestamp next to your handle was always accurate, instead of just reflecting when I loaded the page.
Easier implementation of otherwise difficult features. Doing a client-side autocomplete of people's names is trivial when you already have all of the other accounts in your organization modeled in JavaScript ... but not as fun when you have to ajax for server-rendered HTML for it.
Live updates of pushed data. When another reader +1's a post that you're currently looking at, if the server pushes that data to you, and you have a model for that post -- it's easy to increment the counter. If the server has to push the re-rendered HTML for the entire post, it's much more difficult.
This can be done without the radical switch to modeling everything client-side. I mainly do java web apps with JSP's (which everyone seems to think is stodgy and past its prime) but its rather simple to do something like what you're describing with keeping things up-to-date. I simply have broken my project down into "widgets" that self-load so a page actually is made up of many widgets that use javascript to call various server-side controllers, you can then set them to do whatever you want (such as refresh every 30 seconds) and it doesn't require some massive client-side data model.
I agree, there are cases where you would want a more robust client side data model, but, IMO people are a bit too quick to jump to using this and it can make more work in the end and make a project more costly to maintain. I think the cases where it is useful are mainly for highly interactive sites like those that would have previously done in flash, with animations (such as games, etc), for just regular CRUD webapps I feel its a bit of overkill
Since Diaspora is based on federation, you might even be able to do some neat tricks having your browser bypass your current servers and talk directly to other Diaspora servers.
There are probably tons of security issues, but for some read-only cases it might be useful.
Ok, maybe not tons of security issues, but maybe some pitfalls.
My imagined use-case is where your friend on a different server posts an event, it gets put into your feed, and when you view your feed you're able to see an updated list of people attending by directly querying the friend's server. Would be pretty slick.
This is the biggest win for Diaspora? Really? Sounds like they've been deluded by all those NYU CS professors who think knowing how to add 32 bit floating point numbers is the key to success.
Just dump everything, spend 1% of your time rewriting everything in PHP, and the rest actually doing some marketing, not imaginary work.
They're running a fairly large scale app, which has a lot of users and recently experienced serious capacity issues (after Ilya's fairly well publicized death). Optimizing it doesn't seem premature at all.
Maybe this is a really stupid question, but wasn't Diaspora supposed to be decentralized? In the sense that anybody could set up their own hub? (so people would not have to accept features pushed by some big corporation and things like that)
Then what is this whole database backend for? Is it like the "main" Diaspora hub? And could anyone set up their own secondary one? Would they run into the same difficulties they're trying to solve here?
Though I get the feeling I'm probably misunderstanding the entire Diaspora project, here.
Each hub has its own database to store the data about its own user(s). Then the hubs can communicate with each other so you can "friend/follow" users from other hubs.
Sorry, I don't know what "Backbone" is. I see links to New Relic and Pivotal Labs, but no links to anything called "Backbone".
I guess this blog post was maybe not meant for purely technical people, but it would be nice to understand what "Backbone" is, and exactly why it would solve their problems.
It's entirely possible that you don't get the same search results on Google as I do if we're logged in because of what Google 'remembers' as our browsing habits and interests.
Which gave me the first hit of www.backbonemedia.com/
In fact, backbone.js is no where on the first page for that search.
That's amazing it comes up first for "backbone". I did actually find that link eventually, and try out the "example" todo list, but that gave me no information about how this would fix a garbage collection problem. Does anyone know how it fixed their GC problem?
It fixed their garbage collection problem by reducing the amount of server-side processing they had to do (and hence, reducing the amount of garbage that required collection). Rather than send completely rendered HTML to the browser, they're using javascript in the browser to turn the data coming from the server into the rendered page.
If I had to guess, they're running with the stock Ruby GC parameters, which are fairly horrible for a Rails app, and can result in multiple GC cycles per request. Ruby's GC is painfully slow, so if you're kicking into it, you're going to slaughter your response times.
REE and Ruby 1.9.3 both offer GC tuning parameters, which let you instruct Ruby that you're going to shove a huge app at it and to not garbage collect every time someone sneezes, which can have a pretty massive impact on response times.
It sounds like they were jumping to conclusions regarding the templating being responsible for creating a large number of objects.
ERB, Haml (which is what Diaspora uses), and any other templating engine I've seen use either concat or << when rendering a template. These never create a new object, they mutate (and perhaps resize) the original string.
Maybe next time they should profile better before following their gut feeling and rewriting their front end ;)
I'm coming from a Python perspective; no idea if this is easy/possible in Ruby.
Maybe this is a really stupid idea, but what if we gasp disabled the GC during the course of each request and did a collection run after the request is completed and before the next one is accepted? With a sufficient number of workers, wouldn't this solve some of the problems they were having?
I just remembered that Unicorn actually has an option for out-of-band garbage collection. It works best when you're at < 50% CPU utilization, and improves response times with a possible overall performance decrease.
Diaspora has always been a joke, ever since that first awful naive release.
It's a good idea (and older than the diaspora project) and kudos to them for pushing forward, but I really hope someone else implements this idea correctly.
Rewriting the front-end using Javascript and Backbone won't fix the backend problems necessarily. How about reducing the objects creation and tuning the backend too?
They discussed this in the post. They found that the templating was the source of many of the objects, as Ruby creates new objects when performing string concatenation.
The speed benefits that Diaspora is seeing from distributing their HTML rendering is probably only the first step. There are so many more interesting interactions you can accomplish once your data is being modeled in the browser.