Over time, NewTwitter does incur a large memory footprint. I've been traveling for a month and haven't looked into the problem in depth, but I'd bet the major culprit is our aggressive caching techniques. It's also worth investigating the number of polls and their frequencies. Timeline refreshes are dropped back to 90 seconds when the tab isn't focused, but not sure about other ones.
As most people have reported, infinite scroll is an easy way to introduce a slowdown over time. The sheer number of DOM nodes becomes quite significant - there are things we can do to ameliorate that issue, as well. Will take a look tomorrow. The feedback is greatly appreciated.
Infinite scroll is also a way to make scrolling infuriating, especially if you are wanting to click links and fan out from that location: it forces you to avoid your browser's carefully designed history stack like the plague (opening new tabs or windows for navigation instead) lest you lose your incredibly valuable document you've carefully been scrolling down in (and which is now so large it is threatening to cripple your browser anyway). What happened to normal "next page" paradigms? Does user testing really show that infinite scroll is better?
That's not a problem inherent to infinite scroll at all. It's easily fixed by setting a hash value on the current page (the #foo in a URL) corresponding to the current scroll state, and updating it each time infinite scroll happens.
Why Twitter doesn't implement something like this is a whole different issue.
You can imagine attempting to support something like this when the user has scrolled 20 pages worth through a timeline. We could, of course, backfill newer tweets subsequently up to the most recent (taking a minimum of two requests and a maximum of 20), but this would be expensive.
I mentioned somewhere else in this thread that we could maintain an accurate scroll height but detach more recent tweets when a timeline is scrolled a long distance. We could do the same thing here, and backfill as the user scrolls up. There are some challenging things to get right though if the user decides to jump to the top, etc. Definitely non-trivial.
Thanks for the response, I thought the reality would be much more complicated due to issues of scale.
This is primarily about supporting the back button, and I don't think users expect the latest data when going back through their browser history - generally pages aren't refreshed, and that's not an unusual or unexpected behavior. So you could cache each page state, without worrying about displaying newer tweets, though the caching mechanism itself incurs some nontrivial cost.
I don't have a breakdown with precise numbers for you, but some bloat comes from the fact that we include all of our templates in the phoenix bundle. Some savings could come from fetching certain components and pages on demand, but right now, we pull down everything. Gzipped it drops to ~200k, which isn't terrible considering that this is fetched in parallel with API resources and the initialization of our cross-domain iFrame. We're currently working on getting our bootstrap time down, so the bundle size may be reduced as a result of that effort.
Re: DOM issues, when you switch around between pages and/or timelines, we detach the timeline DOM elements and keep a reference in memory. Upon returning to a previously loaded timeline, we do a repaint (not bad). Right now, we don't employ any such technique for an active timeline that has been scrolled a long way down. One thing we can do is detach large blocks of tweets that aren't in the viewport and splice them back in (while maintaining an accurate scroll height) if the user scrolls back up.
Good deal russ, thanks. There's a fine balance to maintain between combining and lazy loading files. Still seems to be one of the harder front end problems to solve since client side architectures are all relatively unique. Thanks again sir.
It also doesn't work in the version of Arora I have. I imagine it will suck as much as Facebook does on the Kindle browser. Is there a version of Twitter that will work in a plain HTML browser with cookies and no JS? Please?
As much as I love the minimal UI, the mobile version is crippled by comparison. If they flesh out the features (which can be done without bloating it) or they deprecate the old UI like they're threatening, then I'll use it constantly.
The old UI works great and does not monopolize CPU, unlike the new one which, on load, eats 100% of one core on whatever machine I'm using (to say nothing of the really wide horizontal resolution required).
Trouble with the mobile version is it doesn't support https for some reason and so can't be used safely on an untrusted network. The normal version supports https though...
Yeah, I've been using it today. I want to love it but I can't. No DM count on the screen (to save three bytes?!), no user icons, no way to tell that I've gone over 140 characters.
Are you using a really low end user agent to try to access the site? I don't know what's turned off for lower end browsers, but I just tested in desktop Webkit. There's a character count in the upper right of the tweet box. There are user icons, unless you clicked the "images off" link.
As for total DM count, that isn't really possible right now. The app is just another API client, and the DM API endpoint only provides the last 20 DMs.
The thing I noticed in the slow new Twitter is that typing a new status seems to involve continuous autofill-like behavior for each character typed, which seemed to introduce round-trip lag (or javascript-parsing, or...) and causes characters to appear at a rate of ~1/second. Unusable.
And this is one way that dominant sites fall: business considerations are given priority over the user experience. Maybe they just have Flickr-itis and just can't adapt to leaving well enough alone, but for the time being I'm on old-Twitter and not contributing much to the Twitterverse until this is sorted out. If it never winds up being fixed, I've got a head start in living without them.
There is also a possible Second System Effect amongst Twitter's UX gods. It's interesting that after having switched to the new Twitter a couplefew weeks ago, last week I started seeing a "Wanna switch back to the old Twitter?" header. Of course I did, but to me this possibly points to a level of complaints that I did not realize.
I don't think so. I'm not a huge tweeter and so anything that gets in the way of the rare things really jumps out at me, of which the slow status box is no small issue. Other than that the redesign is just that, an expanded sidebar with some fleshed out boxes. No big deal. That it's causing different problems for different people tells me that maybe something else is going on behind the hood. Heck, maybe they just want to be the network and aren't interested in providing a groovy webapp interface anymore, who knows.
I've been keeping twitter open in its own Firefox browser on one screen (of three). It routinely refreshes itself and there is some kind of memory leak whereas after a day it is consuming about half a gig of memory, forcing me to kill and restart.
Firefox has crazy memory leaks. Don't blame it on twitter, regardless how bad their code may be.
A tab should be completely and utterly destroyed when refreshed. It doesn't matter how much memory it used, refreshing it or navigating away from that page should (ignoring optimizations, caching, etc. etc. etc.) be like you were never there.
I have been using it in Chrome and new Twitter has been unreasonably slow. A friend who works at Twitter has acknowledged that they know about the issue and are working on it.
I routinely leave my browser open for days on end and simply cycle through tabs, closing and opening new ones as needed, with usually ~10 open at a given time. I, too, noticed that firefox was eating up about half a gig of ram even when it wasn't doing anything, didn't have any youtube tabs open, etc. It got to the point where I would have to close firefox and restart after a day or two. Chrome fixed that. Now, I just use firefox with the modify headers extension to watch domain-specific video, and have no plans to go back.
Now, maybe I'm to blame here, using the nightly build of chromium, but webkit is a walking memory leak. Just for a comparison, I have the same amount of tabs on firefox and chromium and firefox is using 500mb of ram, while chromium is using 3.5 GIGABYTES. And I'm not even using it much.
But really, the problem is easily reproducible. Just open whatever webkit based browser you want, and keep reloading the same moderately big page. Ram goes up, never to return. That works with a trivial program that embeds webkit via gtk, too.
Or you can just look at the commits on webkit: a couple of leaks fixes every week.
They're not confined to a single tab for me. It seems like child tabs go in the parent process, for example, so I usually get 2-5 tabs crashing at once.
I also get more crashes with Chrome than I used to get with Firefox, but maybe I'm just opening more tabs.
I left Firefox for Safari because Safari was prettier, faster and used less memory.
Lately, I'm close to leaving Safari for Chrome because it's faster, uses less memory and crashes less (in fact, it's never crashed so far, vs 100's of times total probably with Safari.) Unfortunately, Chrome is a little uglier, and has certain less user conveniences. So it's not an easy jump to make.
You think Chrome is ugly and inconvenient? I think it's the best looking browser out there, and the omnibox is fantastic. The only thing I don't like about Chrome on OSX is some flash players don't play nice with it.
Really? I have to restart Firefox 3 much more often than I ever restarted Firefox 2. I routinely left Firefox 2 running for a month at a time with 100+ tabs spread across several open windows in different workspaces. Firefox 3 only lasts for about a week in a similar situation. Chrome uses more memory per tab, so 100+ tabs isn't feasible with it. (Maybe it is now, I haven't tried recently)
Exactly. If these memory leaks were really there, why hasn't anyone gone through the code base and found them? From what I understand, the reason the amount of ram Firefox uses grows over time is because of caching, and if another program needs the ram, Firefox trims it's cache. What's the point of having 4 GBs of ram if you're only going to use 1.5?
I can't help thinking that the tacked-on whine about this is why people want iPads and not PCs would make more sense if this post were not about a web-app that would probably slow down any tablet it was accessed on.
The "the tacked-on whine" as you call it was referring to having to check for the latest version of this that and the other bloody plug-in. Point being, people just Want Shit To Work.
Yes, they do...and then they want it to do the Snazzy New Thing that didn't even exist when the thing was built (and integrate with the Wonderful Service and the other Hot Newness).
These are the two omnipresent and conflicting goals that people want out of any computer that's even vaguely general-purpose. Wait long enough, and you'll see it happen on iPads, too.
ETA: I mean, Hell, we're talking about a page in a web browser. This is an example of both the idea of doing everything simply in a self-contained way that Just Works with no fuss...and the post-1994 reality that people want to make web pages do everything, including building applications out of them and live-updating information. And that's even without involving browser plug-ins.
Yes, and I do sympathize. This stuff isn't like a light switch that you turn on and have the room all lit up despite whatever changes have been made to the backend with the transmission lines and generators, etc. But sometimes we want it to act that way.
Agree that ipad promotion at the end is not really a fair comparison. I'm assuming he's speaking of the twitter app on the ipad. For it to be fair he should install a desktop client for twitter on his pc or possibly tweetdeck and make a comparison.
I noticed the same thing on my girlfriends CR-48, twitter wont scroll properly (studders) and everything else hangs a bit. She filed a bug report with google chrome os support, but I suspect twitter is doing a lot of heavy stuff in the background.
I've noticed the same thing on mine. It seems like the site is only functional when you first load it. After new tweets start rolling in, it becomes worse and worse. I don't think it's the fault of the CR-48, I think we're just seeing the effects worse on this hardware.
I believe the scrolling issues are tied to their use of position: fixed for the sidebar where that DIV has its own internally scrollable frame via overflow: scroll. The combination of those two are taxing on the browser's overall layout system which causes a dramatic slowdown on less powerful machines. Heck, it's even a little sluggish on my iMac.
Agreed, the new twitter is way demanding compared to the older version.
Once they stop allowing the old version, I'll probably have twitter loaded much less. On a core 2 duo, even scrolling in new twitter is way below par.
Is there a low cpu-usage client? I tried one of the flash based ones, but it brought the fans permanently on on my MacBook... Yeah, that's not going to fly.
While it's better than the web UI, it suffers from the "mp3 player" problem of trying to make up its own widget library and UI style, and actually is kind of lame in a lot of ways (there is no top-level "new tweet" button, which is kind of a feature of all the mobile clients, and it's definitely a pain to use for DM conversations. It pops up weird non-standard shaped windows all over. I'd consider it a respectable indy effort, but pretty disappointing for Twitter, Inc.
I've experimented with the new twitter web app on both Chrome and Safari (Mac).
It automatically loads a chunk of data when I scroll to the bottom. The CPU use seems to be proportional to the number of chunks that have been loaded. Initially about 2% and increasing to 50% with 20 chunks.
The new twitter almost unusable because of this. Hoping for a fix soon.
The point of my post is also this: Something changed in the behavior of New Twitter. None of this happened when I was first given access to NT months ago. The problems seemed to crop up simulataneously -- coincidentally -- with my moving to Opera 11 beta. The problems have since spread to all 3 of the browsers I wound up cycling through. So I think Twitter changed something on their end and are unaware of the ripple effect on some users. Someone, on Twitter, suggested they were once polling every 90 seconds in the beginning and now that the switch is complete, we are getting "live streaming." That might be too much for some systems to handle. Perhaps a user option to throttle back polling? [typo edit]
Currently, we're not live streaming. We poll every 30 seconds when focused and fall back to 90 seconds unfocused. When we do push, we'll be using User Streams (http://dev.twitter.com/pages/user_streams) with either Websockets or Flash. Performance should improve when that happens. There are also some other neat tricks we'll be able to do to make the site even faster.
But something changed somewhere. Scrolling used to be a smooth experience. The entire use of NT was exciting and not cloggy. Now it is hell scrolling and very cloggy and even autocomplete of IDs is a torment. What did you guys do over there to the code?
I didn't notice the excessive CPU usage until very recently, meaning that it may be a recent patch or change causing this problem. Safari still operates quite well as long as I don't have the Twitter tab open, and even then I don't notice a slow down until a try to scroll the page.
That said, because of the problems with the web version I have switched to the new standalone Twitter application for Mac OS X. I think this is better as well as it allows me to use Growl notifications, notification sounds, and other nice features.
This is the least scientific analysis I think anyone could get away with. Why doesn't the author just profile the Twitter page to see how much time his CPU spends executing Javascript code and how much memory it consumes while running?
I dropped Shockwave in there as yet another example of something else that needs to be checked for the latest version. Not every plug-in is good about saying there's an update and most of the time the update notifications come in the middle of getting work done, so people tend to disable that distraction.
Yes, I know that. You are taking it too literally, that one little bit. I was simply listing a bunch of updates that techs always ask about having been updated.
well, i just deleted all twitter apps on my mac because they became too much of a time hog, and well new twitter on the web sucks currently so this means one thing ... i'm free
However, this is because I use Google Reader to follow feeds and only use Twitter to keep up to date with friends and acquaintances. That's not the typical use-case.
I'm still looking for a reason to even use twitter. I understand _what_ it is, just not any useful purposes for me.
I did think of it as a way to send broadcast "secret" messages, or something along those lines, or when you are posting mundane stuff (went 2 bathroom had shit). Possibly it could be used as a form of a system downtime monitor, similar to what status.4chan.org is.
But use it for me? That's what emails, txts, and phone calls are for.
The pronounced slowness was primarily due to an upgrade from jQuery 1.4.2 to 1.4.4 (we're going to downgrade). In addition, we're moving from listening to the scroll event to a time-based dispatcher for loading subsequent pages of tweets.
New Twitter has a login popup everytime I open it. I am already logged in. I can close this popup and my twitter experience is not effected. Why is this thing even there?
Is the popup a HTTP Basic Auth window? If so, are you using Firefox? This is a ridiculously elusive bug that a couple of us have been trying to diagnose. It appears to be buried in the FF implementation of XHR when a reference to an XHR instance is shared across iFrames. We're working on it.
As most people have reported, infinite scroll is an easy way to introduce a slowdown over time. The sheer number of DOM nodes becomes quite significant - there are things we can do to ameliorate that issue, as well. Will take a look tomorrow. The feedback is greatly appreciated.