Hacker News new | past | comments | ask | show | jobs | submit login
Facebook Does It Again. Cheating Dalvik (mohitkanwal.com)
90 points by redDragon on Aug 11, 2014 | hide | past | favorite | 65 comments



There's a weird and arbitrary limitation in Dalvik, and the Facebook guys needed to work around it. The author makes it sound like it's some horrible sin. How about shaming the Google developers who thought 65k methods was enough for anyone? Wouldn't be good proper either, because they probably had a good reason for it.

Software development isn't always stacking neat abstractions on top of each other. Sometimes you have real hardware and real legacy and you just got to deal with it, one way or another.


This limitation is not "weird or arbitrary" but based on the technical capabilities of Android phones in these days.

And those limits were later changed and will surely be changed in the future as smartphones get more powerful.

All kinds of limitations are in place both on Android and iOS, particularly when it's about the amount of memory an application can access or the amount of background processing an app can make.

If you don't have these limitations, then apps could affect not only the functioning of other apps, by grabbing and keeping all available memory, but also that of the phone itself, by draining the battery in 1 hour.

That's not something that the average smartphone user would expect or desire his phone to do.

Edit: I'd also like to add, that almost anything a developer creates is usually created with some limit, or at least some expectancy of costs in mind.

For example I couldn't imagine that someone creates a memory cache, where the limit = infinite without a really good reason. Or an API where the request limit/sec for any client = infinite.

You usually develop something in a way, that reduces the cost of some computation as far as possible, and when it still causes problem - introduce hard limits. Especially when it's a whole ecosystem where many companies/developers participate.


The bug they hit isn't the 65k limit, but the smaller LinearAlloc buffer size on older versions of the OS.


Yep i think the blog post referred the stuff as being LinearAlloc == dex method count, I have made it clear now.


I imagine the devs and management don't care much as they're going to migrate to ART soon, which I imagine does not have Dalvik's circa 2003 design decisions. Google hasn't been investing in Dalvik because its days are numbered.

edit: why is this being downvoted? Do we really think ART will have these limitations? Android has been hot to get off Dalvik for many reasons and they're finally doing it in L. This is very good news and makes Andy Rubin-era Android decisions, which made sense for a 2003 phone, replaced with more modern thinking.

edit2: Apparantly, I'm wrong. Looks like the 16 bit limitation in still there in the current version of ART. Maybe this will change in the final production version.


The limit is being addressed, though not fully in L. It's discussed in the podcast at http://androidbackstage.blogspot.com/2014/08/android-develop...

Basically they're aware it's an issue and they're working on it both for the future of the platform but also in a backwards compatible way (using multi-dex and reflection).


I believe ART still uses DEX as an intermediate format, so it's possible the limitations will still exist.



2^16 counts as "weird and arbitrary"?


Since we stopped using 16-bit CPUs, yes, it does.


Since Dalvik Executables run on a virtual machine, and the Dalvik virtual machine's instruction set is 16-bit, it makes perfect sense.


We have this problem. We have a large, mostly-Java game for Android. It won't run on gingerbread, because we just have too much code.

While the official solution looks quite brief, retrofitting it is a nightmare. Unless Facebook got around to rewriting everything again, I can understand why they'd do it their way.


I'm in the end stages of developing my first Android app. Throughout development I used the Android compatibility libraries to get it working on Gingerbread. However, now that I'm almost finished, I realize that many Gingerbread devices have a hard limit on the number of simultaneous active MediaPlayer objects that means my app will not (and cannot) function properly. (All devices tested on ICS++ seem to work fine). It works on some Gingerbread devices (and, annoyingly, all emulated Gingerbread devices I've tried) but not all. So now I'm in the awkward position of either changing the minimum API level of my app to exclude all Gingerbread devices (even though it works on some) or annoying some Gingerbread users by getting them to download my (large) apk only to give them a "sorry, your device is not supported" message when they first open it.

I know nothing about iOS development, but I'm envious of the narrow range of hardware and software in that ecosystem.


IMHO, Google (Android) seems to be at fault here, as they lack a proper way to determine whether an app is compatible with a particular device, without false negatives.

Instead of getting angry at developers (eg. calling it "cheating" and a "horrible hack"), who want to maximize the compatibility of their app, the Android team should add a standard way for apps to define what they need to be compatible with a device, without getting false negatives.


I wouldn't be too worried. Gingerbread devices are something like <13% of all Android devices (out of something like 800 million). I'm going to go out on a limb here and assume that people still running 2.3 are probably not people who will be buying apps anyways.


If it's too much of a hassle to make your app work on Gingerbread, just don't bother with it anymore. I bet Google themselves will end all support for Gingerbread next year when version M comes out (as they should). It just doesn't make sense to support it longer than that, and by then the market share of Gingerbread should be under 15-20 percent, too. Gingerbread is the XP of Android - an old popular version, right before major architectural changes.

Android L seems to be the next major architectural change, but I don't think KitKat will be the new Gingerbread, because Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old, which will soon mean (going by the new release cycle) just 1 version behind. So the transition for new versions should happen a lot smoother in the future.


> by then the market share of Gingerbread should be under 15-20 percent, too

According to the Android dashboard [1], among users of Google Play Gingerbread has a market share of 13.5%

[1] https://developer.android.com/about/dashboards/index.html


I'd like to see what percentage of app sales come from Gingerbread. My guess is that people that are willing to buy apps are not* using a 3 year old phone. (*edited)


A gingerbread phone can by of any age. That's why google is moving towards a new contract for its gapps. Going forward you can only release the current Android or go as far as 2 versions back. So we're on 4.4 now, thus making 4.2 the oldest version for a phone released today (assuming the new contracts are in place).

A few manufacturers went a bit overboard mass producing GB-only phones lately because of how cheap their components are. I'm not sure what this means for the budget phone market now that they need to be 4.2 or higher on release date. From what I've seen, the new budget phone is the low-end Nokia Windows Phone with carriers like Cricket literally giving them away with rebates or charging next to nothing for them like $50, and that's on a no contract plan!

Getting Android out of the GB-era budget ghetto is a good thing. If the Moto E can handle 4.4 then so can everyone else. Supposedly 4.4 uses less ram than previous versions of the 4.x line. The head of google claimed that it can run on 512mb devices. They really, really want to kill GB. I imagine there will always be a android budget phone out there, just not no name junk running a near 4 year old OS.

The real question is how many more GB phones are there in the supply chain that will be sold this year and next? Millions? How long will these things be in play? It really does look like there is a real abandoned version of Android that can't survive fragmentation now. Many apps simply will not work. Shame really because I'm sure no one explained to the budget phone buyers that they were buying an ancient OS that can't be upgraded on that system with so little ram.


People are not willing, right? Gingerbread phones are old and underpowered and can't handle the demands of flashy new apps.

I should buy one.


> Google is now requiring OEMs to only launch devices that are 2 versions and/or 9 months old

Are they requiring anything from carriers or phone shops, though? Because everywhere in my town is still selling Gingerbread devices to anyone that doesn't know better.


There is a way: https://github.com/creativepsyco/secondary-dex-gradle

The new gradle build system really simplifies a lot of things in terms of the build cycle


When did it come out ? I suspect it wasn't adopted widely a year ago.


Consider switching to Xamarin + MonoGame. You will have to pay, but features might be worth it (better yet familiar language more mainstream in gamedev, decent cross-platform support). Alternatively try using Unity3d if you don't mind scripting instead of programming.


I don't think anyone with a phone running 2.x can run games. I have an HTC One or whatever (4 years old; got it from a friend) and I could never ever imagine playing a game on it. Even the home screen lags!


If you code in C++, which most games do, the Android version is not that important, given the little exposure of Android APIs to the NDK.

Everything that matters, besides the existing graphics, sensors and audio APIs, can be built with standard C++ libraries.


The problem lies with the hardware, I think. Truly, there's never a smooth moment with my phone. Sometimes it hangs for a minute. Random shutdowns and similar bugs in software as well.


I've got an old cheap 4 year old Acer phone running 2.3 as a backup phone and it plays lots of games fine. Sure not games using a lot of 3d graphics or games which require very precise timing, but it plays lots of other games.


I just played through Swordigo on a Galaxy S (the first one!) and it ran very smoothly. I am, it may be worth noting, using Cyanogenmod 11 nightlies.


I own a Xperia Play, that don't support anything after 2.3, and I have lots of gaming here, including shooters, platformers and racing


Oh you will be surprised: Not every game requires higher Android version, they are mostly satisfied with c++, http://imgur.com/iX4dcL5

Card Games: Gingerbread (14%)


I'm a bit confused by this article, the author mentions:

    While the official standard way to to fix this problem is described in the offical Android Blog.
    Seems like Facebook missed it. What a waste.
However, in the post linked by facebook, they explain why they couldn't use it (vaguely)

    After a bit of panic, we realized that we could work around this problem by breaking our app into multiple dex files, using the technique described here (http://android-developers.blogspot.com/2011/07/custom-class-loading-in-dalvik.html), which focuses on using secondary dex files for extension modules, not core parts of the app.


Yes i sort of forgot about it, the facebook article was old, i read it a long time ago, i rediscovered the issue when running FB on one of the Gingerbread devices.


Google early on made decisions about code size that made sense at the time, but now seem to be a serious limitation (at least in the interim). FB's decision on getting around this architectural limitation isn't all that bad. What I don't understand is how their solution can affect other applications. As an iOS dev screwing up my app is one thing but being able to screw up other apps seems like an even worse architectural problem.


I'm confused on how it can screw up another application? Aren't all the applications sandboxed?


This hack can't screw up another application, the author is wrong about that. It's changing the local process only and has no influence on any other apps (which are not just sandboxed into their own processes but also their own user ids)


Given the state of Operating Systems 20 years ago, it would be very ironic if today the Apple OS doesn't let one app crash another, but the non-Apple OS does.


Zuckerberg 2012: Our biggest mistake was betting too much on HTML5

Zuckerberg 2014: Our biggest mistake was betting too much on Dalvik


Sounds like his gambling problem is the issue.


Below are some choice quotes from the last time this came up almost 1.5 years ago; interesting how the peanut gallery has mellowed over time. https://news.ycombinator.com/item?id=5321634

> I can't believe an app requires 8M RAM just for method names! - https://news.ycombinator.com/item?id=5323153

> They bloated the app so badly, they would have had to monkey patch the OS to let it run at all -- https://news.ycombinator.com/item?id=5323930

> Basically these engineers decided to be really clever, painted themselves into a corner, smashed a hole in the wall so as to escape from said corner, and then bragged about how great they are. -- https://news.ycombinator.com/item?id=5322286

Also, the bug that was filed was also referenced: Dexopt fails with "LinearAlloc exceeded" for deep interface hierarchies https://code.google.com/p/android/issues/detail?id=22586


I don't think we've mellowed, just that we have become more sympathetic to Facebook's problem. I think the reality is that the 65k method limit problem has turned into something Android Developers put squarely at Google's feet. At the time I thought, what the hell is Facebook doing? Now, I've run into the 65k limit myself on an app written by 4 people over 18 months. Sure it's not a throwaway weekend project, but that is a pretty standard app.

Google either needs to blow away the 65k method limit, or figure out a way to make Google Play Services a lot lighter. As it stands right now, if you want just push notifications you are giving up ~20k methods of your 65k limit for that alone.

Just look at this post from Jake Wharton (leading Android open source developer): http://jakewharton.com/play-services-is-a-monolith/


Slimming down Play Services just punts on the problem for a while. My view is that Google needs to do two things with this:

1) Split up the Play Services client libraries. 2) Figure out some solution for developers hitting the 65K limit.

I'm sort of assuming that the solution would come in the form of a framework or tooling that we'd have to implement. Such a solution should allow us to build fully working apps with multiple dex files, with some decent documentation. It should also work fine for debug builds without proguard, and also without bumping build times up beyond a two minutes. What's more, we should be able to split off pieces that are defined in the manifest into secondary dex files, and fire intents at them.

That, to me, seems like a reasonable response to this problem by Google. The company I work for has been hitting the limit for the last few months, and some of our other dependencies are becoming more and more expensive with newer releases (things that our users actually like.) So far, we've been able to build with a stripped down Play Services jar, but I'm not terribly happy about that approach.

Regardless, I'm not going to blame Facebook for a problem that Google caused. If Facebook can find a solution, and tell the community how they did it, I'm all ears.


I think it's quite common for developers to expect restrictions on APIs to ease as the API matures. I've also heard, anecdotally, about more developers hitting this method limit since facebook first talked about it.

Personally, I think it's silly that Android apps work one way up to 65k methods, and then require you to re-think how your app is architected due to packing limits. I think Google is just as guilty as Facebook of selfish engineering choices.


Its almost as if requirements went up over time. Google Play Services has also made the problem worse over that time.


On the 65K methods restriction, how many lines of code per method do these projects have on average? That just seems like a lot of code. Is this a result of the Java acessor-and-mutator-methods-for-everything approach?

For reference, Tomcat and Maven are both 1M lines of code (would be ~15 lines of code per method @ 65K methods). Lucene is <500K.


I used to think this was a non issue. 65k methods is a lot right? Well I ran into the 65k method limit with my one developer game I'm working on. Can't remember if it while i was using gradle or maven(recent switched) but it would print out all the methods for each package. Method counts add up quick when you are importing libraries. For example google guice by it self(if memory servers me) had ~6k methods. Thats one library. Starting adding different Apache libraries and its possible. Granted the majority of these get stripped out by Proguard, but having to run proguard for every build is a major pain. I have since refactored and am now well under the limit, but this issue is a lot more prevalent then i thought.


I understand, Dalvik uses 2 bytes to select a method found in the list at the start of the dex file and then to execute it. Is there any particular reason it is limited to 2 bytes ?


This is just me guessing but one of the initial goals of Dalvik was to use as little space as possible (due to memory constraints of hardware at that time). Perhaps they thought saving a a byte or two on method identifiers was a good idea. This would not be a problem if it weren't for the fact that dx also squashes all classes into one single class, meaning that instead of being 65K methods per class you now get 65K per apk.


Ah, mobile. Recapitulating all the horribly constrained hacks of the early PC era, in handheld size containers.


Note that the Facebook blog post about this hack is from 4 March 2013, i.e. more than a year ago.


Yes and still FB relies on this patch to run its app, and sometimes it does fail as happened on my device.


I don't see how facebook cheated Dalvik 'again'? Seems like it's still just the one 'patch'/'hack'. Am I missing something?


What they do "again" is pull some nice engineering feat that's also a bit of a hack, and therefore fun to read about. I assume the author is impressed, and likens it to some of Facebook's similar bend-the-world-to-my-turd undertakings such as patching git to be able to deal with Facebook's single gigantic (!) source code repository.


Oh, I see. I assumed the 'it' in the post title referred to the 'Cheating Dalvik' that followed. Not that it referred to a more generic concept of bad-practise.


This article seems to conflate the limit on number of methods vs the amount of memory required for one of the dexopt steps.


I understand blackberry messenger also has the 65k method hack in its android app.


I wonder if this 65k limit is part of the reason they made messenger its own app?


I'm sure it was a consideration. Also from a team/project structure standpoint it probably a lot easier.


They made Messenger its own app and made the separate app required for messaging on iOS first, and later on Android. So it really doesn't make sense if any significant part of the motivation was working around an Android platform issue.


Mirror?


Maybe it would have helped missing out in the mandatory spying features, and providing only the required customer features.

Those features really take a lot of memory:

* “Record audio with the microphone … at any time without your confirmation”

* Take videos and photos using the camera

* Access the phone’s call log

* Read data about contacts stored on the phone, “including the frequency with which you’ve called, emailed or communicated in other ways with specific individuals”


> * “Record audio with the microphone … at any time without your confirmation”

There's no way in Android to record audio with the microphone at only certain times with your confirmation. Facebook needs microphone access for video and voice calling.

> * Take videos and photos using the camera

The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access?

I can't speak for the other two (I use iOS myself) but Androids permissions are generally way too broad (stuff like to be able to pause music when the user is making a call you have to ask for access to their phone number and the phone numbers of everyone calling them) and users have to accept the kitchen sink. The "Facebook spying" stuff is an Android bug, not Facebook.


The Facebook app lets you take photos and videos to post on your feed. How else would they be able to do that without camera access?

Isn't the point of the Android app model of breaking everything up into Activities that you can call another Activity which does the video/photo recording, and then you just deal with the result?

(That's not a rhetorical question; I may be misunderstanding the intent of the design, or how people use the design in reality, or both.)


Yes, you're right that activities are broken up like that. Permissions are defined in the application manifest, and apply to all activities within the application.

Facebook would have to break their application into a calling app and newsfeed app to split the permissions apart. It would still request permission to record and take videos for both, though, if they implemented video recording on the newsfeed app.


> How else would they be able to do that without camera access?

Open gallery, click share, select facebook from list.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: