This is a vulnerability in the authentication scheme used in the MySQL wire protocol, meaning attackers need to be able to connect to your MySQL database directly to exploit it. Attackers should never, ever be able to connect directly to your MySQL database directly. If you can connect to your MySQL instance directly from your Macbook in your living room, fix it right now.
―Attackers should never, ever be able to connect directly to your MySQL database directly.
mySQLgame[1] demonstrates that domain logic can be successfully implemented within a publicly accessible database[2]. It would be better to reword your statement to:
Minimise the attack surface by preventing unnecessary access
There are times where public access to a database server make perfect sense. It is the reason why database servers implement TLS client certificate verification, Role Based Access Control (RBAC) and other security features which a typical application (with domain logic at the application layer) has no hope of implementing correctly.
I would generally avoid building applications that assume clients are going to speak directly to the database. We see a couple of them every year (it's a common pattern in enterprise applications) and they tend to be horrorshows.
In any case, the typical web app deployed by HN readers has no business having an exposed MySQL port.
Actually that's exactly what we do with LedgerSMB. The database is the final enforcer of security matters and the authoritative voice in authentication (the web app logs into the db with user-supplied credentials). A lot of traditional middleware functions are pushed into the db (as stored procedures) where these can be reasonably represented as set operations against the database. As an ERP application, this means that this includes just about everything except generating printable invoices or HTML.
This works well. There are some things we are still working on improving, but it's getting there. Moreover it means multiple clients on multiple codebases are possible and we don't have to worry as much about the implications of what happens when someone writes a secondary client to hit the database.
At the same time, we use PostgreSQL, which I have a bit more confidence in than MySQL. Of course I would still suggest limiting access only to those IP address ranges where that is necessary.
Oh god you poor man. We took a look at using SMB Ledger (the parent of LedgerSMB) years back and after peeling back the covers it just looked horrific both from a security and general code quality perspective. This was around the time of the fork, and I couldn't help feel that SMB Ledger had it's work cut out. Hopefully you guys have been able to get through to the other side.
1) Going to a db-centric security model has allowed us not to trust the SQL-Ledger code (good thing too).
2) We are refactoring/removing SQL-Ledger code as quickly as possible. As of 1.3 this means payment logic, reconciliation logic, contact management logic and more. 1.4 will hopefully rip out and replace all search and reporting functions. It will take us a few more years to get the codebase where we want it though.
3) The bad thing about bad code is that bad code is contagious. When you spend a lot of your time debugging bad code it is very hard to write good code. Most of what we wrote for 1.3 will need to be rewritten again. I am pretty happy with the code we are writing for 1.4.....
I think we have come through the worst of it. Pace of development is speeding up which is a good sign and much more of the application is subjected to unit tests.
As a side note, when we first added unit tests to the number rounding tests for the code we inherited from SQL-Ledger, there were failures. We replaced that logic very quickly.
Yeah, it was pretty bad... Now its getting better.
Edit: Also it seems to me the worst never seems so bad when you are in it. I don't think that I could see how many problems we had from this until I am here, half way from 1.3 to 1.4, asking why 1.3 took five years to release (we beat Perl 6 and HURD though, I guess Duke Nukem Forever in fact beat us date-wise by a couple months).
Looking back at the customers of mine who have had problems, or projects that went way over budget or took too long because of difficulties here, I can see how much that hurt us. At the time though, it was just one of those keep working on it kind of things.
Either use iptables to lock it down to a particular IP or set of IPs. Or setup ssh tunnelling to expose a remote mysql server as a local port. You should assume mysql is insecure and needs protecting.
Having OpenSSH running is pretty much essential for any machines you remote administer. Some things you can do: disable password auth, use public keys. Disallow root logins. Listen on a non-standard port. Configure a hardened "jumphost" as your interface between your machines and the outside world.
Also: SSH is considerably more battle-tested in this configuration. There's a lot to be said for being aligned with how the developers imagine a program is used
The counter argument being that if you expose lots of MySQL servers directly to the Internet, you end up with a product much more suitable to being exposed to the Internet.
From a security perspective it's good advice. I've never seen a shared web hosting provider whose boxes couldn't be owned by an account that could drop code and execute. Local priv escalation holes seem to be a dime a dozen and you don't know if they've even attempted to lock down their users' data.
Now, do you really care if some community bulletin board's database gets owned? Probably not. But I wouldn't run a shopping cart on a shared hoster.
If you don't really care about security --- and I agree there are times when you shouldn't --- then by all means use the cheapest possible hosting option available to you. But if you care even a little about security, avoid shared hosting.
I really don't understand Mike Cardwell's objection; I don't think what I'm saying is controversial at all. I actually thought I was making a relatively banal point.
I think you're having trouble with the intended target of the word "you" in my comment. I'm not writing to people hosting Minecraft forums on Dreamhost.
Perhaps you should have been more careful with your wording. It demonstrated a lack of understanding of real world configurations and requirements and implied that if you're doing it that way, you're doing it wrong. I'd guess that most websites live in shared hosting systems.
EDIT: You could have just replied to my original comment agreeing with me that shared hosting systems work that way, and that it's ok for certain types of site. It would have made more sense than your comment "Don't use shared web hosting."
I'm really not sure what you're hoping to have me concede here. If you operate the kind of application that people on HN tend to operate, you should avoid shared hosting. I work with and enjoy talking to people who are serious about running applications, and I provide advice to people who are at least somewhat serious about security.
If you don't fit either of those molds, I don't think any less of you, but I'm not going to tailor my advice to you either.
It really sounds like you're just looking for something to be indignant about. I don't know you or anything about you, so I had no expectation that you were that kind of person. Consider addressing your objections to the thread, instead of aiming them at me, if you'd like to avoid that appearance. For truly, I do not care whether you like shared hosting or your friends are struggling indie shared hosting operators. That's not relevant to me even a little.
A less personal way to frame your objection, rather than "Are you commenting just so you can be downvoted to oblivion", would be to write a comment that starts with the words "There is another side to this that readers should consider..." and go from there.
I'm not using shared hosting, no. Although I did work for a company that provided it a few years back. I'm just trying to figure out if you're actually being serious with your claim that everyone with a database backed website should be running it from isolated servers? If you actually understand the implications of this or if you're just making nonsensical off the cuff remarks? I wonder how many tens or hundreds of millions of servers that would add to the Internet.
Of course I'm being serious. Don't run serious applications from the shared MySQL databases at shared hosting providers. You seem shocked that I'm saying this, but we work with a lot of very young startups and I have never met one running their app off a Dreamhost-style shared server.
Tens or hundreds of millions of database servers? That's hyperbolic.
Do you really think that we've come close to eliminating all the vulnerabilities inside a MySQL session, post-authentication? Because what you're arguing is effectively that application owners should trust that MySQL is resilient against attackers who can get an authenticated handle to their own database and run nearly arbitrary SQL statements against it. You think all the code in the MySQL query parser, the planner, and the various storage backends have been fully audited? This is a project that didn't even get authentication right.
Again, you're completely ignoring shared hosting systems. You have a very strange understanding of what "real-world applications" means when it comes to MySQL.
One term for the applications you're thinking about is "certified pre-owned". Michel Zalewski just made a handy logo for those people to slap on the bottom of their front page:
Best? Probably an infrastructure VPN across databases. Have you database listen on the MPLS link and limit access to the service based on IP addresses and strong authentication.
Cheapest? SSH tunnel or openvpn point to point link.
I agree with the sentiment, but just to be clear: There are valid reasons to allow MySQL to talk to the wider internet, the most important one being off-site replication.
If you do need your mysql to be exposed to the internet you can use bind_address=x.x.x.x or some kind of firewall.
Before we forked LedgerSMB, the SQL-Ledger author's attitude was that unless you can exploit the software without logging in it doesn't count. It's only accounting software anyway so why would anyone want to break in? Not only that but timestamps were perfectly acceptable as session id's and they didn't even have to be stored on the server, just checked to see if they were recent.
Thus began years of efforts on our part of security fixes, which I would not have started except that I had customers to support.
From the mysql commit: Date: 2012-04-06 09:04:07 UTC
That's two months ago. Looking at the changelog (http://dev.mysql.com/doc/refman/5.1/en/news-5-1-63.html), they piled in a bunch of other changes like "use less disk space". This should have gone out pronto. I feel it's not the kind of thing you sit on until your next quarterly release is scheduled.
[oh wait, this is worse. mysql 5.1.63 was actually released a month ago. But they only now tell us what the security bug was? Meanwhile the bad people have had a month to diff sources? Double unhappy.]
What ever happened to public disclosure as soon as a patch is released?
I have been bit twice by problems relating to difficulties getting security fixes/announcements out in time. In my defence I was trying to patch a difficult codebase and this just took a long time. For example, there was a full SQL injection audit that took us two months to complete early on, and there was an issue of XSRF vulnerabilities which could not be effectively patched in a production release.
But waiting a month to announce the security issue after the release was out strikes me as hard to justify.
Unfortunately Oracle's stewardship of MySQL appears to be a closed model. There is no public access to their bug tracker, and distributions struggle to keep up with security updates because the details of their fixes in the source are not published. The future of MySQL appears to be in one of the MySQL forks.
This is also important in other environments, for instance shared hosting where you may connect to localhost, or places where you may have given non-admin shell access to a developer (assuming they could not connect to mysql root user).
This is a serious vulnerability. Especially since the latest ubuntu seems to be affected(I'm on mint 13, and it is)
See Ready shodanhq query for latest mysql version:
This vulnerability could also assist with local privilege escalation. If an attacker managers to use a vulnerability in a web application to execute code as a web user account, they can likely access the MySQL server instance via the loopback interface (perhaps a Unix domain socket too?).
I've been trying this on lots of our customers' boxes and can't exploit it - no matter how many times I've tried I always get turned away when retrying root's password, e.g. trying "while true; do mysql -u root mysql --password=baha; done" does not yield access on any of:
Debian lenny 32-bit 5.0.51a-24+lenny5
Debian lenny 64-bit 5.0.51a-24+lenny5
Debian lenny 64-bit 5.1.51-1-log
Debian squeeze 64-bit 5.1.49-3-log
Debian squeeze 32-bit 5.1.61-0+squeeze1
Debian squeeze 64-bit 5.1.61-0+squeeze1
Ubuntu lucid 64-bit 5.1.62-0ubuntu0.10.04.1
So I'm not inclined to think it's as bad as made out by the simple exploit above.
It sounds like they were casting the result of a memcmp to a char. A char only has a range of -128 to 127. The resulting overflow means that an arbitrary password hash has a 1/255 chance of landing on 0, but you still have to try a bunch to hit one.
memcmp will probably return always the same value for the same parameters, which may not trigger the error. Trying different values may yield a value whose mod is 0.
Whether a particular build of MySQL or MariaDB is vulnerable, depends on
how and where it was built. A prerequisite is a memcmp() that can return
an arbitrary integer (outside of -128..127 range). To my knowledge gcc
builtin memcmp is safe, BSD libc memcmp is safe. Linux glibc
sse-optimized memcmp is not safe, but gcc usually uses the inlined
builtin version.
How do you know, how the Ubuntu devs compiled their mysql server?
Try to connect to MySQL as root with a made-up password several hundred times. If you successfully connect, the bug is present and you know how it was compiled.
Well, the Ubuntu part came from HD Moore [1]. I haven't been able to confirm it on my Ubuntu 12.04 virtual machine instance though, nor does my virtual machine appear to trigger the bug using the CVE-2012-2122 checker [2].
But, that is just my single VM instance and I would assume HD Moore knows what he is doing.
Yep, if you're blocking remote hosts to authenticate on 3306 (or any other port you're running mysqld on) you're safe. The attacking host can't authenticate itself, so it's unable to exploit this bug.
There is a check_scramble function which returns a my_bool, presumably a char typedef. That function itself directly returns the result of memcmp. If your memcmp implementation returns the full range of int values (allowed), 1/256 of them will have a 0 low order byte, which will then compare equal to zero when the check_scramble call is tested.
This is why real C programmers use int as their bool type. :)
I was at first tempted to complain about useless pedantry, but you're right. Even if some of us still like to party like it's 1989. I should have said the real lesson is don't reimplement standard C types, poorly.
Has anyone managed to actually repro this. I've tried it on a wide variety of systems I run and no repro. Just looking for anecdotal data on how many systems are affected. To me it doesn't seem like a high percentage.
I am trying to make it work on my Ubuntu 12.04 VBox VM. I have mysqld Ver 5.5.22-0ubuntu1, which is the supposedly affected version. I can not reproduce it.
I thought at first it might be because I do not have a root password (it's a VM) so trying to use any password at all returns a failure before the faulty code is reached. Then I tried to login as debian-sys-maint and some other users that do have passwords, and it still did not work.
I couldn't get it to reproduce on a VM (VirtualBox) either. I'm wondering if the SSE-optimized version of the glibc doesn't work the same way in a VM as it does on host hardware (i.e. SSE instructions are virtualized to some degree). Since the hardware doesn't support it, the library falls back to a version that won't trigger the bug.
It seems that SSE4 extensions are needed to be vulnerable, otherwise memcmp() is doing classical computations.
So you need a 64 bits system, and be sure that you are not using a virtualisation system which does clear SSSE4 flag in /proc/cpuinfo (VirtualBox does).
Because when the return value is truncated to a char, the last two bytes in the int would have to be 0x00, which is 1/256 assuming the return value is a completley random int.
Ubuntu always ship fucked up, broken, shitty MySQL versions.
Look at the one that is current HEAD on 10.04 LTS. It's got so much broken stuff in it, we had to move everything to a spare windows machine where we could stick a later version on without screwing up the machine (DBs for: team city, jira, crucible).
Why migrate to a completely different environment? I'd recommend just installing some version of percona server (mySQL flavour) instead. You get both better software and upstream version instead of a repackaged one.
It is but they've fixed it in MySQL and Ubuntu have pinned the HEAD mysql build to the broken one and have pretty much stopped supporting it. LTS my arse.
The bug is most definitely in mysql. Its manifestation depends on compiler/library, and it appears ubuntu is the most prominent afflicted platform, but there could be others. Relying on the bug being limited is very thin ice.
It depends what version they are using. My local dev machine (using Precise) is vulnerable, so Slicehost customers running a more recent version than Hardy might be affected.
That's a horrid assumption, is verifiably wrong and what in God's name are you doing on Hardy? You just screamed "ignore me" three times in two sentences.
I'm not the poster, but Hardy is an LTS release that is still supported in server configuration. It's quite a bit newer than Centos/RHEL 5, which is widely deployed on servers (potentially the most popular platform for existing deployments, still).
