I have wondered why FB didn't use a proper language with proper typing to begin with. I mean, I "understand" logistically: they already had a giant codebase in PHP, migrating a codebase is expensive, and it's difficult to hire and train 1000s of hackers in e.g., OCaml. (They do have some OCaml people, but they are outliers. OCaml was my favorite thing to write there, though it didn't afford some of the same niceties and interactivity as the PHP code they had, only because the support was down by several orders of magnitude.)

But at the same time, layering FP with a home rolled static type checking server (??) is bug prone and is certainly yak shaving (which they have time and money to do). Now they've written (1) a compiler to C++, (2) a compiler to VM byte code, (3) a corresponding runtime for each, (4) extensions to PHP, (5) a type checker, and (6) an inference engine. That's a lot of stuff. And in the end, it's still PHP, which is duly disliked. (Though Facebookers don't seem to care. The prevalent attitude toward it is that "PHP, as it's coded here, is mostly like C++, and that's OK.")

Writing correct type checkers and inference engines is kind of difficult. They seemed to take the approach of just building onto it incrementally until it just seems to work. That approach led to many bugs in many cases that just simply aren't thought of when one is trying to build inference engines by hand, as opposed according to theory. Type checking and inference is an area ripe with theory and attached formal, mathematical semantics. Standard ML's standard is perhaps the most infamous; it's a collection of mathematical statements about the language. That way, the compiler is now almost an engine to prove your code is correct. I don't see how the same guarantee can be made with something that is just cobbled together.

Because perfect is not the enemy of the good? Because "build atop a crumbling foundation" has demonstrated time and again to be, by far, the most successful way to accomplish anything in computing? Unless you have some example of perfect, now dominant, technologies that have been created ex nihilo that I'm missing? I mean we (facebook) are still using PHP and MySql, improving both. And when we need to break things out, we head into C++, the queen mother of castles on broken foundations.

> But at the same time, layering FP with a home rolled static type checking server (??) is bug prone and is certainly yak shaving (which they have time and money to do). Now they've written (1) a compiler to C++, (2) a compiler to VM byte code, (3) a corresponding runtime for each, (4) extensions to PHP, (5) a type checker, and (6) an inference engine. That's a lot of stuff.

All languages, runtimes, and standard libraries (and databases, and source control, and on and on) are "broken" at sufficient scale. You're going to be spending time rebuilding things other people take for granted no matter who you are and what language and technology you are working in. The underlying assumption that a "proper language" gives you these things for free is completely false.

> Writing correct type checkers and inference engines is kind of difficult.

Just so we are clear, you ask why Facebook didn't rewrite 10,000 human-years of code into a mythical unnamed "proper" language, but you consider writing a type checker to be "difficult". I think you might have vastly inaccurate pictures of what is and isn't "difficult".

> That way, the compiler is now almost an engine to prove your code is correct. I don't see how the same guarantee can be made with something that is just cobbled together.

Computing history is littered with dead projects from people who believed that anything less than perfect is unworkable or non-valuable.

I can't imagine where that sort of conclusion comes from. Building on a crumbling foundation seems to be just about the most proven, reliable way to ensure your software project won't survive more than a short time without needing serious effort just to maintain it and/or a big rewrite.

Much of the world still runs on C, a language that was created long before many of us here were born. Other languages have risen, peaked, and fallen into relative obscurity since then, yet C endures, because for all its faults, it is simple and predictable, the epitome of a sound foundation.

Large amounts of COBOL is still driving back office systems in large organisations. The cost of hiring people with the skills to maintain them is probably horrific, but those systems are still there, doing their jobs decades later.

You can still run applications from nearly 20 years ago on Windows today, in no small part because of Microsoft's persistent focus on compatibility and keeping the basics reliable over that time. Similar stories can be told in *nix world.

What major accomplishments in computing that have been built atop crumbling foundations can claim anything even close to the scale of success of these examples? You surely can't be talking about the move fast and break stuff philosophy that seems to drive everything at trendy software shops like Facebook and Google, or the kind of MVP/lean start-up hype we hear about ad nauseam on HN and the lasts-just-long-enough-to-exit web apps that result.

You aren't actually going to claim that Windows was well-conceived and theoretically well-founded, are you? Windows has been a never ending refinement on exceptionally shaky ground.

Incidentally, my point about languages like C or operating systems like Windows was not that they are theoretically wonderful under the hood, merely that they provided a reliable foundation. C has been standardised for a long time and is widely portable. Code that was designed for early incarnations of Windows will often run with little modification even on today's systems because the essential underlying models and APIs have been diligently preserved over the years even as many other changes were going on around them.

Or is it that maintenance is difficult and programmers will run into all sorts of ugly corner cases and features that are just grafted onto each other? Because those apply to C and C++ and Win32 and Google and basically every other large software system as well. That's the point the grandparent is making - if you look at pretty much any successful, evolving software system under the hood, you'll see a byzantine mess of complexity, and it's a wonder that it ever works.

Clearly this isn't an absolute scale. As our industry evolves and we develop more reliable ways to achieve our goals, something that we regarded as being a relatively stable foundation in the past may no longer be regarded as such in the future when our standards have risen. Moreover, what constitutes "long term" might vary wildly among different projects.

I suppose my basic objection to the original claim (that building atop crumbling foundations is by far the most successful way to accomplish anything in computing) is that significant achievements in computing tend not to happen overnight but rather to develop over time, and the more stable your foundations, the better chance your project has of developing far enough to achieve significant things.

I think the other posters are using a definition of "crumbling foundation" as "one which most engineers hate, which slows them down through excess complexity". And by that definition, almost all successful projects are built on crumbling foundations, because the fact that the project was successful leads you to add features to it, and adapt it in ways that the original architecture didn't anticipate. This process only ends when the software becomes so complex that all further attempts at modification fail, at which point everybody hates the codebase and it is, by pretty much any definition, a "crumbling foundation".

Not at all. I'm arguing that in computing, worthwhile results often take time to achieve, and therefore that foundations that are likely to be around for longer will improve the chances of achieving such results. Alternatively, from the opposite point of view, the odds of achieving something worthwhile go down significantly if you have only a short time time to achieve it, as inevitably you will if you are building on foundations that aren't themselves going to be around for long (whatever we choose to call them).

What point are you trying to make?

I had this question in mind while reading your reply. What point are you trying to make with your comment? For my part I wanted to point out that it's unacceptable to say PHP powers 70% of the web. (Check my previous comment's parent.) It has nothing to do with Facebook. And I didn't allude to your inference that choice of language doesn't affect the overall dependability of web applications.

In this case you are mostly hand-waving, and what you say amounts to "I don't like X language".

I think you need to go back and read the message you replied to. He is not arguing that at all. He's pointing to a long range of other things that he is asserting are also crumbling foundations.

Most of the time, you're talking to Win32/64 or high-level services based on DCOM or .NET, where the "well-conceived" and "theoretically well-founded" stuff doesn't turn up. You can go your whole career without knowing that there's a well-designed kernel under all that cruft.

I'd guess that less than half of Windows developers could say what the object manager does.

PHP could have been easily done in any language with native compiler toolchains.

Yes. Or rather, other people can be critical too, but language designer' opinions have far more validity.

In other words, anybody can say whatever uninformed BS he wants (it's a free country). But that's no replacement for being an expert in what you're discussing.

>Since programmers can vote with their feet and gravitate towards better languages (is it miraculous that almost all programmers have a distaste for PHP?)

You'd be surprised. PHP is one of the most popular languages and one of the most used languages, so far from "all programmers have a distaste for PHP". So if we were to use that "voting" argument alone (which I find wrong), PHP should be considered a very good language. Not the intention you had, I guess.

While PHP has it warts, it's mainly the less pragmatic and more fad prone programmers that have issues with PHP, those who look for silver bullets and like to feel superior by choice of programming language, editor and the like.

As for programmers "voting with their feet", well, they don't do quite a good job at it. The best languages (like LISP, Smalltalk, OcamL, Haskell, to name but a few etc) are seldom the most popular too.

Here's the case we're talking about the work of a language designer. By the way, not all people capable of creating a programming language have created one yet. Marc-André Cournoyer, who wrote the book that Jeremy Ashkenas learnt language design from to write CoffeeScript, has not created a language himself.

There were far more evolved and elegant languages at the time C was created (and during the time it took for C to rise, even more were made). LISP for one, but also languages with the same performance characteristics and systems programming capabilities as C.

C definitely wasn't seen as a "great language design" -- just a very useful and pragmatic one (e.g see also the classic "Worse is better" essay).

Compared to C++ a language made of explosions and guttural sounds is an elegant way of communication.

When your compiled C program sometimes runs on x86 and sometimes doesn't, that is the definition of reliable.

But these days, most interesting applications run on C++, which started from the arguably crumbling foundation (from C++'s point of view, not per se) of C, and grew organically over several major revisions into something hideously complex.

This trait of C++ is not a good thing, but the amount of successful software written in it seems to prove that it's not fatal either.

> You can still run applications from nearly 20 years ago on Windows today, in no small part because of Microsoft's persistent focus on compatibility and keeping the basics reliable over that time.

Layers of hacks on hacks to keep old software running correctly is exactly "building on a crumbling foundation", and probably the reason Microsoft is trying to get rid of Win32, having severely limited its availability on ARM in favor of the WinRT APIs. But I'd say the venerable success of Win32 demonstrates that the crumbling foundation works.

That depends. It could also be argued that our current tools are limiting what we can accomplish and at some point we'll need a revolution, otherwise we'll hit a ceiling, just like when concrete replaced the need to carve and place rocks on top of each other.

For example living organisms are way more adaptable, more self-healing than anything we've ever built. Our own body should be a text-book example of massive parallelism involving trillions of independent agents that cooperate with each other. In particular, the process of wound healing that happens when you cut yourself is quite fascinating: https://en.wikipedia.org/wiki/Wound_healing

If we wanted to simulate the human body, or at least the human brain, since that's the most interesting part, somehow I'm not seeing C++ in that picture.

Wikipedia and Facebook?

Its like trolling a raw cast iron hammer used to build a furniture factory and then saying the whole factory is entirely useless just because some invented a stainless steel hammer.

The opposite opinion is basically indefensible. Sure, you can still dig a trench with a spoon (and if you have enough money to wield a bunch of workers with a spoon), even if a shovel would do a better job.

Let's begin.

1) PHP autocraptastically converts strings that look like numbers, into numbers, resulting in all sorts of weirdness like this: https://eval.in/111886

2) PHP 5.4's OWN TEST SUITE has 91 failures and only 70% coverage. There is NOTHING more "WTF" than that! Why even bother having a test suite?? http://gcov.php.net/viewer.php?version=PHP_5_4

3) Why the fuck are all of these different things equal, and how does this NOT result in problems? http://i.imgur.com/pyDTn2i.png

4) String increment is dumb to begin with, but why does it not even match the behavior of string decrement? https://eval.in/60631

5) Why the hell can you jump back into a try block from a catch block? Recipe for disaster: http://phpmanualmasterpieces.tumblr.com/post/33091353115/the...

6) PHP comparison operators. I'm sorry, but this level of complexity might make you feel smart once you master all its idiotsyncrasies [sic], but it's actually dumb: http://stackoverflow.com/questions/15813490/php-type-jugglin...

That's a small fraction of not-thought-out PHP language features that result in REAL bugs and security holes. Which consume large swathes of programmer time. Which, apparently, Facebook can afford to swallow.

I'm sorry, but your position, as valiant as you are defending it, is literally indefensible. And I don't give a fuck how big Facebook is, they would STILL be better-served by switching SOME of their code to a different language. ANY modern programming language wouldn't suffer from this imbecilic, immature language design.

The question more pertinent to your argument is, did they make a mistake years ago choosing PHP? That's when the could have conceivably gone with language X.

BTW the types of stuff you're listing are documented here. So thorough it's amusing to read: http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-de...

And yet Twitter has slowly migrated on the backend from Ruby to Scala/Java. They still use Ruby for the frontend, though it's not clear to what extent, since they've also migrated to a single-page, fat client design on the desktop. And at the very least, their choice of using Ruby when prototyping, was at least sane.

I understand that large codebases can't be migrated easily. But you can migrate individual components when needed, if you have a modular, service-oriented design.

Also - building new functionality in PHP is indefensible, unless their code-base is one big monolithic hairball, which I doubt it is.

An excellent point. Which in fact is an argument in favor of modularizing your code as much as possible, as early as possible. There are tools now like Apache Thrift which make this easier: http://thrift.apache.org/

Option #2 ended up getting a lot of publicity for the engineers. If they chose option #1, not only is it a lot of hard boring and possibly error prone work, there is no chance for any of the engineers to get the type of publicity they are currently getting. All in all, engineers tend to work on whatever will make them get noticed, not necessarily the better technical choice nor what is in the best interest of the company... this is especially true in companies like Facebook and Google where there are a lot of very smart engineers doing relatively mundane work. So there you have it Hack/HHVM are all just publicity stunts, the more you feed it, the happier their PR gets.

You don't need to rewrite anything (at least, not all at once.) Personally, I'd have expected you to make something akin to CoffeeScript or ClojureScript that targets PHP, and can "link with" your existing PHP modules (or rather, with their HHVM bytecode representations.) Then treat the PHP code as a constantly-dwindling Big Ball of Mud (http://laputan.org/mud/).

We took a similar approach to what you described. Then we called it Hack.

This is what we mean by seamless inter-operation: the HHVM runtime understands both syntaxes and runs both <?php and <?hh code in the same process. Whether Hack integrates into the runtime at the parsing (current) or bytecode (future?) layer is an implementation detail.

If you mean that future versions of Hack will evolve to have a different syntax, while still targeting the runtime... then you'll still have code around from the intermediate era, and you'll have to interoperate with that too, won't you? You'll have PHP, PHP-looking-Hack, and actually-nice-to-code-in-Hack.

That's not to say that syntax doesn't matter, but semantics and pragmatics tends to matter more, despite Wadler's Law of Language Design stating that most people don't understand this.

I also wouldn't count Clojurescript here. Its a whole different language that just happensto compile down to JS.

That's only because the currency for building things on top of crumbing foundations has been sweat and man-power. We aren't that far off from the Egyptians that were using hundreds of thousands of slaves per pyramid. It's a good thing that we've transcended the necessity for hundreds of thousands of slaves when raising buildings, don't you think?

And yet, here you are, claiming that building stuff with broken tools is the most successful way to accomplish anything in computing. Actually I view it as nothing short of a miracle, showing human determination in action ;-)

> All languages, runtimes, and standard libraries (and databases, and source control, and on and on) are "broken" at sufficient scale.

That's a fallacy. Just because both X and Y are broken, that doesn't mean they are equal, as some things are more broken than others and PHP is more broken than anything else mainstream (C++ at least has reasons). Also, I don't see how "at scale" changes things in PHP's favor, I really don't.

If you're trying to argue that "at scale" the level of brokenness converges to the same levels, then that's a stupid thing to say. After all, Twitter didn't had to build its own JVM and the stuff they run on top of the JVM is probably more power efficient than you'll ever be with HHVM. Probably saner too.

False, they were skilled, paid laborers: http://en.wikipedia.org/wiki/Egyptian_pyramid_construction_t...

Apologies for being irrelevant to the main point, but this is a tired myth.

Facebook can keep PHP for the thin web layer that renders the page, but they could have migrate the meat of their code to a safer and more robust language.

a) try your new code by saving in your editor, and hitting reload in your web browser.

b) it's very approachable. People who only know HTML and CSS can be expected to do a little bit of PHP work to integrate their changes. If you setup the right network mounts, they just need to edit files and reload (see a)

c) it's not super high overhead at runtime. If you're not using a framework, and you don't build up a crazy object hierarchy, it's not too hard to get your page out with about 10 ms of overhead beyond data fetching. For very simple webservices (fetch data, possibly from multiple sources, and do a little formatting for the consumer), I was able to get the overhead down to 2ms. You can certainly do better with other languages, but you can usually get better throughput improvement by working on getting data quickly. Btw, all the frameworks are terrible; many of them add 100 ms to the page just for the privilege of loading the includes; PHP is a framework for web programming thank you very much.

d) cleanup; you don't have to worry about it. If you don't do anything weird (c extensions, with non-preferred malloc), at the end of the request, everything is thrown away.

That said, there are plenty of things PHP isn't good at: I wouldn't run a long running process in PHP; and multithreaded PHP sounds like a bad idea.

As for a), many (most?) frameworks are able to monitor the source files and reload the application when they change, in order to enable that workflow. For example: http://cherrypy.readthedocs.org/en/latest/refman/process/plu...

P.S. The vast majority of arguments against PHP here leads me to conclusion that most of (not all) debaters don't understand how PHP works well enough.

It is a job scheduler which takes jobs from a Redis queue and executes it.

It does fork a new process for 1 type of task (image resizing in this case) to make sure it doesn't leak memory.

It also reuses these forked processes so it doesn't need to fork for every task.

It was last started on Jan. 2013 (because of maintenance) and still running fine.

The reason ? I didn't want to introduce an extra language for project for which all the server code was already written in PHP.

So if it fits the task, you can do it.

The only language for which that works is client-side Javascript. For PHP, you forgot the part in which you have to install and run a web server, then point your browser to localhost. Plus, to get anything useful done, you'll also need some sort of database to go along with it. I remember the first time I did that and it was pretty intimidating.

> c) it's not super high overhead at runtime ... it's not too hard to get your page out with about 10 ms of overhead beyond data fetching

To me much more interesting is the total time it took for the client to receive the response, possibly when multiple concurrent requests are happening. The comparison here should be versus a static page served by Nginx of course.

The best throughput possible that I got for an otherwise complex business logic happened on the JVM. I basically rewrote a web-service built on top of Django/Python, with a redesign with emphasis towards in-memory caching, parallelism and async I/O and the result was a server that was able to process more than 10,000 requests per second with an average of about 5ms per request (actually in production the instances ended up processing about 2000 reqs/sec of real traffic, since c1.medium EC2 instances don't have enough CPU power).

Of course, people that just need to slap something together, don't need this level of throughput. If a request takes 400ms for a dumb web form on a low traffic website, that's of no consequence to most people. The problem happens of course when such a piece of software evolves to something much bigger, like Wordpress. I'm always amazed at the gimmicks that people do just to keep their Wordpress powered blog alive.

Is there anything about PHP, the language, that lends itself to this style of development, or could you get these same benefits with, say, Ruby or Python?

b) Not everyone has programmers. That was his point

I was to defend myself in small claims court

I want to learn some first aid

I want to build a shed behind my house

> I have wondered why FB didn't use a proper language with proper typing to begin with.

I would suggest watching Keith Adams's talk, "Taking PHP Seriously": http://www.infoq.com/presentations/php-history

He goes through why Facebook uses PHP and decided to build upon it to create Hack. I highly recommend watching the whole thing, but the main three things he points to are:

1. Frictionless programmer workflow with a short feedback cycle

2. All PHP requests start out with the same consistent state by default

3. Rigid style of concurrency

1. I have never seen a framework that didn't go to great lengths to update to new changes quickly in development

2. Which means you lose resources between requests, unless you stuff them into the interpreter/httpd itself. And anyway this is only a problem for PHP, where by default everything runs in the top-level namespace, versus in separate functions.

3. That's a funny way to spin "no concurrency", but you can get that in any language by just not deploying it threaded.

um. have you used java?

The language definition and semantic checker are difficult, but are stereotypically tasks that cannot be distributed to many engineers; instead, a few senior engineers took that task to the benefit of all others.

Congratulations for admitting your ignorance, and lending open ears to experts as to why they made certain engineering decisions.

Oh wait, you weren't doing that, you were just warming up to go on a diatribe about how stupid Facebook engineers must be.

Yes, they made certain engineering decisions now because the decisions they made back then were stupid, and they have to dig themselves out.

That Zuck wrote the first version of thefacebook.com in PHP, the language he was the most productive at?

That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase? If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be. I've personally experienced and heard only horror stories about rewrites. We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.

And it was also written in PHP. And we rewrote in Ruby (not rails, at the time it was not mature in the ways we needed it to be). We made a lot of great technical achievements in doing this and our codebase became much much better. On a technical level I don't regret us doing that one bit. We went long on schedule and we made mistakes, but some of the best work of my career went into that and I'm immensely proud of it.

But then in a matter of months of Facebook going to being open to people who weren't at a university or big organization (remember that?) our users, who got bored with our site not changing while we rebuilt all the tech, completely abandoned us and we went from profitable (having never taken any outside investment) to dead in another couple of years.

This isn't to say that this wouldn't have happened to Facebook had they done something like this, but it is always a huge risk, even if you do everything right.

Using PHP to begin with was a bad technical decision. Failing to establish a reasonable migration strategy was a bad business decision likely rooted in bad engineering management that fell out of starting with bad technical decisions.

It's much harder to hire people that can pull you out of a mess like PHP when, at the same time, you have to hire people that can keep writing PHP for you.

> That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase?

That would have been a good time to bring on new engineering blood as part of scaling out, which would have provided opportunities to enact mitigation and transition strategies. Imagine if the massive amount of talent currently devoted to HHVM had been devoted to Facebook's actual business?

There are migration strategies other than "rewrite everything immediately", and in fact, I'd bet that's exactly what HHVM is. It's just a shame they waited so long that the most cost-effective migration strategy was to tackle an enormously difficult computer science and engineering problem that the world's biggest software companies already invest hundreds of millions of dollars on and provide to the world largely for free.

> If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be.

Yes, and I've also (repeatedly) been the team brought in to rewrite the mess of a code base that was about to torpedo the growth phase.

There's not much correlation between funding, initial success, and engineering talent. Which is why you so often wind up with a mess that has to be cleaned up once you can hire people who know what the hell they're doing, instead of the ones you happened to be stuck with because you didn't know how to grow an engineering team.

> We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.

Messiness is never there for good reasons other than that replacing it is more expensive than not touching it. You don't strive for hidden wisdom and inescapable messiness -- that's just what you get when you let engineering slip up.

Which language would have been a good technical decision in 2002-2003? It needs to be fast enough in terms of iteration. It needs to not require more resources than PHP. It must be easy to onboard people who don't know it onto. It needs to be easy to operate, and not be costly to deploy on the tens, hundreds, and then thousands of servers necessary. (Spending time learning a new technology that others think is cool or which seems cool, while trying to also build a product, has sunk more than a few startups...)

What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth. What was bad with the decision to let hundreds of software engineers continue to build the PHP code-base while a much smaller group of people work on projects to improve the efficiency of both the execution environment but also the tooling and developer efficiency on that code-base? Their contribution there is multiplied out by the improved efficiency of those hundreds of developers and the code they wrote.

Seemingly Facebook survived its growth phase fairly well and didn't need a tiger team from outside to handle it - and without the reliability problems others who chose to use languages other than PHP for their front-end and decided to rewrite their much smaller surface area in other languages.

As much as people may dislike PHP (and I'm one of them), it was definitely "good enough". Many languages may not have been, even if they are nicer languages in some objective way.

(Disclaimer: I actually have to write code in Facebook's PHP code-base every once in a while. But most of the code I work with is Python, Java, or C++.)

You mean, like the JVM? 2003 wasn't the pliocene epoch, we had a working JVM. If you remember back to the last bubble, in the 90s, we were shipping "easy to operate, not costly to deploy" software on Java post-1998. Java 1.4 was released in 2002, and Java 1.5 -- what most people would say is modern Java -- was only 2004.

Scala 2.0 was released in 2006, Clojure in 2007 -- that's 8 and 7 years ago, respectively.

You really don't think there were alternatives during that long period?

> What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth.

In 2004? Nothing. In 2005-2008? Things should have been reassessed, especially before building out a millstone of an engineering team around PHP. Instead, Facebook doubled-down on an actively bad language with HPHP, and the results were hilarious:

   HPHPc required a very different push process,
   requiring a bigger than 1 GB binary to be compiled
   and distributed to many machines in short order.

That's a span of 6 years, and at the end of it, Facebook has functionality they could have gotten for free in 2003. On top of that, the intervening years allowed the PHP mess to become only more entrenched -- who on earth do you think the engineers are that accept a job writing PHP, for Facebook or otherwise?

If I had to hazard a guess, I'd guess that HHVM exists because of a large amount of political inertia in the organization that has everything to lose by PHP being eliminated entirely, and the lack of a strong hand by upper management.

I'd guess that lack of a strong hand by upper management came in no small part from hiring straight-out-of-college graduate Adam D'Angelo -- who had literally zero experience -- to serve as CTO from 2006-2008.

By the time FriendFeed was acquired and Bret Taylor along with it (2009), my guess would be entrenched interests made for a very difficult position for anyone wanting to change the ship's course.

They specifically choose to do it and not move to an other language. One of the reasons was: PHP programmers are cheap and plenty full and can do quick iterations.

Sounds a bit like you are complaining about other peoples choices, it really is their choice. :-)

I'm not saying it is possible to move to an other language. Just look at Paypal they moved their customer-facing code from Java to node.js and got a very large productivity increase: http://www.youtube.com/watch?v=V5yk5SZxWX4

Obviously the reason Paypal choose node.js are similar to why Facebook choose PHP: quick iterations, means more iterations, which means more experimentation and better results.

In 2002-2003 I'd have quit in disgust if anyone tried to introduce the JVM anywhere I worked.... I still probably would, frankly.

Facebook is a ten year old company with a market capitalization of $170 billion, so "bad" is probably not the most accurate word to describe their technical and business decisions.

How does that follow, exactly? They haven't failed, so any inefficient or sub-optimal decisions were the correct decisions?

Facebook is surprisingly easy to underestimate even as someone using it a fair bit.

Just try find every single interface in the front-end. For your own profile. Feed is front-and-center, as is timeline. Then look at events, groups, &c.. Look at messaging.

Then look at the interfaces for managing your privacy and permissions. The apps that you're using and information about when last they requested information. Security like login approvals. Then think about the flows involved in reporting content as abusive or inappropriate. For verifying your identity if you've forgotten your password. For adding more security if you log in from a new computer or from a new location.

Then look at pages, including insights and scheduled posts and so forth. Then look at advertising - boosting individual posts, creating campaigns, &c.. Then look at the interfaces for developers. For translators. Interest lists.

The backends for most of this are in C++ and Java. There's a large amount of data processing happening to track hidden things like spam and scam prevention. But the front-end surface area is quite clearly an order of magnitude or maybe two larger than Reddit (at least where it was when this happened).

Rewrite is more likely to succeed if you are not in a high growth phase, but even then it's risky.

Trying to build a reasonable forward-looking high-performance managed language runtime platform is hard.

Trying to build it atop of PHP is harder.

The only way I could see this as being a smart long-term strategy is if the eventual goal is to isolate and retire PHP projects entirely (and PHP usage itself) over time.

However, even then, with PHP gone, and Hack no longer necessary, you're still stuck maintaining your own incompatible VM / runtime. Is Facebook signing up to reproduce the CLR? Or do they have long term plans to somehow bridge the gap between HHVM and more established VMs/runtimes, where they can better share the maintenance load with the wider industry.

As someone who has worked on a VM, I couldn't disagree more. It can take years to hash out things as simple as ideally performing primitives for a target architecture, and then things change.

Add to that the complexity of optimizing compilers, specification of byte code formats and a consistent virtual machine memory model that can be relied upon across architectures, and the art and science of highly concurrent garbage collectors, and your "no big deal" is a load of hogwash.

Hotspot alone is nearly 20 years of big deal.

> A team made of relatively small number of people (high quality, but low quantity) can and do maintain VMs ...

The number of people doesn't matter in this equation; your small team of (expensive, rare, high quality people) can't build a world-class VM in a day. Or a month. Or a year. Maybe in 5 or 10 years, just ask Microsoft.

> ... like HotSpot and V8. LuaJIT is maintained by a single person.

LuaJIT's said "single person" has been working on it for what, 10 years? It's an extremely impressive implementation and I don't want to bag on it, but even still, it lags in certain areas, eg, its GC implementation isn't up to par with the state of the art.

The author's skillset is extremely rare, and LuaJIT itself is an anomaly in the field. Using such a one-off example doesn't really hold water to prove that it's ideal for a company to internalize maintaining a VM for their own custom language built on top of PHP.

On the other hand, I maintain it still is no big deal compared to rewriting Facebook. I also maintain while skillset is rare, Facebook apparently had no trouble so far and will have no trouble in the future finding (I remind you, small number of) people to work on VM. I also remind you Facebook has been working on alternative PHP implementation for 6 years now, 2 years in private(2008~2010) and 4 years in public(2010~2014). It has been profitable for them for 6 years, will be profitable in the future, and profitability does not need "sharing maintenance load with the wider industry". They can maintain it fine thank you very much. Because, in the end, VM is no big deal.

They're also continuing to propagate an outwardly facing engineering culture that will make it even harder to hire people to help dig them out of the PHP hole -- perpetuating this further.

Your argument is simply another take on survivor bias fallacy.

> I get paid to do these stuffs, and I am on my third VM/PL project now ... Because, in the end, VM is no big deal.

You keep saying that, and yet, there keep being so few high quality VMs.

Adaptive JIT and generational GC would be a good baseline. Limiting myself to open source VM, I think (at least) HotSpot, Mono, V8, JavaScriptCore, PyPy, SBCL, Racket qualify. J9, CLR, Chakra, Allegro CL also qualify, but not open source. SpiderMonkey, LuaJIT, HHVM lack generational GC. All these projects are actively developed, and there are doubtlessly more, e.g. I am not faimilar with Smalltalk VM, some of which are commercial. Research VM like JikesRVM, Maxine qualify. I believe Bartok qualifies too.

I am not sure what you are arguing for. If you are arguing for Quercus route(PHP-on-JVM), I think it's unclear Quercus route is better than HHVM route. If you are arguing for not running existing PHP codebase, I think you are being unrealistic.

PHP's APIs are ugly. It's language semantics are a bit hairy until you get the hang of it. But there are parts of PHP that are extremely elegant and easy to reason about. It's OOP support provides all that you need to produce re-usable and easily understood code.

Facebook's work on PHP has focused on largely two dimensions: reducing CPU cycles and increasing static/runtime type checking. The former is something that only really matters at massive scale: PHP is generally fast enough since most of the time PHP processes are I/O bound reading from a database or memcached. It's only for sites like Facebook where if you squeeze out an additional 10% TPS from your boxes that you will start seeing large absolute cost reduction that this level of optimization starts to make sense. On the type checking side, this is something you might start to want in any dynamically typed language when you have millions of lines of code and want to ensure basic guarantees that it will run, and is something that you'd probably see Facebook doing if they were a Ruby or Python shop anyway. It has nothing to do with PHP but with the classic dynamic vs static typing tradeoff.

Should you be writing your chat server in PHP? No. But 90% of the code you write for a large website is HTTP response code rendering HTML or JSON. PHP excels at this and you can pretty much hire any developer off the street to start cranking out code if you give them a solid foundation to build on.

It's possible that they could eventually get a total rewrite to give a better return, but frankly I don't think you have any idea of the enormity of trying to convert a multi-million line production platform from one language to another.

In any case, one does not preclude the other. Arguably, many of the changes they have made, such as gradual typing, and their ability to now slowly introduce other changes without breaking their existing codebase, means they have a platform for slowly firming up their codebase and migrate it towards a position where a full rewrite (should they decide to do one in the future) could be made substantially less painful.

I detest PHP, but I've still more than once made the choice to do apps in PHP motivated by developer availability alone.

It's not a great language, but with some discipline it is also not nearly as awful as some people like to think, and you can make up for a lot of awful with the difference in ability to hire experienced engineers who know PHP vs. many of the less common languages.

Neither Ruby or Python provide that for instance.

Though Ruby is in my opinion well designed,and with duck typing,you might not need all that Java like OOP, I dont know,I feel like having interfaces makes me understand code faster and better,

Want to understand what an object does? just read the interface,no magics,no bullshit,it's self documenting and that's important,that's why you can have these huge codebases like Zend,Symfony or Doctrine and still understand how complex elements work together. And even figure out how to use them without a doc,just like Java.

I feel I cant go to Django's source code or Rails source code and understand everything easily just by reading function signatures.

And the write/refresh cycle makes iteration pretty fast during development. But yeah,PHP basic syntax sucks for sure.

HHVM replaced the execution environment for their code with a more robust code generation/ runtime system.

Hack allows them to bootstrap their code base into higher degrees of reliability without a mass rewrite.

Also Brian O'Sullivan is one of the best people on the face of the earth to be trying to find practical ways on integrating PL research into practical engineering.

Ok, I cede the point. PHP has the retro chic and Hack's type system is fixed-gear.

Because there are factors like an existing codebase that works that trump BS idealism.

Do you also argue that C++ is built on top of a crumbling foundation, because it's based on C, a ancient language with almost no type-safety?

It admittedly doesn't have a very advanced type system, that's true.

And typedefs. Given that there's no parametric types, you run into void* quite frequently as well, so saying "only" inaccurately minimizes the scale of how much C code isn't strictly type safe.

And may I ask why you say "typedef" is unsafe? It is merely a type alias, like e.g. Haskell's and ML's "type", or isn't it?

It is, but it freely allows conversion between the aliased types.

    typedef int Feet;
    typedef int Meters;

    int main() {
      Feet height = 6;
      Meters inEngland = height; // <-- OK. :(
    }

λ> :{

Prelude| type Feet = Int

Prelude| type Metres = Int

Prelude| :}

λ> let a = 5 :: Feet

λ> let b = 4 :: Metres

λ> :t a

a :: Feet

λ> :t b

b :: Metres

λ> a + b

9

λ> let f = id :: Feet -> Feet

λ> f b

4

Actually, building on a foundation of solid statistics and probability will often result in an algorithm that essentially counts views and applies weighting rules.

As another example, Pagerank has a nice theoretical basis, but power iteration reduces to perhaps 5-10 lines of C.

http://hacklang.org/manual/en/hack.annotations.types.php

I think you're using a wrong metaphor. Facebook foundations can't be crumbling just because they're made in PHP. There wouldn't be Facebook as we know it today otherwise. You might say they used a "low quality" material to build them. I see Hack more as a better material, that can also bind with the previous one and make it stronger.

That's not really true, HPHP and HHVM share the runtime (mostly).

Disclaimer: I have no zero experience with PHP.

Clearly the home rolled stuff is working out for facebook. Twitter was/is on RoR and continues(!!) to have major downtime issues.

They blamed RoR a lot because they needed an explanation for their problems, but "many to many" messaging even at their scale is a "solved" problem and has been for decades and is fairly easy to scale.

(Think of Twitter as a bunch of mailing lists and mailboxes; you scale it by decomposing it: map accounts to virtual "buckets" that becomes the domain part, and map tweets to messages; break apart large follower lists into smaller ones and introduce a forwarding reflector; break apart large following lists by splitting "mailboxes" and doing zipper merges on reading it; add a caching layer -- this is not rocket science, and you could do it properly in any language)

Note: I think RoR was a horrible choice for them, though I love Ruby, but I also don't for a second believe RoR was their real problem.

Edit: Your overall point stands, though. Especially given that Facebook is a far more complicated application.

You might dislike it, but that doesn't mean it's disliked. PHP has a giant base of programmers, scales, is easy to learn, is extremely versatile and powerful, and as you point out, the code was already in PHP. Only an idiot would rewrite a giant working codebase simply to have it in a language that's "difficult to hire and train" in. Or any language for that matter. Perhaps they could have pulled a netscape but instead decided to serve billions.

I think the most likely outcome is that PHP will adopt some of HHVM's additional features, but remain a separate project. IMO that's a great outcome, since we'll both likely drive the other to be better.

Do you think Hack is a good language to start a new project in, compared to non-PHP languages? Are you using Hack for things besides the main web page?

Yeah, I think Hack is a good language to start a new project in. For as much flak as PHP gets, there are actually a lot of good things about the language. The fast development cycle -- edit php script, refresh -- is something amazing that you don't get in a lot of statically typed languages, which usually have a compilation step. The crazy dynamic things you can do also occasionally have their place, though it's certainly easy to shoot yourself in the foot.

On the other hand, a lot of the time you want the safety that strong static typing can give you. Even just the null propagation checking can immediately find tons and tons of silly little bugs without even running the code, and ensure that the code stays consistent as a "mini unit test" if you will.

Hack hits the sweet spot of both. Wiring the Hack typechecker into vim was really revolutionary for me -- having both the immediate feedback of the type system for all the silly bugs that I was writing, along with the fast reload/test cycle from PHP, is great.

Once you have type annotations, it wouldn't be too much of a stretch to enforce them statically with a separate tool. You could even go as far as rejecting first-party code if you can't statically determine every single value's type. Pylint's underlying astroid library has a bunch of inference tools you could perhaps build on top of.

FB already had a PyLint-like tool earlier that could do some static analysis, namely pfff (also open source and written in OCaml), but it did not provide a full-on static type system like Hack. (Background: I used pfff when I was in bootcamp at FB itself. This was however prior to hack, I worked solely on C++, Java, and a bit of Python at FB after bootcamp).

I am sure if FB started off with Python, a similar solution could have been found, but if you're looking for a tool that exists _right_ now, Hack is actually quite decent.

Creating a static type system, implement local type inference, as well as working out "gradual typing" and associated problems (all while being able to do type-checking at speeds developer _expect_) is not a trivial problem.

For that matter, can you call a typed function from an untyped one? Or, worse, a typed method? If the typing is purely static, there's no way to know the method you're calling is actually typed, so there's no actual guarantee that it receives the types it's declared unless your entire program is typed. It doesn't seem like a very strong guarantee if both the caller and the callee have to opt into the typing.

If you're looking for a tool that exists right now, you either have an existing codebase and can't port it to Hack if it's not already PHP anyway (for the same reason Facebook couldn't port away from PHP), or you're starting from nothing and could just use a statically-typed language in the first place.

I don't know if I'd even be excited about the prospect of optional static typing in Python. (It hasn't gotten me interested in Dart, for example.) I'd kinda rather see the effort poured into something that could do static duck-typed analysis/inference, e.g. balk if I pass an argument that could be a non-string into a function that tries to call `.startswith` on it. (Ah, but maybe it could theoretically be a string or None, and I only know it isn't None for reasons the type system can't see, and now I hate the type system.)

I didn't say it was a trivial problem. I just don't feel excited by the solution.

Another mode (you specify the mode per file/compilation unit) is "strict". In strict mode, you can not code any un-typed code (note, the standard library is typed with hack).

(There is a bit more nuance here, but you can read that in the documentation.)

So the idea is to eventually migrate most of the code to strict, but code that relies on legacy can remain partial and you can write new code without waiting for a re-write to finish.

See http://docs.hhvm.com/manual/en/hack.modes.strict.php http://docs.hhvm.com/manual/en/hack.modes.partial.php

"Shapes" are also a neat feature specifically for parts where static typing can be frustrating for dealing with HTTP requests specifically: http://docs.hhvm.com/manual/en/hack.shapes.php

(FWIW I don't see myself using Hack, but I'm not a web developer. I'd say the ML family languages are my favourite, but for what I do day-to-day it's not really an option.)

I'm intrigued, because it's such a good idea (especially when null's originator claimed it was a "billion dollar mistake"), though Java doesn't have it. I'm wondering if there's some subtle problem with it...?

Also, how do you make your vim typechecker fast enough? Usually, even syntax colouring is local to ensure adequate performance - and a typechecker with inference/propagation would be very non-local.

and an Optional<T> class (references to which can still be null for maximum hilarity): http://download.java.net/jdk8/docs/api/java/util/Optional.ht...

I'm stoked. And disappointed the "elvis" operator didn't make it in.

I hope some day Java breaks backwards compatibility and eliminates null entirely. Then again, that's already happened with the proliferation of other JVM languages. But that doesn't me at my day job, where we have a large legacy code base... which would need to be ported to a backwards-incompatible version of non-null Java anyway. Hm.

Well, with Java 8 I can at least start to grow null-safe code within our codebase.

You seriously think that? That's how we do haskell web development. Both yesod and snap do this out of the box. That's how every java developer I know works.

Edit: I misread what you wrote. I thought you were saying there was something fundamentally different about the type checker, but more that reload experience is different then what you get with Yesod.

How big is your code base, and is it all in one huge file or something? Our stuff is reloaded and ready before I've alt+tabbed back and hit refresh.

How extensively is Facebook using Hack at this point? Is it in production?

What has been the biggest learning/unlearning you've needed to do going from PHP to Hack?

The biggest learning step for our engineering teams was to treat type errors from Hack as actual logic errors. We have a collection of "linters" that provide advice on code style and other nice-to-have factors. Some people (quite reasonably) initially thought of Hack errors as lint-like stuff that it was safe to ignore, when in fact they indicate real logical inconsistencies in code.

Interesting, thanks!

One followup: what has it been like from an ops perspective? Similar to PHP, or is there a better frame of reference?

HHVM is its own web server (although it supports FastCGI for easier use with existing infrastructure like nginx), and that's how we use it in production. It's hard to compare memory usage, except at scale, where it benefits from not having a whole bunch of interpreters (in different processes) running at the same time and some other benefits by using more appropriate types to store values through type inference.

On a webserver, that goes further since HHVM is single-process/multi-thread, whereas PHP (in its typical setup) is multi-process/single-thread. This cuts HHVM's memory overhead much further.

Yes, PHP can run multi-threaded, but it still has a number of instability issues in that mode.

TypeScript added classes, interfaces, modules, and arrow functions.

A question about the type inference mechanism: in PHP, while it is possible to define object interfaces which classes may be defined against, by the dynamic nature of the language, functions don't necessarily need that interface specification to accept an object conforming to it, explicitely or implicitely. OCaml provides something "similar" with its object system, but much more powerful with static inference of an object (super)type from its usage. will Hack be able to infer an object interface as well from its usage in a function?

It would be pretty much impossible to implement total type-inference without loosing separate compilation in Hack. PHP projects are not organized around a module system, which means that you have "spaghetti" dependencies, and even better, cyclic dependencies all over the place.

So trying to implement total type-inference would be a bad idea, you would not be able to separate the code in independent entities and the checker would not scale.

2) Does Hack support structural sub-typing? Answer: No, but not for obvious reasons.

Fun fact, the first version of the type-checker was implementing structural sub-typing. And it was not scaling, for subtle reasons.

Hack allows covariant return types, so if we implemented structural sub-typing we would have to "dive" into the return types of each method to see if they are compatible. But in turn, these objects could have covariant return types etc ... The process of checking that was too inefficient. Caching is a bad idea (or at least a non trivial idea to implement), because of constraints and type-variables.

Since disallowing covariant return types was not an option (it was crucial to make a lot of code work), we had to kill structural sub-typing.

I hope this answers your question. As a big OCaml fan myself, I like the features you just mentioned (Well, Hack is written in OCaml), but they really didn't seem to be a good fit due to the nature of the language and the kind of checking speed we were shooting for.

We don't do any type inference across function boundaries, so we largely dodge the issue that I think you are getting at. (Please elaborate if I misunderstand!) We rely on interface and class definitions in order to know what methods are available, and even though the runtime resolves everything at runtime so you can call any method that happens to exist at that time ("duck typing"), we enforce statically that methods do exist where we can. So for example, the following code will work at runtime since the method `g` does exist, our static typechecker will reject it since it does not exist on type `I`.

  <?hh // strict
  interface I { public function f(): void; }
  class C implements I {
    public function f(): void {}
    public function g(): void {}
  }
  function f(I $i): void {
    $i->g();
  }
  f(new C());

What's your motivation, aside from modernizing Facebook's code base? What language niche does Hack serve which is not served by other languages? Why Facebook at all (aside from the rarity of finding a company to pay you to write a compiler :))?

This lets you choose the pace and extent to which you want to adopt the safety of static typing, while preserving your dynamically typed code -- and without sacrificing the rapid turnaround of PHP.

That's a unique combination, in my experience.

Typed Racket also includes more types, like ": Integer [more precisely: Negative-Fixnum]" (from docs), as well as polymorphic data structures and higher-order functions. I don't know if this is a "fundamental difference" but it might mean Facebook's type checker can optimize in certain ways that Racket's cannot.

The Typed Racket type checker is much slower than Hack, though.

Hack does seem to have polymorphic data structures and higher-order functions.

Would you consider adding even more expressive features like algebraic types or hygienic macros to the language?

Also how likely do you think it is that typical PHP will eventually be overtaken by HHVM? I'm hoping it will!

public string $x = '';

instead of

public $x:string = '';

It seems inconsistent to me, probably because I've used AS3/Haxe.

    function foobar() : int { .. }

    function int foobar() { .. }

Property types are declared before the property name, while function return types are declared after (and with a colon).

Very strange.

That syntax in hack seems more "functional" to me. That said, I must admit that I kind of like the C syntax, because of its oddity.

The incosistency drives me a bit crazy.

Do you expect Hack to be stable without large breaking changes going forward?

The documentation doesn't say much about the scheduling for asynchronous tasks. Can async functions be used to batch requests to e.g. caches and databases? Can Awaitable be used to interface with code that expects chainable Future or Promise-style interfaces?

Can HHVM/Hack use standard PHP extensions, or how much work is it to port an extension?

Is there a Hack plugin for IntelliJ?

Does the Hack project have a mailing list or forum?

Our internal plumbing is different, so the extension APIs aren't the same, thus PHP extensions can't normally be used with HHVM. Paul has a source-compatability transformer which works for simple extensions (no fancy stuff allowed), but we recommend anyone with an extension consider porting the code.

If your extension is one of those "This was written in PHP but we wanted it to be faster so we turned it into C" types, you should consider going back to PHP for it. In practice, we find that HHVM can JIT PHP code into running about as fast as (sometimes faster than) C/C++ extension code.

If you do need to dip into C/C++ (because you're calling an external library), the API we've designed is a LOT easier to work with than PHP's. And I say that with the authority of having written THE book on writing PHP extensions: http://amazon.com/dp/067232704X .

I'm working on proper documentation for extensions as we speak, but for now you can find some very basic info at https://github.com/facebook/hhvm/wiki/Extension-API .

(I'm not a php developer so excuse my ignorance)

It seems you only have 1 method 'await', to check if the async function has completed its job (it self, if I'm understanding is a blocking operation in which ever thread/worker its called).

So is there a way to run something more similar to

      $i = 1;
      while(doingSomeThing)
      {
             $b = "";
             if(completed gen_foo())
             {
                    $b += await gen_foo();
                    do_critical_task($b);
             }
             else
             {
                    do_something(do_item[i]);
             }
       $i++; 
       }

       gen_foo(get_user_data); //async
       $x = do_something_for_a_while(); //do something while gen foo processes
       do_new_thing($x, await gen_foo()); //do something with rendezvous result

Again if I'm completely off the mark please let me know.

The way it works is quite different from your example:

  async function gen_foo(): Awaitable<string> {
    echo "until we get to an await, eager execution\n";
    // ...
    await gen_bar();
    // this code is only executed after await
    return 'result';
  }
  $x = gen_foo(); // x is a handle, suspended at the point of its first 'await'
  await $x; // ... and now it's resumed
  

  list($x, $y, $z) = await genva(
    gen_foo(), 
    gen_bar(),
    gen_baz();
  ); // genva creates a wait handle for awaiting its args

  // ... and when we get here $x, $y, and $z are all assigned

Can I embed it? Or extend it via compiled binaries written in C/C++?

As the author of pretty much THE book on writing PHP Extensions ( http://amazon.com/dp/067232704X ), I can say with a fair bit of authority that writing HHVM extensions is SO SO SO much easier.

Are you planning a Windows build ?

do you think Hack could replace all PHP stacks (PHP cloud stacks , not talking about shared host wordpress toys) in a 5 years time frame ?

What do you think about the current state of PHP?

what would you tell PHP core devs about that?

Does an async frameworks like React runs with Hack?

What can you answer to people that are concerned that it is Facebook who is developping this?

Yes. auroraeosrose is working on it (the same person who did the port for php) https://github.com/auroraeosrose/hhvm/tree/win32_start

[1] - http://hacklang.org/manual/en/hack.annotations.summary.php