I was trying to explain to some "civilians" just how much computer power Facebook dumped into the training of their AI models. First off, it's worth mentioning that the GPUs they're using are on the same kind of "banned for export" list as parts used in nuclear weapons. Facebook bought enough of them that the equivalent in weaponry would be a decent sized fleet of ICBMs with multiple warheads (MIRV) each.
They didn't spend this cash for laughs, they're buying it for a purpose, and that purpose existed before ChatGPT was a big hit. Both Zuckerberg and Yann LeCun have stated in interviews that they're invested this money into AI for Facebook feed recommendations and for competing directly with TikTok in their Reels app. In other words, Meta aimed the digital equivalent of a weapon of mass-destruction at your kids' brains, with the express aim of getting them addicted to their products just like TikTok, but even more.
If you want a visualisation of the kind of raw power that your teenager is up against, I came up with this:
- A billion can be visualised as as 1m x 1m x 1m block subdivided into cubic-millimetre tiny little cubes. Think very coarse sand or very small pebbles. Just picture yourself trying to count out the grains in a block that size!
- A billion-billion is a cubic kilometre of the same sized grains.
- Facebook used two-trillion-trillion computations to train one of the smaller Llama 3 models! That's 2 million times the amount above, about the same as 1000 km x 1000 km covered 2 km deep. That's an area the size Egypt. Alternatively, Greenland covered 1km deep, or Texas 3 km (2 miles) deep.
These are all facts coming from publicly available information, such as the Llama 3 paper and recent interviews. I'd dig up the specific timestamps, but I'm working at the moment and can't spend the next hour scrolling through long interviews.
your handle is also appropriate given the epic amounts of electricity needed to power all of that gear. so many articles on the energy consumption of cryptocurrency, yet so few for model training.
Meta / Facebook has 650,000 GPUs. Including overheads for the networking and the servers, these pull about 500 watts each. That adds up to “just” 325 megawatts, which is nowhere near crypto mining levels. (Tens of gigawatts)
Also, most of their compute goes towards inference, not training.
They didn't spend this cash for laughs, they're buying it for a purpose, and that purpose existed before ChatGPT was a big hit. Both Zuckerberg and Yann LeCun have stated in interviews that they're invested this money into AI for Facebook feed recommendations and for competing directly with TikTok in their Reels app. In other words, Meta aimed the digital equivalent of a weapon of mass-destruction at your kids' brains, with the express aim of getting them addicted to their products just like TikTok, but even more.
If you want a visualisation of the kind of raw power that your teenager is up against, I came up with this:
- A billion can be visualised as as 1m x 1m x 1m block subdivided into cubic-millimetre tiny little cubes. Think very coarse sand or very small pebbles. Just picture yourself trying to count out the grains in a block that size!
- A billion-billion is a cubic kilometre of the same sized grains.
- Facebook used two-trillion-trillion computations to train one of the smaller Llama 3 models! That's 2 million times the amount above, about the same as 1000 km x 1000 km covered 2 km deep. That's an area the size Egypt. Alternatively, Greenland covered 1km deep, or Texas 3 km (2 miles) deep.
That's what your kid is up against.
Good luck.