When I first saw reporting of these UAE ventures some days ago, it was about suspicion that the owners would eventually exploit the forest or the minerals underneath. It is a frequent phenomenon in sub-Saharan Africa and Madagascar that land is first restricted and named a protected reserve, but after ribbon-cutting ceremonies are over and attention moves elsewhere, logging is done with wood being sold to e.g. China’s furniture industry, or the local population’s slash-and-burn practices encroach regardless.
Curious thought: at some point a competitor’s AI might become so advanced, you can just ask it to tell you how to create your own, analogous system. Easier than trying to catch up on your own. Corporations will have to include their own trade secrets among the things that AIs aren’t presently allowed to talk about like medical issues or sex.
As someone who doesn’t know much about how these models work or are created I’d love to see some kind of breakdown that shows what % of the power of GPT4 is due to how it’s modelled (layers or whatever) vs training data and the computing resources associated with it.
This isn't precisely knowable now, but it might be something academics figure out years from now. Of course, first principles of 'garbage in garbage out' would put data integrity very high, the LLM code itself is supposedly not even 100k lines of code, and the HW is crazy advanced.
so the ordering is probably data, HW, LLM model
This also fits the general ordering of
data = all human knowledge
HW = integrated complexity of most technologists
LLM = small team
Still requires the small team to figure out what to do with the first two, but it only happened now because the HW is good enough.
LLMs would have been invented by Turing and Shannon et al. almost certainly nearly 100 years ago if they had access to the first two.
That’s true now, but maybe GPT6 will be able to tell you how to build GPT7 on an old laptop, and you’ll be able to summon GPT8 with a toothpick and three cc’s of mouse blood.
What is inherent about AIs that requires spending a billion dollars?
Humans learn a lot of things from very little input. Seems to me there's no reason, in principle, that AIs could not do the same. We just haven't figured out how to build them yet.
What we have right now, with LLMs, is a very crude brute-force method. That suggests to me that we really don't understand how cognition works, and much of this brute computation is actually unnecessary.
Maybe not $1 billion, but you'd want quite a few million.
According to [1] a 70B model needs $1.7 million of GPU time.
And when you spend that - you don't know if your model will be a damp squib like Bard's original release. Or if you've scraped the wrong stuff from the internet, and you'll get shitty results because you didn't train on a million pirated ebooks. Or if your competitors have a multimodal model, and you really ought to be training on images too.
So you'd want to be ready to spend $1.7 million more than once.
You'll also probably want $$$$ to pay a bunch of humans to choose between responses for human feedback to fine-tune the results. And you can't use the cheapest workers for that, if you need great english language skills and want them to evaluate long responses.
And if you become successful, maybe you'll also want $$$$ for lawyers after you trained on all those pirated ebooks.
And of course you'll need employees - the kind of employees who are very much in demand right now.
You might not need billions, but $10M would be a shoestring budget.
And when you spend that - you don't know if your model will be a damp squib like Bard's original release. Or if you've scraped the wrong stuff from the internet, and you'll get shitty results because you didn't train on a million pirated ebooks.
This just screams to me that we don’t have a clue what we’re doing. We know how to build various model architectures and train them, but if we can’t even roughly predict how they’ll perform then that really says a lot about our lack of understanding.
Most of the people replying to my original comment seem to have dropped the “in principle” qualifier when interpreting my remarks. That’s quite frustrating because it changes the whole meaning of my comment. I think the answer is that there isn’t anything in principle stopping us from cheaply training powerful AIs. We just don’t know how to do it at this point.
>Humans learn a lot of things from very little input
And also takes 8 hours of sleep per day, and are mostly worthless for the first 18 years. Oh, also they may tell you to fuck off while they go on a 3000 mile nature walk for 2 years because they like the idea of free love better.
Knowing how birds fly ready doesn't make a useful aircraft that can carry 50 tons of supplies, or one that can go over the speed of sound.
This is the power of machines and bacteria. Throwing massive numbers at the problem. Being able to solve problems of cognition by throwing 1GW of power at it will absolutely solve the problem of how our brain does it with 20 watts in a faster period of time.
> Transistors used to cost a billion times more than they do now
However you would still need billions of dollars if you want state of the art chips today, say 3nm.
Similarly, LLM may at some point not require a billion dollars, you may be able to get one, on par or surpass GPT4, easily for cheap. The state of the art AI will still require substantial investment.
Because that billion dollars gets you the R&D to know how to do it?
The original point was that an “AI” might become so advanced that it would be able to describe how to create a brain on a chip. This is flawed for two main reasons.
1. The models we have today aren’t able to do this. We are able to model existing patterns fairly well but making new discoveries is still out of reach.
2. Any company capable of creating a model which had singularity-like properties would discover them first, simply by virtue of the fact that they have first access. Then they would use their superior resources to write the algorithm and train the next-gen model before you even procured your first H100.
I agree about training time, but bear in mind LLMs like GPT4 and Mistral also have noisy recall of vastly more written knowledge than any human can read in their lifetime, and this is one of the features people like about LLMs.
You can't replace those types of LLM with a human, the same way you can't replace Google Search (or GitHub Search) with a human.
Acquiring and preparing that data may end up being the most expensive part.
Your concern about total environmental collapse makes me wonder why there has been so little talk recently about biomedical advances as a hedge against the effects of global warming. Twenty-odd years ago, Ray Kurzweil’s vision of the human race moving from organic to machine bodies was all the rage in nerd circles. (I don’t share his optimism myself, I just think it curious that it totally disappeared from the discourse.) Machine bodies don’t rely on a world kept within such a narrow temperature range, they don’t rely on all the species that are going extinct, they don’t even need a biosphere at all.
So much was revealed in the European Parliament's ECHELON report back in 2000 that I found it hard to understand why Snowden made the big splash that he did. It all seemed pretty old hat to me.
The chattering classes love counter-cultural packaging. That is why they embraced Greta Thunberg much more than they embraced Al Gore despite the messaging being the same.
The ECHELON report revelations were packaged into a formal (boring) European Parliament report. Meanwhile Edward Snowden had the counter-cultural packaging of a cool dissident hacker.
Japanese considering themselves victims in the war is something that has been present in Japanese cinema since the moment American-occupation censorship was lifted in the early 1950s. It’s present here and there in films by Kurosawa and Ozu that are now in the canon of world cinema. It seems a bit late to be offended by it.
I wonder if it is easier for fast food to stay consistently vegan than a sit-down restaurant. For example, holes in the wall or food trucks that sell takeaway Middle Eastern staples like falafel or koshary. Sit-down places have too much space that, with soaring rents in so many countries, costs too much money.
The article didn’t say “race” at all. Its wording tries to simplify, for a mass audience, the research that found at this Balkan site DNA that is mainly associated with sites in the east of Europe. Across archaeology and linguistics, there has been an archaeogenetics revolution in the last 15 years or so that enables tracing historical migrations like never before.
I don’t know why you think “attracted to the wealth Rome invested in its frontier zone” is a wording that implies jobseekers. It can obviously mean opportunities for pillaging, which peoples of the Eurasian steppe did for centuries. It can mean making use of convenient infrastructure left behind when the Roman military retreated from certain holdings.
Unfortunately none of that is passed on to said audience, as can been in comments that claim ancient europe followed burial or genetic patterns aligned with a cold war era political classification of the region.
It would be comical if not tragic that we actually fund people producing such “research”. I am less and less surprised that some think we can replace them with chat bots given how low quality the output is.
Your posts here are so incoherent, it’s not clear what you are complaining about. Is it the term "Eastern European"? That isn’t a “Cold War-era classification” at all. The use of “Eastern European” in the archaeology and linguistics of the region goes back to the term Osteuropa in the foundational 19th-century literature.
If you lack any familiarity with this field enough to know that, then it would be wise to refrain from making pronouncements on the worthiness of this research. Also, the HN submission is a pop-sci article created by that university’s PR, it is not the actual research. The actual research can be found in the mentioned journal.
Have you ever traveled in Western Sahara, on both sides of the berm? Especially over a span of time that would allow you to witness the changes that have occurred in the region’s demography? The indigenous Saharawi population is so small now it could hardly form a viable state, especially one able to resist migrant flows from further south. As the other poster said, European states’ foreign policy now is strongly driven by migration concerns. Yes, the marginalization of the Saharawi from their own region is a result of Morocco’s occupation, and that can be lamented, but the damage is already done.
You’ll notice that I already spoke of the Moroccan occupation in my post. Repeating the point in more strident language only makes it look like you are trying to engage in political battle on HN.
No, the genetic factor itself is something new here, and interesting even to this reader with an academic background in a related field. Florin Curta (a very respected scholar overall) has held out against the mainstream view of a Slavic migration into the Balkans on the basis of a supposed lack of archaeological evidence. So, I suspect that many scholars, as soon as they learn of this news about the ancient DNA, will immediately think “Hmm, I wonder what Curta has to say to that.”
The poster has 16905 karma and has been around since 2013, though. Obviously a bona fide member of the HN community and not a spammer. (Well, unless his account was hacked.)
