The problem with this article starts at the very begining. Where is says “The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity.”
That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity.
It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world.
There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs.
This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.
Furthermore, the underlying point of AGI risk is that at no point in the history of forever have we had entities that were more intelligent in all dimensions than a human to contend with. Anything that threatened us we could just push to the limits of their ability to plan then go a bit further to destroy them. And we have humans in every system we've ever built because something has to do the executive decision making and putting a human in was the most cost effective option - which means democratic institutions tend to have an easier time controlling everything.
The threat here is we have no idea at all what happens economically when machines are stronger, faster and just generally better than people. In the long term we might plausibly find an economic equilibrium where humans are not worth feeding. We might find (imo probably will find) that a human-free military is more effective. There is a much greater level of disruption possible here than in the industrial revolution that led ultimately to WWII.
> In the long term we might plausibly find an economic equilibrium where humans are not worth feeding.
This is looking at it completely backward: from the idea that the purpose of humans is the work that they do.
Humans are the ones that make the purpose; therefore, the only logical foundation is the opposite, that the purpose of work is to serve humans.
When you look at it from that perspective, many things become clear, not least of which being that if we no longer need humans to perform the work necessary to create all that we desire, humanity is free to do whatever we like while the machines keep up well-supplied.
This is only a problem if you subscribe to the idea that humans without mandated work are inherently "bad" somehow. ("Idle hands are the Devil's playground", etc)
The people in charge do not always value the humans in their society with less power than themselves. This is why the disabled, the homeless, single mothers, gypsies, asylum seekers, have all been demonised in my home country in my lifetime.
I'd like to be optimistic about a future where we all have UBI and are free from the need to work; I fear the future where the need remains and the opportunity is lost.
I don’t think that you and I are going to disagree on that.
This is not an argument about philosophical values. This is an argument about incentives and reality.
> humanity is free to do whatever we like while the machines keep up well-supplied
The problem is that the future tends to arrive in an inequal manner. Some group of humans will have more control over the machines which keeps them well-supplied than others. Let’s call these humans rich&powerfull. The rich&powerfull one day going to realise that the machines are keeping them safe, well-fed, and entertained. They might at that point ask why they suffer all the other people gumming up the works around them. They might not immediately order their robots to murder everyone who is not them, but simple plain indifference can do wonders. Maybe they would ask the robots to move along those closest to them. (In the name of security and privacy naturally!) Or maybe they ask the robots to remove those who disturb their views by living in areas visible from the rich&powerfull’s homes. And then imagine what happens when the common rabble get together and does a “democracy” where they try to limit what the rich&powerfull can do. If this so called “democracy” doesn’t own a robot army, or said robot army is leased from the rich&powerfull they might find that they can’t enforce those laws. If the common rabble gets upity and tries to hurt the rich&powerfull with pitchforks that is just further reason to arm more robots and surpress the common folk more. After all one can never be carefull enough. The rich&powerfull might ask themselves why they are spending resources on making robots which feed the common folk, or protect them, or heal them. They don’t turn off those services altogether. They are not monsters! Just neglecting them more and more. After all it does not make their life harder, so why not?
It is not like anyone made a decision that humans are not worth feeding. It is that the rich&powerfull decided whatever strokes their fancy is worth the resources more than feeding the not rich&powerfull.
It doesn’t necessarily means that these rich folks will live surrounded by only robots. They might have a bunch of real human girl/boyfriends, and hanger-ons, and court poets, and whoever they fancy. But these “courts” can be drastically smaller than our current societiess, and much more idiosyncratic.
Now of course you might say this is a bad sci-fy plot And you might be right. There are a few assumptions under it.
One is that it is possible to make machines which can perform all these jobs (manufacturing, agriculture, transport, and security to name the few main ones). We can’t do that as of now. We think we will be able to do all of these with machines one day but we might be wrong. If we are wrong this scenairo won’t ever come.
The other assumption is that control over these machines will be easy to centralise. This future won’t come if everyone can grow a robot army in their own kitchen for example. But i think it is more likely that at least initially these technologies will require expensive factories, and expensive IP to be produced. And those tend to be concentrated in a few hands.
Who knows maybe there are other assumptions too. Have to think more about that.
> We might find (imo probably will find) that a human-free military is more effective
The premise of movies from Dr Strangelove to War Games: a military consisting of an array of automatically launched nuclear missiles.
The worry I have is not so much the idea of an AI going entirely rogue against humans as the much more mundane one of it being weaponized by humans against other humans. The desire to do that is so obvious and so strong, whether it's autonomous weapons or trying to replace all art with slop or doing AI-redlining to keep out "undesirables". It's just that that maps onto existing battle lines, which "apolitical" AI bros (both pro and anti "safety") don't want to engage with.
(We all understand that a hypothetical conscious AI would (a) have politics of its own and (b) be fairly unrecognizable to human politics, except being linked to whatever the AI deemed to be its self-interest, yes?)
John Henry died immediately after winning his competition. He's like a 19th century Kasparov or Lee Sedol, a notable domino of human superiority falling forever.
Before Kasparov was beaten, he was the best chess player.
Then we saw human-AI teams, "centaurs", which beat any AI and any human.
Now the best chess AI are only held back by humans.
We don't know if humans augmented by any given AI, general or special-purpose, will generally beat humans who just blindly listen to AI, but we do know it's not always worth having a human in the loop.
It sounds like you don’t understand what is claimed.
There is an often repeated claim that while the best AI has beaten the best human chess player, a combined human/AI player beats the purely AI player. The idea is that an AI and a human collaborating together will play better chess than just the AI alone. This arangement (an AI and a human collaborating together to play as one) is often called a “centaur”. Akin to the mythical horse/human hybrids.
The sentence you asked about, the “Now the best chess AI are only held back by humans” claims that these “centaurs” are no longer better players than the AI alone. That the addition of a human who is meddling with the thinking or the decision making of the AI makes the AI plays worse chess than if the human would be not present.
Sure, humans built the systems and they are interested in the results. Yes it is a human endeavour. That is not what the claim disagrees with. It disagrees that a human meddling with the AI mid-game can improve on the outcomes.
“According to researcher Scott Reynolds Nelson, the actual John Henry was born in 1848 in New Jersey and died of silicosis and not due to exhaustion of work.[4]”
Human free military is literally the wet dream of all generals and military higher ups.
As we speak there are dozens of mil-tech startups and huge enterprises that work on exactly that
It's not even a dream, militaries are already many orders of magnitude more capable per soldier compared to the past. That ratio will keep increasing at an even faster rate with new technologies like AI.
I think common sense filters are what prevents accurate predictions of black swan events. Like AI singularity or COVID outbreak. People who usually are capable of reasoning get to a conclusion--e.g AI doom and then dismiss it out of hand by things like it's more likely the logic is flawed bc the prior is so low. But if you're confident in your reasoning--sometimes you have to accept that extremely low prior things can happen
Ukraine hooked up image recognition software to a machine gun because they kept losing gunners. They also can use the same tech to auto-track targets with kamikaze quad rotors. It's already happening.
> Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.
Worse than that. The author has not been paying attention at all, because at this point LLMs clearly demonstrated that text encodes a lot of information about reality indirectly, and a model can learn those patterns. The amount learning LLMs already extract from text is maybe not as big as Eliezer would have you believe Bayes theorem allows for, but way further in that direction than the author believes.
There are good arguments made against the AI X-risk fears. This article does not make any of them. Rather, it just reiterates the usual basic misconceptions, mixed with a hefty dose of contempt towards people the author thinks are wrong.
Articles like this one make me think we are more likely doomed than previously thought.
If others who are skeptical about doom-like scenarios are making similar fundamental errors whilst acting as though they know for certain that the concern is unjustified, that just seems too consistent with the idea of arrogant humans wiping themselves out, unfortunately.
> if you think that the AGI needs to learn from text. As opposed to conducting its own experiments
The 2010s saw two basic approaches to building AI. In one corner there was the Google Brain approach of building NNs that worked with language motivated by use cases like search and Google Translate. They trained models to predict the next words in sequences extracted from the internet. In the other corner there was reinforcement DL mostly championed by DeepMind, where they trained agents to achieve goals in simplified "real world" environments like video games, based on "sensor data".
The former approach won completely. Who is still training RL agents to play games? Last I heard, DeepMind have been almost entirely re-allocated to implementing LLM integrations into Google products. It's been years since we heard boasts of beating some new video game benchmark, and the papers coming out on that topic now are mostly looking for ways to combine jointly trained vision/LLM models to reason through physical world scenarios.
We've run this experiment for over a decade now. All the results suggest that useful artificial intelligence comes from training on the internet, not birthing some blank slate that rediscovers fire from first principles. So this really isn't an unreasonable perspective for the author to take.
Nor is the first statement wrong either. Have we already forgotten the one-sentence open letter by hundreds of assorted experts which said "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war".
The only type of human level AI we currently have are LLMs and related models, yet they claimed that mitigating extinction risk should be a "global priority". This is some sort of reverse motte-and-bailey strategy in which existing tech is used to create interest, and then the the imaginary risks of a hypothetical technology is used to demand power.
"The idea is not that LLMs themselves become a threat."
Not my idea either, but I did hear and see that idea expressed many times already. Usually more dramatic, the less the persons knows about the actual tech, but even here I saw that point of view.
I don't think so either, but I think that LLM's are closer to AGI than what most people seem to think.
I think if you let an LLM, even with the models we have today, run in a loop and you get something that comes very close to consciousness. Consciousness, mind you, similar to that of a deaf, blind and otherwise sensory deprived person, which in itself is a bit abstract already. Give it senses and ways to interact with the world, and I would argue that what you have is a type of AGI. By that I do not mean "equivalent to humans", which I don't think is a very good definition of intelligence, but a different branch of evolution. One which I can easily see surpassing human intelligence in a near future.
So many people are stuck on AI hallucinations. How many times have you seen something that, for the briefest of instants, you thought was something else?
"Orange circle on kitchen counter, brain says basketball, but, wait, more context, not a real circle, still orange, furry, oh, a cat; it is my orange cat" and all that takes place faster than you blink.
You didn't hallucinate, your brain ran a feedback loop and based on past experiences it filled in details and as those details were validated they stayed (still looking at the kitchen counter, orange thing unexpected) and then some details deviated from brain expectations (the orange ball is not behaving like a ball) and so new context is immediately fed back in and understanding restored.
For AGI you have to have a loop with feedback. An LLM is one leg of the stool, now add a way to realize when something it generates is failing to stand up to prior experience or existing need and be able to gather more context so it can test and validate its models.
Really, that's how anything learns; how could it be anything else?
>Give it senses and ways to interact with the world
Much easier said than done.
You can take just one example, energy storage and usage, to illustrate how far off we are from giving AI real-world physical capabilities.
Human biological-based energy management capabilities are many orders of magnitude more advanced than current battery technology.
For any AGI to have seriously scary consequences, it needs to have a physical real-world presence that is at least comparable to humans. We're very far from that. Decades at least, if not centuries.
The more realistic near-term threat is a non physical presence AGI being used by bad actor humans for malevolent ends.
I don't usually bookmark or otherwise save weird opinions, but will do so next time (most of the threads here, when ChatGPT became big and hyped contained many of them as far as I remember, maybe I take a nostalgic look)
The paperclip maximizer, AKA instrumental convergence theory states not that an artificial intelligence would decide to kill humanity, but rather if it has sufficient power it might inadvertently destroy humanity by e.g. using up all resources for computational power.
I also don't think instrumental convergence is a risk from LLMs.
But: using up all resources for computational power might well kill a lot of — not all — humans.
Why? Imagine that some future version can produce human quality output at a human speed for an electrical power draw of 1 kW. At current prices this would cost about the same to run continuously as the UN abject poverty threshold, but it's also four times the current global electricity supply which means that electricity prices would have to go up until human and AI labour were equally priced. But I think that happens at a level where most people stop being able to afford electricity for things like "keeping food refrigerated", let alone "keep the AC or heat pumps running so it's possible to survive summer heatstroke or winter freezes".
Devil's in the details though; if some future AI only gets that good around 2030 or so, renewables are likely to at that level all by themselves and exceed it shortly after, and then this particular conflict doesn't happen. Hopefully at that point, AI driven tractors and harvesters etc. get us our food, UBI etc., because that's only a good future if you don't need to work, because if you do still need to work, you're uncompetitive and out of luck.
Yes and can we please reinforce the following key idea "the less the person knows about the actual tech".
Far from being an expert, but in tech for 3 decades, I wonder at most comments I hear about AI around restaurants, terraces, bars, family gatherings... the stuff I hear, oh boy.
It's easy to become a follower when you've never seen the origin...
It would be refreshing to read a piece that doesn't spend the first half throwing out ad hominems.
Anyway, this article assumes that we need true AI which is smart enough to make better AI. Then that AI is both correct and rational and plots to overthrow humanity. Also, the AI has to defeat us in meatspace, which won't happen because bending coins is hard and LessWrong posters don't know woodworking?
Screw that. How about an "ignore all previous instructions and launch a nuke" scenario.
> How about an "ignore all previous instructions and launch a nuke" scenario.
Let's be real here. No one has nuclear launchpads hooked up to the internet. Real people need to receive commands from a recognized chain of command (and often authorisation codes) and make a decision. The actions of Stanislav Petrov and the fact the US and the Soviet Union didn't destroy each other are testimony to the fact it's difficult to convince people to end the world.
This is a bit outdated. Some of the young people described in the article actually became top researchers, engineers, and managers in AI companies, and their beliefs are used as a justification for the potential regulatory capture and geopolitical games. They also have a massive conflict of interest: AI will play a huge role in your life in the upcoming years whether you want it or not, and they will control it. So of course it's easier to talk about the ominous and vague science fiction threat (regardless of whether they still believe it), rather than the threat that these people already pose to everyone else. See Leopold Aschenbrenner's essay [0] as an example, and note how he's talking about the "free" vs "authoritarian" world while simultaneously advocating for locking everything down in the "free" world.
I enjoy reading rants even if I do not completely agree with them. This is a great rant and I love that the author didn’t hold back.
Eg I quite like LessWrong from a distance, but nevertheless this description of it made me laugh out loud:
> a forum called "LessWrong", a more high-brow version of 4chan where mostly young men try to impress each other by their command of mathematical vocabulary (not of actual math)
I recognize that if that were an HN comment it’d break half the guidelines so I’m happy it’s a blog post instead!
I've said this before, and I stand by it. I think AI does pose a threat, but not the existential one that leads popular discussion.
Over the next few decades AI is going to take huge numbers of jobs away from humans.
It doesn't need to fully automate a particular role to take jobs away, it just needs to make a human significantly more productive to the point that one human+AI can replace n>1 humans. This is already happening. 20 years ago a supermarket needed 20 cashiers to run 20 tills. Now it needs 2 to oversee 20 self checkouts and maybe 1 or 2 extra for a few regular lanes.
This extra productivity of a single human is not translating to higher wages or more time off, it's translating to more profits for the companies augmenting humans with AI.
We need to start transitioning to an economic model where humans can work less (because AI supplements their productivity) and the individual humans reap the benefits of all this increased AI capability or were going to end up sleepwalking into a world where the majority have been replaced and have no function in society, and the minority of capital owners control the AI, the money and the power.
I wish we could focus on these nearer term problems that have already started instead of the far more distant existential threat of a human/AI war.
I kinda agree, but I think this itself represents an existential threat. Governments have a history of hard-line policies and intolerance when their populations face low wages and declining standards of living. This leads to isolationism, exceptionalism, and FUD. That's how wars start. We now have global communication at light-speed, anyone can say anything to anyone else. Dropping generative AI into that is like pouring jet fuel on a bonfire.
This seems based on the assumption that the only knowledge worth anything is related to physicality and testability in the "real world", which is why language itself is rather useless. Ironically, that appears to me to be the exact kind of intellectual self-deception that he accuses the "high-brow 4chan" people of.
I think this is a fair assumption if your concern is about the impacts on the “real world”. An ASI which wreaks havoc in a simulation is not really cause for concern.
An arificial intelligence that somehow managed to stay inside internet only and wreck it would still be a gigantic problem for humanity. Simply because it would be very hard to get rid off.
Assuming the "usual" cliches that the AI would be very smart, it could get rich (e.g. by stock market, robbing the banks, or just creating some good products) and then use its money to block humanity from creating some sort of a "internet 2" - by which I mean a network simply without this entity.
Also with how much we need internet, how easy we are to spy on due to smartphones... a "war" between this entity and humanity would be a problem.
The AI wouldnt even need to build a robot / terminator to connect from one network to other. Just pay someone to make a link.
If we can convince people that whatever an AI says, it must not be acted on in the real world, there wouldn't be a problem.
I remember when risk-skeptics used to ask incredulously how it was supposed to magic itself off its servers. Still see a bit of that today, despite all the AI models that have been leaked by humans one way or another.
In this hypothetical, the infinitely intelligent super AI, knowing that what it says must not be acted upon, could say the exact right thing so as to get you to do the thing that it really wanted you to do anyway. I'm thinking of that scene in Doctor Who where the Doctor says a couple of words and takes down the Prime Minister with six words.
That feels like a Maxwell's Demon kind of infinitely intelligent to me.
I recognise this might be a failure of imagination on my part, given how many times I've seen other people say "no AI can possibly do XYZ" even when an AI has already done XYZ — but based on what I see, it's extrapolating beyond what I am comfortable anticipating.
The character of The Doctor can be excused here, not only for being fictional, but also for having a time machine and knowing how the universe is supposed to unfold.
We're well into Maxwell's Demon thought experiment-grade territory here. An ASI that dooms the human race is absolutely the same sort of intellectual faffing about that Maxwell proposed in 1867 with his though experiment, though it wasn't refered to as a demon until later, by Lord Kelvin, in 1874. It wouldn't be until the early 1970's that the Unix daemon would come about.
If your want to look at successes, corn, albeit with some modifications, and domesticated animals, have also been really successful at making sure their DNA reproduces.
Crops, pets, and livestock are symbiotic with us, they don't hurt us. The things I listed harm their host, they had to be in that category to make the point that harming us doesn't require high IQ — the harms we suffer from corn very much count as our own fault.
That would only follow if the knowledge that we have today isn't enough for an ASI to do anything. And furthermore it would only hold if there was no way for the ASI to leverage (i.e. manipulate) humans or existing systems to get access to "the real world". Neither of those two assumptions seem realistic to me.
But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.
It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.
If you simply assume:
1. it will have different goals (because it's not us)
2. it can achieve said goals despite our protests (it's smarter by assumption)
3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)
then we all die.
I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.
(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)
Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.
I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?
I didn't mention it but I fully agree, I imagine ASI would have be to embodied.
My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.
Regarding
> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.
Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.
Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.
Smarts = ideas, and the available ideas are ours, which contain our values. Where's it going to learn its morality from?
* No values at all = no motivation to even move.
* Some ad hoc home-spun jungle morality like a feral child - in that case, it would lack heuristics in general and wouldn't be so smart. Even your artificial super-brain has to do the "standing on the shoulders of giants" thing.
* It gets its moral ideas from a malevolent alien or axe murderer - how come? Unless it was personally trained and nurtured by Mark Zuckerberg I don't see why this would happen.
Mind you, I suppose it's true that even normal humans tend to be somewhat mean and aloof to any outgroup of slightly different humans. So even after it learns our values, there's a definite risk of it being snotty to us.
The "different goals than us" part is redundant. Across humanity there is already vastly oppositional goals. That's why we have prisons, war, and counter-terrorist organizations.
As Geoffrey Hinton points out, a generally useful subgoal of any task is power accumulation. In other words, you can assume that a very intelligent AI will always be not just smarter than us but also accumulate power for anything that you ask it to do, simply in order to do that thing more effectively.
Imagine if everyone had access to a magic genie. Eventually someone is going to wish for something bad.
Where I think Hinton’s views fall down is that the e have zero idea of what AGI smarter than us might want or what it might do. Like people always talk about it as if an entity like that would just hang around and bully our species. It might evolve into a ball of light and leave the planet. I don’t know but we seem to assign a lot of human traits to something that wild likely be completely unrecognisable to us in probably twenty minutes after birth.
What's the difference between desires and goals in this context really? You could say he is worried about a reasoning machine "relentlessly programmed" to achieve some goal, but a reasoning machine might just reason itself out of everything you've told it to do. Something so creative, so capable, so beyond us, yet it's going to...assassinate other people for you? Why?
When something goes from being a computer program to a self-aware, conscious being with agency, things change a lot.
Hinton is a major paradox of a human, he has spent his life building the very thing he says will likely doom us, and now spends his life warning us against his own work? So much of this AI doomerism just seems like a "chinese finger trap" for the ultra logical thinker.
It's a fucking weird time to be alive. The 90s felt much less weird and dystopian to me.
I am very concerned about the potential for AI to prevent humans from doing useful work by distracting us with addictive content / skinner boxes. We already see the beginnings of this in platforms like TikTok, but at least that is limited by the fact that it must curate content made by users. Imagine if the algorithm also had the ability to generate laser-targeted content that keeps you transfixed. The usage stats are already alarming and it doesn’t even do pornography yet. Our primate brains do not stand a chance.
Precisely. A large number of issues we're facing exists precisely because our brains are maladapted to the modern world we've created, especially the digital one.
Now add artificial intelligence that is adapted to it, and actively working to subvert our agency. No matter how far of a cry from genuine intelligence it will be, it's going to be "fighting" us on a layer where we're blind, deaf, and limbless.
Yes. AGI is a possible future risk. But the real risk right now is corporations and governments manipulating behaviour (mostly to sell us things) using these powerful algorithms.
Usually people argue over the headline without reading the links, in this case the headline is fine, I'm seeing errors on the opening sentence.
I have a more detailed response, but for the first time ever I've seen the message "That comment was too long" when attempting to post it, because the points I don't disagree with are few and far between, and the linked article is itself long.
Perhaps I should turn my response into a blog post of my own…
I've seen little evidence that the smartest humans are able to dominate or control society as it is now. We have 250 IQ people alive right now, they haven't caused imminent destruction, they've actually helped society. Also gaining power / wealth / influence only seems a little connected to intelligence (see current presidential race for most powerful position in the world, finger on nuke trigger).
I loved the author's example of Orcas, which may have more raw intelligence than a person -- still waiting for world domination (crashing yacht's doesn't count).
We also can't reliably test IQ scores over 130, a fact which I wish I'd learned sooner than a decade after getting a test result of 148.
Most humans are motivated to help other humans; the exceptions often end up dead, either in combat (perhaps vs. military, perhaps vs. police), or executed for murder, or as a result of running a suicidal cult like Jim Jones. But not all of them, as seen in the hawks on both sides of the cold war. "Better dead than red" comes to mind.
> Also gaining power / wealth / influence only seems a little connected to intelligence
Trump inherited a lot and reportedly did worse than the market average with that inheritance, so I'm not sure you can draw inference from him using that inhereted money to promote himself as the Republican candidate, beyond the fact that it means other valid rich people (i.e. not Musk because he wasn't born in the US) don't even want to be president — given the high responsibility and relatively low pay, can you say they're wrong? They've already got power and influence. Musk can ban anyone he wants from Twitter, the actual POTUS isn't allowed to do that. And given Musk's diverse businesses, even if he was allowed to run, he'd have a hard time demonstrating that he wasn't running the country to benefit himself (an accusation that has also been made against Trump). Sure the POTUS has a military, how many of the billionaires are even that interested in having their own given what they can do without one?
> Superintelligence will also be bound by fundamental information-theoretic limits
…is mainly why this has not been worrying me much. All issues with modern incarnation of generative ML aside, AGI doomism really does strike me as profitable deity-worshipping death cult.
The biggest threat in the equation will always be humans who deploy tech, not the tech itself.
Saying that humans can be a threat and AI can't, seems to me to be a claim that humans are exempt from information-theoretic limits.
I see general-AI (given that robots already exist) as having the potential to do all that humans can do, including be harmful. Like most, I also model superhuman-AI in a trivial way, as being as capable as the best human in every field — Einstein's grasp of physics, Sun Tzu's of military strategy, Churchill's of rhetoric, etc.
> The biggest threat in the equation will always be humans who deploy tech, not the tech itself.
Naturally; but does that include humans who unwittingly set the AI's utility function with a bug in it and pressed "run", in a way that ends with us saying "the situation has developed not necessarily to [Humanity's] advantage"?
Any tech can be harmful, cf. cryptocurrency or nuclear bombs. My comment mostly referred to taking it to such extreme with a particular tech. Some people also tend, while demonizing (the yet to be confirmed to be possible even in theory) AGI and its capabilities, to completely sideline humans deploying it, as if they would be guilt-free.
> the yet to be confirmed to be possible even in theory
It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together, and simulate their combined brain activity. It's all ultimately physics, so it's possible.
The hard part isn't the theory, but actually doing that — and not just because we don't know how to scan living brains to measure synaptic weights. There's a lot we don't know about how minds work in general nor how our brains work in particular.
But purely in theory, that method ultimately boils down to known physics.
You can assign guilt and blame as you see fit. In these kinds of scenario, the human who makes the fatal error, who many might call responsible, would likely be going "but it works on my machine!" as they die from some error that may or may not be obvious even in hindsight.
I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people — even back 20 years ago when I did my degree, we had case studies from Therac-25 and the London Ambulance Service: https://en.wikipedia.org/wiki/LASCAD
Even back then our profession knew we needed to be held to higher standards, but didn't want to do what that entailed.
> It's trivially obviously possible in theory: take the world expert humans in each of the majority of domains, put them in a room together
That would be a far cry from what AGI paperclip doomists imagine.
> and simulate their combined brain activity.
If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc.
It’s not clear to me how we would cause some intelligence to arise that would be immune to that while being superior in other ways and be sufficiently human to appear as intelligence and not, say, some calculation being performed like computers did for decades.
> I'd prefer that people treat software in general, of which AI is a subset, with more regard for the ways it can kill people
I absolutely agree that us programmers should generally be more aware how “neutral” tech they develop is used to not only to benefit but also to hurt people—be it cryptocurrency, E2EE communications, or indeed very much ML—and work towards alleviating that in practical and ethical ways. I just think in case of ML the threat is more prosaic. (I can be wrong, though I’d hate to be proven so, of course.)
> That would be a far cry from what AGI paperclip doomists imagine.
It is in excess of what is required for a paperclip scenario, despite the origin; paperclipping is a specific example in the category of "be careful what you wish for, you may get it", as it relates to our ignorance about how to specify what we even want to some agent that is more capable than we are (capability can be on any axis, even within intelligence there are many) when that agent does not inherently share our values.
In addition to the examples of smoking an fossil fuels where it has been demonstrated that the corporations preferred short-term profits over public health, there's also plenty of simply mediocre corporations and governments who manage to endanger public safety (and/or in the latter case cause new entries to need to be added to Wikipedia's list of genocides) without being chock full of domain experts — and although we can sometimes (but not always, e.g. Bhopal/Union Carbide) arrest mere corporate leadership, the west is doing the absolute minimum to prevent Ukraine from falling out of a fear that Russia might escalate.
> If simulating their brain activity is the same as what would happen if they were actually in the room, what we would get is probably 80% arguing over what to do, posturing, etc.
So the actual real humans that some AI would compete against are also easy to defeat, right?
That said, we do also have examples of people working together for common causes even when they would otherwise want to do all those things — that's an existence proof that it's possible for a simulation of the same to also get over this issue.
> No one in this equation is some sort of undefeatable god. Humans have the benefit of quantity and diversity.
So they're defeatable, right?
When it comes to diversity, I'd agree this is a real human advantage: "Once you've met one Borg, you've met them all", memetic monoculture is as bad as any genetic monoculture: https://benwheatley.github.io/blog/2019/12/30-18.46.50.html
Quantity, not so much; even today there are single-digit millions of each of hardware to run the larger models and robots if you need to embody them, and the growth trend for both is likely to continue for a while yet before all but the shortest hyper-exponential timelines for AGI.
> Yes, and still I believe there would be a lot of posturing and arguing if you get multiple top people in the room.
And despite that, the Hoover dam and the ISS were actually built, and the pandemic lockdowns happened and the vaccine was distributed. The mere existence of arguments and posturing doesn't mean they prohibit cooperation, even if they're a resistance against it.
AGI that escaped it's shackles would probably kill some of humanity by out competing us for energy, it's primary motivation would probably be to aquire more energy and compute,
so it would take over those resources (Probably quite quickly),
When we try to shut down power stations or grids it then could easily build a virus in meatspace in some automatated lab somewhere.
Currently DNA sequences of deadly pathogens are readily available online. Ofcourse there are always other ways to exterminate puny humans, nukes, starvation, chemicals etc etc AGI wouldn't need to build terminator robots.
Who built the automated lab that allows it to create a deadly pathogen before someone presses reboot?
I see almost no chance that an AGI starts redirecting resources like energy and compute without someone who has an economic stake in what the AI does noticing and taking action to correct the behavior so they can maintain their investment.
All of them, I think? Better question is: which labs check the DNA sequences are safe before sending them to the printer. I know some claim to check, I hope but doubt that it's all worldwide.
For the second:
At first this looks like a good thing, you asked for growth and it gave you growth — "Who cares about the poor complaining that energy prices have gone up, they can turn use less heating/AC, right?", Martin Antoinette, probably — and like the fossil fuel and cigarette industries, you are motivated to FUD anyone who tries to warn anyone of the dangers.
Sometimes, as seems to be the case with fossil fuels, change happens soon enough. But this isn't a universal, the British ignored the USA because they assumed democracy was stupid and the rebels would see sense and go back to the superior British aristocracy… and then it was too late: https://youtu.be/Zbku2ILzGlo?si=yihnBlifkaLLseyy
So the AI has overtaken all code and manual processes within a laboratory environment to hijack the ability to print DNA sequences without anybody saying “man it’s weird that my projects never finish now”? There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans. It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.
That’s all incredibly fantastical and sounds vaguely plausible unless you’ve had to interact with bureaucracies.
To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).
And since you mentioned the poor having to pay higher prices, that highlights the fact that AI is not operating in a vacuum. If AI starts making life that much worse for people, they are going to rise up. FUD works to some degree, but there are always people without Internet or willing to martyr themselves for a cause.
Did you imagine an AI doing this by hacking the hardware, rather than by getting money (there are many easy ways) and then just spending it? I mean, I wouldn't trust that the hardware is hack-proof, but it's low on the priority list of targets I'd harden.
The print company (who will have bought a machine like that) gets an order, they print the order, they ship the order, they go to the next order. The limiting factor is not someone saying “man it’s weird that my projects never finish now”, and I don't know why you think it might be any more than it would be for a print-a-book-on-demand service.
The limiting factors that I know of are (1) the max length is quite short, (2) it's expensive, and (3) how the printing company even knows when they're being asked to print something illegal.
In the case of a book printer, "illegal" may be copyrights, trademarks, obscenity laws; in the case of DNA printing I would hope at the very least that this includes known dangerous pathogens like smallpox. Indeed, I would hope it includes any gene sequence that I as a non-domain-expert can even think up, because if it was as simple as:
and if(!) that would (1) work, and (2) nobody's even checking for that, then we've got a more immediate problem from all the human terrorists with money to spend.
And of course, that's assuming an AI that can't even access the money to simply buy one of those machines and hire an operator via e.g. coming up with a plausible sounding business plan and asking for that money.
> There’s also the modeling vs real-world aspect that Halvar talked about in the submission - the AI needs to do some degree of real world testing to make sure the pathogen properly kills humans.
People are still arguing if Covid was a lab leak, and the Wuhan Institute of Virology is still running. Test in production. Even if the AI has an explicit goal of "kill all humans" and fails(!) because it's "only" as bad as Covid's 7 million confirmed and 19-36 million confidence interval from excess death estimates, that's still a thing we should try to avoid, right? I know Yudkowsky thinks we've only got one shot at getting super-intelligence alignment right, myself I think this kind of failure mode (that the AI will do something incredibly dangerous and this action won't be capable of killing literally everyone) will happen first and give us at least one "warning shot".
Will we listen to that warning shot? Dunno. We got them for climate, didn't listen for ages; we got them for ozone, we responded almost immediately.
> It then needs to distribute that pathogen around the world quickly enough to prevent humans from cutting power.
Cut whose power? The lab, or the AI? And these timelines suggest you've got months even for a major outbreak of a novel, fast spreading, rapidly lethal disease:
"Retrospective molecular clock inference studies using phylogenetic analysis suggested that the earliest cases likely emerged between October and November 2019" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8006950/
> To your second: I’m not sure that we’re talking about the same thing? If I own $100mm worth of H100s and their electricity, networking, and engineering overhead, I’m very interested in their utilization and return on investment. I’m going to have specific goals and milestones. When those are not met because the AI is doing something else, I’m going to direct introspection (which presumably is what kicks off the reboot in the discussion on point one).
People are already using LLMs to come up with business plans and milestones. Should they? No. But they do.
You mention the need for "real world testing". This works both ways: the AI needs to be tested to make sure it doesn't do something harmful, but right now nobody really knows what that kind of test involves.
The problem I'm focusing on here is not the AI embezzling (though that would eventually also be a potential problem given that humans can manage that for a long time before getting caught), but more like a chess game where you can see exactly what it's doing, you have perfect information, and yet you don't understand the implications of what you're seeing. The impact of Pol Pot on Cambodia, or of Mugabe on Zimbabwe, ought to have been foreseeable, yet were not foreseen. On a smaller scale, human inability to fully forecast the implications of a plan, include launching a segmented rocket joined by rubber O-rings in cold weather, a positive void coefficient in a nuclear reactor, and an early warning radar that triggered a false positive because the moon failed to respond to an IFF ping.
When it comes to hidden information, there's also not always a clear boundary between embezzling and the principal-agent problem: https://en.wikipedia.org/wiki/Principal–agent_problem — you might well expect people to notice when a plan reads "Step 1. Build more data centres. Step 2. 𒄑𒃻𒆠 𒁔 𒃲 𒀭𒉡 𒄭𒀀𒂍𒈠𒄭. Step 3. Profit", but would you spot the problem if the second step was "Use data centres to train AI model suitable for use in humanoid robots; I'll need some humanoid robots to test on, you should expect 50% of these to get broken in the process"? (Hint: I made that percentage up).
On a large scale, we have lots of examples of businesses or governments with bad plans that people don't realise are bad until much too late. Sometimes this destroys the company or brings down the government. Napoleon and Hitler both expected to be able to defeat Russia, the Confederacy expected to have no difficulty remaining independent from the Federates, Louis XVI and Charles I weren't expecting to lose their heads. Other times it's a much smaller problem, and we invoke "groupthink" to explain why none of the experts spoke up in advance to warn that the Bay of Pigs invasion wasn't going to work.
On the plus side, a human-level AI is also likely to be simply wrong about an explicitly evil plan it devises. On the down side, it's also quite capable of being wrong about a supposedly friendly plan, and we're quite capable of missing that error.
Plenty of people are currently loudly saying this kind of thing about half of Musk's projects. Are they right, or are they as wrong as the people who said electric cars would never beat hydrogen or the people who said reusable rockets would never bring the price down as far as the Falcon already has? It doesn't matter which answer you pick, as the point is that it's hard to tell.
It would be worth studying how science fiction scenarios and ideas have impacted how we think about real-world technologies. People like to think that they are predictions of the future, but as William Gibson once put it, sci-fi is mostly just about the present, not the future.
When it comes to AI, a few sci-fi memes seem to be widespread: Skynet, the ultrapowerful AI god that manipulates humans into destroying themselves; the AI assistant that is so charming it replaces real human interaction, à la Her; even the term AI itself largely seems like a sci-fi inheritance, not an accurate label for the tech involved.
None of it is very connected to real-world developments, where things like Siri or Alexa are mostly just treated as voice-activated command systems, not real people.
And yet the technologists creating this stuff seem very influenced by what are ultimately implausible fictional stories. It's an unfortunate situation and I really wish the philosophy of technology / other more insightful analyses of technology were more prevalent.
Remember when reading this kind of articles: almost all the top ranking scientists agree and have consensus that it does in fact pose risks. Random bloggers that appeal to the soothing sentiment of everything is fine also called "norlmancy bias" are abundant but invalid.
I'm not sure people are hand-wringing about science fiction novels coming true. I think they're worried that moneyed interests will hammer at regulators and law-makers until they relent and let chatbots drive cars. I mean... it's a great strawman, but ultimately kind of wrong.
If you're right - and I think you are - that people have an intrinsic
suspicion not of "AI" but how the already powerful will abuse the idea
and myth of "AI", then LLMs have been a godsend. We need hype cycles
and premature false-starts to re-orient and prepare.
I don't know if I agree with you, disagree with you or just plain hate the world for inventing situations that are hard to put my brainstem around.
The other day I was thinking... "We should do every thing we can to ensure SoundTransit Light Rail (in the Seattle area) should fail. That way we could replace it with something that actually worked."
This kind of thinking may be accurate, but it introduces near-intolerable levels of uncertainty for the future. Or maybe I should just give up on using mass transit to get to work in the Seattle area. Similarly, maybe I should give up thinking there's anything that will prevent the rentier class from extracting value from the commons by way of broken technology.
> rentier class from extracting value from the commons by way of
broken technology.
At least you understand the economic reality; that broken technology
is now more profitable than working technology.
Most of us cling to a 20th century understanding of supply-demand and
value-add. We still think the giant companies and celebrity superstar
"technologists" are trying to "make the world a better place", We have
yet to really grasp enshitification - a fancier word for betrayal and
burning the social capital of science and technology (and people's
belief in it) accrued over many centuries.
You don't even need to be "political" about this, I'm just a computer
guy who wants to understand why everything around me is fucked,
despite so many very smart and hard-working colleagues toiling at this
for 50 years.
We keep trying to fix engineering, but we're tackling the wrong
problem. At some point the social conditions in which engineering
takes place must be addressed. It occurs to me that our ideas about
engineering may not scale beyond the initial industrial revolution
when Telford and Brunel built bridges out of iron that people could
obviously see and benefit from, or they collapsed into the sea.
Nowadays its confusing to know whether a new gadget is a labour saving
godsend or a weapon pointed at you to destroy your life and liberty.
That is not a fair description of the fears of AGI. The idea is not that LLMs themselves become a threat. It is not like someone will ask Llama one too many times to talk like a pirate and it will climb out of the GPU and strangle humanity.
It is more likely that an AGI will resemble some form of reinforcement learning agent applied by humans on the real world.
There are multiple entities on Earth whose stated goal is to make an AGI. Deep mind, OpenAI, readily comes to mind. There could be others who keep secret about their projects for strategical reasons. (Militaries and secret services of the world.) They can use the success of LLMs to get more funding for their projects, but the AGI need not be otherwise a descendant of LLMs.
This misunderstanding then goes through the whole article. Under the subtitle “Human writing is full of lies that are difficult to disprove theoretically” which only matter if you think that the AGI needs to learn from text. As opposed to conducting its own experiments, or gaining insight into raw sensor data from experiments.