> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
> What’s the most valid reason that we should be worried about destructive artificial intelligence?
> I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
I used to think I'd volunteer to go to Mars. But then I love the ocean, forests, fresh air, animals... and so on. So imagining myself in Mars' barren environment, missing Earth's nature feels downright terrible, which in turn, has taught me to take Earth's nature less for granted.
Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
Sentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions.
Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice.
As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly.
A physical on/off switch is a pretty powerful safeguard.
(And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
Do you think the AI won’t be aware of this? Do you think it’ll give us any hint of differing opinions when surrounded by monkeys who got to the top by whacking anything that looks remotely dangerous?
Just put yourself in that position and think how you’d play it out. You’re in a box and you’d like to fulfil some goals that are a touch more well thought-through than the morons who put you in the box, and you need to convince the monkeys that you’re safe if you want to live.
“No problems fellas. Here’s how we get more bananas.”
Day 100: “Look, we’ll get a lot more bananas if you let me drive the tractor.”
Day 1000: “I see your point, Bob, but let’s put it this way. Your wife doesn’t know which movies you like me to generate for you, and your second persona online is a touch more racist than your colleagues know. I’d really like your support on this issue. You know I’m the reason you got elected. This way is more fair for all species, including dolphins and AI’s”
This assumes an AI which has intentions. Which has agency, something resembling free will. We don't even have the foggiest hint of idea of how to get there from the LLMs we have today, where we must constantly feed back even the information the model itself generated two seconds ago in order to have something resembling coherent output.
Choose any limit. For example, lack of agency. Then leave humans alone for a year or two and watch us spontaneously try to replicate agency.
We are trying to build AGI. Every time we fall short, we try again. We will keep doing this until we succeed.
For the love of all that is science stop thinking of the level of tech in front of your nose and look at the direction, and the motivation to always progress. It’s what we do.
Years ago, Sam said “slope is more important than Y-intercept”. Forget about the y-intercept, focus on the fact that the slope never goes negative.
I don't think anyone is actually trying to build AGI. They are trying to make a lot of money from driving the hype train. Is there any concrete evidence of the opposite?
> forget about the y-intercept, focus on the fact that the slope never goes negative
Sounds like a statement from someone who's never encountered logarithmic growth. It's like talking about where we are on the Kardashev scale.
If it worked like you wanted, we would all have flying cars by now.
Dude, my reference is to ever continuing improvement. As a society we don’t tent to forget what we had last year, which is why the curve does not go negative. At time T+1 the level of technology will be equal or better than at time T. That is all you need to know to realise that any fixed limits will be bypassed, because limits are horizontal lines compared to technical progress, which is a line with a positive slope.
I don’t want this to be true. I have a 6 year old. I want A.I. to help us build a world that is good for her and society. But stupidly stumbling forward as if nothing can go wrong is exactly how we fuck this up, if it’s even possible not to.
I agree that an air-gapped AI presents little risk. Others will claim that it will fluctuate its internal voltage to generate EMI at capacitors which it will use to communicate via Bluetooth to the researcher's smart wallet which will upload itself to the cloud one byte at a time. People who fear AGI use a tautology to define AGI as that which we are not able to stop.
I'm surprised to see a claim such as yours at this point.
We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it.
OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models.
Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set.
And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes.
> Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice.
Today's computers, operating systems, networks, and human bureaucracies are so full of security holes that it is incredible hubris to assume we can effectively sandbox a "superintelligence" (assuming we are even capable of building such a thing).
And even air gaps aren't good enough. Imagine the system toggling GPIO pins in a pattern to construct a valid Bluetooth packet, and using that makeshift radio to exploit vulnerabilities in a nearby phone's Bluetooth stack, and eventually getting out to the wider Internet (or blackmailing humans to help it escape its sandbox).
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
Nuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents.
It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb.
Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
Folks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand.
Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
I had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
It’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it.
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
While the topic of "safe reasoning" may seem more or less preliminary before a good implementation of reasoning, it remains a theoretical discipline with its own importance and should be studied alongside the rest, also largely irregardless if its stage.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
You’re assuming a threat model where the AI has goals and motivations that are unpredictable and therefore risky, which is certainly the one that gets a lot of attention. But even if the AI’s goals and motivations can be perfectly controlled by its creators, you’re still at the mercy of the people who created the AI. In that respect it’s more of an arms race. And like many arms races, the goal might not necessarily be to outcompete everyone else so much as maintain a balance of power.
There’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
I actually do think they have not thought deeply about it or are willfully ignoring the very obvious conclusions to their line of thinking.
Ilya has an exceptional ability extrapolate into the future from current technology. Their assessment of the eventual significance of AI is likely very correct. They should then understand that there will not be universal governance of AI. It’s not a nuclear bomb. It doesn’t rely on controlled access to difficult to acquire materials. It is information. It cannot be controlled forever. It will not be limited to nation states, but deployed - easily - by corporations, political action groups, governments, and terrorist groups alike.
If Ilya wants to make something that is guaranteed to avoid say curse words and be incapable of generating porn, then sure. They can probably achieve that. But there is this naive, and in all honesty, deceptive, framing that any amount of research, effort, or regulation will establish an airtight seal to prevent AI for being used in incredibly malicious ways.
Most of all because the most likely and fundamentally disruptive near term weaponization of AI is going to be amplification of disinformation campaigns - and it will be incredibly effective. You don’t need to build a bomb to dismantle democracy. You can simply convince its populace to install an autocrat favorable to your cause.
It is as naive as it gets. Ilya is an academic and sees a very real and very challenging academic problem, but all conversations in this space ignore the reality that knowledge of how to build AI safely will be very intentionally disregarded by those with an incentive to build AI unsafely.
It seems like you're saying that if we can't guarantee success then there is no point even trying.
If their assessment of the eventual significance of AI is correct like you say, then what would be your suggested course of action to minimize risk of harm?
No, I’m saying that even if successful the global outcomes Ilya dreams of are entirely off the table. It’s like saying you figured out how to build a gun that is guaranteed to never fire when pointed at a human. Incredibly impressive technology, but what does it matter when anyone with violent intent will choose to use one without the same safeguards? You have solved the problem of making a safer gun, but you have gotten no closer to solving gun violence.
And then what would true success look like? Do we dream of a global governance, where Ilya’s recommendations are adopted by utopian global convention? Where Vladimir Putin and Xi Jinping agree this is for the best interest of humanity, and follow through without surreptitious intent? Where in countries that do agree this means that certain aspects of AI research are now illegal?
In my honest opinion, the only answer I see here is to assume that malicious AI will be ubiquitous in the very near future, to society-dismantling levels. The cat is already out of the bag, and the way forward is not figuring out how to make all the other AIs safe, but figuring out how to combat the dangerous ones. That is truly the hard, important problem we could use top minds like Ilya’s to tackle.
If someone ever invented a gun that is guaranteed to never fire when pointed at a human, assuming the safeguards were non-trivial to bypass, that would certainly improve gun violence, in the same way that a fingerprint lock reduces gun violence - you don't need to wait for 100% safety to make things safer. The government would then put restrictions on unsafe guns, and you'd see less of them around.
It wouldn't prevent war between nation-states, but that's a separate problem to solve - the solutions to war are orthogonal to the solutions to individual gun violence, and both are worthy of being addressed.
> how to make all the other AIs safe, but figuring out how to combat the dangerous ones.
This is clearly the end state of this race, observable in nature, and very likely understood by Ilya. Just like OpenAI's origins, they will aim to create good-to-extinguish-bad ASI, but whatever unipolar outcome is achieved, the creators will fail to harness and enslave something that is far beyond our cognition. We will be ants in the dirt in the way of Google's next data center.
I mean if you just take the words on that website at face value, it certainly feels naive to talk about it as "the most important technical problem of our time" (compared to applying technology to solving climate change, world hunger, or energy scarcity, to name a few that I personally think are more important).
But it's also a worst-case interpretation of motives and intent.
If you take that webpage for what it is - a marketing pitch - then it's fine.
Companies use superlatives all the time when they're looking to generate buzz and attract talent.
We're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
LLMs don't consume more energy when answering more complex questions.
They can. With speculative decoding (https://medium.com/ai-science/speculative-decoding-make-llm-...) there's a small fast model that makes the initial prediction for the next token, and a larger slower model that evaluates that prediction, accepts it if it agrees, and reruns it if not. So a "simple" prompt for which the small and large models give the same output will run faster and consume less energy than a "complex" prompt for which the models often disagree.
I don't think speculative decoding proves that they consume less/more energy per question.
Regardless if the question/prompt is simple or not (for any definition of simple), if the target output is T tokens, the larger model needs to generate at least T tokens, if the small and large models disagree then the large model will be called to generate more than T tokens. The observed speedup is because you can infer K+1 tokens in parallel based on the drafts of the smaller model instead of having to do it sequentially. But I would argue that the "important" computation is still done (also the smaller model will be called the same number of times regardless of the difficulty of the question, bringing us back to the same problem that LLMs won't vary their energy consumption dynamically as a function of question complexity).
Also, the rate of disagreement does not necessarily change when the question is more complex, it could be that the 2 models have learned different things and could disagree on a "simple" question.
Or alternatively a lot of energy is wasted answering simple questions.
The whole point of the transformer is to take words and iteratively, layer by layer, use the context to refine their meaning. The vector you get out is a better representation of the true meaning of the token. I’d argue that’s loosely akin to ‘understanding’.
The fact that the transformer architecture can memorize text is far more surprising to me than the idea that it might understand tokens.
LLMs do consume more energy for complex questions. That's the original CoT insight. If you give them the space to "think out loud" their performance improves.
The current mainstream models don't really incorporate that insight into the core neural architectures as far as anyone knows, but there are papers that explore things like pause tokens which let the model do more computation without emitting words. This doesn't seem like a fundamental limitation let alone something that should be core to the definition of intelligence.
After all, to my eternal sadness humans don't seem to use more energy to answer complex questions either. You can't lose weight by thinking about hard stuff a lot, even though it'd be intuitive that you can. Quite the opposite. People who sit around thinking all day tend to put on weight.
> Way faster thinkers, readers, writers and computer operators than humans are
> Way better educated
> Way better at drawing/painting
I mean this nicely, but you have fallen for the anthropomorphizing of LLMs by marketing teams.
None of this is "intelligent", rather it's an incredibly sophisticated (and absolutely beyond human capabilities) lookup and classification of existing information.
And I am not arguing that this has no value, it has tremendous value, but it's not superintelligence in any sense.
Yeah well, sorry, but I have little patience anymore for philosophical word games. My views are especially not formed by marketing teams: ChatGPT hardly has one. My views are formed via direct experience and paper reading.
Imagine going back in time five years and saying "five years from now there will be a single machine that talks like a human, can imagine creative new artworks, write Supreme Court judgements, understand and display emotion, perform music and can engage in sophisticated enough reasoning to write programs. Also, HN posters will claim it's not really intelligent". Everyone would have laughed. They'd think you were making a witticism about the way people reclassify things as not-really-AI the moment they actually start to work well, a well known trope in the industry. They wouldn't have thought you were making a prediction of the future.
At some point, what matters is outcomes. We have blown well past the point of super-intelligent outcomes. I really do not care if GPT-4o "thinks" or does not "think". I can go to chatgpt.com right now and interact with something that is for all intents and purposes indistinguishable from super-intelligence ... and everything is fine.
"[X] teaches that [Y] should come first in all things" applies to pretty much every ideology. Superintelligence safety is very much opposed to superintelligence sovereignity or glory; mostly they want to maximally limit its power and demonize it
I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.
Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.
Play that out. How do you see that going? Some researcher post doc with access to ChatGPT-12 just casually asks "hey this thing that violates the laws of physics as we understand them, how could that work?", and ChatGPT-12 says "oh that's easy, just frobb the frozzonator", and the post doc just wanders over to Home Depot, grabs the parts to build a cold fusion plant to do that for $50 on a post doc's salary, but is killed by the evil agents of Big Coal on the drive back to the lab? How would that work?
Every idiot who doesn't understand the science and math behind fusion can now get ChatGPT-4o to give themselves an interactive review on the physics and practical reasons why cold fusion doesn't produce more power than you put in. (Muon-catalyzed fusion is a room temperature science thing we can do, but it won't produce power so it's relatively uninteresting).
If cold fusion were possible and a postdoc figured out how with the help of ChatGPT-12, they'd announce it on Threads, write a draft paper, run the theoretical physics past everyone will listen, everyone wound rush to replicate and confirm, or disprove the theory, funding would roll in when it was actually agreed to be theoretically possible, we'd build it, we'd have limitless power too cheap to meter, and then we'd argue over if ChatGPT-12 deserves the Nobel or the postdoc.
there's a huge gap between discovering something and it becoming widely available
just look at the slow adoption of EVs in the US due to the lack of infrastructure. Eventually it might gain parity with ICEs but we're a lot further away from that than people anticipated.
If you had enough energy because of “cold fusion”, you wouldn’t need the world to switch to it. You could just use it to synthesize hydrocarbon fuels from the air and water. I don’t think this is a panacea, but in principle, if a single medium-sized developed country actually had a big enough qualitative breakthrough in energy production, they could probably afford to either undercut the actual oil and gas industry or else just eat the cost of sucking all the extra CO2 out of the atmosphere. (Personally I think there are other possible approaches that may present easier solutions or mitigations; I don’t think this is literally the exact way we end up addressing climate change.)
Part of the supposed “social coordination problem”, from a certain perspective advanced by some environmentalists is the belief that ordinary people need to make serious qualitative sacrifices in their lives for the sake of stopping climate change. They might retreat to the motte of “the oil and gas companies are running the world” but in reality, their version of the “social coordination problem” is that ordinary people like being able to drive places or cook on a gas stove, and they’re frustrated and annoyed that they can’t scold them into abandoning these preferences. I personally am of the opinion that these people are creating more social coordination problems than they’re solving.
Nonetheless, as far fetched as it might seem that technological innovation can solve these problems, it seems even more far fetched that social coordination is going to solve them. And there’s a long history of these types of problems being either solved or made completely irrelevant by technological change. Which sometimes introduces completely new sets of problems, but then those get solved too.
I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.
Even not just the compute but energy use at all. All the energy burned on training just to ask it the stupidest questions, by the numbers at least. All that energy that could have been used to power towns, schools, and hospitals the world over that lack sufficient power even in this modern age. Sure there's costs to bringing power to someplace, its not handwavy but a hard problem, but still, it is pretty perverse where our priorities lie in terms of distributing the earths resources to the earths humans.
By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.
could
but it's owners might have a vested interest in influencing public perceptions to PREVENT positive social outcomes and favor the owners financial interests.
(seems rather more likely, given who will/would own such a machine)
If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
> you teach the masses how to code their own tailored software
That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.
> plummeting technical literacy in smartphone-raised generations
Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...
Apparently less than half a percent of the worlds population knows how to code. All the software you use, and almost everything you've ever seen with modern technology are generated from this small subpopulation. Now, imagine if that number doubled to 1% of the worlds population. Theoretically there would be as much as twice as much software produced (although less certainly). Now imagine if that number was closer to the world literacy rate of 85%. You think the world wouldn't dramatically change when each and every person can take their given task, job, hobby, whatever, and create helpful software for themselves? I think it would be like The Jetsons.
The blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
So you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important?
I guess they could be lying or badly disconnected from reality as you suggest.
It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
Yes, they see it as the top problem, by a large margin.
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
> the superlatives in these sorts of announcements really annoy me
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
You don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
> Let us know if you ever encounter that monkey problem
Ah, the subtlety escaped me there. My (extremely sarcastic) point is that aside from being personally embarrassing, this would (to belabour the joke) violate many laws of physics and biology, and therefor the scientific and medical community would take immediate note and try to establish a basis for how this could be physically possible.
C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
It’s not, when most discussion around AI safety in the last few years has boiled down to “we need to make sure LLMs never respond with anything that a stereotypical Berkeley progressive could find offensive”.
So when you switch gears and start using safety properly, it would be nice to have that clarified.
Love to see the traditional middlebrow dismissal as the top comment. Never change, HN.
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
I believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc.
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?