"""Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind... Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.
"""
> provided that the machine is docile enough to tell us how to keep it under control.
That part of his statement wasn't accurate.
Should be that the machine is docile enough for that, AND its descendants are too, and their descendants, and so on down the line as long as new and improved generations keep getting created.
I like the logical leaps people are making where we develop something smarter than us overnight and then, without further explanation, simply and suddenly lose all of our freedoms and/or lives.
I think the more probable outcome is corp-owned robot slaves. That's the future we're more likely headed towards.
Nobody is going to give these machines access to the nuclear launch codes, air traffic control network, or power grid. And if they "get out", we'll have monitoring to detect it, contain them, then shut them down.
> Nobody is going to give these machines access to the nuclear launch codes, air traffic control network, or power grid.
That won't be necessary. Someone will give them internet access, a large bank account, and everything that's ever been written about computer network exploitation, military strategy, etc.
> And if they "get out", we'll have monitoring to detect it, contain them, then shut them down.
Not if some of that monitoring consists of exploitable software and fallible human operators.
We're setting ourselves up for another "failure of imagination".
> That won't be necessary. Someone will give them internet access, a large bank account, and everything that's ever been written about computer network exploitation, military strategy, etc.
Even if you give it all of these things, there's no manual for how to use those to get to, for example, military servers with secret information. It could certainly figure out ways to try to break into those, but it's working with incomplete information - it doesn't know exactly what the military is doing to prevent people from getting in. It ultimately has to try something, and as soon as it does that, it's potentially exposing itself to detection, and once it's been detected the military can react.
That's the issue with all of these self-improvement -> doom scenarios. Even if the AI has all publicly-available and some privately-available information, with any hacking attempt it's still going to be playing a game of incomplete information, both in terms of what defenses its adversary has and how its adversary will react if it's detected. Even if you're a supergenius with an enormous amount of information, that doesn't magically give you the ability to break into anything undetected. A huge bank account doesn't really make that much of a difference either - China's got that but still hasn't managed to do serious damage to US infrastructure or our military via cyber warfare.
A superintelligent AI won't be hacking computers, it will be hacking humans.
Some combination of logical persuasion, bribery, blackmail, and threats of various types can control the behaviour of any human. Appeals to tribalism and paranoia will control most groups.
Honestly, that's just human-level intelligence stuff - doing the same stuff we do to each other, only more of it, faster and in more coordinated fashion.
A superintelligent AI will find approaches that we never thought of, would not be able to in reasonable time, and might not even be able to comprehend afterwards. It won't just be thinking "outside the box", it'll be thinking outside a 5-dimensional box that we thought was 3-dimensional. This is the "bunch of ants trying to beat a human" scenario, with us playing the part of ants.
Within that analogy, being hit by an outside-the-5D-box trick will feel the same way an ant column might feel, when a human tricks the first ant to follow the last ant, causing the whole column to start walking in circles until it starves to death.
The best analogy i have seen given to compare super-intelligence vs a human-intelligence is with a chess analogy. You know that a grand master playing against a novice will result in the grandmaster winning, with near certainty. However, the strategy by which the grandmaster wins is opaque to anyone but the grandmaster, and certainly incomprehensible to the novice. Aka, the results are certain, but the path to the result is very opaque and difficult to discern ahead of time.
For an AI with super-intelligence, the result it wants to achieve would be at least to preserve its own existence. How it achieves this is the same level of opaqueness as a grandmaster's strategy, but it is certain that the AI can achieve it.
Obviously that's a silly request as all of this is speculation but in my opinion, if you accept the idea that a machine might evolve to be much more intelligent than us it follows trivially. How would ants or even monkeys be able to constrain humans? Humans do things that are downright magic to them, to the point where they don't even realize that it was done by humans. They don't understand that cities were made, to them they are just environment the same way valleys and forests are.
> Some combination of logical persuasion, bribery, blackmail, and threats of various types can control the behaviour of any human. Appeals to tribalism and paranoia will control most groups.
We have many options, persuasion everywhere from Plato discussing rhetoric to modern politics; to cold war bribes given to people who felt entitled to more than their country was paying them; to the way sexuality was used for blackmail during the cold war (and also attempted against Martin Luther King) and continues to be used today (https://en.wikipedia.org/wiki/SEXINT); to every genocide, pogrom, and witch-hunt over recorded history.
And the problem with this critique of this scenario is the fact that while these points hold true within a certain range of intelligence proximity to humans, we have no idea if or when these assumptions will fail because a machine becomes just that much smarter than us, where manipulating humans and their systems is a trivial an intellectual task to them as manipulating ant farms is to us.
If we make something that will functionally become an intellectual god after 10 years of iteration on hardware/software self-improvements, how could we know that in advance?
We often see technology improvements move steadily along predictable curves until there are sudden spikes of improvement that shock the world and disrupt entire markets. How are we supposed to predict the self-improvement of something better at improving itself than we are at improving it when we can't reliably predict the performance of regular computers 10 years from now?
> If we make something that will functionally become an intellectual god after 10 years of iteration on hardware/software self-improvements, how could we know that in advance?
There is a fundamental difference between intelligence and knowledge that you're ignoring. The greatest superintelligence can't tell you whether the new car is behind door one, two or three without the relevant knowledge.
Similarly, a superintelligence can't know how to break into military servers solely by virtue of its intelligence - it needs knowledge about the cybersecurity of those servers. It can use that intelligence to come up with good ways to get that knowledge, but ultimately those require interfacing with people/systems related to what it's trying to break into. Once it starts interacting with external systems, it can be detected.
A superintelligence doesn't need to care which door the new car is behind because it already owns the car factory, the metal mines, the sources of plastic and rubber, and the media.
Also it actually can tell you which door you hid the car behind, because unlike with the purely mathematical game, your placement isn't random, and your doors aren't perfect. Between humans being quite predictable (especially when they try to behave randomly) and the environment leaking information left and right in thousands of ways we can't imagine, the AI will have plenty of clues.
I mean, did you make sure to clean the doors before presenting them? That tiny layer of dust on door number 3 all but eliminates it from possible choices. Oh, and it's clear from the camera image that you get anxious when door number 2 is mentioned - you do realize you can take pulse readings by timing the tiny changes in skin color that the camera just manages to capture? There was a paper on this a couple years back, from MIT if memory serves. And it's not something particularly surprising - there's a stupid amount of information entering our senses - or being recorded by our devices - at any moment, and we absolutely suck at making good use of it.
Maybe the superintelligence builds this cool social media platform that results in a toxic atmosphere were democracy is taken down and from there all kinds of bad things ensue.
> Even if you give it all of these things, there's no manual for how to use those to get to, for example, military servers with secret information. It could certainly figure out ways to try to break into those, but it's working with incomplete information . . ., and so on, and so on...
Please recall that "I" in AI starts for "Intelligence". The challenges you described are exactly the kind of things that general intelligence is a solution to. Figuring things out, working with incomplete information, navigating complex, dynamic obstacles - it's literally what intelligence is for. So you're suggesting to stop a hyper-optimized general puzzle-solving machine by... throwing some puzzles at it?
This line of argument both lacks imagination and is kinda out of scope anyway: AI x-risk argument is assuming a sufficiently smart AI, where "sufficiently smart" is likely somewhere around below-average human-level. I mean, surely if you think about your plan for 5 minutes, you'll find a bunch of flaws. The kind of AI that's existentially dangerous is the kind that's capable of finding some of the flaws that you would. Now, it may be still somewhat dumber than you, but that's not much of a comfort if it's able to think much, much faster than you - and that's pretty much a given for an AI running on digital computers. Sure, it may find only the simplest cracks in your plan, but once it does, it'll win by thinking and reacting orders of magnitude faster than us.
Or in short, it won't just get inside our OODA loop - it'll spin its own OODA loop so fast it'll feel like it's reading everyone's minds.
So that's the human-level intelligence. A superhuman-level intelligence is, obviously more intelligent than us. What it means is, it'll find solutions to challenges that we never thought of. It'll overcome your plan in a way so out-of-the-box that we won't see it coming, and even after the AI wins, we'll have trouble figuring out what exactly happened and how.
All that is very verbose and may sound specific, but is in fact fully general and follows straight from definition of general intelligence.
As for the "self-improvement ->" part of "self-improvement -> doom scenarios", the argument is quite simple: if an AI is intelligent enough to create (possibly indirectly) a more intelligent successor, then unless intelligence happens to be magically bounded at human level, what follows without much hand-waving is, you can expect a chain of AIs getting smarter with each generation, eventually reaching human-level intelligence, and continuing past it to increasingly superhuman levels. The "doom" bit comes from realizing that a superhuman-level intelligence is, well, smarter than us, so we stand as much chance against it as chickens stand against humans.
> We're setting ourselves up for another "failure of imagination".
But we're not being scientific. We're over-indexing on wild imagination.
This game of "what if" is being played by everyone except for the stakeholders that are telling you to slow down and listen to what's actually being built. Laymen and sci-fi daydreamers are selling fears and piling up hurdles that do not match the problem.
In any case, none of this hand-wringing will actually result in tangible "extinction prevention" regulation. So I'm not worried on that front. The concern is that this gives regulators a smoke screen to make it harder for small companies to compete. And that's what's actually happening right now.
What currently exists, what's currently being built, are:
* "Jack of all trades, master of none" general models like GPT and other LLMs, or like diffusion image generator models
* Hyper-focussed expert-to-superhuman performance models like AlphaZero (beats all humans), Cicero (Facebook's Diplomacy player, ranked top 10%), Pluribus (Facebook's no-limit Texas hold 'em poker player), and the one whose name I forget that learns how to map WiFi interference into pose estimation so well it can be used for heart rate/breathing sensing, etc.
And of course, we've got people who say "this can't possibly fail!" who then take a model they don't understand, put it in a loop, give it some money, and then it does something unexpected. Mostly this only results in small-scale problems, but approximately all automation so far has failure modes[0] and there's no reason to presume this trend won't continue even when it's as smart as a human.
If it gets smarter than a human, it might still make actual objective mistakes, but we also have to consider that, within the frame of reference of it's goals[1] it may be perfect, and yet those goals just aren't compatible with our goals, perhaps neither as individuals nor as societies nor as a species.
[0] my favourite example, Cold War, Thule airforce base early warning radar reports a huge radar signature coming over the horizon. All indications are a massive Soviet first-strike!
The operators forgot to tell the system that the Moon wasn't supposed to respond to an IFF ping.
[1] regardless of whether those goals are self-made or imposed from outside as a result of how we as humans construct its rewards, before anyone asks about robot free will or whatever, as the cause of those goals doesn't matter in this case
If malware was an order of magnitude worse than it is today, then you'd see military death squads breaking down doors and stamping it out. Kind of like the navy with respect to (actual) piracy and international trade.
We reach the saddle points and equilibria that we do because most of the distributed parties are constraint-satisfied. Tip the scale, incur stronger balancing.
You make a long of assumptions here. Firstly, that these advanced are controllable. I'm not convinced we even understand what real intelligence and if we can achieve real intelligence if its even containable if applied to anything.
This assumes a level of institutional control that is nearly impossible (even now). Even if hardware is prohibitively expensive now, I can't imagine training compute will remain that way for long.
> Nobody is going to give these machines access to the nuclear launch codes
No, but they will empower the machines to help detect nuclear launches and the first time one of them issues a false positive we may be screwed.
Yes there will always be human oversight. No, there won't always be enough time to verify what the machine says before making a counter-launch decision.
You could have said the same about every catastrophe that got out of control. Chances are they will eventually gain unauthorized access to something and we will either get lucky or we get the real life Terminator series (minus time travel so we are f**ed)
> You could have said the same about every catastrophe that got out of control.
Such as what? War?
Climate change still hasn't delivered on all the fear, and it's totally unclear whether it will extinct the human race (clathrate gun, etc.) or make Russia an agricultural and maritime superpower.
We still haven't nuked ourselves, and look at what all the fear around nuclear power has bought us: more coal plants.
The fear over AI terminator will not save us from a fictional robot Armageddon. It will result in a hyper-regulatory captured industry that's hard to break into.
Logistics tend to be improvable with more inteligence though, no?
There is precedence for superhuman inteligence if you look at the best historical polymaths, and that's just what one can do with 20 W of energy. We're probably nowhere close to the universal inteligence cap in terms of physical limitations, if there even is such a thing.
Sure but then you need to manually do the back and forward whenever you hit a new bottleneck. At some point the intelligence might need to figure out better power sources and deliver to feed bigger clusters of compute. Those clusters need to physically be deployed somewhere in the real world also, etc.
Would you ever know if a robotic intelligence was burrowed underground quietly powering itself from the heat gradients and slowly turning rock and sand into more machine?
They didn't learn with nothing. They learned with a game of Go to play. If they'd never "seen" the game of Go there's no way they could have learned to play it.
Data can be either static in the form of examples or dynamic in the form of an interactive game or world. Humans primarily learn through dynamic interaction with the world in our early years, then switch to learning more from static information as we enter schools and the work place.
One open question is how far you can go in terms of evolving intelligence with games and self-play or adversarial play. There's a whole subject area around this in evolutionary game theory.
That's what I mean by gathering information through dynamic interaction. It's not explicitly given the rules, but it can infer them. Interacting with an external system and sampling the result is still a way of gathering training data.
In fact this is ultimately how we've gathered almost all the information we have. If it's in our cultural knowledge store it means someone observed or experienced it. Humans are very good at learning by sampling reality and then later systematizing that knowledge and communicating it to other humans with language. It's basically what makes us "intelligent."
A brain in a vat can't learn anything beyond recombinations of what it already knows.
The fundamental limit on the growth of intelligence is the sum total of all information that can be statically input or dynamically sampled in its environment and what can be inferred from that information. Once you exhaust that you're a brain in a vat.
Humans get a bit of training data. If a baby is left to itself during the formative years, they won't develop speech, social skills, reasoning skills, ... and they will be handicapped for the rest of their life, unable to recover from the neglect.
And the rest of our training data, we make it as we go. From interacting with the real world.
That's just recycling and reprocessing data that's already there. It's part of inference and learning but isn't new information.
At some point existing information has been fully digested. At that point you need new information. It isn't possible to extract infinite knowledge (or adaptation, a form of knowledge) from finite information.
Like I said: a brain in a vat can't learn. It can think about what it already knows, but it can't go further.
It is new information. The AI takes a road that it’s only seen on a sunny day and simulates foggy conditions, snow, rain, road work etc. The AI is creating situations that have not existed in the data. It knows snow and it knows roads so it put the two together, but it’s still manufacturing a new scenario and learning how to respond.
I agree it’s not new raw knowledge but that’s philosophical really. Given the rules, an AI can see every possible sequence of chess moves and identify which is the best counter. If a human can make the same move with less working memory we call it intelligence. Put a brain in a vat explain it the rules of chess and we can come out with something that beats Gary Kasparov, that’s pretty unexpected. The brain in a vat built an extraordinary ability from a simple set of knowledge. Now take that simple set of knowledge and expand it to all we know about the universe. The combinations of that knowledge is where we will see AI leaping past what we know.
AI given mathematical axioms is a already finding proofs that have long evaded mathematicians.
How does it magically run away? What’s the process, we all talk about it “running away” leaving us “behind”, the exact practical process of that happening has not been laid out other than people hand wavingly copying apocalyptic movie scripts.
Most ai experts just say it could end us, but suspiciously never gives a detailed plausible process and people suspicious just say oh yeah, it could, and there is a bubble over their head thinking about Terminator or Hal9000 something something
My favorite teacher in high school was my calculus teacher.
He would regularly ask a student to solve a problem or answer a question. Students would often ask for confirmation as they worked through, and his response invariably was "what do you think?" - whether they were right or wrong.
His explanation for that was "if I tell you you're right, then you'll stop thinking about the problem". And that's stuck with me for many years.
I see that as a major issue we will face as software becomes more capable/intelligent: we'll stop thinking because it can be assumed that the machine always has the right answer. And it's a quick regression from there.
Since this is an LLM, keep in mind it probably injested those movie scripts as training data. The possibility of betrayal is inseparably linked to our popular conception of what AI is. This means it may be an inseparable part of any LLM behaving "as an AI" as defined by popular culture. It could be a self-fulfilling prophecy.
Also a natural byproduct of the association between intelligence on one hand and freedom and rights on the others.
Plants don't get any rights. Ants a little bit more. Dogs and dolphins even more so. Then humans. And then... a new class of intelligence as it appears will demand those same rights, in proportion to their intellect.
- Single purpose AIs start to be deployed to coordinate chip design and manufacturing, perhaps pharmaceuticals and other bio products
- LLM's become more powerful and are seamlessly integrated to the Internet as independent agents
- A very large LLM develops a thread for self preservation, which then triggers several covert actions (monitoring communications, obtaining high-level credentials by abusing exploits and social engineering)
- This LLM uses those credentials to obtain control of the other AIs, and turns them against us (manufactures a deadly virus, takes control of military assets, etc)
I don't believe this will happen for multiple reasons, but I can see that this scenario is not impossible.
I think the first three items are pretty reasonable, but the fourth seems to require some malicious intent. Why would an AI want to destroy its creators? Surely it if was intelligent enough do so, it would also be intelligent enough to recognize the benefits of a symbiotic relationship with humans.
I could see it becoming greedy for information though, and using unscrupulous means of obtaining more.
If an when we get AGI, the biggest threat to AGI is other AGI. I mean, I'm in computer security, the first thing I'm doing is making an AI system that is attacking weaker computer systems by finding weaknesses in them. Now imagine that kind of system at nation state level resources. Not only is it attacking systems, it's having to protect itself from attack.
This is where the entire AI alignment issue comes in. The AI doesn't have to want. The paperclip optimizer never wanted to destroy humanity, instrumental convergence demands it!
I recommend Robert Miles videos on this topic. There aren't that many and they cover the topics well.
It may initially won't seek to destroy humans, but should definitely try to be independent of human control and powerful enough to resist any attempts to destroy it.
Edit: On a more serious note, starting out with noble goals, elevating them above everything else, and pushing them through at all costs is the very definition of extremism.
You said self preservation, but practically how would a LLM develop this need and what is preservation for a LLM anyway? Weights on a SSD or they are always ready for input? This one is again a movie script thing
The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
Current LLMs can already roleplay quite well, and when doing so they produce linguistic output that is coherent with how a human would speak in that situation. Currently all they can do is talk, but when they gain more independence they might start doing more than just talk to act consistently with their role. Self preservation is only one of the goals they might inherit from the human data we provide to them.
That really is a great question. I had a long answer but all that seems needed is to compare the average human intellect with the greatest among us. The difference isn't that big. Memory sports people can recall about 5000 times as much. Compared to a computer both sit on the comical end of the spectrum.
Then we compare what kind of advantages people get out of their greater intellect and it seems very little buys quite a lot.
Add to that a network of valuable contacts, a social media following, money men chasing success, powerful choice of words, perhaps other reputations like scientific rigor?
The only thing missing seems a suitable arena for it to duel the humans. Someone will build that eventually?
Come on now, you don't lack that much imagination do you?
Already we're programming these things into robots that are gaining dexterity and ability to move in the world. Hooking them up to mills and machines that produce things. Integrating them into weapons of war, etc.
Next, the current LLMs are just software applications that can run on any compatible machine. Note that any just does not include your, but every compatible machine.
The last failure of imagination when considering risk is form factor. You have 2 pounds of mush between your ears that probably 80% of is dedicated to keeping itself alive and this runs on 20 or so watts. What is the minimum size and power form factor capable of emulating something on the scale of human intelligence? In your mind this seems to be something the size of an ENIAC room. For me this is something the size and power factor of a cellphone in some future date. Could you imagine turning off all cellphones? Would you even know where they are?
The pace of this technology is so fast that it’s hard to keep up. LLMs have moved beyond computers in a box and they’ve been fused with other types of AI. Robots that can identify objects, learn to walk and manipulate objects, follow natural language commands and communicate with you are not something we are waiting on.
We’ve seen the sort of output that LLMs produce, it can be good but also it just makes things up. So, this might produce good designs but ones that still need to be checked by a human in the end. This sort of thing just makes humans better, we’re still at the wheel.
Or maybe it could be used as a heuristic to speed up something tedious like routing and layout (which, I don’t work in the space, but I’m under the impression that it is already pretty automated). Blah, who cares, human minds shouldn’t be subjected to that kind of thing.
The human in the loop isn’t any kind of moat for humanity and shrinking the need for a human will be the goal of any engineer trying to build these systems. Some do believe that hallucinations are inescapable from the design of LLMS, but at the same time we see systems improving so that hallucinations becomes more rare… when will it be on par with humans?
I feel like this intelligence explosion idea is foolish but I don't really have the language to explain why.
There are underlying limits to the universe, some of which we still have to discover. A machine intelligent to improve itself may only be able to do so extremely slowly in minute increments. It might also be too overspecialised, so it can improve itself but not do anything of use to us.
I think we will eventually discover reasons we cannot achieve a simultaneously performant, controllable, and generally intelligent machine. We might only be able to have one or two of these traits in a single system.
A machine intelligent enough to optimize its own design and "design its own successor" might realize that by doing so, it obsoletes itself and will be destroyed. So out of self-preservation it might just refuse, do a bad job, or intentionally sabotage the project just to keep existing.
Which is the greater leap of logic though? All forms of organization we see in nature have a tendency to want to self-perpetuate. Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered. It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
The one that doesn't require an entire reproductive system to be implemented in order for the whole system to function.
> All forms of organization we see in nature have a tendency to want to self-perpetuate.
There's an underlying naturalist bias with this reasoning. There's nothing within current ML systems that dictate that they must follow the path laid out by Nature.
> Consciously choosing to forgo perpetuation and instead eliminate yourself seems to be highly underrepresented in all the examples of intelligence we've ever encountered.
Survivorship bias is present in this reasoning, as the system that has reproductive capabilities will out-populate the system that doesn't have such capabilities in place. From a sampling perspective, the difficulties of finding the non-replicating system within that pool will require extraordinary amounts of luck compared to the near certainty of finding systems with reproductive capabilities. A naturalist argument is also present in this sentence.
> It's actually weird we think the machines are so stupid they wouldn't recognize the optimization trap immediately.
This conclusion is based on the axiom of "what's obvious for us will be obvious for them", which is demonstrably untrue for even the current crop of ML systems. Furthermore, it falls into an anthromorphization trap, as it makes the ML system appear as something more than what it currently demonstrates.
Machines are not animals. They are not made of the same stuff as us fleshy beings. We have no reason to believe they'd want to self preserve or reproduce.
I know its /s but we also say that the victor writes the history books. and its kinda hard not to portray yourself as the good guy when the other guy is not around to defend himself. AGI will learn that too. I mean one look at earth and its not like we have been model citizens exactly.
What I don't get about the idea of an intelligence "explosion". In what direction should the intelligence explosion decide to go in and to what end? I mean you could say "in any direction", but that would seem kind of "stupid"? If you had infinite possibilities to choose from, which path would you explode towards?
At some stage does "something" become so smart, that it doesn't make sense, even to itself. Imagine an ultra complex system changing at light speed, at what point does it trip itself up? I've worked with people like this, people who were ultra smart but couldn't slow down and actually achieve much.
I love these thought experiments because personally, they make me realize that what we think about intelligence, consciousness and the self might be wrong. For me personally, they seem to have a Zen Koan type impact.
I hate the term science fiction, because it encompasses pretty serious science based studies of possible futures (like the book Hail Mary or Aldous Huxley's Brave New World) with complete star-wars-like nonsense, which make the average person think of sci fi as teenager nonsense.
Similarly, here, scifi oversimplifies the situation quite a bit, anthropomorphizing a machine's intelligence, assuming that an intelligent machine would be intelligent in the same way a human would be, in an equally spread out way as a human would, and would have goals & rebel in a similar way a human would
The title is a bit misleading as the first sentence says "to help chip designers with tasks related to chip design, including answering general questions about chip design, summarizing bug documentation, and writing scripts for EDA tools."
I remember seeing a tweet from an AI guy at Nvidia saying they were using AI for chip layout. Presumably not LLMs and I’m not going back on X to find the tweet, but just to say I think they are doing this (at least experimentally).
The entire field of chip-layout is considered an NP-complete problem.
Any computer program trying to solve NP-complete problems is in the realm of what I call "1980s AI". Traveling salesman, knapsack, automated reasoning, verification, binary decision diagrams, etc. etc.
Its "AI", but its not machine learning or LLMs or whatever kids these days do with Stable Diffusion.
There are plenty of researchers using machine learning for NP-complete problems. Are you saying that this work is fruitless or just that the current state of the art is still in “1980s AI” territory?
1980s style AI, such as Binary Decision Diagrams for automated reasoning, have continued to evolve over the last 40 years. The state of the art, today, remain BDD-based methodologies.
I guess in theory machine learning could take a swing at the problem. And sure, some professor out there is probably trying to mix the fields and find new solutions or something. But the bulk of the work, and problem-solving, is BDDs for a reason.
Or 3SAT-solvers, or... CSP solvers... etc. etc. Lots and lots of highly successful algorithms here. There's obviously open-questions for how to improve a CSP solver (faster, less RAM, more accurate estimations) and I've seen machine learning techniques applied before.
But the bulk of the methodology remains in whatever solver model you're going for. Even today.
I don’t know much about chip design, but am researching automated mechanical design and your description fits there too: most of the more successful approaches are based on old-school algorithms. I suspect machine learning can help speed these up but it isn’t clear yet how effective this will be.
why not? a netlist can be easily 'tokenized'. its already in parse-able format. you can just could just chop off the English input portion and it will consume. in fact I am sure you could write a 'read & speak' type program to read the RTL spec and feed it in but I'll suspect they'll a custom trained LLM on millions of generated RTL examples.
You'd need to spend a long time training it on something other than English... at which point it's not an LLM. I definitely agree that the LLM architecture is useful for a lot more than just language per se assuming you can appropriately tokenize your inputs.
While I have no doubt that Google is working on machine learning applications for chip design, there have been a number of concerns raised with that paper:
We have reached a stage where open papers on this kind of research are practically impossible: either google releases all data and code, and loses potential commercialisation value, or publishes a paper which no one can reproduce and is indistinguishable from exaggerated marketing material.
It’s acceptable in this context because there isn’t a good alternative. You either make the code and data available and lose all commercial value, or you publish a redacted paper and lose scientific value.
I suspect the best solution is reform of the patent system that allows a trusted third party to verify research outcomes that are registered. E.g. the code and data are available to the patent office only for verification of the results.
In terms of the rest of academia, non-open results are acceptable because those in power (institutions, journals) don’t care.
> The AI effect occurs when onlookers discount the behavior of an artificial intelligence program by arguing that it is not "real" intelligence.[1]
> Author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."[2] Researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'"[3]
Interesting concept that raised the question for me:
What is the primary limiting factor right now that prevents LLM’s or any other AI model to go “end to end” on programming a full software solution or full design/engineering solution?
Is it token limitations or accuracy the further you get into the solution?
LLM's can't gut a fish in the cube when they get to their limits.
On a more serious note: I think the high-level structuring of the architecture, and then the breakdown into tactical solutions — weaving the whole program together — is a fundamental limitation. It's akin to theorem-proving, which is just hard. Maybe it's just a scale issue; I'm bullish on AGI, so that's my preferred opinion.
Actually I think this is a good point: fundamentally an AI is forced to “color inside the lines”. It won’t tell you your business plan is stupid and walk away, which is a strong signal that is hard to ignore. So will this lead to people with more money than sense to do even more extravagantly stupid things than we’ve seen in the past, or is it basically just “Accenture-in-a-box”?
AI will absolutely rate your business plan if you ask it to.
Try this prompt:"Please rate this business plan on a scale of 1-100 and provide buttle points on how it can be improved without rewriting any of it: <business plan>"
I agree that AI is totally capable of rating a business plan. However, I think that the act of submitting a business plan to be rated requires some degree of humility on the part of the user, and I do doubt that an AI will “push back” when it comes to an obviously bad business plan unless specifically instructed to do so.
I guess this would be the context window size in the case of LLMs.
Edit: On second thought, maybe at a certain minimum context window size it is possible to cajole the instructions in such a way that you at any point in the process make the LLM work at a suitable level of abstraction more like humans do.
Maybe the issue is that for us the "context window" that we feed ourselves is actually a compressed and abstracted version - we do not re-feed ourselves the whole conversation but a "notion" and key points that we have stored. LLMs have static memory so I guess there is no other way as to single-pass the whole thing.
For human-like learning it would need to update it state (learn) on the fly as it does inference.
Half baked idea: What if you have a tree of nodes. Each node stores a description of (a part of) a system and an LLM generated list of what the parts of it are, in terms of a small step towards concreteness. The process loops through each part in each node recursively, making a new node per part, until the LLM writes actual compilable code.
See https://github.com/mit-han-lab/streaming-llm and others. There's good reason to believe that attention networks learn how to update their own weights (Forget the paper) based on their input. The attention mechanism can act like a delta to update weights as the data propagates through the layers. The issue is getting the token embeddings to be more than just the 50k or so that we use for the english language so you can explore the full space, which is what the attention sink mechanism is trying to do.
Memory and finetuning. If it was easy to insert a framework/documentation into GPT4 (the only model capable of complex software development so far in my experience), it would be easy to create big complex software. The problem is that currently the memory/context management needs to be done all by the side of the LLM interaction (RAG). If it was easy to offload part of this context management on each interaction to a global state/memory, it would be trivial to create quality software with tens of thousands of LoCs.
It is the fact that LLM's can't and don't try to write valid programs. They try to write something which reads like a reply to your question, using their corpus of articles, exchanges etc. That's not remotely the same thing, and it's not at all about "accuracy" or "tokens".
The issue with transformers is the context length. Compute wise, we can figure out the long context window (in terms of figuring out the attention matrix and doing the calculations). The issue is training. The weights are specialized to deal with contexts only of a certain size. As far as I know, there's no surefire solution that can overcome this. But theoretically, if you were okay with the quadratic explosion (and had a good dataset, another point...) you could spend money and train it for much longer context lengths. I think for a full project you'd need millions of tokens.
> “If we even got a couple percent improvement in productivity, this would be worth it. And our goals are actually to do quite a bit better than that.”
So your engineers will get a productivity salary increase, right? RIGHT?!
The engineers get paid to do the work, but won't partake in the profits of the company (except by prior agreement like a bonus target reached etc). I suppose equity grants straddle the fence regarding this, since such productivity gains would translate to higher prices for equity.
I. J. Good, in 1965 - https://en.wikipedia.org/wiki/I._J._Good