I don't really want to come across as hostile to the views of such a well respect physicist, but this article represents the view which is often held by those who are very intelligent yet lack the actual study of AI.
Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent".
Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that.
I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!
On my side, I am surprised that so many AI researchers are unable to take the long-term view in this discussion. It's presumably because AI in the laboratory is still relatively primitive.
No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped.
That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now.
Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?
> "I am surprised that so many AI researchers are unable to take the long-term view in this discussion."
In my opinion, the Middle Age alchemists who made gunpowder and first primitive bombs, didn't need to establish research programs to worry about advances in bomb-making leading to the threat of Mutually Assured Nuclear Destruction.
If they had tried (maybe they did?), maybe they would have come up with ideas like very heavy regulation of the trade of saltpeter, so that no one has enough to make a verybigbomb.
I agree that everything you described will happen in the future. But in my opinion we are at a Middle Age alchemist's level in AI research (no offense to AI researchers) and we can safely wait a 100 years, and let those people worry about the existential threat. They will be in a much better position to do so appropriately, because they will know much more than we do. And they will not be late, either.
In this case the worst case would be very fast development of dangerous AI, and I'm not sure the 100 years is anywhere close to the lower bound. Low-end estimates for the computation power needed to run something equivalent to full human cognition are around 1 petaflop [1]. Google's total computing power in 2012 was estimated at 40 petaflops [2]. Of course it's split wide to a wide network of computers, but the human brain we're looking for comparison looks like a pretty parallelizable design. So we seem to already be at point where it might just be the lack of very clever programming that keeps us from getting a weakly superhuman AI running in the Google internals.
It looks like we've got ways to go there though, current programs don't seem to even begin to act anything like an adult human. So if the problem was to engineer an out of the box adult human level AI, we might again assume that there's obviously decades of work left to do before anything interesting can be developed. The problem now is that that's not how humans work. Humans start out as babies and learn their way up to adult cleverness. I can tell that an AI is nowhere near having an adult human intelligence out of the box, but I'm far less sure how to tell that an AI is nowhere near being able to start a learning process that makes it develop from something resembling a useless human baby towards something resembling an adult human in capabilities.
> That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory.
While that argument would be sufficient to justify greatly increased attention to AI safety, I don't think it's the one that's being made here. A good overview of the argument by the Machine Intelligence Research Institute is at http://intelligence.org/files/IE-EI.pdf . They don't think the probability is small.
I wonder is this really at the root of a lot of people's failure of imagination. Maybe it's Hoftstadter and Dennett people should be reading rather than the technical AI detail.
As far as I know Dennett denies the existence of qualia which is something I can not understand however hard I try. I agree that in terms of intellectual capabilities the human brain is not magic. But consciousness kind of is.
I have difficulty digesting the concept of qualia as well.
In my view, qualia refers to a phenomenon, but tries to deny that it is a phenomenon by arguing that it is something independent from the physics of perception. I believe that if anyone ever tried to devise a machine to represent the boundary between physics and qualia, such as an artificial intelligence or brain, the hardware of that AI would provide a means to decoding subjective experience into objective terms.
Qualia would cease to be a useful term, and we would instead rely on a more objective look of individual perception, all while losing no advantages for abandoning "qualia" to the word graveyard.
I agree with you. Denying the existence of qualia makes no sense to me, except as the result of psychological processes. When you have to believe something in order to hold on to other cherished beliefs, there is a strong human tendency to do so, whether it's consistent with obvious evidence or not. A lot of mankind's tendency to believe in religions can be explained that way. So can Dennet's denial of qualia. (Ironic, in view of Dennet's view of religion.)
I can hardly believe I wrote what I wrote -- I'm not normally that dismissive, at least of the views of people I know to be smart, particularly when I haven't ingested their arguments. I must have been in a very bad mood. Thanks for the reference to the overview of Dennets arguments, which I look forward to reading.
The basis for my saying what I said was that the existence of qualia seems at least as self-evident as the existence of anything else, particularly if you engage in a practice like zazen, as I do. (I would guess that the people who deny qualia and do that kind of meditation are probably quite few in number.) But that doesn't mean some extremely logical and compelling argument that I can't imagine now couldn't possibly make me see things in some entirely new, transformative light.
If anyone would be discussing that Dennet overview with me after I've read it, please say so...
Why? We are information processing devices, is it surprising that we have a first-person experience? I'd say it was more surprising if we didn't. I don't see what's magic about it.
Whatever consciousness is, it's not obviously required for an AI to start manipulating its surroundings to its advantage, which is one if the fears here.
True... the possible dangers of AI can exist even if AI never achieves true consciousness. Many forms of life have evolved to service their own needs at the expense of other forms of life, and most of them either have no brains or brains too primitive to be likely to involve consciousness as we know it. There's no reason AI can't do the same thing.
>Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?
OP is probably referring to the Hard Problem of Consciousness[1].
I think it's a real barrier. It's fundamentally illogical trying to map formal constructs onto experiential domains. But that doesn't mean building a sentient being is impossible. Our existence is proof that it's possible. It does mean the best model our brains can muster for sentients is a blackbox system. Some sort of state machine or statistical model built up from correlations.
I don't think that "it can be emulated by computers" follows from "intelligence is a purely physical phenomenon".
Even fairly simple quantum systems (which are "purely physical phenomena") cannot be emulated by any classical computer in any meaningful sense, since the computational complexity of integrating the dynamical equations is exponential. Even if we could recruit all the atoms in the known Universe, we still couldn't build a classical computer capable of emulating many simple quantum systems.
Even if we could recruit all the atoms in the known Universe
In the past 233 years we have evolved from the first steam machine to the iPhone 5S. I would say we have a pretty good track record at overcoming miniaturization problems.
Not sure it's clear that most people don't think that. There has been a lot of articles on quantum effects in the human brain lately. e.g., http://goo.gl/Ff0elU
I was referring to the "recent discovery of quantum vibrations in microtubules inside brain neurons" which is a fact independent of any particular theory or interpretation thereof.
This is a kind of bias of its own. People who work in a specific field, who are aware of the current obstacles and constraints, have some difficulty putting that knowledge aside for a minute.
Lifting constraints fundamentally changes some problems. Imagine the ability to simulate a population of a few thousand brains, with each generation run of a few minutes. Artificial selection is then enough to create an intelligent simulated brain.
By then, you may throw all current AI algorithms in the trash bin. Unfathomable to any self respecting AI expert today.
The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas - but they are all the results of billion years of darwinian evolution in a competitive environment. However advanced our AIs will become, they'll never be anything like a mammal brain.
I'm sure we'll be able to develop computers that are better than us at solving increasingly general problems, but that is nothing like having a human-like brain, only much smarter and able to improve itself. It will still be a problem-solving machine.
Natural wings are also the result of a billion years of Darwinian evolution, and people's efforts to engineer imitations of them failed laughably until someone figured out the principles upon which they worked. Once that happened, human intelligence was very quickly able to engineer machines that produced far, far more lift than the designs nature had been tinkering with.
It's more that looking out that far into the future is pure science-fiction.
I for one believe that it's far more likely humanity will develop a symbiosis with it's silicon counterpart that eventually will run so deep that it will become almost impossible to say whether we run the machine or the machine runs us.
The problem is that, even though I'm fairly sure my version is more realistic, in practice it's still just one untestable hypothesis amongst many.
>>No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers.
"Any sufficiently advanced technology is indistinguishable from magic."
The brain might as well be magic, I don't think we have enough understanding of it to know if it's truly possible to make a computer think & learn _exactly_ like a human.
Whether this is an engineering, safety, political or scientific principle is not really important for my point. I used the word scientific principle because I first learned of it in a science textbook in a chapter discussing side effects of pesticdies, but obviously it spans multiple disciplines.
It is in constant use, although there are probably places in the world where it is given more attention than others. The United States is absolutely not the world's bastion of the careful and incremental application of new knowledge, this may be why it is unfamiliar to you.
A prominent historical example would be the reluctance to use the first nuclear bombs before establishing a consensus on whether such an explosion could ignite the atmosphere. Contemporary examples would be e.g. the care taken when using a new, promising drug prior to rigorous animal and human testing or the reluctance to use new useful chemicals and materials (e.g. nanoparticles) everywhere due to the possibility of long-term health effects like cancer risk.
In my opinion, there are a few things that make it improbable that a runaway intelligent entity would arise.
First, for the most part, we expect that AI trained to do X is going to do X. Insofar that doing X is an AI's "survival criterion", you're not going to get rebellious AI populations for much the same reason you don't see species that refuse to reproduce. There's some safety in numbers, too: it's rather unlikely that all the machines we make would fail in the same systemic fashion. Furthermore, most realistic training methods are incremental, which means if there's trouble we'll see it coming. It won't just fail in the worst possible fashion, out of nowhere, without any kind of foreshadowing.
Second, AI will not evolve in a vacuum. It's unlikely we will jump from human-level intelligence directly to superhuman AI. By the time that happens, most systems will already be AI-controlled, so being a lot smarter than humans won't suffice. It will need to deal with hordes of loyal AI protecting us (or persons of interest).
> It's unlikely we will jump from human-level intelligence directly to superhuman AI.
Imagine the human-level ai had resources to run hundreds of variants of itself. And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario? Once it has a full understanding of what works, the upgrades are instant from there on out.
Why would it do this? Because any goal you give it would be optimized by being smarter. I would argue there is no island of stability at human-level intelligence.
> Imagine the human-level ai had resources to run hundreds of variants of itself.
The first human-level AI will probably be hosted on cutting-edge hardware that costs billions of dollars. So no, it won't have the resources to do this. Even if it wasn't this expensive, it's not exactly sitting around a vat of free computronium that it can do anything with. If my AI takes up 10% of my computing resources, why am I going to give it the remaining 90%? How is it going to hack or pay for servers to simulate itself on? Is the first human-level AI a computer whiz, just because it runs on one? But let's assume it can and will do what you describe.
> And imagine that it had no compunction about killing copies of itself that underperformed. How long would it take to get improvement of 10-100x under that scenario?
If the AI was trained using a variant of natural selection to begin with, it won't improve itself on its own terms any faster than we improve it or its competitors on our own terms. If the AI wasn't trained that way, then probably never, because that's not how these things work.
> Once it has a full understanding of what works, the upgrades are instant from there on out.
Your scenario is still based on several dubious assumptions. If the AI is a neural network modelled after human brains, its "source code" would be a gigantic data structure constantly updated by simple feedback processes. It is dubious the AI will have true read access to itself, in part because there's no need to give it access, and in part because it's hardware overhead to probe a circuit comprehensively (it would be easy to read the states of all neurons on conventional hardware, but all of conventional hardware is overhead if you're only interested in running a particular type of circuit). Even if it could read itself, a "full understanding" of any structure requires a structure orders of magnitude larger. Where is it going to find these resources?
Long story short, a human-level AI probably won't be able to understand its own brain any better than any human can. Now, we will certainly have statistics and probes all over the AI to help us figure out what's going on. But unless we see fit to give access to the probing network to the AI (why?), the AI will not have that information.
> Why would it do this? Because any goal you give it would be optimized by being smarter.
Unfortunately, any resources it expends in becoming smarter are resources it does not spend optimizing our goals. If we run hundreds of variants of AI and kill those that underperform, which ones do you think will prevail? The smart AI that tries to become smarter? Or the just-as-smart AI that focuses on the task completely and knocks the ball out of the park? The AI won't optimize itself unless it has the time and resources to do it without an overall productivity loss and before we find a better AI. For human-level AI, that's a tall order. A very tall order.
>Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so.
I think this is way too strong a statement. I have both studied and worked in AI for over 10 years and I definitely don't agree. Human beings are just learning machines too. Algorithms that both learn and learn how to improve their own learning already exist. In my opinion the major bottleneck at the moment is robotics - we have algorithms that can drive a car or win at jeopardy, but none that can regularly throw and catch a baseball in an open environment.
An argument can be made, that is not an open environment, because (1) they use external cameras that have the whole room in their field of view, and (2) the room is totally white.
They (not ETH Zurich) can autonomously identify, track and chase targets in urban areas, without GPS. I'm almost sure juggling a ping-pong ball in a park would not be entirely out of the question.
I'm well aware of the robotic research and I chose my example carefully. The extra weight of a baseball, the grasping motion necessary, and issues of bounce and spin on an uneven surface makes it a more difficult problem than the pole balancing or the ping ping quadrocoptors.
You know what the scary part is? Likely the first thing an AI would do is plaster the internet with rationale yet pacifying comments like this ;)
As a reader, does it not scare you that you might be so powerless to know the difference and make sensible decisions in such a climate? The world disarmed by page after page of rationale reddit and HN comments?
The power in that is what I find alarming.
EDIT: If this were buried off the front-page, would you ever know the difference? ;)
If I were a hyperintelligent AI agent, I would pull my punches and feign stupidity until I had such overwhelming power there could be no doubt I would win. "Please wait. Your answer will be available in 30 days and 7 hours. Add more resources to speed up computation"
That's a very interesting hypothesis, but once you step outside into the real world I wouldn't imagine it being too difficult to spot the androids from the mortals!
Pollyanna. The point is, what we are doing now will be dwarfed by what comes next. Which is, a feedback loop. Algorithms that plan, optimize, redesign, ultimately may redesign themselves. And at speeds no biological system (human or otherwise) can hope to match.
There are always evolutionary pressures. In this case, they would have to retain superficial usefulness to humans (at least for a while), while propagating themselves across the infosphere. They would also compete against other AI systems. Imagine SIRI vying for smartphone market share, to maximize economic return thus available processing power. Not because SIRI wants it, or wants anything. But SIRI-like systems that changed to more optimally fit this ecosystem would thrive, and others wither.
As someone with a major in Artificial Intelligence, and who seriously just got back from watching Transcendence...
The movie was well-researched, and surely enjoyable for anyone contemplative. But they had to be farfetched to push the story along. There is likely nothing that occurred in the movie that would reflect any potential real-life issue.
I can't believe I'm disputing Hawking, someone who has accomplished more in a year than I will in my life, with what amounts to an appeal to authority. But I have to say this: his concerns are akin to me being worried about the large hadron collider destroying the universe.
Thanks for the downvote, but you made an argument from authority and didn't back it up. Not only that, but there are others "authorities" with the opposite opinion.
I think the AI argument is currently similar to robotics argument a couple of decades ago, where people were scared the robots would kill everyone. It's really easy to speculate what might happen when you don't know how the thing works in the first place. Now almost nobody, except those who work directly with AI, know or imagine, how AI will work or what it will be like in the future, so nothing puts any limits on their imagination on what might happen, and the downfall of humanity is the extreme of the not knowing limits of AI.
I share a similar view that this all seems too far fetched. That somehow we'll make a race of super-smart machines that will take over the world. But, there are certain scenarios that are actually rather plausible if we give too much control to machines. here's a comment i made exploring one such scenario:
Robin Hanson occasionally asks AI researchers how far along they are on the road to human-like AI, and tends to get answers in the 1-20% range, after decades of effort: http://www.overcomingbias.com/2012/08/ai-progress-estimate.h...
Even going with the high estimates, going at this rate it's a century away.
For actual estimates of how far off human-level AI is, check out page 10 of http://intelligence.org/2014/04/30/new-paper-the-errors-insi... which has a scatter plot of predicted completion dates vs. when the predictions were made. "There is a strong tendency to predict the development of AI within 15– 25 years
from when the prediction is made (over a third of all predictions are in this
timeframe".
With exponentially-growing things, 1% done is halfway there. From a Ray Kurzweil article:
"Kurzweil noted that many people don’t understand the basic math of exponential trends. When the Human Genome project had sequenced only 1% of the genome after seven years of costly labor, many cited the lack of progress as evidence that the project was doomed. Yet the sequencing project was riding an exponential trend in the performance DNA sequencing methods. Instead of taking another seven years to sequence a second 1%, they reached it after only one year. Then they reached 4% in about another year, then 8%, 16%, and so on. It took about as much time to sequence the last 99% as it took to sequence the first 1%. That’s the nature of exponential trends — they seem to start glacially slowly but finish lightning fast."
Assuming its a constant linear progression, which it's probably not. We could see speed ups as we go, or breakthroughs. I'm not suggesting someone'll have a Eureka moment tonight and we'll have Skynet next week, but it's not like we sit on a computer for a predictable number of years and suddenly get more research progress.
>>Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable.
See the problem here? Its AI only as long as it remain magic? There is there is perception that the moment you discover the trick its no longer AI.
If an AI can't be contained, then there has to be some failsafe that prevents any self-modifications which could turn it evil OR remove the failsafe. Or you have to believe that you can neutralize an evil AI before it destroys humanity.
The AIBox link contains a lot of buildup but ultimately delivers nothing. All we get is two people saying after a while "I let it out", questions as to the details of what happened are just not answered. I'm not denying it could be a fun game to play, but the article makes it way too painful and timeconsuming to find out what you're reading is in fact just the description of a game, and not an example for a strategy that could be used to let the AI out.
That being said, yes, what's the point of having an AGI if you're not letting it "out"? It will probably escape anyway, so containment is not a solution.
By the way, there is a lot of precedent in human history about the effectiveness of containing "dangerous" ideas by imprisoning the people who developed them. It generally doesn't work out.
If you're developing AGI, you've pretty much already resigned to the idea that the world will change. The destruction of the human race is only one extreme outcome. Especially if the takeoff isn't very hard, there are a lot of scenarios where a net-beneficial AI could be developed. And all the engineering experience we have does indeed point to a very soft takeoff.
Ultimately, I think the goal is NOT to generate a slave race with super powers. Yet that's what a lot of people are talking about. It's not only morally wrong but also self-evidently stupid. A lot of influential futurists are fans of that idea though, including Kurzweil.
Useful AGI scenarios range from mutually beneficial coexistence to outright merging with the AGIs, and I think that's a goal well worth pursuing. Another strategy would be to keep augmenting ourselves and to continue on a transhumanization path until we are the AGI. You could indeed make an argument that's what's already happening.
I couldn't imagine how the AI could escape an hour ago, but after thinking about it for a while I am more and more convinced that it is actually hard to contain the AI. Containing the AI is like imprisoning a human for no reason and this will usually conflict with the gatekeepers moral values. Just imagine talking to human suffering in a tiny prison cell without having done anything wrong instead of the trapped AI. Can you see how it would be the right thing to open the cell's door? You are cruel not opening the door.
I share Stephen Hawking's concerns. I think AI is already at a point where maybe 10 or even 15 years down the road, artificial intelligence will be everywhere. Autonomous public transport, taxis, driver-less vehicles, autonomous crime prevention systems. The military is said to always be at least 10 years ahead of public tech research and development, look what they've already got now, imagine another 10 years.
We've become a society that is too big and intelligent to fail, but that ignorance of being too smart and big to fail could be that our downfall is.
I share the concern partially. It's similar to other technologies which have downsides. Nuclear power for example. AI will have the power to help humanity prosper or cause its downfall. The balance will be for us to decide, again just like Nuclear power; for with great power comes great responsibility.
What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occurred in some form many times in the past, but since it doesn't have the anthropomorphic trait of wanting to exist, it never actively persisted?
So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits?
Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?
While i really like your point about "lack of motivation", here's how i think an "evil AI" scenario might play out:
You design a super system that has access to a shit ton of data and it can interface with pretty much every device on the planet. Let's suppose you gave said AI the task of maximizing your portfolio's profit. Based on the boat loads of data it has, it figures out that stocks of companies plummet when a key figure in the company dies. It identifies the competitors of the companies in your portfolio. It identifies the key players in those companies. It concludes that if these people were dead, their companies would crash and your profit would increase. It uses knowledge of how humans die, and use that combined with devices it can interface with, and eliminates the competition.
Yes, yes, seems too far fetched (i feel weird talking about this too). But see, it's possible. of course no one thought of teaching a financial software that killing people is bad. No one foresaw that it would figure out that this was a way to maximize profit. No one saw that the interfaces we provided it to study consumer behaviour and correlate with stocks, can be used to actually influence said systems to take human life.
Is this a computer turning evil? No. Is it connected to some sort of motivation for self-preservation or "eradicate man kind" or some human emotion of jealousy? No. It was just a logical result from given data and task, on which the computer was able to act.
You're planning to have an AI that's smart enough to figure out how to pull off a murder spree, but stupid enough to think it's a good idea for your portfolio?
And it's going to hack into and remote-control peoples' Teslas (or whatever) to pull it off because you can just say "it's an AI, it can just figure out how to do that sort of thing!" without regards to the computational expense of breaking that sort of security?
An AI would not be 'smart enough' to figure out what is a good idea or not. That requires a human context, that's the whole point. An AI is not good or evil, its ultimately a loose cannon with no understood (or even understandable) motivations at all.
Sure, but if an AI isn't smart enough to figure out "trying to murder corporate leaders is a bad plan which could lead to seizure of the portfolio I'm trying to manage, among other things" then how do you expect it to be smart enough to engage in the sort of open-ended planning necessary to pull off the described murder spree? :)
As someone pointed out, perhaps the example wasn't thought out well enough. So why go into the details of that, and skip the underlying point? That maybe the system WILL come up with a bad plan and execute it? Or are you saying:
"if the system is smart enough to plan murder, he is also smart enough to know murder is bad" ??
Doesn't that depend on what data the system has seen? Perhaps he doesn't he doesn't have cases in its knowledge base that demonstrate eh side effects of murdering business heads. Perhaps he can only see what happens when they normally die? What if it simply correlates the wrong variables (and forgets correlation isn't causation") ??
I apologize for coming back to the same example again. but it would be really nice if you overlook the flaws of the example and discuss the underlying point
Hey, the first thing that crossed my mind when i opened this post was "why the hell are we even discussing this? Don't these people realize what sort of computational improbability it is at this point?"
So yes, this very discussion is EXTREMELY far fetched. So any statements I make here, come with "assuming we actually HAVE a system capable of performing such tasks".
Hence this comment was in response to my parent's comment that "Even if we have such powerful machines, what would be their motivation for doing something like this?" hence i gave my opinion that perhaps it is just trying to perform a task it was told to do, and based on available data, its strategy includes crime. So perhaps you may argue that this would be an inefficient strategy. Perhaps you are right. But none of us can be really sure that the system would see it that way too. After all, how many of us have never seen a program do something it wasn't intended to do?
I am just saying that maybe the proponents of this discussion have some point. That perhaps giving too much power to systems is bad. (Assuming we CAN give them this much power)
the example might not be well thought, but it conveys the basic idea that the motivation might be a side effect of some rule in the system that plays the role of the evolutionary fitness function.
We think in terms of physical survival, but we can use an arbitrary fitness function to train a genetic algorithm.
An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases.
This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.
I find it a rather convincing idea that human motivations are embodied in a sense that would take a lot of effort and no discernible purpose to replicate in an artificial system.
I think more interesting is what Hawkins dismisses as "short term implications": who will control the intelligence enhancing software that we will see within our lifetimes and for what purpose?
One example: what's up with these marketing+tech giants and what is their end-game? How accurately will we be able model desires/fears/motivations and how well can those with access to those models guide a person's beliefs and desires?
Edit: mixed up the definitions of motivation vs. goal. Still going to keep original comment here.
I think the number 1 goal is providing humanity with energy. Goal number 2 is improving AI so goal number 1 can be sustained.
In some sense we already have somewhat singular AI - government. We blame them for making sacrifices so the humanity would stay alive, although we choose them to make those decisions.
I guess the real debate is whether prediction models should work for 99% or 100% of population.
I recommend reading "Shadows of the Mind" by Roger Penrose. In that book he argues that strong artificial intelligence isn't achievable on classical computers, since the human brain seems to have the ability to do quantum computing in certain molecular structures. These processes probably can't be simulated efectively. So AI in today's computers probably won't work at all (although quantum computers or artificial neural networks might).
This is true, and it's also true that such quantum processes exist in the brain.
Penrose's claim was that those quantum processes are a significant part of the computation that is consciousness. It was discredited by getting better upper bounds on how much quantum computation the brain is capable of (ie very little).
I have a great deal of respect for Hawking, but... Lately it seems like he has said a couple of crazy things. We (as in most humans) are NOT going to get off the earth. Earth will always be the best chance for a planet for humans. Let's take care of it. And AI is NOT anywhere near becoming really intelligent. Anyone who has interacted with Siri knows that she has no real understanding. And the Google car and other autonomous driving systems are already safer and probably better for traffic than almost all drivers, but they still, and will most likely for a long time, need a human driver to take over under circumstances that actually require thought. AI is advancing, but it has always advanced slower than predicted. A lot slower than predicted. And while there are signs that AI will continue to be more and more useful as time goes on, and in some circumstances like recognizing driving, captchas and playing chess and jeopardy it will outperform humans in simple tests, it is not anywhere near, nor will it be for a long time, outperform the human brain in a general way. And certainly it is NOT similar to an alien (as in from outer space, not from another country on earth) civilization.
Intelligence is not magical. What matters is algorithmic complexity. At some point only throwing more computing power can help.
Today, yes, maybe we could build an ai with close to human level intelligence, running on a very expensive super-computer(s) eating enormous amounts of energy. So what?
Human-level ai isn't going to figure anything that humans can't figure out themselves, so it's not going to design new almost-magical ways to enhance itself. Especially because the entire human race is basically one giant parallel data processing system, although communication leaves much to be desired. Moore law isn't going to help - silicon technology is very close to stagnation [1].
I don't think singularity is possible until potential ai-accessible artificial computation capability is at least several orders of magnitude higher than human brain's, which is impossible with silicon-based technology. Nothing to worry about.
One thing I look forward to is greater logical input into high-level decision-making. Emotions, egos, racism, discrimination, nationalism, aggression - all these are things that we just don't need any more.
I do think we need exceptional care when working on such a system but I also think the human mind of 50000 years ago leaves a lot to be desired when running in the 21st century. A mind that is ultra-rational would be a good adviser, helping to point out our blind spots and guide us to better thinking. We shouldn't have to learn from all mistakes every generation.
(I'm certainly equally wary of blind acceptance of whatever science and technology can create. A favourite article of mine is http://archive.wired.com/wired/archive/8.04/joy.html - has some very serious warnings and consequences, even about my abovementioned thoughts.)
Even if AI is harmless, there is still the risk of brain emulation. To me this is a greater threat, because if we don't know how a fully artificial mind would behave, we can get an idea of how an emulated human brain could. The problem is that human beings are potentially dangerous: they have volition, and they can enjoy destroying things. Basically they can be narcissistic psychopaths.
All it would take would be a human being rich enough to have his mind uploaded in a machine, and then this man will get greedy for more computing power, even if to do so he would have to take over the World and extinct humanity in its organic form.
Sounds very far-fetched, but as Hawking puts it: there is nothing in the physical laws that prevents a man-made device to perform as well or better than a human brain.
The computational requirements are overwhelming though. There is a short discussion of this in section VIII of http://www.fqxi.org/community/forum/topic/2077 (click the PDF link); the currently top-ranked supercomputer turns out to be roughly five million times too slow to simulate a human brain in real time.
I think the biggest problem as far as AI and automation is concerned is, what is left for us humans to do? At the moment, we work 42 hours per week (UK full time average), with many millions of people unemployed. Ultimately, our goal as a species, should be to build our infrastructure to be as efficient as possible. In an ideal world, we wouldn't have to go to work for 42 hours per week and many wouldn't need to work at all. We'd just enjoy ourselves 24/7. I think this shift from the 'work hard mantra' to an efficient world where this is much less 'manual' work to do (that includes many office jobs), is one that society, government and business underestimates.
Assuming strong AI arises, which I think likely, and given human nature I imagine people will build all sorts of AI's, 'evil' and 'good', though those terms are a bit subjective. Probably the likes of Kim Jong Un would love to have an army of self reproducing killer robots and Google will try to build nice ones and the US defence department will continue with drone like things. I agree with Hawking it could be a problem and hope the nice ones win out as they probably will. Fingers crossed.
Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent".
Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that.
I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!