The decisive moment for AI will come when a program can run a corporation better than humans. That's not an unreasonable near-term achievement. A corporation is a system for maximizing a well-defined metric. Maximizing a well-defined metric is what machine learning systems do well.
If a system listened in on all communication within a company, for traffic, sentiment analysis, who responds to whom how fast, and what customers, customer service, and sales are all saying and doing, it would generate more data than a CEO could ever process. Somewhere in there are indicators of what's working and what isn't. That may be the next phase after processing customer data, which has kind of been mined out.
If this starts working, and companies run by algorithms start outperforming ones run by humans, stockholders will put their money into the companies that peform better. The machines will be in charge.
> The decisive moment for AI will come when a program can run a corporation better than humans. That's not an unreasonable near-term achievement. A corporation is a system for maximizing a well-defined metric. Maximizing a well-defined metric is what machine learning systems do well.
Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
As soon as an AI CEO starts trying to maximize a single, well-defined metric, the humans working there will start finding ways to satisfy that metric at the expense of everything else. There is no single mesaure, no matter how well-defined, that will remain a good measure in perpetuity as long as humans are capable of interpreting it in the light of their own self-interest.
You can also make crappy decisions that increase profit in the short term but burn out your employees and run the company into the ground in the long term.
It will be painfully obvious within the decade that this is exactly what's happening right now. Corporations are AI optimizing a single metric. They only use humans so much as they need them to improve the metric. As they get better, automation improves and humans are needed less and less. There's a reason so many feel like cogs in a machine.
This already happened in 19th century. That's why we need the state (which is another AI when you think about it, with some hardcoded rules and periodical review (elections)).
People do a lot of things for short term pleasure that risk heavy long term consequences. Societies don't have the right to decide for others what their choices should be.
Maybe outlaw ice cream next? Or how about marijuana smoking? (Oops, already tried that.) How about loud music (long term hearing damage)? How about motorcycles?
> Societies don't have the right to decide for others what their choices should be.
If an AGI starts inventing new and wonderful ways in which we can destroy ourselves, and we are taken in by it, we will have to restrict that, it's non optional.
You can make the argument that, say, current hard drugs should be legal - but I don't think there's a way to defend the position that any possible future 'thing' should always be legal/permitted regardless of negative effect.
An awful lot of harm has been done to people via one group who are sure they know what is best for others, and thereby are justified in forcing it upon them.
In fact, likely much more harm than those with evil intent.
I tend to invest in companies whose products and behavior I like. It's not terribly surprising that they've done well - pleasing customers is good business. I've dumped stock in companies that began a business model of suing their customers to make money - and those companies (again unsurprisingly) tilted down.
Buy companies that customers love, sell companies that customers do business with only because they have to.
> A corporation is a system for maximizing a well-defined metric.
Interesting theory. My perspective is that once a corporation thinks of itself that way, it's already going downhill. Once a measure becomes a target, it ceases to be a good measure.
Now I assume you're imagining the well-defined metric to be profit, in attempt to dodge the aforementioned problem. However, even leaving aside the problem of short- vs long-term profit, I don't think that metric has a sufficiently well-defined link with actions to be optimized with anything we can build in the near term. There's too much "common sense" involved that, AFAICT, we still don't have any idea how to handle.
Ed: I guess I sort of did the thing the OP told me not to. But I still don't think the ability to phrase the problem of running a corporation as "optimize profit" makes it likely to be accomplished sooner than any other AGI application.
But surely a general intelligence would recognize that maximizing short term profit at the expense of long term profits has its own set of trade offs. And indeed, that common sense or “je ne sais quoi” would also reasonably indicate general intelligence.
I think the “running a corporation” problem is actually a great example of how to measure a general intelligence not only because it has well defined success criteria but also because of the very open ended and general means by which it must be accomplished. Running a corporation requires communication, setting goals, creativity, number crunching, and lots of soft skills like branding or understanding how consumers/customers will react to things. But there is nothing uniquely human about it except that it needs to be able to make money from humans. Presumably if I’m paying for some well defined service like Search Ads I don’t particularly care if the corporation is run by a robot.
I do agree though that there is a bit of ambiguity here on what “run a corporation” means. I assume that with a lot of thought I could write a very procedural script that somehow incorporates, purchases a property, and rents it through a property management group.
The book Blockchain Revolution covers this well. It postulates a decentralized autonomous organization (a bot, basically) that controls its money and can trade with other bots via smart contracts in cryptocurrency and self improve via machine learning. Its goal is to maximize profit, which it can do in various ways, both savory and unsavory to the human race.
If a system listened in on all communication within a company, for traffic, sentiment analysis, who responds to whom how fast, and what customers, customer service, and sales are all saying and doing, it would generate more data than a CEO could ever process. Somewhere in there are indicators of what's working and what isn't. That may be the next phase after processing customer data, which has kind of been mined out.
If this starts working, and companies run by algorithms start outperforming ones run by humans, stockholders will put their money into the companies that peform better. The machines will be in charge.
This is perhaps the destiny of the corporation.