Everyone has this same feeling that we don’t know what the hell is going on or what is going to happen. In troubled times there’s a buck to be made by confidently asserting you know what’s going on and what’s coming next.
One of the problems is that people treat experts on AI technology as if they were also experts on AI philosophy, which leads to poorly edited thought salads being published in a respectable context.
Just because someone understands variational autoencoders doesn't mean they have a clue about how the field of AI will look like 5 years from now, and it certainly doesn't mean they can anticipate the societal and political impacts of those technologies any better than the average (intelligent) Joe.
I don't know what 'AI philosophy' is, if it even exists, but if it does exist then poor quality articles might reflect the state of AI philosophy, or it might be just crappy articles.
I'm not comfortable with the idea that philosophy is intrinsically nebulous and poor quality, any more than observing amateur footballers being not very good would lead you to assume that there can be no such thing as a good footballer; of course there are, but there are a damn sight rarer than your local kids having a kick around for fun.
AI philosophy is its own academic discipline. Has been since around the 1970s. A lot of early academic experiments involving AI, e.g. ELIZA, most ALife research, Prolog-based expert systems, etc., can be best categorized (retroacively) as research into AI philosophy. No novel Computer Science principles were being explored; rather, what was being explored was the impact that certain novel applications of existing CS principles would have upon the world.
AI philosophy wasn't a very popular / widely-researched field, though, until recently, when AI ethics (ethics is considered part of philosophy) — and a specific subfield of that called "alignment research" — became something that a good number of philosophers became very concerned with.
Now there are many AI philosophers, employed not just in academia, but also in think-tank-like arms of AI technology companies like OpenAI (mostly because the AI tech companies know they're perceived as being irresponsible with how quickly they're iterating toward more-powerful AI, and so use employing AI philosophers as something like carbon credits to offset that perception.)
There is a lot of very good work done in AI philosophy; with many of the insights from alignment research specifically, being incorporated into the work that the AI tech companies are doing.
But none of this really "surfaces" in articles about AI that you might see floating about, because alignment research is really high-context — it's not really something you can write a fluff piece about; it's all stuff that requires knowledge of both philosophical (e.g. linguistics, decision theory) and computer science concepts to understand. Instead, all the normal articles about AI philosophy, are written by people doing amateur AI philosophy — usually resulting in them badly retreading the same ground that was already thoroughly explored in the 1970s.
Because many people write for virality and clicks and pieces with strong opinions usually have higher reach. “We don’t know” narrative doesn’t sell as well.
There are many people who write deep nuanced articles too, but since they have lower virality, we are usually exposed to the first type of shallower articles and it might seem that there are more of them.
The last few weeks I have been calling this a Crystal Ball moment. We really don’t know where we are with this. Unfortunately humans don’t do very well handling uncertainty and especially when the incentive to wing it and to be the next big AI player is so great. Greed is good according to some apparently.
On some levels this feels like our Prometheus moment, where we are able to create perhaps new forms of intelligence but we cannot at this point say what it will look like or what impact it will have, and we might certainly be ill equipped to handle the impact on society.
But you don't need to know to test yourself, and empirically, the author seems right, at least for image generation. The most specific your Lisa is, the less noisy images you get.
My intuition is the same as the author, I feel like we are at a limit (how much data can we really feed the models?), and most llm (except gpt4 apparently) are often too wrong to be useful (unless you need intern-level work). I think the next step is deepening the LLMs and specialize them (at least for code generation).
> Nobody knows how even the current generation of LLMs work (at the plumbing layer, sure, but the high-level behavior is a complete mystery).
i dont think thats fair to say tbh. clearly language has alot of patterns in it. so it should be possible to come up with a network that finds them. if the loss converges to something reasonable and the amount of training data is greater than what the network is able to memorize, then the only possibility is that the network learned whatever general patterns the data happens to have.
When I don't know something, I think of different possibilities, guess at probabilities, learn more, revise what's possible and not, revise probabilities, etc. Call it circling around the topic.
Other people, like this author, seem to pick one view and assert it as certain and incontrovertible, and wait for rebuttals to debate X or Y or Z externally rather than with themselves. Cann it a pinball style, relying on exterior factors to redirect.
It isn’t a crazy hypothesis to say products will require AI to perform specific tasks repeatedly. How many products that can be improved by AI today need a generalist chatbot trained on the entire internet.
Almost none, they all have a specific task that a specifically trained bot could do way cheaper.
The counter to this is AI is so cheap and easy people lazily use it for everything even though it is using .1% of its training data.
Whether or not a specialized solution could do it better or cheaper, it might not matter if the generalized solution does it well enough. A sufficiently powerful generalized solution can erase entire product categories and industries. Think of how many different devices and services were replaced by a single device we now all carry around in our pocket. Some industries just need to pivot slightly, others need to pivot hard, but it’s really hard to predict ahead of time.
The question isn't whether a generalist chatbot is needed. The question is whether a generalist chatbot is the easiest and cheapest way to go. And so far, looks to be the case.
>Nobody knows how even the current generation of LLMs work
we know exactly how they work. all they do is use statistical probabilities with limited randomization to synthesize new outputs from massive samples of inputs.
essentially a machine for plagiarizing spam.
gpt, and its general approach, will never have much value other than a few limited fields, because it can never produce anything that wasn't in its original input set and it is incapable of reasoning and analysis.
the world is already drowning is junk information. gpt has very limited commercial applicability because it is the opposite of what people actually want and need, which is a way to filter and analyze the massive pile of information they already have. gpt does the opposite and creates more junk information based on an analysis free synthesis of existing junk.
Even if this would be the case (it is not; gpt already already is producing value and replacing humans outside the domain you mention), people seem to love crap, spam, lies and chewing gum. TikTok and instagram are proof we want this type of content. We want copies of stuff we already like, in small packages and easy to consume. LLMs are great in providing that content.
>TikTok and instagram are proof we want this type of content
really inspiring take here. this is like saying fentanyl is a great product because people really like it. but i think you are wrong. these apps are mostly used by children and young adults who will grow out of them and turn away from using digital experiences in general because they are addictive and harmful. that is not a good thing for the industry in the long term.
Of course it is not good for anything and in my view it should be forbidden outright, but plenty over 18 want to consume this crap and don’t grow out of it until middle aged.
But also ‘news’; people like sensationalist copy/paste news and they don’t grow out of that. Not sure how many people watch fox, but I would be surprised if most of that is not already written by AI.
Nobody knows how even the current generation of LLMs work (at the plumbing layer, sure, but the high-level behavior is a complete mystery).
Yet there is an endless stream of opinion pieces telling us what future LLMs won't be able to do.
Here's the most truthful 'article' on AI you will ever read: "We have no idea where we are, and we have no idea where we are going."
I struggle to understand why this is so incredibly hard to admit for some people.