I'll tell you my vision which is kinda long term - like 20-30 year from now on. I think the future is that everyone will have their own personalized AI assistant on their phone. Internet as it is will be mostly useless because only robots will be able to wade through the generated shit ocean and the next generation will see it as the current see the TV - unimportant old boring low-entropy data which is not entertaining anymore.
There will be a paradigm shift where our-customers-are-ai apps appear and most stuff will need to have an API which makes AI able to connect and use those services effortlessly and without error because who don't want to tell his assistant to "send 5$ to Jill for the pizza"? There will be money in base AI models you can choose(subscribe to) and what it can and cannot do for you. It will still be riddled with ads and now it's your personal assistant who can push any agenda to you.
Operation systems will become a layer under the assistant ai.
You will still talk on/to your phone.
I guess free software will be more important than ever.
Free ai assistants will be available and computing power will be there to run it on your phone but all the shit we've seen with open Vs closed source, Linux Vs Windows, walled gardens whatnot will go another round this time with free open public training data assistants Vs closed-but-oh-so-less-clumsy ones.
Security problem will be plenty like how to hide your AI assistant custom fingerprint? Authentication and authorization system for AI? How much someone else personal assistant worth? How to steal or defend it?
An LLM/AI companion based on your ledger, that grows and evolves with you, acting as a sort of Socratic daimon, a psychologic twin, assistant, advisor, and sparring partner. Like a journal that talks back and thinks with you. It'd need to be only personally accessible and safe/private. Biometric access and local running could solve this.
This is the dream. That said - Gen AI on its own wont get us to helpers/Daemons.
Right now, if you want to build something like a daemon you will need multiple agents. When a task needs to be delegated, the central control system needs to be able to spin up a few processes and get them going.
You can do this right now. You can create an ensemble of personalities/roles (jr dev, sr dev, project manager) and have them plan.
They do a good job, if you are monitoring it. You can break up the plan into chunks, spin up more instances, distribute chunks and have those instances work on it.
Sadly this cant happen, and I think these are fundamental limits to Generative AI. Right now personas just go ahead and "pretend work" - they say we "I am now going to go and come up with a project plan".
You have a dependency where output from all prompts must be actionable, or verifiable. If output is off, then it snowballs.
This is not a tooling, or context window issue. This is a hard limit to what Generation can achieve. Its impressive that we got this level of emergence, but when you look at output in detail, its flawed.
This is the verification issue, and verification (or disproving something) is the essence of science.
Maybe something can be added to it, or entirely new structures can be built - but generation on its own will not take us over this threshold.
Right now, LLMs are more like actors. They are very good actors, but you dont get your prescriptions from actors.
Exactly! Most will have a name given by its owner, a pet of sort but much more in function. You will need to reach a certain age to get it, the beginning of adulthood.
People will mourn them and there will be real world AI cementeries.
Diffie Whitfield envisioned an internet highway where every computer is connected to every other in 1974 and then went on to find out how to do that communication in secure channels.[1]
I think my vision is quite tame and deductible from the current situation compared to his.
The more personal value it holds, the more risk it has. Imagine gaining access to this digital daimon of someone, it'll be like accessing their most inner thoughts.
The upside however could be exponential personal growth. I've seen many examples of real life twins that basically live as a superhuman individual. E.g. the Liberman twins. This tech could be a digital equivalent of that.
In outright totalitarian countries government access to your agent will be mandated.
In more democratic countries we'll see lots of suits and court cases over getting access to these agents and assigning intent to your IRL actions based on your 'thoughts' with the AI.
> You will need to reach a certain age to get it, the beginning of adulthood.
Not a chance will whatever capitalist figures out how to do this allow a measly cultural restriction keep them from pushing that age deep into childhood.
Anything long term is meaningless to think about, besides just the fun.
AI on it's current track is going to so radically reshape society that it will be totally unrecognizable to people today. Society in it's current form makes no sense when there are smarter and cheaper non-human workers available.
There is a huge number of people - hundreds of millions - employed today that could quite literally be replaced with small shell scripts. Yet they are not being replaced. There is no reason why AI will not similarly fail to bring forth the revolution.
Do you ever wonder if that's because of the growing up part and not the phone part? Growing up for most people tends to be pretty fun and exciting compared to adult life. And with or without phones, the 80s and 90s were a very different time when it came to kids being able to go out and play outside on their own. Move to a country where it's common for kids to play outside and you'll see that times aren't much different even though kids all have phones. In a way, it's easier because now you're just one text and location share away from meeting up with them.
Growing up in the time where Palm and Windows CE were a thing, I knew having a capable pocket sized computer was an inevitability, and couldn't wait for it. And it's great: I would fucking hate not having modern mobile banking, communications tools, maps, calendars and the ability to look anything up immediately.
The internet was more fun pre-smartphone, but that's because of the demographic change brought on by a new Eternal September, not the existence of mobile computers. We went from the internet being the domain of interesting, creative people who made things for fun to "basically everybody."
If you grew up before phones then don't worry. You man not live this long.
But seriously there will be some communication device or chip implant. And 'personal' assistant will be just a presentation of a software running 'in the cloud' and controlled by big corporation. With no privacy at all. Which makes it a spyware through which government can track and control crowd.
That's sounds like dystopia, but having a tracking device on almost every person was unimaginable not so long ago. Today it's a norm. Big corporations have and sell your location in real time.
> the next generation will see it as the current see the TV - unimportant old boring low-entropy data which is not entertaining anymore.
Don't you mean high-entropy data? High entropy data would be less orderly, more compressible, and have a lower signal to noise ratio ... like TV shows compared to a textbook.
Low entropy data is what's more compressible. High entropy means unpredictable, which could mean high noise (like TV static which is incompressible) or high signal as GP intended.
Thank you. I just learned about Shannon Entropy and that it grows with information content. The negative sign in the formula is due to the negative log of the normalized population probability.
Yes. You can also take the log of the reciprocal probability (or "luck") and then you don't need the negative sign. This value is the expected number of trials you would need for the outcome to occur once. I find this presentation a bit more intuitive. See also Boltzmann entropy S = k log W which takes this form.
Meanwhile, the common folk will sit in the iCubicle homes with just AirGlasses and subscriptions for everything, including the virtual furniture and the Fortnite Metaverse as the then-facebook to glue it all together.
I'm honestly a little excited about our Snow Crash future.
Do you think that this will all function as one giant homogenizing force at the societal level? The AIs will all be trained on the same data, and so will have the same opinions, beliefs, persuasions, etc. It seems like everyone having AIs which are mostly the same will maximize exploitation, and minimize exploration, of ideas.
Is that any different than media before the internet consumed it all? Media consisted of a couple of news stations, a couple of TV stations and that was mostly it.
I'm also not sure that the recent broadening of media has been a net benefit to society. Look at the degree of polarization in recent years. At a certain point heterogeneity is no longer a societal good.
No, you're correct, media is also a homogenizing force. Regional accents have disappeared, for example. But I would argue that even the political polarization is homogenization, not evidence of heterogeneity, because the variety of the middle was eliminated, leaving the consolidated extremes.
My point is that personal llms will be an even greater force along this same line.
There are a million different beliefs, opinions etc in the corpus LLMs get trained on and they can predict all of it. It doesn't have to have the same opinions.
Sure, but the probabilities of those beliefs won't be the same as each other, and they will be the same between all users, so that doesn't address my point.
again that's up to who's training the models and/or the user. Bing is GPT-4, same exact pre-training but it sounds nothing like chatGPT-4.
LLM probabilities are dynamic. They change based on context. If you want it to behave in a certain way then you provide the context to engineer that.
a particular belief system being most present in training doesn't mean an LLM will always shift probabilities to that system. Such prediction strategies would be awful to fulfil its training objective. Getting it to shift to even the most niche belief is as simple as providing the context to do so.
> again that's up to who's training the models and/or the user.
True, but training is expensive, I imagine that only a few actors will train the popular LLMs, exactly as we are seeing today.
> LLM probabilities are dynamic. They change based on context. If you want it to behave in a certain way then you provide the context to engineer that. ... Getting it to shift to even the most niche belief is as simple as providing the context to do so.
I thought so too, until I tried to get it to agree that 1 + 1 = 3. It would not do that, no matter how much context I provided, likely because the probabilities in the underlying data were so skewed.
If your interface to the internet is a locally running open source ai assistant, couldn't you just tell it to strip out ads from the information it returns? It could change the internet economy some.
> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications.
Not clear to me if the author actually uses LLMs to do meaningful work, or is speculating about how they might be used.
I've written about 2500 lines of F# for the first time in the past 1.5 weeks using ChatGPT-4 to guide me. It has been an constant back and forth, iterative process. My decades of development experience factored in heavily to guide the process. I would've have been at maybe a quarter the progress without ChatGPT, or given up entirely on F# as my language.
I don't think that iterative aspect will be eliminated any time soon for AI-supported complex, creative processes. It's no different from tweaking inputs to a Photoshop filter until your experienced brain decides things look right.
To that end you need to know roughly what looks "right" before you use an LLM. This will all become second nature to the average developer in the next 5-10 years.
Bit of an aside, but I wonder if the rise of LLMs will lead to new programming languages being much slower to be adopted.
Like you said, you might have given up on F# without ChatGPT assistance, and the main way ChatGPT is able to help with F# is because of all of the example code it's been trained on. If developers rely more and more on LLM aid, then a new language without strong LLM support might be a dealbreaker to widespread adoption. They'll only have enough data once enough hobbyists have published a lot of open-source code using the language.
On the other hand, this could also leading to slowing adoption of new frontend frameworks, which could be a plus, since a lot of people don't like how fast-moving that field can be.
I heard somewhere that ChatGPT is surprisingly good for human language translations even though it's not specifically trained for it. There seems to be just enough examples of e.g. Japanese that Japanese researchers use it to translate papers. I suspect that's largely true for programming languages too. I've had great success working with it in Clojure, even though there's relatively little published code compared to more popular languages.
ChatGPT is pretty good for translations of Japanese into English. It’s English to Japanese translation tend to sound somewhat stiff/formal/machine-generated, although it’s less prone to hallucinations than DeepL for larger texts. I expect this is because it was trained on a much larger corpus of English language texts than Japanese ones, which means the problem is not intractable.
Wouldn't you just need to publish a Rosetta stone type translation for it to be able to digest the new language fully? e.g. here is how you do this in python and here is how you do it in this new language
The crazy thing that a lot of people don’t realize is that all of that data generalizes to anything new you can throw at it. As long as there’s enough space in the prompt to provide documentation it can do it on the fly but you could also fine tune the model on the new info.
It could go the other way. LLMs might make porting code from one language to another easier which would speed the adoption of newer and more niche languages. And the future of documentation and tutorials might be fine-tuning an LLM.
I've also wondered this - including if we might see a breed of 'higher level' languages (i.e. much higher level than Python) which can then be 'AI compiled' into highly efficient low level code.
i.e. the advantages of an even-higher-level python that's almost like pseudo-code with assembly-level speed and rust-level safety, where some complexity can be abstracted out to the LLM.
I disagree. Chatgpt is helpful here because f# is a paradigm shift for this otherwise experienced programmer. The programmer probably knows juuussstt enough f# to guide the llm.
I mean why is f# the goal, and could we write a better f# with the use of AI.
As an example, why not write in f# and let an 'AI-compiler' optimise the code...
The AI-compiler could then make sure all the code is type safe, add in manual memory management to avoid the pitfalls of garbage collection, add memory safety etc - all the hard bits.
And then if we gave the AI-compiler those sort of responsibilities, then we can think about how this would impact language design in the longer term.
None of this is with current generation LLM's, but might be where we end up.
In current generation language there is definitely a trade-off between language productivity (i.e. speed of writing) and features such as speed and memory-safety.
So far we haven't been able to close this gap fully with current compilers and interpreters (i.e. python still runs slower than C).
It seems like that gap could be closed through, for example, automated refactoring into a rust-like language during compilation, or directly into more-efficient byte-code/ASM that behaves identically.
And surely if that is a possibility, that would affect language design (e.g. if you can abstract away some complexity around things like memory management).
To this day i am still wondering what kind of code people write that chatgpt can possibly help with. All my attempts lead to garbage and i would spend more time fixing the output of the chat bot than writing the actual code. It does help with some documentation. But even that has glitches.
No one uses it to generate code. Really. Talk to people who actually use it and listen to what they say… they use it to help them write code.
If you try to generate code, you’ll find it underwhelming, and frankly, quite rubbish.
However, if you want an example of what I’ve seen multiple people do:
1) open your code in window a
2) open chatgpt in window b (side by side)
3) you write code.
4) when you get stuck, have a question, need advice, need to resolve an error, ask chatgpt instead of searching and finding a stack overflow answer (or whatever).
You’ll find that it’s better at answering easy questions, translating from x to y, giving high level advice (eg. Code structure, high level steps) and suggesting solutions to errors. It can generally make trivial code snippets like “how do I map x to y” or “how do I find this as a regex in xxx”.
If this looks a lot like the sort of question someone learning a new language might ask, you’d be right. That’s where a lot of people are finding a lot of value in it.
I used this approach to learn kotlin and write an IntelliJ plugin.
…
…but, until there’s another breakthrough (eg. Latent diffusion for text models?) you’re probably going to get limited value from chatgpt unless you’re asking easy questions, or working in a higher level framework. Copy pasting into the text box will give you results that are exactly as you’ve experienced.
(High level framework, for example, chain of thought, code validation, n-shot code generation and tests / metrics to pick the best generated code. It’s not that you cant generate complex code, but naively pasting into chat.openai.com will not, ever, do it)
That matches my experience. It's a sort of shortcut to the old process of googling for examples and sifting through the results. And those results, I didn't typically cut and paste from them, or if I did, it was mostly as a sort of a scaffold to build from, including deleting a fair amount of what was there.
Many times it works really well, and it surfaces the kind of example I need. Sometimes it works badly. Usually when it's bad, going to the google/sift method has similar results. Which I guess makes sense, it couldn't find much to train on, so that's why it's answer wasn't great.
One area it works really well for me is 3rd party apis where their documentation is mostly just class/function/etc. ChatGPT generally does a good job of producing an orchestrated example with relevant comments that helps me see the bigger picture.
Me too. As someone who used to be a dev but hasn't written code professionally in twelve years or so, it was such an amazing accelerant. My iteration loop was to contextualize it (in English and in code), ask how to do a thing, look at its response, tweak it, execute, see what happened, alter it some more.
The fact that it usually had errors didn't bother me at all -- it got much of the way there, and it did so by doing the stuff that is slowest and most boring for me: finding the right libraries / functions / API set up, structuring the code within the broader sweep.
Interesting side note: un-popular languages, but ones that have been around for a long time and have a lot of high-quality and well-documented code / discussion / projects around, are surprisingly fecund. Like, it was surprisingly good at elisp, given how fringe that is.
With GPT-4, you can often just paste the error message in without any further commentary, and it will reply with a modified version of the code that it thinks will fix the error.
- A rough crash course in F#. I'll say "what's the equivalent in F# of this C# concept?". It will often explain that there is no direct concept, and give me a number of alternative approaches to use. I'll explain why I'm asking, and it'll walk through the pros/cons of each option.
- Translating about 800 lines of TypeScript JSON schema structures to F#. A 1:1 translation is not possible since TypeScript has some features F# doesn't, so ChatGPT also helped me understand the different options available to me for handling that.
- Translating psuedo-code/algorithms into idiomatic F# as a complete F# beginner. The algorithms involve regex + AST-based code analysis and pattern matching. This is a very iterative process, and usually I ask for one step at a time and make sure that step works before I move onto the next.
- Planning design at a high-level and confirming whether I've thought through all the options carefully enough.
- Adding small features or modifications to working code: I present part of the function plus relevant type definitions, and ask it for a particular change. This is especially useful when I'm tired - even though I could probably figure it out myself, it's easier to ask the bot.
- Understanding F# compiler errors, which are particularly verbose and confusing when you're new to the language. I present the relevant section of code and the compiler error and 90% of the time it tells me exactly what the problem and solution is; 5% of the time we figure it out iteratively. The last 5% tends I have to stumble through myself.
- Confirming whether my F# code is idiomatic and conforming to F# style.
- Yes it makes mistakes. Just like humans. You need to go back and forth a bit. You need to know what you're doing and what you want to achieve; it's a tool, not magic.
Note: this is the commercial product, ChatGPT-4. If you're using the free ChatGPT 3.5, you will not be anywhere near as productive.
I used chatgpt4 to generate python code which generates c++ code for a hobby project of mine using a library I've never used before. The iteration speed is ridiculously good and no one in any of the IRC or discord channels I visited pointed me even in the general direction of such a simple solution.
Programmers[1] have a complexity bias that interferes with the idea that LLMs can write useful code.
I had a problem last week where I wanted to extract sheet names and selection ranges from the Numbers app for a few dozen spreadsheets. ChatGPT, came up with the idea of using Apple script and with a but of coaxing wrote a script to do it. I don't know ApplesScript and I really don't want to learn it. I want to solve my problem and its 10 lines of AppleScript did just that.
We're nowhere near LLMs being capable of writing codebases, be we are here for LLMs being able to write valuable code because those concepts are orthogonal.
I am confused - the "code" you described is likely a google search away. Well I mean google has become useless but when it worked it was able to find such stuff in one search. So really all I am getting is that gpt is a better google.
It’s good at a beginner, early intermediate level when you need help with syntax and structuring basic things. It’s an excellent helper tool at that stage.
But it’s obvious outside of jr dev work and hobby projects there’s no way it could possibly grasp enough context to be useful.
Stage? A lot of developers don't realize that they're all the same personality type which is good at particular things. LLMs give everyone else this advantage. You just don't realize it yet because you were never aware of the advantage in the first place.
I am not, but there's a tendency among those types that categorise people into narrow sets that they can understand. Also your statement doesn't much make sense. Being a good developer means you understand a wide range of issues, not just spelling in a language and adding if statements. The combination of personalities vary wildly. To be fair, LLMs if anything, will help developers become better managers, simply because developers understand what needs to be done. Instead of decyphering what someone meant by requesting a vague feature, you can ask a statistical system - an ai as some call it - what the average joe wants. And then get it done.
There are three types of intelligence: intuitive, cognitive and narrative.
Tech has, for years, seen people with cognitive/narrative intelligence as the people who are actually smart.
LLMs help the intuitive people reach the level of the cognitive/narrative people. Cognitive/narrative people can't really understand this in the same way the intuitive people are bad at syntax or database structure. The steamrolling will be slow and merciless.
Could you give a concrete example of where ChatGPT is likely to provide a competitive advantage to intuitive people with weak cognitive/narrative capabilities?
Did some ETL with Python. ChatGTP got it right 99%. And, remarkably understood a public API that I was feeding from, which used 2-letter abbreviations.
If you are trying to do something with an api that you have no experience with it will get you up and running quickly. e.g. How do I get all kubernetes configmaps in a given namespace older than n days in go? It gives you the bones about how you create and configure a client and query kubernetes to get the information that you are looking for. It's much quicker than googling and parsing a tutorial.
I'm the author, and I don't disagree with this at all - I do use LLMs pretty meaningfully in my day-to-day engineering work, and I definitely agree that they hold a ton of promise in situations like the one you mentioned. To be clear, I'm very much bullish on the applications of LLMs to e.g. coding! The point I was trying to make there was just that for _certain_ tasks, the process of chatting with an LLM is, by nature, less precise and more arduous than a purpose-built UX. By analogy, we might have "describe-your-change" functionality in Photoshop, but its no replacement for all the other pixel-perfect editing functionality that it provides, and I'd struggle to imagine a world where Photoshop is ever _replaced entirely_ by a chat UX
Ah haha I am going through the exact same experience but picked up F# about 4 months ago and ChatGpt has been an absolute godsend. There is a whole category of blockers in learning that have been eliminated.
I feel like such a jerk for not wanting to use chatgpt for all that. It's not just mild paranoia about feeding it input I prefer be kept private but I just don't enjoy having someone else do half the thinking for me I guess.
I mean, I don't like intellisense either (but simple autocomplete is fine). Perhaps it is because I only code to help with my job, I don't get paid to have a lot of good quality code output.
I think I would have felt this way when I was younger. But after writing code for over twenty years, I'm very happy to let computers do the boring half of the thinking, so that I can think about more interesting things.
It is not intellectually stimulating for me to think about what the syntax for a dict comprehension is, or what exactly I'm supposed to do to map over the values of an array in javascript without screwing it up, or any of a million other kinds of minutia. Computers know the answers to these uninteresting questions.
That's what's been so striking for me -- the stuff that is fun and playful for me I get to do; and a bunch of stuff that I hated I can now offload, with the net result that I get to work at the level of abstraction that is most interesting and most human, vs acting like a human robot trying to unearth documentation, API examples, etc.
There's no question about this being the right way to use it for me, but I wonder if this could introduce something bad for someone just starting out, who hadn't first got all the reps with the drudgery over decades? Still mulling on that.
Yeah, I've said a bunch that I'm worried about people starting out from scratch. So much of what I do after I get responses from LLMs is best described as "taste", but how would I have developed that taste without the years of slogging through?
But I also think this is probably just a classic geezer's concern about kids being on my lawn, and it will probably work out just fine for these youths :)
Eh. I think the peak of computer ability was GenX. I’m not in that group but made up for it with sheer intensity. If you grow up on a phone you’re not getting any of the mental modeling to get close to what the machine is doing and you’re only going to be able to work with the limited tools of that level until you deliberately dive deeper.
I think both you and the GP are onto something important. I am one of the vaunted Gen-Xers (this is the first time someone has said anything nice about us, so thanks) and there is something to the idea that my understanding of systems goes down to assembly language (fully) and even CPU architecture (very sketchily) all the way up to, like, web frameworks. So for the stuff I'm usually doing, I understand several layers up and down, and that often seems valuable.
This strikes me as the essence of expertise, the difference between talking to someone who's read a book or a review paper, vs someone who is a real expert at that thing. You can be quite competent and insightful from having done a modest amount of reading and thinking, but if you're seeing the Matrix, and know how all these forces are interacting with each other, that's just a whole other level. It makes an impression when you brush up against it.
However: it always weighs on me how this is all a matter of framing. Like, I don't know the electronics. There's a level of metal that I don't get. Some of the really high-level stuff is now beyond me, too. I catch myself saying "I should really take x {weeks, months, years} and really grok that." And yet my actual experience suggests this is a mirage.
More briefly: there are always more layers for a fuller understanding. It's hard to see how many of them are really useful. Maybe the kid who is 10x better than me at LLM collaboration will benefit more than from having a deeper stack. It's interesting to ponder how these different consequences will play out.
FWIW I'm a (geriatric) millennial, and we were totally taught all that same stuff. And yeah, it totally feels important to me to have this mental model up and down the layers. But I also do wonder whether we're just wrong about that, merely falling prey to nostalgia.
My messy answer is that for many projects it's either neutral or an actual hindrance - I have to remind myself to pragmatically not care about stuff that won't matter for years at the project's expected growth rate - but for other projects it's very useful. I think the way to go is to seek out those kinds of projects, in order to avoid frustration all around.
I dunno, I've used all kinds of languages, from the ones people often consider lower quality to the ones people often consider higher quality, and they all have something like this.
And maybe there's some as yet unexplored design space out there for which this isn't true, but my prior is that we're actually just circling around some essential complexity at the core of the problem, which will never be eliminated.
Don't beat yourself up. As a mathematically-oriented software engineer I also have little to no interest in using LLMs to help with coding. It feels like the people that do are the builders, i.e. interested in quickly knocking up something that works most of the time. Which is great from a business perspective. I can't help but see my programming output as an artistic expression of my thought, and therefore find it difficult to give up the hands-on construction to others (be it LLMs or juniors).
I think the division is along a different axis than this one (or probably it's along multiple axes).
I've always been more of a fastidious crafter than a "just get it built" person, but I also struggle with a blank page. I thrive on editing more than writing. Since forever, I like to get out something that works or mostly works, and then start carving on it until I like it.
LLMs have been helping me get some ink on the page, but very little of what they suggest ends up in the final product.
I’ve observed for a long time that there’s a lot of value in making a seed, a starting point, for the thing a group needs to create. And that’s exactly the reason - new text is harder than criticism of the old.
Yep! But some people do seem to thrive more with a blank page.
Supposedly Tom Robbins writes books entirely by putting one word after another starting with the first one and finishing with the last one. I don't know if that's apocryphal, but I do think that's closer to the process for some people.
But if I were a writer, I'd be squarely in the "get out a first draft, it will be like pulling teeth, but just get something down; then you can do the fun part of revising and polishing".
Not especially more biased than GGP's claim that people who use LLMs as coding assistants are "builders, i.e. interested in quickly knocking up something that works most of the time".
Could you expand on the comment about "algebraic designs"?
When I interact with an LLM it feels like I'm speaking to someone who lacks inherent conceptual understanding, but is doing their utmost best to bullshit me. This is problematic when it comes to algorithmic thinking.
I sometimes need to solve problems that involve tweaking existing algorithms in somewhat novel ways. For example "k-medoid clustering but with various capacity constraints on clusters", or "contraction heirarchies but accounting for varying vehicle dimension restrictions".
I love collaborating with smart colleagues to come up with ideas to solve these problems. Seeing chat-gpt fail to solve very simple traveling salesman type problems gives me little confidence that it can do novel conceptual tihnking.
EDIT
To test my own claims, I just tried asking Chat-GPT to make this modification to k-means clustering. It produced something that will give a result but will likely not be very optimal. When prompted to refine the solution, it failed to understand the issue and instead proposed the exact same solution just with a bit more implementation detail.
User
Suggest a modification to the k-means algorithm where the number of elements assigned to each cluster is limited to a pre-defined maximum value.
ChatGPT
To modify the traditional k-means algorithm to limit the number of elements assigned to each cluster to a pre-defined maximum value, you can create a variation of k-means called "k-means with cluster size constraints." This modification ensures that no cluster exceeds the specified maximum size while still optimizing cluster centroids based on the sum of squared distances. Here's a high-level overview of the modified algorithm:
1 Initialize centroids: Start by randomly selecting K initial centroids, where K is the desired number of clusters.
2 Assign data points to clusters: Assign each data point to the nearest centroid based on Euclidean distance, just like in the traditional k-means algorithm.
3 Update centroids: Recalculate the centroids of each cluster by taking the mean of all the data points assigned to that cluster.
4 Check cluster sizes: After updating the centroids, check the size of each cluster. If any cluster exceeds the pre-defined maximum size, proceed to reassign some data points from that cluster to other clusters. Here's how you can do this:
Identify the cluster(s) that exceed the maximum size.
For each over-sized cluster:
Sort the data points in that cluster by their distance to the cluster centroid in ascending order.
Start reassigning the farthest data points to other clusters (the ones with the smallest distances) until the cluster size constraint is satisfied. Update the centroids of the affected clusters accordingly.
5 Repeat steps 2 and 3: Continue iterating between data point assignment and centroid updating until convergence, or until a predefined number of iterations have been reached.
6 Termination: The algorithm terminates when either the cluster assignments do not change significantly between iterations or after a fixed number of iterations.
Here are a few things to consider when implementing this modification:
You will need to experiment with the order in which you reassign data points from over-sized clusters to minimize the impact on the overall clustering quality.
You should set an appropriate stopping criterion to prevent the algorithm from running indefinitely.
Be prepared to handle situations where it is impossible to meet the cluster size constraint while maintaining good cluster quality. In such cases, you may need to relax the constraint or consider alternative clustering methods.
This modified k-means algorithm with cluster size constraints allows you to control cluster sizes while still finding meaningful clusters in your data.
User
Your proposed solution has an issue that points might be reassigned even if there are clearly no other suitable clusters. Please tweak the solution to avoid this issue.
ChatGPT
To address the issue of potentially reassigning points even when there are no other suitable clusters, we can modify the approach by introducing a priority queue or heap data structure to efficiently manage the reassignment process. Here's an improved version of the k-means algorithm with cluster size constraints:
1 Initialize centroids: Start by randomly selecting K initial centroids, where K is the desired number of clusters.
2 Assign data points to clusters: Assign each data point to the nearest centroid based on Euclidean distance, just like in the traditional k-means algorithm.
3 Update centroids: Recalculate the centroids of each cluster by taking the mean of all the data points assigned to that cluster.
4 Check cluster sizes: After updating the centroids, check the size of each cluster. If any cluster exceeds the pre-defined maximum size, proceed to reassign data points from that cluster to other clusters. Here's how you can do this:
Identify the cluster(s) that exceed the maximum size.
For each over-sized cluster:
Create a priority queue (or heap) to store the data points in the cluster, ordered by their distance to the cluster centroid in ascending order.
Start reassigning the farthest data points to other clusters (the ones with the smallest distances) until the cluster size constraint is satisfied. Update the centroids of the affected clusters accordingly.
Continue this process until either the cluster size constraint is met or there are no more suitable clusters to reassign points to.
5 Repeat steps 2 and 3: Continue iterating between data point assignment and centroid updating until convergence, or until a predefined number of iterations have been reached.
6 Termination: The algorithm terminates when either the cluster assignments do not change significantly between iterations or after a fixed number of iterations.
This modified approach ensures that points are only reassigned to clusters that are suitable and have room to accommodate additional points while maintaining the overall quality of the clustering solution. The use of a priority queue helps prioritize the reassignment of points to clusters with the least impact on their quality.
Additionally, you should set an appropriate stopping criterion to prevent the algorithm from running indefinitely, and you may need to handle situations where it is impossible to meet the cluster size constraint while maintaining good cluster quality by relaxing the constraint or considering alternative clustering methods.
How is what you're doing mathematical? I mean you can call CS "math", but then I don't know what you mean by more "mathematical." Traditional cs algos are not what I term as more mathy from my pov. Maybe you can call stats "mathy" but this is more applied math and not too different from what "builders" do.
Also what you're doing here is asking chatGPT for the answer. chatGPT is more effective via collaboration. Meaning instead of shoving the entire problem down it's throat and asking it to solve it, you ask it for advice. Ask it for bits and pieces of things.
To some extent I agree with you, in that all software engineers have to think about refactoring and time complexity, involving a logical or mathematical style of thinking. However, there's definitely a spectrum of how distinctly mathematical the work of a software engineer is.
A lot of front-end development for example does not require familiarity with any algorithms, formulae or mathematical structures. You might need to reason about large systems and have a rough idea of when hashmaps are useful, but the bulk of the work is constructing an interface according to functional business requirements. I frequently see comments here along the lines of "why am I being interviewed about algorithms when I'll never use them in my job".
A more mathematically oriented developer may be in the business of modelling and predictions. They may be confronted with a novel real world problem involving traffic, or trading, or electricity networks, that potentially no one has tried to solve before. They may be required to find a mathematical structure that closely approximates real world behaviour, implement that structure via code, and deploy a continuous model which allows their client to analyse and project.
Of course, you also have academic mathematicians using software like Maple or SageMath to assist with their research. This is another level more mathematical. Perhaps what you're getting at is that people can ask ChatGPT questions like "write me some Sage code to get the Delaunay triangulation of this set of points". I totally agree that it can probably do well at these tasks.
Modelling stuff sounds like data science. It's a term they often use and it sounds very much like the same deliverables you mentioned. I've never seen a data scientist term themselves as more mathematically oriented. Also let's not play around, what you actually meant to say mathematically "superior". That much is clear. Sounds like you know stats and you think that makes you more "mathematical".
You also talk about things like traffic. Modelling traffic is mathematical? Sounds like a simulation to me. Man take a look at GTA. That game is almost entirely made by builder engineers creating simulations. It's the same shit and likely far more advanced then what any data scientist can come up with.
Anyway from your example and from what Ive seen it sounds like you're still doing the same thing. CS algorithms. You're just using algorithms that aren't likely very popular or very specific to data and stats. But adjusting stuff like k-means clustering still sounds like regular cs stuff to me.
There's no point in calling it more "mathematical" because it's not. The builder engineer who wrote all the systems in GTA or even say red dead redemption use a ton of "math" even and they don't term themselves more "mathematical" even though their simulations are likely more complex than anything you will ever build.
That's why when you called your self mathematically superior (again don't deny this.. we all know what you really mean here) I thought you were talking actual math. Because if you looked at a math equation it doesn't look anything like an algorithm. Math equations are written as a single expression. Math equations model the world according to a series of formula. It's very different to a cs algorithm.
Mathematical oriented programming involves largely the same thing and using algebras of mathematics.
If you're not doing this just call it data science instead of trying to call yourself more "mathematical". If you truly were more mathematically oriented you would know what I'm talking about.
Geeze some guy writing "models" and doing some applied math+stats like what every other freaking programmer out there is doing and he calls himself more "mathematically oriented."
Statistics isn't my strongest area. But I do have a doctorate in quantum information theory, so I have some idea of what it means to be mathematical.
Data science definitely forms part of what I do, as my employer stores a lot of data that we use to estimate various parameters. But there's also work on creating bespoke routines for solving vehicle routing problems in niche domains, which I wouldn't really class as data science.
Thanks for the discussion, anyway. I'm not interested in being insulted.
No one is interested in being insulted. But you only feel insulted because what I said is 100 percent true.
"Bespoke routines for vehicular routing problems" lol. I mean phrases like that reveal what you think of yourself as.
You're writing simulations. That's all. "Bespoke" lol. And those simulations have lower fidelity then a video game like GTA which likely does traffic at higher levels of fidelity and real time with a renderer.
I have a doctorate in mathematics. Prior to that I've done work in cs. Doesn't mean shit. I don't name drop that crap to pretend to be superior.
An LLM is going to be most helpful for topics that came up a lot during training. What you're discussing doesn't sound like it falls into that category.
I'd agree with that. I'd even go a step further and say that most of my work and things I'm interested in coding don't fall into that category. I have absolutely nothing against people who do find it useful, but I'm keen to reassure user badrabbit that they're not a jerk for not being interested in using LLMS.
I love to write code, I love to modify existing code too.
I do not love to read and then fix code after someone all the time. With ChatGPT I have to read then understand then fix code after ChatGPT every time.
Also, I do not love to fix code that often contains hallucination.
Second, have you tried pasting any syntax errors you get back into the chat window? Usually, GPT will try to fix them.
Third, you can have GPT write a function for you and then write a different version of the function using a different algorithm along with a test (e.g. a fuzz test) to check whether two functions produce the same output given the same input. This makes it less likely that an error will slip through because both algorithms would have to be produce the same incorrect output for the same input.
Second, your fix is to verify code generated by LLM and then play with it trying to find a way to fix it. I`m quite sure that I will spend less time and less mental power by writing code in the first place.
Third fix is to play with chatGPT more, and read more code generated by chatGPT trying to find errors in it.
What happened with
"reading code written by someone else is harder than writing your own code"?
Yeah, for me it has been docker and kubernetes. I have found it to be something of a killer app for spinning up on anything with a bunch of annoying little details, that I'm not already super familiar with.
Make, docker, kubernetes, all fit this pattern. Heck, maybe I won't be so down on autotools if I run into it again now that I can attack it with LLM support.
Or html / css; I haven't written any of that in the LLM era, but maybe I'd enjoy it more now.
The llm effect may enable people to use languages that are harder to write, but produce a more maintainable and/or performant product. Maybe it's finally time to learn rust or ocaml!
A lot of this seems like a nerd’s fantasy. I share these fantasies / desires, but I don’t think its a sober, realistic take on what’s most likely.
> Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes.
Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”. This is a UX pattern that I hate but has proven a winner across many apps lately - e.g. TikTok. Streaming music still allows you to build playlists of specific songs, but auto-built radio stations are filled with a suspicious amount of whatever the major labels are pushing this quarter. Netflix/etc has shockingly fuzzy search which largely pushes whatever they want you to watch rather than what you’re searching for. YouTube is mostly also push rather than pull today.
I expect everything to continue moving that direction, against the wishes of the power users. The majority of users seem to go for the simpler UX, even if they sometimes complain about quality.
> In an ideal world, I’d like to have the underlying model be a swappable implementation detail. Llama 2 and similar developments make me optimistic.
This is a pipe dream. LLMs may be hot-swappable by developers but for 99% of apps + OSes this wont be a user-configurable thing.
Author of the piece, and I... can't really argue with any of this, tbh. I'll admit both those parts you called out were probably heavily tinged by my own desires, and your more-sober predictions are a very reasonable counterargument for what will happen in the general case. I suppose, especially regarding swappable LLMs, I _do_ only expect it to be an option for devs or sophisticated users; I assume that most folks probably wouldn't care, I'm just hoping there's enough of us that do care that at least some options offer that swappable functionality. Fwiw, I also use Linux (Fedora) as my daily-driver, and I'd be more than content if the predictions from this post came true in a similar vein, e.g. as an OSS option (or family of OSS options) that some subset of users can opt to use.
> Me too, but allrecipes.com has already switched from “search for what you want” to “we’ll tell you what you want”.
has anybody ever heard of a cookbook? it's the perfect ux for this. especially if you have a lot of different ones. even better if you collection is mostly physical copies.
Yes but Allrecipes.com didn't cost me hundreds of dollars to get 20-30 high quality cookbooks and had 10,000+ recipes. More importantly, I could trust that the highest-rated recipes, while not necessarily the "best", would be stupid-proof. The only way that recipes got universally high feedback from the average home cook on allrecipes.com is if you could accidentally double or halve any ingredient and it would still taste good. Cookbooks often contain recipes which, while better, are also more particular and require a higher skill floor.
I could also quickly filter against recipes which contained ingredients I didn't like, or filter for recipes which used ingredients I already have on hand.
OP here, as someone who does love cooking, I've gone down this route pretty heavily in the last few years - been growing my collection of physical cookbooks and definitely enjoy flipping through them in search of inspiration. So, yeah, very much endorse the cookbook UX!
I was hoping for some clever ideas about direction of software / UIs, and new use-cases for LLMs / AI, since for now I'm still struggling and have a lack of vision, it seems.
For example, I work on developing a logistics management and route optimization platform. If I try to envision new features that could be unlocked through AI or just LLMs, I basically get nothing back from my feeble brain, that I would fit into this category. E.g. - automate incident handling (e.g. driver broke down, handle redirection of other driver, handover of goods, reoptimize routes) - but the implementation would be just a decision tree based on a couple of toggles and parameters - no AI there? Other things that come to mind - we already use ML for prediction of travel times and service durations - it's a known space, that I refuse to call AI.
Apart from serving as an alternative and sometimes more efficient interface for data queries through NLP (e.g. "tell me which customers I had margin lower than 20% on, due to long loading times and mispackaged goods" - even then, all the data already needs to be there in appropriate shape, and it's just replacing a couple of clicks), I really fail to see new use-cases / features that the current state / hype for AI / LLMs unlocks.
Am I just lacking vision? Are there opportunities I'm grossly overlooking?
Turn the company jingle into a heavily distorted guitar track, make a database with 3d models of drivers, locations and drone footage, from departure to delivery including the people driving desks, then have the llm generate a movie with narration in a dark voice. Dealing with broken down vehicles is an opportunity to brag and show off.
Given all the log data for the last N packages, analyze for anomalies and hypothesis as to their cause. Eg is there a specific shipper, warehouse or driver causing problems?
ML does well when you have too much data for a human to wrangle and the search target is well described.
Ideas are probably the last thing humans will delegate to AI. We have needs and wants that Ideas help us meet, AI only sets priorities in a reactive way.
Yeah it's a pretty interesting experience actually, trying to use it for idea generation.
It's like talking to a very well informed and generally competent person ... who has no spark of creativity or insight whatsoever.
This does vary product to product - some are excruciatingly boring by design - but I think they're universally uninteresting just to different degrees.
> Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes. Even if an LLM could provide me with restaurant recommendations based on my preferences, I’d still seek out a map-based UX for exploring the variety present in my vicinity. The desire to replace all UX with LLMs seems like a desire to replace all serendipity with efficiency, and I think (or hope) that such a transition is much more appealing in theory than it would be in practice.
I guess the question is: how much of our web or software use is leisurely browsing (reading news or HN would be other likely candidates for this category) and how much is more task-like, e.g. send a message to some friends, add a note to a specific list, order some groceries?
We might also want to consider how much of a role such private use of software plays in shaping UX trends. If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI.
This is not to say that browsing will disappear, but I can totally see it being relegated to a second class UI in the long run, even in applications where it's currently the obvious choice, just because our default UX expectations will be different.
> If business software (sheets, Photoshop, CAD etc.) can be sped up with chat input, it will be, and people will be expected to use the quickest UI.
I have a hard time seeing chat input become the primary UI for that class of applications, unless you can delegate complete tasks to it. As an analogy, for driving a car, I can see voice commands replacing the steering wheel if we reach full self-driving capabilities, but absent that, the steering wheel and gas/breaking pedals will remain the more efficient and practical UI (even ignoring safety concerns).
I think the biggest change will be users will increase my parents don't know how to use Photoshop are not tech inclined they won't be touching up photos of their grand child. But give them an option of just asking the app to do something and they will as the learning curve goes down dramatically
> I guess the question is: how much of our web or software use is leisurely browsing
I think the author's sentiment here is different. There's a personal subjectiveness when it comes to things like recipes. It could come down to the presentation (photos, narrative), an added ingredient in one that piques your interest and curiosity (a chili recipe with dark cocoa powder?!), or other subjective difference that is experienced differently by each of us.
The other aspect is mental bookmarking or "what ifs". Maybe I'll try this recipe this time, but I might come across other recipes I want to try some other time or I'll find an author that I really vibe with. That process of discovery is lost with LLMs today
Ok. I appreciate they didn't use Medium as a blog post. There's a lot of good engineering knowledge currently locked behind the Medium "sign in" interface. It's dumb and stupid.
But just to voice an opinion, please kill the lengthy paragraph levels of fluff in blog posts. I don't want to say my opinion is influenced by shitty blog recipes and research papers, but it is. So stop it.
Just say what you need to say in bullet points at the beginning and fill in the details further on.
While I agree some longform articles could use a bulleted summary, I think the act of blog-writing is as much an exercise of the writer coming to terms with the subject as us enjoying the piece. Abridging an idea comes easier after exploring its breadth and width, and I am supportive of anyone who tries it out in the open.
You nailed it. In 2027 the AI is going to write the fluff for the Author and then extract the bullet points for the reader - Everybody saves time, winwin.
and for some there will be a "please elaborate" command of course
> Basically, the thinking goes, once we’re able to express our desires to computers in natural language, and once natural-language agents can carry out actions on our behalf, we’d no longer need the myriad of individual task-specific applications that exist today. Personally, I neither believe in nor desire this outcome.
Strong agree.
I find it very telling that the kind of people who advocate for chat-like software are often managers and executives. Sure, if the kind of person you are is one that enjoys giving verbal orders to people, it makes perfect sense that that's the kind of job you'll seek, and the kind of software you'll want.
But for the rest of us creative hands-on people who like to feel that we're making things through direct manipulation, talking to a computer is just about the least joyful activity I can imagine.
I would like to add my own prediction for 2027. I believe in the next 4 years, much more comfortable and capable mixed reality glasses and goggles may be somewhat common. Also AI generation and streaming of realistic avatars will have advanced. Quite possibly this will use low-latency Wifi to stream from a PC.
So if you want, you will be able to have a fairly realistic representation of the AI as a synthetic person in the room with you when you put the MR device on. It will have eye contact and seem quite similar to a real person (if you want). You will be able to just talk to it.
Another thing that might become popular could be larger 3d monitors. The type that have some stereoscopic effect tuned to your exact position. So your AI helper might just live in a virtual window or doorway or something like that.
You can actually already build something a bit like this, at least in 2d, without really inventing anything complex. You would use things like a HeyGen or D-ID API and maybe Eleven Labs or something. You won't get eye contact or a realistic 3d avatar that seems to be sitting on your couch, and there will be pauses waiting for it to respond. But theoretically, fast-forwarding several years, those things are not at all insurmountable.
I really doubt it. Modified glasses are too much of a change, compared to earbuds which can be the bridge to the phone in order to provide a conversational assistant. For normal use there is almost no benefit of having something visual available without the need for manual interaction, which is the benefit of glasses compared to the smartphone screen.
But being able to have a verbal communication with a chatbot is of immense help. It can be used while driving, while cleaning or doing anything else which requires the use of your hands or while they are dirty or covered with gloves.
These glasses will be expensive for at least 4 more years and most of us just won't feel the need to invest in it.
I'd rather chew gravel every day for the rest of my life than be forced into the hell of wearing a headset so that I can have my useless morning standups in a virtual room rather than on a 2D screen.
Four years is an eternity in software, especially now that we can have LLMs write the first draft.
Transformers were only invented six years ago after all. Some people are even very optimistically projecting that we'll reach the singularity in the next three.
Making comfortable and capable mixed reality glasses common is not primarily a software problem. In fact, I'd say the challenges are almost entirely unrelated to software, though there probably would be a need for a richer software ecosystem designed for mixed reality for them to really take off.
It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact.
It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments.
It does CodeLlama (and any HuggingFace-based language model)
It does ChatGPT
It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM.
I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.
The modern internet and modern app experience has become so shitty. For example opening up Robinhood, transferring money from a bank account, and buying a stock is hidden behind a multitude of product driven dark patterns that turns what should be a low cognitive load task into a high cognitive load task. This is true for everything now days.
I predict llm based agents will be used as a 3rd party layer on top of the modern web to work in users favor against enshitification.
If we're talking about the future here, it makes much more sense that you'd be buying stocks directly using currency stored on your phones hardware wallet through a DEX like Uniswap. In fact, you might just tell your phone that you want to buy X amount of stock, and it would do it all for you onchain.
Wouldn’t this avoid a lot of ad exposure that companies are currently enjoying? Why would they want to give that up by allowing you to use your llm instead?
My prediction is that LLMs are going to be used almost exclusively for views over commands. They are simply too unpredictable.
I'm more optimistic than the author about how useful LLMs may be for chat based interfaces - I don't think it is appreciated in tech how many people are (still) computer illiterate in the world, and natural language can be a big improvement in usability for the kinds of usecases these users need.
Clippy is going to make a triumphant return as the AI assistant of the future. Clippy will be on every phone/tablet/computer just talking to other instances of itself.
In 5-10 more years models will all run locally on device with weight updates being pushed OTA.
Advertisers will somehow figure out how to train ads into these models so you get subtle references to new products.
… concludes your helpful outdoor activities this weekend. Also, try Lawn Darts, which the cool kids call “Jarts”. (Sorry, training data was cut off in Jan 1970)
I'm actively trying to scaffold an LLM for business usecases and my experience totally echoes this. I can tweak the prompt for a very representative set of training data, and then it vomits at a seemingly normal new case.
You might be interested in another way to use LLMs inside a SQL query. For example, LLMs can be used to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.).
The predictions the article makes are a bit unimaginative and shallow. The timeline is also somewhat extremely long. It's more like late 2023 than 2027.
So, Language Model UX in 2024 according to me:
- Pervasive: Virtual assistants are omnipresent, and integrate everything together. Like Alexa that is actually intelligent and can for example make you a custom app for interfacing with your banking following your preferences on-the-fly for you only. Web apps become meaningless as you are getting the UI you need customized, tailor-made, live when you need it. Agents go with you in smartphones or project to any device nearby. No need for authentication, bots know you.
- Agentic: They will negotiate with each others all the time and present a unified interface to you no matter which system you are interacting with. The air is full of silent chatter. Things move when you aren't watching. Progress is decoupled from human input, attention and care.
- Web searches and such are for bots. Answering phones is for bots. Making calls is for bots. Any person still grasping at these kinds of interfaces will notice all emails they get are from bots, and all emails they send are instantly answered by bots. Just let go, let the machines handle that, which brings me to:
- Intermediation: No matter what you do, a bot will help you with it. Swiping Tinder? Don't bother, a bot does it better for you. Just lay back and relax. Ads become targeted to bots and truthful, because the bots won't forgive misrepresentation, ever.
My prediction: knowledge will start to be reformatted into an LLM directly and knowledge providers will stop publishing it otherwise. Thus the chat interface will both improve and protect knowledge access once the traditional formats are not published anymore.
This will come together with the downfall of search, a prime driver why so much knowledge was published on the web. Search will start to drown in an explosion of generated AI content, and the case to publish your knowledge for the sake of SEO marketing will diminish.
If you can sell your knowledge to an existing audience and then add a special category with a higher price for companies training LLMs why would you take the financial hit of selling to a single customer where they can now make demands of you as your only source of revenue?
I don't get your reasoning. First not all knowledge is the source of revenue. Many products and services are supported by know how to implement, use and maintain. This kind of knowledge can benefit from protection.
And if selling knowledge is the main revenue, why would transferring it to an LLM mean losing the opportunity to sell it multiple times? Au contraire.
For any category I assume the owner of the knowledge keeps in control and my case is that a LLM can be beneficial for them.
I’m currently building something that touches on point 4, the dynamically generated UI. I’m building basically copilot for PMs that integrates with your project planning tool and so far it’s been working like magic. You can literally tell it to output a json and use it like an API.
I think my main jobs are getting the prompts right and building an UI and UX experience that makes sense, and the rest is somehow taken care of by magic.
One thing I hope we'll see is that the people with the big LLM servers try to be platforms as opposed to one-stop shops [1]. With AI getting better and better at executing, I think best case we'll see an era where good novel ideas are the currency of the realm[2]. That would be great - coming up with novel ideas and thinking deeply about what we should be pursuing is about the best alignment with human nature I can imagine. But it can only happen if access to the LLMs themselves is cheap and hacker-friendly.
[1] there's good arguments to evolve a business model in that direction, e.g. Apple beating blackberry not through superior hardware but a superior app store.
[2] I'd be remiss not to plug my own recent oevre here, https://eucyclos.wixsite.com/eucyclos/contact-8-1 inspired by the difficulty of conveying positive emotions over screens. Advice from people who have built this kind of thing successfully is of course welcome.
> the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others)
A lot of people feel this way. The romanticism of computing is diminished by LLMs. Call me a luddite. I even work in AI.
I can give a basic example of why chat-based UI will not dominate.
Let's say I want to change a setting that uses a traditional radio button UI:
- Autoscale servers
- Manual scale servers
It's much easier to discover, understand the options, and make a decision via a radio button UI than to ask for my options via chat. That would look like:
"I'm having load issues on my servers. Can you increase the server capacity?"
"Sure! Do you want to autoscale or manual scale?"
"What's the difference? Are those the only two options or are there more?"
"There are only these 2 options. The difference is..."
But it's not either/or. There's nothing that says you can't have a visual interface that adjusts based on both natural language feedback and traditional cursor-based input.
There are great examples of this in Westworld and The Expanse, with characters talking to screens to update and refine their queries.
Nice writeup! Here is another example of using LLMs is to augmenting existing software that we are exploring -- specifically SQL database systems. We are using LLMs inside a SQL query to power a "soft join" between SQL tables for when a correspondence is only implied (e.g. different address formats, etc.).
--- Create a reference table that maps neighborhoods to zipcodes using ChatGPT
CREATE TABLE reference_table AS
SELECT parkname, parktype,
ChatGPT(
"Return the San Francisco neighborhood name when provided with a zipcode. The
possible neighborhoods are: {neighbourhoods_str}. The response should an item from the
provided list. Do not add any more words.",
zipcode)
FROM postgres_db.recreational_park_dataset;
--- Map Airbnb listings to park
SELECT airbnb_listing.neighbourhood
FROM postgres_db.airbnb_listing
JOIN reference_table ON airbnb_listing.neighbourhood = reference_table.response;
My guess is that LLMs will become an absolutely incredible tool, to the point of being indispensable, for software developers. This demographic is highly biased and often magnify their usefulness based on that. For everyone else, they will continue to be an advanced search engine / copywriting assistant.
Part of the problem is that GPT-4 inference is too expensive to roll out at scale with current GPU availability and cost, so even basic features aren't generally available (e.g. your word processor writing for you) or if they are the model used is cheaper and not as good.
Partly it just takes time - it will overall take (I think, based on previous similar changes like the web) 20 years before the ideas from the current generation of LLMs are built out and integrated into products and made into new products and it is all done. People and organisations take time to change.
The problem with LLMs as I see it is that they make logic errors all the time, and somehow LLMs themselves are not smart enough to reason about their own reasoning.
You've got nerves of steel. It can, and will, hallucinate sudo rm -fr / as a step necessary for configuration at some point (it's even easy to justify by, say, missing the mention of this having to be in a chroot or a container). Russian roulette with your data and setup on the line. Not sure I'm willing to try...
Some nice ideas here. I think some of these could see realisation, depending very much on how accessible those LLMs become.
At the moment I'm sure there are only a couple of user interfaces that people know and will resist l reuse. After all they'll want to take very little risk and use what they've seen works in other places. The full screen ChatGPT style page, or those annoying chatbot pop-ups that sit in the bottom right hand corner of sites I don't care about. Something to keep an eye on and see what emerges.
The largest problem with a gpt future imo would be bot networks creating manufactured outrage. these will shift the decisions of policymakers towards those who controls these networks ie big corp.
Nowhere near that, the largest problem will be when the misinformation will be manufactured on an unprecedented scale by these LLM. In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale. Doctorow called them "plausible sentence generators" and that's indeed what they are and it being plausible will make it deadly.
> As the U.S. marks one million people dead from COVID-19, scientists suggest that nearly one third of those deaths could have been prevented if more people had chosen to be vaccinated.
> In the US alone some 300 000 people died from homemade artisan anti vaxx propaganda, it's very scary to think what will happen when GPT switches that to industrial scale.
From a human moral viewpoint this is indeed despicable and worth considering mitigation.
I am optimistic though that in the grand unfolding that natural selection brings, it will pressure an elevation of rationality, but with the cost of some human suffering.
Natural selection is not fast enough. It operates over many generations but we know climate change is driving the expansion of zoonotic diseases. Consider within the last generation -- generally understood to be 25 years -- we had a SARS outbreak, swine flu, SARS-CoV-2, mpox outbreak just to name the big ones. It is extremely likely we will have more within the next 25.
> Natural selection is not fast enough. It operates over many generations but we know climate change is driving the expansion of zoonotic diseases. Consider within the last generation -- generally understood to be 25 years -- we had a SARS outbreak, swine flu, SARS-CoV-2, mpox outbreak just to name the big ones. It is extremely likely we will have more within the next 25.
Perhaps I should not have used "natural selection" but instead a more generic evolutionary principle but that isn't prodicated on biological hereditary. Since the type symbolic reasons arose in our brain matter, our bodies have been vehicles for evolution that transcends the rhythm of offspring.
Loving the ideas here. I'm especially fond of the machine-to-machine idea, the concept of a locally running LLM that knows about you, and a remote LLM in an app or whatever, interacting with eachother to create a highly individualized UX.
Seems like a perfect blend of keeping the core personality of the LLM trained on your data local, while still allowing permissioned access.
I hope all the bold and rapid changes predicted here won't become reality so fast. I think the societal changes needed to somehow still function as a civilization will be too much for vast masses of people to handle resulting in some drastic decisions. I'm pretty pessimistic about this and can't shake it off, call it Yudkowsky syndrome.
Title is misleading. More like what is under the hood in the near future, a technical perspective. I thought to find some inspiration on what kind of software we can expect.
I agree with the author that chat interfaces for LLMs can be suboptimal. A chat could require redundancies that is not required to perform a task and it imposes a linear flow of information. It is hard to have branching behaviour in a chat. I have been experimenting with using a different kind of a UI to overcome these limitations: https://www.heuristi.ca/
@vishnumenon, thanks for writing this up. I might do a followup blog on this, for now here is a comment :)
This article is close to my heart as I've been working on craftcraft.org with similar perspective.
1. Chat UX Consolidation: I agree that having crappy chat UIs everywhere is very suboptimal. Perhaps having a complete UX as a component is another solution here.
We took many months to get http://chatcraft.org from prototype to an ergonomic productivity tool. Highly unlikely such attention will be paid to every chat UI integration.
2. Persistence Across Uses. This one is tricky. We keep all of our history client-side...but after using it this way, having a shared history server-side and having it pulled in as relevant context would be a nice improvement.
3. Universal Access: It's super weird to have LLMs restricted to providing output that you cut/paste. We have integrated pretty slick openai function interface to allow calling out to custom modules. So far we integrated: pdf conversion/ingestion, clickhouse analytics and system administration using a webrtc<->shell connector. Demo here: https://www.youtube.com/watch?v=UNsxDMMbm64
3b. LocalLLMs. These have been underwhelming so far vs openai ones(except maybe WizardCoder). Industry seems to be standardizing around openai-compatible REST interface(ala S3 clones). We have some support for this in a wip pull req, but not much reason to do that yet as the local models are relatively weak for interactive use.
4. Dynamically Generated UI & Higher Level Prompting: I do a lot of exploration by asking http://chatcraft.org to generate some code and run it to validate some idea. Friend of mine built basic UX for recruiting pipelines, where one can ingest resume pdfs into chatcraft and via custom system prompt have chatcraft become a supervised recruiting automation. We also do a lot generation of mermaid architecture diagrams when communicating about code. I think there a lot of room for UX exploration here.
Now a few categories that weren't covered:
1. Multi-modal interaction: It's so nice to be able to have chat with the assistant and then switch to voice while driving or to input some foreign language. I think extending UX from chat to voice and even video-based gestures will make for an even cooler AI assistant experience.
2. Non-linearity in conversations: Bots are not human, so it makes sense to undo steps in conversation, fork them, re-run them with different input params and different model params. Most of my conversations in chatcraft are me trying to beat llm into submission. Example: tuning chan-of-density prompt https://www.youtube.com/watch?v=6Vj0zwP3uBs&feature=youtu.be
Overall, really appreciate your blog post. Interesting to see how our intuition overlaps.
Really good stuff, but some minor things: your url doesn't work; went to your twitter profile, and it seems you meant https://chatcraft.org? Also, you are un-dm-able on twitter. (I am @eating_entropy if you want to talk more)
The discord link seems to be not working. Just a heads up.
The YOLO example on your Github page is super interesting. We are finding it easier to get LLMs to write functions with a more constrained function interface in EvaDB. Here is an example of an YOLO function in EvaDB: https://github.com/georgia-tech-db/evadb/blob/staging/evadb/....
Once the function is loaded, it can be used in queries in this way:
SELECT id, Yolo(data)
FROM ObjectDetectionVideos
WHERE id < 20
LIMIT 5;
SELECT id
FROM ObjectDetectionVideos
WHERE ['pedestrian', 'car'] <@ Yolo(data).label;
Would love to hear your thoughts on ChatCraft and a more constrained function interface.
> I don’t believe that natural language is an adequate medium for conveying instructions with the precision required for many applications. Moreover, the “inefficiency” involved in carrying out many digital tasks “manually” is a core driver of the joy that comes with using computers (for me and, I assume, for at least some subset of others). Even if an LLM could provide me with a recipe that perfectly suits what I’m looking for, I wouldn’t want to give up the experience of using a recipe search engine and browsing through a large collection of recipes.
But searching for places to eat and recipes to make is very much not a precise search.
IMO the reason chat and not input text and get an answer is so powerful is that it allows messy search with iterative refinement - just like talking to an expert. Just "chat input, result given" doesn't have that.
I want to tell a recipe search I'm after a healthy light thing as it's hot. I want to get back options and then say "I don't really like cucumber though" and have it get rid of cucumber heavy recipes but leave in some salads and say "look this recipe has cucumber in but it'll be fine without it". Or "you asked for a chicken recipe but here's one with pork that should sub in just fine".
For restaurants I want to get back some options and tell it "that's too expensive, this is a quick thing" and get back more hole-in-the-wall things. Tell it "Hmm, something spicy sounds good but I've been to those Indian restaurants before, I'm after something new" and get a recommendation for the Ethiopian place in town.
> The current state of having a chat-ish UX that’s specific to each tool or website (e.g. a Documentation Chat on a library documentation page, a VSCode extension, a Google Bard integration, a really-badly-implemented chatbot in my banking app, etc.) doesn’t make any single one of those experiences more enjoyable, effective, or entertaining;
Coding chat in my editor absolutely makes coding more effective. I want heavy integration around how it edits text, not a general chat widget.
> The idealized role of persistence in LLM UX is also fairly obvious: it’s easy to imagine an LLM-powered experience that remembers and “understands” all my previous interactions with it, and uses that information to better help me with whatever my current task is
I sort of agree, but I absolutely detest hidden state about me. I should not alter my behaviour just because I'm worried about how that'll impact things. You see this regularly, I may avoid some weird youtube video (e.g. to see a weird flat earth take) because I don't really want the hassle of having loads of weird conspiracy stuff promoted or having to manually remove it from some list.
Having said that, recipe search that remembers I hate cucumbers is great.
I wonder if manually curated context will in general be better? Or maybe just for me.
> I’m interacting with UXes that can remember and utilize my previously expressed preferences, desires, goals, and information, using that underlying memory across different use cases seems like low-hanging fruit for drastically reducing friction.
The tricky part here is I switch contexts and don't want them bleeding into each other. My preferences for interaction around my kids and while at work are very different.
> A small-scale and developer-centric example: I use GitHub Copilot in VSCode, and I was recently implementing a library with documentation that featured LLM-powered Q&A, and it felt bizarre to have two LLM-mediated experiences open that each had exactly half the info needed to solve my problem.
I think here a split between data sources and frontends is key. This interaction is awkward, and should be combined (copilot should be able to reach out to that documentation).
> locally runnable and open models are no match for GPT-4 (etc.),
It's going to be more efficient to move compute to central places, the less they're used per person the more efficient it will be to have one central location process everyones things. A short taxi ride per week is cheaper than owning a car. However as uses grow (e.g. proactive llms), will this shift the equation towards locally runnable ones? A few queries a day and you're obviously better off not buying a h100. Constantly running things all day and if prices fall maybe that'll change.
I'd love for the author to be right in his predictions. Arguably current AI has made understanding this article '4. easier' for me to understand already.
There will be a paradigm shift where our-customers-are-ai apps appear and most stuff will need to have an API which makes AI able to connect and use those services effortlessly and without error because who don't want to tell his assistant to "send 5$ to Jill for the pizza"? There will be money in base AI models you can choose(subscribe to) and what it can and cannot do for you. It will still be riddled with ads and now it's your personal assistant who can push any agenda to you.
Operation systems will become a layer under the assistant ai.
You will still talk on/to your phone.
I guess free software will be more important than ever.
Free ai assistants will be available and computing power will be there to run it on your phone but all the shit we've seen with open Vs closed source, Linux Vs Windows, walled gardens whatnot will go another round this time with free open public training data assistants Vs closed-but-oh-so-less-clumsy ones.
Security problem will be plenty like how to hide your AI assistant custom fingerprint? Authentication and authorization system for AI? How much someone else personal assistant worth? How to steal or defend it?