Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters.
Enterprises I have spoken to says they are getting pitched by 20 startups offering similar things on a weekly basis. They are confused on what to go with.
From my vantage point (and may be wrong), the problem is many startups ended up doing the easy things - things which could be done by an internal team too, and while it's a good starting point for many businesses, but hard to justify costs in the long term. At this point, two clear demarcations appear:
1/ You make an API call to OpenAI, Anthropic, Google, Together etc. where your contribution is the prompt/RAG support etc.
2/ You deploy a model on prem/private VPC where you make the same calls w RAG etc. (focused on data security and privacy)
First one is very cheap, and you end up competing with Open AI and hundred different startups offering it. Plus internal teams w confidence that they can do it themselves. Second one is interesting, but overhead costs are about $10,000 (for hosting) and any customer would expect more value than what a typical RAG provides. Difficult to provide that kind of value when you do not have a deep understanding and under pressure to generate revenue.
I don't fully believe infra startups are a tarpit idea. Just that, we havent explored the layers where we can truly find a valuable thing that is hard to build for internal teams.
Pretty much this, 18 months ago my CEO told me we HAD to get into this space, and I told him that basically our money came from our private product and that the only way our big enterprise customers were going to play game with us was either ironclad agreements that went all the way to openai, or more likely a completely single tenant system, which would cost far more than they were willing to pay.
Of course they went with both, and as far as I can tell both are a major disaster post layoffs :)
I fully expect in somewhere around 3-6 months the dam will burst and we're going to start hearing more and more about all the teams out there that are pouring tens of millions of dollars into AI and all they have to show for it is a worse version of whatever it is they were doing.
To placate the AI fans, that's not because AI isn't interesting, it's because that's how these hype cycles always go. I remember when everything had to be XML'd. XML has its uses, but a lot of money was wasted jamming it everywhere because XML Was Cool. AI has its uses, but it is still an engineering tool; it has a grain, it has things it is good at, it has things it can't just wave a magic wand and improve, the demarcation between those two things is very, very complicated, and people are being actively discouraged from thinking about those lines right now.
But there really isn't any skipping the Trough of Disillusionment on your way to the Plateau of Productivity.
That was probably before my time. Was it really "cool"? Like big data, cloud and agile cool? Or more like ... dunno ... some design pattern? So hard to think about XML as having been cool.
Can you go into details (as much as you are comfortable) on what happened with the single tenant system? I have seen a few things, but I find it hard to put a finger on what went wrong except the ROI wasnt there. Would love to understand your experience.
Mostly capacity issues with openai made super early stuff infeasible for our customer base, and really it was because they would only give an extremely small number of total resources per azure account, which of course each would be vetted.
I pushed for local first models but the cost tradeoff just did not make sense for anything but the biggest clients, and they were constantly swapping back and forth whether openai would be acceptable or not for "insert sensitive use case here"
Sorry for being vague. I meant LoRA, but used Apple as an example because their demo showed the potential. At a conceptual level, you can finetune a base model to be good at a specific task - eg: summarization, proofreading, generation etc. These finetuned weights are at the top layer and can be replaced by other weights for a different task as needed. Apple demoed different tasks by showcasing how their model identifies the task and then chooses the right set of finetuned weights. Apple called it Adapters as it comes via LoRA (Low Rank Adapters). It's around for some time, but only shot into prominence after people got some idea on how to use it.
> Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We utilize adapters, small neural network modules that can be plugged into various layers of the pre-trained model, to fine-tune our models for specific tasks. For our models we adapt the attention matrices, the attention projection matrix, and the fully connected layers in the point-wise feedforward networks for a suitable set of the decoding layers of the transformer architecture.
Not Apple adapters per-se, but LoRA adapters. It’s a way of fine tuning a model such that you keep the base weights unchanged but then keep a smaller set of tuned weights to help you on specific tasks.
(Edit) Apple is using them in their Apple Intelligence, hence the association. But the technique was around before.
Its rent seeking and grifting. As technology has become easier to get into, a huge number of "startups" are low level people just looking to make noise, get their cut, and bail. Its a bad look, up there with fast fashion.
An acquisition here amounts to teams luck surface.
Enterprises I have spoken to says they are getting pitched by 20 startups offering similar things on a weekly basis. They are confused on what to go with.
From my vantage point (and may be wrong), the problem is many startups ended up doing the easy things - things which could be done by an internal team too, and while it's a good starting point for many businesses, but hard to justify costs in the long term. At this point, two clear demarcations appear:
1/ You make an API call to OpenAI, Anthropic, Google, Together etc. where your contribution is the prompt/RAG support etc.
2/ You deploy a model on prem/private VPC where you make the same calls w RAG etc. (focused on data security and privacy)
First one is very cheap, and you end up competing with Open AI and hundred different startups offering it. Plus internal teams w confidence that they can do it themselves. Second one is interesting, but overhead costs are about $10,000 (for hosting) and any customer would expect more value than what a typical RAG provides. Difficult to provide that kind of value when you do not have a deep understanding and under pressure to generate revenue.
I don't fully believe infra startups are a tarpit idea. Just that, we havent explored the layers where we can truly find a valuable thing that is hard to build for internal teams.