These patterns appear in many fields. I take it as a sign that the tooling in the field is underdeveloped.
This leads to a split between domain problem solvers, who are driven to solve the field's actual problems at all costs (including unreliable code that produces false results) and software engineers, who keep things tidy but are too risk-averse to attempt any real problems.
I encourage folks with interests in both software and an area of application to look at what Hadley Wickham did for tabular data analysis and think about what it would look like to do that for your field.
Everyone compares against llama.cpp because it's easy mode. Llama.cpp is slow! Everyone should know this. They should compare against exllamav2 or other optimized implementations.
ROCm, which is AMD's equivalent of CUDA. The thing is you don't have to directly interface with CUDA or ROCm. Once the framework you want to use supports these, you're done.
AMD is consistently getting used on the TOP500 machines, and this gives them insane amounts of money to improve ROCm. CUDA has an ecosystem and hardware moat, but not because they're vastly superior, but because AMD both prioritized processors first, and NVIDIA played dirty (OpenCL support and performance shenanigans, anyone?).
This moat is bound to be eaten away by both Intel and AMD, and compute will be commoditized. NVIDIA foresaw this and bought Mellanox and wanted ARM to be a complete black box, but it didn't work out.
Ethernet consortium woke up and got agitated by the fact that only ultra-low latency fabric provider is not independent anymore, so they're started to build alternatives to Infiniband ecosystem.
Interesting times are ahead.
Also there's OneAPI, but I'm not very knowledgeable about it. It's a superAPI which can target many compute platforms, like OpenCL, but takes a different approach. It compiles to platform native artifacts (CUDA/ROCm/Intel/x86/custom, etc.) IIRC.
The bottleneck is probably the availability of lithography machines that can make ubiquitous chips for processing that much data quickly enough without heating or drawing electricity too much.
Not too far in the future every device will have some 5nm or better tech LLM chip inside and devices understanding natural language will be the norm.
By dumb machines, people will mean machines that have to be programmed by people using ancient techniques where everything the machine is supposed to do is written step by step in a low level computer language like JavaScript.
Nerds will be making demos of doing something incredibly fast by directly writing the algorithms by hand, and will be annoyed by the fact that something that can be done in 20 lines of code on few hundred MB of RAM in NodeJS now requires a terabyte of RAM.
Dumb phone will be something like iPhone 15 pro or Pixel 8 Pro where you have separate apps for each thing you do and you can't simply ask the device to do it for you.
It's not mentioned in the paper but this month OpenChat 3.5 released the first 7b model that achieves results comparable to ChatGPT in March 2023 [1]. Only 8k context window, but personally I've been very impressed with it so far. On the chatbot arena leaderboard it ranks above Llama-2-70b-chat [2].
In many ways open source LLMs are actually leading the industry, especially in terms of parameter efficiency and shipping useful models that consumers can run on their own hardware.
I have a midrange Zojirushi rice cooker, a Vermicular Musui Kamado cast iron induction cooker, and a Kamado-san double-lid donabe rice cooker, all from Japan.
The Zojirushi makes respectable rice with minimal effort, as in we're about to walk the dog but we need to set up rice for dinner.
The Musui Kamado is a dramatic upgrade over an Instant Pot for every use except pressure cooking, for which we prefer a stovetop Fissler pressure cooker. Carefully following directions, it makes better rice than the Zojirushi. We use it whenever we want a feedback loop for fixed temperature, not unattended flame, such as cooking beans or stews. For tacos, I have to remember to transfer the cooked nixtamal corn for fresh masa, in time for my wife to start beans.
The Kamado-san double-lid donabe rice cooker takes the most care, and makes in my opinion the best rice, with a controllable bit of scorch as desired.
At home, out of expediency we use the Zojirushi to make our rice. I find myself avoiding meals that involve rice, out of boredom. In my work apartment, I haven't touched the Zojirushi or Musui Kamado for rice since adopting the Kamado-san donabe, and I look forward to rice.
If all sushi bar rice tastes the same to you, stick to the Zojirushi. If only food at the limits of your abilities captivates you, get a Kamado-san donabe. The Musui Kamado is the most versatile compromise. There's no novel physics here; its edge over both a Zojirushi and an Instant Pot can be explained by its precision-machined cast iron pot, of a better quality than Le Creuset or Staub (I have all three).
You do realize how possible it is to fine tune a task like this (along with a hundred others in a similar vein) on a tiny model you can scale on your own hardware?
I've run hundreds of millions (150m so far in a couple of weeks of non-continuous running as I tweaked things) of tokens through my 2x 3090 with a 13b llama2 model I fine tuned on tasks like: summary, knowledge graph generation, writing using the knowledge graph, grammar, spelling, and transcription correction, etc.
This type of stuff is going to be done at scale with a modest budget if you have the skills to tune more efficient and faster models to your use cases.
Can I quote zach's 15 year old comment on Python vs Ruby? Maybe it will have some relevant insights:
> Ruby has clever syntax. Python has pure syntax.
> Ruby has method aliases. Python does not allow a string to capitalize itself.
> Ruby uses Ruby methods within Ruby classes to extend Ruby. Python has decorators so you can write functions that return functions that return functions to create a new function.
> Ruby has strict object-oriented encapsulation. Python is laid-back about objects, because you probably know what's going on inside them anyway.
> Ruby lets you leave off parentheses so you don't miss objects having attributes too much. Python will let you mix tabs and spaces for indentation, but passive-aggressively mess up your scoping as punishment.
> Ruby has seven kinds of closures. Python has one, in the unlikely case a list comprehension won't do.
> Ruby's C implementation is a mess of support for language-level flexibility. Python's C implementation is so clean you get the unsettling thought that you could probably write Python using C macros.
> Ruby supports metaprogramming for cases when programmers find it more descriptive. Python supports metaprogramming for cases when programmers find it necessary.
> Ruby is expressive. Python is direct.
> Ruby is English. Python is Esperanto.
> Ruby is verse. Python is prose.
> Ruby is beautiful. Python is useful.
> I like Python, but coming to it after using Ruby for seven years, well, I think it's like dog people and cat people. You can like having one species around, but you're always thinking -- why they can't be more like the other?
Every time I travel, I spend insane amounts of time browsing AirBnB to find a suitable place.
For example, I miss a "good table" option on AirBnB.
I have to manually discard over 90% of the listings after looking at the images because there is no table suitable for work. That is so time-consuming.
It's not that I want to work at home all the time, but still I need a proper table to put my laptop on. A proper chair and a monitor would be golden of course.
And there are a bunch of additional things I do manually too. For example reading the reviews. Meanwhile, I am pretty good at predicting the quality of a place by reading the reviews. The stars are somewhat of an indication. But not reliable at all. I had really bad stays at places that got 100 reviews and an average of 4.7 stars. Hosts have too much influence on this metric.
Here are two examples of positive signals that help me find nice places:
1) How long the reviews are.
Hosts can remove bad reviews. And they can manipulate their guests into writing 5-star reviews. But they cannot really manipulate them to write long, enthusiastic reviews with many details.
2) How many of the reviews say "I have stayed at many AirBnBs and this one was one of the best".
This is something I usually write when I particularly liked a place. And it turned out a good signal to find nice places.
I often wonder if one could build a business by creating a site that lists a small subset of AirBnBs that are manually vetted like this. I already wrote my own web based tool to organize and sort AirBnBs that I am interested in. So it would not be a lot of work to make these lists public.
Also, he narrates that book but not the new one. Which is too bad because his narration is truly additive to the book.