Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt):
# prints the first ten prime numbers
def print_primes():
i = 2
num_printed = 0 # end of prompt
while num_printed < 10:
if is_prime(i):
print(i)
num_printed += 1
i += 1
def is_prime(n):
i = 2
while i * i <= n:
if n % i == 0:
return False
i += 1
return True
def main():
print_primes()
if __name__ == '__main__':
main()
It will be interesting to see how the larger models perform, especially after community tuning and with better context/prompting.
def primes_upto(limit: int):
"""Generate prime numbers < *limit*."""
# Sieve of Eratosthene
is_prime = [True] * limit
for n in range(2, limit):
if is_prime[n]:
yield n # found prime number
for c in range(n*n, limit, n): # start with square, less values are marked already
is_prime[c] = False # mark composites
if __name__ == "__main__":
from itertools import islice
print(*islice(primes_upto(100), 10)) # -> 2 3 5 7 11 13 17 19 23 29
Yeah, but yours was generated by the "post unoptimized code to HN and wait for someone to optimize it" model, which, although free and doesn't require a GPU, is a much slower model.
Someone should turn this into a product! You highlight the code you want to optimize, and it posts it to hn as a semi-contextually-appropriate comment to invite code golfing, and the highest rated reply gets posted back to your repo as a PR.
But, unless you are trying to find a prime number low enough that you might as well look it up in a pre-generated table, it might still be end-to-end more efficient?
This is a good point; English alone is an awful specification language. Everyone knows what you mean by "print the first ten prime numbers" but only by idiom and cultural context. This is why LLM is a good fit here, because -- when it works -- it includes all three aspects.
I suppose it's indeed possible to read the assignment as "print whatever you like as long as it includes 10 or more primes" but I'd fail you for that based on not being able to communicate with humans.
I have been waiting for weeks, and am still waiting, to get access to Llama2 (released a month+ ago), and access to this model goes through the same form, so I'm not very hopeful. Are you getting it from other methods?
Interesting here and in some of the other comments in this thread is that 1 is not a prime number ¹)!
Now granted, that's more or less by definition and I don't doubt there's communities and fields where it is considered one, but still shows some of the subtleties at play when using language models.
Whether 1 should be a prime number or not wasn't clear-cut for centuries.
Current consensus has settled on excluding 1 from the definition, but there are examples of publications well into the 20th century that still included 1 as prime.
Rough calculation: typical quantization is 4 bit, so 7b weights fit in in 3.6GB, then my rule of thumb would be 2GB for the activations and attention cache (not usually quantized). So 6 or 8 GB VRAM would probably do it. llama.cpp will let you offload your choice of layers to GPU, so you could probably get quite a way with 4GB.
> "I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing"
Congratulations! You must be that arrogant guy everybody hates interviewing with, the one with the superiority complex.
How about instead of just failing people over literally nothing (wasting everybody's time and money) - just ask the candidate whether they could somehow reduce the search space by utilizing the properties of a prime number?
Are we including it just to poke the bear? Prime numbers are typically defined as numbers with no positive divisors other than one and the number in question.
Well, I’m questioning it… not that it really matters.
-1, 0, and 1 are no good.
Two divisors for positive integer X:
1, X
I recently demonstrated GPT-4 via by having it explain phases of matter metaphorically with animals. Though impressive, it points to loose abstraction being more in range than firm abstraction. Let’s get that model a python interpreter and see how far we can take this party. Until then, I’m going to stick to explaining concepts.
————- included here for fun
Alright! Let's dive into the wild world of matter using some animal friends as examples:
1. *Solids* - Think of solids like a herd of elephants standing closely together. The elephants are packed in tight, barely moving, just maybe swaying a bit. They're sturdy and strong, just like solid things in our world. This is because the particles in a solid don't move around much; they just vibrate in place.
2. *Liquids* - Imagine a school of fish swimming in a pond. They're free to move around, weaving in and out, but they still stay pretty close to each other. They aren't packed as tightly as our elephant friends, but they aren't completely free either. This is like liquids: the particles are close, but they can move around and flow, just like water in a glass.
3. *Gases* - Now, picture a flock of birds soaring in the sky, free to fly in all directions. These birds aren't sticking close to one another; they're spread out, enjoying the vast space of the sky. In gases, the particles are like these birds, very spread out and moving all over the place.
4. *Plasma* - Think of plasma like dragons (I know they're mythical, but bear with me). These dragons breathe fire, and that fire is so hot and energetic that it can change the way things behave. Plasma is like that – it's gas that's become so hot that the particles are super energized and can even glow, like neon lights.
5. *Bose-Einstein Condensate (BEC)* - This one's a bit trickier, but imagine penguins in Antarctica. They huddle closely together to keep warm. BEC is like the coldest group of particles ever, where they start acting in strange, uniform ways, almost like one giant super-particle. It's like all the penguins moving together as one.
So, next time you think of matter, just remember our animal friends: the sturdy elephants, the flowing fish, the free-flying birds, the fiery dragons, and the huddled penguins!
Again, 1 fits, because it has two divisors: 1, and 1. You never said X != 1, nor does the definition upthread.
This isn't a silly gotcha, these things matter in math. For instance, when solving a quadratic equation, allowing for the solutions to be equal lets you avoid special-casing your understanding - instead of memorizing when the equation has zero, one or two solutions, you just learn it has two or zero (real) solutions, and the two solutions are allowed to be equal.
It's perfectly reasonable understanding. Inequality isn't a natural implicit assumption. E.g. if I say you have two variables:
int a;
int b;
I doubt you'd be insisting that `a != b` at all times.
No. In the set of integers, 1 is not two divisors if repeated. It is one divisor.
An element to set relationship is that element A is or is not in set B. So, if the set of divisors only contains 1, there is only one divisor. If 1 and 1 made two divisors, 1 and 1 and 1 and… would make infinite divisors, rendering the concept of counting divisors (i.e., the cardinal number of the set of divisors) meaningless.
M is a divisor of N if it is a number that divides N without a remainder. While divisors can be negative, they are conventionally limited to non-negative integers in primality and factoring.
If you’d wanted to dig in on negative vs. positive divisors, that quickly provides an avenue for clearer formality, but piling on 1 and saying it’s not a silly gotcha is pretty fruitless. And please don’t bother to say “you didn’t say it has only two divisors”, as that would, again, be a silly argument.
So wind back and really formalize the definition if you want: A prime number is a natural number with only two divisors in the set of natural numbers, 1 and itself.
While set theory is axiomatic, it’s not practical for me (or anyone else) to explain conventional foundations to avoid someone feeling like they need to wiggle out of a prior bad argument.
Just say “ah, okay” or stop replying and move on. Feel free to read up in Wikipedia or any other texts (or ping me privately if you’d like to discuss further), but this thread isn’t looking like it’s going to meaningfully contribute to the broader discussion. Accordingly, I’ll leave it here unless something meaningful comes up.
Downvoting this is not enough. I'd pay $5 (reddit style) to have this properly killfiled if HN allowed for that. Besides, not "everybody hates them, only those inmature enough to still need intellectual babysitting.
Finally we meet the lifeless drone that everybody complains about in the interviews.
My suggestion for your next interview: decide to hire them just based on their leetcode score, but invite to the interview just to flex that you're still better at puzzle solving :-D
So I take it you typically produce fully optimized, thoughtful, and correct code on the first iteration while being actively judged by a stranger, yes?
I assume you meant that you should add 2? If yes, that's such a mind boggling basic thing to do that I agree with you, and it makes no sense that you're being crucified.
Probably because there's significant overlap in the Venn diagram of people with years experience who professionally develop products that generate $millions in wealth/value, and people who would fail that interview.
Or we have worked with junior developers who have really grown and flourished under our care, who would never have gotten that chance with such insane Draconian judgements.
It's such an obvious "GOTCHA!!" setting someone up for failure.
The way it's framed is very cringy because it signals that they don't care in their interviews about determining how objectively effective a software developer is.
I don't get it. This is an extremely basic fact that most people can figure out after thinking about primes for a minute. Maybe if you ask for "what's an easy optimisation here?" This would make the candidate think more closely about invariants that their code should hold, which in itself is a very valuable skill.
Because I know enough not to write prime testing code that resemble anything like that loop to have to care about reducing its search space. If you actually want to test my knowledge about prime number, you can ask and I will tell you about using some probabilistic choice instead, and that I know fast deterministic one might exist, but I am not up to date on the state of the art.
If I have to write the loop above, I am assuming it is the Fizzbuzz equivalent of your company to show that I know how to write a while loop. I am not thinking about reducing the search space because I am writing the code semi-unconscious and frankly just want to get to the next question.
Same. Also, if I'm suspecting it's a "can they into loops" fizzbuzz test, I'd be wary of reducing the search space for the simple reason that it makes the code slightly more complex, introducing a chance to make e.g. off-by-one error, which would lose me points if the reviewer has the kind of "pedntic over irrelevant things" attitude this subthread is criticizing.
I've done that (maybe it was fizzbuzz, now that I'm thinking about it) and boy howdy does that get the people you're interviewing with agitated. Saying "I'm interviewing for a architect level container orchestration position. If I'm reinventing the wheel writing algorithms, something is terribly wrong" shuts them up, but doesn't make them any happier.
what does and container orchestration architect do ? Something like this cluster should use envoy and Prometheus. The new clusters rate isn’t usually high enough for the stack to change.
Real question I love these non conventional (swe, sre, pm, manager ) roles in tech
> I'd fail an interview candidate that suggested adding 1 each time for subsequent prime testing
The problem a team that always seeks the optimal solution is that never they get shit done. And that’s rarely optimal in a business context. Your view does not strike me to be nearly as arrogant as it is short-sighted.
I think on a team of one I want the guy who gets it done without thinking. On a team of two I want a guy that’s somewhere in the middle. And on a team of three, that’s when I want my optimiser. Because in the time that guy number 3 has written let’s say 3 files of optimal code, guy number 10 files of not optimal code. And you rarely need guy number 3 to fix all ten, you just need him to fix the one or two files that actually matter.
Clients rarely ask “is this optimal?”. But they always ask “is this done?”.
All three developers have different approaches. All three are assets.
I think on some level then you’re making the same mistake that we could say the “just add one guy” made if your comment is honest- not factoring in (his) speed.
I think code readability, rather than code optimisation is far more important thing to get hung up on in an interview (and is, I must remind some of you, not to be confused with conciseness). And you can see this in the end result. But if you’re following along and the interviewee already knows you know what’s going on because of that, you can see it in smaller ways- it could be as simple as going back and changing a variable name such as open (maybe a function?) to isOpen (almost always a Boolean value).
I think most of us are in this position pretty often where we’re writing and figuring out something at the same time, maybe we just see if it works first and don’t give a name much thought or maybe we change the actual value or type of a variable in the process and the variable name becomes ambiguous. I’d look for small indicators that shoe me that this person is considering readability. But I still don’t necessarily expect it to be as readable as it would be in a more natural setting I mean I think 90% of these sorts of things start off with the person saying “I swear I can type” within the first five minutes of being watched- if they get flustered while being watched that it effects their typing, it certainly also effects their coding as well.
You'd reject a candidate that is willing and legally able to work for free while also cloning themselves so they can pair program with every one of your employees at once?
Having "extremely basic understanding" of prime numbers immediately at one's command is important for approximately 0% of software engineering jobs. If you instant-fail a candidate for this, it says a lot more about you and your organization than the candidate.
Approx 0% of devs need to know what the earth is, but from lots of interviews I've given I've found consistent correlation between lack of basic knowledge and lack of ability to solve many things. It was so strong we found it much more cost effective to cut people early that didn't know at least a few of some standard knowledge items.
This is some really good advice here.
It's always a good idea to throw out all candidates that can't immediately recall what the first theoretical result of the rest mass of a Higgs boson was in the first paper describing was. Basic knowledge like this just correlates so well with ability to make proper decisions in API architecture.
I'd also save time and money cutting people that read as poorly as you're demonstrating.
Try actually measuring basic knowledge with competency at programming before thinking your opinion is better than measured data. Peer reviewed research finds similar results [1].
And yes, we tested all this carefully before enacting it. Interviews cost time and money, so giving 100% on every candidate despite quick signals is a waste of time and money that would be better spent on other candidates. If you want the best outcome then you allocate scarce resources based on expected returns, not on unfounded beliefs.
You probably do not have a child of 7 years old because they do not know at that age what is a prime number.
Second, basic math still that you never or rarely use or with very large time between usage might get rusty. You may understand the concept but not find the optimal solution. The way you are responding here shows quite a lot about how you are short sighted by instant-failing someone with a single question instead of trying to asses the whole person as much as you can. On you side, you are wasting opportunity to have a great person that could be a key player in your team by bringing other set of skill on the table.
> You probably do not have a child of 7 years old because they do not know at that age what is a prime number.
it's part of the curriculum for children of this age where I grew up (I did check)
> The way you are responding here shows quite a lot about how you are short sighted by instant-failing someone with a single question instead of trying to asses the whole person as much as you can. On you side, you are wasting opportunity to have a great person that could be a key player in your team by bringing other set of skill on the table.
it may also be the case that I have more in depth knowledge about the roles that I've interviewed candidates for
most recently: hiring people to work for quants
not instantly knowing that even numbers (other than 2) are not prime is a very strong signal
You're not testing for "basic math skills" here. What you're testing for is more like "immediately retrieves an irrelevant math fact after many years of having no need to think about it."
Look, if you think this sort of thing allows you to identify great candidates, good for you. But in my experience, not only is this kind of practice stupid on its face, but it leads to engineering orgs packed with people who are good at memorizing trivia but terrible at solving real problems.
I think the key problem here is that is is a bad programming question. If you know anything about prime numbers then coming up with an answer is trivial. If you expect a more optimized solution, then you are really only gauging the interviewee’s understanding of prime numbers. So effectively the interview is more about mathematics than it is about programming or problem solving.
You happen to remember a particular piece of knowledge, so you project that expectation onto others. Theory of mind.
> yes, we expect professional software developers to have basic maths skills
Skill != knowledge. "What is a prime number" can be looked up and understood by any competent programmer in <5 minutes.
> "what is a prime number" is taught to 7 year olds, it's not vector calculus
Then it's reasonable to expect that an interviewee would be able to learn it as well, given the same resources. It does not however follow that an interviewee would inherently have that knowledge, just because 7 year olds are taught it.
Bottom line is, you're making too many assumptions about complete strangers.
If they know "prime number" is some technical term to look up. They might confuse it with amazon prime or anything else depending on context. You waste time explaining, they get indignant they are supposed to coding not do maths, complete mess.
I'm mad at myself now that it has eaten 15 minutes of my time trying to come up with the right optimization. What's the trick? 2, +1, and then +2 from there on seems obvious but once you get to 9 is it worth building a list of nonprimes to skip?
Well it doesn’t double the speed, since anything with a factor of 2 undergoes only one loop iteration inside is_prime. It basically just saves a function call
Exactly, multiples of 2 can't be a prime, so skipping this saves a function call (that itself only does one comparison), so it is a very simple and relatively obvious optimization. But actually every multiple of a number per definition cannot be a prime, you can't just change the step size, but there are some nifty tricks to approach the problem from a different angle.
Common approach is to use square roots, this reduces the runtime. Recommend checking out project euler if you like solving hard math-code-o(n)-puzzles.
I didn't want to cheat by looking on S.O. but thanks ;)
Yes it makes sense (in the GPT code) that you'd only go up to i * i ... although looking at pythonic while: statements is just gross to me in this context, it would feel a lot more readable to say, e.g. in PHP:
for ($i=2;$i<sqrt($n);) {
$i+=($i==2 ? 1 : 2); //although the first one should just be outside the loop
}
There's lots of more advanced optimizations, which would be an interesting avenue for discussion in an interview, but the drain dead algorithm would just use +2 instead of +1
> The Code Llama models provide stable generations with up to 100,000 tokens of context. All models are trained on sequences of 16,000 tokens and show improvements on inputs with up to 100,000 tokens.
Edit: Reading the paper, key retrieval accuracy really deteriorates after 16k tokens, so it remains to be seen how useful the 100k context is.
Looks like they aren't releasing a pretty interesting model too. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane.
Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.
The paper states it was instruction fine tuned with synthetic data (LLM generated instructions) ala another paper (“Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor”).
The github repo associated with that paper is linked below. It links to the paper on arxiv, but also has some data in the repo.
I've seen this argued a lot but is it fact? OpenAI was able to train on data from other platforms and surely, those platforms weren't letting their data go if they could help it. Unless some new laws have been passed, I don't think OpenAI can legally prevent others from using their data to train models. OpenAI can't have their cake and eat it too. After all, any content generated by AI can't be copyrighted.
It is indeed a fact that OpenAI's Terms of Use do state that you can't use their service to develop competing models: Section 2.c.iii -
https://openai.com/policies/terms-of-use
Now of course, the terms are not the law (so don't govern the use of the generated data by any third party), they are an agreement between two parties. If you did click "agree" then that's a binding agreement and there could be legal/contractual repercussions (some of which are outlined in the terms).
That seems like a likely explanation, probably won't get into legal trouble for using an OpenAI model for a research paper but redistributing said model may be upsetting enough for OpenAI trigger a legal challenge.
Unnatural language used davinci-002 although that was a while ago, they only say "similarly" in this paper and don't specify what they used. I can't see a reason why they wouldn't be releasing it if the unnatural prompts were generated by LLaMA2-family.
In any case, replicating this training seems trivial and very cheap compute-wise for anyone who wanted to do it.
I thought this specific model was referring to self-instruction using both synthetic prompts (generated from few-shot in-context prompting of presumably some OpenAI model, the original paper used text-davinci-002) as well as synthetic code (presumably Code Llama 7 like for self-instruct) subsequently validated with execution?
The differences being it's not just training on unvalidated synthetic data and this specific method (per the unnatural questions paper) results in increased instruction diversity which confers some added advantage and I'm assuming explains the performance gain over the also synthetic self-instruct code?
I may be misunderstanding but this seems more nuanced than just training on synthetically AI-generated code and is more validating of synthetic instructions (i.e. low resource setting) rather than synthetic code (i.e. high resource setting).
Curious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding?
I'm pretty sure they don't do that, but for code the relevant relationship between two tokens is easy to determine with the semantics of the language alone (for instance you can say that tokens related to a local variable have no relationship with tokens outside), so it would lead to a sparse matrix in the transformer, reducing the cost of big contexts by a lot. But it would require language specific preprocessing, and whether you can make it fast is also dubious. I don't think it's been tried so far.
Copilot X is just their name for their project to bring AI to more areas of VSCode. I don’t believe they can use GPT-4 for completions because it’s a chat-optimized model. It seems that they are using something else, that blog post seems to imply it’s a custom-trained model.
I use GPT-4 for code completion all the time! There are many Neovim extensions[1][2][3] (and I'm sure there are many VSCode extensions) which call the GPT-4 API directly for code completion. I'm pretty sure the only reason that Microsoft might avoid using GPT-4 for Copilot is cost.
>Even the 7B model of code llama seems to be competitive with Codex, the model behind copilot
It's extremely good. I keep a terminal tab open with 7b running for all of my "how do I do this random thing" questions while coding. It's pretty much replaced Google/SO for me.
It was made available internally, I believe. So this is one of the many Meta engineers on this site —- after all, Facebook is now less hated than Google here ;)
No one who has been using any model for just the past 30 minutes would say that it has "pretty much replaced Google/SO" for them, unless they were being facetious.
They said 7b llama which I read as the base LLaMa model, not this one specifically. All of these LLMs are trained on Stack Overflow so it makes sense that they’d be good out of the box.
I've been using this or something similar internally for months and love it. The thing that gets downright spooky is the comments believe it or not. I'll have some method with a short variable name in a larger program and not only does it often suggest a pretty good snippet of code the comments will be correct and explain what the intent behind the code is. It's just a LLM but you really start to get the feeling the whole is greater than the sum of the parts.
I just don’t understand how anyone is making practical use of local code completion models. Is there a VS Code extension that I’ve been unable to find? HuggingFace released one that is meant to use their service for inference, not your local GPU.
The instruct version of code llama could certainly be run locally without trouble, and that’s interesting too, but I keep wanting to test out a local CoPilot alternative that uses these nice, new completion models.
There are a bunch of VSCode extensions that make use of local models. Tabby seems to be the most friendly right now, but I admittedly haven't tried it myself: https://tabbyml.github.io/tabby/
An M1 Max with 64GB of RAM allows me to run multiple models simultaneously, on top of stable diffusion generating images non-stop + normal chrome, vscode, etc. Definitely feeling the heat, but it's working. Well worth the investment.
A 7B model at 8-bit quantization takes up 7 GB of RAM. Less if you use a 6-bit quantization, which is nearly as good. Otherwise it's just a question of having enough system RAM and CPU cores, plus maybe a small discrete GPU.
You’ll need a bit more than 7GB (~1 GB or so), even at 8 bit quantization, because of the KV-cache. LLM inference is notoriously inefficient without it, because it’s autoregressive.
Some projects such as lmdeploy[0] can quantize the KV cache[1] as well to save some VRAM.
Speaking of lmdeploy, it doesn't seem to be widely known but it also supports quantization with AWQ[2] which appears to be superior to the more widely used GPTQ.
The serving backend is Nvidia Triton Inference Server. Not only is Triton extremely fast and efficient, they have a custom TurboMind backend for Triton. With this lmdeploy delivers the best performance I've seen[3].
On my development workstation with an RTX 4090, llama2-chat-13b, AWQ int4, and KV cache int8:
8 concurrent sessions (batch 1): 580 tokens/s
1 concurrent session (batch 1): 105 tokens/s
This is out of the box, I haven't spent any time further optimizing it.
6-bit quantizations are supposed to be nearly equivalent to 8-bit, and that does chop 1.5 GB off the model size. I think a 6-bit model should therefore fit, or if that doesn't, 5-bit medium or 5-bit small surely will.
There is always an option to go down the list of available quantizations notch by notch until you find the largest model that works. llama.cpp offers a lot of flexibility in that regard.
On Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.
Code llama Python is very interesting. Specifically tuned for Python.
I wonder if we could make such specific LLMs (one that is proficient in all things Rust, another- all things Linux, all things genomics, all things physics modeling etc) and have them talk to each other to collaboratively solve problems.
That would be a crazy future thing! Putting machines truly to work..
I think this is called "mixture of experts" and also there's a lot of speculation that it's how GPT-4 works, although probably with just a few large models rather than many small ones.
I think more attempts havent been made because base llama is not that great at coding in general, relative to its other strengths, and stuff like Starcoder has flown under the radar.
Start with a CodeLlama for C, and start treating these systems as natural language compilers. C is low level enough and still readable for those rare moments
The best model, Unnatural Code Llama, is not released. Likely because it's trained on GPT4 based data, and might violate OpenAI TOS, because as per the "Unnatural" paper [1], the "unnatural" data is generated with the help of some LLM -- and you would want to use as good of an LLM as possible.
TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090.
while it supports it, so far I've only managed to get infinite streams of near nonsense from the ollama models (codellama:7b-q4_0 and codellama:latest)
my questions were asking how to construct an indexam for postgres in c, how to write an r-tree in javascript, and how to write a binary tree in javascript.
I was up and running from clone/build-from-scratch/download in ~5m.
It's running on my M1.. it knows WebGL JS APIs better than I do, makes a passable attempt at VT100 ascii art, and well, should read more about Wolfram Automata, but does seem to know Game of Life!
They have a server/client model. The binary comes with a basic terminal front-end but you can just create your own self-hosted GUI or editor integration against the API[1]:
I haven’t quantized these myself since TheBloke has been the main provider for all the quantized models. But when I did a 8 bit quantization to see how it compares to the transformers library load_in_8bit 4 months ago(?), it didn’t use my GPU but loaded each shard into the RAM during the conversion. I had an old 4C/8T CPU and the conversion took like 30 mins for a 13B.
I can quantize models up to 70B just fine with around 40-50 GB of system RAM, using the GGMLv3 format.
GGUF seems not optimised yet, since quantizing with a newer version of llama.cpp supporting the format fails on the same hardware. I expect that to be fixed shortly.
For inference, I understand that the hardware requirements will be identical as before.
Replicate has already hosted Llama2 13B, the chat version. My guess is, in a short span of days or weeks they will host the code version too. They charge a dollar for 2000 generations if i am not mistaken.
“Normal laptop” is kind of hard to gauge but if you have a M series MacBook with 16GB+ RAM, you will be able to run 7B comfortably and 13B but stretching your RAM (cause of the unified RAM) at 4 bit quantization. These go all the way down to 2 bit but I personally I find the model noticeably deteriorate anything below 4 bit. You can see how much (V)RAM you need here [1].
To run Code Llama locally, the 7B parameter quantized version can be downloaded and run with the open-source tool Ollama: https://github.com/jmorganca/ollama
ollama run codellama "write a python function to add two numbers"
More models coming soon (completion, python and more parameter counts)
>The Code Llama models provide stable generations with up to 100,000 tokens of context.
Not a bad context window, but makes me wonder how embedded code models would pick that context when dealing with a codebase larger than 100K tokens.
And this makes me further wonder if, when coding with such a tool (or at least a knowledge that they’re becoming more widely used and leaned on), are there some new considerations that we should be applying (or at least starting to think about) when programming? Perhaps having more or fewer comments, perhaps more terse and less readable code that would consume fewer tokens, perhaps different file structures, or even more deliberate naming conventions (like Hungarian notation but for code models) to facilitate searching or token pattern matching of some kind. Ultimately, in what ways could (or should) we adapt to make the most of these tools?
You can, I suppose, contract your code so that it’s context free and uses less tokens, but that makes it more confusing for humans and language models.
Taken to the extreme, you can see obviously with one letter functions and variables like i, j, k the model will be able to infer literally nothing and, thus, produce arbitrary nonsense.
Clearly the solution is to do what we already do to manage complexity which is to decompose large tasks into smaller black box modules with an api where the (large number of tokens) implementation is hidden and not known or relevant to using it.
If you give an LLM a function signature and good description, maybe some usage examples, it doesn’t need the implementation to use it.
Terseness decreases the ability of LLMs to process code; it doesn’t solve context length, and even at best it doesn’t scale.
The process of decomposing the task into smaller steps and generate each step independently seems to be the correct way in my experience too. It works very well with GPT (chatGPT or GPT4).
>100k tokens is plenty.
The context window can be really helpful, in case there is a release of a new library and the user wants to generate code targeting the API of the library. When the training date stops at August 2023, any library released after that date is not known to the engine.
My general opinion in regards to context window, is that 1 trillion tokens context window still may not be enough for all use cases.
Your developer tool already maps out the entire code base in useful ways, such as knowing all the symbols available in the current context and the structure of classes. This information can be distilled for presentation to the LLM. For instance, if you’re wanting to generate a method implementation inside a C++ class, the LLM can be given a condensed version of the header files that the compiler would have access to on compiling that specific class. Removing white space and comments and boiling macros down saves a lot of tokens.
You can also probably skip including standard library headers since those will be well known to the LLM through its fine tuning.
Either way, consider that a typical preprocessed C++ file would push against the 100K limit even with some optimizations. You will definitely want to have some middleware doing additional refinement before presenting that file to the LLM.
I’ve found the utility of the coding LLMs gets a lot higher when you’ve got code comments and descriptive variable and function names - the LLM makes better inferences and suggestions. We’ve seen similar on data - properly tagged data and descriptive field names helps the LLM to produce much more useful responses. I’m secretly hoping the spread of these tools will finally lead my fellow developers to comment their code and stop using three character variable names.
This is my ultimate (short term) AI fear - letting it get into a feedback loop with itself, leading to perverse and incorrect results.
To state my position more clearly: I don’t think an AI could comment code from scratch very well - how would it know all the decisions made, business logic considerations, historical conventions, micro-industry standards, etc?
A good benchmark I was told once was “if a human expert couldn’t do it, an AI probably can’t either”. And commenting code I didn’t write would certainly test the bounds of my abilities
I built a VS code extension a while back that I still use that wraps GPT-4 and writes code directly in my editor.
The method I used to choose which files to feed GPT-4 was embeddings-based. I got an embedding for each file and then an embedding from the instruction + some simple processing to pick the files more likely to be relevant. It isn't perfect but good enough most of the time in medium-sized codebases (not very large ones).
The one thing I started doing because of how I implemented this is make files shorter and move stuff into different files. Having a 1k+ LOC file is prohibitive because it eats up all the context window (although with 100k context window maybe less so). I think it's a good idea to keep files short anyways.
There's other smarter things that can be done (like embed and pass individual functions/classes instead of entire files) so I have no doubt someone will build something smarter soon. You'll likely not have to change your coding patterns at all to make use of AI.
This sounds like a job for middleware. Condensing split code into a single huge file, shortening comments, removing whitespace and such can be done by a preprocessor for the llm.
You dont even need an llm for trimming whitespace, just a smart parser with language rules like ide code checkers already use. Existing llms are fine at summarizing comments, especially with language specific grammar constraints.
A good practice is to have a prompt file where you keep the information you want the model to have at its disposal. Then you put it in the start of your conversations with GPT-4. It's also good documentation for people.
You start a project by defining the task. Then as you iterate, you can add new information to the prompt. But it can be also partially automated - the model can have a view of the file structure, classes, routes, assets and latest errors.
I was really hoping that the one year update of Codex would be that - a LLM that can see deep into the project, not just code, but runtime execution, debugging, inspecting and monitoring. Something that can iterate like autoGPT. Unfortunately it didn't improve much and has weird conflicts with the native code completion in VSCode, you get freezes or doubled brackets.
Solutions exist that feed LLMS ctags, and seem to work well. The function signatures and symbols names for a code base are much smaller than the actual code.
Copilot has been working great for me thus far, but it's limited by its interface. It seems like it only knows how to make predictions for the next bit of text.
Is anyone working on a code AI that can suggest refactorings?
"You should pull these lines into a function, it's repetitive"
"You should change this structure so it is easier to use"
With Cody you can create embeddings for your entire repo, so Cody will have much greater context about your code base and the problems you're trying to solve.
Disclaimer: I just joined Sourcegraph a few weeks ago.
Cody is great, it had become my go-to (and I pay for Github Co-pilot).
With that said, they have recently changed the architecture, with the local install required, and I have not managed (yet) to get it working with NixOS. Once I have some more time, I will try again - it looks like there will be some hoops to go through. https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-appimageT...
Kudos to the Source Graph team, Source Graph's original product was nicely thought out and ahead of it's time. Nice to see how the original product gave a nice basis for building out Cody.
Hello, Sourcegrapher here! We are making some simplifications to eliminate the need for the separate local app install. Should be out in the next month or so. Until then, apologies for the added friction here, we're working hard to smooth it out.
Neither of those tasks require AI. IntelliJ IDEA will happily suggest both for you today, locally. It can find large chunks of duplicated code and automatically refactor them out to functions for you. And it has many inspections that suggest refactorings to make code clearer.
Last I heard they are in beta and don't work very well (even on the examples page: the "add types" brush is too strict, since `a` and `b` are checked for `null`, and the "fix simple bug" is a typo)
I've been using Cursor (https://www.cursor.so/) and it can do embeddings of the entire codebase, refactoring entire classes, etc. I had it rewrite a UI to add state to show one item at a time and have a selection list to the left and it executed it perfectly in MUI controls, first try.
Sounds like what's needed is a bit of tooling in the background consistently asking the LLM "How would you improve this code?" so you don't need to actually ask it.
I've used Ollama to run Llama 2 (all variants) on my 2020 Intel MacBook Pro - it's incredibly easy. You just install the app and run a couple of shell commands. I'm guessing soon-ish this model will be available too and then you'd be able to use it with the Continue VS Code extension.
Edited to add: Though somewhat slow, swap seems to have been a good enough replacement for not having the loads of RAM required. Ollama says "32 GB to run the 13B models", but I'm running the llama2:13b model on a 16 GB MBP.
Apple Silicon, especially an M1 Max Studio seems to be an interesting machine to hang on to as the models become more and more efficient with using less and less.
If there's nay other opinions or thoughts on this, I'd be very happy to learn as well. I have considered the eGPU route connected to a 1L PC such as a thinkcentre m80/90.
I have a 64 GB M1 Max MBP, and I'd say unless you really have some academic interest towards messing with open models, for now accessing SOTA models via a REST API has better latency for a given quality.
Claude 1.2 instant is as fast as 3.5, follows instructions at a quality closer to 4, and has a 100k context window. Hard to compete with that with an open source model right now.
How does open source compete with the Claude API? Easy: actually let you use the model. From the signup page:
> Anthropic is rolling out Claude slowly and incrementally, as we work to ensure the safety and scalability of it, in alignment with our company values.
> We're working with select partners to roll out Claude in their products. If you're interested in becoming one of those partners, we are accepting applications. Keep in mind that, due to the overwhelming interest we've received so far, we may take a while to reply.
No thanks, I'd much rather not wait months to see if my app deserves their oh-so-limited attention, or "aligns with the values" of a company taking $400m from Sam Bankman-Fried.
To be more charitable to your underlying point, Claude 2 is free to chat with via Anthropic's website, Poe, or Slack, and the GPT-4 API is open to use. If you're building a prototype or just need a chatbot, these do have better results and dev experience, at least for now. But I don't think picking on your Claude API example is unfair. These companies could randomly refuse your prompts via some opaque "moderation API" (that all GPT fine-tuning data goes through!), train on your company's proprietary data, spy on your most intimate questions, or just not find you worth the trouble and cut you off, at any time. THAT is why open source beats proprietary hands down: My device, my data, my weights, my own business.
Awkward tie-ins between SBF and value systems (?) have no effect on practical usage.
A theoretical concern they might train on my API data after saying they won't doesn't either. Amazon might be training on everything not bolted down in S3, not worth wasting brain power on that.
The moderation API isn't some magic gotcha, it's documented. They don't want to deal with people fine tuning for porn. Maybe you have some ideological disagreement on that but it's not of practical relevance when trying to write code.
At the end of the day you're not alone in these opinions. But some of us prefer pragmatism over hype. Until someone catches OpenAI or Anthropic trying to kill their golden goose by breaking their GDPR, HIPPA, and SOC2 certifications, I'm going to take delivered value over theoretical harm.
34B should be able to run on 24GiB consumer graphics card, or 32GiB Mac (M1 / M2 chips) with quantization (5~6bit) (and 7B should be able to run on your smart toaster).
Matthew Berman has a tutorial on YT showing how to use TheBloke's docker containers on runpod. Sam Witteveen has done videos on together and replicate, they both offer cloud-hosted LLM inference as a service.
If you want to run them fast, a 12GB GPU (e.g 3060) for the 13B and a 24GB GPU for the 34B (e.g 3090). Otherwise llama.cpp CPU inference would work on most machines.
How are people using these local code models? I would much prefer using these in-context in an editor, but most of them seem to be deployed just in an instruction context. There's a lot of value to not having to context switch, or have a conversation.
I see the GitHub copilot extensions gets a new release one every few days, so is it just that the way they're integrated is more complicated so not worth the effort?
You can use Continue as a drop-in replacement for Copilot Chat with Code Llama. We've released a short tutorial here: https://continue.dev/docs/walkthroughs/codellama. It should save you a lot of time context-switching; you can just highlight code and ask questions or make edits, all with keyboard shortcuts
This works well for me except the 15B+ don't run fast enough on a 4090 - hopefully exllama supports non-llama models, or maybe it'll support CodeLLaMa already I'm not sure.
http://cursor.sh integrates GPT-4 into vscode in a sensible way. Just swapping this in place of GPT-4 would likely work perfectly. Has anyone cloned the OpenAI HTTP API yet?
I was tasked with a massive project over the last month and I'm not sure I could have done it as fast as I have without Cursor. Also check out the Warp terminal replacement. Together it's a winning combo!
That’s just one perspective… Another perspective is that LLMs enable programmers to skip a lot of the routine and boring aspects of coding - looking up stuff, essentially - so they can focus on the fun parts that engage creativity.
But it won't stop there. Why would it stop at some arbitrarily defined boundary? The savings associated with no longer having to pay programmers the amounts of money that they believe they are worth (high enough to result in collusion between employers) are just too tempting.
Some form of AI will eventually take over almost all existing jobs. Whether those jobs evolve or not somehow and new jobs replace them, we will see.
But it's definitely not just programmers. And it will take time.
Society needs to adjust. Stopping progress would not be a solution and is not possible.
However, hopefully we can pause before we create digital animals with hyperspeed reasoning and typical animal instincts like self-preservation. Researchers like LeCun are already moving on from things like LLMs and working on approaches that really imitate animal cognition (like humans) and will eventually blow all existing techniques out of the water.
The path that we are on seems to make humans obsolete within three generations or so.
So the long term concern is not jobs, but for humans to lose control of the planet in less than a century.
On the way there we might be able to manage a new golden age -- a crescendo for human civilization.
Humans don’t become obsolete, we become bored. This tech will make us bored. When humans get too bored and need shit to stir up, we’ll start a war. Take US and China, global prosperity is not enough right? We need to stoke the flames of war over Taiwan.
In the next 300 years we’ll wipe out most of each other in some ridiculous war, and then rebuild.
I agree that WWIII is a concern but I don't think it will be brought about by boredom.
"Global prosperity" might be true in a very long-term historical sense, but it's misleading to apply it to the immediate situation.
Taiwan is not just a talking point. Control over Taiwan is critical for maintaining hegemony. When that is no longer assured, there will likely be a bloody battle before China is given the free reign that it desires.
WWIII is likely to fully break out within the next 3-30 years. We don't really have the facilities to imagine what 300 years from now will look like, but it will likely be posthuman.
I’ll go with the 30 year mark. Countries like Russia or China don’t get humbled in a loss (like Germany didn’t in WW1). Russia will negotiate some terms for Ukraine (or maintain perpetual war), but I believe it will become a military state that will funnel all money into the defense sector. The same with Iran, and the same with China.
Iran supplies Russia with drones. I can promise you Russia will help Iran enrich their uranium. They are both pariah states, what do they have to lose? Nuclear Iran, here enters Israel.
Okay, think about it this way. This thing helps generate tons and tons of code. The more code people (or this thing) writes, the more shit there is to debug. More and more code, each calling each other means more and more insane bugs.
We’re going to move from debugging some crap the last developer wrote to debugging an order of magnitude more code the last developer generated.
It’s going to be wonderful for job prospects really.
The only thing that takes anyone's job is demand shortfalls. Productivity increases certainly don't do it. It's like saying getting a raise makes you poorer.
Actually, in my country, Portugal, if your salary is the Minimum wage, you are exempt from paying taxes, but if you get a raise as little as 15€ or so, you move one category up and start to pay taxes, and you will receive less money than what you used to get before the raise.
One coachman to the other: "Another perspective about this car thing, you can skip all the routine and boring trips - they are done with cars. You can focus on the nice trips that make you feel good".
This should be the only goal of mankind so we can smell the flowers instead of wasting our years in some cubicle. Some people will always want to work, but it shouldn't be the norm. What's the point really unless we're doing something we're passionate about? The economy?
The best interpretation of this is you mean eventually ML/AI will put programmers out of a job, and not Code LLama specifically.
However it is hard to tell how that might pan out. Can such an ML/AI do all the parts of the job effectively? A lot of non-coding skill bleed into the coder's job. For example talking to people who need an input to the task and finding out what they are really asking for, and beyond that, what the best solution is that solves the underlying problem of what they ask for, while meeting nonfunctional requirements such as performance, reliability, code complexity, and is a good fit for the business.
On the other hand eventually the end users of a lot of services might be bots. You are more likely to have a pricing.json than a pricing.html page, and bots discover the services they need from searches, negotiate deals, read contracts and sue each other etc.
Once the programming job (which is really a "technical problem solver" job) is replaced either it will just be same-but-different (like how most programmers use high level languages not C) or we have invented AGI that will take many other jobs.
In which case the "job" aspect of it is almost moot. Since we will be living in post-scarcity and you would need to figure out the "power" aspect and what it means to even be sentient/human.
I understand the fear of losing your job or becoming less relevant, but many of us love this work because we're passionate about technology, programming, science, and the whole world of possibilities that this makes... possible.
That's why we're so excited to see these extraordinary advances that I personally didn't think I'd see in my lifetime.
The fear is legitimate and I respect the opinions of those who oppose these advances because they have children to provide for and have worked a lifetime to get where they are. But at least in my case, the curiosity and excitement to see what will happen is far greater than my little personal garden.
Damn, we are living what we used to read in the most entertaining sci-fi literature!
(And that's not to say that I don't see the risks in all of this... in fact, I think there will be consequences far more serious than just "losing a job," but I could be wrong)
If we get to the point where these large language models can create complete applications and software solutions from design specs alone, then there's no reason to believe that this would be limited to merely replacing software devs.
It would likely impact a far larger swath of the engineering / design industry.
Interesting that there's a 34B model. That was missing from the original Llama 2 release. I wonder if it's still usable for general non-code chat tasks or if the code fine tuning destroyed that. It should be the best model that would still fit on 24GB gaming GPUs with quantization, because 70B doesn't fit.
Theoretically this is an even better size, as it would fit on a 20GB-24GB GPU with more relaxed quantization and much longer context.
Metrics are slightly below 13B, but the theory is that the higher parameter count is more amenable to finetuning. If you search for 22B on huggingface, you can see that frankenllama experiments are ongoing:
Looks like they left out another model though. In the paper they mention a "Unnatural Code Llama" which wipes the floor with every other model/finetune on every benchmark except for slightly losing to Code Llama Python on MBPP pass@100 and slightly losing to GPT-4 on HumanEval pass@1 which is insane.
Meta says later on that they aren't releasing it and give no explanation. I wonder why given how incredible it seems to be.
Between this, ideogram.ai (image generator which can spell, from former Google Imagen team member and others), and ChatGPT fine-tuning, this has been a truly epic week.
I would argue that many teams will have to reevaluate their LLM strategy _again_ for the second time in a week.
SDXL and DeepFloyd can spell. It's more or less just a matter of having a good enough text encoder.
I tried Ideogram yesterday and it felt too much like existing generators (base SD and Midjourney). DALLE2 actually has some interestingly different outputs, the problem is they never update it or fix the bad image quality.
Editor plugins are fantastic about completing based on a pattern. That's the main thing you're missing out on imo - it's worth it to hit tab, but not to copy/paste and say "finish this line for me, it looks almost like the one above."
There's also the real-time aspect where you can see that it's wrong via the virtual text, type a few characters, then it gets what you're doing and you can tab complete the rest of the line.
It's faster to converse with when you don't have to actually have a conversation, if that makes sense? The feedback loop is much shorter and doesn't require natural language, or nearly as much context switching.
Seriously, I was expecting to read the article and them be on a level on-par with GPT-4 or higher. For all this chat of how long Google/Facebook have been in the AI space longer than OpenAI, their products don't speak to that..
I can't wait for some models fine tuned on other languages. I'm not a Python developer, so I downloaded the 13B-instruct variant (4 bit quantized Q4_K_M) and it's pretty bad at doing javascript. I asked it to write me a basic React Native component that has a name prop and displays that name. Once it returned a regular React component, and when I asked it to make sure it uses React Native components, it said sure and outputted a bunch of random CSS and an HTML file that was initializing a React project.
It might be the quantization or my lacklustre prompting skills affecting it, though. To be fair I did get it to output a little bit of useful code after trying a few times.
Anyone know of a docker image that provides an HTTP API interface to Llama? I'm looking for a super simple sort of 'drop-in' solution like that which I can add to my web stack, to enable LLM in my web app.
This is great for asking questions like "how do I do x with y" and this code <<some code>> isn't working, whats wrong? Much faster that googling, or a great basis for forming a more accurate google search.
Where its a bit shit is when its used to provide auto suggest. It hallucinates plausible sounding functions/names, which for me personally are hard to stop if they are wrong (I suspect that's a function of the plugin)
hallucinations can be resuces by incorporating 'retrieval automated generation' , RAG, on the front end. likely function library defs could be automagically entered as prompt/memory inputs.
Why wouldn’t they provide a hosted version? Seems like a no brainer… they have the money, the hardware, the bandwidth, the people to build support for it, and they could design the experience and gather more learning data about usage in the initial stages, while putting a dent in ChatGPT commercial prospects, and all while still letting others host and use it elsewhere. I don’t get it. Maybe it was just the fastest option?
But Yann LeCun seems to think the safety problems of eventual AGI will be solved somehow.
Nobody is saying this model is AGI obviously.
But this would be an entry point into researching one small sliver of the alignment problem. If you follow my thinking, it’s odd that he professes confidence that AI safety is a non issue, yet from this he seems to want no part in understanding it.
I realize their research interest may just be the optimization / mathy research… that’s their prerogative but it’s odd imho.
It’s not that odd and I think you’re overestimating the importance of user submitted data for the purposes of alignment research. In particular because it’s more liability for them to try to be responsible for outputs. Really though, this way they get a bunch of free work from volunteers in open source/ML communities.
What I found interesting in Meta's paper is the mention of HumanEval[1] and MBPP[2] as benchmarks for code quality. (Admittedly maybe they're well-known to those working in the field.)
I haven't yet read the whole paper (nor have I looked at the benchmark docs which might very well cover this) but curious how these are designed to avoid issues with overfitting. My thinking here is that canned algorithm type problems common in software engineering interviews are probably over represented in the training data used for these models. Which might point to artificially better performance by LLMs versus their performance on more domain-specific type tasks they might be used for in day-to-day work.
It's really sad how everyone here is fawning over tech that will destroy you own livelihoods. "AI won't take your job, those who use AI will" is purely short term, myopic thinking. These tools are not aimed to help workers, the end goal is to make it so you don't need to be an engineer to build software, just let the project manager or director describe the system they want and boom there it is.
You can scream that this is progress all you want, and I'll grant you that these tools will greatly speed up the generation of code. But more code won't make any of these businesses provide better services to people, lower their prices, or pay workers more. They are just a means to keep money from flowing out of the hands of the C-Suite and investor classes.
If software engineering becomes a solved problem then fine, we probably shouldn't continue to get paid huge salaries to write it anymore, but please stop acting like this is a better future for any of us normal folks.
It’s less of a concern if you are in mid career. But someone should warn all these college kids that are going into comp sci. I don’t think this will be the kind of lucrative field they think it’s going to be over the course of a 40 year career.
The days of getting paid well for making crud are numbered (which most of us do, even in the most interesting problem spaces).
# need a front end boilerplate that hits a backend with the following end points. REST api for movies catalogue, and a corresponding ui. Oh, unit tests please. Go with a responsive design and also make a React Native version (matter of fact provision it to my iPhone). Decide between Heroku or AWS, set up deploy with git hooks.
# scrape IMDb for initial population of the db
# I think a Reddit like comment system would be good to add, so add it. No upvote/downvote though
# handle user login with google/fb/email
# also make an admin page to manage all this
I guess the technical product designer will be the new unicorn.
You can say this about every major invention. The loom destroyed jobs! The engine destroyed jobs! So on and so forth.
This view is critically flawed in two major ways:
1) AI is not anywhere near being able to replace the majority of what developers do on a product team. We are decades away from a PM at Facebook being able to type "make a twitter clone that uses instagram login and can scale to 1 billion" and have an AI just do it.
2) Programming and product work is not zero sum. The more we can do means the more product we can make. It means more products can be made overall. After the loom came out, we simply made more clothes than ever before and in the process created a ton of jobs. We are not at some peak software point where we've completely saturated all humanity's need for software or profitable software and thus tools that increase efficiency don't put us out of work.
And frankly, if we develop the kind of general AI that accept a query like "make a facebook competitor capable of scaling to 10 billion" and simply do it, inventing whatever languages, frameworks, hardware, processors, patterns, methodologies, global datacenters handling global politics and law, etc, etc necessary to accomplish such a task, then so be it. I welcome the overlords!
I agree. This will ultimately become another way to extract more value straight into the pockets of the owners.
Unfortunately, I don't believe there's a way to stop (or even slow down) this train. We can't defeat it, so the only logical answer is to join it.
It's the classical issue with progress removing jobs. In today's world, since mostly everyone (aside from the capitalists themselves) relies on jobs to survive, barring a complete switch from capitalism (which will not happen in our lifetimes), we're fucked.
Next best thing we can do is to try and democratize it enough so that not only the rich have access to it.
The problem with 2 is that the people in power, and with immense wealth, remain there because of capitalism. They have the political power, and the resources, to enact the change ... but they also lose the most (unless you count altruism as gain, which of it were true the World would be so different).
We could structure things so that LLM, and the generalised AIs to come, benefit the whole of society ... but we know that those with the power to make that happen want only to widen the poverty gap.
Yes but the common man has won before! There has never been a perfect revolution/paradigm shift (personally anti utopia-through-intense-bloodshed, so hesitant to use the former term alone), but there have been many, all of which were against the wishes of those in power.
Plus, if these AIs are enough to change everything, that kinda implies that we've developed flexible, reliable AGI systems. In such a world, everything changes - maybe the calculus of The Powerful Few vs. The Oppressed Masses changes in too! It might even change in our favor, if we're terribly lucky...
Curious if there are projects to enable working with these things self-hosted, tuned to a git repo as context on the cli, like a Unix filter - or with editors like vim? (I'd love to use this with Helix)
I see both vscode and netbeans have a concept of "inference URL" - are there any efforts like language server (lsp) - but for inference?
I want "safety" to be opt-in due to the inaccuracy it introduces. I don't want to pay that tax just because someone is afraid I can ask it how to make a bomb when I can just Google that and get pretty close to the same answer already, and I certainly don't care about being offended by its answers.
Feels like we're like a year away from local LLMs that can debug code reliably (via being hooked into console error output as well) which will be quite the exciting day.
Have you tried Code Llama? How do you know it can't do it already?
In my applications, GPT-4 connected to a VM or SQL engine can and does debug code when given error messages. "Reliably" is very subjective. The main problem I have seen is that it can be stubborn about trying to use outdated APIs and it's not easy to give it a search result with the correct API. But with a good web search and up to date APIs, it can do it.
I'm interested to see general coding benchmarks for Code Llama versus GPT-4.
The 34b Python model is quite close to GPT4 on HumanEval pass@1. Small specialised models are catching up to GPT4 slowly. Why not train a 70b model though?
Given this can produce code when prompted, could it also be used to interpret html from a crawler and then be used to scrape arbitrary URLs and extract structured attributes? Basically like MarkupLM but with massively more token context?
This is probably a stupid question, but would it be possible to use these models to rate existing code and point to possible problems, rather than generating new code? That would be extremely useful to some use cases I'm working on.
Random tangential question given this is about llama, but how do you get llama.cpp or kobold (or whatever tool you use) to make use of multiple GPUs if you don't have NVlink in place?
Is there somewhere docs to show you how to run this on your local machine and can you make it port it a script between languages? Gpt4 can do that pretty well but its context is too small for advanced purposes.
Those charts remind me just how insanely good GPT-4 is. It's almost 5 months since its release and I am still at awe with its capabilities. The way it helps with coding is just crazy.
There are several different ways, but the easiest way in my (clearly biased) opinion is just got to ollama.ai, download it, and start playing around. It works out of the box w/ newer Macs, but there are versions for Linux and Windows in the works.
ggml quantization is very easy with the official llama.cpp repo. Its quick and mostly dependency free, and you can pick the perfect size for your CPU/GPU pool.
Some of the newer models have slightly different architectures, so he explains any differences and shows a llama.cpp invocation. Plus you can avoid pulling the larger dataset.
Llama is a very cool language model, it being used for coding was all but inevitable. I especially love it being released open for everyone.
I do wonder about how much use it'll get, seeing as running a heavy language model on local hardware is kinda unlikely for most developers. Not everyone is runnning a system powerful enough to equip big AIs like this. I also doubt that companies are going to set up large AIs for their devs. It's just a weird positioning.
12Gb of VRAM lets you run 13B models (4-bit quantized) with reasonable speed, and can be had for under $300 if you go for previous-generation NVidia hardware. Plenty of developers around with M1 and M2 Macs, as well.
Better reasoning and general performance than 13b by far (if llama1 was any indication), and like the other user said, can fit on a single 24gb vram gaming card, and can be peft fine-tuned with 2x 24gb cards.
Llama-1-33B was trained on 40% more tokens than LLama-1-13B; this explained some of the disparity. This time around they both have the same data scale (2T pretraining + 500B code finetune), but 34B is also using GQA which is slightly more noisy than MHA. Furthermore, there have been some weird indications in the original LLama-2 paper that 34B base model is something… even more special, it's been trained on a separate internal cluster with undervolted/underclocked GPUs (though this in itself can't hurt training results), its scores are below expectations, it's been less "aligned". Here, Code-Llama-Instruct-13B is superior to 34B on HumanEval@1. So yes, it's desirable but I wouldn't get my hopes up.
To a certain extent, I think it's just IBM disease. A company the size of Meta is expected to have an AI research department like Microsoft or Google, even if their core business (social media) derives relatively less benefit from the technology.
Pretend you're an uncreative PM on an AI team; what part of Facebook or VR could you feasibly improve by iterating on LLMs? Perhaps the content moderation system... but that would require wrangling with the company ethics comittee and someone else at the company probably already took ownership that idea. You've gotta do something compelling or else your ML engineers are going to run off somewhere else.
If I were to ask my ML engineers about what they wanted to work on, they're going to avoid areas where their model is outgunned (i.e.: chat) and instead prefer lower hanging fruit which generalizes well on a resume (i.e.: "Pioneered and published key innovations in LLM code-generation").
Of course, the alternative answer is that Meta wants to replace all of their jr. developers with GPUs, but I think their leadership is a little too preoccupied with VR to be actively pushing for such a transformative initiative in anything more than a very uninvested capacity (e.g.: "Sure I'll greenlight this. Even if it doesn't pay off I don't have any better ideas")
The ability to work less historically has always came as a byproduct of individuals earning more per hour through productivity increases.
The end goal of AI isn't to make your labour more productive, but to not need your labour at all.
As your labour becomes less useful if anything you'll find you need to work more. At some point you may be as useful to the labour market as someone with 60 IQ today. At this point most of the world will become entirely financially dependent on the wealth redistribution of the few who own the AI companies producing all the wealth – assuming they take pity on you or there's something governments can actually do to force them to pay 90%+ tax rates, of course.
Business opportunity: I'd pay money for NICE desktop software that can run all these different models (non-subscription, "2-year updates included, then discount pricing" modal perhaps). My wishlist:
- Easy plug & play model installation, and trivial to change which model once installed.
- Runs a local web server, so I can interact with it via any browser
- Ability to feed a model a document or multiple documents and be able to ask questions about them (or build a database of some kind?).
- Absolute privacy guarantees. Nothing goes off-machine from my prompt/responses (USP over existing cloud/online ones). Routine license/update checks are fine though.
I'm not trying to throw shade at the existing ways to running LLMs locally, just saying there may be room for an OPTIONAL commercial piece of software in this space. Most of them are designed for academics to do academic things. I am talking about a turn-key piece of software for everyone else that can give you an "almost" ChatGPT or "almost" CoPilot-like experience for a one time fee that you can feed sensitive private information to.
I work for a Fortune 100 company with 80,000+ employees. All of us are explicitly forbidden from using any sort of AI/LLM tool without written permission from the head of legal AND the CEO. In other words, nobody is going to get permission.
The concerns are 2 fold - 1. We might inadvertently use someone else’s intellectual property. 2. Someone else might gain access to our intellectual property.
What you are describing would help alleviate the concern about issue 2, but I’m not sure if it would help alleviate the concerns with issue 1.
It's basically the same thing in our company, too. They basically put a similar rule in place that prevents anyone from using e.g. Chat GPT. Little do they know that all software devs within the company are using co-pilot and the company is even paying for it. It's quite a funny situation tbh..
> Little do they know that all software devs within the company are using co-pilot and the company is even paying for it.
Just like annual sexual harassment training - it's mostly corporate CYA on liability. If it ever goes to court, they'll plead ignorance and blame the employees who should have known better as they were trained/informed on what they ought not to do.
Paying for co-pilot could bite them though, so I suspect it's a case were the one part of the organization isn't aware of what the other is doing
All of your assumptions are exactly right. They (mostly managers with little to no IT background) want to cover their own asses in case shit hits the fan (unlikely scenario if you ask me, because the company is just overrating the value of their data. Nobody gives a fuck about us anyway...) and many parts of this company have developed their own habits... The company is just very big and I can understand why they might be afraid, but come on, nobody will take that policy seriously forever. You need to eventually put some reasonable rules in place that allow you to make use of such Innovations...
Except, at my company they block software like that. Not only do they block it, but if you try to go to it a security person will immediately call your manager and ask what you are doing.
we try to eliminate this problem by using code models trained only on permissevely licensed code, then you can run them locally without sending code anywhere
Change company. Honestly. If you go as far as to forbid your partners in crime (workers sigh..) to explore new uncharted territory at all - well ya know someone will/might just win by not doing that.
This is particular, specifically problematic territory. I cannot imagine handing over proprietary data to a third party without a contract in place for how that data is stored and used. It’s not about innovation, it’s about using someone else’s tools without ownership. For the other case, it’s both about integrity in owning your own work, and a shield from legal consequences. These things should be very relevant to any business.
I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy. I am more bullish on LLMs than most of my coworkers. I think there is a lot of potential there. I do not see that potential in the current commercial offerings, and the financial outlay to fine-tune an open-source model and run it at scale is…prohibitive.
> I also don’t know any professional devs who have used tools like copilot and said they were anything but a toy.
Really? I'm academic now but I find Copilot at least moderately helpful when I'm writing a library. It's pretty good a lot of boilerplate functions, docstrings, regex, etc. I certainly don't want to go back to not using it, my code is a lot closer to production quality now and looks nicer.
Thinking back to my days in back-end it seems like it would have been very helpful/sped things up so I'm surprised to hear it's just a toy but I've been out of the professional game for a while now. What's the main criticism?
Agreed. After several rounds of setting up various python environments and tinkering with directory structures and debugging glitches and quantizing models just to end up playing around for a few minutes and getting bored, it would be nice to have the experience just be seamless. I wouldn't try to set up a workflow around seriously using what's out there to run on localhost now.
That said, non-subscription is essential, and that's probably going to be a heavy lift considering how quickly things are evolving.
I've been trying to push things in that direction with my LLM tool - the idea is to have Python plugins which you can install that do all of the irritating details to get a model setup.
I've not yet been able to solve the challenge of needing CUDA etc for some models though!
Cool! I've followed your instructions and your blog quite a bit as I've experimented with running local LLMs as well as stable diffusion. It's been especially helpful, as python is not my language or usual environment. Your patience at hacking your way through each new iteration and presenting what's important about them is astonishing; I personally think I'd have gone mad, but you've done great work in charting the territory.
Oobabooga is a great tool but it still has a long way to go in term of user-friendliness. It's absolutely not plug and play the way that chatgpt is; It requires research, trial and error, and knowledge of the tech to make the model work to its full potential. It's great once you finish setting it up, but it does not compare to what you would expect from a commercial product aimed at normal end-users.
Things like bad default values, no tooltips, an no curated model list to one-click download is what separates a tool like Oobabooga from a paid commercial product. These things require time/money and it would be very unlikely that an open source tool could find resources for all the testing and R&D.
I think there is a big market for products where you pay and can just start chatting with the model without having to ever go to the settings tab or google anything unless you need to do something out of the ordinary.
Looks really promising. I wonder if the similar pricing to OpenAI means that Gradient is also(?) bleeding money even if they get a good customer base. Or are these prices sustainable over time?
Yeah it's even cheaper. Although it looks like it's about the same in proportion to approx model size/expected quality? They haven't launched any >13B model yet, although they plan to.
we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE
Does anyone have a good explanation for Meta's strategy with AI?
The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?
They also don't have the same economic setup and DNA as MS/OpenAI. Large corporate customers don't pay for access to the FB cloud, nor are they likely to -- Ellison has spent years building out Oracle Cloud, and he's on the FB board, for example. And I bet you didn't think of using Oracle's Cloud for your last project.
So, your company DNA is free-to-all social based on ad monetization, with a large bet on metaverse / AR / experiential social compute being next. You aren't a trusted corporate partner for anything but gatekeeping your immense community through ad sales.
And, it's clear you a) have some of the most interesting private social data in the world, including photos and DMs and texts, and b) this AI thing is huge.
A play that doesn't f with your existing corporate structure too much is to build this stuff, give it away, keep publishing, build your AI team internally, and see where it takes you.
This isn't the only play, but I think it's reasonable. It's pretty clear large enterprises are going to need their own, internally built / owned, Foundation models to be competitive in a bunch of arenas in the next decade. In this case, if FB can get a little mindshare, keep the conversation going, and as a sidenote, be a disruptor by lowering Azure/OpenAI revs with open releases at-the-edge, that's probably a strategy win.
If I were in charge of AI strategy at FB, I'd probably double down more on generative AI, and I'd be working hard on realtime multimodal stuff -- their recent very large multimodal speech to text in multiple languages work is good. If a team could eyeball realtime-ish video chat with translations, that would be something the platform has a natural advantage in pushing out. Generative hits existing customers, and metaverse asset creation, which is going to experience radical changes in costs and productivity over the next few years, and impact Oculus 100% no matter what anybody wishes were true.
That’s interesting. I tend to lump FB, Amazon, Google, and MS in my head when thinking about the tech giants, but you’re right, FB is the only one of those not offering a commercial platform. For them, building out the capabilities of the LLMs is something to be done in the open with community involvement, because they’re not going to monetize the models themselves.
They’re also getting a fantastic amount of press from all this, which is good for attracting talent and helping improve their image, at least among the nerd set.
FB is unlike the other BigTech(tm) since Zuck never sold out and has a controlling equity stake. Amazon, Google, and MS are all controlled by and beholden to institutional investors.
FB can release these for no other reason than Zuck’s ego or desire to kill OpenAI. Same deal as him going off on a tangent with the Metaverse thing.
Given that OpenAI finished training GPT4 a year ago, and no models today (including these) can beat it, I highly doubt anyone is capable of killing Open AI in the near future. I’m guessing by the time GPT5 is out, someone will finally catch up with GPT4.
Depends what you mean by platform and depends what you mean by FB. If by FB you mean Meta, they have also https://www.workplace.com/ (which is like an internal facebook), instagram, whatsapp and some others. Integration of LLMs technology in those "platform" might give them some advantage.
Right, but they’re not competing directly on offering the LLM - they benefit from having a better LLM as a feature, but their value add is elsewhere in the product.
Meta absolutely could not overcome the barriers to entry and technical mismatch for any sort of traditional IAS style product, and it would be foolish for them to try. They might be able to pull off some sort of next generation Heroku style service aimed at smaller shops with built in facebook integration and authn/z management, but that's tangential.
I don’t believe they’re going for the same hosted monetization as Oracle or Google. I’m sure they’ll play around with assistant AIs but you can imagine them leveraging their graph and data for this.
Who is better positioned to answer a question like, “What should I get my friend Sophia for her birthday?” Facebook/Instagram already have huge volumes of data to specifically target ads. They can feed those into a chat interface pretty easily.
Customers would then buy per impression by describing their product and trusting Facebook to place it correctly. They already do this today, it’s just a different medium.
> Who is better positioned to answer a question like, “What should I get my friend Sophia for her birthday?” Facebook/Instagram already have huge volumes of data to specifically target ads. They can feed those into a chat interface pretty easily.
Interesting idea but sounds risky and intrusive in practice.
I think this suggestion lacks subtlety. More likely, around the time leading up to Sophia's birthday, you will see more ads for things (maybe even gift idea ads) that just so happen to be things Sophia would love (at least, according to their data).
Yes. But it also needs a very different org structure to support that. Their internal infra from what I heard is dated (monolithic PHP binary deployment, no federated authorization management etc.). It is doable (FAIR's org structure was very different in the first a few years), but would also be a distraction for a long time.
I would add that having open source gen AI will enable the creation of content for metaverse / AR / VR, which will improve the chances that all of that will take off.
Right, exactly this. Ratcheting the costs down two orders of magnitude in both dollar and expertise/human costs is going to make huge changes. You better believe FB is thinking about this hard.
Yeah, there are tons of opportunities for AI to do something with facebooks private user data and sell new services. For users to create engagement - and for ad companies to get very good targeted ads delivered. It is of course a challenge, to update the models on the fly, to include the latest private data, but then you can tailor an ad, that has subtil references to the latest shared wishes of the user. Probably quite effective.
So for now they mainly need top talent, to make some of it work. And open source is the best bet, for creating a ecosystem they can control and get talents who already trained on their tools. And they loose allmost nothing, because yes, they ain't in the cloud buisness.
So I will continue to not use facebook. But the models I will try.
You ought to think about using Oracle Cloud for your next LLM/GPU project, because they sell access to A100/H100s for cheap and they actually have them in stock!
Commercially it's not clear if there is a reliable "ahead", I'd be surprised if copyright lawsuits don't start hitting MS/OAI when publishers wake up and if you take out that training data where does it leave their models?
Countries putting copyright above AI progress will just fall behind. It's one thing to demand no exact replication of copyrighted content, another to forbid training on copyrighted works. Ideas were not supposed to be under copyright, only expression, from what I remember.
The argument that copyright abuse is required for "AI progress" is sus. It is required for quick easy buck to be made by the likes of Microsoft-- that I agree...
Clearly the research team at Meta knows the domain as well anybody, has access to a data trove as large as anybody and their distribution capability is as large scale as anyone's.
If their choice right now is not to try to overtly monetize these capabilities but instead commoditize and "democratize" what others are offering it suggests they think that a proprietary monetization route is not available to them. In other words they do not leave money on the table. They think that (at least right now) there is no money on the table that they can get to.
Rather than remaining quiet and isolated, the best alternative - their conjectured thinking goes - is to show up as they do, buying up good will with various stakeholders, maintaining mindshare internally and externally etc.
Assuming that the above reading is correct it still leaves various options as to why they may have come to that conclusion: For example reasoning about the future of this sector they might be thinking that there is no real technical moat and they simply accelerate that reality to gain some brownie points.
It may be also idiosyncratic reasons specific to their own business model (data privacy challenges and how any AI monetization will mesh with all that). The drawback of being the elephant in the room is that there is not much room to move.
The nature of their long game depends on which of the decision branches carries more weight. Maybe it is wait-and-see until others clear up the regulatory hurdles. Or keep the engines running until the real and irreducible added value of LLM algos and the like becomes clear.
There really is no technical moat. Any new architectures are going to be published because that's 100% the culture and AI folks won't work somewhere where that's not true. Training details/hyperparameters/model "build-ons" aren't published but those are a very weak moat.
The only moat that is meaningful is data and they've got that more than any other player save maybe google. Publishing models doesn't erode that moat, and it's not going anywhere as long as facebook/whatsapp/instagram rule "interactive" social.
Well Facebook is a walled garden, perhaps the board hopes free highly capable LLMs will continue degrading the internet outside those walls thus acting as a moat for their money printer.
Retention project to keep their top ML/AI staff engaged and not straying away?
Working towards NLU that can solve content moderation once and for all? Contrast with tiktok which is clearly using word filters that are easily worked around with phrases like "un-alived" or "corn".
They want to replace influencers and your friends with chatbots and keep you scrolling through an infinite feed of ads and AI generated content?
There has been some shuffling of seats but from what I am hearing FAIR is the best setup as far as staffing and funding that they have been in quite some time. Mark is pivoting hard to stay competitive in AI and is providing the resourcing to do so, the results speak for themselves.
As a business strategy I would see it as preventing themselves from being hemmed in from the market leaders. By open sourcing and raising the bar for commodity AI, they get to crowd source improvement to their models and techniques to get ahead in their own uses by co-opting open source improvement. I would say to sage this is working amazingly well - the amount of interest around open source models from meta is immense. I also think, as do I, the majority of uses in the future will be from fine tuned RAG capable models embedded in devices, not pangalactic planet sized computers running septillion parameter models. Llamacpp is a perfect illustration of where that’s working.
We followed a similar model under more duress at Netscape. When you use Firefox that’s the fruit of that effort. It didn’t save Netscape, but Meta has a better and more diversified revenue base.
I watched a good talk from Yann LeCun who is Chief AI Scientist at Meta, and he explained that the thinking is that open source AI models will be the long-term winner, so it's best for them to work in that arena.
Yann wants to cement his position as a leader in AI and while he clearly does not appreciate LLMS at all, he realizes that he needs to make waves in this area.
Mark needs a generative product and has invested tremendously in the infrastructure for AI in general (for recommendation). He needs researchers to use that infrastructure to create a generative product(s).
Yann sees this going on, realizes that he has a very powerful (research+recruiting) position and tells mark that he will only sign on if Meta gives away a good deal of research and Mark concedes, with the condition that he wants his generative product by end of 2023 or start of 2024.
It’s not just ego. It’s accelerationism. Giving this stuff away from free is probably going to accelerate AI a decade faster than if it was kept locked up behind closed doors at Google, OpenAI, etc. And if you’re an optimist then that actually might make the world a better place much faster.
I wish that Meta would release models like SeamlessM4T[0] under the same license as llama2, or an even better one. I don't understand the rationale for keeping it under a completely non-commercial license, but I agree that is better than not releasing anything at all.
There seem to be opportunities for people to use technology like SeamlessM4T to improve lives, if it were licensed correctly, and I don't see how any commercial offering from smaller companies would compete with anything that Meta does. Last I checked, Meta has never offered any kind of translation or transcription API that third parties can use.
Whisper is licensed more permissively and does a great job with speech to text in some languages, and it can translate to English only. However, it can't translate between a large number of languages, and it doesn't have any kind of text to speech or speech to speech capabilities. SeamlessM4T seems like it would be an all-around upgrade.
Yeah - different projects have different goals and licenses aren't one size fits all. Depending on the project, type of technology, goals, etc.. we will select or even develop the right license that aligns with those goals. Hope this helps :)
Facebook Connect is what used to be called Oculus Connect. Kinda their equivalent of Apple's WWDC, I guess. It's when and where the Quest 3 will be officially unveiled in full, for example.
Probably just talent acquisition. As Google and OpenAI start sharing and publishing less, they become less attractive to scientists. No scientist wants to fall into a black hole and not publish for 8 years.
Exactly. The Google and OpenAI engineers who published their groundbreaking research 5 years ago are now rockstars. Those who create great research but can't share it often get frustrated.
The problem is also companies bragging about AI, but not releasing anything behind (like most of the recent Google announcements).
If nobody except the researcher can reproduce an AI paper, and there is no source-code, and no demos that the public can access, then it's almost like if it doesn't exist.
I wouldn't want to work in a company that would throw away my research and just use it for PR purposes.
Meta has a clear channel to leverage generative AI in profitable ways in their ads. At some point in the probably not so far future, everybody's going to have custom ads generated for them that are optimized to get that particular person to click/buy/etc. Those will convert well, and the better ads convert, the more businesses will be willing to pay Meta for a given ad.
This compares favorably with Google, which is as likely to cannibalize its search business with generative AI as to create new value for itself.
Thus, for all the gen AI stuff like this, for which Meta doesn't have an obvious path to commercialization, it makes sense to release it publicly. They get plenty of benefits from this - for one, engineers (and smart people generally) who are working on really complex problems like to be able to talk about the work they're doing. If you're picking between jobs at Meta and Google, the fact that Meta's going to release your stuff publicly might well be the deciding factor.
I would also argue that there's an economic incentive. Right now, being seen as an AI company is definitely a positive for your multiple. I think the movement of Meta's stock price over the last 12 months relative to their change in profit and revenue is certainly driven in part by the perception that they're a leader in AI.
It makes sense to me that Facebook is releasing these models similarly to the way that Google releases Android OS. Google's advertising model benefits from as many people being online as possible and their mobile operating system furthers that aim. Similarly, Facebook's advertising model benefits from having loads of content being generated to then be posted in their various products' feeds.
> Does anyone have a good explanation for Meta's strategy with AI?
Yes. I said it many times. Meta is already at the finish line in the AI race to zero. All the other cloud-based AI models cannot increase their prices given that a $0 free AI model is available to be self-hosted or used on-device for private / compliance reasons.
Cloud-based AI models cannot afford to compete with free or close to free. It costs Meta close to nothing to release a readily available $0 AI model which is good enough for most use-cases that ChatGPT has already done.
> The only thing I've been able to think is they're trying to commoditize this new category before Microsoft and Google can lock it in, but where to from there? Is it just to block the others from a new revenue source, or do they have a longer game they're playing?
Mostly benefits the PyTorch ecosystem which Meta has an active community around it.
vessenes and rvz kind of sum the idea I think they're going for to me.
AI has no moat, but many players are in denial about this still. Microsoft and other might have tight enough control they can use a product dumping strategy to get people dependent upon their implementation such they can start charging, but that isn't a delusion Meta can have.
That max revenue license they used with the models seemed fairly clever to me. It will seed the environment with players that base their product on Meta tech in return for them being born with a poison pill preventing their use by big players (other than Meta) buying them. This is a long term play that may not really work but it creates the potential for big opportunities. And even if it doesn't work out, denying easy wins for their powerful competitors might be worth the price on its own.
I posit it is similar to how Adobe lets students pirate Photoshop, because when they join the workforce that is what they know and need their employers to buy Adobe services, which for corporate is very expensive.
Meta by democratizing AI access is generating more capable developers which will make the Metaverse a reality, where FB leads. They have already realized they have a losing gambit with Google, Apple, Microsoft (also X?) having an antagonistic monopoly against Meta product advancement
the only beneficiary of this are the hardware vendors.. nvidia and amd. and startups which get these foundation models for free.
because language models are a complementary product, and the complement must be commoditized as a strategy.
I see AMD as a bigger beneficiary, since, very soon, amd will equal nvidia for inference and fine-tuning, but amd has a long way to go to equal in foundation model training.
> and startups which get these foundation models for free.
It's licensed non-commercially, so I'm not sure what those startups stand to gain.
> since, very soon, amd will equal nvidia for inference and fine-tuning
Source? If you're referring to Olive, it is indeed impressive but also has caveats:
1. It is just as proprietary as CUDA or CoreML.
2. You need a copy of Windows and licensed DirectX to use those optimizations.
3. AMD only matches Nvidia's inferencing performance when comparing Olive to Pytorch. Olive-to-Olive comparisons will still reflect an Nvidia lead.
I don't think AMD has the capability to equal Nvidia in the short-term. It will take longtime software investments from across the industry to shake Nvidia's yoke.
Microsoft is a hosting partner, there's an Azure service for hosted private LLaMa inference for business. Being a go-to hosting provider for SoTA AI is of course a very good thing for Microsoft.
Here's some output from q4_0 quantization of CodeLlama-7b-Python (first four lines are the prompt):
It will be interesting to see how the larger models perform, especially after community tuning and with better context/prompting.