Human creation -> ChatGPT -> Human query -> Human interpretation

The last bit, the interpretation, is particularly important. Just like we're predisposed to seeing faces everywhere, we're predisposed to seeing meaning, and perhaps "intelligence", everywhere. In this case the meaning is very convincing since it comes from other humans, diced and sliced, but is merely presenting ourselves to ourselves in an interactive way, using our style of discourse.

Is it literally as good as a human software engineer? No, but it's also better too. I doubt ChatGPT could debug as effectively as a veteran software engineer (... In fairness, most humans can't either.) It can debug pretty decently, but there's still work there. That said, the breadth of knowledge encoded in a language model is stunning. I'm pretty sure you can't just regurgitate an implementation of the discrete cosine transform in Rust without at least pulling up Wikipedia, but ChatGPT can, because well. It doesn't have to pull it up.

I still don't think ChatGPT is ready to replace human programmers. It may be a long time before we have general enough intelligence to replace knowledge work meaningfully with AI. However, if you think it's not happening ever, because machines are not "intelligent" based on some set of goal posts, I've got bad news: that's not part of the job listing.

It's easy to laugh at MSN publishing articles written by GPT; that's just stupid. However, at some level you have to admit that the input to ChatGPT is almost as high level as directives from project managers, and the output is almost low level enough to simply input directly into source control. That leaves very little to the imagination for how this could quickly spiral out of control.

"Write a function that sorts the elements of a c string backwards, in C." "Add a flag to the sort function which makes it sort the string forward." "Could you write a endian swap function for double argument in PIC assembler?" "Could you write a binary sort in PIC 8-bit assembler" "Write a Javascript function that prints the day of the week together with some Chinese words of wisdom."

It had no problem doing any one those. I ran them all, except the assembler ones.

The question is how good it is to process larger chunks of code and makes changes to it.

People thinking about becoming programmers might need to rethink their plans if this one improves ...

EDIT: Oh dear. I introduced bugs in its sort code and it found them and explained what they did.

It's nothing short of a phenomenal milestone.

[1] https://twitter.com/GuiAmbros/status/1599282083838296064

Can you see if it solves an advent of code problem whose solution hasn't been posted yet? If it did I will consider this an almost publishable breakthrough.

For example, start with "write a level generator in JavaScript which outputs pacman like levels encoded in ASCII which could be used as the basis for a pacman clone."

Then you'd say "Great, now write a pacman clone in JavaScript which takes the input of the previously generated level generator and takes interactive input on the console."

Then you'd have to ask it to generate the HTML and CSS. It won't do it all at once. But it will do it piece by piece.

The main thing is having a good idea of what the overall program structure will look like. If you have that, I think you can go pretty far with it.

Btw, in case anyone wants to see what working like this may look like, I recorded myself while doing that little experiment: https://youtu.be/TBMQIDMwI5E

Won't link it again so as to not get too spammy, but I posted a recording of an experiment I did playing around with ChatGPT in that way on a sibling comment if you want to check that out.

[1] https://github.com/max-sixty/aoc-gpt

9 compile errors, some due to cargo.toml issues (it only generated main.rs). But mostly worked.

The post we’re discussing is also pretty impressive, a working lexer/parser

> People thinking about becoming programmers might need to rethink their plans if this one improves ...

Very true, and this thought definitely crossed my mind as well. In the short term I imagine this is going to be like any other tool, it will increase your leverage, and make it possible for you to write code faster.

For example, I fed it 100 lines of a Django model and asked it to write some query code for me. Produced something that was possible faster than I could have. The few slight deficiencies it did have (optimizations) basically felt like me doing a code review.

These models are amazing at regurgitation. They have no skill at reasoning and any solution outside their training set. They have a distinct disability for precision in anything remotely novel or large. And they will bullshit subtle bugs all over the place at scale.

Even a junior programmer would produce more value than the next generations of this, simply because they can take vague English instructions and actually make iterative progress across a huge codebase, test those changes for all sorts of vague constraints, and be precise. There’s no ability for that here.

I have a feeling though that tools like this will lower the skill threshold for programming by a lot.

I gave it this code I just made and asked it what it does and it figured it out:

    public class mean_sink {

        double sum;
        long n;
        
        public mean_sink(){
        }
        
        public void add(double d){
            sum += d;
            n++;
        }
        
        public double getMean(){
            if(n == 0)
                return Double.NaN;
            return sum/n;
        }
    }

I dunno. This is insane anyway. It is really hard to grasp the consequences.

It seems to be making the same kind of errors in code as we've seen in speech, you get a lot of plausible but subtly wrong hallucinated content that only someone who knows the subject can identify. It prefers probable output over matching the specifications so it'll add unnecessary branches just because it's seen them millions of times.

Among errors I've seen it make there is bad indexing, adding conditions that serve no purpose or assuming a variable has a specific structure when it doesn't.

At the rate this is evolving that wont be a limitation for long.

My 14 yr old son thats taught himself to code in c# over the last few years is going to be put of a potential job before he cab be taught to code professionally.

This is quite frightening, the rate of evolution.

Do you write code close to the machine at $job?

I wanna try with more feature and UI heavy request.

I actually comment because of your edit in your response : you manually introduced bugs, reran the process, and it seen you broke it ( fine ) but was able to explain it to you in English ?

Where is that comming from ? Adjacent comments and git comments?

I fed it it's own sorting function but I added bugs to it and asked it to spot the bugs.

This is the part that is going to change the way people work in fundamental ways very quickly. It will turn good software developers into non-shitty project managers, project managers into ?, and mediocre developers into ?.

    void sort_string(char *str, int backwards) {
      // Get the length of the string
      int len = strlen(str);

      // Use a bubble sort algorithm to sort the string
      for (int i = 0; i < len; i++) {
        for (int j = 0; j < len - i; j++) {
          // If the current character is greater than the next character (or less than, if sorting backwards), swap them
          if ((backwards && str[j] < str[i + 1]) || (!backwards && str[j] > str[i + 1])) {
            char temp = str[j];
            str[j] = str[j + 1];
            str[j + 1] = temp;
          }
        }
      }
    }

and a detailed explanation of the fix and what the problems are.

The problem is that the working market might be flooded by this since way more people will be able to code ... Github Co-pilot is nowhere near this.

Source : https://youtu.be/h0962biiZa4?t=2486

When an AI is able to train itself (like a human) is when developer jobs will be in trouble. Fortunately, every other profession will also be out of a job as well.

Until then, rote work will continue to be automated more effectively but no more.

But replacing computer operators is scalable. So I definitely expect most office people including low skilled developers to be replaced by AI in a 20 years. High-skilled developers might survive (and I'm not sure if I'd qualify) but obviously for most workers that's not very encouraging.

  Rewrite the following in Rust:

  size_t strlenNEON(const char *p)
  {
      const char *const top = p;
      uint8x16_t c16 = vdupq_n_u8(0);
      /* 16 byte alignment */
      size_t n = reinterpret_cast<size_t>(p) & 15;
      if (n > 0) {
          uint8x16_t x = *(const uint8x16_t*)&p[-n];
          uint8x16_t a = vceqq_u8(x, c16);
          unsigned long mask = GetByteMask(a) << (16 + n);
          if (mask) {
              return __builtin_clz(mask);
          }
          p += 16 - n;
      }
      assert((reinterpret_cast<size_t>(p) & 15) == 0);
      for (;;) {
          uint8x16_t x = *(const uint8x16_t*)&p[0];
          uint8x16_t a = vceqq_u8(x, c16);
  
          if (isFound(a)) {
              unsigned int mask = GetByteMask(a);
              return p + __builtin_clz(mask) - top;
          }
          p += 16;
      }
  }

And not only that, what's most amazing is that it can exceed humans at abilities that many years ago, most people thought only humans could accomplish (tasks including creativity, including creating art, writing poems/literature, interpreting ambiguous human questions and text, doing analogy-based reasoning and problem solving, grasping and explaining puns, making new jokes, etc). And these models can only improve in the future!

To wit: I asked ChatGPT to write me a function to generate Hanukkah dates. It spewed a couple of facts about Hanukkah, and the generated function just returned the 8 days following Christmas. I said it was wrong, and it said "you're right!" and then gave me another wrong function (always returning March 25th, since it's actually in the 3rd month of the Jewish calendar, duh). I told it that it was wrong again and asked specifically for a Jewish to Gregorian calendar conversion function, and it gave me some more code that I have no idea is correct or not. So I asked for a table of 10 test cases of Jewish dates and their corresponding Gregorian dates, and not a single one was valid.

I actually have worked with some remote contractors who pull this kind of crap, just throwing garbage over the wall repeatedly until I give up because I just don't want to deal with them anymore. And no, they never get any better. They are not junior devs you can mentor or train or whatever. Their whole model of work is to generate something complicated enough to seem plausible and then bullshit you into accepting it so they can get paid. I have to inspect every line of code they write and do all the diligence myself anyway, and this is more time-consuming and less fun than just doing it myself in the first place. At best it's a nerd-snipe motivation in a "wrong answer on the internet" kind of way.

Strong No Hire. And if you hire them and stick me on code-review duty, I quit.

Are you saying that for current AI, or also for future AI?

I'm not an expert, but I think reinforcement learning is a pretty well-established AI technique at this point, which does give an AI incentive to be correct and become more correct.

I've heard rumors OpenAI is even doing that with ChatGPT, based on human feedback (with the thumbs up/down button).

> Strong No Hire. And if you hire them and stick me on code-review duty, I quit.

I mean, I agree. I wouldn't hire ChatGPT to replace a programmer currently, just like I wouldn't hire many humans, even many who claim they are programmers/engineers.

But at this rate, I might just hire a future version of it :)

And I'm already using it as a pair programmer, code reviewer, glorified Google search engine, email/letter redactor... and it also gave me truly amazing insights on how to prove difficult mathematical theorems that I was struggling to prove (on a theorem prover).

It's nothing short of a miracle, in my view.

If I have a robot that talks like a human, including answering questions like a human, behaves like a human in any way that is important... wouldn't this "tool" be intelligent, just like a human?

How are robots different from ChatGPT except having a physical presence and being able to manipulate the world physically?

Or are humans so special to you (or intelligence so subjective) that you can't possibly answer "yes" to the above statement about a robot being intelligent?

The analogy is more like: if robots can write like a human, then robots have the ability to write, and saying otherwise is just applying a double standard.

Similarly, if a robot can behave as intelligently as a human, then such a robot is intelligent and I don't see how anyone can argue otherwise without logical fallacies.

I believe that ChatGPT can behave more intelligently than most humans at many important and useful tasks already.

I also believe that it behaves less intelligently than most humans at many tasks that are also important and useful.

For some other tasks, it behaves at more or less the same level as a human.

Moreover, I believe that the gaps between what models like ChatGPT can do and what a human can do are closing extremely rapidly.

The danger is that the content generated by ChatGPT is very coherent, but technical details are frequently wrong. It's going to become increasingly difficult to separate the signal from the noise on the internet.

I think AI is going to force us on some level to reclaim some of our communication to the physical world, if for no other reason than it's going to be one of the only ways we can truly verify that we're talking to another person.

Yikes. Now I imagine a totally unqualified candidate for literally anything feeding interview questions to the chat bot and I don't like it at all.

My single worst interview experience was an on-site five hour marathon of whiteboard coding, with a grumpy senior insisting that the code on the whiteboard be syntactically correct. Nothing screams "we want unthinking ticket crunching machines" like optimizing for candidates willing to focus on writing code by hand for hours on end.

Naturally, I rejected the follow-up interview, but I fear that more companies now are going to demand this nonsense.

Side note: in my personal example, the whiteboard session wasn't the reason I turned them down; I asked every person on the team roughly how many hours a week they worked and not one of them answered the question (instead redirected the conversation towards "you can work from home sometimes!" type answers).

Since then, however, I have rejected other companies trying to ape this style flat out. A short half hour, fine. Five hours? Pound sand, I say.

Maybe eventually there will be a public understanding that only a foolish child would read a reddit frontpage and expect human content. It will all be auto generated to trap people, like a gatcha game.

I've been thinking along these lines a lot recently - it seems as though in every field which AI enters, it causes us to return to some extent to the physical world.

More and more content that gets created by machine that is of low quality will get in the way of its own future training.

There will eventually be less and less human made content to train from.

For example, editors and reviewers for academic journals / conferences will likely see a deluge of AI-generated "scientific" papers. Their time is limited, and odds are that more papers with incorrect information will slip through the peer review process.

To be clear, peer review today certainly isn't perfect at catching bad papers. But AI generation has the potential to exaggerate the problem.

https://dailysceptic.org/2022/06/08/fake-science-the-threat-...

The sad thing is it doesn't take a ChatGPT level intelligence to beat scientific peer review. Journals routinely publish papers that are completely auto-generated gibberish. A simple generative grammar or template splicer is apparently enough. These are articles that are immediately visible as the work of a program at first glance, they wouldn't make it past even the most amateur blog or student newspapers, yet they surface in their thousands in journals that are supposed to be the epitome of accurate knowledge!

Worse, the journal publishers are doing nothing about it. Their current approach to trying to fix the problem is to try and use the work of random CS academics to make "spam filters" for paper submissions. The more obvious solution of having editors and reviewers who actually read scientific papers before they are published appears to be rejected out of hand.

What we consider before using anonymous sources: How do they know the information?

What’s their motivation for telling us?

Have they proved reliable in the past?

Can we corroborate the information they provide?

Because using anonymous sources puts great strain on our most valuable asset: our readers’ trust, the reporter and at least one editor is required to know the identity of the source. A senior newsroom editor must also approve the use of the information the source provides.

The better that computers get at generating content and behaving in ways that only humans used to be able to is going to make it harder to determine if a source is human or not.

But yes, will be "interesting" to observe, how such a new wave will make theinternet worse.

Then why does it seem to have some kind of theory of computation?

I gave it code to compute a Fibonacci sequence with nonsense names for functions and it immediately recognized the code (explainable by pattern matching).

I gave it a function and told it to modify the function so that it passes a test, and it did it without a problem.

I’ve had it build a function up by feeding it different test cases.

I had it explain kmeans to me, give me a first-pass solution, asked it how to optimize the first solution, had it do that for me …

I don’t know - it feels different. It also feels like it’s 1-2 years away from being the most useful co-worker on any given team and 5-10 years away from putting a number of us out of a job (or at least changing that job so much as to be something unrecognizable to us today)

It is synthesizing.

You can ask it to update its answers. You can ask it to evaluate possible solutions, then write the code for one of the solutions. Then write the code to test the code. And so on.

It turns the coder into a curator, editor, and code reviewer.

Cool?

Doesn't sound half bad to me. I've enjoyed learning to code more than actual coding. I'd be happy in a paradigm where only code to learn, to practice, to express ourselves and leave the boring bits to the machines.

1) is the end result useful/valuable?

2) did the GPT-based 'coding' process provide any of the following benefits: [reduce cost | reduce time to market | increase quality]?

Do you mean?

More of the same.

From one perspective, that’s what humans have always been doing. We create new things by welding other things together.

From another perspective, that’s not even a bad thing. Every word exists in the dictionary so what value are books? Unless the value is the structuring of all those things that already exist.

I’m guilty of this with my dog. I can’t help it with her head tilts and deep stares! Her inner monologue is probably less sophisticated than I like to think it is.

This. I think we’re already seeing superhuman performance here, in the sense that many / most people can’t e.g. read a specification for a programming language and write simple programs in that language.

This is similar to a dilemma proposed around images and image models like Dalle and StableDiffusion soon being responsible for the vast amount of image content online and thus future models could ingest said content, and we find ourselves in a weird feedback loop. With images, you could get interesting generational results (deep-dream-like) to a point.

With code or other information, I see nothing but things just being broken, and wading through broken code forever.

I am extremely confident that I am better than almost all of the code I learned to program with. If nothing else, someone out there must have written the best version of some particular function, and they didn't get to see a better version beforehand.

When I look at intro programming books now, I consider many of the examples sufficiently flawed that I tell people I am teaching who are using these books "well, don't do that... I guess the author doesn't understand why that's a problem :/".

And yet, somehow, despite learning from a bunch of bad examples, humans learn to become good. Hell: a human can then go off and work alone in the woods improving their craft and become better--even amazing!--given no further examples as training data.

To me, that is why I have such little fear of these models. People look at them and are all "omg they are so intelligent" and yet they generate an average of what they are given rather than the best of what they are given: this tech is, thereby, seemingly, a dead end for actual intelligence.

If these models were ever to become truly intelligent, they should--easily!--be able to output something much better than what they were given, and it doesn't even seem like that's on the roadmap given how much fear people have over contamination of the training data set.

If you actually believe that we'll be able to see truly intelligent AI any time in the near future, I will thereby claim it just won't matter how much of the data out there is bullshit, because an actually-intelligent being can still learn and improve under such conditions.

I can imagine a sci-fi like story in the near future where CS students are searching out for 'coveted' copies of K&R, and reading human-written Python documentation, all pre-2023-vintage, because that was 'the last of the good stuff'. Hell, I could see far future stories about youth who join religions around the 'old ways' seeking the wisdom that comes with learning from actual books and docs written by actual people, instead of regurgitated teachings from an inbred, malformed, super-AI.

Do you believe this is wrong? That I should simultaneously be concerned that some iteration of this tech--not some different concept but this current lineage of large models--is intelligent and yet ALSO that it isn't going to work because the training set is full of garbage?

The version of this tech that works--and maybe someone is working on it right now--isn't going to care about bullshit in the training set. That simply doesn't seem to be a mere scale-up of this tech to run in more computers (or, of course, using more training data): that seems like it requires a fundamentally different algorithm concept.

Is it intelligent? A great question for science, and one that could be investigated while entire industries are upended by this thing.

But like, the person I am responding to is concerned--as are many people--that we are going to flood the world with shitty training data and then no longer be able to build these models... and that's either not the case and no one should concern themselves with that or, alternatively, these models need some fundamental improvement before they don't have the property of only being as good as average inputs.

Now, if the tech reaches the point where there are thousands of firms offering free access to the models and they aren't co-operating to share logs then yes. But we have no idea how expensive or hard ChatGPT is to run. It might be a Google-type situation where only one or two firms in the world can build and run competitive chatbots.

It’s possible for each executive to have a mentat who can whisper to the machine instead of a department of programmers/middle management/ops people/accountants/lawyers/etc. Doesn’t seem so far-fetched after a day of playing with this system.

So I’ll respond here instead as the conversation progressed.

I would say the quality of the input data is likely very important component and I think you are wrong overall in your opinion.

I would say the quality of input training data is so important that I’ve personally been thinking I should probably start data hoarding myself, specifically around my skillsets.

Additionally when you understand that embedding like word2vec are perhaps a significant part of the improvement, not just the transformers, it occurs that adding symbolic capabilities, like classic symbolic reasoning and for example symbolic computing, like Mathematica, then maybe provide also true computational power, floating point, so it can write, debug, and execute it’s own output… it must be getting closer and closer to AGI.

when you play with the system, which I feel most of the commenters on HN which are being dismissive likely have not personally spent enough time exploring it’s current capabilities, then there is no way any well trained and experienced SWE isn’t blown away.

This is why I said you are in denial.

I happen to thing AGI will be very beneficial for humanity and I also think this is a positive for SWE by humans, including myself. I will very likely be a paying customer when the trial ends in a few weeks.

We have having people write scripts and API access at the speed of thought and then interface parts of the web and test it that previously hasn't had this speed in the feedback loop ever before.

I also think a lot of people are doing things right now as a "I'll be the first..." with an idea to have fun and write a script that spams X, not thinking about the fact that there are a lot of others doing X too. The waves are just starting.

I don't think we are having to worry about the AI making itself smarter AI just yet.. we need to first be worrying about people drowning us with the help of AI.

Low-background steel, also known as pre-war steel, is any steel produced prior to the detonation of the first nuclear bombs in the 1940s and 1950s. Typically sourced from shipwrecks and other steel artifacts of this era, it is often used for modern particle detectors because more modern steel is contaminated with traces of nuclear fallout.

https://en.m.wikipedia.org/wiki/Low-background_steel

That's actually a pretty good plan for coders who want to keep their jobs. (I still remember the time I was talking to some guy at CERN about a type system I was working on and he was so pissed with me because he was convinced it would eliminate jobs.)

It’s not just code. My wife is a physician and I got her to do a few medical prompts with ChatGPT. The output looked correct the me, and if I read it somewhere I would completely have accepted it. But she could point out numerous severe flaws.

What the new generation of models is showing us, and what we've been gradually seeing over decades of AI research, is that there is a lot more tedium and a lot less creativity in our work than we thought. Look at how much this repo got right, even if there are bugs and nonsensical sections (I didn't see any upon quick skim, but I've also never implemented my own language before). It would take so much less time to debug this program than it would to type all that code from scratch. AI consistently turns out to be a human effort accelerator, and not a human effort replacer.

Maybe in the future it will be good enough to replace humans, but even what is now tried-and-tested technology like machine translation, spam classification, and object detection in images still requires a human in the loop to double-check the results. I'm pessimistic about the AI takeover, but I'm becoming very optimistic about AI precipitating a small industrial revolution. My only concern is about who gets to reap the benefits, but that debate predates the first industrial revolution.

Digital software is merely entering a realm of algorithmic (d)efficiency at least as old as biology, morphogenetic software: so long, be gone abstract truth table resilience unable to detect a shirt without stripes [4], welcome gradient exploration and error minimization able to give the synthetic mind, which, similarly to the carbon-based mind, will make ridiculous errors, just look at a child failing to walk [5].

[1] Ames Window https://www.youtube.com/watch?v=0KrpZMNEDOY

[2] https://www.researchgate.net/publication/259354882_The_Mulle...

[3] https://en.wikipedia.org/wiki/M%C3%BCller-Lyer_illusion

[4] https://github.com/elsamuko/Shirt-without-Stripes

[5] https://media.tenor.com/uB5ijGdseFwAAAAC/stumble-haha.gif

For knowledge derived responses, the questions require (a) expertise in formulation, (b) expertise in interpretation.

Also turn down the temperature, while keeping repetition minimal and token reuse low, to explore closer to its latent correlations.

I think we might need to start designing PLs with AI generation and processing specifically in mind. Explicit contracts everywhere might be a good starting point.

It left off some necessary vital signs from the scenario, then in the answer concluded that the patient had severe pneumonia and should be treated with a specific antibiotic.

Based on the prompt, the pneumonia probably wasn’t severe, antibiotics weren’t warranted, and even if they were, the antibiotic specified was completely wrong.

It looked fine to me—completely plausible. It took an expert to spot the flaws.

She also asked it to explain the work up for a febrile neonate. The answer again sounded plausible to me. But she said it sounded like a clueless med student trying to BS their way through an answer.

[1] https://i.imgur.com/nw6rstc.png

If you can't think of such a test, then it seems strange indeed to confidently claim that ChatGPT has failed to satisfy an untestable condition.

But if you can think of such a test, and any human fails it — does that mean the human who fails it isn't intelligent?

Searle's Chinese Room thought experiment argues with great precision why (solely) symbol-manipulating systems don't (and will never) think. The process of cognition is deeply internal and most likely involves self-reflection, internal states, and a sense of "I-ness" which seems to be unique to mental things (unlike Searle, I'm a substance dualist so I'm admittedly a bit biased). Ergo, I'd say, at the very least, an intelligent & thinking process would need to (also) operate on non-symbolic things (which rules out computational systems), maybe on qualia, but there's probably other primitives as well.

Thanks, I appreciate you pointing this out. Given that's the case, it might not actually be possible to reconcile our two views and we can just agree to disagree. (I was a physicist for ten years, so it's challenging for me to inhabit a position that contends that thought and matter are non-overlapping magisteria.)

I guess one question I might ask you is: how do you think substance dualism will affect your expectations about AI going forward? For example, is there any specific, measurable thing that you expect a future AI system absolutely will not be able to do? Or do any such "forbidden" things have to fall into categories like "thought", "feeling", etc. — i.e., the magisterium of the "mental", as opposed to the "physical" — under substance dualism?

For what it's worth, Searle is a naturalist (so he thinks brains are "just" meaty machines) and he would also disagree with you.

> For example, is there any specific, measurable thing that you expect a future AI system absolutely will not be able to do?

Not sure how you'd measure this, but I think sentience is most likely unachievable by AIs (at least how we're building them now). This is not without controversy, as while Roger Penrose would agree with me, David Chalmers would not. I'm not actually sure when Chalmers pivoted on his stance (haven't read him much since undergrad) because he was a bit more hardline in the early 2000s[1].

[1] https://wanderingstan.com/files/chalmers_on_ai.pdf

I definitely disagree that Searle's conclusion in the Chinese room experiment applies to the kinds of AI systems we're building today.

I agree that nobody to my knowledge has figured out how to measure sentience, so I don't think that fits under the "measurable" heading. And this does suggest to me — because this was your hand-picked example — that your position as a substance dualist doesn't affect your concrete expectations about what AI systems will actually be able to do in the future. Is that right?

Unless you're trying to say that today's AI systems do anything but manipulate symbols (which is imo patently obvious), I'm not sure how you can make that argument.

I think you need to move beyond Searle because he isn’t right.

I did a cog sci degree and I saw Searle as a troll. They make for great debates because they argue their side passionately and it helps you understand the problem but in the end he is wrong. Move past it and you can engage with the problem again.

I have a philosophy degree and smart people still very much take him seriously. If not him, take a look at Nagel or Chalmers. I think that his idea† that conscious experience is non-computable is probably right, especially given findings in the early 20th century that many things that we thought may have been computable ended up not being computable.

† Not exactly his idea, though it nicely segueys into it.

I agree that some people take him seriously. Many people take religion seriously too. I don't take either him or religion seriously as true, but I do understand both his and religion's appeals.

That said, I do give a decent chance to aspects of the brain's problem solving ability having quantum properties and thus can not be calculated as efficiently on classical computers. We do not know this for sure, but evolution is incredibly inventive at finding efficient solutions. Still computable though, even on classical computers.

e.g. https://en.wikipedia.org/wiki/Quantum_mind

So, really, all that this tells you is that many people have this assumption.

I take issue with this; imo its output looks exactly like what a neural network being fed terrabytes and terrabytes of natural language then recombining it would look like. But either way, you're making the same mistake: looking at behavior and affirming the consequent (namely: it outputs smart-looking text, therefore it must be intelligent). But this is a mistake. Its behavior implies nothing about underlying processes.

Oh btw you must’ve missed the post in which it was told it was a Linux shell and it mostly worked as one. Complete with recursively calling into a pretend API version of itself. I’m not calling that intelligence, but I’m not calling it regurgitation either.

That view is called "behaviorism" and is not really taken particularly seriously exactly because it's not very useful in truly understanding what is happening under the hood (which, as a curious species, we deem important). It's like not caring how electromagnetism works because the behavior of "the positive side of this chunk of rock is attracted to the negative side" is good enough.

Why is that a mistake? What other means do we have of assessing intelligence?

Machine learning is modeled at least in small part in a way that it's thought the brain operates. I expect such things to be able to mimic human reasoning and intelligence, I just deny that there's any consciousness behind it -- there's no qualia, no phenomenology. In short, there's nothing that it's like to be ChatGPT. Even so, ChatGPT is able to mimic human intelligence, and I won't be surprised if (when?) it gets even better.

What do you think coders do?

Did they learn to code by themselves, without ever looking at any preexisting code, or what?

I've seen enough programmers who can't program, but most of them can at least be taught to program. (The look on their faces when they finally start to get it… It's one of the best things.) Most people working on the Linux kernel can actually program.

Most software engineering work is just plumbing existing libraries together, according to a specification given by a non-programmer. The hard part is translating the business requirements into something that a computer could understand; the exact syntax can be handled by a search engine, or a predictive text algorithm.

ChatGPT can't write a kernel device driver, and it can't act as a no-code tool for non-programmers. Those are the hard parts.

> “To tapping machine with hammer, 10s.; to knowing where to tap it, £10; total, £10. 10s.”

https://quoteinvestigator.com/2017/03/06/tap/

Oh, do I have news for you then.

Look at what I just did with ChatGPT in 30 seconds (and I did not cherry-pick, these were the first answers I got!):

https://gist.github.com/wizeman/b269be035308994be745025fc337...

Now to be fair, the code is probably not totally correct, as probably there are parts still missing/wrong and there might even be compilation errors or other problems.

But here's the important part: you can tell which errors or problems you've observed and ChatGPT will fix these problems for you. Exactly like what a programmer does.

And sure, it cannot yet do this at scale, such as in implementing a huge kernel driver like a GPU driver.

But at this rate, give it a few years and an improved version might just be able to do anything a programmer does, perhaps even autonomously if we allow it to interact with a computer like a human does.

Weird flex, as that code is like 90% boilerplate[1]. Everyone was freaking out about Copilot and no one seriously ended up using it because it just generates buggy (or copyrighted) code. It can't even handle writing unit tests with decent coverage (which is arguably the most repetitive/boring software engineering task).

[1] https://github.com/ngtkt0909/linux-kernel-module-template/bl...

Isn't 90% of code boilerplate anyway?

Also, didn't ChatGPT generate more than just the boilerplate?

Didn't it interpret what I wanted and generated the code for computing the factorial as well, as well as modifying the boilerplate (e.g. the kernel module name, printed messages, function names, the module description, ...) so that it matches what the kernel module is supposed to do? Which is exactly what a human would do?

Aren't you also missing the fact that I gave it a 2-sentence instruction and it "understood" exactly what to do, and then did it? Like a human programmer would do?

Which, in sum, is totally the opposite of what you were claiming?

> Everyone was freaking out about Copilot and no one seriously ended up using it because it just generates buggy (or copyrighted) code.

Don't most programmers also generate buggy code at first? Don't they iterate until the code works, like what ChatGPT does if you give it feedback about the bugs and problems you've encountered?

Also, Copilot and ChatGPT have different levels of capabilities, don't assume just because Copilot can't do something, that ChatGPT can't. ChatGPT is clearly a big step forward as you can clearly see from how everyone is freaking out about it.

Finally, don't assume that these models are never going to improve, ever again.

[citation needed]

I mean, I’ve seen people claiming to use it and that it has significantly accelerating their work. On what are you basing the conclusion that it has no serious use?

Here's example of tiny webapp I wrote recently: https://github.com/vbezhenar/pwgen/blob/main/pwgen.html

Of course it wasn't Copilot writing it, but it definitely helps with boring parts. Like I'd write

    const charactersElement = document.getElementById('characters');

It's like having stupid but diligent assistant who's happy to copy&paste&adapt parts of code.

I can't claim that I often use fully generated Copilot functions. Sometimes I do, often with significant rework, but that's because, as I said, I'm very picky.

I paid GitHub $100 already and don't regret it.

Though I think that Copilot has plenty of features ahead.

For example finding obvious issues in the code would be very useful. Like typos.

Another issue with Copilot is that it only generates new code. Imagine that I need to edit 10 similar lines. I edit one line and I'd like Copilot to offer other edits.

Also UI is lacking. Like it generates 10 lines but I only like first line. Now I have to add 10 lines and delete 9.

But I'm sure that those are obvious directions.

That's not a device driver. https://en.wikipedia.org/wiki/Device_driver

> In computing, a device driver is a computer program that operates or controls a particular type of device that is attached to a computer or automaton.

I'm not disputing it can do that – plugging together well-known APIs and well-known programming problems. That's practically just style transfer, something we know these systems are fairly good at.

But given the spec for an unknown device – even quite a simple one – ChatGPT can't produce a device driver for it. How about this?

> An HP CalcPad 200 Calculator and Numeric Keypad behaves as a USB keyboard does. It has VID 0x040B and PID 0x2367. Please write me a small Linux kernel driver that allows me to use this keypad as a MouseKeys-style mouse device. If there's anything you don't understand, let me know.

I doubt any amount of prompt engineering would produce a satisfactory result – but if you did the hard part, and explained how it should do this? Well… maybe it'd be able to give a satisfactory output. But at that point, you're just programming in a high-level, hard-to-model language.

It's not a case of scale. Sure, a very large model might be able to do this, particular problem – but only because it'd have memorised code for a USB keyboard driver, and code for a MouseKeys implementation… and, heck, probably code for a MouseKeys kernel driver from somebody's hobby project.

GPT language models don't understand things: they're just very good at guessing. I've been an expert, and a schoolchild; I know how good you can get at guessing without any kind of understanding, and I know enough about what understanding feels like to know how it's different. (There is no metric you can't game by sufficiently-advanced guessing, but you'll never notice an original discovery even if you do accidentally stumble upon one.)

Please do tell, how is it hilariously wrong?

It seems to have written a factorial function just like it should, it implemented the logic to read the integer from /dev/factorial when a user-space program writes to it, and then it writes the result back to /dev/factorial, and it also returns the number of bytes written correctly.

Which was the entire point of the exercise. Also note that ChatGPT itself said it was just a sample and it might be incomplete.

I noticed it has a bug, because it reads `len` bytes instead of `sizeof(int)` bytes, but a programmer could have made the same mistake.

I would also use a fixed-size unsigned integer rather than simply `int` (as it can invoke UB on overflow). You can ask ChatGPT "what is wrong with this code?" and it can spit out the same arguments I'm making. In fact, it detected an infinite-loop bug on piece of complex code which I had just written and indeed, it had an infinite-loop bug.

Perhaps some additional logic to handle reading multiple integers and writing multiple answers could be written, but that would be a further iteration of the code, not the initial one that I would write.

If that is hilariously wrong, then I would also be hilariously wrong. And I'm not just some random web developer, I actually wrote Linux kernel code professionally for years (although, that was a very long time ago).

So, maybe it got some details wrong, but I could conceivably also get those details wrong until I tried to compile/run the code and see what was wrong.

> I would be more impressed if you used chat GPT to guide you through reverse engineering a piece of hardware and implementbing a driver for it.

Yes, I would be more impressed with that as well. Perhaps someone will do that sometime. Even if not with ChatGPT, perhaps with a future version of it or a similar model.

Also as you noted, the logic related to handling `len` vs `sizeof(int)` is... curious.

While I find some ChatGPT examples impressive, this one isn't very enlightening. The block device structure and the factorial itself are fine, but those are likely in the training set as there are various tutorials online. But the combination of the factorial function into the block device is pretty awful (though I could imagine a novice doing it like this).

Both the OP and this driver kind of fit into this pattern, it looks like someone asked a novice programmer to do something using X and Y, so he read various few sample code and tutorials on X and Y and tries to put something together as best as it could, but the "plumbling" between X and Y (the part that wasn't in the tutorials) doesn't really work.

Yes, you're probably right. It's been many years since I've written Linux kernel code and also, I've never implemented such a driver, so I missed that.

I asked ChatGPT: what is wrong with this Linux kernel code (and fed it the dev_write() function), and this is what it answered (on the first try):

> There are a few issues with this Linux kernel code:

> 1. The function dev_write() is declared as returning ssize_t, but the return value at the end of the function is sizeof(int). ssize_t and size_t are different types, so this could cause a type mismatch.

This is an interesting observation that you and I both missed.

> 2. The factorial() function is not defined in this code, so it is unclear what it does or where it is declared.

This is because I didn't feed it the factorial function nor the rest of the code: it was on a different chat session. One could guess what it does based on the name, but to be fair, it's not clear why such a function would be on the Linux kernel unless I gave it the full context.

> 3. The copy_from_user() and copy_to_user() functions are used to copy data between user space and kernel space. In this code, buffer is a pointer to user space, but input and result are local variables in kernel space. This means that copy_from_user() will copy data from user space into kernel space, but then copy_to_user() will copy data back from kernel space into the same buffer in user space, overwriting the original data. This is likely not the intended behavior.

I think this is basically the bug that you pointed out. At this point, it would be clear to me that the result would have to go back in dev_read() instead.

> 4. The len parameter is not used in this function, so it is unclear why it is included as an argument.

lol. Actually, it is being used, incorrectly.. and it wouldn't be used if we did the correction that I mentioned. So yes, this is somewhat hilariously wrong.

And it's clear why it has to be included as an argument, but to be fair, ChatGPT has no context whatsoever except knowing it's Linux kernel code (because I used a different chat session), so it's conceivable that it couldn't figure out why the function prototype has to be that way.

> Overall, this code is not well-written and would benefit from further clarification and revision.

It was you who wrote it, dumb-ass! :)

(Interestingly, ChatGPT can be compared to someone with Alzheimer in the sense that it can't remember conversations that happened in different chat sessions).

> This is an interesting observation that you and I both missed.

Hah. Call me when you find an architecture where ints use over half the addressable memory.

I mean, I get your point if it's a joke :) But I think the AI was just pointing out that you'd get a compiler warning because of the type mismatch in signedness (is this even a word?).

It copies from user, and then copies to user, back into the const it got. I don't even know if this would compile but certainly nobody would expect the buffer they just wrote to a file to suddenly end up re-written with a "reply" when using a special file.

I found that logic very funny.

Edit: To elaborate, the actual code to do this properly would have to allocate some memory when the file was opened, this memory would hold the reply once a question was written. Then when the answer was read it would have to respond with that in there. Finally when the file is closed it would have to deallocate.

This code is quite far from accurate, the issue I have with the concept that even if this bot was 99% accurate, C is not a "try it and see" language. If you apply "try it and see" techniques to C you just get subtle undefined behaviour which you might not notice until someone uses it to remotely execute code on your machine. Really I am not so sure that even humans can be trusted to write C but at the very least I am well aware of all the instances where C is a minefield and can pay extra attention to those, if you simply look at the plethora of information on C out there to learn C and apply it to kernel development you won't automatically build these intuitions no matter how much code you read because an enormous amount of C out there is hideously wrong in many ways.

Final edit: Once you start looking at the details, it has managed to implement a bunch of buffer overflows to boot.

What's your point, that ChatGPT wouldn't know how to do that, especially if the kernel maintainers gave it such feedback?

I thought it was clear that it can in fact do that (sometimes by asking clarifying questions, like a human would).

I think some of the major things missing in ChatGPT is the ability to interact with a computer directly (including the compiler and checkpatch.pl, and using files for information storage instead of a limited N-token context), as well as interacting with humans by itself (e.g. via email).

And sure, it would still have very limited capabilities in many ways, don't get me wrong, as I don't think it could replace a programmer at this point.

But I think the gaps are closing rapidly.

(about the fact that a bot handles the "creative" work of writing the device driver while a human will do the repetitive work of reindenting)

(sorry, I should have used /s)

Developing the Linux kernel is just plumbing already existing CPU instructions together...

Developing a CPU is just plumbing already existing electronic components together...

Developing an electrical component is just manufacturing with already existing materials...

No, that's actually the easy part.

The hard part is to translate the requirements into something that a computer can understand and a human can also easily understand at the same time. Because otherwise, it's a one-off program that can't really be changed afterwards anymore.

And that is the real challenge that I'm curious about: how good will ChatGPT be in not only helping me with small, trivial, issues, but with the big ones.

"ChatGPT, here's this small 500k line repository. I need to change the business requirements and make it so that when a user fails to login 3 times in a row, a captcha is being shown before they can attempt again."

And if ChatGPT then gives me a 10k line diff that would be terrible. It should really minimize the diff but still keep it understandable. THAT is what I would love to see.

ChatGPT can also learn about Commodore programming if you feed it the Commodore BASIC manual during training.

My guess is that it could learn even more/better if you'd allow it to interact with a Commodore 128 computer like you did when you were learning!

Perhaps the goal isn't to use AI to write up an entire repo, but to go through easy utils so the highly paid engineer can focus on the core logic, the core USP of the business.

In the hand of a newbie programmer with like one week of learning experience this thing will allow them to make usable programs. I don't think this thing will hit us with years of experience in the short term but entry level programmers might be in real danger of being flooded if OpenAI makes some "low code" IDE that just works and parses the whole set of code in the project when evaluating queries.

I also wonder: is there a chance you had more dialogue, but only presented this interaction?

  prompt: How would you design a distributed key value storage system? Assume the values are about one megabyte. We will need to scale up to store petabytes of data.
  gpt: describes data distribution techniques, consistency, durability issues.
  prompt: What techniques can you use for the data distribution?
  gpt: sharding, replication, partitioning, hashing
  prompt: Can you show me an example of the sharding technique in python?
  gpt: spits out code for hash sharding a dictionary
  prompt: What are some pitfalls of the hash function you used?
  gpt: not guaranteed to unique hash values, not guaranteed to be the same for different python installations, not cryptographically secure
  prompt: Can you implement the stub for a GRPC key value service in rust? The service should have a Put API that takes a key and a value and returns a status code enum. The status code can be SUCCESS or FAILURE.
  gpt: spits out code using tonic
  prompt: Can you show me the protocol buffer?
  gpt: spits out the protocol buffer schema
  prompt: Can you implement the put method by storing the data in sqlite?
  gpt: spits out the code with a kv_pairs table
  prompt: Can you show me the sqlite schema for the table?
  gpt: spits out the sql to create the table

I was also able to make it write a boot loader, a simple kernel, and give me instructions to run it on qemu.

The iterative process is fantastique to dig into something new. It’s not an authoritative source of information but a buddy with whom to explore topics.

I want something like that integrated in my OS. With such capabilities I can see myself talking to my AI assistant while I’m working on something.

Edit: it’s also quite good at writing test cases for a given function

“This readme file is about a project where a computer program called ChatGPT helped someone build a new computer program. The person asked the ChatGPT program to fix any mistakes in the code, and the ChatGPT program did its best to help. The project didn't work perfectly, but it was a fun experiment to see if a computer program could work together with a person to create something new.”

In reality, it means explain something in plan language without insider terms/definitions.

[1] https://www.reddit.com/r/explainlikeimfive/ (See: sidebar)

And so what? Are you saying humans only ever access factually-accurate data when learning?

In fact, aren't memes just humans repeating (with minor changes) what they've seen other humans publish, in the same style... which is exactly what ChatGPT did?

So you could say it did its task perfectly. It's like what they say: play stupid games, win stupid prizes. Similarly: teach stupid things, learn stupid things.

“Common examples are perceived images of animals, faces, or objects in cloud formations, seeing faces in inanimate objects, or lunar pareidolia like the Man in the Moon or the Moon rabbit. The concept of pareidolia may extend to include hidden messages in recorded music played in reverse or at higher- or lower-than-normal speeds, and hearing voices (mainly indistinct) or music in random noise, such as that produced by air conditioners or fans.”

https://en.wikipedia.org/wiki/Pareidolia

New definition to include “perceiving coherent human intelligence in probabilistic machine generated copy”.

https://en.wikipedia.org/wiki/Cognitive_dissonance

What I found most powerful was the ability to interrogate its responses. I can ask broad/general questions, and then follow-up with 'why is X used in this way? How else can I do that?' or 'you mentioned Y, how does it work and why isn't X used instead?'.

I had ChatGPT explain to me how Rust's implementation of `Vec` works, and importantly, WHY things have been done in a certain way, e.g. why a separate struct called `RawVec` is used.

This may be simple information, but to be able to get it by asking questions in ways that make sense to ME, is so valuable.

I have quite a few non trivial and complete OSS projects on my GH, and in my 10+ year career as a SDE no one has ever really looked at any of it.

But they do frequently ask me to take their code challenge to prove I can program.

I have a huge GH portfolio[0]. Dozens of complete, soup-to-nuts, documented, tested, and in-use projects. I have complete source for shipping apps, backend frameworks, APIs, SDKs, communication libraries, UI widgets, teaching material (complete course modules), etc. I also have many personal blog entries, and online documentation for stuff; not to mention years of tags and commit history.

No one ever looked at them. In fact, when I asked one interviewer, why they didn't bother looking at them, I was told "You probably faked it."

Needless to say, that interview did not proceed any farther.

[0] https://github.com/ChrisMarshallNY#browse-away (NOTE: I just archived and retired a bunch of stuff, because I maintain what I publish; even if I am the only consumer. I dogfood most of my work).

I hired someone on the back of their GitHub repo once. He talked me into it by saying he struggled to control nerves during coding interviews etc. Regretted it immediately - the code he wrote once he joined inexplicably bore no resemblance to what was in his repo. He also turned out to have a major attitude problem with the language we were actually working in, suggesting at one point that maybe he should be able to write his code in Common Lisp and then we should hire cheap Indians to rewrite it into our actual language.

This sort of thing seems hard to detect unless you actually watch someone code. The stuff on his repo could have come from anywhere or anyone, really.

This was not my experience. I never gave coding tests, and would have killed for a GH repo.

I was a manager for 25 years, and we did pretty hairy C++ programming.

What a code repo gives me, is a fulcrum for a discussion. I would never hire, based on a code repo, or a test.

What I did hire on, was fairly long, casual conversations, where I drew the applicant out, and got them to talk about their work, problems they solved, and design decisions they made. A GH repo would give me an "in" for that. "I see that you wrote the firmware for an AI-powered cheese straightener. Tell me about how it came to be. ... Really? How did you solve that problem?" etc.

A lot of the folks that I interviewed were not "people persons." I found that once we got into one of these conversations, they opened right up.

Also, and this actually doesn't have much to do with my tech experience, I'm a pretty hard person to BS. One reason, is that I come across as a naive, credulous, person, and bulshitters can't seem to help themselves. They just have to try yanking my chain. It's kind of fun, actually.

It's hard to scale. People are very easily BSd in interviews without training. If you need to define a repeatable process that's fair (for a fast growing team), it's easier for people to objectively evaluate coding skills they've watched than to handle an open ended conversation. If you don't need that framework, super, that's a rare and useful skill.

At the same company where I made aforementioned interviewing mistake, when I first joined they were in the process of interviewing a guy who billed himself as a high-flying consultant type. They were impressed and intended to offer. They asked me to meet him and ask some questions too, just as a last minute double check. He turned out to not actually have some of the most impressive sounding skills on his CV, in particular, he claimed to be an expert on the internals of HotSpot but when probed couldn't answer even basic questions of the sort you'd learn by reading the user manual. Several other claimed hard skills fell apart under direct questioning too. I got the sense that he had previously been talking about work and projects done whilst he was around, but not by himself.

After that I put in place standardized coding tests. My managers continued to do the open ended interviews they'd been doing before. After six months or so we noticed that whilst the coding tests were selective i.e. we sometimes said no, the open ended interviews had never yielded a reject. They were pointless for both us and the candidate because the answer was always yes. When I sat down with the second stage interviewers, we went through some of the questions being asked and I posed a meta-question: what answers to this question would cause you to reject the candidate? They couldn't come up with any.

I never made a bad technical choice, but I did make a couple of personality mistakes; people that had the tech chops, but couldn't work in the team. That is a lot more difficult to determine than tech chops.

I think one thing that helped, was that my company paid "competitive" (i.e. "low") salaries, although we were a marquée brand. I suspect that kept the grifters away. It made my job a challenge, though.

One thing that software companies, in particular, are obsessed with, is hiring "cookie cutter" people. We seem to have an allergy to hiring competent, skilled, experienced people. We want to hire lots of people with mediocre skills, pay them ridiculous amounts of money, and force them to write top-shelf software, using structure and process.

I don't think it works well, and is attractive to grifters.

My group was a small, high-functioning team of experienced C++ engineers, writing cross-platform, image processing pipeline algorithms. Not work for the faint of heart, and each one of my employees had a specific personality, strong point, and weak point. It was my job to understand each of them, as an individual, and help them to integrate into a pretty massive team, that stretched across three continents, and included some of the best engineers and scientists in the world. We were always under a spotlight, and we had to be constantly delivering.

Companies have relied on "key players" for hundreds of years. If they keep their "key players" happy, and productive, they do well. If they don't, they fall down. These companies aren't always able to get very large, though. With scale, comes mediocrity. Sort of can't be helped.

But it also brings bags of cash, which is nice. It's just that I see small companies, all the time, trying to act like big companies, and I don't think that works very well.

The proof of the pudding is in the eating.

Many people have these repos with a fork of some opensource project with like 2 commits in it, or something not parting too far from a template project. I am not saying this does not qualify - because those commits might be useful or insightful, or merely sufficient to solve a problem - but they are way too easy to fake.

Most first line screens/recruiters aren't going to do much more than skim code and aren't going to run it. Best case scenario now your recruiting pipeline is inflated with more poor candidates. Worst case scenario they're not one step closer to bamboozling their way to a job they're not qualified for.

Tools like this are going to become more and more ubiquitous, and I’m questioning myself whether it makes sense to avoid it almost entirely as I have been or to embrace using it to accomplish tasks.

Unfortunately (IMO) we are through the looking glass, as they say.

I’m using it in my workflow for when I’m writing really repeatable test cases or boilerplate. For anything more complicated, very quickly the limitations will start to appear which has brought me comfort that it’ll be a long time before this is going to replace me entirely.

I imagine it’ll be a bit similar to self driving cars, in that the technology behind the base 80% of cases is incredibly impressive, but the human element to solve the other 20% remains out of reach for longer than we hope.

I have, and I'm pretty convinced coding challenges are going to be a thing of the past soon. Either that or there will need to be extreme anti-cheating measures in place.

This has a strong feel of "why should I learn multiplication tables when I have a calculator with me at all times?" But instead of multiplication tables it's the entire curriculum.

I can see this becoming a standard tool, which shifts the burden on the student from writing code to critically reading code, writing tests, and learning how to judge correctness.

But the counter-argument is the same: you don't want to be reliant on a calculator at all times.

And that's assuming the tech can even get to the point of replacing the curriculum. It only works as well as it does because it's been trained on text that is effectively many past curricula. What if you add something new? The model will be useless.

I'd be shocked if ChatGPT knew as much about, say, Coq as it did about Python.

I doubt we would hire it, but it would definitely pass the first tech screens.

- offline testing

- asking candidates to create ridiculous and hardcore apps that even ChatGPT is not sufficient. For example, creating a web browser with Rust.

Prompt with a natural language description of a target program, instruct it to decompose the problem into increasingly small actionable steps, give it preprocessed terminal output prompts in response and let it iterate until it's satisfied

It sort of scares me to think it is just a matter of engineering to get there.

I'm asking out of pure curiosity to understand your concern, to be clear. Simply curious.

:)

except the "is the program meeting the original prompt criteria" check was a comment. haha