I pay for it because I can afford it, but honestly I haven't been super impressed.
GPT-4 is definitely better at some tasks and moderately less prone to hallucination, but it botched the first couple scripting tasks I gave it, so it didn't exactly wow me. The main difference I've observed is that it gives wordier responses than GPT-3.5.
That said, I might have a different opinion if usage of GPT-4 weren't so severely rate-limited. Currently, you can only send 25 requests every 3 hours, so it's hard to experiment with it as freely as I want.
The only task GPT-4 "botched" for me was trying to write an IRC bot with Rust.
12 attempts and it still couldn't connect. After 6 attempts (of me pasting back compilation errors) it compiled, but didn't run. The last 6 attempts were me trying to get Rust to ignore TLS errors so it could connect. I gave up after that.
I had a set of ancient (10+ year old) shell scripts and Python controlling Deluge. I wanted to move to Transmission and still retain the same functionality.
Except for using an API from 2021, the code it gave me was 100% correct and properly named and even commented. I had to spend 5 minutes correcting the import and changing a few named variables (user -> username type stuff) and it just worked.
Use the API if you want some more time with it. You’ll pay for the tokens you use, but twenty bucks gets you some hours of experimentation depending on how fast you’re going.
Not the person you asked, but the OpenAI playground is a great place to start messing with their API, I wrote this to help: https://www.inputoutput.ai/tutorials/openai-playground-guide (note website is still a work in progress just got it going bear with me haha)
However, you need access to the GPT-4 API which you have to request separately. So even if you have GPT-4 with chatGPT Plus you might not have access to GPT-4 through the playground/API.
The 25 requests isn't too much of a hindrance for me but the user experience around its long response times definitely is. Needs notifications, or at least just have it wait and then send the text all at once rather than drip feeding it.
Although if they did that then I'd probably hit the 25 request limit a lot more.
As someone who spent a lot of time playing with VB6 as a kid, Gambas is an amazing throwback. It feels like the VB6 I remember, even though my memories are vague and tinted with nostalgia.
In other words, it's not a perfect recreation of VB6 -- it's even better, because it includes a lot of modern conveniences that make it feel more comfortable to use. The IDE feels surprisingly polished and, while the language is definitely still BASIC, it has some nice enhancements that make it feel a bit less clunky IMO.
I'm probably not going to implement anything important in it, but it's a fun way to relive my VB6 nostalgia when I'm in the mood for it.
As mentioned, the example you quoted is from a pure-AWK JSON parser. I don't dispute that AWK has issues, but AWK is one of those languages that magically coerces strings to numbers, so you can just write `"1" + 2 + "3.5"` and it'll work.
Same here haha. StackOverflow informed me that it means GitHub is down, and indeed it was.
I have to wonder if Git could somehow report this better. I guess it depends on exactly how GitHub is down, but "fatal error in commit_refs" made me worry that my local repo was somehow hosed.
It reports what it has, if it manages to connect but fetching the metadata or whatever fails, that’s what it’s going to report.
If it can’t even connect it’ll tell you that, but I would assume on github the client will always manage to connect unless their entire network is down.
* You don't have experience with basic technical decision making -- e.g. why would you pick Java vs. C vs. Python vs. Go vs. any other language, and how would that decision change if you were building a web app vs. mobile app vs. embedded software?
* You don't have a familiar set of tools and established preferences for the aforementioned decisions -- i.e. if an experienced programmer is building a web app, there are probably specific languages and tools they're inclined to reach for.
* You don't know about application deployment and productionization. How do you get your project somewhere people can use it? You don't know about deploying code to servers or running it in Docker or K8s or PaaS.
* You don't have the practiced logical thought process for things like algorithm design, debugging, figuring out user bug reports, identifying edge cases, predicting user behavior, etc.
* You don't have the experience with things that experienced programmers find "obvious" -- e.g. if you're building a web app, you need an authentication system, registration flow, password reset, etc.
I think the main effort to address this will be LLM-driven low-code solutions. Imagine something like Airtable, but with an integrated LLM assistant that can make changes to your app and write code to implement business rules and automation. This gets rid of problems like choosing your tech stack, deployment, etc. and it minimizes the amount of code the LLM has to create.
I imagine this will have effects comparable to those of spreadsheet software: non-programmers will be able to build custom tools with a newfound ease and some will end up learning a lot of programming skills in the process.
And, much like with spreadsheets, these tools will end up becoming business critical in unexpected ways and will turn out to be a huge pain in the ass to maintain. They'll find that parts of their business logic don't work as expected and that they can't figure out how to prompt the LLM to get it to implement the rules the right way. The users of these tools will basically have to learn some programming anyway or they'll have to hire experienced programmers to figure the problems out for them.
Yeah, I think this nails it. I think about the non-trivial parts of writing code, and I think about all the previous failed attempts at this sort of thing, and I remain unimpressed so far.
I think specifically about things like optimizations and wonder how an LLM would deal with that. In the last week I’ve written code that dealt with file I/O slowness, thread contention, thread deadlock, etc. In order to even begin fixing these problems, I had to first identify what the cause was by debugging them. Then I had to reason about how the program works to think up a good fix for the issue in our specific case.
There are no stack overflow answers for some of the problems I fixed this week. Heck, at one point in the past, I even wrote some code that we thought about getting patented. It was an interesting solution to the problem. And the problem itself was not a common one (but not unheard of). There’s a small amount of literature on it, but not a ton. I don’t see LLMs being a threat to this type of work. But in talking about it, I do see how some other type of AI might assist with it.
Anecdotally, the times I've felt most impressed by ChatGPT usually ended up being exactly like this.
Even as an LLM skeptic, I've had chat sessions with ChatGPT that felt unbelievable at the time, where it almost seemed to be generating incredibly cogent ideas and arguments all on its own. Then, after a few days, I went and reread the transcript, only to realize I'd been hinting and nudging it along much more than I consciously thought at the time. I only remembered the answers that were the most impressive and forgot about the ones that missed the mark.
ChatGPT is really good at picking out the key ideas in the prompt and responding to them. It's really easy to inadvertently nudge it to give a certain type of response, which of course makes it feel all the more astounding when it gives you exactly the type of answer you already unconsciously expect.
Indeed; I've been trying to describe these models as being consummate bullshitters. Or consummate improvisers, if you want to be kinder. Since its core task model is "autocomplete", it's less "trying to answer a question" than it is "trying to predict the most likely next thing to say after a question". Doing its best to sound like the most likely thing that comes after the thing prompting it, which in this case, often looks like an 'answer'. And when you have access to as much context as these models do, that can get as specific and nuanced as "giving you the answer you were expecting".
It has got me wondering if this might be an interesting fundamental model of consciousness. A bullshit generator modulated by other systems that may have more "logical" qualities.
I discuss this topic briefly in a post I recently made where I call it "decorative knowledge" and try to pinpoint exactly what GPT is doing and fails to do.
GPT-4 is definitely better at some tasks and moderately less prone to hallucination, but it botched the first couple scripting tasks I gave it, so it didn't exactly wow me. The main difference I've observed is that it gives wordier responses than GPT-3.5.
That said, I might have a different opinion if usage of GPT-4 weren't so severely rate-limited. Currently, you can only send 25 requests every 3 hours, so it's hard to experiment with it as freely as I want.