It's not a "bug" in the traditional sense, but a consequence of how language models work. Probability distributions over token chains are very powerful tools for generating human-like written text, but they perform poorly at symbolic inference. Humans have the same limitations, which is why math is a separate topic taught in schools that focuses not on knowledge but on algorithms. Understanding natural language syntax and semantics doesn't necessarily enable you to calculate.

Perhaps I should have added how I noticed in the first place. I gave it a block of code with a few nested loops and asked it how many times would it run. For a small number 5 x 5 x 5, it calculated correctly. But for 5000 x 5000 x 5000 it gave the wrong answer.

I'd like to see an integration into wolfram alpha or something. That would be amazing.

if you ask it a question and it gives you a confident answer that's 100% wrong that's a terrible experience and a non-working product. "it's not a bug in the traditional sense" is technically correct but missing the point entirely.

ChatGPT is not a product, at least not for the purpose of answering arbitrary questions with a high degree of correctness. Nobody is claiming that ChatGPT is, or is supposed to be, the ultimate answer generating machine. If you are relying on the current generation of language models for that purpose you are making a huge mistake. They are a technology demonstration for what future AI products suitable for the mass market might look like, nothing less and nothing more.

Plenty of terrible experiences and non-working products are still very valuable to people. We're dealing with new magic here, so YMMV. I appreciate that you're trying to push to product forward, but if you have learned any background of how GPTs work, what you're asking for isn't really understood yet. The product might be too new for you.

and that's after the recent improvements

I noticed the announcement in the latest update that it was supposedly better at math, but the first simple calculations I gave it were all wrong. Not sure how it was improved at all.