Eight months in and ChatGPT is worse than it’s ever been, study finds

ChatGPT needs no introduction. Since its release in November last year, it has quickly become the most widely talked about software application and a household name in offices, academia, government and political institutions. It’s heralded a boom in AI development and technological innovation, partly due to the powerful capabilities the software offers, however, a recent study has found that as time goes on, ChatGPT is slowly getting worse.

According to the report, “GPT-4 (March 2023) was very good at identifying prime numbers (accuracy 97.6%) but GPT-4 (June 2023) was very poor on these same questions (accuracy 2.4%)”, alongside the fact that “GPT-4 was less willing to answer sensitive questions in June than in March, and both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March.” However, the study makes it clear that there are interesting variations in reasoning ability over time, with GPT-3.5 showing improvements at times.

The above image is taken from the report and perhaps the most surprisingly result is that GPT-4 (June 2023) was unable to correctly identify whether or not 17077 is a prime number or not (it is, by the way). While this result could be expected from a more complex question with multiple variables, prime number evaluation is a boolean result and it’s astonishing GPT-4 was unable to consistently provide a right answer here.

This follows on from more findings that ChatGPT’s ability to provide executable code has been deteriorating. While studying ChatGPT’s code generation ability, the study finds that it provides “more verbose and less directly executable” code. It has long been the subject of chatter among online communities that ChatGPT’s coding skills are getting worse, however this only reiterates that theory.

There’s not a clear explanation why ChatGPT’s abilities are changing, though the paper reiterates the importance of developers keeping a close eye on how they use the chatbot, as it seems to be constantly changing in use.

You can read the full study here.

About the Author

Amaar Chowdhury

Amaar is a gaming journalist with an interest in covering the industry's corporations. Aside from that, he has a hankering interest in retro games that few people care about anymore.