Surprisingly, the study revealed that ChatGPT's performance in providing accurate responses to programming questions was subpar, with the model delivering correct answers only 52% of the time.

The results showed that out of the 512 questions, 259 (52%) of ChatGPT's answers were incorrect and only 248 (48%) were correct. Moreover, a whopping 77% of the answers were verbose.