ChatGPT 4.0 Changed 31 Answers on an Accounting Ethics Exam

An AI-generated ethics answer can sound convincing and cite the law. Before accepting it, ask students to check two things separately: is the answer defensible, and does the cited authority actually support it?

A study of ChatGPT 4.0 on a Portuguese accounting ethics exam shows why that distinction matters.

What the researchers tested

Fábio Albuquerque and Paula Gomes dos Santos gave the model 99 valid questions from the October 2024 exam of Portugal's Order of Certified Accountants. Each question offered four options. The researchers supplied the relevant statutes and ethics code and checked the answers against the official key.

They asked every question twice, then used further prompts on persistent errors, including requests to reconsider and a change from questions to statements.

The answer and its justification can fail separately

The model scored 61% on the first attempt and 55% on the second. It changed 31 answers between attempts; 15 changed from right to wrong.

Its precise legal citations also needed scrutiny. Ten of the 30 specific article citations in the first round were wrong. In six of those cases, a faulty citation accompanied a correct answer.

For assessment design, that raises a useful question: would your marking scheme distinguish a correct option from a sound justification? The study did not test a classroom activity, but it gives lecturers a concrete failure to discuss.

Limits and context

This was one model version, one exam and one legal setting, scored by two researchers. It tested neither students nor current AI systems generally. Prompt wording and previous interactions may have affected the responses; later rounds also covered fewer questions, so their percentages are not directly comparable.

Evidence strength: limited and preliminary. This is one exploratory test of one ChatGPT version on one exam. Its numbers are fully reported, but it has no statistical tests, no comparison with people or other tools and no student outcomes.