Will ChatGPT pass the online quizzes? Adapting an assessment strategy in the age of generative AI

Abstract

As generative AI (artificial intelligence) technologies, such as ChatGPT, become increasingly available, traditional online assessments must be re-evaluated to maintain their educational value. Open-book online quizzes have long been an effective tool for engaging students, effectively supporting learning, and reinforcing fundamental knowledge and skills. However, the ease of using AI to complete these quizzes may undermine their intended purpose. This article explores the initial findings of using ChatGPT to answer twelve online quizzes used for continuous assessment in two first-year quantitative techniques modules on business programmes in an Irish technological university. ChatGPT, along with suitable plugins, is increasingly accurate in answering the online quizzes, with ChatGPT-3.5 scoring an average of 35%, ChatGPT-4 47% and ChatGPT-4 with Wolfram plugin 78%. Most of the incorrect responses are due to calculation errors; if these are corrected by simply checking the arithmetic with a calculator, the averages increase to ChatGPT-3.5 scoring 72%, ChatGPT-4 76% and ChatGPT-4 with Wolfram plugin 80%. Thus, the online quizzes on these modules can be quickly completed with the assistance of ChatGPT with a high level of success. The implications of this for using online quizzes as an assessment strategy are discussed; potential assessment redesigns are outlined, including how to integrate generative AI into the learning and assessment process in an ethical and constructive manner. Although generative AI provides a challenge to traditional online quizzes, it also has the potential to aid student comprehension and learning, and the skills of prompt engineering are likely to become increasingly relevant and useful.