Online citations, reference lists, and bibliographies.

A Systematic Review Of Automatic Question Generation For Educational Purposes

Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, Salam Al-Emari
Published 2019 · Computer Science
Cite This
Download PDF
Analyze on Scholarcy
Share
While exam-style questions are a fundamental educational tool serving a variety of purposes, manual construction of questions is a complex process that requires training, experience, and resources. This, in turn, hinders and slows down the use of educational activities (e.g. providing practice questions) and new advances (e.g. adaptive testing) that require a large pool of questions. To reduce the expenses associated with manual construction of questions and to satisfy the need for a continuous supply of new questions, automatic question generation (AQG) techniques were introduced. This review extends a previous review on AQG literature that has been published up to late 2014. It includes 93 papers that were between 2015 and early 2019 and tackle the automatic generation of questions for educational purposes. The aims of this review are to: provide an overview of the AQG community and its activities, summarise the current trends and advances in AQG, highlight the changes that the area has undergone in the recent years, and suggest areas for improvement and future opportunities for AQG. Similar to what was found previously, there is little focus in the current literature on generating questions of controlled difficulty, enriching question forms and structures, automating template construction, improving presentation, and generating feedback. Our findings also suggest the need to further improve experimental reporting, harmonise evaluation metrics, and investigate other evaluation methods that are more feasible.
This paper references
10.1007/s40593-018-00172-w
Ontology-Based Generation of Medical, Multi-term MCQs
Jared Leo (2019)
Aid to the evaluation of therapeutic studies.
J. Reisch (1989)
10.1145/3227609.3227656
Automatic Generation of Multiple Choice Questions from Slide Content using Linked Data
Ainuddin Faizan (2018)
A Novel Multiple Choice Question Generation Strategy: Alternative Uses for Controlled Vocabulary Thesauri in Biomedical-Sciences Education
Marcelo A. Lopetegui (2015)
10.1016/j.websem.2015.05.005
A novel approach to generate MCQs from domain ontology: Considering DL semantics and open-world assumption
V VinuE. (2015)
10.18653/v1/W18-0533
Distractor Generation for Multiple Choice Questions Using Learning to Rank
Chen Liang (2018)
10.1136/jech.52.6.377
The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
Sara H. Downs (1998)
10.18653/v1/W18-0548
Automatic Distractor Suggestion for Multiple-Choice Tests Using Concept Embeddings and Information Retrieval
Le An Ha (2018)
10.1109/FIE.2015.7344291
Automatic fill-the-blank question generator for student self-assessment
Girish Kumar (2015)
10.1108/IJILT-09-2017-0085
Generating story problems via controlled parameters in a web-based intelligent tutoring system
Nabila Khodeir (2018)
10.1109/ICOSC.2015.7050809
Aligning automatically generated questions to instructor goals and learner behaviour
Lydia Odilinye (2015)
10.1109/ICALT.2014.105
TEDQuiz: Automatic Quiz Generation for TED Talks Video Clips to Assess Listening Comprehension
Yi-Ting Huang (2014)
10.1145/3148011.3154463
Distractor Generation with Generative Adversarial Nets for Automatically Creating Fill-in-the-blank Questions
Chen Liang (2017)
10.1007/978-3-540-30139-4_45
Can Automated Questions Scaffold Children's Reading Comprehension?
Joseph E. Beck (2004)
A framework for automated generation of questions across formal domains
R. Singhal (2015)
Personalized Mathematical Word Problem Generation
Oleksandr Polozov (2015)
10.1109/LaTiCE.2016.8
Generating DFA Construction Problems Automatically
Varun Shenoy (2016)
10.18653/v1/W17-5038
Question Generation for Language Learning: From ensuring texts are read to supporting learning
Maria Chinkina (2017)
10.1109/JCSSE.2016.7748891
Automatic multiple-choice question generation from Thai text
Chonlathorn Kwankajornkiet (2016)
AUTOMATIC QUESTION GENERATION FOR 5W-1H OPEN DOMAIN OF INDONESIAN QUESTIONS BY USING SYNTACTICAL TEMPLATE-BASED FEATURES FROM ACADEMIC TEXTBOOKS
S. T M.T. Basuki (2018)
10.3115/1225403.1225404
FAST - An Automatic Generation System for Grammar Tests
Chia-Yin Chen (2006)
10.1109/COGINFOCOM.2013.6719278
Complexity-based generation of multi-choice tests in AQG systems
Laszlo Kovacs (2013)
Semantic Question Generation Using Artificial Immunity
Ibrahim Eldesoky Fattoh (2015)
10.22219/KINETIK.V3I2.650
Generating Indonesian Question Automatically Based on Bloom’s Taxonomy Using Template Based Method
Selvia Ferdiana Kusuma (2018)
Evaluating the Quality of Learning: The SOLO Taxonomy
Kevin F. Collis (1982)
A comparative study of methods for a priori prediction of MCQ difficulty. the Semantic Web journal
G Kurdi (2019)
Improving Large-Scale Assessment Tests by Ontology Based Approach
V VinuE. (2015)
10.3402/RLT.V20I0.19198
Automatic generation of analogy questions for student assessment: an Ontology-based approach
Tahani Alsubait (2012)
10.1109/FUZZ-IEEE.2018.8491624
Computational Intelligence Framework for Automatic Quiz Question Generation
Akhil Killawala (2018)
Multiple choice question generation for slides
A. Faizan (2017)
10.1007/978-3-319-93846-2_48
Automatic Generation of Multiple-Choice Fill-in-the-Blank Question Using Document Embedding
Junghyuk Park (2018)
10.1080/0304379950200213
Distance Education in Australia
Terry Evans (1995)
Implementation and assessment of the automatic question generation
L. Bednarik (2012)
Dimensionally Guided Synthesis of Mathematical Word Problems
Ke Wang (2016)
Domain-specific Question Generation from a Knowledge Base
Linfeng Song (2016)
10.1109/ICACCI.2015.7275833
Automated Question Generation tool for structured data
A. Shirude (2015)
10.18653/v1/W16-0503
Automatic Generation of Context-Based Fill-in-the-Blank Exercises Using Co-occurrence Likelihoods and Google n-grams
Jennifer Hill (2016)
10.1016/j.nedt.2006.07.006
The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments.
Marie Tarrant (2006)
10.5087/D
G-Asks: An Intelligent Automatic Question Generation System for Academic Writing Support
Ming Liu (2012)
10.3115/1073083.1073135
Bleu: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni (2002)
10.1145/3121050.3121073
Knowledge Questions from Knowledge Graphs
Dominic Seyler (2017)
10.18653/v1/W16-6609
Infusing NLU into Automatic Question Generation
Karen Mazidi (2016)
10.1016/J.WEBSEM.2015.05.005
A novel approach to generate MCQs from domain ontology
E V Vinu (2015)
10.1007/10639.1573-7608
Education and information technologies
Arend J. Visscher (1997)
10.1109/TLT.2018.2889100
Automatic Multiple Choice Question Generation From Text: A Survey
D. Ch (2020)
10.1371/journal.pone.0183591
A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review
Seunghye Han (2017)
10.1080/08832329709601623
Quality Multiple-Choice Test Questions: Item-Writing Guidelines and an Analysis of Auditing Testbanks
James D. Hansen (1997)
10.1155/2014/274949
Ontology-Based Multiple Choice Question Generation
Maha M. Al-Yahya (2014)
10.5087/D
A Detailed Account of The First Question Generation Shared Task Evaluation Challenge
Vasile Rus (2012)
10.1186/s41039-016-0031-7
How do machine-generated questions compare to human-generated questions?
Lishan Zhang (2016)
Modeling of Item-Difficulty for Ontology-based MCQs
V VinuE. (2016)
10.1504/IJSISE.2017.10005435
Automatic generation of multiple choice questions for e-assessment
A. Santhanavijayan (2017)
10.7554/eLife.13615
Bias in the reporting of sex and age in biomedical research on mouse models
Oscar Flórez-Vargas (2016)
Analysis of one-best MCQs: the difficulty index, discrimination index and distractor efficiency.
Mozaffer Rahim Hingorjo (2012)
10.5220/0005775502670274
Item Difficulty Analysis of English Vocabulary Questions
Yuni Susanti (2016)
A REVIEW ON AUTOMATIC QUESTION GENERATION SYSTEM FROM A GIVEN HINDI TEXT
Jaspreet Kaur (2015)
10.1007/978-3-319-54627-8_3
An Experimental Evaluation of Automatically Generated Multiple Choice Questions from Ontologies
Ghader Kurdi (2016)
Survey on e-Learning Implementation in Eastern-Europe - Spotlight on Romania
Ioana Raluca Goldbach (2017)
Literature Review of Automatic Question Generation Systems
Sheetal Rakangor (2015)
Automatic question generation system for English exercise for secondary students. In: the 25th international conference on computers in education
T. Soonklang (2017)
The learning benefits of questions
W. Thalheimer (2003)
10.1007/978-3-319-61425-0_22
Improving Reading Comprehension with Automatically Generated Cloze Item Practice
Andrew Olney (2017)
10.1587/TRANSINF.2016EDP7417
SOLS: An LOD Based Semantically Enhanced Open Learning Space Supporting Self-Directed Learning of History
C. Jouault (2017)
10.1109/ICTAI.2014.129
Automated Generation of Region Based Geometric Questions
Rahul Singhal (2014)
10.1109/ICTAI.2016.0129
User-Defined Difficulty Levels for Automated Question Generation
Rahul Singhal (2016)
Difficulty-level Modeling of Ontology-based Factual Questions
V VinuE. (2017)
10.4103/0976-500X.107697
Critical appraisal skills programme
Jatinder Singh (2013)
10.1145/1134285.1134500
Performing systematic literature reviews in software engineering
D. Budgen (2006)
10.2307/3119730
Taxonomy of Educational Objectives. Handbook I: Cognitive Domain
B. Bloom (1966)
10.1017/S1351324906004177
A computer-aided environment for generating multiple-choice test items
R. Mitkov (2006)
10.1145/3287324.3287492
Stochastic Tree-Based Generation of Program-Tracing Practice Questions
Anderson Thomas (2019)
E-Learning in European Higher Education Institutions: Results of a Mapping Survey Conducted in October-December 2013.
Michael Gaebel (2014)
Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts
Jun Araki (2016)
10.18653/v1/P16-1056
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Iulian Serban (2016)
10.1109/BIBM.2018.8621162
Domain specific automatic Chinese multiple-type question generation
Tianlin Zhang (2018)
10.18653/v1/W18-0530
A Semantic Role-based Approach to Open-Domain Automatic Question Generation
Michael Flor (2018)
Determining Alignment of Expectations and Assessments in Mathematics and Science Education.
Norman L. Webb (1997)
10.1186/S12909-016-0773-3
The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value
Bonnie R Rush (2016)
Doing a Systematic Review: A Student's Guide
Gemma Cherry (2017)
10.1007/978-3-642-21043-3_4
Unsupervised Relation Extraction Using Dependency Trees for Automatic Generation of Multiple-Choice Questions
Naveed Afzal (2011)
10.1007/978-3-319-19773-9_26
Leveraging Multiple Views of Text for Automatic Question Generation
Karen Mazidi (2015)
Semantic Web and Question Generation: An Overview of the State of the Art.
Andreas Papasalouros (2018)
Drill and practice in learning (and Beyond)
C S Lim (2012)
Automatic question generation system for English exercise for secondary students
T Soonklang (2017)
10.1016/J.IERI.2012.06.085
Algorithms for Automatic Generation of Logical Questions on Mobile Devices
Kun Wang (2012)
10.18653/v1/D16-1264
SQuAD: 100, 000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar (2016)
Recognizing Question Entailment for Medical Question Answering
Asma Ben Abacha (2016)
10.1145/3184558.3191534
Automatic Generation of Quizzes from DBpedia According to Educational Standards
Oscar Rodríguez Rocha (2018)
10.1007/978-3-319-64206-2_12
Machine Learning Approach to the Process of Question Generation
Miroslav Blsták (2017)
10.1504/IJTEL.2012.051580
Next generation of e-assessment: automatic generation of questions
Tahani Alsubait (2012)
10.1109/IALP.2015.7451552
A Kanji learning system based on automatic question sentence generation
Jianwei Zhang (2015)
10.1109/ICELETE.2013.6644389
A similarity-based theory of controlling MCQ difficulty
Tahani Alsubait (2013)
Using Automated Questions to Assess Reading Comprehension, Vocabulary, and Effects of Tutorial Interventions
J. Mostow (2004)
Item analysis of published MCQs.
Yogesh Kumar Sarin (1998)
Automatic Question Generation Based on Sentence Structure Analysis
Miroslav Blšták (2018)
Good Question! Statistical Ranking for Question Generation
Michael Heilman (2010)
10.5220/0006320203790386
Automatic Generation of English Reference Question by Utilising Nonrestrictive Relative Clause
Arief Yudha Satria (2017)
10.1109/TLT.2017.2679009
Automatic Chinese Multiple Choice Question Generation Using Mixed Similarity Strategy
Ming Liu (2018)
Package 'irr
M Gamer (2019)
10.18653/v1/W17-5008
Evaluation of Automatically Generated Pronoun Reference Questions
Arief Yudha Satria (2017)
A computer-aided environment
R. Mitkov (2006)
10.1093/nar/gkh061
The Unified Medical Language System (UMLS): integrating biomedical terminology
O. Bodenreider (2004)
Generating multiple questions from ontologies: How far can we go?
T Alsubait (2014)
Generating Multiple Choice Questions From Ontologies: Lessons Learnt
Tahani Alsubait (2014)
10.1186/s41239-017-0060-3
Factual open cloze question generation for assessment of learner’s knowledge
Bidyut Jyoti Das (2017)
10.1007/978-981-13-0298-5_14
Distance Education in Australia, Europe and the Americas
Adnan Qayyum (2018)
10.1109/TLT.2016.2565477
Automatic Chinese Factual Question Generation
Ming Liu (2017)
10.1016/J.SBSPRO.2015.01.685
An Overview of the World of MOOCs
Meltem Huri Baturay (2015)
Can LOD Based Question Generation Support Work in a Learning Environment for History Learning
Jouault Corentin (2016)
10.3233/978-1-60750-028-5-465
Generating Instruction Automatically for the Reading Strategy of Self-Questioning
J. Mostow (2009)
10.1109/COGINFOCOM.2012.6421938
Implementation and assessment of the automatic question generation module
László Bednarik (2012)
10.1007/978-3-319-39583-8_3
Automatic Question Generation: From NLU to NLG
Karen Mazidi (2016)
10.1145/3051457.3053995
Intelligent Math Tutor: Problem-Based Approach to Create Cognizance
Monika Gupta (2017)
Drill and practice in learning (and Beyond), Springer US, pp
C. S. Lim (2012)
10.3115/v1/W15-0623
Judging the Quality of Automatically Generated Gap-fill Question using Active Learning
Nobal B. Niraula (2015)
10.18653/v1/W17-5015
Distractor Generation for Chinese Fill-in-the-blank Items
Shu Jiang (2017)
10.1007/s10639-018-9814-3
A hybrid approach for automatic generation of named entity distractors for multiple choice questions
Rakesh Patra (2018)
10.1093/ijl/3.4.235
Introduction to WordNet: An On-line Lexical Database
G. Miller (1990)
10.3115/v1/P14-2053
Linguistic Considerations in Automatic Question Generation
Karen Mazidi (2014)
10.1145/3159450.3159608
On the Use of Semantic-Based AIG to Automatically Generate Programming Exercises
Laura Zavala (2018)
10.1109/TENCON.2017.8227909
Web authoriser tool to build assessments using Wikipedia articles
Sri Sai Adithya (2017)
A Method for Generating History Questions using LOD and Its Evaluation
Jouault Corentin (2015)
CASP qualitative checklist
Critical Appraisal Skills Programme (2018)
ROUGE: A Package for Automatic Evaluation of Summaries
Chin-Yew Lin (2004)
10.3115/1598834.1598846
A Web-Based System for Automatic Language Skill Assessment: EVALING
Cédrick Fairon (1999)
Ontology-based question generation
T. Alsubait (2015)
10.1186/1472-6920-9-40
An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis
Marie Tarrant (2009)
10.1007/978-981-10-7871-2_49
Automatic Generation of Named Entity Distractors of Multiple Choice Questions Using Web Information
Rakesh Patra (2018)
10.5220/0005437200770087
Automatic Generation of English Vocabulary Tests
Yuni Susanti (2015)
10.1007/978-3-319-06569-4_24
Automatic Question Generation for Educational Applications - The State of Art
Nguyen-Thinh Le (2014)
10.1007/978-3-319-52977-6_4
A Review of Recent Advances in Adaptive Assessment
Jill-Jênn Vie (2017)
Difficulty Controllable Question Generation for Reading Comprehension
Yifan Gao (2018)
Automatic question generation system for Punjabi
A Kaur (2017)
Criteria for Alignment of Expectations and Assessments in Mathematics and Science Education. Research Monograph No. 6.
Norman L. Webb (1997)
10.1007/978-3-319-93040-4_27
Automating Reading Comprehension by Generating Question and Answer Pairs
Vishwajeet Kumar (2018)
Guidelines for performing systematic literature reviews in software
B. Kitchenham (2007)
10.1109/SACI.2012.6250000
Automated EA-type question generation from annotated texts
László Bednarik (2012)
10.1007/978-3-319-20609-7_32
Generating Quizzes for History Learning Based on Wikipedia Articles
Yoshihiro Tamura (2015)
10.1186/s41039-017-0051-y
Evaluation of automatically generated English vocabulary questions
Yuni Susanti (2017)
10.3115/v1/W15-0618
RevUP: Automatic Gap-Fill Question Generation from Educational Texts
Girish Kumar (2015)
10.1145/2815833.2815842
Automatically Predicting Quiz Difficulty Level Using Similarity Measures
Chenghua Lin (2015)
Automatic factual question generation from text
Noah A. Smith (2011)
10.1007/978-3-319-96565-9_14
A Semantic Graph-Based Japanese Vocabulary Learning Game
Ratsameetip Wita (2018)
10.1109/DISA.2018.8490637
Building an Agent for Factual Question Generation Task
Miroslav Blšták (2018)
Automatic Generation and Ranking of Questions for Critical Review
Ming Liu (2014)
10.1017/S1351324915000455
Automatic generation of short answer questions for reading comprehension assessment
Yan Huang (2016)
10.1007/978-3-642-30950-2_47
Using Information Extraction to Generate Trigger Questions for Academic Writing Support
Ming Liu (2012)
10.1007/978-3-319-19773-9_16
Evaluating Human and Automated Generation of Distractors for Diagnostic Multiple-Choice Cloze Questions to Assess Children's Reading Comprehension
Yi-Ting Huang (2015)
Generating Multiple Questions From Ontologies: How Far Can We Go?
Tahani Alsubait (2014)
10.1109/TENCON.2012.6412273
Automated question generator for Tagalog informational texts using case markers
Chuchi Montenegro (2012)
10.18653/v1/W17-5034
Multiple Choice Question Generation Utilizing An Ontology
Katherine Stasaski (2017)
10.18653/v1/W17-4413
Crowdsourcing Multiple Choice Science Questions
Johannes Welbl (2017)
10.14705/rpnet.2017.eurocall2017.692
Automatically Generating Questions to Support the Acquisition of Particle Verbs: Evaluating via Crowdsourcing.
Maria Chinkina (2017)
10.1109/CSCITA.2017.8066538
Automatic question generation for intelligent tutoring systems
Riken Shah (2017)
Mining Ontologies for Analogy Questions: A Similarity-based Approach
Tahani Alsubait (2012)
SNOMED-CT: The advanced terminology and coding system for eHealth.
Kevin M. Donnelly (2006)
10.1109/TLT.2012.5
Using Wikipedia and Conceptual Graph Structures to Generate Questions for Academic Writing Support
Ming Liu (2012)
Taxonomy of educational objectives, handbook i: The cognitive domain vol 19
B. S. Bloom (1956)
Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations
10.5087/D
Semantics-based Question Generation and Implementation
Xuchen Yao (2012)
10.1186/s41039-017-0065-5
Controlling item difficulty for automatic vocabulary question generation
Yuni Susanti (2017)
10.1007/s00354-016-0404-x
Content-Dependent Question Generation Using LOD for History Learning in Open Learning Space
C. Jouault (2016)
10.1007/978-3-319-19773-9_114
A Framework for Automated Generation of Questions Based on First-Order Logic
Rahul Singhal (2015)
Ranking Automatically Generated Questions as a Shared Task
Michael Heilman (2009)
10.18653/v1/W15-4405
Semi-automatic Generation of Multiple-Choice Tests from Mentions of Semantic Relations
Renlong Ai (2015)
10.1109/IRI.2018.00078
Automatic Question Generation from Children’s Stories for Companion Chatbot
Che-Hao Lee (2018)
Biology Question Generation from a Semantic Network
Lishan Zhang (2015)
10.18653/v1/W18-3722
Learning to Automatically Generate Fill-In-The-Blank Quizzes
Edison Marrese-Taylor (2018)
Rating Computer-Generated Questions with Mechanical Turk
Michael Heilman (2010)
10.1145/3231644.3231654
QG-net: a data-driven question generation model for educational content
Zichao Wang (2018)
Automatic question generation system for Punjabi. In: The international conference on recent innovations in science, Agriculture, Engineering and Management
A. Kaur (2017)
10.3233/SW-170252
Automated generation of assessment tests from domain ontologies
V VinuE. (2017)
A comparative study of methods for a priori prediction of MCQ difficulty
G. Kurdi (2019)
Semantic web and question generation: An overview
A. Papasalouros (2018)
Automatic Generation of Multiple Choice Questions using Surface-based Semantic Relations
afzal. naveed (2015)
10.1007/978-3-319-19773-9_86
Quality of LOD Based Semantically Generated Questions
C. Jouault (2015)
10.18653/v1/W15-4410
A System for Generating Multiple Choice Questions: With a Novel Approach for Sentence Selection
M. Majumder (2015)
10.1007/978-3-319-07221-0_37
Content-Dependent Question Generation for History Learning in Semantic Open Learning Space
Corentin Jouault (2014)
10.1007/s00500-013-1141-4
Automatic generation of multiple choice questions using dependency-based semantic relations
Naveed Afzal (2014)
Question Generation from a Knowledge Base with Web Exploration
Linfeng Song (2016)
10.1007/978-3-319-77116-8_49
Automatic Question Generation From Passages
Karen Mazidi (2017)
10.1080/01421590802155597
Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations
James Harold Ware (2009)
10.3115/1118894.1118897
Computer-aided generation of multiple-choice tests
Ruslan Mitkov (2003)
10.1017/S1351324916000024
Developing, evaluating, and refining an automatic generator of diagnostic multiple choice cloze questions to assess children's comprehension while reading
Jack Mostow (2017)
10.1016/j.nepr.2006.07.002
The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments.
M. Tarrant (2006)
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments
S. Banerjee (2005)
Understanding interobserver agreement: the kappa statistic.
A. Viera (2005)



This paper is referenced by
Semantic Scholar Logo Some data provided by SemanticScholar