Adaptive Testing for programming logic skills assessment
DOI:
https://doi.org/10.47236/2594-7036.2026.v10.1905Keywords:
Adaptive Testing, Education, Item response theory, Programming logic, Maximum likelihood estimationAbstract
Adaptive Testing (AT) enhances learning outcomes by adjusting assessments to students’ proficiency levels. This paper presents adaptive methods for evaluating programming logic skills, implemented in an open-source system named MCTest. In this system, teachers create ATs tailored for their students. Three adaptive methods were developed: Semi-AT (SAT), Weighted Probability of Correction (WPC), and Maximum Likelihood Estimation (MLE). Six tests were designed, including a non-adaptive baseline, with multiple-choice questions classified according to Bloom’s Taxonomy. These tests validated item calibrations using Item Response Theory. The method was applied in two classes with 72 students, and a final questionnaire with 17 respondents statistically confirmed its perceived effectiveness.Downloads
References
ALVES, Laura Filállepe et al. Continuous assessment, a teaching methodology for reducing retention and dropout rates in higher education calculus courses. Revista Sítio Novo, Palmas, v. 6, n. 4, p. 51-60, 2022. DOI: 10.47236/2594-7036.2022.v6.i4.51-60p.
ALVES, Lynn et al. Remote education: between illusion and reality. Interfaces Científicas-Educação, v. 8, n. 3, p. 348-365, 2020.
ALVES, Welington Domingos; SANTOS, Luiz Gustavo Fernandes dos. Playing with mathematics: using games to mediate the teaching and learning of mathematical content. Revista Sítio Novo, Palmas, v. 6, n. 4, p. 84-93, 2022. DOI: 10.47236/2594-7036.2022.v6.i4.84-93p.
BAKER, Frank B. et al. The basics of item response theory using R. v. 969. Springer, 2017.
BAYLARI, Ahmad; MONTAZER, Gh A. Design a personalized e-learning system based on item response theory and artificial neural network approach. Expert Systems with Applications, v. 36, n. 4, p. 8013-8021, 2009.
BECKER, Samantha Adams et al. NMC horizon report: 2018 higher education edition. Louisville, CO: Educause, 2018.
BINH, Hoang Tieu; DUY, Bui The. Student ability estimation based on IRT. In: National Foundation For Science And Technology Development Conference On Information And Computer Science (NICS), 3., 2016. [S. l.], 2016. p. 56-61.
CAI, Li et al. Item response theory. Annual Review of Statistics and Its Application, v. 3, p. 297-321, 2016.
CHEN, Keyu. A comparison of fixed item parameter calibration methods and reporting score scales in the development of an item pool. 2019. Tese (PhD) - University of lowa, [S. l.], 2019.
CHOI, Younyoung; MCCLENEN, Cayce. Development of adaptive formative assessment system using computerized adaptive testing and dynamic bayesian networks. Applied Sciences, v. 10, n. 22, p. 8196, 2020.
COHEN, Jacob. Statistical power analysis for the behavioral sciences. 2. ed. Hillsdale, NJ: Erlbaum, 1988.
COSTA, Rebeca Soler et al. Personalized and adaptive learning: educational practice and technological impact. Texto Livre, v. 14, p. e33445, 2022.
GALVAO, Ailton Fonseca et al. An intelligent model for item selection in computerized adaptive testing. 2013. Dissertation (Master's) - Federal University of Juiz de Fora 2013. (UFJF), Juiz de Fora, MG, 2013.
GHAVIFEKR, Simin; ROSDY, Wan Athirah Wan. Teaching and learning with technology: Effectiveness of ICT integration in schools. International journal of research in education and science, v. 1, n. 2, p. 175-191, 2015.
GROS, Begoña. The design of smart educational environments. Smart learning environments, v. 3, p. 1-11, 2016.
HAMDARE, S. An adaptive evaluation system to test student caliber using item response theory. International Journal of Modern Trends in Engineering and Research, v. 1, n. 5, p. 329-333, 2014.
HAMMOND, Flora et al. Handbook for clinical research: design, statistics, and implementation. Demos Medical Publishing, 2014.
HOLM, Sture. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, v. 6, n. 2, p. 65-70, 1979.
JOHNSON, Amy M. et al. Challenges and solutions when using technologies in the classroom. In: Adaptive educational technologies for literacy instruction. Routledge, 2016. p. 13-30.
KARINO, Camila Akemi; SOUSA, Eduardo Carvalho. Understanding your ENEM score - Participant's Guide. [S. l.: s. n.], 2012.
KRATHWOHL, David R. A Revision of Bloom's Taxonomy: An Overview. Theory Into Practice, v. 41, n. 4, p. 212-218, 2002.
LAZARINIS, Fotis et al. Creating personalized assessments based on learner knowledge and objectives in a hypermedia Web testing application. Computers & Education, v. 55, n. 4, p. 1732-1743, 2010.
LORD, Frederic M. Applications of item response theory to practical testing problems. Routledge, 1980.
MEIJER, Rob R.; NERING, Michael L. Computerized adaptive testing: Overview and introduction. Applied psychological measurement, 1999.
MIN, Shangchao; ARYADOUST, Vahid. A systematic review of item response theory in language assessment: Implications for the dimensionality of language ability. Studies in Educational Evaluation, v. 68, p. 100963, 2021.
MORAN, José. Hybrid education: a key concept for education today. In: Hybrid teaching: personalization and technology in education. Porto Alegre: Penso, 2015.
MOREIRA, José António; SCHLEMMER, Eliane.
Towards a new concept and paradigm for onlife digital education. Revista uFG, v. 20, 2020.
NEVES, Rogério; ZAMPIROLLI, Francisco de Assis. Processing information: a practical book on language-independent programming. São Bernardo do Campo: EdUFABC, 2017.
OLIVEIRA, Plínio Cardoso de; SOUZA, Wallysonn Alves de; ALVES, José Robson Mariano. Contesting: software to stimulate critical thinking, engagement, and promote autonomy in professional and technological education. Revista Sítio Novo, Palmas, v. 9, p. e1611, 2025.
PARAMYTHIS, Alexandros; LOIDL-REISINGER, Susanne. Adaptive learning environments and e-learning standards. In: European Conference On E-learning, 2., 2003. [S. l.], 2003. p. 369-379.
PELLEGRINO, James W.; QUELLMALZ, Edys S. Perspectives on the integration of technology and assessment. Journal of Research on Technology in Education, v. 43, n. 2, p. 119-134, 2010.
PONTES, Paulo Ricardo da Silva; VICTOR, Valci Ferreira. Educational robotics: a practical approach to teaching programming logic. Revista Sítio Novo, Palmas, v. 6, n. 1, p. 57-71, 2022. DOI: 10.47236/2594-7036.2022.v6.i1.57-71p.
PUGLIESE, Lou. Adaptive learning systems: Surviving the storm. Educause review, v. 10, n. 7, 2016.
ROSENTHAL, Robert. Meta-analytic procedures for social research. Beverly Hills: Sage, 1984.
SHAPIRO, Samuel S.; WILK, Martin B. An analysis of variance test for normality (complete samples). Biometrika, v. 52, n. 3-4, p. 591-611, 1965.
SOARES, Ronald Ruan Pereira et al. Development of a virtual assistant as academic support for the bachelor's degree program in Computer Science using customized generative AI and RAG. Revista Sítio Novo, Palmas, v. 9, p. e1757, 2025.
TUKEY, John W. Box-and-whisker plots. In: Exploratory data analysis. [S. l.: s. n.], 1977. p. 39-43.
WAINER, Howard et al. Computerized adaptive testing: A primer. Lawrence Erlbaum Associates, Inc, 1990.
WANG, Feng-Hsu. Application of componential IRT model for diagnostic test in a standard-conformant eLearning system. In: IEEE International Conference On Advanced Learning Technologies (ICALT'06), 6., 2006. [S. l.], 2006. p. 237-241.
WATERS, John K. The great adaptive learning experiment. Campus Technology, v. 16, 2014.
WILCOXON, Frank. Individual comparisons by ranking methods. Biometrics Bulletin, v. 1, n. 6, p. 80-83, 1945.
YANG, Albert C. M. et al. Adaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning. Computers and Education: Artificial Intelligence, v. 3, p. 100104, 2022.
ZAMPIROLLI, Francisco de Assis. MCTest: How to create and correct automatically parameterized exams. Brazil: Independently Published, 2023.
ZAMPIROLLI, Francisco de Assis et al. An experience of automated assessment in a large-scale introduction programming course. Computer Applications in Engineering Education, p. 1284-1299, 2021.
ZAMPIROLLI, Francisco de Assis et al. Evaluation process for an introductory programming course using blended learning in engineering education. Computer Applications in Engineering Education, 2018. p. 1-13.
ZHENG, Yi. New methods of online calibration for item bank replenishment. 2014. Tese (PhD) - University of Illinois at Urbana-Champaign, [S. l.], 2014.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Lucas Montagnani Calil Elias, Francisco de Assis Zampirolli
This work is licensed under a Creative Commons Attribution 4.0 International License.
It allows sharing, adaptation, and use for any purpose, including commercial use, provided proper attribution is given to the authors and to Revista Sítio Novo.
The authors declare that the work is original and has not been previously published, in whole or in part, except on recognized preprint servers, if declared, and that no other similar manuscript authored by them is published or under review by another journal, whether printed or electronic.
They declare that they have not violated or infringed upon any proprietary rights of others, and that all citations in the text are factual or based on research with scientifically significant accuracy.
The authors retain the copyright of the manuscripts published in this journal, allowing unrestricted use of their content, provided that the original authorship and the publication source are properly cited.
