Adaptive Testing for programming logic skills assessment

Lucas Montagnani Calil Elias; Francisco de Assis Zampirolli

doi:10.47236/2594-7036.2026.v10.1905

Authors

Lucas Montagnani Calil Elias Federal University of ABC https://orcid.org/0009-0006-4746-1551
Francisco de Assis Zampirolli Federal University of ABC https://orcid.org/0000-0002-7707-1793

DOI:

https://doi.org/10.47236/2594-7036.2026.v10.1905

Keywords:

Adaptive Testing, Education, Item response theory, Programming logic, Maximum likelihood estimation

Abstract

Adaptive Testing (AT) enhances learning outcomes by adjusting assessments to students’ proficiency levels. This paper presents adaptive methods for evaluating programming logic skills, implemented in an open-source system named MCTest. In this system, teachers create ATs tailored for their students. Three adaptive methods were developed: Semi-AT (SAT), Weighted Probability of Correction (WPC), and Maximum Likelihood Estimation (MLE). Six tests were designed, including a non-adaptive baseline, with multiple-choice questions classified according to Bloom’s Taxonomy. These tests validated item calibrations using Item Response Theory. The method was applied in two classes with 72 students, and a final questionnaire with 17 respondents statistically confirmed its perceived effectiveness.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

Author Biographies

Lucas Montagnani Calil Elias, Federal University of ABC

Bachelor’s Degree in Computer Science at the Federal University of ABC. Santo André, São Paulo, Brazil. Email address: lucas.montagnani@aluno.ufabc.edu.br. Orcid: https://orcid.org/0009-0006-4746-1551. Lattes Curriculum: http://lattes.cnpq.br/3276534519963546.

Francisco de Assis Zampirolli, Federal University of ABC

Ph.D. in Electrical Engineering from State University of Campinas. Full Professor of Computer Science at the Federal University of ABC. Santo André, São Paulo, Brazil. Email address: fzampirolli@ufabc.edu.br. Orcid: https://orcid.org/0000-0002-7707-1793. Lattes Curriculum: http://lattes.cnpq.br/4127260763254001.

References

ALVES, Laura Filállepe et al. Continuous assessment, a teaching methodology for reducing retention and dropout rates in higher education calculus courses. Revista Sítio Novo, Palmas, v. 6, n. 4, p. 51-60, 2022. DOI: 10.47236/2594-7036.2022.v6.i4.51-60p. DOI: https://doi.org/10.47236/2594-7036.2022.v6.i4.51-60p

ALVES, Lynn et al. Remote education: between illusion and reality. Interfaces Científicas-Educação, v. 8, n. 3, p. 348-365, 2020. DOI: https://doi.org/10.17564/2316-3828.2020v8n3p348-365

ALVES, Welington Domingos; SANTOS, Luiz Gustavo Fernandes dos. Playing with mathematics: using games to mediate the teaching and learning of mathematical content. Revista Sítio Novo, Palmas, v. 6, n. 4, p. 84-93, 2022. DOI: 10.47236/2594-7036.2022.v6.i4.84-93p. DOI: https://doi.org/10.47236/2594-7036.2022.v6.i4.84-93p

BAKER, Frank B. et al. The basics of item response theory using R. v. 969. Springer, 2017. DOI: https://doi.org/10.1007/978-3-319-54205-8

BAYLARI, Ahmad; MONTAZER, Gh A. Design a personalized e-learning system based on item response theory and artificial neural network approach. Expert Systems with Applications, v. 36, n. 4, p. 8013-8021, 2009. DOI: https://doi.org/10.1016/j.eswa.2008.10.080

BECKER, Samantha Adams et al. NMC horizon report: 2018 higher education edition. Louisville, CO: Educause, 2018.

BINH, Hoang Tieu; DUY, Bui The. Student ability estimation based on IRT. In: National Foundation For Science And Technology Development Conference On Information And Computer Science (NICS), 3., 2016. [S. l.], 2016. p. 56-61. DOI: https://doi.org/10.1109/NICS.2016.7725667

CAI, Li et al. Item response theory. Annual Review of Statistics and Its Application, v. 3, p. 297-321, 2016. DOI: https://doi.org/10.1146/annurev-statistics-041715-033702

CHEN, Keyu. A comparison of fixed item parameter calibration methods and reporting score scales in the development of an item pool. 2019. Tese (PhD) - University of lowa, [S. l.], 2019.

CHOI, Younyoung; MCCLENEN, Cayce. Development of adaptive formative assessment system using computerized adaptive testing and dynamic bayesian networks. Applied Sciences, v. 10, n. 22, p. 8196, 2020. DOI: https://doi.org/10.3390/app10228196

COHEN, Jacob. Statistical power analysis for the behavioral sciences. 2. ed. Hillsdale, NJ: Erlbaum, 1988.

COSTA, Rebeca Soler et al. Personalized and adaptive learning: educational practice and technological impact. Texto Livre, v. 14, p. e33445, 2022. DOI: https://doi.org/10.35699/1983-3652.2021.33445

GALVAO, Ailton Fonseca et al. An intelligent model for item selection in computerized adaptive testing. 2013. Dissertation (Master's) - Federal University of Juiz de Fora 2013. (UFJF), Juiz de Fora, MG, 2013.

GHAVIFEKR, Simin; ROSDY, Wan Athirah Wan. Teaching and learning with technology: Effectiveness of ICT integration in schools. International journal of research in education and science, v. 1, n. 2, p. 175-191, 2015. DOI: https://doi.org/10.21890/ijres.23596

GROS, Begoña. The design of smart educational environments. Smart learning environments, v. 3, p. 1-11, 2016. DOI: https://doi.org/10.1186/s40561-016-0039-x

HAMDARE, S. An adaptive evaluation system to test student caliber using item response theory. International Journal of Modern Trends in Engineering and Research, v. 1, n. 5, p. 329-333, 2014.

HAMMOND, Flora et al. Handbook for clinical research: design, statistics, and implementation. Demos Medical Publishing, 2014.

HOLM, Sture. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, v. 6, n. 2, p. 65-70, 1979.

JOHNSON, Amy M. et al. Challenges and solutions when using technologies in the classroom. In: Adaptive educational technologies for literacy instruction. Routledge, 2016. p. 13-30. DOI: https://doi.org/10.4324/9781315647500-2

KARINO, Camila Akemi; SOUSA, Eduardo Carvalho. Understanding your ENEM score - Participant's Guide. [S. l.: s. n.], 2012.

KRATHWOHL, David R. A Revision of Bloom's Taxonomy: An Overview. Theory Into Practice, v. 41, n. 4, p. 212-218, 2002. DOI: https://doi.org/10.1207/s15430421tip4104_2

LAZARINIS, Fotis et al. Creating personalized assessments based on learner knowledge and objectives in a hypermedia Web testing application. Computers & Education, v. 55, n. 4, p. 1732-1743, 2010. DOI: https://doi.org/10.1016/j.compedu.2010.07.019

LORD, Frederic M. Applications of item response theory to practical testing problems. Routledge, 1980.

MEIJER, Rob R.; NERING, Michael L. Computerized adaptive testing: Overview and introduction. Applied psychological measurement, 1999. DOI: https://doi.org/10.1177/01466219922031310

MIN, Shangchao; ARYADOUST, Vahid. A systematic review of item response theory in language assessment: Implications for the dimensionality of language ability. Studies in Educational Evaluation, v. 68, p. 100963, 2021. DOI: https://doi.org/10.1016/j.stueduc.2020.100963

MORAN, José. Hybrid education: a key concept for education today. In: Hybrid teaching: personalization and technology in education. Porto Alegre: Penso, 2015.

MOREIRA, José António; SCHLEMMER, Eliane.

Towards a new concept and paradigm for onlife digital education. Revista uFG, v. 20, 2020.

NEVES, Rogério; ZAMPIROLLI, Francisco de Assis. Processing information: a practical book on language-independent programming. São Bernardo do Campo: EdUFABC, 2017.

OLIVEIRA, Plínio Cardoso de; SOUZA, Wallysonn Alves de; ALVES, José Robson Mariano. Contesting: software to stimulate critical thinking, engagement, and promote autonomy in professional and technological education. Revista Sítio Novo, Palmas, v. 9, p. e1611, 2025. DOI: https://doi.org/10.47236/2594-7036.2025.v9.1611

PARAMYTHIS, Alexandros; LOIDL-REISINGER, Susanne. Adaptive learning environments and e-learning standards. In: European Conference On E-learning, 2., 2003. [S. l.], 2003. p. 369-379.

PELLEGRINO, James W.; QUELLMALZ, Edys S. Perspectives on the integration of technology and assessment. Journal of Research on Technology in Education, v. 43, n. 2, p. 119-134, 2010. DOI: https://doi.org/10.1080/15391523.2010.10782565

PONTES, Paulo Ricardo da Silva; VICTOR, Valci Ferreira. Educational robotics: a practical approach to teaching programming logic. Revista Sítio Novo, Palmas, v. 6, n. 1, p. 57-71, 2022. DOI: 10.47236/2594-7036.2022.v6.i1.57-71p. DOI: https://doi.org/10.47236/2594-7036.2022.v6.i1.57-71p

PUGLIESE, Lou. Adaptive learning systems: Surviving the storm. Educause review, v. 10, n. 7, 2016.

ROSENTHAL, Robert. Meta-analytic procedures for social research. Beverly Hills: Sage, 1984.

SHAPIRO, Samuel S.; WILK, Martin B. An analysis of variance test for normality (complete samples). Biometrika, v. 52, n. 3-4, p. 591-611, 1965. DOI: https://doi.org/10.1093/biomet/52.3-4.591

SOARES, Ronald Ruan Pereira et al. Development of a virtual assistant as academic support for the bachelor's degree program in Computer Science using customized generative AI and RAG. Revista Sítio Novo, Palmas, v. 9, p. e1757, 2025. DOI: https://doi.org/10.47236/2594-7036.2025.v9.1757

TUKEY, John W. Box-and-whisker plots. In: Exploratory data analysis. [S. l.: s. n.], 1977. p. 39-43.

WAINER, Howard et al. Computerized adaptive testing: A primer. Lawrence Erlbaum Associates, Inc, 1990.

WANG, Feng-Hsu. Application of componential IRT model for diagnostic test in a standard-conformant eLearning system. In: IEEE International Conference On Advanced Learning Technologies (ICALT'06), 6., 2006. [S. l.], 2006. p. 237-241. DOI: https://doi.org/10.1109/ICALT.2006.1652414

WATERS, John K. The great adaptive learning experiment. Campus Technology, v. 16, 2014.

WILCOXON, Frank. Individual comparisons by ranking methods. Biometrics Bulletin, v. 1, n. 6, p. 80-83, 1945. DOI: https://doi.org/10.2307/3001968

YANG, Albert C. M. et al. Adaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning. Computers and Education: Artificial Intelligence, v. 3, p. 100104, 2022. DOI: https://doi.org/10.1016/j.caeai.2022.100104

ZAMPIROLLI, Francisco de Assis. MCTest: How to create and correct automatically parameterized exams. Brazil: Independently Published, 2023.

ZAMPIROLLI, Francisco de Assis et al. An experience of automated assessment in a large-scale introduction programming course. Computer Applications in Engineering Education, p. 1284-1299, 2021. DOI: https://doi.org/10.1002/cae.22385

ZAMPIROLLI, Francisco de Assis et al. Evaluation process for an introductory programming course using blended learning in engineering education. Computer Applications in Engineering Education, 2018. p. 1-13.

ZHENG, Yi. New methods of online calibration for item bank replenishment. 2014. Tese (PhD) - University of Illinois at Urbana-Champaign, [S. l.], 2014.