Item Analysis of an English Achievement Test Using Rasch Model and Classical Test Theory
DOI:
https://doi.org/10.59247/ijase.v3i1.180Keywords:
English Test, Classical Test Theory, IRT (Rasch Model)Abstract
This study aimed to evaluate the quality of English test items developed by the English Teachers Working Group (MGMP) in Muna District, Indonesia. The study specifically examined the psychometric characteristics of the items in terms of item difficulty, discrimination, distractor effectiveness, and item fit using Classical Test Theory (CTT) and Item Response Theory (IRT). This study employed a quantitative psychometric approach. The data consisted of responses to an English achievement test developed and administered by English teachers participating in the MGMP in Muna District. Item quality was initially examined using Classical Test Theory based on item difficulty, discrimination index, and distractor effectiveness. Further analysis was conducted using the Rasch model of Item Response Theory with the Quest program to evaluate item difficulty and item fit based on INFIT Mean Square (MNSQ) statistics. The results of both approaches were used to determine whether individual items were acceptable, required revision, or should be rejected. The analysis revealed variation in the psychometric quality of the test items. The CTT analysis showed differences in item difficulty, discrimination, and distractor effectiveness, while the Rasch analysis revealed variation in item difficulty and item fit. The combined findings indicated that some items demonstrated acceptable psychometric characteristics, whereas others required revision. In addition, several items could not be evaluated using Quest. These findings indicate that although more than half of the items demonstrated acceptable psychometric characteristics, a substantial proportion still requires improvement before being used as a high-quality English assessment.
References
Z. Edger-lacoursière et al., “The MacHAND performance assessment (MPA): Development and psychometric testing of the short English version (MPA-S) for the traumatic hand injury population,” J. Hand Ther., vol. 39, no. 1, pp. 156–167, 2026, https://doi.org/10.1016/j.jht.2025.04.008.
M. Kawata, M. Tsuruta-hamamura, and H. Hasegawa, “Assessment of speech transmission index and reverberation time in standardized English as a foreign language test rooms,” Appl. Acoust., vol. 202, p. 109093, 2022, https://doi.org/10.1016/j.apacoust.2022.109093.
M. Paquot et al., “The Varieties of English for Specific Purposes dAtabase (VESPA): Towards a multi-L1 and multi-register learner corpus of disciplinary writing,” vol. 10, no. 2, pp. 1–15, 2022, https://doi.org/10.32714/ricl.10.02.02.
V. Gamboa, M. P. Paixão, and S. Neves de Jesus, “Internship quality predicts career exploration of high school students,” J. Vocat. Behav., vol. 83, no. 1, pp. 78–87, 2013, https://doi.org/10.1016/j.jvb.2013.02.009.
V. Gamboa, M. P. Paixão, and S. N. de Jesus, “Vocational profiles and internship quality among Portuguese VET students,” Int. J. Educ. Vocat. Guid., vol. 14, no. 2, pp. 221–244, 2014, https://doi.org/10.1007/s10775-014-9268-0.
R. Handapangoda and J. Kormos, “Scene behind the unseen: Special adjustments for visually impaired students in L2 English reading tests,” System, vol. 140, p. 104049, 2026, https://doi.org/10.1016/j.system.2026.104049.
Q. Si, “Deep learning-based English speech recognition and pronunciation assessment system,” Syst. Soft Comput., vol. 9, p. 200570, 2026, https://doi.org/10.1016/j.sasc.2026.200570.
S. Wang, S. Min, H. Wang, X. Gao, and N. Ding, “Computers and Education : Artificial Intelligence Standardized assessment of LLM English proficiency,” Comput. Educ. Artif. Intell., vol. 11, p. 100656, 2026, https://doi.org/10.1016/j.caeai.2026.100656.
Z. Liu, B. Igried, M. Aljaidi, A. Abdullah, A. Fayez, and M. Khishe, “A pedagogically-guided wavelet-enhanced autoencoder for intelligent assessment of english language teaching,” J. Eng. Res., 2026, https://doi.org/10.1016/j.jer.2026.08.014.
L. Qi and P. Shi, “Study on multi-dimensional assessment model of English reading ability based on SHAP and LIME interpretation,” Syst. Soft Comput., vol. 9, p. 200591, 2026, https://doi.org/10.1016/j.sasc.2026.200591.
M. P. Saadi et al., “Assessment of Interrater Reliability in Point-of-Care Ultrasound for Assessing Congestion in Cardiovascular Intensive Care,” ABC Imagem Cardiovascular, 38, no. 2, 2025, https://doi.org/10.36660/abcimg.20250022i.
S. Park, S. B. Piasta, and P. M. Sayer, “Early Childhood Research Quarterly An initial yet rigorous test of multisensory alphabet instruction for english monolingual and emergent bilingual children,” Early Child. Res. Q., vol. 72, pp. 328–338, 2025, https://doi.org/10.1016/j.ecresq.2025.04.008.
W. Zhang et al., “Evaluation of the effectiveness of using flipped classroom in puncture skills teaching,” BMC Med. Educ., vol. 24, no. 1, pp. 1–11, 2024, https://doi.org/10.1186/s12909-024-05132-8.
Y. Kuo, S. I. C. Wang, E. Zhi, and F. Liu, “Peer assessment using one-to-one tablets to improve the English oral presentation of fourth-grade students in Taiwan,” Comput. Educ. Open, vol. 8, p. 100262, 2025, https://doi.org/10.1016/j.caeo.2025.100262.
Y. Liu et al., “Development of the Short Form of the Specific Module of QLICD-BPH (V2.0) for Assessing Quality of Life with Benign Prostatic Hyperplasia by Classical Test Theory, Item Response Theor, and Generalizability Theory,” Eur. Urol. Open Sci., vol. 91, pp. 32–40, 2026, https://doi.org/10.1016/j.euros.2026.05.022.
M. Adler, J. Skånberg, U. Brodin, and F. Lenhard, “Journal of Affective Disorders Reports An item response theory analysis of PHQ-9,” J. Affect. Disord. Reports, vol. 26, p. 101150, 2026, https://doi.org/10.1016/j.jadr.2026.101150.
J. A. Marmara and D. Zarate, “An item response theory examination of the functionality appreciation scale,” Body Image, vol. 58, p. 102133, 2026, https://doi.org/10.1016/j.bodyim.2026.102133.
M. Paz and L. Ezquerra, “Measuring overconfidence: An item response theory approach,” J. Math. Psychol., vol. 130, p. 103003, 2026, https://doi.org/10.1016/j.jmp.2026.103003.
E. Yusron, H. Retnawati, and I. Rafi, “Bagaimana hasil penyetaraan paket tes USBN pada mata pelajaran matematika dengan teori respon butir?,” J. Ris. Pendidik. Mat., vol. 7, no. 1, pp. 1–12, 2020, https://doi.org/10.21831/jrpm.v7i1.31221.
M. Molaei, P. Fadakar, and M. Shafieetabar, “Psychometric properties and factor structure of the cognitive distortions questionnaire (CD Quest),” Psychiatry Res. Commun., vol. 5, no. 4, p. 100241, 2025, https://doi.org/10.1016/j.psycom.2025.100241.
S. A. Livingston, “Equating Test Scores (without IRT),” ETS Rep., pp. 1–80, 2004, [Online]. Available: papers3://publication/uuid/753FF7E7-6A9F-4F37-99FA-5FC927542973
O. N. Bakytbekovich et al., “Distractor Analysis in Multiple-Choice Items Using the Rasch Model,” Int. J. Lang. Test., vol. 13, pp. 69–78, 2023, https://www.ijlt.ir/article_169049.html.
J. R. Rico-juan and M. Arevalillo-herráez, “D-CTT : A deterministic and scalable heuristic for real-time item calibration in adaptive learning systems,” Expert Syst. Appl., vol. 333, p. 134180, 2027, https://doi.org/10.1016/j.eswa.2026.134180.
Y. Altın, B. Ozkurt, and H. F. Küçükibis, “Acta Psychologica Examination of the psychometric properties of physical activity enjoyment scale (PACES) based on classical test theory and item response theory,” vol. 261, 2025, https://doi.org/10.1016/j.actpsy.2025.105855.
L. Kun, Y. Yi, Y. Bing, M. Rui, and G. Lei, “International Journal of Nursing Studies Advances Validation of the short index of job satisfaction in Chinese nurses: classical test theory and item response theory,” Int. J. Nurs. Stud. Adv., vol. 8, p. 100321, 2025, https://doi.org/10.1016/j.ijnsa.2025.100321.
Y. Dai, C. Chen, Z. Lin, J. Qin, and Y. Guo, “Computers & Education Development and validation of a scenario-based multiple-choice test of AI ethical competence (AIEC) using item response theory,” Comput. Educ., vol. 251, p. 105641, 2026, https://doi.org/10.1016/j.compedu.2026.105641.
P. Chen, Y. Li, M. Xiong, and Y. Zhang, “Acta Psychologica Psychometric evaluation of the behavioral inhibition/activation system scales in older adults in Mainland China: A classical test theory and item response theory approach,” vol. 258, no. 783, 2025, https://doi.org/10.1016/j.actpsy.2025.105280.
N. S. Thamrin et al., “Eclectic Approach : A Search for More Effective Assessment Ways to Meet EFL Assessment Principles,” vol. 10, no. 2, pp. 668–685, 2023, https://doi.org/10.24815/siele.v10i2.26682.
L. Huang et al., “The Journal of nutrition, health and aging Development validation of a ultra-brief screening tool for sarcopenia in Chinese community-dwelling older adults,” J. Nutr. Heal. aging, vol. 30, no. 9, p. 100928, 2026, https://doi.org/10.1016/j.jnha.2026.100928.
A. Maharani and N. Putro, “Item Analysis of English Final Semester Test,” Indones. J. EFL Linguist., vol. 5, p. 491, Dec. 2020, https://doi.org/10.21462/ijefl.v5i2.302.
A. Argianti and H. Retnawati, “Characteristics of Math national-standardized school exam test items in junior high school: What must be considered?,” J. Penelit. dan Eval. Pendidik., vol. 24, no. 2, pp. 156–165, 2020, https://doi.org/10.21831/pep.v24i2.32547.
H. H. Dewi, S. M. Damio, and S. Sukarno, “Item analysis of reading comprehension questions for English proficiency test using Rasch model,” Res. Eval. Educ., vol. 9, no. 1, pp. 24–36, Jun. 2023, https://doi.org/10.21831/reid.v9i1.53514.
W. Hartono, S. Hadi, R. Rosnawati, and H. Retnawati, “Uji Kecocokan Model Parameter Logistik Soal Diagnosa Kemampuan Matematika Dasar,” JNPM (Jurnal Nas. Pendidik. Mat.), vol. 6, no. 1, p. 125, 2022, https://doi.org/10.33603/jnpm.v6i1.5899.
Z. Erdil-Moody and A. S. Thompson, “Exploring motivational strategies in higher education: Student and instructor perceptions,” Eurasian J. Appl. Linguist., vol. 6, no. 3, pp. 387–413, 2020, https://doi.org/10.32601/ejal.834670.
S. Handayani, "Analisis uas biologi kelas x dengan teori tes klasik dan item response theory (rasch model)," Bio-Pedagogi: Jurnal Pembelajaran Biologi, vol. 11, no. 2, pp. 76-90, 2022, https://doi.org/10.20961/bio-pedagogi.v11i2.63177.
Kartianom, H. Retnawati, K. Hidayati, and A. H. Dewantara, "Understanding Teachers' Experiences with Large-Scale Diagnostic Assessments in Indonesian Islamic Primary Schools: A Phenomenological Study," Journal of Teaching and Learning, vol. 20, no. 2, pp. 376-389, 2026, https://doi.org/10.22329/jtl.v20i1.9250.
E. Istiyono, W. S. B. Dwandaru, R. Setiawan, and I. Megawati, “Developing of computerized adaptive testing to measure physics higher order thinking skills of senior high school students and its feasibility of use,” Eur. J. Educ. Res., vol. 9, no. 1, pp. 91–101, 2020, https://doi.org/10.12973/eu-jer.9.1.91.
