A Classical Test Theory Analysis of an English Achievement Test
DOI:
https://doi.org/10.59247/ijase.v5i1.181Keywords:
Classical Test Theory, English Test, Assessment, Reliability, Item Difficulty, Item DiscriminationAbstract
The quality of assessment items is an important aspect of ensuring that English language tests provide reliable and meaningful information about students’ achievement. This study aimed to analyze the quality of a 15 item English language achievement test using Classical Test Theory (CTT). The analysis focused on test reliability, item difficulty, item discrimination, and the effect of item deletion on internal consistency. A quantitative approach was employed using responses from 96 students from two public senior high schools in Muna Regency, Southeast Sulawesi, Indonesia. The data were analyzed using descriptive item analysis and Cronbach’s alpha to evaluate the internal consistency of the assessment. The results indicated that the assessment demonstrated satisfactory internal consistency. The item difficulty analysis showed variation across the items, with items ranging from very difficult to moderate. The discrimination analysis revealed that several items demonstrated good to very good discrimination, while several items showed poor discrimination and require further review. The alpha if deleted analysis indicated that removing individual items did not improve the overall internal consistency of the assessment. These findings suggest that the assessment has an acceptable foundation but still requires item-level refinement. Items with satisfactory discrimination can be retained, whereas poorly discriminating items should be reviewed in terms of item wording, answer keys, distractors, and alignment with the intended learning objectives. The study highlights the usefulness of CTT as a practical approach for identifying problematic items and supporting the improvement of English language assessment quality.
References
Y. Liu et al., “International Psychogeriatrics Harmonized Cognitive Assessment Protocol in the English Longitudinal Study of Ageing: Contrasting approaches to evaluation of factor structure,” Int. Psychogeriatrics, p. 100042, 2025, https://doi.org/10.1016/j.inpsyc.2025.100042.
A. Müller, A. Brenner, and M. Barghamadi, “Benchmarking English proficiency standards in International Nursing Education: A cross-sectional analysis of registration and university entry requirements,” J. Prof. Nurs., vol. 67, pp. 256–266, 2026, https://doi.org/10.1016/j.profnurs.2026.08.019.
W. Yao, “Multimodal Collaborative Assessment of English Speech Based on LLM and Fuzzy Logic,” Syst. Soft Comput., vol. 9, p. 200620, 2026, https://doi.org/10.1016/j.sasc.2026.200620.
P. A. Chala-bejarano, M. Rodríguez-uribe, and S. Patricia, “Pre-service english teachers’ perceptions of language assessment in a colombian language teacher education program,” International Journal of Educational Research Open, vol. 8, 2025, https://doi.org/10.1016/j.ijedro.2024.100405.
L. Stoffelsma, I. N. Mwinlaaru, and W. Spooren, “Testing university students’ morphological awareness of English as a second-language,” System, vol. 130, p. 103626, 2025, https://doi.org/10.1016/j.system.2025.103626.
S. Park, S. B. Piasta, and P. M. Sayer, “Early Childhood Research Quarterly An initial yet rigorous test of multisensory alphabet instruction for english monolingual and emergent bilingual children,” Early Child. Res. Q., vol. 72, pp. 328–338, 2025, https://doi.org/10.1016/j.ecresq.2025.04.008.
C. Mallart et al., “Research Methods in Applied Linguistics Assessing the validity of syntactic alternations as criterial features of proficiency in L2 writings in English,” Res. Methods Appl. Linguist., vol. 4, no. 3, p. 100238, 2025, https://doi.org/10.1016/j.rmal.2025.100238.
B. Naismith, Y. Attali, and G. T. Laflair, “The impact of task duration on the scoring of independent writing responses of adult L2-English writers,” Assess. Writ., vol. 62, p. 100895, 2024, https://doi.org/10.1016/j.asw.2024.100895.
L. Chih-chung and Y. Fu-yun, “Social Sciences & Humanities Open Impact of scaffold structures on student-generated questions in English grammar learning,” Soc. Sci. Humanit. Open, vol. 12, no. 1, p. 102126, 2025, https://doi.org/10.1016/j.ssaho.2025.102126.
K. Barkaoui, J. Holmes, and S. Van Viegen, “Journal of English for Academic Purposes Pre-admission EAP course completion as evidence of English language proficiency for university admission: A longitudinal evaluation study,” J. English Acad. Purp., vol. 77, p. 101555, 2025, https://doi.org/10.1016/j.jeap.2025.101555.
S. Nabhan and A. Habok, “Development and validation of the Teacher Multiliteracies Scale (TMS) for English language teacher education,” System, vol. 131, 2025, https://doi.org/10.1016/j.system.2025.103673.
E. Yisihak, A. Damtew, and K. Terms, “Social Sciences & Humanities Open Primary school English teachers’ practice of teaching early grade reading,” Soc. Sci. Humanit. Open, vol. 9, p. 100840, 2024, https://doi.org/10.1016/j.ssaho.2024.100840.
A. Benati and G. Benati, “The online primary and secondary transfer-of-training effects of structured input on the acquisition of English past tense and third person singular present tense markers: A self-paced reading study measuring accuracy and response time,” Ampersand, vol. 15, p. 100229, 2025, https://doi.org/10.1016/j.amper.2025.100229.
P. Heisterkamp, A. Schüppert, V. Baaijen, and M. Michel, “Learning in English : A research synthesis on European countries with advanced English skills,” Journal of English for Academic Purposes, vol. 77, 2025, https://doi.org/10.1016/j.jeap.2025.101564.
E. C. M. Simatupang, “Social Sciences & Humanities Open Task-based learning and translanguaging: A dual approach to empowering hotel staff in real-world English communication,” Soc. Sci. Humanit. Open, vol. 12, p. 102005, 2025, https://doi.org/10.1016/j.ssaho.2025.102005.
R. Zhang, D. Zou, H. Xie, and F. Lee, “Computers and Education: Artificial Intelligence Engagements with GPT responses and learner prompts in ChatGPT-based learning of English argumentative writing logic and their impacts,” Comput. Educ. Artif. Intell., vol. 9, p. 100489, 2025, https://doi.org/10.1016/j.caeai.2025.100489.
A. Montakantiwong and N. Funada, “Measuring the Global Englishes mindset: Development and validation of the GEM-Q for EFL learners,” System, vol. 134, p. 103847, 2025, https://doi.org/10.1016/j.system.2025.103847.
H. Tran, P. Crosthwaite, Q. Huynh, and P. Pham, “Journal of English for Academic Purposes Students’ self-determination in using machine translation and generative AI tools for English for academic purposes,” J. English Acad. Purp., vol. 78, p. 101578, 2025, https://doi.org/10.1016/j.jeap.2025.101578.
A. Mizumoto, N. Shintani, M. Sasaki, and M. Feng, “Research Methods in Applied Linguistics Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment,” Res. Methods Appl. Linguist., vol. 3, no. 2, p. 100116, 2024, https://doi.org/10.1016/j.rmal.2024.100116.
M. Sánchez-Torrón, E. Ipek, and V. E. Raído, "Creating terminological correspondence recognition tests with GPT-4: a case study in English-to-Turkish translations in the engineering domain," International Journal of Artificial Intelligence in Education, vol. 35, no. 4, pp. 2185-2201, 2025, https://doi.org/10.1007/s40593-025-00465-x.
M. P. Saadi et al., “Assessment of Interrater Reliability in Point-of-Care Ultrasound for Assessing Congestion in Cardiovascular Intensive Care,” ABC Imagem Cardiovascular, vol. 38, no. 2, 2025, https://doi.org/10.36660/abcimg.20250022i.
M. Ibnu, B. Indriyani, H. Inayatullah, and Y. Guntara, “Aplikasi Rasch Model: Pengembangan Instrumen Tes Untuk Mengukur Miskonsepsi Mahasiswa Pada Materi Mekanika,” Prosiding Seminar Nasional Pendidikan FKIP, vol. 2, no. 1, pp. 205–210, 2019, https://jurnal.untirta.ac.id/index.php/psnp/article/view/5669.
P. Levrai and P. García, “Journal of English for Academic Purposes Understanding the collaborative assessment identity of English for academic purposes practitioners in higher education,” J. English Acad. Purp., vol. 79, p. 101621, 2026, https://doi.org/10.1016/j.jeap.2025.101621.
Y. Hwang and H. Lee, “Computers & Education: X Reality An exploratory study on the development of pre-service English teachers’ VR-contextualized testing prototype,” Comput. Educ. X Real., vol. 8, p. 100126, 2026, https://doi.org/10.1016/j.cexr.2025.100126.
Y. Kuo, S. I. C. Wang, E. Zhi, and F. Liu, “Peer assessment using one-to-one tablets to improve the English oral presentation of fourth-grade students in Taiwan,” Comput. Educ. Open, vol. 8, p. 100262, 2025, https://doi.org/10.1016/j.caeo.2025.100262.
Z. Erdil-Moody and A. S. Thompson, “Exploring motivational strategies in higher education: Student and instructor perceptions,” Eurasian J. Appl. Linguist., vol. 6, no. 3, pp. 387–413, 2020, https://doi.org/10.32601/ejal.834670.
Y. Altın, B. Ozkurt, and H. F. Küçükibis, “Examination of the psychometric properties of physical activity enjoyment scale (PACES) based on classical test theory and item response theory,” Acta Psychologica, vol. 261, 2025, https://doi.org/10.1016/j.actpsy.2025.105855.
F. Lalot, J. Räikkönen, and S. Ahvenharju, "An item response theory approach to measurement in environmental psychology: A practical example with environmental risk perception," Journal of Environmental Psychology, vol. 101, 102520, 2025, https://doi.org/10.1016/j.jenvp.2025.102520.
N. Trajtenberg and P. Ezquerra, “An item response theory approach to punitive attitudes,” J. Crim. Justice, vol. 98, p. 102402, 2025, https://doi.org/10.1016/j.jcrimjus.2025.102402.
N. Azizah, M. Suseno, and B. Hayat, “Item analysis of the rasch model items in the final semester exam indonesian language lesson,” World J. English Lang., vol. 12, no. 1, pp. 15–26, Mar. 2022, https://doi.org/10.5430/wjel.v12n1p15.
S. Park, S. B. Piasta, and P. M. Sayer, “Early Childhood Research Quarterly An initial yet rigorous test of multisensory alphabet instruction for english monolingual and emergent bilingual children,” Early Child. Res. Q., vol. 72, pp. 328–338, 2025, https://doi.org/10.1016/j.ecresq.2025.04.008.
L. Y. Tay, C. P. Lim, S. Y. Lye, K. J. Ng, and S. K. Lim, “Open-source learning management system and Web 2.0 online social software applications as learning platforms for an elementary school in Singapore,” Learn. Media Technol., vol. 36, no. 4, pp. 349–365, 2011, https://doi.org/10.1080/17439884.2011.615322.
R. A. N. Grande et al., “Psychometric testing of the nurse professional competence scale Arabic version: An exploratory and confirmatory factor analyses,” Nurse Educ. Pract., vol. 70, 2023, https://doi.org/10.1016/j.nepr.2023.103679.
X. Tang, B. D. Schalet, J. D. Peipert, and D. Cella, “Does Scoring Method Impact Estimation of Significant Individual Changes Assessed by Patient-Reported Outcome Measures? Comparing Classical Test Theory Versus Item Response Theory,” Value Heal., vol. 26, no. 10, pp. 1518–1524, 2023, https://doi.org/10.1016/j.jval.2023.06.002.
O. G. Yildirim and N. Ozdener, “Development and Validation of the Programming Anxiety Scale,” Int. J. Comput. Sci. Educ. Sch., vol. 5, no. 3, pp. 17–34, May 2022, https://doi.org/10.21585/ijcses.v5i3.140.
A. P. Bal, R. Yilmaz, and V. Atas, “Development of self-efficacy scale of differentiated instruction for teachers,” JRAMathEdu (Journal Res. Adv. Math. Educ.), vol. 7, no. 2, pp. 93–104, Apr. 2022, https://doi.org/10.23917/jramathedu.v7i2.16204.
H. Alfiana, H. Karyono, and W. Gunawan, “Analisis Butir Tes Keterampilan Berpikir Kritis dan Pengetahuan Prosedural Grammar Bahasa Inggris,” JINOTEP (Jurnal Inov. dan Teknol. Pembelajaran) Kaji. dan Ris. Dalam Teknol. Pembelajaran, vol. 8, no. 1, pp. 20–29, 2021, https://doi.org/10.17977/um031v8i12021p020.
M. Muchlisin, D. Mardapi, and F. A. Setiawati, “An analysis of Javanese language test characteristic using the Rasch model in R program,” Res. Eval. Educ., vol. 5, no. 1, pp. 61–74, Jun. 2019, https://doi.org/10.21831/reid.v5i1.23773.
S. Menggo and H. C. Darong, “LLT Journal: A Journal on Language and Language Learning http://e-journal.usd.ac.id/index.php/LLT Sanata Dharma University, Yogyakarta, Indonesia,” LLT J. A J. Lang. Lang. Learn., vol. 25, no. 1, pp. 132–148, 2022, https://doi.org/10.24071/llt.v25i1.4159.
T. Bahrani, “Technology as an Assessment Tool in Language Learning,” Int. J. English Linguist., vol. 1, no. 2, pp. 295–298, 2011, https://doi.org/10.5539/ijel.v1n2p295.
A. S. Konca, Ö. Baltaci, and Ö. F. Akbulut, “Problematic Technology Use Scale for Young Children (PTUS-YC): Validity and Reliability Study,” Int. J. Assess. Tools Educ., vol. 9, no. 2, pp. 267–289, Jun. 2022, https://doi.org/10.21449/ijate.888936.
D. Pratama, “Pengembangan Skala Thurstone Metode Equal Appearing Interval untuk Mengukur Sikap Moderasi Beragama Siswa Sekolah Menengah Atas,” J. Psikol. Teor. dan Terap., vol. 11, no. 1, p. 71, 2020, https://doi.org/10.26740/jptt.v11n1.p71-82.
J. R. Rico-juan and M. Arevalillo-herráez, “D-CTT: A deterministic and scalable heuristic for real-time item calibration in adaptive learning systems,” Expert Syst. Appl., vol. 333, p. 134180, 2027, https://doi.org/10.1016/j.eswa.2026.134180.
A. Muhson, B. Lestari, S. Supriyanto, and K. Baroroh, “The development of practical item analysis program for indonesian teachers,” Int. J. Instr., vol. 10, no. 2, pp. 199–210, 2017, https://doi.org/10.12973/iji.2017.10213a.
M. Molaei, P. Fadakar, and M. Shafieetabar, “Psychometric properties and factor structure of the cognitive distortions questionnaire (CD Quest),” Psychiatry Res. Commun., vol. 5, no. 4, p. 100241, 2025, https://doi.org/10.1016/j.psycom.2025.100241.
L. Huang et al., “The Journal of nutrition, health and aging Development validation of a ultra-brief screening tool for sarcopenia in Chinese community-dwelling older adults,” J. Nutr. Heal. aging, vol. 30, no. 9, p. 100928, 2026, https://doi.org/10.1016/j.jnha.2026.100928.
Z. J. Kunicki, M. R. Schick, N. S. Spillane, and L. L. Harlow, “Addictive Behaviors Reports Creation and validation of the barriers to alcohol reduction (BAR) scale using classical test theory and item response theory,” Addict. Behav. Reports, vol. 7, pp. 47–52, 2018, https://doi.org/10.1016/j.abrep.2018.01.004.
W. Y. Hwang, T. K. Shih, Z. H. Ma, R. Shadiev, and S. Y. Chen, “Evaluating listening and speaking skills in a mobile game-based learning environment with situational contexts,” Comput. Assist. Lang. Learn., vol. 29, no. 4, pp. 639–657, 2016, https://doi.org/10.1080/09588221.2015.1016438.
H. Stadthagen-González, L. López, M. C. Parafita Couto, and C. A. Párraga, “Using two-alternative forced choice tasks and Thurstone’s law of comparative judgments for code-switching research,” Linguist. Approaches to Biling., vol. 8, no. 1, pp. 67–97, 2018, https://doi.org/10.1075/lab.16030.sta.
