2023-11-10 Jeffrey Buckley
How can we increase the credibility of technology education research?
The volume of research being conducted and published in technology education is growing steadily. Even if there is no indication of a problem, it is important that efforts are invested in making sure the work that is conducted is of high quality and that trust can be placed in published results. With this in mind, over the last 2 years I, with colleagues, have conducted analyses into the replicability and transparency of technology education research. The results suggest that for qualitative research there are methodological details which are often not reported, and for quantitative research it is estimated that approximately 45% of published results will not replicate. Now it is time to ask where do we go from here?
Increasing credibility can take many forms, including conducting replication studies, implementing measures to ensure data analysis is reproducible, following open science practices such as data sharing or preregistering studies, and aligning with transparency criteria when reporting studies. Different stakeholders, such as authors, reviewers, and editors, can all take on different roles, with the major decision being should effort be invested. For example, for a single study a journal reviewer or editor may ask should effort be invested in reproducing the analysis prior to publication to check the reported results. For replication studies there is another question to ask in addition to the investment of effort; where should the effort be invested? There are approximately 2000 articles published in dedicated English language technology education journals, so if a decision is made to invest effort in replicating a study, which of these studies should be selected? With finite resources available, this is an important conversation to have across the field.
Join this seminar to learn about the previous transparency and replicability analyses conducted, to discuss ways in which research can be made more credible and their potential value, and to contribute to the conversation on how to decide which studies should be subjected to a replication attempt. While technology education will serve as the context for this discussion, the questions being asked and solutions being posed are relevant more broadly across STEM education research fields.
About the speaker
Jeff Buckley, PhD, is a Lecturer in Research Pedagogy at the Technological University of the Shannon in Ireland. He received his PhD from the Department of Learning at KTH in 2018, and has since been conducting research in technology and engineering education. His main research interests relate to spatial ability and learning, and educational assessment. However, the ongoing discussion about the replication crisis and credibility revolution in the broader social sciences turned his interests towards finding ways to improve/maintain research credibility in technology education.
Preparatory reading
Buckley, J., Hyland, T., & Seery, N. (2023). Estimating the replicability of technology education research. International Journal of Technology and Design Education, 33(4), 1243–1264. https://doi.org/10.1007/s10798-022-09787-6
Buckley, J., Adams, L., Aribilola, I., Arshad, I., Azeem, M., Bracken, L., Breheny, C., Buckley, C., Chimello, I., Fagan, A., Fitzpatrick, D. P., Garza Herrera, D., Gomes, G. D., Grassick, S., Halligan, E., Hirway, A., Hyland, T., Imtiaz, M. B., Khan, M. B., … Zhang, L. (2022). An assessment of the transparency of contemporary technology education research employing interview-based methodologies. International Journal of Technology and Design Education, 32(4), 1963–1982. https://doi.org/10.1007/s10798-021-09695-1