Fiona Concannon*1
Eamon Costello 2
Tom Farrelly 3
1 National University of Ireland Galway, Galway, Ireland
2 Dublin City University, Dublin, Ireland
3 Tralee Institute of Technology, Tralee, Ireland
Educational technology as a broad and applied interdisciplinary research field faces challenges in achieving consensus on what constitutes good quality research. As the field is embedded in many other disciplines, considering what evidence matters and the optimal methodologies to conduct inquiry is continually evolving and maturing. Inhabiting a boundary between education, and computer science, and viewed through numerous theoretical lenses ranging from disciplines of sociology, politics, psychology, the learning systems, curriculum development, digital humanities, and beyond, the number of approaches contributing to the field is vast. The validity, trustworthiness and integrity of over two decades of research in this domain is continually questioned. Furthermore, as technology itself also changes, there are differing opinions on how best to explore and understand the role it plays in education. How we define, research and evaluate our evidence is central to our understanding of how we learn and how this is enhanced with and through technology in various ways. Whilst scholars continue to critique and debate the veracity of findings, educational technology journals play an important role in allowing us to collectively peer review, and publish the best quality research studies. Changes in the open access publishing world and in the open science movement have the potential to address some of shortfalls in how our understandings are evaluated, critiqued and judged in this domain.
Recent critiques of educational technology have focused on researcher links with commercial vendors (Selwyn, 2017; Watters, 2014) weak theory (Hew, Lan, Tang, Jia, & Lo, 2019; Jones & Czerniewicz, 2011) and methodological shortcomings (Baker, 2016; Bulfin, Henderson, Johnson, & Selwyn, 2014). Technology is also continually evolving with reports each year heralding trends in the latest new tool (Bryan Alexander et al., 2019). This concern around research quality isn’t confined to this domain. Related fields of psychology have similarly encountered a reproducibility crisis (Baker, 2016). Researchers have identified questionable practices, such as failing to report all dependent measures, selectively reporting studies that “worked”, or claiming to have predicted unexpected findings. (John, Loewenstein, & Prelec, 2012).
Some of the causes of these practices include the perverse incentives in academic work, where publish or perish holds strong, and new faculty positions and tenure are directly linked to a researchers publication rate. Along with promotions and tenure being linked to publication records, commercial journal publishers benefit from free labour, which in turn creates a culture where incentives for publishing research findings with high statistical significance and impact rather than also failures and non-significant results (Dobele & Rundle‐Theile, 2015; Santos & Horta, 2018). Some have taken an even stronger position, suggesting that the result of this culture of research has resulted in a situation where most published research findings are false, relying on low statistical power, poor control for bias and questionable research practices (Ioannidis, 2005).
In the last twenty years, there have been key developments based on Open Science and Open Scholarship, and a resurgence in open research practices. The Budapest Open Access Initiative (BOAI, 2002) was a pivotal moment in the definition of openness, and a call to action for openness of all publicly funded work. Open access differs from subscription-based publication, where articles are published but inaccessible due to copyright, licencing and paywall constraints.
2013) authors are enabled to publish scholarly articles in a free and open access journal, without charging them fees. Through peer review, publicly funded scholars giving their time and
expertise. Via the support of the Irish Edtech community, hosting and associated publishing costs are met, so that authors retain copyright and license their work through a permissive creative commons license (CC-BY 4.0), without any lower publishing standards. Furthermore, the openness helps authors amplify their impact.
More recently, a global movement to make research and educational practices more transparent, under the Open Science umbrella term, promisingly, include a set of practices and principles that have extended into all aspects of the research cycle. These have the potential to address some of the challenges and problems noted above that are fundamental to teaching and learning research quality/success, and will be considered next.
A potential solution arising from Open Science, with direct application to educational research is to instil scientific rigor at the grassroots level. Working collaboratively across institutions can help us to achieve better quality research outputs (current and future) and help us to increase power, generalisability and transparency of research data. Practical research practices, such as pre- registration, open data, open resources, are central to this change. Collectively, written protocol & a- priori statistical analysis plans can be published on the Open Science Framework before data collection starts, to improve the efficiency and access to research, and resultant outputs. Additional solutions proposed include protocol and analysis plan pre-registration and FAIR or CONSORT reporting guidelines, and consortia pooling resources to maximise sample size and large scale collaborations. Furthermore, systematic reviews encourage transparent and clear reporting of data and methods. Others have argued for more equity and partnerships in the process of educational research including a clearer definition of roles, including practitioners, researchers, and funders to promote larger scale research findings, with participatory design goals, such as those proposed by the Design Based Implementation Research (Penuel, 2019). It is clear that there are immediate improvements from the Open Science movement in how we support and carry out our research, that we later publish openly as peer-reviewed research work, as free to read, redistribute and reuse, with some restrictions as a journal article.
The intention of open access journals such as the Irish Journal of Technology Enhanced Learning is to negate claims made that are exaggerated, unsubstantiated, contradictory, and even untrustworthy.
Diverse methodologies, and a critical analysis will continue to be a feature of research in this meta domain, but more transparency and openness into all aspects of the research process, including the published outputs will be a very positive direction for the future. The challenge now is for us to work together towards creating a shared understanding of what works and doesn’t in the field of educational technology and how practices from the Open Science movement might become a supported and defined part of our educational research practices, resulting in better quality published research findings that are open and accessible to all.
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452-454,. BOAI. (2002). Budapest Open Access Initiative. Retrieved May 18, 2019, from Read the Budapest open access initiative. website: https://www.budapestopenaccessinitiative.org/read
Bryan Alexander, Kevin Ashford-Rowe, Noreen Barajas-Murphy, Gregory Dobbin, Jessica Knott, Mark McCormack, … Nicole Weber. (2019). NMC horizon report: 2019 higher education edition. The New Media Consortium.
Bulfin, S., Henderson, M., Johnson, N. F., & Selwyn, N. (2014). Methodological capacity within the field of “educational technology” research: An initial investigation. British Journal of Educational Technology, 45(3), 403–414.
Dobele, A. R., & Rundle‐Theile, S. (2015). Progression through academic ranks: A longitudinal examination of internal promotion drivers. Higher Education Quarterly, 69(4), 410–429.
Fuchs, C., & Sandoval, M. (2013). The diamond model of open access publishing: Why policy makers, scholars, universities, libraries, labour unions and the publishing world need to take non-commercial, non-profit open access serious. TripleC: Communication, Capitalism & Critique. 11(2), 428–443.
Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the “theory” within the field of educational technology research? British Journal of Educational Technology, 50(3), 956–971. https://doi.org/10.1111/bjet.12770
Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.
Jones, C., & Czerniewicz, L. (2011). Theory in learning technology. Research in Learning Technology, 19(3).
Penuel, W. R. (2019). Infrastructuring as a Practice of Design-Based Research for Supporting and Studying Equitable Implementation and Sustainability of Innovations. Journal of the Learning Sciences, 1–19. https://doi.org/10.1080/10508406.2018.1552151
Santos, J. M., & Horta, H. (2018). The research agenda setting of higher education researchers.
Higher Education, 76(4), 649–668.
Selwyn, N. (2017). Education and technology: key issues and debates (Second edition). London New York: Bloomsbury Academic.
Watters, A. (2014). The monsters of education technology. CreateSpace Independent Publishing Platform.
* Corresponding author. Email: Fiona.concannnon@nuigalway.ie Irish Journal of Technology Enhanced Learning Ireland, 2019.© F. Concannon, E. Costello, & T. Farrelly, 2019 The Irish Journal of Technology Enhanced Learning Ireland is the journal of the Irish Learning Technology Association, an Irish-based professional and scholarly society and membership organisation. (CRO# 520231) http://www.ilta.ie/ . This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.