Fishing for relevance or how to handle large and heterogeneous datasets. Lessons learned from 20 critical reviews in education research
Wed-03
Presented by: Annika Wilmers
The presentation reflects the review processes of 20 critical reviews in the field of digitalization in education, and wants to discuss some of the methodological aspects that are linked to the research field and research topic, namely the handling of large and heterogeneous literature sets and risks of bias (cf. Sutton et al., 2019 for a differentiation of review types and Zawacki-Richter et al., 2020 and Gough et al., 2017 for research syntheses in education science). The reviews are conducted in the meta-project Digi-EBF (collaboration of UDE, DIE, IWM and DIPF). Tasks within this meta-project include support of all research projects on digital education that are funded by the Ministry of Education and Research, reflection on research methodology, evaluation of the funding program, facilitation of knowledge transfer and finally compilation of a set of 20 research syntheses.
To this end, four general topics were chosen that deal with digitalization and education from different perspectives: Competences (reviews 1-5), organizational development (6-10), teaching and learning (11-15) and social learning and participation (16-20). These four broad topics were refined by more specific questions for each single review. Across the team of reviewers, five educational sectors are represented and the workload is divided according to centralized elements (coordination, literature search and transfer) and decentralized elements (screening, analysis and specific transfer activities).
For each set of reviews, we developed a search strategy and then adapted it to the specific research question of each single review including the selection of search terms in German and English, a test search, the main search in databases and additional search tactics like hand searches. Regardless of the scope of the review question (narrow or broader), we found it very difficult to predict any numbers of findings. While at the beginning strategies were needed to improve the recall and precision in the vast field of digitalization, it was then necessary to develop a strategy for dealing with large datasets of findings. In the end, in most cases we had very high numbers of findings but rather low numbers of included studies – with some variations due to characteristics of the specific educational sectors.
In the presentation, we want to discuss the need for a flexible research strategy that can reflect the dynamic research field of digitalization and education as well as risks of bias that can occur at several points, for example when translating highly contextualized search terms. The presentation will conclude by pointing out open science strategies that are linked to this review project, such as data accessibility via the Research Data Centre Education at DIPF.
References
Gough, D., Oliver, S. & Thomas, J. (eds.). (2017). An introduction to systematic reviews. London: SAGE.
Sutton, A., Preston, L. & Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. Health Information & Libraries Journal, 36, pp. 202–222.
Zawacki-Richter, O., Kerres, M., Bedenlier, S., Bond, M. & Buntins, K. (eds.). (2020). Systematic Reviews in Educational Research. Methodology, Perspectives and Application. Wiesbaden: Springer.
To this end, four general topics were chosen that deal with digitalization and education from different perspectives: Competences (reviews 1-5), organizational development (6-10), teaching and learning (11-15) and social learning and participation (16-20). These four broad topics were refined by more specific questions for each single review. Across the team of reviewers, five educational sectors are represented and the workload is divided according to centralized elements (coordination, literature search and transfer) and decentralized elements (screening, analysis and specific transfer activities).
For each set of reviews, we developed a search strategy and then adapted it to the specific research question of each single review including the selection of search terms in German and English, a test search, the main search in databases and additional search tactics like hand searches. Regardless of the scope of the review question (narrow or broader), we found it very difficult to predict any numbers of findings. While at the beginning strategies were needed to improve the recall and precision in the vast field of digitalization, it was then necessary to develop a strategy for dealing with large datasets of findings. In the end, in most cases we had very high numbers of findings but rather low numbers of included studies – with some variations due to characteristics of the specific educational sectors.
In the presentation, we want to discuss the need for a flexible research strategy that can reflect the dynamic research field of digitalization and education as well as risks of bias that can occur at several points, for example when translating highly contextualized search terms. The presentation will conclude by pointing out open science strategies that are linked to this review project, such as data accessibility via the Research Data Centre Education at DIPF.
References
Gough, D., Oliver, S. & Thomas, J. (eds.). (2017). An introduction to systematic reviews. London: SAGE.
Sutton, A., Preston, L. & Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements. Health Information & Libraries Journal, 36, pp. 202–222.
Zawacki-Richter, O., Kerres, M., Bedenlier, S., Bond, M. & Buntins, K. (eds.). (2020). Systematic Reviews in Educational Research. Methodology, Perspectives and Application. Wiesbaden: Springer.