How do you compare apples with oranges? Or even apples with oranges with cherries with bananas with peaches?
This isn’t a joke question that has a pithy punchline.
Is it possible to compare a wide range of very different items and still draw meaningful conclusions?
In terms of research, some might say this is what meta-analysis in education attempts to do.
Combining rigorous studies
Meta-analysis is a research process used to systematically synthesise or combine the results from multiple studies on a particular topic.
Rather than simply collecting the data from various smaller studies, meta-analyses use the most rigorous studies that relate to a particular question or intervention, then take the quantified results and use statistical techniques to combine or integrate them often into a single average value (e.g. effect sizes), therefore providing a map of the evidence.
Meta-analyses are now a common feature of educational research. Considered to be at the highest level of evidence available, researchers, policy makers and practitioners are increasingly engaging with these findings.
However, HJ Eyesnick (1978) dubbed meta-analysis ‘an exercise in mega-silliness’ and Robert Slavin (1986) criticised the approach of integrating results from different studies with different designs and different features stating it ‘combined apples with oranges’.
But to what extent are the metrics from good-quality experiments and meta-analyses really helping us to improve education in practice?
What has worked or what is likely to work?
When using evidence to inform thinking, practice and policy in education, and to make a valid decision about what is likely to work (or not work) in our classrooms, ideally we should not rely on the results obtained from a single study – results can vary from one study to another for a number of reasons.
By combining the data from individual studies broadly related to each in other in a meta-analysis, we can establish an overview of what is known about a given area of research.
The aggregation of findings in meta-analysis offers a generalised picture – as Glass (2000) indicated: ‘Of course it mixes apples and oranges; in the study of fruit nothing else is sensible; comparing apples and oranges is the only endeavour worthy of true scientists; comparing apples to apples is trivial.’
Limitations
There is little doubt that these ‘gold standard’ methods hold real promise for helping us to understand what works.
However, there are risks involved with simplifying so much complex information and there are many who point out the weaknesses of the approach, citing issues such as:
Dylan Wiliam (2016) stated that ‘right now, meta-analysis is simply not a suitable technique for summarizing the relative effectiveness of different approaches to improving student learning’.
The Big Evidence Debate
On the 4 June 2019 CEM hosted The Big Evidence Debate in which invited guests, academics, practitioners and knowledge-brokers, had a chance to share their views and add their voice in panel discussions to some of the issues around meta-analysis in education.
- Is meta-analysis the best we can do? A critical discussion of the best approaches to synthesising research findings.
- Professor Steve Higgins, Professor Nancy Cartwright, Professor Catherine Hewitt, Ben Styles
- Where is the value in meta-analysis in education? A discussion about the benefits and limitations of meta-analysis, and discovering what works in practice.
- Dr Niki Kaiser, Phil Stock, Megan Dixon
- ‘Don’t do meta-analysis’, or ‘Do it more carefully’? Where do we go from here?
- Professor Rob Coe, Professor Stuart Kime, Professor Lee Elliot-Major, Philippa Cordingley
Find out more
Follow us on Twitter @CEMatCambridge and the event using #TBED2019.
Sign up for updates to find out more about CEM's future events.
