Systematic reviews and meta-analyses – what’s all the fuss?

Should we be using systematic reviews and meta-analyses as teachers and school leaders to support the decisions we make, or is that a luxury? How do they actually help?

To be able to answer that question, it is helpful to understand what systematic reviews and meta-analyses actually are.

Researchers have always summarised the findings of many studies, trying to understand not just the individual findings but trends – let’s call them the “golden threads of ideas”- that run through.

Often these attempts to synthesise ideas are presented in a literature review, with the researcher highlighting studies they feel are pertinent and have important messages. This is a painstaking process, relying on researchers finding studies that are relevant and timely.

We all tend to emphasise things we agree with and play down aspects that are a little more challenging to our views and this can result in an unbalanced interpretation of the evidence.

A systematic review, and meta-analysis, is a process of synthesising all the findings and identifying the golden threads that run through them in a structured way. The processes that are applied are designed to reduce any unconscious (or conscious) biases we might have, presenting an honest picture of the evidence, and identifying areas where we do not seem to know very much.

It is systematic

There are agreed ways of conducting a systematic review that ensure every effort is taken to ensure no study is ignored. All studies that are relevant are found, read and the interesting and important ideas are collated to identify the golden threads. Once the studies have been identified and unpicked, a statistical analysis, the meta-analysis, can be applied to calculate the average effect of all the studies together.

So, for a teacher or school leader, a systematic review and meta-analysis is a very useful thing. It points us in the right direction, identifying what the evidence suggests are the best bets.

They will never replace the nuanced understanding we have of the context and situations of our schools, teachers and pupils, but they enhance the decisions we make about what to focus on and what we might stop. They are the starting point, not the goal in themselves.

Strategic decision-making

The multi-academy trust I work for uses both systematic reviews and meta-analyses all the time! They underpin the strategic decisions that are made concerning teaching and learning - that is pedagogical decisions about practice in our schools.

We start with a meta-analysis of meta-analyses (generally the EEF’s Teaching and Learning Toolkit and the Early Years Toolkit) to help focus our attention, to decide which areas would be helpful to work on.

From there, we dig deeper into the evidence (maybe through a systematic review, a guidance report, or individual research studies) to help us work up the pedagogy and understand what it looks like, sounds like and feels like in the classroom. This gives us a clear picture of what we should focus our attention on and the direction we are travelling in.

We do this at many levels; in a classroom, across a school, between and beyond schools.

An example…

Three years ago we were approached by the Local Authority EYFS team to collaborate on a piece of work aimed at raising attainment in early years classrooms, in literacy, for the schools in the LA with the greatest numbers of children in receipt of pupil premium.

Our trust schools generally have good outcomes, with great progress for our youngest, vulnerable learners – we think it is really important.

So…the first thing we did was go to the EEF Early Years Toolkit and look at the headline summary about what would be a best bet to focus on. From there, we identified several areas of promise:

• teaching of early phonics with letter knowledge and phonological awareness
• storytelling and group reading
• developing early writing: communication and language approaches (including explicit approaches to developing vocabulary).

Once we had identified these themes we started to look at the research in more detail (including the EEF Guidance report on Preparing for Literacy).

An audit for teachers

We began to develop a programme of professional learning for schools to be involved in.

We developed an audit for the teachers to use in their classrooms. The audit was intended to help them evaluate their teaching in relation to the practices the research had identified as being effective for young learners in this context. Completing the audit helped the teachers evaluate what was successful for their children and areas they wished to develop.

The next step was the training options we provided. Again, carefully following the research, we developed 6 training courses that the teachers could choose from. They selected two to attend and worked together, with a mentor to develop the new practices they learnt in their own classrooms. At the end of the year, we held a conference for everyone to share their experiences.

Success

The success of the project was shown in three different and important ways:

Firstly the teachers all maintained their involvement for the whole year.

Secondly, we were asked to carry on the project for a second (and now a third) year.

Thirdly, the numerical data showed that the children in the classes we had worked with had made more progress and had some better outcomes at the end of the reception year than the other schools across the LA.

It was a really exciting case of evidence to practice!

About the author:

Megan is the North West Regional Lead, working in the Dissemination and Impact team. Her role includes working strategically across the North West to widen the reach and impact of the EEF and the Research School Network.

Megan joined the EEF, on secondment in January 2019, from her substantive role as Director of the Aspirer Research School and Director of English for the Aspire Educational Trust. With almost twenty years’ experience in the classroom, she is a specialist literacy teacher and trainer, with considerable expertise in teaching children who find it hardest to learn. Prior to working at the Aspire Educational Trust, she has worked as a class teacher, Local Authority Literacy consultant and school improvement advisor and trainer.

Find out more:

Follow the Big Evidence Debate on Twitter #TBED2019

Read more from Megan: Why does Vocabulary Matter? and Translating Evidence into Practice