Where is the value in meta-analysis?

Before I trained as a teacher, I worked as a researcher, investigating the effect of phytoplankton processes in the ocean. It appeared that in certain regions, plant-life was acting as a source of carbon dioxide to the atmosphere, which would have serious implications because it's a greenhouse gas. So I sailed the Atlantic, taking samples of seawater en-route, capturing a snapshot of the gas-balance at each point.

But the oceans are vast, covering ~70% of the Earth’s surface so it would be impossible to physically sample it all. And even if we could, the oceans are dynamic systems, which are affected by factors such as the seasons so our snapshot wouldn’t be representative of the whole year.

On the other hand, we can make continuous, worldwide measurements via satellite, but they offer only a proxy, and the fine detail is lost. So climate scientists work together, using the best available evidence from a number of sources, and use these to help model future climate change.

Using the best available evidence

Similarly, I feel quite comfortable taking overriding messages from a variety of educational research studies and using these to guide my practice.

I understand that we couldn’t possibly hope to include each pupil in every classroom within a particular study, and I realise that we’ll only ever capture a snapshot, which will have to be extrapolated more widely.

But as with climate research, where many people work on separate pieces of the same puzzle, I do believe it’s a valid approach.

The alternative would be that we learn only from our own experience, or from the experience of those around us, both of which are valuable, but neither of which are robust enough to use in isolation.

Meta-analysis as a starting point

For my first few years of teaching, I pretty much did whatever my more experienced colleagues suggested I should do, and I spent a great deal of (very useful) time observing other teachers. But I soon realised that, rather than copying other teachers directly, I needed to work out what the ‘essence’ of their successful teaching was, and find ways of transferring this to my own practice.

This is why I’m interested in using research to inform my practice, and why I think that meta-analysis, despite its drawbacks, is still a useful starting point.

When I first became more interested in educational research, I wondered how we could possibly hope to measure anything. Every class is different and can be affected by all sorts of contemporaneous factors. But this is the point: if an approach is robust enough to survive the blips and bumps that occur every day in the classroom, and if it is shown to be effective overall in a range of settings, then it has to be worth trying.

‘Best bets’

I like the language of “best bets”. An approach that has worked (on average) in a number of settings (in the past) is a better bet than something I happen to see someone else doing, or something I read about on Twitter.

This isn’t to say that we should ignore professional experience, or that advice from others isn’t worth listening to. But because every class is different; because every teacher is an individual; and because teaching conditions change daily, meta-analysis can help us to distil the essential, effective ingredients, so we can tailor them to our own classrooms.

The results from any study are not the end of the story; they are a starting point.

Firstly, it’s important to look below the surface and beyond headline conclusions. Any analysis will have a deeper, more complex story behind it, which leads to its overall effect size.

A good example is feedback, which is known to be very effective, but can be implemented badly, and turn into laborious triple marking or shallow checklists.

Secondly, caution is one of your biggest friends, as a teacher. You need to think about when and how you’re going to implement approaches, and you should evaluate the impact that it has. This doesn’t necessarily mean producing spreadsheets of pupil data, showing steadily increasing grades; sometimes, it’s just as important to evaluate the process. How easy was it to use? Would you use it again?

And most importantly, you need to be prepared to stop doing something, or at least to change the way you’re doing it. It’s very easy to become attached to things that seem shiny, new and valuable on paper. But even the most robust studies can only tell you what has happened previously, not what will happen in your specific context.

As a teacher, my priority is always the children sat in front of me. I’m not an expert in meta-analysis or effect sizes so, to a certain extent, I have to trust the people that have this expertise.

But I am the person that knows my class best, and I do believe that by combining this knowledge with that of many others, I am giving them the best chance to succeed.

About the author:

Niki Kaiser is a teacher and Research Lead, teaching Chemistry to students aged 11-18 and supporting teachers to embed educational research in the classroom. Prior to becoming a teacher, she carried out post-doctoral research in Marine Biogeochemistry, and Niki has combined her interest in research with her love for teaching in a number of ways. She started Journal Clubs for students and colleagues, she was a member of the expert advisory panel for the Education Endowment Foundation’s Improving Secondary Science report, and she was recently awarded a teacher-researcher fellowship from the Royal Society of Chemistry’s Chemical Education Research Group.

Niki mentors school leaders in Norwich to help them integrate and evaluate research-informed practice, she coordinates the Norwich Research Leads Network, and she helped to form #CogSciSci, a peer-support network for teachers with over 700 members. She also serves on the Royal Society of Chemistry’s Education Division Council, and chairs their 11-19 Curriculum and Assessment Working Group.

Find out more:

Follow the Big Evidence Debate on Twitter #TBED2019

Read more about meta-analysis on the CEM blog. Subscribe for CEM updates.