Evaluating Impact

Each year, the education system spends astonishing amounts of money on implementing educational initiatives which often yield only negligible impact on learners.

There is a danger that time-poor educators spend too many of their precious hours putting new things in place with little real evidence that what they are doing is having the impact that they want to have or adding any educational value. Teachers want to do what is best for their pupils and schools but with so much noise in the system about this program or that intervention its difficult for them to make the right choices for their school.

What’s the difference between monitoring progress and evaluating impact?

Monitoring individuals’ progress is a familiar process in all schools and when we think about measuring progress we are thinking about making sure that each child, no matter what their background and situation, are able to make an amount of progress that is appropriate for them.

Evaluating impact is about progress too but is about understanding which are the most beneficial ways of helping children make progress in particular areas. Evaluating impact is about understanding what tools, instruments, strategies or interventions you can put in place that maximise the progress that each student will benefit from.

How do you know it is giving you the impact you think it is giving?

There are many tools commercially available that will tell you they are specifically designed to help you implement strategies to bring improvements in specific areas such as, phonological awareness, reading or spelling, and the chances are that if you have paid money for an intervention you are more likely to think it has been successful, than if you don’t pay money for it. It's human nature - you are not really thinking about it objectively. Whether you have paid for something or not it is still important to be objective in deciding whether it has worked for you.

There are many tools that have already been evaluated and have lots of evidence behind them, but not all children, schools, classrooms, teachers are the same. What works in one school won’t necessarily work in another. It’s really important to evaluate each new initiative so that you know it is working for you and your pupils. There are more and more good resources to support teachers in designing their own evaluation.

Challenges to good evaluation

There are many benefits of evaluating impact for teachers, learners and schools. Benefits include gaining a deeper understanding of what works for your learners in your context, developing expertise in collecting evidence, continual development and flexibility of your practice and a greater sense of empowerment.

However, effective evaluation is not without its challenges.

There is a knowledge challenge. There are few instances (if any) of research methods being taught in initial teacher training courses, so how would you know where to start? What methods do you use? What kinds of data do you gather? How will your data be recorded?

Even if you do know what a good evaluation looks like, the gold standard of evaluations is large scale studies and Randomised Control Trials (RCT). However, this can be very difficult to navigate in the classroom and there are lots of moral and ethical debates surrounding it. For example, as a class teacher, you might not have the luxury of having a control group as it’s difficult to say that you will involve half the class in an intervention and the other half won’t get it.

That doesn’t mean it can’t be done, it can. But there are many challenges that need to be overcome.

How can schools evaluate impact?

It’s important to decide what you want to evaluate the impact of and to make the evaluation specific, focussed and achievable. Do you want to measure progress in relation to specific learning goals? Do you want to understand how your students compare to similar students? Do you want to evaluate the impact of a new resource?

1. Use a baseline assessment. Baseline data is important as it makes sure you have a clear starting point. You might want to use end of topic test scores, previous exam results, end of term or semester grades.
   
   Even better is looking at external, objective data. A standardised baseline assessment can be quickly and easily administered, and you get a nationally representative comparison as well. For example, do the standardised assessment in one year, put your intervention in place and repeat the assessment in the following year to able to see how much your students have gone on to improve over that timeframe and compare that amount of progress against a larger population, including similar schools in similar situations.
   
   
2. Look at previously published evidence. There is a wide range of trusted educational research available.
   
   Take a look at the EEF’s Teaching and Learning Toolkit which provides a summary of the research evidence and summarises the important considerations.
   
   
3. Gather different forms of evidence from different sources. Standardised assessment scores, teacher observation, and external examination results can all be triangulated to give you a more thorough evaluation.
   
   
4. Remember that it is often much easier to say what doesn’t work than what does work. Evaluating impact can help prevent continual squandering of valuable time and money on strategies and methods that simply don’t give the impact you want.
   
   It’s just as important for an evaluation to guide decisions about what not to do.
   
   
5. The Cambridge Assessment International Education’s Teaching and Learning Team have put together a really useful Getting Started with Evaluating Impact document which clearly presents considerations around evaluation as well as practical 'how-to' and 'next steps' guidance.

More information:

Find out how CEM’s baseline assessments can help improve your students’ outcomes.