How can we trust research?

By Dr Gary Jones

A major challenge for anyone interested in the use of research within both schools and the wider education system is to try and make some kind of judgment about the trustworthiness and quality of research.

Indeed, this is a real problem, as it is highly unlikely that teachers, heads of department, or school leaders will have the required level of expertise to critically examine the claims that are being made.

So, to help get around this problem of a lack of expertise and increase the chances of being able to judge the usefulness of research, in my new book, Evidence-based School Leadership and Management, I have looked at a helpful framework which provides you with a useful scaffold to help you make efficient and effective use of research evidence in bringing about improvements within your classroom, school or educational setting:

The 6 A’s of the usefulness of research evidence

Professor Steve Higgins, of Durham University, developed the framework the '6 A's of the usefulness of research' by asking a number of questions:

Is the research you are attempting to use:

• Accurate
• Accessible
• Applicable
• Acceptable
• Appropriate
• Actionable

I have subsequently worked with Professor Higgins to develop a more fleshed-out version of the 6 A’s, which draws upon both the TAPUPAS framework developed by Pawson, Boaz, et al. (2003) and Critical Reading and Writing for Postgraduates (Third Edition) by Wallace and Wray (2016) and which is illustrated in the following table.

The 6 A’s framework in use

It needs to be remembered that the ‘6 A's' is just a framework to help you think through the usefulness of a piece of research report for your setting and associated problems of practice. 

With that in mind, the framework should not be used in a ‘tick-box’ manner, answering all the sub-questions. Rather it should be used as an aide-memoire, which is to be dipped into as time, knowledge and understanding allows. That said, we would recommend that you attempt to answer one sub-question within each section.

Amended from Jones (2018)

Some observations about the framework

Observant readers will notice that the original order of the first two A’s has been switched, with accessibility now coming before accuracy. This was done for a very practical reason, you are unlikely to be able to judge the accuracy of the research unless you have first been able to access it.

Furthermore, it needs to be remembered that this is not the ‘only framework in town’ which you can use to appraise the usefulness of research.

Willingham (2012) has come up with the following four-step process:

First, ‘strip it’, get rid of the fluff surrounding the idea and get right to the heart of the claim being made. What specific intervention, strategy or actions is the school leader being asked to adopt and what outcomes, for student learning or achievement, staff well-being or other outcomes are being promised?

Second, ‘trace it’, where did the idea come from? Is the idea supported by a leading educational authority? Unfortunately, in education, this can be a weak indicator of validity and reliability? Do other ‘experts’ support the idea?

Third, ‘analyse it’, what are you – the evidence-based school leader - being asked to believe? What is the evidence to support the claims being made? How does this evidence relate to your experience as a school leader?

Fourth, ‘should I do it?’ Is it something which is already being done? Is it an old idea wrapped in new language and terminology? Has it worked previously in other settings with other students? Has it failed previously in other settings with other students? What are the opportunity costs – the things necessary to give up or forego - in pursuing this intervention or strategy?

And finally

What is hopefully clear from this blog post – no matter your knowledge and skill level - there are ways in which you can make structured judgments about the usefulness of research.

However, what is great about these frameworks is that the more you use them and become familiar with them, the better you will get at using them, which will in turn positively contribute to helping you make efficient and effective use of research evidence in the improvement of schools.

References

Further reading

Higgins, S. (2018) Improving Learning: Meta-analysis of Intervention Research in Education Cambridge: Cambridge University Press