Review: “Improving Education: A triumph of hope over experience”

calvin 1

Posted by Stephen. I recently came across a pdf of Professor Robert Coe’s inaugural lecture as Professor of Education and Director of the Centre for evaluation and Monitoring (CEM) at Durham Univeristy.  Delivered in June 2013 it is nearly two years old now but in light of Anne Glennie’s recent post about education standards, it’s worth a re-visit.

This is the abstract for the piece:

“Despite the apparently plausible and widespread belief to the contrary, the evidence that levels of attainment in schools in England have systematically improved over the last 30 years is unconvincing. Much of what is claimed as school improvement is illusory, and many of the most commonly advocated strategies for improvement are not robustly proven to work. Even the claims of school effectiveness research – that we can identify good schools and teachers, and the practices that make them good – seem not to stand up to critical scrutiny. Recent growth of interest in evidence-based practice and policy appears to offer a way forward; but the evidence from past attempts to implement evidence-based approaches is rather disappointing. Overall, an honest and critical appraisal of our experience of trying to improve education is that, despite the best intentions and huge investment, we have failed – so far – to achieve it.
Nevertheless, we have reason to be hopeful, provided we are willing to learn from our experience. Specifically, I will argue that we need to do four things: to be clear what kinds of learning we value; to evaluate, and measure properly, teaching quality; to invest in high-quality professional development; and to evaluate robustly the impact of changes we make”.

As a former teacher himself, Professor Coe is not implying criticism of teachers, he realises how hard they work to improve things in schools.  He argues that there is insufficient data to robustly conclude that standards have risen despite all the efforts to drive the system harder.  Given what data is available he draws a bleak conclusions that our good intentions have failed to produce results. He lists some of the weaknesses of school improvement research methodologies and has a figure listing 7 things you could do to ‘mistake school improvement’ and make it look as if your improvement project has worked.

One might think that the content is unnecessarily cynical, but I believe that it presents a real wake up call to the mass of flimsy research and poor attribution of causation and effectiveness that is out there.  For example, value added is not necessarily the same as effectiveness, and the usual suspects in lists of characteristics of ‘good’ schools are often poorly defined and the correlations ‘generally pretty low’.

Many strategies that schools claim to be doing that ‘work’ (for example assessment for learning) are prone to a lack of proper contextual evaluation and subject to problematic intervention. In other words, there is a poor understanding of how supporting factors in one school may be different in another; also, “we do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable” (p.xi).

Despite this bleak assessment of the state of play, Robert Coe argues there is hope and suggests four strategies:

  1. Think hard about learning
  2. Invest in effective professional development
  3. Evaluate teacher quality
  4. Evaluate Impact of changes

Under each heading a little more detail is provided. One interesting and important one being a list of ‘proxies for learning’: those easily visible things we observe that lead us to conclude learning might be taking place when in fact it is not (e.g busy classroom, students seem to be engaged).  Teaching and learning may be distinct activities and not necessarily related.

Coe suggest this link to use for teachers wishing to conduct small scale evaluation projects: Coe, R., Kime, S., Nevill, C. and Coleman, R. (2013) ‘The DIY Evaluation Guide’. London: Education Endowment Foundation. [Available at evaluation-guide]

This is Coe’s conclusion: ” I have made four suggestions. The first three are just suggestions; I’ll be happy to concede them if others have better suggestions. But the fourth is non-negotiable: the fourth is the one that distinguishes what I am recommending from what we have done before. Education has existed in a pre-scientific world, where good measurement of anything important is rare and evaluation is done badly or not at all. It is time we established a more scientific approach”.

A strong argument, but I am still left wondering whether a ‘scientific approach’ is either possible or productive, or whether we’re just chasing another myth? An open question – perhaps someone would care to make for an argument one way or another to post on this blog.

The pdf of the speech can be downloaded here: ImprovingEducation2013