The False Proxy Trap

How do we stop OfSTED falling into a ‘false proxy trap’?

Sometimes, we can’t measure what we need, so we invent a proxy, something that’s easier to measure and stands in as an approximation. (Seth Godin)

After reading Amanda Spielman’s speech at ResearchED, I’ve had to ‘pause the blog’ button to write this!

OfSTED is considering a new approach to book scrutiny, in which off-site inspectors will look at scanned versions of pupils’ books. (Schools Week)

Now, I appreciate I was not in the room and there will be much more to understand from this announcement, but is this an indication of a broken system or an attempt to up-skill the inspectorate? With too much in-house variation and with an inspectorate that is currently not fit for purpose, I hope that this decision is a move to develop inspectors for the better.

But it has made me think. At what cost will this decision impact on schools?

For those who do work in schools, they will truly understand the dangers of conducting a book scrutiny. Imagine looking at books without the teacher present? It happens all the time! Or assuming ‘little Ahmed’s’ work is incomplete and of poor quality, because his prior data says so, but failing to discuss with him or the teacher that he’s just missed the last 2 months of school because his mother has passed away.

Looking at students’ books without context is dangerous. Doing this remotely could be catastrophic! I hope this will not form part of inspections, but merely for training inspectors without increasing school leaders’ workload.

Bias and Mistakes

For the past 17 years of looking in exercise books as a senior and middle leader, I’ve made hundreds of mistakes. Here are some examples:

• Walking quietly into the room, I choose not to disturb the lesson and find an empty chair. I end up sitting next to one or two students by chance and flick through their books and ask a few (random) planned questions.
• During an INSET day, I find an empty classroom and grab a set of class books, deciding to flick through them all and tick-box what I see / don’t see. There’s no teacher or student in sight.
• I call a ‘book look’ for the next middle leadership team meeting, fully expecting each person to bring – not their own – but a sample of their best books from their departments.
• “Anyone else want to conduct a book look with me?” Silence …
• On duty around the school, I choose a department and decide to ‘pop in’ to each class, making a concerted effort to pre-select student books based on their data – ensuring I have a discussion with the student and teacher where possible.
• On a planned book look schedule, a huge stack of books arrive for sampling. The workload is too high, so a quick flick through will be fine.

If I have made these mistakes on-site, what assumptions or errors could I make remotely?

Over the past 10 years, I’ve tried all sorts of methodologies – refining and tinkering with forms over and over again – hoping to be as fair as possible for those students and teachers who will be assessed. I even posted this video inside this post: Reflections on Whole-School Marking, causing an outrage online by a minority of bloggers. This was clearly because ‘those not in the process’ and not in the school were fearful of seeing endless piles of books stacked up in one room. The sampling was a powerful CPD for the leadership team within the school – something many are reluctant to get involved in – to reduce in-house variation, but it was at the expense of sharing our work online and facing much criticism.

In the past 3 years, I’ve heard some terrible rumours of schools now grading teachers for their marking, replacing lesson observations with one for the other. Looking in student books is not something to be taken lightly. It can be conducted with weak reliability and worse, lead to teacher-capability.

Remote Book Looks

The thought of OfSTED inspectors logging into an online portal OfSTED, god forbid with inspectors demanding various documents on the day of inspection, even if it is a work of fiction, I can see thousands of school leaders / heads of department frantically, running around school, grabbing students’ books to be scanned and uploaded during the inspection process. I hope this is not where this announcement is heading, rather than as a tool to train inspectors.

Spielman says:

“There are things we can do to isolate the book scrutiny process – both to test whether inspectors’ conclusions are consistent with each others’, and whether it gives a valid picture of teaching quality and pupils’ progress.”

So, it appears we are moving to a system where we are testing the testers.

Why? Well, rightly, we need to ensure that school inspectors are top of their game. That they know what to look for and that there is greater consistency between inspection teams. But more importantly, many of those who inspect rarely teach in the classroom, so although it irks me somewhat to admit this may be a good thing for inspectorate professional development, I cannot help but think, this is yet another indication of a broken process. Something that should be left to schools and school leaders to do.

There’s rather good scanning quality now, so the views of other inspectors, remote from the live inspection, could be drawn in to test out the judgments of inspectors on site,” Spielman said. “That could be used both in the context of a particular inspection, for training and for research processes, and to test the overall reliability and validity. We might get several things from the same bit of scrutiny … to evaluate curriculum quality … [to] make sure we have the kind of curriculum that we want across the full range of subjects”.

Some of the most powerful professional development I have been involved in, was when our teachers gathered to discuss student exercise books as a team. Within the hour, gone were any misconceptions about ‘what not to do’ and how ‘specific and focused’ questions were critical.

Warning

How can we improve the process for schools, teachers and observers when looking at students’ books? Well, it’s good to see that OfSTED want to improve the quality of looking at students’ books, but this is not without warning.

Pam Fearnley (Pupils First Ltd.) calls this “fishing without the bait” which was powerful professional development for me when I first heard it and was shown ‘how to look at student books’ more reliably. Whether this same methodology can be used remotely, via a scanner, and if this will lead to more reliable assessments about the quality of teaching and learning, I suspect not.

This is another small step in the direction of OfSTED taking teaching and learning outside of the inspection process. Remove graded lessons? Yes, we’ll do that for you. Learning walks? Yes, just a sample and we’ll triangulate this with your school’s data before we arrive. What about book looks? Well, we’ll do this remotely too, asking you to upload a number of scans to a portal for ‘other inspectors’ to verify the quality of your curriculum, teaching and learning, data and marking.

Oh, I see. So, there’s not much else you’ll do on the day at all, is there?

Yes that’s right. We’ll just talk to students and a few members of staff. We don’t need to give teaching and learning a separate judgement anymore, because we can see over time, long before we arrive, what’s going on and spend less time in classrooms trying to marry up all of the evidence.

Please, let’s not forget, teachers and students work tirelessly – outside of books – providing verbal feedback, working on performances, sport and countless other pieces of work that is not captured in exercise books. Between the front and back cover, learning captured on to lined pages only offer a glimpse into the complicated life of learning and school life. Let’s not get fixated with book scanning to determine the quality of what’s going on.

If we’re going to inspect schools, let’s get inside our classrooms and talk to the students. Otherwise, what’s the point?

N.b. I wonder if OfSTED’s push on curriculum has anything to do with a struggling English Baccalaureate that is proving difficult to implement.