Inspecting The Curriculum: Deep Dive Unreliability


Reading time: 4
Deep Dive Work Scrutiny

@TeacherToolkit

In 2010, Ross Morrison McGill founded @TeacherToolkit from a simple Twitter account through which he rapidly became the 'most followed teacher on social media in the UK'. In 2015, he was nominated as one of the '500 Most Influential People in Britain' by The Sunday...
Read more about @TeacherToolkit

Can Ofsted inspectors look through pupils’ books with any degree of reliability?

I have already articulated many thoughts on the reliability of work scrutiny and lesson observation through countless blogs on my website, as well as recently on Twitter. This also includes two decades of trial and error working in school leadership.

In this post, I ask some key questions we should all be asking Ofsted inspectors.

I’ve been conducting ‘deep dives‘ in secondary schools my entire career. It took several years of senior leadership to even get close to something I would call ‘reliable’, which involved many months of development, trial and error and triangulation. To resemble any degree of reliability took weeks of conversation and departmental reviews, often lost in one-off observations and isolated book scrutinies. Having a secure overview of what pupils had learned, or what teachers intended on teaching, was certainly not something that could be achieved in one day.

When I put my ‘research methods’ hat on and consider Ofsted’s ‘deep dive graphic’ (which explains the process schools have to go through before inspectors are ‘bringing it together’), I question if anyone can conduct this process with any degree of reliability in 1.5 days of school life – worse, particularly during a school inspection.

Bringing it together

Firstly, any conversation about curriculum, moving away from data and a less ‘high-stakes’ feel about the whole process is of course, welcome. And although shorter and friendly Oftsed reports are already being published, the emerging mood from colleagues working in our primary schools is mixed.  In terms of reducing or increasing workload, the jury is still out. What impact does this new process have on workload? One teacher recently said about the new inspection framework:

“It did increase the workload. Lesson planning didn’t change for me personally but having all relevant documents to hand was time-consuming. For me it was positive, but it is tough when you’re the only subject teacher…”

What I hope to do with from this blog post, is a set of thoughts and questions to support teachers to use when being inspected or developing ‘deep dive’ methods. I hope the questions that follow put yourself in a stronger position when conducting the process within your school, or particularly to challenge orthodoxical methods used when inspectors visit for inspection.

Deep Dives Ofsted Inspecting The Curriculum

Questions of reliability

  1. What is the inspector’s subject specialism?
  2. What happens if the inspector is not a subject expert?
  3. Was there an agreed focus before the inspector/observer visited the classroom?
  4. Was the teacher informed?
  5. What questions are asked of pupils? And are these questions public knowledge?
  6. Are the same questions asked of pupils in the classroom next door?
  7. Is there a conversation with the classroom teacher?
  8. Is that conversation conducted before, during and after the lesson?
  9. Did the inspector visit several subjects/age groups?
  10. What happens if there is insufficient evidence? What constitutes evidence in classrooms that have no books?

The research

Sometimes, we cannot measure what we need, so we invent a proxy, something that’s easier to measure and stands in as an approximation. Ofsted has chosen (Kappa statistics) to use a very insecure data set for deep dive methodology.

You can read Ofsted’s research paper in which they tested 9 (just nine) inspector’s subject knowledge into reliable methods for assessing the quality of education using lesson observation and work scrutiny.

  1. Most HMI (6/9) felt confident in the bands they awarded.
  2. Subject expertise did not affect the reported ease with which HMI applied the rating scale.
  3. Scrutiny across a single subject/department/year group is helpful in securing greater validity/reliability.
  4. Ofsted selected 4 indicators for workbook scrutiny from a range of indicators designed for the inspection process. They are: prior learning, coverage, progress and practice.
  5. It’s very hard for a classroom teacher to assess these indicators in a lesson, even when they are working with the pupils frequently, how does an observer do this so well?

Should inspectors look in pupils’ books?

My original question was ‘Can Ofsted inspectors look through pupils’ book with any degree of reliability?’ Of course, we should all want pupils’ work to be reviewed and standards to be disseminated, and we should also consider how this process may fuel teacher habits, school leaders’ perceptions of ‘what Ofsted want’ as well as teacher workload.

In a period of time where teacher vacancies are increasing as teacher-numbers stagnate, we are each hoping that Ofsted’s new framework will make a difference to the profession. ‘Does Ofsted raises school standards?’ is another question we do not have sufficient research to suggest that it does, and what I hope we can all achieve from the new framework, is that methods used by visitors are fair and reasonable and address some of the national issues we face.

Inspectors must understand what they can and cannot observe from ‘snapshot’ observations. Any observer cannot conduct book looks with any certainty, or, make claims about progression in a classroom – without actually being in the classroom over a period of time – and addressing some of the reliability questions I have posed above. Even when you work within the school itself, I have demonstrated how difficult triangulation is to conduct, reliably.

Communicate the focus!

For me, the methods used, the focus and expertise are essential when gathering any conclusions from the classroom. The degree of precision is most critical and without this foci, making an assumption of what is happening in the lesson is the best we can hope for. It is critical that what we ‘look at’ in lesson observations and deep dives is agreed in advance, communicated, and that the focus is as narrow as possible.

There are better ways to raise standards. I’ve offered my services to Ofsted – to provide perspective as a critical friend – for the greater good. I’m still waiting for the phone to ring.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.