Can Machine Learning Predict A School Inspection?

Reading Time: 5 minutes

How do OfSTED determine which schools to inspect? 

On Wednesday 11th April, I attended an NAHT meeting, a new commission on accountability, spanning every phase and sector of education. Over the next few months it will canvass the views of some of the foremost thinkers in this area of education policy with the aim to have interim findings before the summer term and to publish our full report in September 2018. This post captures a presentation delivered by an OfSTED representative and not the meeting itself.

Data Drives Inspection

When will [XYZ school] be inspected? This is a question 21,000 headteachers ask themselves several times throughout their headship – some schools will be due for inspection every 2-3 years, whilst other ‘Good’ schools (today) are exceeding a decade since their last inspection! On social media, many teachers, educators and academics have been questioning ‘how OfSTED select schools for inspection?’ with the crux of the debate simply resting on:

  • A school is due for an inspection (and at no time has there been any risk of decline presented by outcomes – or safeguarding concerns)
  • A school’s data has declined or is at risk of decline (and triggers an inspection outside of the typical cycle of inspection).

Teachers have been asking OfSTED the following questions for many years: Does performance data determine when a school will be inspected? If teaching and learning is ‘more than just data’, why do school inspectors rely heavily on the numbers?

At the meeting itself, an OfSTED representative presented insights and accountability methodology. Although I welcomed the transparency and privilege to be part of this commission, with the remit to propose an ‘alternative future to school accountability’, each person within the room is aware that there will need to be some sort of ‘trade off’ for all stakeholders e.g. parents, OfSTED, DfE, schools and communities. With this in mind, our difficult task is to propose something helpful for everyone, whilst also ensuring school accountability, supporting teachers and headteachers to reduce workload and improve recruitment and retention to ensure pupils and families receive the best education possible.

OfSTED’s Shiny New Toy

At some point early on in the meeting, my heart sank. This was the moment when ‘Machine Learning‘ was presented to the room; OfSTED’s new fancy methodology which sets out the risk assessment process that OfSTED uses for primary and secondary maintained schools and academies that are judged to be good and outstanding.

It’s essentially an artificial intelligent tool using data to preempt when a school should be inspected, thus strengthening OfSTED’s position that data indeed, does determine when a school will be inspected and thus, reduces even further, what the profession wants – less emphasis on data and a broader coverage of all the wonderful work that schools do.

I am led to believe that the system has been designed in collaboration with statisticians at the Office for National Statistics and that is has a 63% reliability i.e. it can predict that a set number of schools ‘at risk of decline’ do actually decline based on actual overall outcome found after inspection.

Data analysts who compiled the report found that 65 per cent of ‘requires improvement’ and ‘inadequate’ schools were within the 10 per cent of schools identified as highest risk by the model they built. (The Telegraph, December 2017)

As UCL quotes in this blog, Ofsted’s head of risk assessment (Paul Moore) says: ‘For a number of years Ofsted have risk assessed maintained schools and academies. This risk assessment is used to help put inspection resource into schools where it is most needed.”

How Does Machine Learning Work?

It would appear that Machine Learning does the following:

  1. Uses prior data to build its machine learning database – including below floor standards which are not law.
  2. Input various triggers to identify schools at risk
  3. Output a list of schools ‘at risk of decline’ to inform what schools need to be inspected
  4. This information is kept separate to the inspection team, yet informs OfSTED HQ which schools should be visited.
  5. The OfSTED inspection team arrive to the school with no (apparent) bias to make a conclusion about a school’s performance.

Essentially, OfSTED do not have the physical capacity or funding to inspect every school, so they must to be selective and inspect a) schools on a cycle b) schools at risk of decline c) extreme circumstances.

Promoting Bias

Therefore, Machine Learning raises a question I have posed before – why choose to work in a disadvantage school if a) it puts you and your colleagues under ‘more risk’ because the data doesn’t work in your favour or b) work in a challenging school where the current system does not inspect schools working in different contexts e.g. families of schools.

Just look at this one example. This school is deemed ‘average‘ by the Department for Education’s Progress 8 measure, yet when inspected by OfSTED, it was placed into Special Measures and closed down by the DfE after a directive to join a Multi Academy Trust under a new URN. What hope is there for schools when others are rated average by the same measure, yet are given an ‘Good’ judgement by OfSTED. Does this lead to increased bias? Fairness? Second guessing what may happen to your school? I assume it does strengthens OfSTED’s claim that inspections are more than just about data, but contradicts the points made here, that data actually does drives the start and end points to the inspection process.

Not only is this promoting OfSTED’s unreliability, it is divisive and damaging – and there are hundreds of examples.

What Does Research Suggest?

There is a poll conducted by Paul Garvey in which 1,000+ people responded to this statement, stating what OfSTED does to schools is unfair:

“32% of IDACI ranked 5 secondary schools are grade 4 or 3, compared to only 9% of IDACI ranked 1 schools. Ofsted continue to say that all should be inspected by exactly the same criteria, no matter what their level of disadvantage.”

There has been no public response.

Research quoted in UCLs blog suggests “automating risk assessments seems an intelligent approach to using scarce inspection resources more efficiently, but the recent critique begs the question of whether this combined use of data and human judgement is an actual example of ‘intelligent accountability’?” Crooks (2006) provides an extensive description of ‘intelligent accountability’ in saying that it is a system which:

  • preserves and enhances trust among key participants
  • involves participants in the process
  • offers participants a strong sense of professional responsibility and initiative
  • encourages deep, worthwhile responses
  • provides well-founded and effective feedback to support good decision-making and
  • leaves the majority of educators more enthusiastic and motivated in their work.

Key Questions

The most critical question OfSTED must answer is this:

  1. Do OfSTED want to make judgements about schools working in different contexts, or not?
  2. If Machine Learning will use data from the past to predict when a school will be inspected, do OfSTED recognise that past data – whether accurate or not – could further exacerbate social segregation that we all hope to eradicate from our society?

A spokesperson for NAHT said: “Leaders and teachers need absolute confidence that the inspection system will treat teachers and leaders fairly.” OfSTED are clear about their shiny new toy; Machine Learning algorithms is only used as stage 1 of the risk assessment process and Senior Her Majesty’s Inspector (SHMI) reviews follow on from this and in no way do the algorithm results impact on inspection judgements. Well, we’ll just have to take their word for it!

For me, I don’t think Machine Learning is the answer to fairness or improved reliability. Let me be clear. I want school accountability. We need data to determine standards, but the data must be reliable and fair for everyone. We must go back to the drawing board and ask ‘What is OfSTED for?’ and ‘What do we want OfSTED to report?’

I have another 21 questions I’d like OfSTED to answer if you would like to read more.

@TeacherToolkit

In 2010, Ross Morrison McGill founded @TeacherToolkit from a simple Twitter account in which he rapidly became the 'most followed teacher on social media in the UK'. In 2015, he was nominated for '500 Most Influential People in Britain' in The Sunday Times as one of the most influential in the field of education - he remains the only classroom teacher to feature to this day ... Sharing online as @TeacherToolkit, he rebuilt this website (c2008) into what you are now reading, as one of the 'most influential blogs on education in the UK', winning the number one spot at the UK Blog Awards (2018). Today, he is currently a PGCE tutor and is researching 'social media and its influence on education policy' for his EdD at Cambridge University. In 1993, he started teaching and is an experienced school leader working in some of the toughest schools in London. He is also a former Teaching Awards winner for 'Teacher of the Year in a Secondary School, London' (2004) and has written several books on teaching (2013-2018). Read more...

8 thoughts on “Can Machine Learning Predict A School Inspection?

  • 12th Apr 2018 at 9:26 pm
    Permalink

    I was a head and now provide external evaluation for schools in both highly advantaged and disadvantaged settings and you have nailed the greatest iniquity that our teachers and families face.

    There can be no justice, as we do, in comparing against the same criteria the attainment and progress of students who have been coached intensively and expensively, who visit theatres and museums, live a cosmopolitan life and have every comfort at home, those for whom life is a daily struggle with poverty and multiple disadvantages, whose parents of course want the best for their children, but simply do not have the luxury of security and physical and cultural capital. This is not making excuses but reality.

    We desperately need great teachers to be happy to work in schools where students face the greatest challenges to learning and deserve to have the very best teaching. Those who teach there do an amazing job, but it is ever harder to recruit them and this compounds the difficulties. Why then does DFE create such mammoth disincentives to teachers to teach there? The solutions are clear: stop publically pillorying teachers and students who are doing their best and praise their efforts instead. Does it take astonishing insight to see that we all thrive on support and encouragement rather than criticism and condemnation and that is the way forward for our schools.?

    Reply
    • 13th Apr 2018 at 5:45 am
      Permalink

      Patricia – you hit the nail on the head with your wise commentary. I will pass this on to the DfE in the best format that I can …

      Reply
  • 13th Apr 2018 at 7:46 am
    Permalink

    A comment on Twitter by @HeatherLeatt: “When schools are inspected outside of their time – due to perceived risk identified, inspection teams aren’t told. The implication is that they ‘go in blind’ and therefore unbiased, but they’ll still read all the previous reports and will know what it’s being inspected sooner than expected in the cycle. They’ll look at the results published on website (DfE requires this) and draw their own conclusions about why they’re being sent in early. Thus, they are not being told directly, but it also doesn’t mean they go in without an opinion?”

    Perhaps we should take away some of the details so that bias is removed as much as possible to provide a fairer outcome.

    Reply
  • 13th Apr 2018 at 7:55 am
    Permalink

    As I have understood what machine learning is but What is OfSTED for??Can you clear this for me??
    Thanks for sharing this useful information with us.

    Reply
  • Pingback:The Life of a Deputy Headteacher: The Finale | TeacherToolkit

  • 5th Dec 2018 at 10:00 am
    Permalink

    Completely understood the machine learning but still little confused with OfSTED.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.