Can Machine Learning Predict A School Inspection?


Reading time: 5
Machine Learning

@TeacherToolkit

Ross Morrison McGill founded @TeacherToolkit in 2010, and today, he is one of the 'most followed educators'on social media in the world. In 2015, he was nominated as one of the '500 Most Influential People in Britain' by The Sunday Times as a result of...
Read more about @TeacherToolkit

How do OfSTED determine which schools to inspect? 

On Wednesday 11th April, I attended an NAHT meeting, a new commission on accountability, spanning every phase and sector of education. Over the next few months it will canvass the views of some of the foremost thinkers in this area of education policy with the aim to have interim findings before the summer term and to publish our full report in September 2018. This post captures a presentation delivered by an OfSTED representative and not the meeting itself.

Data Drives Inspection

When will [XYZ school] be inspected? This is a question 21,000 headteachers ask themselves several times throughout their headship – some schools will be due for inspection every 2-3 years, whilst other ‘Good’ schools (today) are exceeding a decade since their last inspection! On social media, many teachers, educators and academics have been questioning ‘how OfSTED select schools for inspection?’ with the crux of the debate simply resting on:

  • A school is due for inspection (and at no time has there been any risk of decline presented by outcomes – or safeguarding concerns)
  • A school’s data has declined or is at risk of decline (and triggers an inspection outside of the typical cycle of inspection).

Teachers have been asking OfSTED the following questions for many years: Does performance data determine when a school will be inspected? If teaching and learning is ‘more than just data’, why do school inspectors rely heavily on the numbers?

At the meeting itself, an OfSTED representative presented insights and accountability methodology. Although I welcomed the transparency and privilege to be part of this commission, with the remit to propose an ‘alternative future to school accountability’, each person within the room is aware that there will need to be some sort of ‘trade-off’ for all stakeholders e.g. parents, OfSTED, DfE, schools and communities. With this in mind, our difficult task is to propose something helpful for everyone, whilst also ensuring school accountability, supporting teachers and headteachers to reduce workload and improve recruitment and retention to ensure pupils and families receive the best education possible.

OfSTED’s Shiny New Toy

At some point early on in the meeting, my heart sank. This was the moment when ‘Machine Learning‘ was presented to the room; OfSTED’s new fancy methodology which sets out the risk assessment process that OfSTED uses for primary and secondary maintained schools and academies that are judged to be good and outstanding.

It’s essentially an artificial intelligent tool using data to preempt when a school should be inspected, thus strengthening OfSTED’s position that data indeed, does determine when a school will be inspected and thus, reduces even further, what the profession wants – less emphasis on data and a broader coverage of all the wonderful work that schools do.

I am led to believe that the system has been designed in collaboration with statisticians at the Office for National Statistics and that is has a 63% reliability i.e. it can predict that a set number of schools ‘at risk of decline’ do actually decline based on actual overall outcome found after inspection.

Data analysts who compiled the report found that 65 per cent of ‘requires improvement’ and ‘inadequate’ schools were within the 10 per cent of schools identified as highest risk by the model they built. (The Telegraph, December 2017)

As UCL quotes in this blog, Ofsted’s head of risk assessment (Paul Moore) says: ‘For a number of years Ofsted have risk assessed maintained schools and academies. This risk assessment is used to help put inspection resource into schools where it is most needed.”

How Does Machine Learning Work?

It would appear that Machine Learning does the following:

  1. Uses prior data to build its machine learning database – including below floor standards which are not law.
  2. Input various triggers to identify schools at risk
  3. Output a list of schools ‘at risk of decline’ to inform what schools need to be inspected
  4. This information is kept separate to the inspection team, yet informs OfSTED HQ which schools should be visited.
  5. The OfSTED inspection team arrive to the school with no (apparent) bias to make a conclusion about a school’s performance.

Essentially, OfSTED do not have the physical capacity or funding to inspect every school, so they must to be selective and inspect a) schools on a cycle b) schools at risk of decline c) extreme circumstances.

Promoting Bias

Therefore, Machine Learning raises a question I have posted before – why choose to work in a disadvantaged school if a) it puts you and your colleagues under ‘more risk’ because the data doesn’t work in your favour or b) work in a challenging school where the current system does not inspect schools working in different contexts e.g. families of schools.

Just look at this one example. This school is deemed ‘average‘ by the Department for Education’s Progress 8 measure, yet when inspected by OfSTED, it was placed into Special Measures and closed down by the DfE after a directive to join a Multi Academy Trust under a new URN. What hope is there for schools when others are rated average by the same measure, yet are given an ‘Good’ judgement by OfSTED. Does this lead to increased bias? Fairness? Second guessing what may happen to your school? I assume it does strengthens OfSTED’s claim that inspections are more than just about data, but contradicts the points made here, that data actually does drives the start and end points to the inspection process.

Not only is this promoting OfSTED’s unreliability, but it is also divisive and damaging – and there are hundreds of examples.

What Does Research Suggest?

There is a poll conducted by Paul Garvey in which 1,000+ people responded to this statement, stating what OfSTED does to schools is unfair:

“32% of IDACI ranked 5 secondary schools are grade 4 or 3, compared to only 9% of IDACI ranked 1 schools. Ofsted continue to say that all should be inspected by exactly the same criteria, no matter what their level of disadvantage.”

There has been no public response.

Research quoted in UCLs blog suggests “automating risk assessments seems an intelligent approach to using scarce inspection resources more efficiently, but the recent critique begs the question of whether this combined use of data and human judgement is an actual example of ‘intelligent accountability’?” Crooks (2006) provides an extensive description of ‘intelligent accountability’ in saying that it is a system which:

  • preserves and enhances trust among key participants
  • involves participants in the process
  • offers participants a strong sense of professional responsibility and initiative
  • encourages deep, worthwhile responses
  • provides well-founded and effective feedback to support good decision-making and
  • leaves the majority of educators more enthusiastic and motivated in their work.

Key Questions

The most critical question OfSTED must answer is this:

  1. Do OfSTED want to make judgements about schools working in different contexts, or not?
  2. If Machine Learning will use data from the past to predict when a school will be inspected, do OfSTED recognise that past data – whether accurate or not – could further exacerbate social segregation that we all hope to eradicate from our society?

A spokesperson for NAHT said: “Leaders and teachers need absolute confidence that the inspection system will treat teachers and leaders fairly.” OfSTED are clear about their shiny new toy; Machine Learning algorithms is only used as stage 1 of the risk assessment process and Senior Her Majesty’s Inspector (SHMI) reviews follow on from this and in no way do the algorithm results impact on inspection judgements. Well, we’ll just have to take their word for it!

For me, I don’t think Machine Learning is the answer to fairness or improved reliability. Let me be clear. I want school accountability. We need data to determine standards, but the data must be reliable and fair for everyone. We must go back to the drawing board and ask ‘What is OfSTED for?’ and ‘What do we want OfSTED to report?’

I have another 21 questions I’d like OfSTED to answer if you would like to read more.


18 thoughts on “Can Machine Learning Predict A School Inspection?

  1. I was a head and now provide external evaluation for schools in both highly advantaged and disadvantaged settings and you have nailed the greatest iniquity that our teachers and families face.

    There can be no justice, as we do, in comparing against the same criteria the attainment and progress of students who have been coached intensively and expensively, who visit theatres and museums, live a cosmopolitan life and have every comfort at home, those for whom life is a daily struggle with poverty and multiple disadvantages, whose parents of course want the best for their children, but simply do not have the luxury of security and physical and cultural capital. This is not making excuses but reality.

    We desperately need great teachers to be happy to work in schools where students face the greatest challenges to learning and deserve to have the very best teaching. Those who teach there do an amazing job, but it is ever harder to recruit them and this compounds the difficulties. Why then does DFE create such mammoth disincentives to teachers to teach there? The solutions are clear: stop publically pillorying teachers and students who are doing their best and praise their efforts instead. Does it take astonishing insight to see that we all thrive on support and encouragement rather than criticism and condemnation and that is the way forward for our schools.?

  2. A comment on Twitter by @HeatherLeatt: “When schools are inspected outside of their time – due to perceived risk identified, inspection teams aren’t told. The implication is that they ‘go in blind’ and therefore unbiased, but they’ll still read all the previous reports and will know what it’s being inspected sooner than expected in the cycle. They’ll look at the results published on website (DfE requires this) and draw their own conclusions about why they’re being sent in early. Thus, they are not being told directly, but it also doesn’t mean they go in without an opinion?”

    Perhaps we should take away some of the details so that bias is removed as much as possible to provide a fairer outcome.

  3. Thanks for the insightful article. Machine learning is going to be part of our teaching future whether we like it or not, so it’s good to know more about it.

  4. Hello Sir, using machine learning into the education system is the greatest idea. Like Ofsted, we all can use in their education systems.

    I believe machine learning could eradicate further exacerbate social segregation from our society.

    Thank you for writing such a great blog. I would like to hear more from you.

  5. Hi, Thanks for this detailed article. Machine learning is the future of online education this technology is adding a quality in elearning which is helping children, office employees, entrepreneurs to gain more knowledge.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.