How do OfSTED determine which schools to inspect?
On Wednesday 11th April, I attended an NAHT meeting, a new commission on accountability, spanning every phase and sector of education. Over the next few months it will canvass the views of some of the foremost thinkers in this area of education policy with the aim to have interim findings before the summer term and to publish our full report in September 2018. This post captures a presentation delivered by an OfSTED representative and not the meeting itself.
Data Drives Inspection
When will [XYZ school] be inspected? This is a question 21,000 headteachers ask themselves several times throughout their headship – some schools will be due for inspection every 2-3 years, whilst other ‘Good’ schools (today) are exceeding a decade since their last inspection! On social media, many teachers, educators and academics have been questioning ‘how OfSTED select schools for inspection?’ with the crux of the debate simply resting on:
- A school is due for inspection (and at no time has there been any risk of decline presented by outcomes – or safeguarding concerns)
- A school’s data has declined or is at risk of decline (and triggers an inspection outside of the typical cycle of inspection).
Teachers have been asking OfSTED the following questions for many years: Does performance data determine when a school will be inspected? If teaching and learning is ‘more than just data’, why do school inspectors rely heavily on the numbers?
At the meeting itself, an OfSTED representative presented insights and accountability methodology. Although I welcomed the transparency and privilege to be part of this commission, with the remit to propose an ‘alternative future to school accountability’, each person within the room is aware that there will need to be some sort of ‘trade-off’ for all stakeholders e.g. parents, OfSTED, DfE, schools and communities. With this in mind, our difficult task is to propose something helpful for everyone, whilst also ensuring school accountability, supporting teachers and headteachers to reduce workload and improve recruitment and retention to ensure pupils and families receive the best education possible.
OfSTED’s Shiny New Toy
At some point early on in the meeting, my heart sank. This was the moment when ‘Machine Learning‘ was presented to the room; OfSTED’s new fancy methodology which sets out the risk assessment process that OfSTED uses for primary and secondary maintained schools and academies that are judged to be good and outstanding.
It’s essentially an artificial intelligent tool using data to preempt when a school should be inspected, thus strengthening OfSTED’s position that data indeed, does determine when a school will be inspected and thus, reduces even further, what the profession wants – less emphasis on data and a broader coverage of all the wonderful work that schools do.
I am led to believe that the system has been designed in collaboration with statisticians at the Office for National Statistics and that is has a 63% reliability i.e. it can predict that a set number of schools ‘at risk of decline’ do actually decline based on actual overall outcome found after inspection.
Data analysts who compiled the report found that 65 per cent of ‘requires improvement’ and ‘inadequate’ schools were within the 10 per cent of schools identified as highest risk by the model they built. (The Telegraph, December 2017)
As UCL quotes in this blog, Ofsted’s head of risk assessment (Paul Moore) says: ‘For a number of years Ofsted have risk assessed maintained schools and academies. This risk assessment is used to help put inspection resource into schools where it is most needed.”
How Does Machine Learning Work?
It would appear that Machine Learning does the following:
- Uses prior data to build its machine learning database – including below floor standards which are not law.
- Input various triggers to identify schools at risk
- Output a list of schools ‘at risk of decline’ to inform what schools need to be inspected
- This information is kept separate to the inspection team, yet informs OfSTED HQ which schools should be visited.
- The OfSTED inspection team arrive to the school with no (apparent) bias to make a conclusion about a school’s performance.
Essentially, OfSTED do not have the physical capacity or funding to inspect every school, so they must to be selective and inspect a) schools on a cycle b) schools at risk of decline c) extreme circumstances.
Therefore, Machine Learning raises a question I have posted before – why choose to work in a disadvantaged school if a) it puts you and your colleagues under ‘more risk’ because the data doesn’t work in your favour or b) work in a challenging school where the current system does not inspect schools working in different contexts e.g. families of schools.
Just look at this one example. This school is deemed ‘average‘ by the Department for Education’s Progress 8 measure, yet when inspected by OfSTED, it was placed into Special Measures and closed down by the DfE after a directive to join a Multi Academy Trust under a new URN. What hope is there for schools when others are rated average by the same measure, yet are given an ‘Good’ judgement by OfSTED. Does this lead to increased bias? Fairness? Second guessing what may happen to your school? I assume it does strengthens OfSTED’s claim that inspections are more than just about data, but contradicts the points made here, that data actually does drives the start and end points to the inspection process.
Not only is this promoting OfSTED’s unreliability, but it is also divisive and damaging – and there are hundreds of examples.
What Does Research Suggest?
There is a poll conducted by Paul Garvey in which 1,000+ people responded to this statement, stating what OfSTED does to schools is unfair:
“32% of IDACI ranked 5 secondary schools are grade 4 or 3, compared to only 9% of IDACI ranked 1 schools. Ofsted continue to say that all should be inspected by exactly the same criteria, no matter what their level of disadvantage.”
There has been no public response.
Research quoted in UCLs blog suggests “automating risk assessments seems an intelligent approach to using scarce inspection resources more efficiently, but the recent critique begs the question of whether this combined use of data and human judgement is an actual example of ‘intelligent accountability’?” Crooks (2006) provides an extensive description of ‘intelligent accountability’ in saying that it is a system which:
- preserves and enhances trust among key participants
- involves participants in the process
- offers participants a strong sense of professional responsibility and initiative
- encourages deep, worthwhile responses
- provides well-founded and effective feedback to support good decision-making and
- leaves the majority of educators more enthusiastic and motivated in their work.
The most critical question OfSTED must answer is this:
- Do OfSTED want to make judgements about schools working in different contexts, or not?
- If Machine Learning will use data from the past to predict when a school will be inspected, do OfSTED recognise that past data – whether accurate or not – could further exacerbate social segregation that we all hope to eradicate from our society?
A spokesperson for NAHT said: “Leaders and teachers need absolute confidence that the inspection system will treat teachers and leaders fairly.” OfSTED are clear about their shiny new toy; Machine Learning algorithms is only used as stage 1 of the risk assessment process and Senior Her Majesty’s Inspector (SHMI) reviews follow on from this and in no way do the algorithm results impact on inspection judgements. Well, we’ll just have to take their word for it!
For me, I don’t think Machine Learning is the answer to fairness or improved reliability. Let me be clear. I want school accountability. We need data to determine standards, but the data must be reliable and fair for everyone. We must go back to the drawing board and ask ‘What is OfSTED for?’ and ‘What do we want OfSTED to report?’
I have another 21 questions I’d like OfSTED to answer if you would like to read more.