How can we measure school examination performance ethically?
As with all exam measurements, performance monitoring evolves despite progress being relative to whatever you decide to measure it against.
Due to the nature of changing goalposts, I know as soon as I publish this blog post, much of it will become obsolete. Hopefully, the message will stay the same…
Progress 8 across English schools has already had a fascinating journey in its short history. It measures average pupil progress made between key stage 2 (age 11) and general certificate of secondary education (GCSE) examinations (age 16).
5 areas of concern
In a new (relatively) thought-piece published by BERA, academics discuss if it is still worth calculating progress 8, inspired by a Review and evaluation of secondary school accountability in England: statistical strengths, weaknesses and challenges for progress 8 raised by COVID-19.
If we factor in off-rolling, exclusions, rising mental health issues and increased EBacc entries – still woefully off target though – as with all exam indicators, there will inevitably be some unintended consequences.
The research argues five areas of concern.
- Choice of pupil outcome attainment measure.
- Potential adjustments for prior attainment and background characteristics.
- Decisions around which schools and pupils are excluded from the measure.
- Presentation of Progress 8 to users, statistical model choice, and uncertainty calculation.
- Issues related to the volatility of school performance over time
Amidst all the government and think-tank (wonk) hyperbole, I recently wrote about genuine reform for schools. I highlighted that some policymakers will ignore this research because it suggests that Progress 8 – the metric we have all relied on so favourable for years – is not working.
Put simply, this metric does not tell us anything about what is working in our schools.
Possible alternatives to Progress 8?
Currently, Progress 8 does not sufficiently communicate that the variation in school Progress 8 scores and accounts for only around a tenth of the overall variation in pupil progress across England. Yes, a tenth! Perhaps all our schools are doing a really good job regardless of status, structure or leadership?
The research offers some possible tweaks:
- The attainment measure is heavily weighted (70:30) in favour of the traditional academic subjects!
- Pupil background is not represented/adjusted to provide greater perspective
- Pupil mobility is not taken into account; holding schools to account for those they have taught before ‘on-rolling’ to another school – this is already changing thanks to Ofsted policy
- The metric creates uncertain predictions as to the future performance of schools (for parents in particular)
- That progress 8 accounts for only around a tenth of overall variation in pupil progress (see graphic below).
A former colleague raised a decent suggestion (for Ofsted too):
Publish data in the interim, the median as well as the mean. This would ensure we can redress schools that [are] run in students’ interests, not the adults who run them. Only then will we get a clearer picture of behaviour, which types of policies are working, and in which types of schools.
Until this happens, we can wait for commonsense to prevail and keep continue to measure apples against pears and vice versa…
Read the blog summary or the full paper.
