Mind the gap: academic warns attainment gap measurements 'let down' poorer pupils

Depending on the moment, politicians have a habit of saying that whatever policy they are talking about that day is their "top priority".

These statements are usually made without any apparent acknowledgement of the inherent contradiction. You can only, of course, have one top priority.

So when Nicola Sturgeon says closing the attainment gap between poorer and better off pupils in Scotland's schools is her 'top priority' should we believe her?

Having interviewed the First Minister on this subject several times, my answer to that is an unequivocal 'yes'. And it is the same for the Deputy First Minister, and education secretary, John Swinney.

They are both are the products of the state education system. They feel this personally. Both think the current system does not do enough to help children from poorer backgrounds. Both want change to close that gap.

And you won't find politicians in any of the other main political parties who disagree with their aim. Where it gets difficult are the proposals the Scottish government has put forward to solve this very difficult to solve problem.

There are now several documents outlining the planned changes in detail but a key policy plank is to give headteachers more power over staffing and curriculum in their schools.

The latest document launched by Mr Swinney earlier in the week stated explicitly that international evidence shows that more autonomy for heads leads to improved standards in schools.

Giving greater power to heads and therefore reducing the power of local authorities - who formally run schools - has been controversial in itself.

And there are other areas where the government has come in for criticism, including the plan to set up 'regional collaboratives' to bring councils together to jointly work on attainment.

It would be easy to go into even more detail of the Scottish government's plans and the arguments for and against the various changes. Previous blogs have touched on these, future ones will too, no doubt.

However, there is a crucial question which has to be answered about these reforms. How will we be able to tell whether they have worked?

How will we be able to tell in, say, five years time, if children from the least well off households are doing better?

As things stand, the basis for measuring poverty in Scotland is what is known as the Scottish Index of Multiple-Deprivation, SIMD to its friends.

This looks at relatively small neighbourhoods across Scotland and measures the deprivation levels within them.

As things stand the Scottish government will track children from the most deprived areas to see if their reforms improve their education - in terms of exams, other qualifications, entrance to University and the like.

But in the view of one academic this plan is flawed. Deeply flawed. Lindsay Paterson, professor of education policy at Edinburgh University, goes so far as to say these measures are "completely useless" for measuring poverty in individual families.

Which means, in Prof Paterson's view, that the government will be unable to tell whether if they see a rise in standards from the poorer neighbourhoods, that is because kids from poorer backgrounds are performing better, or the better off kids from that poorer area are benefitting.

Similarly, he argues that the measurements will miss measuring what happens to the children of poorer families who happen to live in what is overall a more prosperous neighbourhood.

I interviewed Prof Paterson for Representing Border this week. This is what he told me: "The major problem is that if you focus on neighbourhoods only, which is what the Scottish government does, you miss at least a third of children living in actual poverty. These are children living in poverty who don't live in deprived neighbourhoods.

"That's an illusion, it's an illusion that you think all the poverty is concentrated in areas that are dominated by poverty and yet there are important patches of poverty all over society which currently policy just completely misses."

I asked him why this matters? His reply: "It's really important because in the long run we are measuring the experiences of individual children, the difficulties they face, not of whole neighbourhoods. It's children who learn, not whole neighbourhoods.

"Unless we get our measures addressed to the people who actually do the learning, which is the children, then we end up letting down all these children who are not covered by the measures."

Now, the Scottish government would say SIMD is the standard way of measuring deprivation in Scotland?

Prof Paterson agreed, up to a point. "It is true for measuring neighbourhood poverty, but it is completely useless for measuring the poverty of individual families. Let me put it like this. Let's suppose we adopt these measures over the next ten years as a way of tracking whether inequality of attainment has reduced. We might as well be randomly doing it.

"We will see a reduction in poverty, not because poverty has actually changed, but because the measures are so poor, so inadequate that they can't give us a true picture of what is happening."

So does that mean if the Scottish government make a claim in five years time that the attainment gap has been narrowed, it will have no credibility in your view?

Prof Paterson was blunt: "None whatsoever." And what should we do about that?

Prof Paterson added: "What we properly need is a good quality survey of the kind that is done in many countries, including England and Wales, which tracks individual children over time, paying attention to their family circumstances, the support they get from their parents, the fact that some parents are too poor to provide support. That's the only credible social scientific way of measuring the effects of poverty."

I then put these point to Mr Swinney. The Deputy First Minister told me: "I want to make sure we have in place the most effective mechanisms possible to judge the progress that we make in closing the attainment gap."

I put it to Mr Swinney that Prof Paterson says that won't achieve what the government wanted to, because you would be measuring areas and there might be well-off kids in worse-off areas?

Mr Swinney told me: "We're undertaking a consultation on these very questions to make sure that we get that absolutely correct. I want to have in place a framework which will enable us to judge whether we are making progress on closing the poverty-related attainment gap."

Not for some statistical exercise, because I want to be able to say to people in this country that we have interrupted inter-generational poverty in our country, that we have transformed the life-chances of our people and I want people to feel that in their communities."

As this is such an important issue, I asked the Scottish government if it had anything more to say in response to Prof Paterson's claim.

A Scottish Government spokesperson said: “We recognise the challenges involved in measuring deprivation, which is precisely why we are consulting on proposals for a robust package of indicators to determine progress on closing the poverty-related attainment gap.”

I understand that ministers want to stress that their consultation paper 'makes clear', SIMD is a measure of area-based deprivation and - while there is an undeniable relationship between educational attainment and the nature of the area in which young people live - it does not represent all deprived young people in Scotland.

The Scottish government says it has also made clear it does not not think it is sensible or realistic to assess the performance of our education system using a single measure.

Which would suggest that ministers are aware of the problem. The question now is what they do about it.

You can see my Representing Border interview with Prof Paterson, and John Swinney in full here:

http://www.itv.com/news/border/update/2017-11-07/watch-tuesdays-representing-border-online/