Tuesday 21 January 2014


Taking the PISA

I know that the car boot death mayhem on Eastenders and the laughably blood-rich Hollyoaks is the reality of life in London and Chester. MPs know that Benefits Street is the reality of poor people's lives because they assume everyone's a thief. They base this insight on themselves having a higher rate of convicted criminals than the country as a whole and an impressively inclusive honour role for those caught innocently, they say, fiddling their expenses. So it’s empirical, scientific, intellectual and justified nonsense, based on soap operas.

The insulting Michael Gove believes that schools don't teach the poetry of Sassoon, Brooke and Owen, nor do we read the diaries of the soldiers, nor did I ever teach “All Quiet on the Western Front” for a German  point of view. Trips to The Battlefields? Look at the First World War as a “world war” including the world-changing three years of Russian anti Austro-Hungarian imperialism? Nah, I just shoved on the telly and said that Blackadder is the absolute truth.
Gove admires the leadership of, “great patriotic heroes,” generals struggling in an industrial war.

Ignoring the millions of dead already sacrificed, he doesn’t understand satirical TV either. Looking at WW1 and the pity of war makes us, “left wing undergraduate academic cynic(s)” and in his education fiefdom Gove will determine how his history is taught.

Gove has now claimed hero status. Faced with leftie teachers, he felt like “Horatio at the Bridge,” a reference to the pro empire Macauley’s poem where Horatio, holds off 3,000 enemies, and is remembered as the lone warrior against the hordes.( Radio 4 January 2014) Will a statue do, Michael?

Moving on:


Imagine two goalposts so far apart that you have to build a wider stadium to contain them. You have the ball at your feet such a tiny distance from this gaping gap that it is quite impossible not to score . Step forward the Labour spokesman on education, the public schoolboy from the telly,Tristram Hunt. A mere tap and he would score vital election points. With his unnecessary, bureaucratic anti Teacher Licensing Scheme, in one clumsy move, Hunt missed the goal, fouled the spectators and sent the ball into his own goal.

Labour have got nothing to say on education: no leadership, no policy, no charisma and no ears. Because if you can't score against Gove with virtually the entire education crowd roaring you on you deserve historical anonymity. Mind you, Labour seems to be doing better at the “Mine is Bigger than Yours” game on immigration, benefits and public spending cuts.

Last month, the results of the OEDC’s Pisa tests were announced. Labour and Tory politicians played the blame game, The Daily Mail howled whilst teachers worried and wondered. The BBC’s usually thoughtful  Reeta Chakrabarti presented her analysis which pretty much went: I talked to four Korean boys and they found the test easy and then I spoke with two English kids and they said it was hard. Therefore we are thick.

While all sorts of press people inaccurately recorded the UK “plummeting” we stayed in the same mid 20s position. We can do better.

Headteachers did not necessarily take the tests seriously and administered them grudgingly.  I heard from a head who gave no preparation time, did the tests on a Friday afternoon with no preparation with a disgruntled student group. Germany, Switzerland and Spain all reacted to previous poor league standing by prioritizing the content and conduct of the tests - what Gove calls “Gaming” when he doesn’t like the results. Let’s hope a Reading position of 19th satisfies the Germans.

We have no way of knowing which schools took the tests. About 4,000 15 year olds in each of the 65 reporting countries sat the tests. I know only of two secondary modern schools who took part. Secondary moderns are for those kids in places like Kent who have failed a selection test at age 11. Those who pass go to grammar schools and are at the wealthier end of our society, “those who are eligible for free school meals are less likely to secure a place in a grammar school.” (The Sutton Trust 2014)  Could it be that the selection of schools to be tested is skewed academically according to desired outcomes?  We know that public (private) schools take bright kids on scholarships, to bolster their results as their paying intake involves wealth not brains. These kids are barred from Pisa tests but in the high tabled Finland, without private schools, testing their population might be more representative of the country as a whole. Pity Kyrgyzstan who were bottom in the 2011 tables, tested during a prolonged strike by their impoverished teachers.

 Pisa tests 15 year olds so if some15 year olds are not in the school system the results will be skewed, and if you have a high dropout rate they will not be there for the tests either. Who is going to do better at the tests: those who have no secondary schooling or those staying on past the end of compulsory schooling? China will do well in the 2015 tests if it continues to educate around 60% of rural students – with the poorer  children out at work and out of tests.

The tests themselves, the results and their interpretation are riddled with unacceptable weaknesses.

Each Pisa test is not taken by all the 15 year olds tested.  By Pisa’s own admission, it would be impracticable to do the full 4 ½ hour test with all kids. Some countries have a Reading Result when no one took the reading test. The Rasch statisical model used extrapolates from the actual answers how each student would have done in questions they were not asked.

Not surprisingly this model is questioned - Professor Spiegelhalter of Cambridge University: expressed “considerable uncertainty” about the model. Svend Kreiner of Copenhagen University added. “I don’t think it’s reliable at all.”
 
Defending his work Pisa guru, Andreas Schlecher says, “The model is an approximation  of reality.” “Large variations in single country ranking positions is likely,” and “people shouldn’t pay too much attention to the precise country rankings.”

There is much criticism of the model   concurring that  Pisa’s administrators work out what the scores would have been if all the students in all the countries answered the same questions.

If the questions asked in one particular country were switched with another, league standings vary bizarrely, “with the UK finishing anywhere between 14th and 30th, Japan between 8th and 40th and Denmark between 5th and 37th .”(TES Dec 2013) 

In the 2006 tests, half those tested were not asked any questions on reading and half were not tested on Maths, although full rankings were issued for both subjects. Kreiner says “this in itself is ridiculous.”

8 of the 28 reading questions were deleted in some countries  because the OECD itself considered them ”dodgy,” but only in  some countries.  Little wonder that Dr Morrison of Belfast University deems the model, “valueless,” “utterly wrong,” and stated, “it will never work.”

As worrying is the attempt to ask the same questions in different languages. An article on Cloning, “A Copying Machine for Human beings,”  became incomprehensible translated word by word into Norwegian.” I wonder how particular metaphors and other linguistic nuances translate accurately in and out of the various languages of the Pisa test.

Kreiner, a student of Georg Rasch the creator of Pisa’s statistical method condemned the tests in 2011, “I am not actually able to find two items in Pisa’s test that function in exactly the same way in different countries…Therefore you cannot use this model.”                                                                                                                                                                                                                                               
 
Gove has called Pisa delivery man Schleicher, the most revolutionary post war educationalist. What a laugh if all the world’s education politicians get us to teach Pisa style, teach to the test until every country scores brilliantly, whilst their kids can't shake hands or smile with feeling, appreciate art, drama and music or  kick a ball correctly. You may recall the chilling image of the 5 year old Koreans trying to clamber onto the bus taking them to their extra evening schools, which 80% of them do.

The very dangerous Goveite Schools Minister Elizabeth Truss, bound as Gove’successor, as I desperately pun her name, has said that individual schools could take the tests “to compare themselves against the world’s best education systems.” (Nov 2013)The government-friendly NFER is offering trials of this. Soon, we will all be doing these tests as yet another flawed accountability measure, and yet another bloody test.

With a nod of thanks to those in America making 1851 blog hits, I thought I’d mention USA  performance – like ours it’s stagnated with position 24, 28 and 36 in Reading, Science and Maths respectively. Like everywhere, the middle class schools in wealthy areas did better than average and many of my UK doubts apply to the USA. However,  despite such poor Pisa performance America has more millionaires and the highest GDP in the world, scores highly on creativity – the odd film, book and musical item, entrepreneurship and industrial patents  granted and, of course, has more Miss Universe winners than any other country.  

Back to the tests.                                                                                                                       

If you were born in 1983 you were the first year guinea pigs, for tests at the end of KS1 (aged 7), KS2 (11), KS3  (14) new  GCSEs (at 16) and new A Levels at 18. The school was judged by your success in these tests so you were taught to these tests, practised for these tests and judged by them. And, before my 30 year old daughter revisits her, “I was so bored,” rants, the teachers were too. So if we are going to be judged by Pisa testing I will personally guarantee we move above other countries

We get the Department for Education to choose a fair cohort of schools who take the tests seriously, prepare the kids for the sort of approach required, set mocks, give each kid a banana and decent air conditioning and tell them the tests are really important.
With scientific integrity I tested Diane Carey on the easily available Sample Pisa Reading questions. At first she struggled, then, when she saw model answers she soared to total success. And as a Maths teacher she said that an hour’s preparation of her 15 year olds would lead to a 20% increase in scores. It’s like a cryptic crossword: easier once you know the style.

There is nothing inherently wrong with tests: like an MOT car test they can show you what needs improving. However, Goodhart’s Law is instructive: “Once a measure becomes a target it ceases to be a good measure,” and the danger is that Gove finds true love with Pisa: diagnosis, judgement and imposition on flimsy evidence.

Politically, the tests are a ball to be kicked in the direction your team faces. We should kick our way, teach the curriculum that’s best for our students, whilst doing a little prep for the international tests by which they will be judged.  

But if you really want, I can train sheep to answer your questions.