Friday, September 18, 2009

Standardized tests: analysing the analysis

In the Ottawa Citizen today there was an editorial that was critical of the performance of the local school boards in the recently released results of testing done by the Educational Quality and Accountability Office (EQAO) in Ontario.

This surprised me since yesterday I was blogging about a story in the Citizen talking about the same results. That story put a positive spin on the results, even though I felt the analysis was simplistic.

For those who want to look at the actual data instead of the simplistic summaries in the papers, go here and select a grade and year.

I wrote a letter to the editors of the Citizen, pointing out the flaws in the editorial, and I am also going to publish that letter here:

I would like to make a few comments about the editorial “Rising to the Test” in the Citizen on Friday, September 18. For the record, full disclosure: I am a public school board High School teacher in Math and Science (currently supply teaching).

My first comment is about interpretation of the test scores. All the results talk about the percentage of students who meet or exceed the provincial standards. For example, the Grade 3 reading test had 63% of students meet or exceed provincial standards. This makes it sound like 37% of Grade 3 students are failing in reading. However, when you look at the full results, you find that 27% of students achieved a Level 2 result, which is below provincial standards but is about the equivalent of a C. So, 90% of students are either within striking distance of provincial standards, have met the standards, or have exceeded them. Furthermore, in Grade 3 reading 8% of students scored at Level 1, which is approximately a D. In total, 98% of students got a result which would be a pass at school. Looked at one way, only 2% of are Grade 3 students are failing in reading. Yet the result that is put out by the Educational Quality and Accountability Office (EQAO) is 63%. Could that be because if EQAO published a 98% pass rate then people might suggest that we don't need the EQAO?

Another comment I have deals with media interpretation of the results. The editorial states that “most Ottawa schools are underachieving.” Yet an online story in the Citizen yesterday had the headline “Area schools outperform provincial average in reading, writing, tests reveal”. So how did Ottawa area schools go from beating the provincial average to underachieving? The editorial mentions that for the public board “only 73 per cent of Grade 6 students met the provincial standard in reading.” But 26% scored a level 2, Meaning that 99% of public board Grade 6 students are at least close to the standard. That does not sound like underachieving to me. If you want to truly claim that our schools are not up to snuff, you need to offer a better explanation than “ Many students in the nation's capital come from homes where the parents have high levels of education”, a statement that offers nothing to show how much better Ottawa should be doing based on this factor.

A third comment is about the statement “Strangely, some critics respond by questioning the value of standardized tests.” There is plenty of reason to be wary of standardized tests and their results, especially when that seems to be almost the only facet of education on which the media report. As an example, look at the statistic for primary math results published in the editorial. Ten years ago the primary math results were 56% (44% below standards) and now the result is 70% (30% below standard). So, in ten years we have gone from 44% to 30% below standards, almost a 1/3 decrease in poor results! Is this because math is taught so much better now than ten years ago? Not a chance.

Sure, some of the improvement is the result of improved teaching practices, but I am willing to bet a large sum of money that most of the improvement is because teachers have learned how to prepare their students for the test. I believe this is true because as a High School Mathematics teacher I see and hear about the Grade 9 EQAO Math test. A common refrain from teachers is that their students understand the questions quite well but have trouble answering them in the form that the test demands. The result is that teachers are forced to spend time teaching their students how to deal with the test format instead of teaching course content. So, results improve not because of more learning of content, but simply because of teaching to the test. If that does not raise at least some questions about the value of standardized testing, I do not know what will.

I agree that the results of standardized testing offer information that can be valuable when looking at how our well schools and boards are educating students. But the results need to be treated with care because the tests do not cover the whole curriculum, there are issues of teaching to the test, and there are issues with interpretation, especially as the results are often presented in a simplistic manner by the EQAO.

No comments:

Post a Comment