The Globe and Mail had an article about teacher merit pay in their Saturday edition this weekend. The article was similar in many respects to Malcolm Gladwell's article "Most Likely to Succeed", minus the football comparison.
The Globe article highlighted many problems with merit pay, and I won't repeat them here, since I have already blogged on that topic.
One point that I felt that the Globe article missed was that teacher merit is, in the end, about teacher merit. Which teachers are truly helping the majority of students learn and succeed? Those teachers need to be encouraged and rewarded (although not necessarily with money). Which teachers are failing to help and teach the majority of their students? Those teachers need to be helped so that they can improve, or, if they cannot improve enough, they should be gently removed from the profession.
The problem with the idea of teacher merit is that I don't think that a good definition of teacher merit in terms of measurable quantities currently exists. Gladwell and the Globe article discuss the idea of evaluating teacher performance by looking at standardized test results over several years, but there are no specifics. Between the lack of specifics, human inertia, and teacher union resistance, I suspect that a good definition of teacher merit through measurable values is several decades away.
So, until we have that definition, I fear that we are stuck with the status quo. Too bad. I, for one, would be happy to know whether I am actually doing a good job of teaching or whether I just think that I am doing a good job.
Showing posts with label standardized tests. Show all posts
Showing posts with label standardized tests. Show all posts
Monday, February 8, 2010
Friday, September 18, 2009
Standardized tests: analysing the analysis
In the Ottawa Citizen today there was an editorial that was critical of the performance of the local school boards in the recently released results of testing done by the Educational Quality and Accountability Office (EQAO) in Ontario.
This surprised me since yesterday I was blogging about a story in the Citizen talking about the same results. That story put a positive spin on the results, even though I felt the analysis was simplistic.
For those who want to look at the actual data instead of the simplistic summaries in the papers, go here and select a grade and year.
I wrote a letter to the editors of the Citizen, pointing out the flaws in the editorial, and I am also going to publish that letter here:
I would like to make a few comments about the editorial “Rising to the Test” in the Citizen on Friday, September 18. For the record, full disclosure: I am a public school board High School teacher in Math and Science (currently supply teaching).
My first comment is about interpretation of the test scores. All the results talk about the percentage of students who meet or exceed the provincial standards. For example, the Grade 3 reading test had 63% of students meet or exceed provincial standards. This makes it sound like 37% of Grade 3 students are failing in reading. However, when you look at the full results, you find that 27% of students achieved a Level 2 result, which is below provincial standards but is about the equivalent of a C. So, 90% of students are either within striking distance of provincial standards, have met the standards, or have exceeded them. Furthermore, in Grade 3 reading 8% of students scored at Level 1, which is approximately a D. In total, 98% of students got a result which would be a pass at school. Looked at one way, only 2% of are Grade 3 students are failing in reading. Yet the result that is put out by the Educational Quality and Accountability Office (EQAO) is 63%. Could that be because if EQAO published a 98% pass rate then people might suggest that we don't need the EQAO?
Another comment I have deals with media interpretation of the results. The editorial states that “most Ottawa schools are underachieving.” Yet an online story in the Citizen yesterday had the headline “Area schools outperform provincial average in reading, writing, tests reveal”. So how did Ottawa area schools go from beating the provincial average to underachieving? The editorial mentions that for the public board “only 73 per cent of Grade 6 students met the provincial standard in reading.” But 26% scored a level 2, Meaning that 99% of public board Grade 6 students are at least close to the standard. That does not sound like underachieving to me. If you want to truly claim that our schools are not up to snuff, you need to offer a better explanation than “ Many students in the nation's capital come from homes where the parents have high levels of education”, a statement that offers nothing to show how much better Ottawa should be doing based on this factor.
A third comment is about the statement “Strangely, some critics respond by questioning the value of standardized tests.” There is plenty of reason to be wary of standardized tests and their results, especially when that seems to be almost the only facet of education on which the media report. As an example, look at the statistic for primary math results published in the editorial. Ten years ago the primary math results were 56% (44% below standards) and now the result is 70% (30% below standard). So, in ten years we have gone from 44% to 30% below standards, almost a 1/3 decrease in poor results! Is this because math is taught so much better now than ten years ago? Not a chance.
Sure, some of the improvement is the result of improved teaching practices, but I am willing to bet a large sum of money that most of the improvement is because teachers have learned how to prepare their students for the test. I believe this is true because as a High School Mathematics teacher I see and hear about the Grade 9 EQAO Math test. A common refrain from teachers is that their students understand the questions quite well but have trouble answering them in the form that the test demands. The result is that teachers are forced to spend time teaching their students how to deal with the test format instead of teaching course content. So, results improve not because of more learning of content, but simply because of teaching to the test. If that does not raise at least some questions about the value of standardized testing, I do not know what will.
I agree that the results of standardized testing offer information that can be valuable when looking at how our well schools and boards are educating students. But the results need to be treated with care because the tests do not cover the whole curriculum, there are issues of teaching to the test, and there are issues with interpretation, especially as the results are often presented in a simplistic manner by the EQAO.
This surprised me since yesterday I was blogging about a story in the Citizen talking about the same results. That story put a positive spin on the results, even though I felt the analysis was simplistic.
For those who want to look at the actual data instead of the simplistic summaries in the papers, go here and select a grade and year.
I wrote a letter to the editors of the Citizen, pointing out the flaws in the editorial, and I am also going to publish that letter here:
I would like to make a few comments about the editorial “Rising to the Test” in the Citizen on Friday, September 18. For the record, full disclosure: I am a public school board High School teacher in Math and Science (currently supply teaching).
My first comment is about interpretation of the test scores. All the results talk about the percentage of students who meet or exceed the provincial standards. For example, the Grade 3 reading test had 63% of students meet or exceed provincial standards. This makes it sound like 37% of Grade 3 students are failing in reading. However, when you look at the full results, you find that 27% of students achieved a Level 2 result, which is below provincial standards but is about the equivalent of a C. So, 90% of students are either within striking distance of provincial standards, have met the standards, or have exceeded them. Furthermore, in Grade 3 reading 8% of students scored at Level 1, which is approximately a D. In total, 98% of students got a result which would be a pass at school. Looked at one way, only 2% of are Grade 3 students are failing in reading. Yet the result that is put out by the Educational Quality and Accountability Office (EQAO) is 63%. Could that be because if EQAO published a 98% pass rate then people might suggest that we don't need the EQAO?
Another comment I have deals with media interpretation of the results. The editorial states that “most Ottawa schools are underachieving.” Yet an online story in the Citizen yesterday had the headline “Area schools outperform provincial average in reading, writing, tests reveal”. So how did Ottawa area schools go from beating the provincial average to underachieving? The editorial mentions that for the public board “only 73 per cent of Grade 6 students met the provincial standard in reading.” But 26% scored a level 2, Meaning that 99% of public board Grade 6 students are at least close to the standard. That does not sound like underachieving to me. If you want to truly claim that our schools are not up to snuff, you need to offer a better explanation than “ Many students in the nation's capital come from homes where the parents have high levels of education”, a statement that offers nothing to show how much better Ottawa should be doing based on this factor.
A third comment is about the statement “Strangely, some critics respond by questioning the value of standardized tests.” There is plenty of reason to be wary of standardized tests and their results, especially when that seems to be almost the only facet of education on which the media report. As an example, look at the statistic for primary math results published in the editorial. Ten years ago the primary math results were 56% (44% below standards) and now the result is 70% (30% below standard). So, in ten years we have gone from 44% to 30% below standards, almost a 1/3 decrease in poor results! Is this because math is taught so much better now than ten years ago? Not a chance.
Sure, some of the improvement is the result of improved teaching practices, but I am willing to bet a large sum of money that most of the improvement is because teachers have learned how to prepare their students for the test. I believe this is true because as a High School Mathematics teacher I see and hear about the Grade 9 EQAO Math test. A common refrain from teachers is that their students understand the questions quite well but have trouble answering them in the form that the test demands. The result is that teachers are forced to spend time teaching their students how to deal with the test format instead of teaching course content. So, results improve not because of more learning of content, but simply because of teaching to the test. If that does not raise at least some questions about the value of standardized testing, I do not know what will.
I agree that the results of standardized testing offer information that can be valuable when looking at how our well schools and boards are educating students. But the results need to be treated with care because the tests do not cover the whole curriculum, there are issues of teaching to the test, and there are issues with interpretation, especially as the results are often presented in a simplistic manner by the EQAO.
Thursday, September 17, 2009
We want everyone to be above average?
The online version of the Ottawa Citizen just published a story about how well local school boards are doing relative to the provincial average.
As I read through the article, just about every statistic was compared with the provincial average. Essentially the article is saying "above average good, below average bad." This is not completely unreasonable, but it is simplistic and makes education look like a competition between school boards when it should be about simply doing what is best for the students.
The big problem with using average as the yardstick is that, no matter how you slice it, approximately half the school boards are going to be below average. That is how average works. Currently the province has a goal to have 75% of students score at the provincial standard or above on the EQAO (Education Quality and Accountability Office) tests. As a reference, the provincial average for Grade 3 Math results is 70%. However, even if schools across the province made massive gains and all scored better than the 75% goal, still there would be about half the schools below average. Would that mean that those below average schools are bad? Nope, but that is probably what the media would report.
So, the moral of the story is that if you want to truly understand stories in the media, it is important to know about statistics and how the various statistical measures work. Otherwise you may end up trying to figure out a way to achieve the impossible of making everyone above average.
As I read through the article, just about every statistic was compared with the provincial average. Essentially the article is saying "above average good, below average bad." This is not completely unreasonable, but it is simplistic and makes education look like a competition between school boards when it should be about simply doing what is best for the students.
The big problem with using average as the yardstick is that, no matter how you slice it, approximately half the school boards are going to be below average. That is how average works. Currently the province has a goal to have 75% of students score at the provincial standard or above on the EQAO (Education Quality and Accountability Office) tests. As a reference, the provincial average for Grade 3 Math results is 70%. However, even if schools across the province made massive gains and all scored better than the 75% goal, still there would be about half the schools below average. Would that mean that those below average schools are bad? Nope, but that is probably what the media would report.
So, the moral of the story is that if you want to truly understand stories in the media, it is important to know about statistics and how the various statistical measures work. Otherwise you may end up trying to figure out a way to achieve the impossible of making everyone above average.
Subscribe to:
Posts (Atom)
