Waiting for Lex Luthor Part 2

We may all very much like to know the answer to George Bush’s infamous question, “Is our children learning?” but the answer will not come as easily as a comparison of standardized test scores across schools, states, national borders, or time. To understand some of the difficulties in comparing different countries to decide which ones are on the way to displacing the U.S. as an economic super power, consider how hard it is just to assess progress in our own country. 

The “A Nation at Risk” report based its alarmist claims about our failing schools on declining SAT scores, but they failed to take into account the fact that the population of SAT takers has changed significantly over time. Because of social changes in the sixties and seventies, by 1983, it was no longer the case as it had been in the past that only a relatively small proportion of students pursuing future education in elite schools were taking the test. A natural consequence of increasing the proportion of students who go on to post-secondary education (an obvious real good) is lower average SAT scores (a merely apparent bad).

By the same token, ranking educational systems by comparison of state’s average SAT scores is just as problematic. Pennsylvania ranked 42nd in the nation for average SAT scores in 2010 with its average score of 1473. Arkansas, which currently ranks 14th, has a far higher average score of 1684. This more than 200 point difference can be explained by the fact that 71% of students take the SAT in Pennsylvania while only 3% of teens in Arkansas do.  Controlling for the effect of the percent taking the test, a simple linear regression reveals that PA is actually scoring better than Arkansas on the SAT (and not surprisingly, Pennsylvanian’s outscore Arkansians on the less popular ACT, the test that 81% of Arkansians take), but let’s still not jump to any conclusions. Consider that the percent taking the test is just one lurking variable which was not taken into account in the original naive comparison that we happen to have been clever enough to identify. Who knows how many other factors may also be important? There is just no way to know how many other such variables ought to be controlled to avoid making naive judgments about states’ education programs based on SAT scores. We know it is easy to be naive, but we can have no idea how hard it is to be wise.

We have similar difficulties in comparing schools to other schools. Consider that there is a very simple way that a high school can improve its school profile with an increased average SAT score. Instead of seeking ways to increase the proportion of its students who pursue higher education as they generally do, a school could improve its average by actively discouraging its weaker students from taking the SAT. This strategy only makes sense if increasing test scores rather than our children’s best interest is the goal, and these two interests can indeed be in opposition. In the perverse logic of testing, teachers and guidance councilors who are especially good at convincing their students to have high educational goals or to at least consider taking the test now just in case they may change their mind about college in the future are having a negative impact on their school’s test scores.

At this point one might be tempted to point out the No Child Left Behind mandated testing has taken such problems of comparing school to school with different populations into account. Schools are instead compared with themselves and asked to demonstrate improvement on standardized test scores from year to year. The impressive gains in standardized test scores demonstrated by many schools are cited as evidence of the success of the testing fad. The tests themselves are used to justify the testing. 

But let’s not lose sight of the goal here. The corporate model--the Lex Luthor solution--that many would like to see imposed on education has a clear bottom line, namely profits. Test scores are offered as the appropriate analogue to earnings when it comes to school performance. Just as companies can show short term profits at the expense of long term gains, schools can increase test scores at the expense of education.

Consider the following thought experiment to understand how little impressive-looking gains in test scores may be worth. You send your son to a 30 hour SAT prep course, and afterward he increases his score by 100 points (the Princeton Review in fact guarantees that the student’s score will improve by 150 points after taking such a course which amounts to as much as a fifty percentile leap). Would you now believe that your son with his dramatically increased test score is generally better educated in some significant sense than he was before completed the 30 hours of test preparation? Certainly not. Then why would we make anything more of an improvement in the average test score of a school? 

Such companies as the Princeton Review are in the business of raising test scores, not in educating our youth or even preparing them for further academic study in college. There is an important difference here. The test score business is not the same at the education business.