Waiting for Lex Luthor Part 2

We may all very much like to know the answer to George Bush’s infamous question, “Is our children learning?” but the answer will not come as easily as a comparison of standardized test scores across schools, states, national borders, or time. To understand some of the difficulties in comparing different countries to decide which ones are on the way to displacing the U.S. as an economic super power, consider how hard it is just to assess progress in our own country. 

The “A Nation at Risk” report based its alarmist claims about our failing schools on declining SAT scores, but they failed to take into account the fact that the population of SAT takers has changed significantly over time. Because of social changes in the sixties and seventies, by 1983, it was no longer the case as it had been in the past that only a relatively small proportion of students pursuing future education in elite schools were taking the test. A natural consequence of increasing the proportion of students who go on to post-secondary education (an obvious real good) is lower average SAT scores (a merely apparent bad).

By the same token, ranking educational systems by comparison of state’s average SAT scores is just as problematic. Pennsylvania ranked 42nd in the nation for average SAT scores in 2010 with its average score of 1473. Arkansas, which currently ranks 14th, has a far higher average score of 1684. This more than 200 point difference can be explained by the fact that 71% of students take the SAT in Pennsylvania while only 3% of teens in Arkansas do.  Controlling for the effect of the percent taking the test, a simple linear regression reveals that PA is actually scoring better than Arkansas on the SAT (and not surprisingly, Pennsylvanian’s outscore Arkansians on the less popular ACT, the test that 81% of Arkansians take), but let’s still not jump to any conclusions. Consider that the percent taking the test is just one lurking variable which was not taken into account in the original naive comparison that we happen to have been clever enough to identify. Who knows how many other factors may also be important? There is just no way to know how many other such variables ought to be controlled to avoid making naive judgments about states’ education programs based on SAT scores. We know it is easy to be naive, but we can have no idea how hard it is to be wise.

We have similar difficulties in comparing schools to other schools. Consider that there is a very simple way that a high school can improve its school profile with an increased average SAT score. Instead of seeking ways to increase the proportion of its students who pursue higher education as they generally do, a school could improve its average by actively discouraging its weaker students from taking the SAT. This strategy only makes sense if increasing test scores rather than our children’s best interest is the goal, and these two interests can indeed be in opposition. In the perverse logic of testing, teachers and guidance councilors who are especially good at convincing their students to have high educational goals or to at least consider taking the test now just in case they may change their mind about college in the future are having a negative impact on their school’s test scores.

At this point one might be tempted to point out the No Child Left Behind mandated testing has taken such problems of comparing school to school with different populations into account. Schools are instead compared with themselves and asked to demonstrate improvement on standardized test scores from year to year. The impressive gains in standardized test scores demonstrated by many schools are cited as evidence of the success of the testing fad. The tests themselves are used to justify the testing. 

But let’s not lose sight of the goal here. The corporate model--the Lex Luthor solution--that many would like to see imposed on education has a clear bottom line, namely profits. Test scores are offered as the appropriate analogue to earnings when it comes to school performance. Just as companies can show short term profits at the expense of long term gains, schools can increase test scores at the expense of education.

Consider the following thought experiment to understand how little impressive-looking gains in test scores may be worth. You send your son to a 30 hour SAT prep course, and afterward he increases his score by 100 points (the Princeton Review in fact guarantees that the student’s score will improve by 150 points after taking such a course which amounts to as much as a fifty percentile leap). Would you now believe that your son with his dramatically increased test score is generally better educated in some significant sense than he was before completed the 30 hours of test preparation? Certainly not. Then why would we make anything more of an improvement in the average test score of a school? 

Such companies as the Princeton Review are in the business of raising test scores, not in educating our youth or even preparing them for further academic study in college. There is an important difference here. The test score business is not the same at the education business.

Waiting for Lex Luther

If our education system is truly broken as is so often claimed, then we can ignore the Hippocratic maxim “first do no harm.” Doing something--anything at all--will be at least no worse than the status quo. Of course we should always strive to improve public education, but if public education is generally working, if the statistical measures that predict our doom are bunk (and I will argue in later posts that we have good reason to think that at least many of them are), then we have something to lose in considering sweeping education reform proposals, something quite precious. 

If so, we ought to be cautious about imposing “magic bullet” reforms such as those propagandized in the 2010 documentary “Waiting for Superman,” specifically charter schools and union-busting. It was the villain, Lex Luthor, who was the industrialist and mad scientist trying to impose his vision on the world to fit his schemes. It seems to be he likes of him rather than Superman to whom too many are looking for help in improving our system of education.

The notion of “America’s failing schools” or our “broken educational system” is a cherished dogma that few seem incentivized to question. Both Democrats and Republicans find it expedient for pursuing their ends. Democrats use it for seeking greater funding for schools which helps them maintain support from teachers unions. Republicans find it just as useful for fomenting opposition to unions and painting pictures of big government boondoggles to justify tax cuts. Education theoreticians depend on the myth of failing schools to help promote their various reform proposals. Even Richard Dawkins and Sam Harris celebrated for their skepticism of dogmas are quick to attribute America’s “failing schools” to those rare nutty school boards pushing intelligent design. They don’t have an ounce of their tauted skepticism to direct toward the “failing schools” slogan itself. It seems to be one-size-fits-all evidence for support of just about any agenda. The political echo chamber reverberates with the truism that American students rank among the worst educated in math and science when compared to industrialized countries of the world, but where does this notion come from, and is it true?

First consider that claims that the sky is falling when it comes to the preparation of youth to take the lead in the future are nothing new. Growing up in the 1980’s, the big fear in my youth was that the Japanese superior work ethic in schools and factories would drive us to second rate nation status. In the widely heralded 1983 report “A Nation at Risk,” author James J. Harvey summarized the sentiments of the commission members with these memorable opening lines: "the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people.” It went on to claim that “If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war." These words will certainly sound familiar to anyone in education who is at least as old as I am (40 years old with 16 years teaching experience). 

A generation after the report, these claims of mediocre education spelling the doom of American society in retrospect are revealed as no more than the typical complaints that every generation in history has had about the moral and intellectual failings of it’s youth. These are the complaints that Socrates had about the youth of ancient Athens, that our grandparents had about our parent’s generation and the ones that our grandchildren will no doubt someday have about their children.

If “sky is falling” predictions were so wrong 28 ago, what should make us think that they are any more likely to be right when such modern seers forecast decades into the future today? The Communists haven’t invaded. Japanese men in business suits haven’t taken over the world. (Our collective fears on that score are today directed at China and India) That doesn’t mean that our future will necessarily be rosy. Our own banker’s lack of ability to understand the risks they were taking surprisingly turned out to constitute a far greater threat to our economic well-being than any foreign power in recent years. We will likely be surprised over and over again by the future. One thing we should always expect is the unexpected. We should therefore have a lot more humility about our forecasts and recognize that we just aren’t very good at predicting such things. We should try to improve education without lending any credence towards doomsday predictors that may lead us to hasty and drastic measures with perhaps dire unforeseen consequences.