Waiting for Lex Luthor Part 3

The most commonly cited test score evidence in support of today’s doomsday predictions come from the Trends in International Math and Science Study (TIMSS), a study that tests students throughout the world every four years and compares education systems by nation. U.S. News reported on the latest results in 2008 that “the United States has failed to raise student achievement in science over the past decade while Singapore and several other Asian countries continue to score higher in both subjects.” You may have heard such well-publicized claims that the U.S. is lagging behind Singapore, South Korea, Taiwan, Hong Kong, and Japan in math and science achievement as measured by these tests. U.S. News concluded that “the results suggest that the United States is not doing enough to train the next generation of scientists.” They do indeed seem to suggest that, but that claim is probably false

A researcher named Erling Boe at the University of Pennsylvania made an interesting finding using the TIMSS data. In addition to mathematics and science questions, these tests also ask more than one hundred biographical questions about the students and their families. As you might imagine, students asked to take the test frequently can not be bothered to answer some or even any of these questions, and the average number of biographical questions varies from nation to nation. American students, it turns out, tend to be less likely to feel obliged to answer these questions than students in other industrialized nations. It also turns out that the countries where students are most likely to be diligent in answering these preliminary questions score the best on these same tests.

When Boe ranked the nations in terms of the average number of biographical questions answered, he found that the rankings were the same as those for the TIMSS math test results. In other words, it is possible to rank the nations in math test performance by gauging their persistence in answering boring questions about the number of siblings the students have, their parents’ occupations and education, etc. and only based on such questions. If these same rankings can be formed without asking a single math or science question, we should be quite skeptical about believing that the TIMSS rankings are comparing math and science achievement at all. 

Boe’s study illustrates that what the test may be telling us is that students in Singapore, South Korea, Taiwan, Hong Kong, and Japan are more willing to work hard on a test that has absolutely no direct consequences for them whatsoever just because an adult told them to try their best, and that is perhaps all the TIMSS rankings say.

If it is the case that differences we see between nations can largely be explained by willingness to comply with adult requests for no other reason than that they asked for it, then is an inferior ranking a good thing or a bad thing? I think there are pros and cons to weigh. It is generally a good thing for young people to be obedient to their teachers, but perhaps like me you can easily imagine Bill Gates, Steve Jobs, and Mark Zuckerberg as having once been the sort of high school students who would not have seen fit to give their best to such a test. Perhaps the rebellious tendency of American youth as compared to other nations is part of what contributes to the fact that we produce so many great innovators.