Contact The Editor
Articles By This Author
Now think about this number: 11. Because when NPR tried to confirm the incidents reported by the government, that's how many it could verify. And while some of the entries in the Education Department's study could neither be verified nor disproved, NPR's report found that in two-thirds of the cases, the school districts contacted by the news organization said that no shootings had occurred.
How can NPR and the Education Department be getting such clashing results?
NPR's reporting suggests that much of the problem is that school districts simply filled out the forms incorrectly. In Cleveland, for example, whoever was in charge of compiling the data seems to have put the answer to the previous question - which asked about possession of a knife or firearm - into the space designated for the discharge of a firearm on school grounds.
These kinds of data errors inevitably creep into any large survey, as anyone who has ever made a slight mistake on a tax form can attest. And in most contexts, such errors probably don't matter much; they're just a bit of statistical noise in a broadly sound dataset.
But they become a big problem when the phenomenon being studied is relatively rare.
When the incidence is low, the data errors can easily swamp the real effect, making it seem many times larger than it actually is. Small isn't beautiful when it comes to statistical samples.
Such number ugliness isn't restricted to data accidents. It turns up in news reports about polls showing that a modest but startling percentage of people believe something utterly insane - that the world is secretly ruled by lizard people, that Barack Obama is the antichrist, that chocolate milk comes from brown cows. These results often stir handwringing about ignorance - and not about the possibility that the polls are simply measuring the modest but significant percentage of people who will say random things for the sheer joy of messing with pollsters.
But even when the data is sound and the respondents are all very serious, small can still be utterly misleading. Remember the push for smaller schools in the 2000s? That was based on solid data showing that small schools were consistently the highest-performing.
The data was correct; it just didn't show what the researchers thought. Small schools, it turned out, were also the worst-performing, while larger schools tended to cluster in the middle of the pack. Which is just what statistics teaches us to predict if results are normally distributed and driven by random variation.
This effect is best illustrated by height. Say we're measuring the heights of three men drawn randomly from the population, and using those figures to estimate the average height of U.S. citizens. We draw one guy who's 5-foot-6, one who's 5-foot-9 - and Kareem Abdul-Jabbar, who is 7-foot-2. We will conclude that the average height of the U.S. population is 6-foot-2 - five inches taller than it actually is.
But if we keep adding people to the group who mirror the normal distribution of American male height, with its true mean of 5-foot-9, that will tamp down the effect of an individual outlier. By the time we're at 100 people, Abdul-Jabbar's influence on the mean will be less than a fifth of an inch.
But of course, if Patrick Ewing and Shaquille O'Neal, both 7-foot-1, somehow get into the mix, the distortions will be greater. That's what can happen with a limited group, like 100 people. But as the groups being studied expand, the likelihood of accidentally drawing enough outliers to swamp the rest of the group's more normal heights is reduced - there are only so many people over 7 feet tall. So larger samples are more likely than small ones to be close to the true mean - and as with height, so with school performance.
This may seem like a bunch of dry math. But the government was presumably collecting the school-shooting numbers for a reason; they were supposed to help guide policy.
Those surveys of extreme beliefs are frequently used to disparage members of particular groups in ways that further deepen the nation's political and cultural divides.
The findings about the purported benefits of small schools drove the Gates Foundation a decade ago to spend $1 billion on a small-schools initiative, money that could have been spent chasing real results, rather than random chance.
(COMMENT, BELOW)
Megan McArdle is a Washington Post columnist who writes on economics, business and public policy. She is the author of "The Up Side of Down." McArdle previously wrote for Newsweek-the Daily Beast, Bloomberg View,the Atlantic and the Economist.