But like all statistics, StateMaster’s figures have to be taken with a grain of salt. My statistics professor was fond of saying that unless you knew the methods used to gather, filter, and derive the data behind a statistic, the number was essentially meaningless. He warned that without this essential background knowledge we tend to lend unwarranted credibility to statistics cited in by media organizations, companies, academicians, governments, and others.
I happened to click on this StateMaster report showing Utah’s health status ranking. How is it that Utah ranks as #7 in the nation on the health index? StateMaster doesn’t say. And the links it provides to its main source, the Kaiser Family Foundation’s statehealthfacts.org site are all broken. Fortunately you can get to the right place rather easily on your own.
It is unsurprising that Utah has among the nation’s lowest rates for smoking, alcohol consumption, alcohol related fatalities, illicit drug use, and sexually transmitted diseases. And that it has the nation’s highest birth rate. A little more surprising to me is the obesity rate of 18% (making Utah 43 of 51) and 83.1% for physical exercise (2 of 52). Are Utahns really that slender and exercise oriented?
I became more suspicious when the report said that Utah’s prevalence of poor mental health was 41.4%; the worst in the nation. While this may fit a stereotype fostered in certain circles, it doesn’t seem to pass the smell test. Are over 40% of Utahns actually mentally unhealthy?
I started digging into the sources of the health data, and found that most of it came from the Centers for Disease Control Behavioral Risk Factor Surveillance System. The surveillance system is nothing more than an annual telephone survey.
Many of the figures cited on the StateMaster site come from the 2007 survey. I’m not sure why some figures come from different years. The CDC applies various weighting factors to the results. A few of those factors are explained here, but it is not at all clear how Utah’s or any other state’s statistics were actually derived. The CDC provides this document that delves into data quality, but it doesn’t really treat my question.
Telephone surveys are notoriously faulty. They do provide some value for tracking opinion trends, but they are terribly unreliable at pinning down hard facts. The main problem is that respondents have no incentive to treat questions seriously or to answer accurately. The result for the respondent isn’t much different either way. If something serious, such as financial reward, future status, or a personal reputation were riding on the answers, you can bet that you’d see significantly different answers.
Then you have to ask whether the population of people willing to answer an extensive CDC telephone survey closely approximates the general population of a state. It is reasonable to assume that respondents are more likely to be people with telephone land lines and are less likely to have caller-ID. They apparently have enough time on their hands to answer the extensive survey. Are most people in the state like that?
Next you have to question the usefulness of the information you get from these people. When the CDC survey taker asks how much the person weighs without shoes on (question #12.11), how useful is the information received? It is wholly unverifiable. It might mean that Utahns are lighter than their counterparts in most states. Or it could mean that Utahns are more likely to lie about their weight than people in other states. We don’t know. It would be silly to treat these answers as if they had serious scientific value.
The quality of the survey questions is a major factor. The physical exercise statistic comes from a single unverifiable question (#4.1):
“During the past month, other than your regular job, did you participate in any physical activities or exercises such as running, calisthenics, golf, gardening, or walking for exercise?”Possible answers are Yes, No, Don’t know / Not sure, and Refused. How would you answer the question? Even if you answered yes, do you get anywhere near enough physical exercise? Respondents’ answers to this question do not provide any serious indication of a population’s level of physical activity.
Getting back to the mental health issue. Section 16 of the questionnaire includes 10 questions about mental health. Several are related to stress and/or depression. Some are about mental health perceptions. The first and third questions try to pinpoint now nervous and fidgety you have felt in the last 30 days. Despite the answer classifications, these questions are very subjective. In fact, all of the mental health questions are highly subjective.
Although StateMaster says that 41.4% of Utah survey respondents (as weighted) reported “poor mental health,” the Kaiser Foundation site, which is supposed to be the data source, pins the number at 38.1%. Even at this lower number, it can only be assumed that the definition of poor mental health is so broad as to include almost any temporary negative emotional perception; a position that would please those in the mental health activism industry.
As with the questions about weight, the value of the mental health questions relative to other geographic areas is dubious. The answers are completely unverifiable. One person’s feeling “restless” on “most” days may be analogous to another person’s sense of restlessness on “some” or few days. Or not. We simply don’t know. As with questions about weight, the answers provided by respondents have little scientific value.
The CDC survey can be a useful tool for tracking trends in a population’s opinions about some health related issues. But there is too much ambiguity in most of the data gathered to provide any real value with respect to geographical ranking.
The StateMaster site pegs Utah at #7 on the “Health Index.” This index is simply a bunch of factors that a group of elites with various unapparent motivations decided was important. Given that the data plugged into the index was poorly controlled and often ambiguous, we should assume nothing more than entertainment value from this ranking and from the rankings of the categories that make up the index.
Continue reading at the original source →