Comparing Data: How to Use Reliability Measures in Real Life

One common mistake we see when interpreting data is the comparison of two different data points without considering the context the data was collected or is being presented in. For more information about data context, check out our previous post A Guide to Conscious Data Consumption.  One type of statistics that helps compare data points is reliability measures. For definitions of reliability measures like confidence intervals, margins of error, and standard deviation, scroll to the end of this post for a refresher!  Otherwise, keep reading to learn how we use these reliability measures in data analysis.

Reliability Measures in Real Life

The most common place you’ll see a margin of error reported is in political or opinion polls.  Have you ever looked at a survey and wondered how they can only talk to only 400 people and think it’s a reasonable result?  Well, it depends! Most of the time, you’ll also see a number reported like this (+/- 5%), that tells you that the margin of error is within 5 percentage points of the real value, so the real population is somewhere in a 10 point range (it’s with some certainty, but not 100%! Read about confidence levels here).  Let’s look at a real political poll as an example.

The week before the 2016 election, Reuters had a poll of a sample of voters that put Candidate A at 42% and Candidate B at 39% of the votes. Many people looked at this and walked away thinking Candidate A had a strong lead, but didn’t take into account the reliability measures.

The margin of error that Reuters reported was 2.4%.  So if we subtract 2.4% from Candidate A and B’s estimates, we get the lower bound of the confidence interval.  We also add 2.4% to each estimate to get the upper bound. While Candidate A’s sample estimate is 42%, Reuters is saying that ANY number between 39.6% and 44.4% could still be “right” if you spoke to every single eligible voter.

That means Candidate A’s percentage could be as low as 39.6% and high as 44.4% and Candidate B as low 36.7% and as high as 41.3%.  This means that it is possible, when talking to every single voter, that Candidate B has a higher percentage of support than Candidate A.

Reliability Measures and the American Community Survey 

Another place we use reliability measures all the time is the American Community Survey (ACS). While the decennial Census attempts to count every American, the ACS  only surveys a small sample of the entire US population each year so we can have data between the census. Thus, all of the ACS estimates are also reported with a margin of error. 

There are three main ways that we try to compare ACS estimates: 

  1. Category comparisons like male vs. female or under 18 vs. adults
  2. Geography comparisons like Detroit vs. Wayne or Michigan vs. Wisconsin
  3. Time comparisons like Detroit in 2018 to Detroit in 2019

This concept of a margin of error can help us determine whether the ACS estimates represent an actual difference or if the estimates might overlap.  A common mistake we see is not paying attention to margins of error, especially in time comparisons between years. We made up an imaginary city that has had variation in the estimate of the number of children living in it over the past four years.

Looking at these estimates, it’d be easy to make a statement that in 2016, the number of kids living in City A increased significantly from 2015, and grew again in 2018.  You can almost see the 2016 and 2018 newspaper headlines highlighting the additional strain on services for children in City A! However, let’s take a look at the margins of error for each year as well. When you add and subtract the margin of error from an estimate, the distance between the lowest and highest is called a confidence interval.

By charting the confidence intervals, we can quickly see where they overlap for each year’s estimate.  Remember, the confidence interval is calculated by subtracting and adding the margin of error to the estimate to give us the lowest and highest possible estimate for the entire population. 

We quickly see that the confidence intervals for 2015-2017 all overlap, which means that the sample estimate reported in the first chart is not reliable enough to confidently say that there was measured change from 2015 to 2016.  In 2018, the increase to 200 children includes a confidence interval that does not overlap with the other estimates. So while the actual population of children in 2018 could be anywhere from 180 to 220, we can be confident that this is significantly higher than the three previous years where the highest possible estimate was 175 in 2016.

American Community Survey in Detroit

We make it super easy to check margins of error for many data points on State of the Detroit Child.  If you hover over big numbers, the margin of error pops up automatically.  For more detailed data, just click on “show data”.  

Below we have the distribution of youth by sex for Detroit, Wayne County, and Michigan.  We want to answer the following question for each geography: Are there more female or male children?

First, let’s look at Detroit where the ACS reports that there are 85,688 male youth and 85,382 female youth.  It would appear that there are more male youth at first glance, but let’s use the margins of error to check out the confidence intervals!

The actual number of male youth in Detroit could be anywhere between 84,074 and 87,302.

The actual number of female youth in Detroit could be anywhere between 83,984 and 86,780.

Since the lower bound of the male youth confidence interval and upper bound of the female youth confidence interval overlap, we can’t say with reasonable certainty that there were more male youth than female youth in Detroit at the time of they survey.

So what about Wayne County and Michigan?

In Wayne County, the confidence interval for male youth is 213,368 to 217,061 and for female youth is 206,206 to 209,070.  Since the confidence intervals don’t overlap, we can say with reasonable certainty that there are more male youth in Wayne County.

In Michigan, the confidence interval for male youth is 1,126,334 to 1,133,108 and for female youth is 1,075,27 to 1,082,329.  Again, since the confidence intervals don’t overlap, we can say with reasonable certainty that there are more male youth in Michigan.

So we answered the question!  We can’t be sure if there are more male youth than female youth in Detroit, but there are more male youth in Wayne County and Michigan.

How can the same data source provide reliable results for some spaces and not others?  Well, remember the definition of margin of error earlier? Margin of errors take into account sample size.  Since Wayne County and the state of Michigan have larger samples than the city, the ACS is able to better approximate the sex of youth from the survey than in a smaller geography, like the City of Detroit.

This is important to take into consideration especially since the ACS gets reported for much smaller geographies like census tracts and block groups.  You should always double check that the confidence intervals don’t overlap before making decisions based on “different” estimates from a survey!

Important terms

Standard Deviation

When comparing the standard deviation of two samples, the values in the sample with the larger standard deviation vary (deviate from each other) more than the sample with a smaller standard deviation.  Standard deviation is used to calculate the margin of error.

Margin of Error

A margin of error is calculated using complex math, employing the sample size, standard deviation, and other statistics.  It tells us how accurately representative a number is of the entire population.  

Confidence Interval

Using the margin of error, a confidence interval reports the highest and lowest possible value for a population.