As we’ve covered previously, data collection in a pandemic can be challenging. Similar to the Current Population Survey (CPS), the Census Bureau faced multiple challenges collecting data for the American Community Survey (ACS). This resulted in the Census Bureau not publishing the usual 1-year estimates, only experimental data, which relies on a number of statistical inferences and supplemental administrative data. Due to the experimental nature of these 1-year Estimates, D3 is not going to integrate them into our existing workflows. Instead, we will continue to utilize the 5-year estimates for our data analyses.
While many of the same issues exist for the 5-year calculations, since it is an average over 5 years and not a single-year, the impact is much smaller. The Census Bureau did warn about higher than usual margins of error, especially in smaller geographic areas. Since D3 utilizes census tracts frequently with our clients (one of the smallest Census geographic areas), we decided to analyze the margins of error reported in the 2020, 2018, and 2015 5-Year estimates in order to determine the impact on accuracy in the Detroit region. For more on what margins of error can help us understand you can watch our instructional video or read more on this blog post.
We pulled three years of data for four commonly used variables in 10 random Detroit-based census tracts: Median housing value, vacant/occupied housing, total population, and median rent. Then, we calculated how big the margin of error is compared to the ACS estimate in that Census tract by dividing the margin of error by the estimate to get a percentage. You can access all four data points across the ten census tracts here.
For example, looking at the data for median housing value for two census tracts, we can calculate how big the margin of error is as a percentage of the 2015 estimate.
Comparing Tract 5114 to 5652, we can see that the margin of error for tract 5114 is proportionally larger than tract 5652. This helps us compare margins of errors and estimates by creating a standard measure that takes into account the size of the estimates. For example, a margin of error of 50 is much larger proportionally if the estimate is 100, versus if the estimate is 1,000. In 2015, the median rent margins of error in our 10 census tracts ranged from 3.59% to 25.3% of the estimates.
Across the four data points, we found that margins of error in the 2020 dataset did have a higher level of the variance than in 2015 and 2018. However, they’re not solely in one direction (so some of the 2020 ones are actually smaller than prior ACS). For example the margin of error in Census tract 5435 was 24% of the estimate in 2015, 22.9% in 2018, and 10.71% in 2020. This variance suggests that the 2020 estimates aren’t all biased in the same direction despite the additional processing steps required in 2020. There’s also not a large number of estimates where the margins of error substantially increased.
Moving forward, D3 will use the 2020 ACS 5-year estimates, but pay careful attention to the margins of error especially in smaller geographic areas like tracts or block groups. We will be certain to raise the flag with clients and the community if there is a concern about the data quality. You will also notice a note on our data products that use 2020 ACS 5-year estimates which reads: “Due to low response rates in the 2020 American Community Survey, there are higher than normal margins of error in some geographies. Interpret all estimates and comparisons with caution and reach out to AskD3 with any questions.”
Do you have any questions about the impact of the pandemic on data or 2020 ACS data that we haven’t answered? You can submit requests for more information on any data topic to us via AskD3.