Data Collection in a Pandemic

The COVID-19 pandemic has upended the lives of many Americans. People are limiting their contact to the outside world through fewer trips to work and public places, and many have temporarily relocated. Despite this, the federal government continues to collect important data on the American people.

Government officials, researchers, and private and nonprofit organizations all rely on the Current Population Survey (CPS) to collect key information on social and economic trends, used by the Bureau of Labor Statistics (BLS) to publish a monthly jobs report. Based on these numbers, the nationwide unemployment rate skyrocketed from 3.5% in February to 14.7% by April, before dropping to 13.3% in May. But due to COVID-19 limiting the response rate to the CPS, unemployment numbers from this spring may not be as accurate as those from past monthsand this trend could continue in the foreseeable future.

How the CPS is Carried Out

As part of the Current Population Survey, the US Census Bureau normally talks to members of about 60,000 American households each month. The survey is voluntary, meaning those selected don’t have to take part. Interviews take place during the week that includes the 19th of the month. However, respondents are asked about their employment status from the week prior (the week that includes the 12th of the month)

This means that CPS data isn’t based on data from the whole month, but rather from one week in the middle of the month. This is a particularly important point as it relates to COVID-19. Many people lost their jobs during the second half of March. However, because respondents were asked about the week ending on the 14th, the March jobs report showed far fewer people unemployed than was the case by the end of the month.

Once selected, households become part of the survey for two four-month cycles, with an eight-month gap between both cycles; in other words, people are interviewed during the same four months in back-to-back years. Interviewers talk to respondents in-person during the first months of both cycles, while interviews are held over the phone in all other months.

For more information on how the Current Population Survey is conducted, click here.

How COVID-19 Affected the CPS

Officials started interviewing respondents for the March version of the CPS on March 15. During the first four days of the interviewing period, the US Census Bureau conducted interviews as normal, including phone and in-person interviews.

However, on March 20, the Census Bureau stopped holding interviews in-person due to COVID-19. This meant that people being interviewed during the first months of both cycles, who normally would be speaking to an interviewer in-person, were instead asked to talk over the phone for the last three days of data collection. The Census Bureau also closed two call centers that help with interviewing for the CPS.

These changes, along with potential changes in the priorities of household members due to COVID-19, greatly impacted the response rate to the survey. Based on averages from March 2019 to February 2020, the US Census Bureau normally gets responses from about 82.5% of people. However, only 73% of households in the sample responded in March 2020. That’s a decrease of nearly 10 percentage points.

This drop was especially large among respondents who normally would have been interviewed in-person. In February 2020, people from 80.5% of households asked to be interviewed for the first time took part in the survey; this dropped by well over a quarter, to 56.8%, in March. Second-cycle, first-month members responded to the CPS 80.1% of the time in February, but only 68.6% in March.

The Census Bureau held interviews for the April CPS from the 19th to the 25th. No in-person interviews were held, and the call centers that assist with the CPS were still closed.

The overall response rate fell even lower in April, with 70% of households taking part – down three percentage points from March. More than half of new households included in the survey sample chose not to take part, with the response rate falling to 46.7%. However, the proportion of second-cycle, first month memberswho also would have had in-person interviews in a typical monthdidn’t change from March to April. This points to a possible longer-term impact on response rates not only with cohorts first entering the CPS, but also with those re-entering after eight months away.

There is also evidence that cohorts who missed out on having an in-person interview in March had lower April response rates than those who had in-person interviews in January or February. The response rate among second-month members in the first cycle fell from 74.2% in March to 63.5% in April, while the response rate among second-month members in the second cycle had a smaller drop of 2.5 percentage points.

In May, the overall response rate fell to 67.4% – the lowest rate yet. Among those in the first month of each cycle, the proportion of people taking part in the survey stayed consistent from April to May; the response rate even went up rising among first-cycle, first month-members. But response rates for all other months of both cycles dipped to their lowest points all year. The greatest drops happened with first-cycle cohorts that didn’t have in-person interviews in previous months, with second- and third-month response rates in the first cycle falling between 7 and 8 percentage points.

The Potential Impact of Lower Response Rates

The main takeaway from these numbers is that changing the way a survey is conducted can affect how many people participate. With the CPS, when first-month interviews changed from being held in-person to over the phone, fewer respondents took partnot only in the month the in-person interview was supposed to be held, but in future months as well.

Depending on who does not participate in the interviews, this may not be a problem. Let’s imagine for a second that the group of people who are now excluded from the data because they can’t or won’t do interviews over the phone are the same or similar to the people who are able to do first-month phone interviews; i.e., that both groups are just as likely to be unemployed. If this is the case, the survey results will be the same no matter if interviews are held in-person or over the phone.

However, there’s a good chance that’s not what’s happening here. For those people who have been forced to move during the pandemic, the only way for the Census Bureau to stay in contact with them is through their smartphones, as they won’t have access to their in-home telephone. But according to the Pew Research Center, people without smartphones are more likely to be low-income and less educated. 

If these trends hold within our sample, it’s possible that the people who are getting excluded due to the transition from in-person to phone interviews are more likely to be unemployed or otherwise adversely impacted by the pandemic. This could create a problem called “non-response bias”, where the results we get from our sample don’t reflect the population as a whole. In other words, if people who have lost their jobs due to COVID-19 are underrepresented in the sample, the unemployment rate being reported by the BLS will be lower than the actual unemployment rate.

The potential impact of non-response bias seems to have been largely overlooked in media reports on this spring’s job numbers. If the non-response bias over the past few months is significant and unemployed Americans were less likely to respond to the CPS, then it’s possible we didn’t have such a big increase in unemployment after all. It’s also possible unemployment went up higher than reportedand millions more Americans are out of work than the BLS projects. The agency has already stated that many people were misclassified in the April and May reports, and the unemployment rates from both months would have been far higher than reported without this error.

This is a problem that could carry over to other surveys being done during the pandemic and could impact the accuracy of the decennial US Census. The Census relies on door-to-door enumeration of households that did not submit their Census online, by mail or over the phone. In Detroit, the census response rate is not even 50% yet and the Census Bureau delayed outreach activities in the city until the start of this month. You can see neighborhood-level response rates on our 2020 Census Response Rate interactive map and also discover ways to promote the census in your community on our portal.

Another example of critical data collection in the US is the American Community Survey (ACS), through which the Census Bureau asks about things like education, transportation, and internet access. They send the ACS out to about 3.5 million households every month. We’ve talked about the importance of the ACS in detail; you can read that blog here.

If the COVID-19 pandemic hurts response rates to the ACS and Census in a similar way it has to the CPS, they could both be impacted by non-response bias. This means the survey results could overestimate the average American’s level of education, or overreport how many Americans have access to Broadband internet. Luckily, the ACS and Census have much larger  sample sizes and require people to respond to the survey (unlike the CPS), so it should be more insulated from problems of bias.

What Now?

The lack of in-person surveying during the pandemic could significantly hamper the Census Bureau and other researchers. Face-to-face interviews allow interviewers to get greater insight into the answers respondents give through follow-up questions. Interviewers can also talk to respondents and ensure they all have an equal understanding of the question, so the answers they give are all based on the same interpretation of the question’s wording. 

There are ways the in-person format for surveying could be replicated virtually. Respondents could schedule a time to talk with interviewers using an online video conferencing tool like Zoom, or hold a face-to-face phone call. This isn’t a perfect solution, thoughmany people don’t have smartphones or access to the Internet, and for those who do, they may not be willing to disclose their email address or phone number to hold a virtual in-person interview.

In conclusion, it’s important to keep the context of the pandemic in mind when looking at numbers from the CPS and other surveys that have been impacted. As time passes, the effect of the pandemic will lessen as the Census Bureau and other organizations continue to refine their data collection methods and make adjustments to encourage higher response rates and reduce non-response bias. For now, we should keep the inaccuracies inherent to data collection during this time in mind when evaluating the spring unemployment count for Americans.