At D3, we spend a lot of time thinking about making data more useful to people who are making decisions, and sometimes we re-discover just how challenging that can be when we want to use data to make a decision in our own lives.
Making a choice about where to live, especially when you’re a parent or thinking of becoming a parent someday, can be really complicated. Aside from factoring what kind of house, what size, what type of neighborhood, or where in a region you want to be, you also need to think about what the schools are like.
Our colleague Jeff is thinking about moving to a new house. He also has a one-year-old (pictured above). So he’s been spending some time during the search process trying to fact check his anecdotal evidence (amassed over a few decades of living in southeast Michigan) about which school districts have the highest-performing schools. Despite our experience with the difficulties of education data in Michigan, we were surprised that this was much harder than any of us suspected.
Starting at Google we found MI School Data which seemed like a great place to start.
This is where we learned we were in the wrong place. There was a disclaimer next to the map that says:
We went back to MI School Data and tried the “K-12th Grade” menu.
We looked into the School Index, the Postsecondary Outcomes by High School, Our Schools – At A Glance, and the Dashboard & Accountability Scorecard, none of which helped us in our quest. It was then that we recalled from years past a ranking of Michigan schools produced by the state. Since we couldn’t find it in the site, we went back to Google and found it right away (“Top-to-Bottom School Rankings”). By the way, this adventure took us to the Michigan Department of Education website, which is a distinctly different website from MI School Data, which is produced by the Center for Educational Performance and Information.
This was where the real fun for D3 nerds began. We downloaded the “2015-16 Top-to-Bottom Individual School Rankings Lookup Tool” (an Excel file) and opened it up. The default view opened to the “Data Dictionary”, which was pretty intense.
We didn’t understand what matters in this long list that required a lot of Googling to comprehend undefined acronyms like “fay” and “SGP”. For example, here’s a standout definition of one variable out of the 101 listed: “Z-score of best cohort graduation rate from best of 4, 5, or 6 year cohorts or best single year rate of 4, 5, or 6 year cohorts if two year average cohort rates are not available”.
He was not impressed.
We then checked the “Directions for Use” tab.
Interestingly, there was some information we only just discovered when taking the screenshots for this post hiding in cell A2.
There were so many words, and again, this tool offered a single school lookup which didn’t help us figure out any quality comparisons. Luckily, we’re data people, so we moved to the “Data” tab.
It was a little overwhelming, but what can you expect after seeing that data dictionary? We saw a variable earlier that seemed simple enough to let us get a small leg up in terms of comparing quality – “school ranked in the top 5% of the overall school rankings” – and we filtered the table to show just those. Then we narrowed the list down further by filtering for the two Intermediate School Districts where Jeff is looking at houses. This helped us get to a more manageable list of about 50 schools. Some of these schools were surprising, but as we looked at the details we were more intrigued by the schools that were missing from the list. We had to remind ourselves that these were the top 5% of schools in the whole state regardless of grade level and thus excluded some top 5% high schools for elementary/middle schools.
We then unfiltered the data and looked at the other interesting variable – “overall school percentile rank” – and sorted the spreadsheet from highest to lowest percentile. Again, we could’ve looked at the names of the schools and gotten a general sense of how they compare to one another overall, but what would this really mean?
Having gone through all of this, we still didn’t have an answer. We weren’t even sure we were using the right variables to make choices. There’s an overwhelming amount of data, even for a bunch of data geeks familiar with education data.
There are a few takeaways for D3 that we wanted to share, especially as we continue to build tools like State of the Detroit Child for parents and communities to use to answer their own questions.
Design with the user in mind. This means that you have to include the information the user needs and consider how they need to see it. Most parents don’t need 101 data points to make a decision about school quality. Unfortunately, this gigantic dataset was the most direct way we could find to get scores on all the schools that Jeff would possibly be interested in.
Don’t use jargon. This flows from the first point, but also means when you promise someone a dashboard, make sure the page is actually a dashboard. It also means limiting the use of acronyms (like “fay”) when possible and providing clear definitions when it’s not possible.
Package all the relevant information into a format that’s easy to use and interpret. Many parents wouldn’t be familiar with formatting an Excel sheet to dig into the data about schools in their particular area. Data is already hard to understand as professionals. We should make it easy for people who are trying to make important decisions in their lives to access and use data points effectively.
Trying to find this information for a real world problem reminded us that we’re not invincible when it comes to bad data design. We continue to try to understand and interpret data to make it more accessible and useful to the public, while keeping in mind the different barriers that our users might face.