When Data Analysis Doesn’t Tell What You Want to Hear

These days, every organization wants to be data driven—especially education organizations.

There’s a problem, though. Sometimes data doesn’t tell us what we want to hear.

If we want to brag about our good data, we absolutely have to deal with the bad data. I don’t mean messy data, inaccurate data, data that was collected poorly. I’m referring to data that suggests things—findings, outcomes, trends, whatever—that don’t align with what we think should be happening.

Over the last decade or so, I’ve spent a fair amount of time collecting and analyzing education-related data.

I’ve projected anticipated results from students taking standardized tests, measured how reliably district-created assessments assess student knowledge and skills, and evaluated the effectiveness of common academic programs.

Reporting on data analyses in education is not always fun. For instance, imagine the feeling of presenting a superintendent with the number of students you expect to NOT pass a standardized test.

Too often—even with trusted colleagues and friends—data that doesn’t match expectations and desires is cast off as invalid.

In one situation that I’ll never forget, a projection seemed so dismal that the folks I presented it to completely refused to accept it.

I get it. No one wants to acknowledge that things aren’t what they’d hoped.

However, we can’t simply ignore the data that we don’t like. If fact, quite the opposite: sometimes it can be argued that the “bad news data” should be exactly what we seek out to learn, grow, and affect change. 

Allowing the Data to Drive the Work

Over the last two decades, students in Chicago Public Schools (CPS) have increased learning at a faster rate than 96 percent of all school districts in the country, including those with more resources. How? Janice Jackson, CEO of CPS, claims that one of the reasons is the responsible use of data. CPS uses a third-party researcher and are willing to use the “bad” data to drive their work.

Instead of ignoring statistical models and analyses that indicate things aren’t looking good, let’s be accountable and commit to building on that data analysis to proactively plan and adjust our courses of action.

Clearly, the education system asks a lot of our teachers. By providing the data they need—both the good and not so good—teachers  can more realistically plan for how to reach more students. Simply put, if we want our work to be truly data driven, then we must allow the data to drive the decisions we make.

Understanding Why Data Analysis Matters

I can’t help but wonder if part of the reluctance to accept unfavorable or “bad” data is based in the math we’ve been teaching in the U.S. for the last century.

The worlds of math education and statistics have been disconnected for too long. Some people may even go so far as to argue that statistics is not rigorous math.

More and more, though, higher education systems understand the need to provide multiple math pathways for students that align with students’ intended programs of study. Higher ed institutions are increasingly seeing statistics as a valid pathway for students interested in careers in journalism, marketing, nursing, and more. They’re also using statistics as a way to replace the old model of remedial math, which has been a major barrier to students’ degree completion.

Today’s K–12 students also have an opportunity to learn more statistical concepts than in the past. In 2012, Texas adopted standards for Advanced Quantitative Reasoning (AQR), which emphasizes learning around probabilistic and statistical reasoning. Three years later, the state implemented a full Statistics course, which can be taken in place of a legacy course like Algebra II. California allows students to take Statistics and Probability as an alternative to Precalculus for a fourth-year math course. This trend is increasingly promising, especially as states are exploring options for aligning what students learn in their K-12 systems with what they need to know to succeed in higher ed.

The Dana Center’s Advanced Mathematical Decision Making, which is aligned to the Texas AQR standards, and Transition to College Mathematics  courses emphasize key statistical concepts that ground students in the skills they need to make the same projections, measurements, and evaluations that I have made for education systems.

In this age of “fake news,” our students now, more than ever, need to be able to analyze data with rigor and accuracy—not to mention eventually needing this type of understanding in numerous career paths. If we continue to embrace statistics in our higher ed and K–12 systems, I have complete confidence that there will be more willingness to accept what the data tell us. Especially when we don’t like it.