Tuesday, March 1, 2011

Why Data-Informed Trumps Data-Driven

Plot, characters, and setting, the three primary qualities of a story; but imagine a story with only a setting. Well, you no longer have a story, we call that a painting. Data-driven decision-making runs the risk of turning one element of effective education into the primary element and in the process, turning beautiful stories of educational success into static pictures of moments in time.

Data is the setting in our story of education. Within this setting, the characters (students, teachers, parents, etc.) create the plot in their daily interactions. To call for an end to “data-driven decision-making” is sure to raise a few eyebrows. Data is the constant, the solid ground; data takes the guesswork out of what we do. However, focusing on the data to the exclusion of all other elements of our story does not advance the cause of effective education for our students.

That is why I believe in “data-informed decision-making.” Ignoring data would be foolish, but data alone does not tell a story. I am reminded of a quote from Jack Nicholson’s character, Melvin Udell, in the movie “As Good As It Gets.” While discussing his relationship problems with an acquaintance, Udell interrupts—“I’m drowning here, and you’re describing the water.” For the majority of the last two decades, the drive for greater accountability through standardized testing has done a super job of describing the water without offering any tangible ways to provide a lifeboat.

“Data-driven decision-making” has become a tool for power. Politicians use data-driven to point out the flaws in education instead of directing efforts toward areas needing improvement. Edupreneurs use data-driven to sell the next great solution to our education woes. Too often, the data drives decisions that have already been made.

This is how data-informed works:

1) All data is considered- student attendance, variables in the life of the teacher and/or students affecting outcomes, problems with the assessment, instructional methods, past performance of teacher and student.  In other words, a data set should not be analyzed relying on numbers alone. Testing does not happen in controlled settings, therefore we should not mistakenly treat results as dependent variables.

2) Instructional decisions are made in response to meaningful conclusions drawn from the data- just like a few high performers do not indicate good instruction, a few low performers do not indicate poor instruction. A few years ago our high school was on a fixed schedule, and every Friday afternoon I taught the same class of seniors ready to start the weekend.  On top of that, our school gave seniors a ten minute early release every Friday.  They also found less success than their peers on the AP exam.  Data-driven decision-making would increase my anxiety by trying to figure out where I was going wrong with these students.  Data-informed allows us to look at the big picture and balance the benefits of a senior release with student achievement.  Before anxiety leads to poor decisions, all of the potential reasons for low achievement need to be addressed.

3) Teacher’s need to honestly evaluate the data they collect- we cannot, nor should we avoid accountability. The response of “I just know” to the question of how much your students are learning is not good enough. I continually make an effort to compare and evaluate how well my grading structure compares to student performance on end-of-year testing. I’m not satisfied with where I am, but I have used the data collected to inform decisions about the changes made over the years.

4) Data does not need to be punitive to be informative- Dan Pink gives an excellent summary of the 21st century understanding of motivational theory.  The use of standardized student assessment data to evaluate the performance of schools and tying these results to consequences run counter to this theory. Further, the national movement toward “value-added” or “growth models” which tie these results to individual teachers further undermines the quality of education. This point could stand further explanation, but in the interest of brevity, I will save that explanation for a later time.

“Data-informed decision-making” is a way of holistically approaching education.  We realize that students are people and not data points.  Unlike the pigeon in a Skinner box, the tools to isolate specific behaviors and tie them to discrete consequences are not at our disposal. Until that becomes possible (and I'm not even sure it would be desireable) we must continue to rely on human judgment, reason, and ultimately decision-making.

If nothing else, the paradigm of data-informed instills the idea of collaboratively looking at the facts and deciding the best course of action for moving forward as opposed to the data-driven paradigm of top-down one-size-fits-all approaches to education reform.


  1. Talking about data sometimes get clouded in the broad landscape of its use. Let's focus in more closely on one simple yet clear example. We as teachers make recommendation for our student's courses for the upcoming year. Data helps inform us when making these decisions. We consider it and factor it in. Sometimes it is consistent with our judgment and other times it shows a disjoint between things. What would be the result if we did this in a data driven way, only looking at scores on standardized tests? Obviously we also have to use our professional judgment, for better or worse, when making such a decision. That is an imperfect process but using data can help.

  2. Jumping off your Melvin Udall quote (great film), I'm reminded of a couple of Sherlock Holmes quotes when reading your post:

    "It is a capital mistake to theorize before one has data. Insensibly, one begins to twist facts to suit theories, instead of theories to suit facts."

    Like you have already said, the important thing is not to have our minds made up before looking at 'the facts,' AKA 'the data,' AKA information. Your blog brought this to mind: "to decide" is an action, an action not based on facts but on theories. In other words, we don't necessarily do things because of what we see, but rather as a response to the perceived reasons for what we see. We act in response to theories; as a result of action, we see what other facts emerge, which may help us adjust the theories on which we act in the future. (And the cycle continues.)

    "Data, data, data! I cannot make bricks without clay!"

    The decision to act may be the most important thing that each of us does on a daily basis- the "data" that we use to arrive at these decisions (and how we use that data to inform) must be a topic of continuous conversation.

    Thanks for the post sharing your thought process- I've only recently run across your blog, but I really enjoy it.

  3. Thanks for the comment. Oddly enough, I've only recently re-discovered the joy of Sherlock... I downloaded the complete works for free on Kindle.

    I enjoyed the drawings on your blog-- I've heard some people say that if you can't communicate it in writing on a napkin it's not concise enough to communicate. These diagrams speak volumes.