How Big is Your Effect Size?

Dr. Strangelove: Or How I Learned to Stop Worrying and Love Effect Size

I like that title better.

I’ve been intrigued by the concept of effect size for several years.  I am not a quantitative person, but I’m curious. I try to keep an open mind, but I still can’t shake a lack of faith in numbers.  I try to believe, and sometimes a good quantitative person can move me in their direction just a bit, but I’m still a qualitative guy at heart.

Two weeks ago, our school division hosted its annual “Making Connections” conference and Dr. Matt Haas, assistant Superintendent offered a session titled “You Can Calculate Effect Size.”  The fact that many teachers lack basic literacy in research and statistical methods is a detriment to our profession.  First, we fail to apply the results of research in the classroom and second, we fail to adequately participate in the conversations around educational research that drives decisions in our divisions, states, and nation.

In a perfect world, education research would be carefully vetted and practitioners could refer to current research from time to time in order to refine their skills.  In the world as it is, research on education is often agenda-driven and practitioners too often fall prey to ideas that merely sound good. (Anyone still encouraging students to discover their Learning Styles?)

In the world of the classroom, it would do teachers well to understand at least a little of the methods and language that researchers are using to influence the national conversation on education.  Influence that affects universally, such as the movement to use value-added measurements to teacher evaluations. And, influence that affects the classroom in the form of instructional methods teachers are expected to use.

In the absence of any “authoritative body” to filter and condense the growing body of educational research into something productive for American education, teachers need to develop a better understanding for themselves of how to interpret research.

Ready for your first lesson.

Effect Size= “Mean of Data Set Two minus Mean of Data Set One divided by Standard Deviation of Data Set One.” 

If you give a pre-test and a post-test, data set one is the pre-test.  Data set two is the post-test. Sometime between pre-test and post-test you “apply a treatment.” In the case of education, an instructional strategy.  The effect size measures how much difference the treatment made.

If like me, you’re not a number/stats person it’s easy to stop here and pretend that it’s too confusing to waste your time on.  This is too important for that, if you didn’t get it read it again.  An effect size should tell you how much an instructional strategy facilitated or inhibited student growth.  Yes growth (or value-added if you’d rather.)

Take this tool for what it’s worth.  It’s the primary tool used by researchers such as Marzano and other education “meta-analysts” to determine instructional methods that work, the techniques have the greatest effect size on student achievement.

Still, the greatest power in using effect size is informative, not prescriptive. For example, Marzano’s well-known book “Classroom Instruction that Works” presents strategies that have proven, through meta-analysis, to have a higher than average effect size on student learning.  He does not imply (and even directly states otherwise) that a given strategy WILL work on every student in every situation.

That's likely the greatest flaw with this type of research. What should be informative for our educational practices becomes prescriptive through policies and evaluative methods. I imagine that across the country, more than a few teachers have been evaluated on consistently applying the “strategies that work” without regard to immediate evidence of whether the strategies are working or not, leaving them skeptical and critical of the entire body of work that attempts to isolate the most effective classroom strategies.

This is why all of us, from classroom teachers to legislators enacting policy, should have a better understanding of educational research.

The Fallacy of Average Class Size

The average person has one ovary and one testicle! 

If you think that’s ridiculous, then you’ll understand the folly of using average class size data in educational decisions. Statements that are mathematically correct can still be blantantly wrong.

Averages are attractive. In uncertain situations they provide a concrete anchor for understanding our world.

Data from the 2000 United States census indicate that the average household size in the United States is 2.59. Does that describe your family?

I didn’t think so, .59 of a person doesn’t exist.

You would call me a fool if I believed everyone has one ovary and one testicle, or even if I spent my days looking for the extra .59 person that should be living in the house next door. But somehow, when the average looks like something that supports our agenda, it becomes a valid measure of reality.  Kind of like average class size data.

Modern educators are familiar with the “power of zero” discussion. It goes like this: if a student has scores of 100, 100, 100, 100, and 0, they have an average of 80; a ‘B-’ for most, a ‘C’ for some. According to the argument, an 80 does not reflect the achievement of the student, the zero has an undue effect on the “average.” A move to standards-based grading indicates a desire to measure a child’s true achievement that can’t be measured with an average.  So averages aren't good indicators of student acheivement, but it's o.k. to use them as an indicator of how well a system is staffing.

Let’s assume that a school has ten teachers. Four of the teachers have a low class size of say twelve students. Perhaps they teach students who need more support, or they have a class that just met the minimum number for a section. The remaining six teachers each have classes of twenty-nine students. That school has an average class size of 22.2.

What if we looked at a different set of statistics? At this school with an average class size of 22.2, seventy-eight percent of the students are in classes with twenty-eight other students. Sixty percent of the teachers have classes of twenty-nine students. The average class size of 22.2 doesn’t look quite as successful.

What if this small model school were a high school? Each teacher has six classes. We would find six teachers at this school with a load of one-hundred seventy-four students and four teachers with a load of seventy-two student. The average teacher would have one-hundred thirty-three students, but in reality, sixty percent of the staff is teaching one-hundred and seventy-four students. Seventy-eight percent of the students at this school are 1 out of 174 to all of their teachers.

Honestly, we know better. Averages mean very little when divorced from their source, yet we continue to let them drive and/or support our positions. No amount of compiled information can substitute for looking closely at it’s individual parts and an uncritical acceptance of data is a recipe for poor decision-making.

Recently, I presented the following problem to my students:

Three truck drivers went to a hotel. The clerk told them that a room for three would cost $30. Each driver gave the clerk $10, and went to their room. After checking registrations the manager realized he had over-charged the drivers. The cost of the room should have been $25, so the manager gave the clerk $5 and told him to return the difference to the three drivers. On the way to the room, the clerk decided that since the drivers did not know they had been overcharged, he would return $1 to each of them and keep $2 for himself. Now each driver had paid $9 for the room and the clerk kept $2. 9 times three is 27 plus the 2 kept by the clerk totals 29. Where did the extra dollar go?

Just because the data are accurate and the numbers add up doesn't mean they reflect reality.  Sometimes we need to get out of the statistics and into people to find the answers.

Have you're experiences ever been misrepresented by "the average"?

The Greatest Post "N"ever Written

The well publicized rapture came and went on Saturday with no apparent impact.  Or maybe I and everyone I know are bad people.  But wait just a minute, Saturday was not without incident.   My most recent blog post which I sat editing on Saturday at 12:10 EST is missing.  I hit CTRL-Z to undo a recent edit and whoosh...flash of light...tingly feeling...sounds of bells...and I'm left with an empty dialogue box.  Before I could do anything helpful google slaps an autosave on me and....the post is no longer among us.  Perhaps this post was too perfect.  It did after all warrant a compliment from an award winning author after a cursory review. Sadly I cannot recall the majority of content and since I can't remember exactly what it said, it's as if it never existed. Weird Huh?

Thus I have decided to try and share the "abridged" version of my thoughts...at least the parts that are still floating around in my head:

Schools are Under Attack-When I take in all that is happening in reform efforts I am left with one conclusion.  There is a deliberate effort to undermine and even destroy our public school system.  At the least there is an effort to stop funding it, ignoring all the untold good that has been done in our nation's history as a result of free public schools.  I am not a truther or conspiracist...just a teacher who is careful with his observations.

High Stakes Testing is Bad- 10 years worth and nothing really to show for it.  Data can be misleading if interpreted for a purpose.  This testing changes the focus of schools to something that is not good(see previous post). Watching your kids take such a test makes you hate the psychometricians...yeah that's a real job and they are real people and they do a "real" job.  Not sure we need them but they exist.  
Vouchers aren't all bad, but they aren't all THAT either- It is not fair private and charter schools can turn students away.  We as public schools can't and shouldn't. No wonder they sound better compared to public schools.  Also taking PUBLIC money away from schools filled with kids that really need it is a BAD idea. It could potentially undermine one of the strongest social institutions in this nations history and much of the good they've done.

Companies are slowly getting more influence and thus control over our schools.  This is bad since companies exist to make profits.(in that sense at least they are  not like teachers).  When faced with doing what is right with doing what they want...its about the profits.  A colleague shared this recent article which illustrates the problem....$266, 000?...that was their offer for a screw up.  Despite the absence of tenure I suspect no one at Pearson was labeled "bad" and blamed for everything.  But mistakes happen.  Get over it and move on.  Who holds the accountability machine accountable? 
http://www.edweek.org/dd/articles/2011/05/20/430371wypwsproblems_ap.html

People in charge of education reform are certain they are right.  Hey, what if they are not?  I'm pretty sure every teacher I know think they are NOT.  Makes you think huh?  A recent blog post described the nation's chief reformer as "condescending, arrogant, insulting, misleading, patronizing, egotistic, supercilious, haughty, insolent, peremptory, cavalier, imperious, conceited, contemptuous, pompous, audacious, brazen, insincere, superficial, contrived, garish, hollow, pedantic, shallow, swindling, boorish, predictable, duplicitous, pitchy, obtuse, banal, scheming, hackneyed, and quotidian."  I think he even de-friended him on facebook.


I remember linking to this cartoon in the post that was raptured:

But there was a ton more...mostly from those who support them.  And I think for misguided reasons.
http://www.google.com/search?um=1&hl=en&rls=com.microsoft%3Aen-US&rlz=1I7ADRA_en&biw=1088&bih=620&tbm=isch&sa=1&q=school+voucher+cartoon&aq=f&aqi=&aql=&oq=

The abridged version might be like watching a movie trailer where you get the idea but not the substance.  For this I apologize.  Blame Harold Camping.

Data-Driven Decision-Making Kills Crickets!

“Diana Virgo, a math teacher at the Loudoun Academy of Science in Virginia, gives students a more real-world experience with functions. She brings in a bunch of chirping crickets into the classroom and poses a question:” So begins a story related in the book “Made to Stick” by Chip and Dan Heath. They applaud the teacher for providing a concrete lesson to understand the notion of a mathematical “function.”

I learned a different lesson altogether from this story. After gathering all the data relating to chirp rates and temperature, the students plug the information into a software package and--- AHA! The hotter it is, the faster crickets chirp and even better, IT’S PREDICTABLE! Now students have a concrete example of what a function is and what it does. Next comes the point where the story grabs me. The Heath brothers mention (in parenthesis no less, even calling it a side note, as if this isn’t the main point) that “Virgo also warns her students that human judgment is always indispensible.” For example, if you plug the temperature 1000 degrees into the function, you will discover that crickets chirp really fast when it is that hot.

The moral of the story is this: Data-driven decision-making kills crickets!

Unfortunately, it can also kill good instruction. Recently while attending a district-wide work-session on Professional Learning Communities, a nationally recognized consultant suggested reasons why teachers at a small middle school without colleagues in the same subject should collaborate with teachers from other schools in the same subject. He suggested that when these inter-school teams see that one teacher has better data in a given area, the others could learn what that teacher is doing to get such good results.

I’m not against this type of collaboration, but could it be possible that a teacher from one school whose student testing results (data) are not so good is still better than a teacher in a different school with excellent data? For example, might the data at school A look better than school B because students are getting better support at home. Perhaps school B spends more time making sure students are fed and clothed before concentrating on the job of instruction. What if school A has stronger leadership and teacher performance reflects teacher moral, support, or professional development?

Teachers must collaborate and share stories relating to instruction that works, but if student test-data is the only metric used to evaluate effectiveness we are essentially determining that crickets chirp very fast at 1000 degrees. There is a better choice than “data-driven.” Next week I’ll share my thoughts on this alternative and together we can strive to “save the crickets.”

Follow-up Post: Why Data-Informed Trumps Data-Driven