James McDonald: Consider a simple pre-test/post-test study looking at the influence of video game
violence upon aggression.
All that's involved is measuring participants' aggression before the study, having them play
a really violent video game for a month, and then measuring their aggression at the end
of the study.
Note that some of our participants in red are already by nature more aggressive individuals.
Looking at change in aggression between beginning and end will help weed out the influence of
that individual difference.
If there's a significant increase in the average aggression scores of our sample, that'd mean
the video game makes people overall more violent, right?
Let's test it out.
Uh oh, what's this?
Some of the participants were so appalled by the violence in the game, they bailed before
finishing the study!
That's what we call attrition.
And what's more, it was predominantly the naturally aggressive participants that stayed
behind.
That's what we call differential attrition, and that's a big problem!
Since the proportion of naturally aggressive participants taking the second test was higher
than on the first test, that likely raised the average aggression score on the second
test.
So how do we know whether the video game had an effect on aggression?
The problem is that we don't.
Differential attrition has confounded our manipulation with the participants' individual
differences.
Our internal validity is ruined!
Let's say we set things up the same as before with a simple change.
We'll use a slightly less violent video game, so maybe it won't drive so many people off.
We've got our sample of participants ready to go through our study, same as before, and
we send them on through.
Fortunately, this time, none of our participants bail from the study.
Attrition isn't a problem this time, but then, near the end of March, something absolutely
terrible happens.
An irresponsible bio-engineering corporation accidentally releases a Tyrannosaurus Rex
into the streets of your city.
The Tyrannosaurus cut a bloody swath through the city eating everyone in sight until it
attacked a turkey farm and subsequently passed out.
Your participants were thankfully unharmed, but seeing their classmates devoured emotionally
scarred them and changed their outlook on life, at least for the time being.
They finish your study in spite of their distress, and you find that aggression scores are much
lower on the second test.
Does that mean that playing a violent video game for a month reduces aggression?
No, not necessarily, because playing the game is confounded with the Tyrannosaurus incident,
so you don't know whether the game actually led to lower levels of aggression.
Your internal validity has just been chewed up and spit out by an extinct apex predator!
Let's say instead of doing our study between March and April of this past year, we had
instead launched our study in 1980.
We get our participants --ten year olds—and have them play a very violent video game for
the next thirty-five years.
Well, okay, since they're ten year olds, we can't have them play a very violent video
game, and because it's 1980, the most appropriate thing we can get is Pac-Man.
The protagonist is a cannibal, though, so that's pretty violent.
We run them through the procedure like before: they take the first aggression test and start
playing the game.
But, instead of testing them after a month of gameplay, we wait longer this time: thirty-five
years.
We're in no rush to get this research published, apparently.
But, a lot can happen in thirty-five years.
Our participants are now forty-five years old, and three and a half decades may have
changed their temperament.
So, we run the participants through the second aggression test, but what's the outcome?
They're less aggressive on test #2 than on test #1, but we don't know if it's because
a life devoted to Pac-Man promotes lower aggression or if it's because the participants mellowed
with age.
So, maturation has also screwed up our internal validity.
Let's consider a different type of study on aggression.
A scale of aggression is simple and quick, but what does it really tell us?
As real-world problems go, are we really worried about video games causing people to respond
to surveys with hostility?
Wouldn't we rather have a measure of aggression that reflects how aggression might actually
be expressed in the real world, how it might be problematic?
Sure!
Well, what can we do then?
We can't measure aggression by handing out clubs and axes and telling people to go to
town on each other, so what about having participants play a violent video game and then have them
use a driving simulator?
We can observe their performance in the driving simulator and evaluate aggression based on
accidents and such.
So, we'll look at the impact that playing violent video games has on people's driving.
We'll see if playing a violent video game makes someone a more aggressive driver.
But, wait.
One of our reviewers for the last study pointed something out.
We're not really concerned with the effects of playing a video game once; we're interested
in how the effects build up over time, so we ought to look more closely at that the
change in participants' performance over time.
That's not too hard; we can just have them take the driving test several times.
So, we'll run our participants through in a six-hour gaming session on a Friday night.
That'll be fun, right?
Every two hours they'll stop playing the game to take the driving test, where we'll track
the number of accidents they have.
So, let's see how that works out for us.
Ah ha!
Look at that!
After six hours of playing a video game, the average number of accidents the participants
had in the driving simulator nearly tripled!
Although, well, six hours is a long time playing that game, especially at the end of the school
week.
I mean, I can do six hours of hard gaming without a problem, and I'm pretty old, but
I wouldn't necessarily want to do much driving after a night of gaming.
So, maybe we should spread things out a bit more.
So, minor change: instead of a single six-hour-long gaming session, we spread our study out over
four nights, with participants playing for two hours on three nights.
That should be perfect, right?
Let's try it out.
Oh, well, the number of simulated wrecks participants experienced went up after playing the game
the first time, but then it went down.
Could that mean that violent games make people better drivers if they play them long enough?
Well, that's one possibility, yeah.
But, it's also likely that using the driving simulator every night simply gave participants
the opportunity to become proficient with the simulator.
So, in summary, when you use repeated testing to study changes over time, always remember
people change.
So, when you see changes in participants during your study, always ask yourself what else
could be causing changes in these people between my measurements?
It's possible that your sample is changing because people are dropping out of your study.
Or, could some major historical event have intervened?
Is it possible that due to their participation in your study, the participants are starting
to become fatigued?
Or, like artists, are they becoming better with practice?
Or, is it possible that your participants have simply matured with time?