September 18, 2009

Methodology Matters

Via Barbara Drescher comes a lovely analysis behind some rather sensational reporting:

Technology addiction among young people is having a disruptive effect on their learning, researchers have warned.

Their report concluded that modern gadgets worsened pupils' spelling and concentration, encouraged plagiarism and disrupted lessons.

Luckily for those of us who might have some contact with teenagers, education or technology (how many of us is that again?), Anne Marie of Wishful Thinking in Medical Education sprang for the report behind the press release.

With regards to 'tech addiction' this seems to have been a self-assessment based on response to the question: How addicted are you to the internet or your mobile phone? The proportions given in the BBC report are those who stated they were 'quite' or 'very' addicted. Of course, we don't know what the students meant by 'addicted'.

With regards to this addiction harming learning, there is no analysis relating the perception of being addicted to outcomes in learning. In fact very few of the questions are related in any way to learning.

The whole thing is worth a read, as is a comparison between the BBC article and the blog post. It highlights the way that several red flags regarding the research are reported in the article without being emphasized or interpreted in any way for readers who don't already know that these are flags.


Anne Marie said...

Thanks for the post. Really glad you enjoyed. It's good that through blogging we have the opportunity to expose and reply to this kind of sloppy journalism.

Stephanie Zvan said...

It is indeed. And thank you for going that extra step to really dig into this. I'd seen some other studies reporting similar things that were just about as bad, but I hadn't seen this one yet.

Now, if we can only fight the tendency toward, "Oh, maybe this study isn't any good, but it's saying the same thing as all these other [lousy and poorly reported] stories, so there must be something to it." :p