June 17, 2011

With Friends Like These

Times Higher Education has just posted a rather amusing defense of evolutionary psychologist Satoshi Kanazawa. If you managed to miss the poorly analyzed Psychology Today blog post he wrote that put him in a defensive position, I recommend you catch up here before reading the letter.

All set now? Good.

We believe the recent criticisms of Satoshi Kanazawa's work cannot be justified ("Damage limitation: evolutionary psychologists turn on controversial peer", 2 June). Contrary to the assertion that Kanazawa does poor work, he has published 70 articles in peer-reviewed journals in the fields of psychology, sociology, political science, biology and medicine. These are listed on his London School of Economics web page and many of them have been published in top high-impact journals.

I'll let someone from Retraction Watch weigh in on how well peer review guarantees that poor work is never published.

Kanazawa's publications are listed here. I note that whatever "top high-impact journals" Kanazawa has published in, he's also published in Intelligence, which still (unironically) prints papers treating IQ testing as a valid measure for cross-cultural intelligence comparisons. Someone for whom impact factor is a big deal will have to do the research on whether the letter writers are correct, but I would love to see the results.

Why? Because there are a number of fairly staid topics and treatments among Kanazawa's publications. It wouldn't be the first time I'd seen that kind of work used to put someone's name in the "right" places while the iffy political pieces went elsewhere. In fact, Pharyngula had a post up yesterday documenting that kind of behavior in a geologist. If anyone matches articles to impact factor, please let me know.

The critics assert that many of these papers are "bad science" and have been published only as a result of a faulty peer-review process. This cannot be accepted. The editors of journals send the papers submitted to them to reviewers with expertise in the fields in question and publish only those that are deemed to be sound. Thus, all of Kanazawa's papers have been judged as sound by competent reviewers. Others may disagree, and in the case of innovative papers of the kind Kanazawa writes, frequently do. Time eventually tells whether the authors or their detractors are right.

This is just silly. Bad science gets through peer review, even when one's peers don't have the same political bent you do. On the day this letter appeared, Times Higher Education also ran an article about a mathematics journal withdrawing a paper written by a proponent of intelligent design that claimed to disprove the second law of thermodynamics. The editor apologized for even considering it, but the article had passed peer review.

The critics complain that when Kanazawa has a paper rejected by one journal, he sends it to another and publishes it there. Who among the academy's members has not done that? Reviewers frequently misjudge a paper and editors accept their recommendations. The author then sends it elsewhere and it is accepted. If there were anything wrong with this practice, then, as the first online comment under "Damage limitation" puts it: "A few Nobel prizes will have to be returned."

The detractors assert that Kanazawa rarely responds to brickbats. On the contrary, we believe that while he sometimes does not respond immediately, he frequently deals with criticisms in his subsequent work.

Actually, the objection was not that Kanazawa submitted papers after they were rejected. The section in question:

The peer review process is not perfect and appears to have failed when dealing with Kanazawa's poor quality work. Those of us who have reviewed his papers have had experiences where we have rejected papers of his for certain journals on scientific grounds, only to see the papers appear virtually unaltered in print in other journals, despite the detailed critiques of the papers given to Kanazawa by the reviewers and editors of the journals that rejected his papers.

Thus, not only is Kanazawa's work an example of poor science on theoretical and methodological grounds in our view, but we also believe it violates the central purpose of scientific discourse, because he rarely engages with his scientific critics. He rarely considers the criticisms of his work that have been published as well as those given to him during the peer review process: to our knowledge he has published counter-responses on only two occasions to critiques of his work (separate responses to two critiques of a paper published in 2001; and a response to one critique of a paper published in 2002). Since then, he has not published a full length response in the academic literature to any of the numerous critiques which have been published against his work, nor has he published corrections to the papers for which doubt has been cast on the conclusions.

There are legitimate discussions to be had on the role of peer-review feedback in shaping the final published product. However, having that discussion and recasting a complaint about Kanazawa's resistance to incorporating feedback are two very different things. Also, given what the criticism of Kanazawa actually was (that he doesn't interact with feedback prior to publication) it seems a little odd to note that he incorporates feedback into later work. If the criticism is important enough to be dealt with, wouldn't he produce stronger papers by dealing with it up front?

But back to the letter. There are a few short paragraphs providing information about two times Kanazawa later responded to criticism, followed by this closing:

Finally, we believe that the proper place to make criticisms of academic papers is in the journals in which they were published, not in letters to the press where they cannot be adequately answered.

This--this!--is what makes this letter so entertaining. Even forgetting that Kanazawa brought himself and his work into the general public eye by writing a blog post about his "findings," this is the richest vein of irony I've mined in some time. You see, while the idea that scientific ideas and their validity should be hashed out in journals is relatively common among scientists, it's pretty rare among the signatories to this letter.


With the exception of Lynn, who cowrote the book with Vanhanen, that's just one example per signatory for those who were easy to find in a very quick Google search. If there is one thing this group is not, that would be in favor of keeping science discussions contained in journals. The fact that they want everything contained and compartmentalized in this case makes a far stronger argument than anything in Kanazawa's CV that, at least in this field or subfield, there may be some serious problems with peer review.

But it was terribly sweet of them to write a letter and make it obvious.

6 comments:

Eric Crampton said...

I'm totally in favour of blogging, obviously. I just thought it awful that the Times letter writers levied their critique of his academic work in the Times just when popular calls for his ouster (based on his blogging) hit their peak.

I won't defend every method in every one of Satoshi's papers as I've not read them in sufficient depth. And he could have done himself some favours by having written rejoinders rather than incorporating comments into later work. But I can't see that anything he's done justifies the treatment he's since received.

Stephanie Zvan said...

Thanks for the comment, Eric.

If Kanazawa's problematic blogging weren't based on his problematic research, there might be a reason to separate criticisms of the two. However, what I've seen is that his blogging raised the profile of his research to the point where others looked at his methods and analysis. They found neither held up in this particular case.

At that point, I can't see what it is except completely appropriate to note that similar flaws have existed and been pointed out to Kanazawa for years.

Eric Crampton said...

I'd thought that Gelman's academic critique preceded Satoshi's blogging, but I could have that wrong.

Becca said...

Not that it matters, but I would not rate any of those journals as in the field of 'medicine'. And there's only 40 articles in peer-reviewed journals, although several more book chapters.
So the defense of Kanazawa isn't very concerned with literal truth. Much like Kanazawa himself?

That said, I don't see anything that violates actual scientific protocol about how he responds to criticism or anything of that nature. It's not ok to discredit him as a total hack without discrediting several fields of interesting, if controversial and likely inconclusive (at this stage) research. I see why people might have defended him on those grounds.

Ultimately it sounds to me more like Kanazawa is just a compulsive shit-stirrer who then doesn't even have the decency to discuss the science. It's poor form. It's trollish. But it's in a completely different category of problem than any scientific errors.

Temaharay said...

I'd add Henry Harpending to your list of academic signatories who routinely step out of the sphere of academic journals and into the public eye through conferences, interviews, and popular literature/web.

Stephanie Zvan said...

Damn it. I meant to make a special place for Henry, but I forgot because I never opened a tab for it. He even has a history of being provocative en blogge, then deciding that criticism for it should only come at him in particular ways: http://scienceblogs.com/gregladen/2010/01/average_brain_size_for_the_thr.php#comment-2189628