Tuesday, November 30, 2004

Methodology is Everything

I was somewhat excited when I heard on CNN about a study that purported to show differences in MRI brainscans between liars and truthtellers. Jim Lundgren at The Volokh Conspiracy dug a little deeper into the study and found some serious flaws. From the section quoted in his article:

Faro and colleagues tested 10 volunteers. Six of them were asked to shoot a toy gun and then lie and say they didn't do it. Three others who watched told the truth about what happened. One volunteer dropped out of the study.

Aside from the sample size, the issue Mr. Lundgren makes is that there are more than one variable in this arrangement. Those who are lying are also the ones who performed physical action, two areas where there is a difference from the other group. Therefore, it is impossible to say whether it was the lying or the memory of taking physical action that activates the additional brain regions.

A better set up for the experiment would be to have to groups, a group that is set before hand to lie to a particular question and one that will answer all questions honestly. While in the brain scan, each group will get the same list of questions. While answer all of the others truthfully, the first group will lie when given the key question. That way, comparisons can be made between responses in an unreliable individual's telling lies and truth, and between the unreliable person and the reliable one. In fact, you could have a third group instructed to lie all of the time. However it is done, the experiences of the subject will be functionally identical with respect to brain activity.

Update: Thanks to John of Boffoblog for bringing my attention to the more in depth look the study he took in a couple of posts. Looking at his posts, it is clear that it was the reporting on the subject that was shoddy rather than the study. One of the big issues I have is scientific illiteracy. Reporting like this tends to push the sensational, and the idea of a fool-proof lie detector is gold. It does not help the general issue of people uncertain about the real science. With reporting like this, it becomes all to easy for snake oil to sell and real medicines to be demonized.

No comments: