Confirmation bias in the study of human rights
by Jake Wobig
Jay Ulfelder published a post today about what he called “advocascience” – when political scientists have an ideological agenda and this potentially ends up biasing their results. In a connection that I think is interesting, he analogizes this phenomenon to cases where researchers have financial conflicts of interests with their research – a bioresearcher may skew results if s/he thinks it might be lucrative, a political scientist may skew his/her results if that will support a favored normative position or policy prescription.
Ulfelder says sometimes this skew comes from publishing in a forum that favors a particular position (e.g. the National Endowment for Democracy) or because the scholar may want not want to offend a source or some other person s/he has a relationship with (this episode of RadioLab comes to mind as an illustration of that), but it may also come from the scholar sub-consciously interpreting the results in a way that s/he favors.
This is something that comes up a lot in discussions between grad students and faculty at my program at the University of Nebraska, and I think the reason is because NU is a human rights program, and human rights scholarship is a bit sensitive to the problem because of the findings in Hafner-Burton and Ron’s 2009 paper “Seeing Double: Human Rights Impact Through Qualitative and Quantitative Eyes.” Ungated version here. They find that qualitative research on the efficacy of human rights law is significantly more optimistic about the effectiveness of human rights law than quantitative research. One of the reasons they propose for this is that qualitative researchers sometimes privilege cases where human rights appear to work, whether in case selection or in interpretation of data.
The general consensus that has emerged in our grad student bull sessions is that this is not a problem with qualitative research per se – most of us, including me, use qualitative methods in our research – but that this potential bias is something to be aware of when we do our research.
A person does not start studying human rights unless they want to identify ways to change the world for the better. However, wanting something to be so does not make it so, and we scholars do not do anyone any favors by describing the world incorrectly.