Evaluation of scientific rigor in animal research

Faculty of the University of Bern, who conducted two studies on animal experimentation in Switzerland. In a first step, they screened all 1’277 applications for animal experiments approved in 2008, 2010 und 2012, as well as a random sample of 50 scientific publications resulting from studies described in these applications. These were scored for explicit evidence of the use of seven basic measures against risks of bias (including randomization, blinding, and sample size calculation). «Use of these measures is a prerequisite for unbiased, scientifically valid results», says Prof. Hanno Würbel, director of the Division of Animal Welfare.

Explicit evidence of the use of measures against risks of bias was scarce both in applications and publications. Thus, fewer than 20 percent of applications and publications mentioned whether a sample size calculation had been performed (8 percent in applications, 0 percent in publications), whether the animals had been assigned randomly to treatment groups (13 percent in applications, 17 percent in publications), and whether outcome assessment had been conducted blind to treatment (3 percent in applications, 11 percent in publications).

Are measures against risks of bias not used?

Does this mean that in more than 80 percent of the experiments measures against risks of bias were not used, and that animals used in procedures were used for inconclusive research? „No“, says study director Hanno Würbel. „It is possible that the researchers did use these measures but did not mention them in their applications and publications. So we decided to ask the researchers.”

Lack of awareness and insufficient knowledge

In a second step, an online survey was therefore conducted among all 1’891 animal researchers, who were registered in the central online information system of the FSVO and were involved with ongoing experiments at that time. Among others, they were asked which measures against risks of bias they normally use when conducting animal experiments and which of these measures they had reported explicitly in their latest scientific publication. Nearly 30 percent of the researchers participated in the survey. According to the distribution of the data, this is a representative sample.

According to the researchers’ responses, the use of measures against risks of bias is considerably higher than estimated based on evidence found in applications and publications. Thus, 86 percent of the participants claimed to assign animals randomly to treatment groups, but only 44 percent answered that they had reported this in their latest publication. The same applies to the other measures, for example for sample size calculation (69 percent claimed to be doing this, but only 18 percent answered to have reported it in their latest publication) or for blinded outcome assessment (47 percent vs. 27 percent).

From this, the researchers draw two conclusions: on the one hand, their results indicate that estimates based on explicit evidence in applications or publications probably underestimate the actual use of measures against risks of bias considerably. On the other hand, the results also suggest that the researchers tended to overestimate the quality of experimental conduct of their own research. „We found considerably fewer publications with explicit evidence of the use of measures against risks of bias than claimed by the researchers“, says Würbel. For example, 44 percent of the participants claimed to have reported randomization in their latest publication, but Würbel’s team found evidence of randomization in only 17 percent of publications. Furthermore, both the online survey as well as accompanying interviews with selected researchers indicated a lack of awareness of the problem among researchers, and insufficient knowledge about risks of bias and measures to avoid them.

More education and training needed

Animal experiments are being authorized on the explicit understanding that they will provide significant new knowledge, and no unnecessary harm will be imposed on the animals. Thus, scientific rigor is a fundamental prerequisite for the ethical justification of animal experiments. Current practice of authorization of animal experiments rests on confidence rather than evidence of scientific rigor. However, the present results indicate that such confidence may not be sufficiently justified. In order to avoid a loss of confidence and to strengthen the responsible institutions in their tasks, the authors of the studies suggest investing into education and training in good research practice and scientific integrity. They further suggest that the authorization procedure for animal experiments be reviewed for potential for improvement. The two studies are ow being published in the journals PLOS Biology and PLOS ONE.