Science and Reproducibility

Third in a series on the edges of Science.

Reproducibility is at the core of science. If an experimental effect can be reproduced by others, then it confirms the hypothesized theory. Usually, the theory is then treated as Truth—at least, until a countering (or extending) theory can be proven. But how reproducible are the results of any given experiment?

A 2016 survey by Nature surprised the scientific world when it discovered more than 70% of researchers have failed to reproduce results by another scientist. This is true even in the fields of physics and engineering (67%), where experiments are often deemed to be quite repeatable. Not surprisingly, the same scientists (in a classic case of confirmation bias) believe strongly that the original results are still correct. This leads to a mild form of professional schizophrenia, in which scientists who rely on replication are willing to forego replication if it contradicts their perception of scientific truth.

The crisis of replication is of immense magnitude in the “softer” sciences such as psychology, medicine, social, and environmental. In fact, these are often call “soft” primarily because of the problems in repeatability.

In medicine, a study of successful therapies found that only 44% of the original studies had been replicated. When the study was repeated by others, 16% of the results actually contradicted the original study. Yet the therapies were approved for use and were in fact effective 92% of the time. What this says about the underlying science is rather alarming, in that repetition of the studies did not match the actual effectivity of the therapy. In other words, these medical practices are approved based on partial trial-and-error results, “proven” in the name of science.

Psychology is another area with wide variability in results, in which only some 36% to 50% of high-ranking studies could be repeated with any reliability. Statistical means are widely used in psychological studies to hone the highly variable data sets. Scientists in this field often have a far better understanding of statistics than physical scientists because of their desperate need to create order out of chaos.

In the environmental field, one 2019 survey of 1,989 water management studies showed that fewer than 5% could be reproduced. Yet the science of environmental management relies on these studies as Truth.

The result of this replication crisis is that our science often makes claims that are only weakly supported by the studies behind them. Scientists want success. Funding sources want success. Publications want successful reports. Companies want proven products. As in any human endeavor, desire often trumps rigor. 

Recent decades have seen significant reversals in our treatment and understanding of many areas that were once considered proven science. Examples include thalidomide and other drugs, psychological classification of disorders, understanding of sub-atomic particles, environmental theories, and even the attainability of cold fusion.

The demonstrated lack of reproducibility provides a rich area for science fiction stories. Nearly any scientific theory has experiments available proving it wrong; those experiments might be a distinct minority, yet they open the door for new ideas. As a writer, I love the prospect of extending the possibilities.

Doc Honour
September 2023