Oneweb's List: Problems with Research

• Johannessen’s experience, and the project’s as a whole, illustrate that the crisis of “reproducibility” has a double meaning. In one sense, it’s a problem of results: Can a given finding be repeated in another lab? Does the finding tell us something true about the world? In another sense, though, it’s a problem of methodology: Can a given experiment even be repeated in another lab, whatever its results might be? If there’s no way to reproduce experiments, there’s no way to know if we can reproduce results.

• Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.

   

  I’m not a big fan of the whole false-positive, false-negative thing. In this particular case it makes sense because they’re actually working with null data, but ultimately what you’ll want to know is what’s happening to the estimates in the more realistic case that there are nonzero differences amidst the noise. The general message is clear, though: don’t trust FMRI p-values. And let me also point out that this is yet another case of a classical (non-Bayesian) method that is fatally assumption-based.

   

  Perhaps what’s the most disturbing thing about this study is how unsurprising it all is. In one sense, it’s big big news: FMRI is a big part of science nowadays, and if it’s all being done wrong, that’s a problem. But, from another perspective, it’s no surprise at all: we’ve been hearing about “voodoo correlations” in FMRI for nearly a decade now, and I didn’t get much sense that the practitioners of this sort of study were doing much of anything to clean up their act. I pretty much don’t believe FMRI studies on the first try, any more than I believe “gay gene” studies or various other headline-of-the-week auto-science results.
• Those guys at Harvard (but not in the statistics department!) will say, “the replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%.” But they’re innumerate, and they’re wrong. Time for us to move on, time for the scientists to do more science and for the careerists to find new ways to play the game.

• “A lot of what is published is incorrect.”
• the idea that something has gone fundamentally wrong with one of our greatest human creations.
• The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices.

1 more annotation...

• But a study published last month in the Proceedings of the National Academy of Sciences uncovered flaws in the software researchers rely on to analyze fM.R.I. data. The glitch can cause false positives — suggesting brain activity where there is none — up to 70 percent of the time.
• The authors of the paper on the software glitch found that a vast majority of published papers in the field do not make this “multiple comparison” correction. But when they do, they said, the most widely used fM.R.I. data analysis software often doesn’t do it adequately.
• The shocker is that transparency and reproducibility aren’t already required, given that we’re talking about often publicly funded, peer-reviewed, published research. And it’s much the same in other scientific disciplines.
• Nevertheless, the fM.R.I. community seems determined to be an exemplar. The next issue of the journal NueroImage: Clinical will lead with an editorial announcing that it will no longer publish research that has not been corrected for multiple comparisons, and there is a push for other journals to do the same, as well as to require authors to make publicly available their data sets and analyses. Data-sharing platforms such as OpenfMRI and Nuerovault have already been established to make fM.R.I. data and statistical methods more widely accessible. In fact, it was data sharing that revealed the fM.R.I. software glitch.

2 more annotations...

• Dr Eklund now used the current analysis methods and compared 20 healthy people with 20 other healthy people. In other words, there should not have been any differences -- or, in any case only the five percent that chance provides. In total he made three million comparisons of randomly selected groups with data from 499 healthy persons.

   

  "The differences were considerably greater than five percent, up to 60 percent in the worst case," Dr Eklund says.

   

  This means that the analyses could have shown positive results where there shouldn't have been any, thereby indicating brain activity where there was no activity.

   

  He also analysed the same data set with his more calculation-heavy method and obtained a considerably better correspondence, with differences in the expected five percent of cases.

• Irreproducible history

  In 2012, Amgen researchers made headlines when they declared that they had been unable to reproduce the findings  in 47 of 53 'landmark' cancer papers¹. Those papers were never identified
• One study adds to existing criticism of a Science paper that suggested that a cancer drug might be a potential treatment for Alzheimer’s disease²; a second counters earlier findings (including some by Amgen researchers) connecting a gene to insulin sensitivity in mice^{3, 4}; and a third counters a Nature paper reporting that inhibiting one particular protein could enhance degradation of other proteins associated with neurodegenerative diseases⁵.

• Right now, the main way that the scientific community spreads the word about irreproducible research is through innuendo, which is inefficient and unfair to the original researchers, says Ricardo Dolmetsch, global head of neuroscience at Novartis’s Institutes for Biomedical Research in Cambridge, Massachusetts. “Anything we can do to improve the ratio of signal to noise in the literature is very welcome,” he says.

     

   

   

   

  “Non-replication does not necessarily mean 'not true'”

   

   

   

   

  The F1000 initiative is useful, but previous efforts have tried and failed to encourage the reporting of replications and negative results, cautions John Ioannidis, who studies scientific robustness at California's Stanford University. That is because, in general, the scientific community undervalues such work, he says.
• providing industry scientists with a low-barrier way to share their attempts might prove a winning strateg

2 more annotations...

• I'm aware that there's a lot of debate going on around the PACE trial, and, as you will gather, my advice would be to make the data available.
  However, as I note in my blogpost, open data is still a long way from being the norm, and refusal to deposit data seems to be more common than not (see the Ascoli paper I cited). In addition, there was a time when every study that I got ethics approval for had to tell the participants explicitly that their data would *not* be publicly available and would *only* be seen by members of the research team. This was done to protect patients and because of concerns about confidentiality.
  Things are now changing, not least because some patient groups have objected to over-restrictive ethics statements that preclude re-use of data, so studies that are starting out now are more likely to include consent forms and information sheets that make it clear that data-sharing will occur. In such cases, it is of course vital that adequate anonymisation is carried out and that the data does not contain information such as postcodes or dates of birth that could link back to individuals. Even particular symptom patterns could identify someone in some cases. None of these are insuperable problems if data collection and ethics procedures plan for them in advance. But they are issues that affect older studies and can be hard to deal with retrospectively.
  Anyhow, having said all that, I hope it does become possible to fully anonymise and release the PACE data; given the high tension around this study, people are going to think there is some kind of cover-up if the data are withheld.

• “I believe we are in the steroids era of social science,” he says.

• Crowdsourcing research can reveal how conclusions are contingent on analytical choices
• Under the current system, strong storylines win out over messy results. Worse, once a finding has been published in a journal, it becomes difficult to challenge. Ideas become entrenched too quickly, and uprooting them is more disruptive than it ought to be. The crowdsourcing approach gives space to dissenting opinions.

• The stem-cell field holds enormous promise for therapy. As a result, all claims of considerable importance should be verified with utmost care before being made public. The Review suggests that such claims in the field of reprogramming and pluripotency should be demonstrated in more than one experimental model, and encourages their independent replication.

  Nature will endeavour to help the field to achieve its promise, and is looking at ways to support and encourage this reproducibility enterprise. For example, we ask authors to include more details about the methods developed in their studies. We strongly encourage our authors to deposit step-by-step protocols on freely accessible platforms, such as Protocol Exchange (www.nature.com/protocolexchange) — this may be requested for extraordinary claims, at the editor’s discretion. We encourage our authors to verify the origin of the cell lines they use, as we do for cancer cell lines (see Nature 520, 264; 2015).

  The Review concludes: “Science is ultimately a self-correcting process where the scientific community plays a crucial and collective role.” In this case, the stem-cell community has excelled in that role and should be congratulated.

• Tie funding to verified good institutional practice, and robust science will shoot up the agenda, say C. Glenn Begley, Alastair M. Buchan and Ulrich Dirnagl.
• Irreproducible research poses an enormous burden: it delays treatments, wastes patients' and scientists' time, and squanders billions of research dollars. It is also widespread.
• Many parties are addressing the problem. Funding bodies such as the US National Institutes of Health (NIH) have announced training initiatives³ and explicitly instructed grant reviewers to consider whether experimental plans ensure rigour.
• Several journals, including Nature and Science, have updated their guidelines and introduced checklists. These ask scientists whether they followed practices such as randomizing, blinding and calculating appropriate sample size.
• Although researchers want to produce work of long-term value, multiple pressures and prejudices discourage good scientific practices. In many laboratories, the incentives to be first can be stronger than the incentives to be right.
• Amplifying these pressures is a human prejudice in favour of our own ideas. There is a very real temptation to ignore a result that does not conform to our preconceptions, or to recast it so that it does
• Researchers encounter these lists after they have drawn conclusions and are ready to announce them — not when planning their research. There is no mechanism to verify that listed practices were actually employed
• We propose that research institutions that receive public funding should apply the same kind of oversight and support to ensure research integrity as is routinely applied for animal husbandry, biosafety and clinical work.
• Routine discussion of research methods.
• Methods and processes (rather than conclusions) would be debated just as a competitor's paper might be critiqued in a journal club. Primary research material would be available.
• Many investigators determine whether trainees are ready to move on by gauging the number and impact factors of their publications; instead, supervisors should base such decisions on whether their lab members understand research methods and process.
• Compulsory institutional training should ensure a common understanding of rigorous experimental design, research standards and objective evaluation of data. Faculty members and trainees should demonstrate their ability to spot problems such as 'cherry picking' data to make the best story. Compliance with research standards, including data-sharing, should be supported, audited and acknowledged.
• These kinds of changes will require additional money, infrastructure, personnel and paperwork. The load on institutions and investigators will be real, but so is the burden of irreproducible research. Even if it is accompanied by an apparent decrease in productivity, the resulting increase in research quality will be well worth the costs.
• Still, most institutions will not make the necessary moves unless forced. Funding bodies should make GIP a prerequisite for receiving a grant. The concept has gained some traction: last year, Science Foundation Ireland announced plans to conduct external audits on some of the labs that it supports
• Large-scale change is possible. In the 1970s, clinical research had little rigour or oversight. Now clinical trials routinely include concurrent control groups, double-blinding, pre-specified endpoints, power calculations to determine patient numbers and analysis plans that thwart bias. In addition, primary data are available for independent statistical analysis by regulatory authorities. At the time, these changes were controversial; many physicians believed them to be unnecessary and regressive.

13 more annotations...

• A large portion of replications  produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance  for methodological fidelity, and high statistical power to detect the original effect sizes

• The 39% figure derives from the team's subjective assessments of success or failure (see graphic, 'Reliability test'). Another method assessed whether a statistically significant effect could be found, and produced an even bleaker result. Whereas 97% of the original studies found a significant effect, only 36% of replication studies found significant results
• The team also found that the average size of the effects found in the replicated studies was only half that reported in the original studies.
• John Ioannidis, an epidemiologist at Stanford University in California, says that the true replication-failure rate could exceed 80%, even higher than Nosek's study suggests. This is because the Reproducibility Project targeted work in highly respected journals, the original scientists worked closely with the replicators, and replicating teams generally opted for papers employing relatively easy methods — all things that should have made replication easier.
• But they will have a harder time shrugging off the latest study, says Gelman. “This is empirical evidence, not a theoretical argument. The value of this project is that hopefully people will be less confident about their claims.”
• Nosek believes that other scientific fields are likely to have much in common with psychology.

3 more annotations...

• But these disparate results don’t mean that studies can’t inch us toward truth. “On the one hand, our study shows that results are heavily reliant on analytic choices,” Uhlmann told me. “On the other hand, it also suggests there’s a there there. It’s hard to look at that data and say there’s no bias against dark-skinned players.” Similarly, most of the permutations you could test in the study of politics and the economy produced, at best, only weak effects, which suggests that if there’s a relationship between the number of Democrats or Republicans in office and the economy, it’s not a strong one.
• , he and some colleagues looked back at papers their journal had already published. “We had to go back about 17 papers before we found one without an error,” he told me. His journal isn’t alone — similar problems have turned up, he said, in anesthesia, pain, pediatrics and numerous other types of journals.
• “Science is great, but it’s low-yield,” Fang told me. “Most experiments fail. That doesn’t mean the challenge isn’t worth it, but we can’t expect every dollar to turn a positive result. Most of the things you try don’t work out — that’s just the nature of the process.” Rather than merely avoiding failure, we need to court truth.

1 more annotation...