Weiye Loh's Library tagged Evidence →  View Popular, Search in Google

The typical mantra of the left is that we don't need to sink to the Republicans' level because we have the truth on our side. But if the other side is utterly immune to the truth -- and indeed, the truth only makes them dig deeper into their fantasy world in which the economy is fundamentally strong and the War in Iraq is a staggering success -- what's a leftie to do?

When arguing with conservatives in front of on-the-fence independents, remember that you're not trying to convince the conservative to actually buy into silly notions like facts and reason. You're highlighting the differences between left and right for the outside observer. If the other guy insists on political views that belong only in Disney World's Fantasyland, other folks will realize what's happening.

  

But if there is no third party, do yourself a favor and save your breath. As the study demonstrates, you're only making matters worse.

Many have no problem with biological evolution, whereas others reject it vociferously. The reasons for rejection are often religious, but then justified with misinformed ideas about evolution ("where are the missing links?", "evolution is just a theory").

For a subset of interviewees, evidence (or lack thereof) plays no role in their rejection of evolution. In their minds, an acceptance of biological evolution will lead to the abandonment of their religion. The cost of accepting evolution may simply be too high for them.

we have to realise there will always be individuals who reject evolution for reasons that have little to do with evolution. If certain beliefs are bound up with personal meaning-making, then an attack on those beliefs threatens what is meaningful for that person. An effort to move them towards more evidence-based thinking may need to untangle meaning-making from such beliefs. Even if our efforts are unsuccessful with these individuals, a deeper appreciation, understanding and sympathy for their beliefs may be essential in communicating better science.

when trying to steer the course between skepticism and gullibility, it makes sense to stay much closer to the Scylla of skepticism than to bring our ship of beliefs within reach of the much larger and more menacing Charybdis of gullibility. The net result of this prudent policy, however, is that even positive skeptics are bound to reject a lot of beliefs, with the side effect that their popularity plunges.

(Caveat: I have actually argued in a technical paper that we should abandon the whole idea of null hypotheses and embrace more sophisticated approaches to the comparisons of competing explanations. But that’s another story, and it doesn’t change the basic reasoning of this post.)

A skeptic in the modern sense of the term, let’s say from Hume forward, is someone who thinks that belief in X ought to be proportional to the amount of evidence supporting X.

So what is X in this case? Is it "The chemical is safe" (X1)? Or is it "The chemical is harmful" (X2)? The manufacturer is likely to claim X1, while the health activist is likely to claim X2.

Whenever confronted with a new claim, it’s reasonable to think that the null hypothesis is that the claim is not true. That is, the default position is one of skepticism.

But in this case, it's not clear which claim should be treated with skepticism.

We should consider all competing claims as separate hypotheses, each to be individually weighed against the available information. Both Bayesian and likelihood analyses do this very well.

"It is no miracle that a man, seemingly in good health, should die on a sudden: because such a kind of death, though more unusual than any other, has yet been frequently observed to happen. But it is a miracle that a dead man should come to life; because that has never been observed in any age or country."

On that basis Hume concluded that there is not a single reliable case of a miracle in all of human history. But how did he explain the fact that people report miracles nonetheless?
 
 M: Here is what he said in response to that question: “The passion of surprise and wonder, arising from miracles, being an agreeable emotion, gives a sensible tendency towards the belief of those events, from which it is derived.”
 
 D: So his opinion was that people are fascinated by strange occurrences, and so readily believe them. And I think he also pointed out that it is easy for some people to take advantage of the credulity of others, or even simply that one feels good about having witnessed such an extraordinary event. All of which explains why people are so easily mistaken or credulous about miracles.

Statistics 101: just because a difference is significant doesn’t necessarily mean that it’s significant.

the finding holds true across a wide variety of metrics:
 
 http://www.livescience.com/7689-women-religious-men.html
 
 “The percent of women (and then men) who:
 
 * Are affiliated with a religion: 86 (79).
 * Have absolutely certain belief in a God or universal spirit: 77 (65).
 * Pray at least daily: 66 (49).
 * Have absolutely certain belief in a personal God: 58 (45).”
 
 This gender disparity also shows up across 7 decades of polls.
 
 All this points to a remarkably robust result which requires, at the least, a great deal of explanation to challenge.

with regard to providing evidence for a particular arbitrary interpretation of statistical data versus another… is there evidence that appealing to p<0.05 provides for more accurate conclusions than p<0.01? Type I or Type II error, one must choose one's poison.

whether an absolute but relatively small difference that appears unlikely to be due to random variance in single population (P<0.whathaveyou) is, for want of a separate word, informative, is thoroughly context-dependent (and no where is this a more important issue to acknowledge than in the already dubious realm of evopsych).

religious institutions are almost all patriarchal, and almost all hang on the prophesies, testimonies, miracles and other arcane meanderings of… men. Which puts a big bloody question mark over the idea that women are, by pure voluntary association, more likely to be religious. The shadow of coercion, so neatly emphasized by such thrilling events as the Salem Witch trials, is strong and it is thoroughly male. That the religious leanings of that extra woman in the group of 20 are due to a profound social pressure does not seem like an outrageous hypothesis. Certainly a better one than “Women are more religious than men because they’re women”.

Me: A p-value of 5% is standard in social science and is a good compromise between Type I and Type II errors. That’s the reason why good research must be replicable – and 7 decades of polls is surely sufficient for that. In any event, this is a very simple research finding (very unlike studies of whether drugs are more effective than placebos) with an effect size larger than what you typically find in research, so I am sure the p-value is far below 1%.

As for biological differences, when did they come into the picture?! What I was referring to was the very uncontroversial finding that women are *more religious* than men, not that there’re inherent biological reasons why this is so. There are various theories put forward to explain why women are more religious than men, and not all of them are biologically grounded.

When you sample size is 20 – 10 in each population – I would not draw any strong conclusions. Yet when we crank up the sample size – in theory 30 is the minimum you need to get a reasonably accurate result – and more importantly, replicate the findings multiple times, attempts at denying findings you don’t like look more and more like delusion (the social baggage that invariably taints such investigations is not always on the part of those seeking to draw conclusions from research).
 
 I’m assuming that you don’t think very highly of social science research in general, since more or less all of it can be objected to on similar grounds.

DSKS: I have no general aversion to social science. Arguably the best and the worst use of statistics occurs within that discipline. (Your last post suggests that you might be of a frequentist disposition, which is interesting because this approach is currently under pressure in the social sciences, assailed as it is by the Bayes brigade.

as I understand it the majority of these studies have not been conducted to the standards of a serious quantitative study by a team of social science researchers anyway; mostly basic Q&A based polls from Gallup and similar outfits. Outfits that have been notoriously wrong in there poll-based predictions for more concrete things like voting patterns. These also tend to return the kinds of numbers that are ripe for cooking in all sorts of ways to yield different strengths of interpretation. e.g. for the following:
 
 * Are affiliated with a religion: 86 (79)
 * Have absolutely certain belief in a God or universal spirit: 77 (65).
 * Pray at least daily: 66 (49).
 * Have absolutely certain belief in a personal God: 58 (45).”,

Assuming equal sampling of men vs women, the female fraction of religiously affiliated, God believing, and praying are 52, 55, and 57% respectively. Suddenly, the differences aren’t so striking, and when we consider that the latter three questions are a little vague (what exactly does an individual consider to be “God” or a “universal spirit” and what constitutes prayer?) they’re even less so simply by virtue of being difficult to parse meaningfully.

At best we can say that there is possibly a weak but persistent trend indicative of higher probability of a randomly selected woman being religious than for man. But simply stating a frequency statistic that, “More of the religious are women” is very different from the statement and conclusion that, “Women are more religious than men”. Given as a hypothesis, the latter is as immediately falsifiable as the statement, “Men are taller than women”.

The widely accepted test size of 5%, after all, means that there is a 5% chance of getting the results in question even if your null hypothesis is true. I’m assuming your bar for “a serious quantitative study” is really high, and would disqualify a good deal of research; despite repeating the mantra of “correlation is not causation” when they don’t like particular research findings, I still see researchers fall prey to it – especially when they like what they find (one example: TV violence).

And just mentioning Bayesianism does not mean that women are somehow not more religious than men – after all, the statistics show that more women are religious than men, which will update your prior probability

While definitions of prayer and gods differ, we are not trying to investigate the research question, “Do more women than men hold to the Nicene Creed” but simply “Are women more religious than men?”. What sort of questions would you ask to determine religiosity, if not these?

Also, differences do not have to be overwhelming in order to make a difference. To turn it around, the gender wage gap in 2008 was 77:100 (this is ignoring very important factors like education, experience, occupation, industry and union membership). So the female fraction of earnings in 2008 was only 44%. Yet, if I said “Suddenly, the difference is not so striking” or that “at best we can say that there is possibly a weak but persistent trend indicative of higher probability of a randomly selected woman earning less than a man”, you can imagine the frosty reception I would receive.

Lastly no one (except those flogging straw men) seriously thinks that the statement “Men are taller than women” means that all men are taller than all women, or that “Women are more religious than men” means that all women are more religious than all men, any more than anyone would say that the claim that “there is racism against blacks in the United States” is falsified by there being a black President.

Dr. Spector doesn’t really know the difference between inadequately rigorous science and pseudoscience! Now, don’t get me wrong. I know that it’s not always easy to distinguish science from pseudoscience, especially at the fringes, but in general bad science has to go a lot further than Dr. Spector thinks to merit the the term “pseudoscience.” It is clear (to me, at least) from his articles that Dr. Spector throws around the term “pseudoscience” around rather more loosely than he should, using it as a pejorative for any clinical science less rigorous than a randomized, double-blind, placebo-controlled trial that meets FDA standards for approval of a drug (his pharma background coming to the fore, no doubt). Pseudoscience, Dr. Spector. You keep using that word. I do not think it means what you think it means. Indeed, I almost get the impression from his articles that Dr. Spector views any study that doesn’t reach FDA-level standards for drug approval to be pseudoscience.

Medical science, when it works well, tends to progress from basic science, to small pilot studies, to larger randomized studies, and then–only then–to those big, rigorous, insanely expensive randomized, double-blind, placebo-controlled trials. Dr. Spector mentions hierarchies of evidence, but he seems to fall into a false dichotomy, namely that if it’s not Level I evidence, it’s crap. The problem is, as Mark pointed out, in medicine we often don’t have Level I evidence for many questions. Indeed, for some questions, we will never have Level I evidence. Clinical medicine involves making decisions in the midst of uncertainty, sometimes extreme uncertainty.

Dr. Spector then proceeds to paint a picture of reckless physicians proceeding on crappy studies to pump women full of hormones. Actually, it was more than a bit more complicated on than that. That was the time when I was in my medical training, and I remember the discussions we had regarding the strength (or lack thereof) of the epidemiological data and the lack of good RCTs looking at HRT. I also remember that nothing works as well to relieve menopausal symptoms as HRT, an observation we have been reminded of again since 2003, which is the year when the first big study came out implicating HRT in increasing the risk of breast cancer (more later).

I found a rather fascinating editorial in the New England Journal of Medicine from more than 20 years ago that discussed the state of the evidence back then with regard to estrogen and breast cancer:

 

Evidence that estrogen increases the risk of breast cancer has been surprisingly difficult to obtain. Clinical and epidemiologic studies and studies in animals strongly suggest that endogenous estrogen plays a part in causing breast cancer. If so, exogenous estrogen should be a potent promoter of breast cancer. Although more than 20 case–control and prospective studies of the relation of breast cancer and noncontraceptive estrogen use have failed to demonstrate the expected association, relatively few women in these studies used estrogen for extended periods. Studies of the use of diethylstilbestrol and oral contraceptives suggest that a long exposure or latency may be necessary to show any association between hormone use and breast cancer. In the Swedish study, only six years of follow-up was needed to demonstrate an increased risk of breast cancer with the postmenopausal use of estradiol. It should be noted, however, that half the women in the subgroup that provided detailed data on the duration of hormone use had taken estrogen for many years before their base-line prescription status was defined. The duration of estrogen exposure in these women before the diagnosis of breast cancer was probably seriously underestimated; a short latency cannot be attributed to estradiol on the basis of these data. Other recent studies of the use of noncontraceptive estrogen suggest a slightly increased risk of breast cancer after 15 to 20 years’ use.

even now, the evidence is conflicting regarding HRT and breast cancer, with the preponderance of evidence suggesting that mixed HRT (estrogen and progestin) significantly increases the risk of breast cancer, while estrogen-alone HRT very well might not increase the risk of breast cancer at all or (more likely) only very little. Indeed, I was just at a conference all day Saturday where data demonstrating this very point were discussed by one of the speakers. None of this stops Dr. Spector from categorically labeling estrogen as a “carcinogen that causes breast cancers that kill women.” Maybe. Maybe not. It’s actually not that clear. The problem, of course, is that, consistent with the first primary reports of WHI results, the preponderance of evidence finding health risks due to HRT have indicted the combined progestin/estrogen combinations as unsafe.

“… one in five scores in the top 30% at O and A levels… One in five goes to university and polys”

 

What’s the data for richer kids?

 

Since the proportion of the entire population going to university and polys has increased substantially, this clearly shows that poorer kids are worse off!

The Minister was quoted as saying:

 

“My  parents had six children.  My first home as a young boy was a rental flat in Zion Road.  We shared it as tenants with other families”

 

Citing individuals who made it, may be of no “statistical” relevance, as what we need are the statistics as to the proportion of poorer kids to richer kids, who get scholarships, proportional to their representation in the population.

“More spent on primary and secondary/JC schools.  This means having significantly more and better teachers, and having more programmes to meet children’s specific needs”

 

What has spending more money, which what most countries do, got to do with the argument whether poorer kids are disadvantaged?

Straits Times journalist, Li XueYing put the crux of the debate in the right perspective:

 

“Dr Ng had noted that ensuring social mobility “cannot mean equal outcomes, because students are inherently different”, But can it be that those from low-income families are consistently “inherently different” to such an extent?”

Relevant statistics

 

Perhaps the most damning statistics that poorer kids are disadvantaged was the chart from the Ministry of Education (provided by the Straits Times), which showed that the percentage of Primary 1 pupils who lived in 1 to 3-room HDB flats and subsequently progressed to University and/or Polytechnic, has been declining since around 1986.

Theorists, on the other hand, emphasize economic models that assume people are highly rational in forming expectations of future inflation. In these models, Fed actions that call its commitment to low inflation into question can cause inflation expectations to spike, leading to actual increases in prices and wages.

For theorists, any rise in an indicator of expected or future inflation, like the recent boom in commodity prices, suggests that the Fed’s credibility is at risk. They fear that general inflation could re-emerge quickly, despite high unemployment.

Philosophers may not agree on what, exactly, intuition is, but that doesn’t stop them from using it. “Intuitions often play the role that observation does in science – they are data that must be explained, confirmers or the falsifiers of theories,” Brian Talbot says.⁵ Typically, the way this works is that a philosopher challenges a theory by applying it to a real or hypothetical case and showing that it yields a result which offends his intuitions (and, he presumes, his readers’ as well).

Ethics leans especially heavily on appeals to intuition, with a whole school of ethicists (“intuitionists”) maintaining that a person can see the truth of general ethical principles not through reason, but because he “just sees without argument that they are and must be true.”⁶

Intuitions are also called upon to rebut ethical theories such as utilitarianism: maximizing overall utility would require you to kill one innocent person if, in so doing, you could harvest her organs and save five people in need of transplants. Such a conclusion is taken as a reductio ad absurdum, requiring utilitarianism to be either abandoned or radically revised – not because the conclusion is logically wrong, but because it strikes nearly everyone as intuitively wrong.

British philosopher G.E. Moore used intuition to argue that the existence of beauty is good irrespective of whether anyone ever gets to see and enjoy that beauty. Imagine two planets, he said, one full of stunning natural wonders – trees, sunsets, rivers, and so on – and the other full of filth. Now suppose that nobody will ever have the opportunity to glimpse either of those two worlds. Moore concluded, “Well, even so, supposing them quite apart from any possible contemplation by human beings; still, is it irrational to hold that it is better that the beautiful world should exist than the one which is ugly? Would it not be well, in any case, to do what we could to produce it rather than the other? Certainly I cannot help thinking that it would."⁷

Although similar appeals to intuition can be found throughout all the philosophical subfields, their validity as evidence has come under increasing scrutiny over the last two decades, from philosophers such as Hilary Kornblith, Robert Cummins, Stephen Stich, Jonathan Weinberg, and Jaakko Hintikka (links go to representative papers from each philosopher on this issue). The severity of their criticisms vary from Weinberg’s warning that “We simply do not know enough about how intuitions work,” to Cummins’ wholesale rejection of philosophical intuition as “epistemologically useless.”

One central concern for the critics is that a single question can inspire totally different, and mutually contradictory, intuitions in different people.

For example, I disagree with Moore’s intuition that it would be better for a beautiful planet to exist than an ugly one even if there were no one around to see it. I can’t understand what the words “better” and “worse,” let alone “beautiful” and “ugly,” could possibly mean outside the domain of the experiences of conscious beings

If we want to take philosophers’ intuitions as reason to believe a proposition, then the existence of opposing intuitions leaves us in the uncomfortable position of having reason to believe both a proposition and its opposite.

“I suspect there is overall less agreement than standard philosophical practice presupposes, because having the ‘right’ intuitions is the entry ticket to various subareas of philosophy,” Weinberg says.

But the problem that intuitions are often not universally shared is overshadowed by another problem: even if an intuition is universally shared, that doesn’t mean it’s accurate. For in fact there are many universal intuitions that are demonstrably false.

People who have not been taught otherwise typically assume that an object dropped out of a moving plane will fall straight down to earth, at exactly the same latitude and longitude from which it was dropped. What will actually happen is that, because the object begins its fall with the same forward momentum it had while it was on the plane, it will continue to travel forward, tracing out a curve as it falls and not a straight line. “Considering the inadequacies of ordinary physical intuitions, it is natural to wonder whether ordinary moral intuitions might be similarly inadequate,” Princeton’s Gilbert Harman has argued,⁹ and the same could be said for our intuitions about consciousness, metaphysics, and so on.

We can’t usually “check” the truth of our philosophical intuitions externally, with an experiment or a proof, the way we can in physics or math. But it’s not clear why we should expect intuitions to be true. If we have an innate tendency towards certain intuitive beliefs, it’s likely because they were useful to our ancestors.

But there’s no reason to expect that the intuitions which were true in the world of our ancestors would also be true in other, unfamiliar contexts

And for some useful intuitions, such as moral ones, “truth” may have been beside the point. It’s not hard to see how moral intuitions in favor of fairness and generosity would have been crucial to the survival of our ancestors’ tribes, as would the intuition to condemn tribe members who betrayed those reciprocal norms. If we can account for the presence of these moral intuitions by the fact that they were useful, is there any reason left to hypothesize that they are also “true”? The same question could be asked of the moral intuitions which Jonathan Haidt has classified as “purity-based” – an aversion to incest, for example, would clearly have been beneficial to our ancestors. Since that fact alone suffices to explain the (widespread) presence of the “incest is morally wrong” intuition, why should we take that intuition as evidence that “incest is morally wrong” is true?

The still-young debate over intuition will likely continue to rage, especially since it’s intertwined with a rapidly growing body of cognitive and social psychological research examining where our intuitions come from and how they vary across time and place.

its resolution bears on the work of literally every field of analytic philosophy, except perhaps logic. Can analytic philosophy survive without intuition? (If so, what would it look like?) And can the debate over the legitimacy of appeals to intuition be resolved with an appeal to intuition?

With some white conservatives, I've noticed, the gulf between what they see and what they hear can widen into an unbridgeable chasm. The affect of Englishness—hauteur, refined behavior and aristocracy (none of which I possess)—is something they aspire to, or at least appreciate. Blackness, on the other hand, is not.

so when I introduce myself as a journalist from England I occasionally prompt a moment of synaptic dysfunction. The overwhelming majority get over it. But every now and then they say, "Really? I don't hear an accent."

"If you beat your head against the wall," the Italian Marxist Antonio Gramsci once wrote, "it is your head that breaks, not the wall."

To avoid an almighty headache I try to shut the conversation down: "Well, I can't explain that. But let's get on with the interview."

 

But they won't let it go. "Where in England?" "Were you born there?" "How long have you been here?"

The sad truth is that even when presented with concrete and irrefutable evidence, some people still prefer the reality they want over the one they actually live in.

You can't win an argument without facts, and we live in a moment when whether you're talking about climate change or WMD, facts seem to matter less and less.

the persistent, stubborn, willful refusal to acknowledge basic, known, verifiable facts and the desire to make misinformation the cornerstone of an agenda.

if Obama can't convince the right of these basic facts, what hope does he have of persuading them to support his economic and foreign policies?

But you can negotiate only with those who engage in good faith. In the absence of that, Obama should expend less effort trying to win the right over and more trying to win us back.

Faced with the option of believing something that's not true or gaining tangible benefits like a job or healthcare, most people will take the latter. However petulant, ignorant or gullible people might be, most would prefer to hold on to their jobs, homes and health than their illusions. The problem is that Obama's failure to deliver gives little incentive to exchange fiction for fact.

(As Paul Simon sang, “A man sees what he wants to see and disregards the rest.”) Most of the time, these distortions are unconscious – we don’t know even we are misperceiving the data. However, even when the distortion is intentional it’s still rarely rises to the level of outright fraud. Consider the story of Mike Rossner. He’s executive director of the Rockefeller University Press, and helps oversee several scientific publications, including The Journal of Cell Biology.  In 2002, while trying to format a scientific image in Photoshop that was going to appear in one of the journals, Rossner noticed that the background of the image contained distinct intensities of pixels. “That’s a hallmark of image manipulation,” Rossner told me. “It means the scientist has gone in and deliberately changed what the data looks like. What’s disturbing is just how easy this is to do.” This led Rossner and his colleagues to begin analyzing every image in every accepted paper. They soon discovered that approximately 25 percent of all papers contained at least one “inappropriately manipulated” picture. Interestingly, the vast, vast majority of these manipulations (~99 percent) didn’t affect the interpretation of the results. Instead, the scientists seemed to be photoshopping the pictures for aesthetic reasons: perhaps a line on a gel was erased, or a background blur was deleted, or the contrast was exaggerated. In other words, they wanted to publish pretty images. That’s a perfectly understandable desire, but it gets problematic when that same basic instinct – we want our data to be neat, our pictures to be clean, our charts to be clear – is transposed across the entire scientific process.

One of the philosophy papers that I kept on thinking about while writing the article was Nancy Cartwright’s essay “Do the Laws of Physics State the Facts?” Cartwright used numerous examples from modern physics to argue that there is often a basic trade-off between scientific “truth” and experimental validity, so that the laws that are the most true are also the most useless. “Despite their great explanatory power, these laws [such as gravity] do not describe reality,” Cartwright writes. “Instead, fundamental laws describe highly idealized objects in models.”  The problem, of course, is that experiments don’t test models. They test reality.

Cartwright’s larger point is that many essential scientific theories – those laws that explain things – are not actually provable, at least in the conventional sense. This doesn’t mean that gravity isn’t true or real. There is, perhaps, no truer idea in all of science. (Feynman famously referred to gravity as the “greatest generalization achieved by the human mind.”) Instead, what the anomalies of physics demonstrate is that there is no single test that can define the truth. Although we often pretend that experiments and peer-review and clinical trials settle the truth for us – that we are mere passive observers, dutifully recording the results – the actuality of science is a lot messier than that. Richard Rorty said it best: “To say that we should drop the idea of truth as out there waiting to be discovered is not to say that we have discovered that, out there, there is no truth.” Of course, the very fact that the facts aren’t obvious, that the truth isn’t “waiting to be discovered,” means that science is intensely human. It requires us to look, to search, to plead with nature for an answer.

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.

At first, he assumed that he’d made an error in experimental design or a statistical miscalculation. But he couldn’t find anything wrong with his research. He then concluded that his initial batch of research subjects must have been unusually susceptible to verbal overshadowing. (John Davis, similarly, has speculated that part of the drop-off in the effectiveness of antipsychotics can be attributed to using subjects who suffer from milder forms of psychosis which are less likely to show dramatic improvement.) “It wasn’t a very satisfying explanation,” Schooler says. “One of my mentors told me that my real mistake was trying to replicate my work. He told me doing that was just setting myself up for disappointment.”

In private, Schooler began referring to the problem as “cosmic habituation,” by analogy to the decrease in response that occurs when individuals habituate to particular stimuli. “Habituation is why you don’t notice the stuff that’s always there,” Schooler says. “It’s an inevitable process of adjustment, a ratcheting down of excitement. I started joking that it was like the cosmos was habituating to my ideas. I took it very personally.”

The most likely explanation for the decline is an obvious one: regression to the mean. As the experiment is repeated, that is, an early statistical fluke gets cancelled out. The extrasensory powers of Schooler’s subjects didn’t decline—they were simply an illusion that vanished over time. And yet Schooler has noticed that many of the data sets that end up declining seem statistically solid—that is, they contain enough data that any regression to the mean shouldn’t be dramatic. “These are the results that pass all the tests,” he says. “The odds of them being random are typically quite remote, like one in a million. This means that the decline effect should almost never happen. But it happens all the time!

this is why Schooler believes that the decline effect deserves more attention: its ubiquity seems to violate the laws of statistics. “Whenever I start talking about this, scientists get very nervous,” he says. “But I still want to know what happened to my results. Like most scientists, I assumed that it would get easier to document my effect over time. I’d get better at doing the experiments, at zeroing in on the conditions that produce verbal overshadowing. So why did the opposite happen? I’m convinced that we can use the tools of science to figure this out. First, though, we have to admit that we’ve got a problem.”

In 2001, Michael Jennions, a biologist at the Australian National University, set out to analyze “temporal trends” across a wide range of subjects in ecology and evolutionary biology. He looked at hundreds of papers and forty-four meta-analyses (that is, statistical syntheses of related studies), and discovered a consistent decline effect over time, as many of the theories seemed to fade into irrelevance. In fact, even when numerous variables were controlled for—Jennions knew, for instance, that the same author might publish several critical papers, which could distort his analysis—there was still a significant decrease in the validity of the hypothesis, often within a year of publication. Jennions admits that his findings are troubling, but expresses a reluctance to talk about them publicly. “This is a very sensitive issue for scientists,” he says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”

the worst part was that when I submitted these null results I had difficulty getting them published. The journals only wanted confirming data. It was too exciting an idea to disprove, at least back then.

the steep rise and slow fall of fluctuating asymmetry is a clear example of a scientific paradigm, one of those intellectual fads that both guide and constrain research: after a new paradigm is proposed, the peer-review process is tilted toward positive results. But then, after a few years, the academic incentives shift—the paradigm has become entrenched—so that the most notable results are now those that disprove the theory.

Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for. A “significant” result is defined as any data point that would be produced by chance less than five per cent of the time. This ubiquitous test was invented in 1922 by the English mathematician Ronald Fisher, who picked five per cent as the boundary line, somewhat arbitrarily, because it made pencil and slide-rule calculations easier. Sterling saw that if ninety-seven per cent of psychology studies were proving their hypotheses, either psychologists were extraordinarily lucky or they published only the outcomes of successful experiments. In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology.

While publication bias almost certainly plays a role in the decline effect, it remains an incomplete explanation. For one thing, it fails to account for the initial prevalence of positive results among studies that never even get submitted to journals. It also fails to explain the experience of people like Schooler, who have been unable to replicate their initial data despite their best efforts

an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. Palmer’s most convincing evidence relies on a statistical tool known as a funnel graph. When a large number of studies have been done on a single subject, the data should follow a pattern: studies with a large sample size should all cluster around a common value—the true result—whereas those with a smaller sample size should exhibit a random scattering, since they’re subject to greater sampling error. This pattern gives the graph its name, since the distribution resembles a funnel.

The funnel graph visually captures the distortions of selective reporting. For instance, after Palmer plotted every study of fluctuating asymmetry, he noticed that the distribution of results with smaller sample sizes wasn’t random at all but instead skewed heavily toward positive results.

Palmer has since documented a similar problem in several other contested subject areas. “Once I realized that selective reporting is everywhere in science, I got quite depressed,” Palmer told me. “As a researcher, you’re always aware that there might be some nonrandom patterns, but I had no idea how widespread it is.” In a recent review article, Palmer summarized the impact of selective reporting on his field: “We cannot escape the troubling conclusion that some—perhaps many—cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori beliefs often repeated.”

Palmer emphasizes that selective reporting is not the same as scientific fraud. Rather, the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Stephen Jay Gould referred to this as the “shoehorning” process. “A lot of scientific measurement is really hard,” Simmons told me. “If you’re talking about fluctuating asymmetry, then it’s a matter of minuscule differences between the right and left sides of an animal. It’s millimetres of a tail feather. And so maybe a researcher knows that he’s measuring a good male”—an animal that has successfully mated—“and he knows that it’s supposed to be symmetrical. Well, that act of measurement is going to be vulnerable to all sorts of perception biases. That’s not a cynical statement. That’s just the way human beings work.”

One of the classic examples of selective reporting concerns the testing of acupuncture in different countries. While acupuncture is widely accepted as a medical treatment in various Asian countries, its use is much more contested in the West. These cultural differences have profoundly influenced the results of clinical trials. Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

John Ioannidis, an epidemiologist at Stanford University, argues that such distortions are a serious issue in biomedical research. “These exaggerations are why the decline has become so common,” he says. “It’d be really great if the initial studies gave us an accurate summary of things. But they don’t. And so what happens is we waste a lot of money treating millions of patients and doing lots of follow-up studies on other themes based on results that are misleading.”

In 2005, Ioannidis published an article in the Journal of the American Medical Association that looked at the forty-nine most cited clinical-research studies in three major medical journals. Forty-five of these studies reported positive results, suggesting that the intervention being tested was effective. Because most of these studies were randomized controlled trials—the “gold standard” of medical evidence—they tended to have a significant impact on clinical practice, and led to the spread of treatments such as hormone replacement therapy for menopausal women and daily low-dose aspirin to prevent heart attacks and strokes. Nevertheless, the data Ioannidis found were disturbing: of the thirty-four claims that had been subject to replication, forty-one per cent had either been directly contradicted or had their effect sizes significantly downgraded.

The situation is even worse when a subject is fashionable. In recent years, for instance, there have been hundreds of studies on the various genes that control the differences in disease risk between men and women. These findings have included everything from the mutations responsible for the increased risk of schizophrenia to the genes underlying hypertension. Ioannidis and his colleagues looked at four hundred and thirty-two of these claims. They quickly discovered that the vast majority had serious flaws. But the most troubling fact emerged when he looked at the test of replication: out of four hundred and thirty-two claims, only a single one was consistently replicable. “This doesn’t mean that none of these claims will turn out to be true,” he says. “But, given that most of them were done badly, I wouldn’t hold my breath.”

the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. “The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,” Ioannidis says. In recent years, Ioannidis has become increasingly blunt about the pervasiveness of the problem. One of his most cited papers has a deliberately provocative title: “Why Most Published Research Findings Are False.”

The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. “It feels good to validate a hypothesis,” Ioannidis said. “It feels even better when you’ve got a financial interest in the idea or your career depends upon it. And that’s why, even after a claim has been systematically disproven”—he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins—“you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”

scientists need to become more rigorous about data collection before they publish. “We’re wasting too much time chasing after bad studies and underpowered experiments,” he says. The current “obsession” with replicability distracts from the real problem, which is faulty design. He notes that nobody even tries to replicate most science papers—there are simply too many. (According to Nature, a third of all studies never even get cited, let alone repeated.)

Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations and document all their results. “I think this would provide a huge increase in access to scientific work and give us a much better way to judge the quality of an experiment,” Schooler says. “It would help us finally deal with all these issues that the decline effect is exposing.”

Although such reforms would mitigate the dangers of publication bias and selective reporting, they still wouldn’t erase the decline effect. This is largely because scientific research will always be shadowed by a force that can’t be curbed, only contained: sheer randomness. Although little research has been done on the experimental dangers of chance and happenstance, the research that exists isn’t encouraging

John Crabbe, a neuroscientist at the Oregon Health and Science University, conducted an experiment that showed how unknowable chance events can skew tests of replicability. He performed a series of experiments on mouse behavior in three different science labs: in Albany, New York; Edmonton, Alberta; and Portland, Oregon. Before he conducted the experiments, he tried to standardize every variable he could think of. The same strains of mice were used in each lab, shipped on the same day from the same supplier. The animals were raised in the same kind of enclosure, with the same brand of sawdust bedding. They had been exposed to the same amount of incandescent light, were living with the same number of littermates, and were fed the exact same type of chow pellets. When the mice were handled, it was with the same kind of surgical glove, and when they were tested it was on the same equipment, at the same time in the morning.

The premise of this test of replicability, of course, is that each of the labs should have generated the same pattern of results. “If any set of experiments should have passed the test, it should have been ours,” Crabbe says. “But that’s not the way it turned out.” In one experiment, Crabbe injected a particular strain of mouse with cocaine. In Portland the mice given the drug moved, on average, six hundred centimetres more than they normally did; in Albany they moved seven hundred and one additional centimetres. But in the Edmonton lab they moved more than five thousand additional centimetres. Similar deviations were observed in a test of anxiety. Furthermore, these inconsistencies didn’t follow any detectable pattern. In Portland one strain of mouse proved most anxious, while in Albany another strain won that distinction.

The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand. The problem, of course, is that such dramatic findings are also the most likely to get published in prestigious journals, since the data are both statistically significant and entirely unexpected. Grants get written, follow-up studies are conducted. The end result is a scientific accident that can take years to unravel.

This suggests that the decline effect is actually a decline of illusion.

While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that. Many scientific theories continue to be considered true even after failing numerous experimental tests. Verbal overshadowing might exhibit the decline effect, but it remains extensively relied upon within the field. The same holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation antipsychotics are still widely prescribed, and our model of the neutron hasn’t changed. The law of gravity remains the same.

Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

Arguably, the recent streak of articles on this topic began with one penned by David Freedman in The Atlantic, and provocatively entitled “Lies, Damned Lies, and Medical Science.” In it, the major character was John Ioannidis, the author of some influential meta-studies about the low degree of replicability and high number of technical flaws in a significant portion of published papers in the biomedical literature.

As Freedman put it in The Atlantic: “80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials.” Ioannidis himself was quoted uttering some sobering words for the medical community (and the public at large): “Science is a noble endeavor, but it’s also a low-yield endeavor. I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”

Julia and I actually addressed this topic during a Rationally Speaking podcast, featuring as guest our friend Steve Novella, of Skeptics’ Guide to the Universe and Science-Based Medicine fame. But while Steve did quibble with the tone of the Atlantic article, he agreed that Ioannidis’ results are well known and accepted by the medical research community. Steve did point out that it should not be surprising that results get better and better as one moves toward more stringent protocols like large randomized trials, but it seems to me that one should be surprised (actually, appalled) by the fact that even there the percentage of flawed studies is high — not to mention the fact that most studies are in fact neither large nor properly randomized.

The second big recent blow to public perception of the reliability of scientific results is an article published in The New Yorker by Jonah Lehrer, entitled “The truth wears off.” Lehrer also mentions Ioannidis, but the bulk of his essay is about findings in psychiatry, psychology and evolutionary biology (and even in research on the paranormal!).

In these disciplines there are now several documented cases of results that were initially spectacularly positive — for instance the effects of second generation antipsychotic drugs, or the hypothesized relationship between a male’s body symmetry and the quality of his genes — that turned out to be increasingly difficult to replicate over time, with the original effect sizes being cut down dramatically, or even disappearing altogether.

As Lehrer concludes at the end of his article: “Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can’t bear to let them go. And this is why the decline effect is so troubling.”

None of this should actually be particularly surprising to any practicing scientist. If you have spent a significant time of your life in labs and reading the technical literature, you will appreciate the difficulties posed by empirical research, not to mention a number of issues such as the fact that few scientists ever actually bother to replicate someone else’s results, for the simple reason that there is no Nobel (or even funded grant, or tenured position) waiting for the guy who arrived second.

n the midst of this I was directed by a tweet by my colleague Neil deGrasse Tyson (who has also appeared on the RS podcast, though in a different context) to a recent ABC News article penned by John Allen Paulos, which meant to explain the decline effect in science.

Paulos’ article is indeed concise and on the mark (though several of the explanations he proposes were already brought up in both the Atlantic and New Yorker essays), but it doesn’t really make things much better.

Paulos suggests that one explanation for the decline effect is the well known statistical phenomenon of the regression toward the mean. This phenomenon is responsible, among other things, for a fair number of superstitions: you’ve probably heard of some athletes’ and other celebrities’ fear of being featured on the cover of a magazine after a particularly impressive series of accomplishments, because this brings “bad luck,” meaning that the following year one will not be able to repeat the performance at the same level. This is actually true, not because of magical reasons, but simply as a result of the regression to the mean: extraordinary performances are the result of a large number of factors that have to line up just right for the spectacular result to be achieved. The statistical chances of such an alignment to repeat itself are low, so inevitably next year’s performance will likely be below par. Paulos correctly argues that this also explains some of the decline effect of scientific results: the first discovery might have been the result of a number of factors that are unlikely to repeat themselves in exactly the same way, thus reducing the effect size when the study is replicated.

nother major determinant of the unreliability of scientific results mentioned by Paulos is the well know problem of publication bias: crudely put, science journals (particularly the high-profile ones, like Nature and Science) are interested only in positive, spectacular, “sexy” results. Which creates a powerful filter against negative, or marginally significant results. What you see in science journals, in other words, isn’t a statistically representative sample of scientific results, but a highly biased one, in favor of positive outcomes. No wonder that when people try to repeat the feat they often come up empty handed.

A third cause for the problem, not mentioned by Paulos but addressed in the New Yorker article, is the selective reporting of results by scientists themselves. This is essentially the same phenomenon as the publication bias, except that this time it is scientists themselves, not editors and reviewers, who don’t bother to submit for publication results that are either negative or not strongly conclusive. Again, the outcome is that what we see in the literature isn’t all the science that we ought to see. And it’s no good to argue that it is the “best” science, because the quality of scientific research is measured by the appropriateness of the experimental protocols (including the use of large samples) and of the data analyses — not by whether the results happen to confirm the scientist’s favorite theory.

The conclusion of all this is not, of course, that we should throw the baby (science) out with the bath water (bad or unreliable results). But scientists should also be under no illusion that these are rare anomalies that do not affect scientific research at large. Too much emphasis is being put on the “publish or perish” culture of modern academia, with the result that graduate students are explicitly instructed to go for the SPU’s — Smallest Publishable Units — when they have to decide how much of their work to submit to a journal. That way they maximize the number of their publications, which maximizes the chances of landing a postdoc position, and then a tenure track one, and then of getting grants funded, and finally of getting tenure. The result is that, according to statistics published by Nature, it turns out that about ⅓ of published studies is never cited (not to mention replicated!).

“Scientists these days tend to keep up the polite fiction that all science is equal. Except for the work of the misguided opponent whose arguments we happen to be refuting at the time, we speak as though every scientist’s field and methods of study are as good as every other scientist’s, and perhaps a little better. This keeps us all cordial when it comes to recommending each other for government grants. ... We speak piously of taking measurements and making small studies that will ‘add another brick to the temple of science.’ Most such bricks lie around the brickyard.”

Most damning of all, however, is the potential effect that all of this may have on science’s already dubious reputation with the general public (think evolution-creation, vaccine-autism, or climate change)

“If we don’t tell the public about these problems, then we’re no better than non-scientists who falsely claim they can heal. If the drugs don’t work and we’re not sure how to treat something, why should we claim differently? Some fear that there may be less funding because we stop claiming we can prove we have miraculous treatments. But if we can’t really provide those miracles, how long will we be able to fool the public anyway? The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing.”

Joseph T. Lapp said... 

 

 

 But is any of this new for science? Perhaps science has operated this way all along, full of fits and starts, mostly duds. How do we know that this isn't the optimal way for science to operate?

My issues are with the understanding of science that high school graduates have, and with the reporting of science.

What seems to have emerged in recent decades is a change in the institutional setting that got science advancing spectacularly since the establishment of the Royal Society. Flaws in the system such as corporate funded research, pal-review instead of peer-review, publication bias, science entangled with policy advocacy, and suchlike, may be distorting the environment, making it less suitable for the production of good science, especially in some fields.

Remedies should exist, but they should evolve rather than being imposed on a reluctant sociological-economic science establishment driven by powerful motives such as professional advance or funding. After all, who or what would have the authority to impose those rules, other than the scientific establishment itself?

Lehrer's piece described an anxious sense, noticeable in many different branches of science, that today the evidence section of a lot of research papers isn't as reliable as assumed. The reasons could be psychological, mathematical (statistics packages are powerful and sophisticated, and the sheer amount of data now is huge), and cultural. The question is open. Gracefully, he avoids claiming to know more than he does.

None of this is a claim that about any theories, one way or the other. It's not about whether I'm right when I describe what I think you see in my telescope. It's about whether the telescope is working as well as it should.

It's not a claim that there's no such thing as evidence. It's not a claim that we should all believe whatever the hell we want. It doesn't privilege ignorance over science. It does what we science journalists should be doing for our readers: Tell them how research is actually done, so that they can better judge for themselves what "science says." It is a lucid explanation of a real phenomenon, whose point, I think, is that scientists have taken notice because they want science to do better.

That didn't work for some readers this week. Instead, they stretched and twisted the piece until it fit into a familiar old quarrel: Do we have to listen to what scientists say, or can we believe anything we want? Lehrer quoted a few here. Another example: My friend John Horgan, who reacted like a secular version of Charlton Heston in The Ten Commandments: "Lehrer's broad-brush critique will no doubt also cheer global-warming deniers, creationists, postmodernists and other pesky challengers of scientific orthodoxy," he wrote, later adding, "the evidence is rock-solid for quantum mechanics, general relativity, the germ theory of infectious disease, the genetic code and many other building blocks of scientific knowledge, which have yielded applications that have transformed our world. There's nothing truthy about a hydrogen bomb."

Horgan's Martha-Stewart-of-science style (this is a Bad Thing, that is a Good Thing) requires that his thumb go either up or down on his subject. To make that work I think he decided that Lehrer's piece was about theories and that he needed to defend them from its heresies.

But it's actually a piece about evidence, which means it gives comfort to no theory over any other: The epistemological jitters it describes will afflict creationists and climate denialists as well as psychologists, evolutionary psychologists and fMRI scanners. Except, I guess, the ones who don't care about evidence at all. But they, by definition, aren't listening. People who do care about science, on the other hand, need have no fear. Lehrer's article is no threat to their "scientific orthodoxy."

There was also a backlash. Many doctors resented bitterly the implication that medicine had not always been based on evidence, while others saw unworthy people like statisticians and epidemiologists replacing the magnificence of clinicians. Many doctors thought evidence-based medicine a plot driven by insurance companies, politicians, and administrators in order to cut costs.

The discomfort of many clinicians comes from the fact that the data are derived mainly from clinical trials, which exclude the elderly and people with multiple problems. Yet in the “real world” of medicine, particularly general practice, most patients are elderly and most have multiple problems. So can the “evidence” be applied to these patients? Unthinking application of multiple evidence-based guidelines may cause serious problems, says Mike Rawlins, chairman of NICE.

There has always been anxiety that the zealots would insist evidence was all that was needed to make a decision, and in its early days NICE seemed to take this line. Critics quickly pointed out, however, that patients had things called values, as did clinicians, and that clinicians and patients needed to blend their values with the evidence in a way that was often a compromise.

Social scientists have tended to be wary of the reductionist approach of evidence-based medicine and have wanted a much broader range of information to be admissible.

Evidence-based medicine has been at its most confident when evaluating drug treatments, but many interventions in health care are far more complex than simply prescribing a drug. Insisting on randomized trials to evaluate these interventions may not only be inappropriate, but also misleading. Interventions may be stamped “ineffective” by the hardliners when they actually might offer substantial benefits. Then there is the constant confusion between “evidence of absence of effectiveness” with “absence of evidence of effectiveness”—two very different things.

even some of the strongest proponents of evidence-based medicine have become uneasy, as we have increasing evidence that drug companies have managed to manipulate data. In the heartland of evidence-based medicine—drug trials—the “evidence” may be unreliable and misleading.

All this doesn’t mean that evidence-based medicine should be abandoned. It means, rather, that we must never forget the complex relationship between evidence and truth.

Nano Nonsense and Cryonics goes for the nitty-gritty right away in the opening paragraph:

To see the flaw in this system, thaw out a can of frozen strawberries. During freezing, the water within each cell expands, crystallizes, and ruptures the cell membranes. When defrosted, all the intracellular goo oozes out, turning your strawberries into runny mush. This is your brain on cryonics.

This sounds convincing, but doesn’t address what cryonicists actually claim. Ben Best, President and CEO of the Cryonics Institute, replies in the comments:

Strawberries (and mammalian tissues) are not turned to mush by freezing because water expands and crystallizes inside the cells. Water crystallizes in the extracellular space because more nucleators are found extracellularly. As water crystallizes in the extracellular space, the extracellular salt concentration increases causing cells to lose water osmotically and shrink. Ultimately the cell membranes are broken by crushing from extracellular ice and/or high extracellular salt concentration. […] Cryonics organizations use vitrification perfusion before cooling to cryogenic temperatures. With good brain perfusion, vitrification can reduce ice formation to negligible amounts.

The Skeptic’s Dictionary entry is no advance. Again, it refers erroneously to a “mushy brain”. It points out that the technology to reanimate those in storage does not already exist, but provides no help for us non-experts in assessing whether it is a plausible future technology, like super-fast computers or fusion power, or whether it is as crazy as the sand-powered tank; it simply asserts baldly and to me counterintuitively that it is the latter. Again, perhaps cryonic reanimation is a sand-powered tank, but I can explain to you why a sand-powered tank is implausible if you don’t already know, and if cryonics is in the same league I’d appreciate hearing the explanation.

Another part of the article points out the well-known difficulties with whole-body freezing — because the focus is on achieving the best possible preservation of the brain, other parts suffer more. But the reason why the brain is the focus is that you can afford to be a lot bolder in repairing other parts of the body — unlike the brain, if my liver doesn’t survive the freezing, it can be replaced altogether.

Further, the article ignores one of the most promising possibilities for reanimation, that of scanning and whole-brain emulation, a route that requires some big advances in computer and scanning technology as well as our understanding of the lowest levels of the brain’s function, but which completely sidesteps any problems with repairing either damage from the freezing process or whatever it was that led to legal death.

Sixteen years later, it seems that hasn’t changed; in fact, as far as the issue of technical feasability goes it is starting to look as if on all the Earth, or at least all the Internet, there is not one person who has ever taken the time to read and understand cryonics claims in any detail, still considers it pseudoscience, and has written a paper, article or even a blog post to rebut anything that cryonics advocates actually say. In fact, the best of the comments on my first blog post on the subject are already a higher standard than anything my searches have turned up.

I don’t have anything useful to add, I just wanted to say that I feel exactly as you do about cryonics and living forever. And I thought that this statement:

  

 

I know that I don’t know enough to judge.

 

  

shows extreme wisdom. If only people wishing to comment on global warming would apply the same test.

WRT global warming, the mistake people make is trying to go direct to the first-order evidence, which is much too complicated and too easy to misrepresent to hope to directly interpret unless you make it your life’s work, and even then only in a particular area. The correct thing to do is to collect second-order evidence, such as that every major scientific academy has backed the IPCC.

To begin with, scientific explanation is a very specific and technical kind of knowledge. It requires patience, pedantry, a narrowing of focus and (in the case of the most profound scientific theories) considerable mathematical knowledge and ability. No-one can understand quantum theory — by any account, the most successful physical theory there has ever been — unless they grasp the underlying mathematics. Anyone who says otherwise is fooling themselves.

Religious belief is a very different kind of thing. It is not restricted only to those with a certain education or knowledge, it does not require years of training, it is not specialized and it is not technical. (I’m talking here about the content of what people who regularly attend church, mosque or synagogue take themselves to be thinking; I’m not talking about how theologians interpret this content.)

while religious belief is widespread, scientific knowledge is not. I would guess that very few people in the world are actually interested in the details of contemporary scientific theories. Why? One obvious reason is that many lack access to this knowledge. Another reason is that even when they have access, these theories require sophisticated knowledge and abilities, which not everyone is capable of getting.

most people aren’t deeply interested in science, even when they have the opportunity and the basic intellectual capacity to learn about it. Of course, educated people who know about science know roughly what Einstein, Newton and Darwin said. Many educated people accept the modern scientific view of the world and understand its main outlines. But this is not the same as being interested in the details of science, or being immersed in scientific thinking.

This lack of interest in science contrasts sharply with the worldwide interest in religion. It’s hard to say whether religion is in decline or growing, partly because it’s hard to identify only one thing as religion — not a question I can address here. But it’s pretty obvious that whatever it is, religion commands and absorbs the passions and intellects of hundreds of millions of people, many more people than science does. Why is this? Is it because — as the new atheists might argue — they want to explain the world in a scientific kind of way, but since they have not been properly educated they haven’t quite got there yet? Or is it because so many people are incurably irrational and are incapable of scientific thinking? Or is something else going on?

Some philosophers have said that religion is so unlike science that it has its own “grammar” or “logic” and should not be held accountable to the same standards as scientific or ordinary empirical belief. When Christians express their belief that “Christ has risen,” for example, they should not be taken as making a factual claim, but as expressing their commitment to what Wittgenstein called a certain “form of life,” a way of seeing significance in the world, a moral and practical outlook which is worlds away from scientific explanation.

 

This view has some merits, as we shall see, but it grossly misrepresents some central phenomena of religion. It is absolutely essential to religions that they make certain factual or historical claims. When Saint Paul says “if Christ is not risen, then our preaching is in vain and our faith is in vain” he is saying that the point of his faith depends on a certain historical occurrence.

Theologians will debate exactly what it means to claim that Christ has risen, what exactly the meaning and significance of this occurrence is, and will give more or less sophisticated accounts of it. But all I am saying is that whatever its specific nature, Christians must hold that there was such an occurrence. Christianity does make factual, historical claims. But this is not the same as being a kind of proto-science. This will become clear if we reflect a bit on what science involves.

The essence of science involves making hypotheses about the causes and natures of things, in order to explain the phenomena we observe around us, and to predict their future behavior. Some sciences — medical science, for example — make hypotheses about the causes of diseases and test them by intervening. Others — cosmology, for example — make hypotheses that are more remote from everyday causes, and involve a high level of mathematical abstraction and idealization. Scientific reasoning involves an obligation to hold a hypothesis only to the extent that the evidence requires it. Scientists should not accept hypotheses which are “ad hoc” — that is, just tailored for one specific situation but cannot be generalized to others. Most scientific theories involve some kind of generalization: they don’t just make claims about one thing, but about things of a general kind. And their hypotheses are designed, on the whole, to make predictions; and if these predictions don’t come out true, then this is something for the scientists to worry about.

Religions do not construct hypotheses in this sense. I said above that Christianity rests upon certain historical claims, like the claim of the resurrection. But this is not enough to make scientific hypotheses central to Christianity, any more than it makes such hypotheses central to history. It is true, as I have just said, that Christianity does place certain historical events at the heart of their conception of the world, and to that extent, one cannot be a Christian unless one believes that these events happened. Speaking for myself, it is because I reject the factual basis of the central Christian doctrines that I consider myself an atheist. But I do not reject these claims because I think they are bad hypotheses in the scientific sense. Not all factual claims are scientific hypotheses. So I disagree with Richard Dawkins when he says “religions make existence claims, and this means scientific claims.”

Taken as hypotheses, religious claims do very badly: they are ad hoc, they are arbitrary, they rarely make predictions and when they do they almost never come true. Yet the striking fact is that it does not worry Christians when this happens. In the gospels Jesus predicts the end of the world and the coming of the kingdom of God. It does not worry believers that Jesus was wrong (even if it causes theologians to reinterpret what is meant by ‘the kingdom of God’). If Jesus was framing something like a scientific hypothesis, then it should worry them. Critics of religion might say that this just shows the manifest irrationality of religion. But what it suggests to me is that that something else is going on, other than hypothesis formation.

Religious belief tolerates a high degree of mystery and ignorance in its understanding of the world. When the devout pray, and their prayers are not answered, they do not take this as evidence which has to be weighed alongside all the other evidence that prayer is effective. They feel no obligation whatsoever to weigh the evidence. If God does not answer their prayers, well, there must be some explanation of this, even though we may never know it. Why do people suffer if an omnipotent God loves them? Many complex answers have been offered, but in the end they come down to this: it’s a mystery.

Science too has its share of mysteries (or rather: things that must simply be accepted without further explanation). But one aim of science is to minimize such things, to reduce the number of primitive concepts or primitive explanations. The religious attitude is very different. It does not seek to minimize mystery. Mysteries are accepted as a consequence of what, for the religious, makes the world meaningful.

Religion is an attempt to make sense of the world, but it does not try and do this in the way science does. Science makes sense of the world by showing how things conform to its hypotheses. The characteristic mode of scientific explanation is showing how events fit into a general pattern.

Religion, on the other hand, attempts to make sense of the world by seeing a kind of meaning or significance in things. This kind of significance does not need laws or generalizations, but just the sense that the everyday world we experience is not all there is, and that behind it all is the mystery of God’s presence. The believer is already convinced that God is present in everything, even if they cannot explain this or support it with evidence. But it makes sense of their life by suffusing it with meaning. This is the attitude (seeing God in everything) expressed in George Herbert’s poem, “The Elixir.” Equipped with this attitude, even the most miserable tasks can come to have value: Who sweeps a room as for Thy laws/ Makes that and th’ action fine.

None of these remarks are intended as being for or against religion. Rather, they are part of an attempt (by an atheist, from the outside) to understand what it is. Those who criticize religion should have an accurate understanding of what it is they are criticizing. But to understand a world view, or a philosophy or system of thought, it is not enough just to understand the propositions it contains. You also have to understand what is central and what is peripheral to the view. Religions do make factual and historical claims, and if these claims are false, then the religions fail. But this dependence on fact does not make religious claims anything like hypotheses in the scientific sense. Hypotheses are not central. Rather, what is central is the commitment to the meaningfulness (and therefore the mystery) of the world.

while religious thinking is widespread in the world, scientific thinking is not. I don’t think that this can be accounted for merely in terms of the ignorance or irrationality of human beings. Rather, it is because of the kind of intellectual, emotional and practical appeal that religion has for people, which is a very different appeal from the kind of appeal that science has.

 

Stephen Jay Gould once argued that religion and science are “non-overlapping magisteria.” If he meant by this that religion makes no factual claims which can be refuted by empirical investigations, then he was wrong. But if he meant that religion and science are very different kinds of attempt to understand the world, then he was certainly right.