Weiye Loh's Library tagged Prediction → View Popular, Search in Google

Jan

2012

Roger Pielke Jr.'s Blog: We are Not as Smart as We'd Like to Think

Writing at VoxEu.org Victor Ginsburgh, of the Université Libre de Bruxelles, notes that experts aren't so expert in many situations:

rogerpielkejr.blogspot.com/...t-as-smart-as-wed-like-to.html

- Preview

Prediction Uncertainty Complex System Expertise

professional musicians are unable to distinguish between the tonal superiority of a violin built by Stradivari (which would cost up to $4 million) from that of a new American instrument (a couple of thousand). . .
there is lack of concordance between wine judges. Hodgson’s (2008) [here in PDF] result is even stronger, since he finds that only about 10% of the judges are able to replicate their score within a single wine medal group.
1 more annotation(s)...
- In artistic skating, evaluation depends on the incentives and the monitoring faced by judges. Lee (2004) points out that they face an “outlier aversion bias” because they may be excluded from further competitions if they cannot explain why their rating is at odds with the mean of other judges. Therefore, they manipulate their ratings to achieve “a targeted level of agreement with the other judges,” which essentially implies that their judgement is based on previous achievements, and not on the one that is unfolding, since they have to cast their votes a couple of seconds after the performance of each skater.

Sep

2011

http://www.ieo-imf.org/ieo/files/completedevaluations/01102011Crisis_IV._Why_Did_the_IMF_Fail_to_Give_Clear_Warning.pdf

Various factors played a role in the IMF’s failure to identify risks and give clear warnings. Many of these factors represent long-standing problems that had been highlighted for over a decade. In this section, these factors are grouped into the following broad categories: analytical weaknesses, organizational impediments, internal governance problems, and political constraints. There are considerable interconnections among these categories, and their relative importance is based on subjective judgments. The IMF’s ability to correctly identify the mounting risks was hindered by a high degree of groupthink, intellectual capture, a general mindset that a major financial crisis in large advanced economies was unlikely, and incomplete analytical approaches. Weak internal governance, including unclear lines of responsibility and accountability, lack of incentives to work across units and raise contrarian views, a review process that did not “connect the dots” or ensure follow-up, and an insular culture also played a big role, while political constraints may have also had some impact.

www.ieo-imf.org/...Fail_to_Give_Clear_Warning.pdf

- Preview

Economics Prediction

Aug

2011

Roger Pielke Jr.'s Blog: How Many Findings of the IPCC AR4 WG I are Incorrect? Answer: 28%

Jonassen, R. and R. Pielke, Jr., 2011. Improving conveyance of uncertainties in the findings of the IPCC, Climatic Change, 9 August, 0165-0009:1-9, http://dx.doi.org/10.1007/s10584-011-0185-7.

Abstract Authors of the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) received guidance on reporting understanding, certainty and/or confidence in findings using a common language, to better communicate with decision makers. However, a review of the IPCC conducted by the InterAcademy Council (2010) found that "the guidance was not consistently followed in AR4, leading to unnecessary errors . . . the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented." Our comprehensive and quantitative analysis of findings and associated uncertainty in the AR4 supports the IAC findings and suggests opportunities for improvement in future assessments.

rogerpielkejr.blogspot.com/...ings-of-ipcc-ar4-wg-i-are.html

- Preview

Climate Change Climate Science Policy Probability Prediction Projection IPCC

If we confine our attention to those findings that refer to the future, one can ask how many IPCC findings can be expected to become verified ultimately as being accurate? For example, if we consider findings that refer to future events with likelihood in the ‘likely’ class (i.e., >66% likelihood) then if these judgments are well calibrated then it would be appropriate to conclude that as many as a third can be expected to not occur. More generally, of the 360 findings reported in the full text of WG1 across all likelihood categories and presented with associated measures of likelihood (i.e., those summarized in Table 2 below), then based on the judgments of likelihood associated with each statement we should logically expect that about 100 of these findings (~28%) will at some point be overturned.
ur paper concludes:

Although the IPCC has made enormous contributions and set an important example for global assessment of a vexing problem of immense ramifications, there remain clear opportunities for improvement in documenting findings and specifying uncertainties. We recommend more care in the definition and determination of uncertainty, more clarity in identifying and presenting findings and a more systematic approach in the entire process, especially from assessment to assessment. We also suggest an independent, dedicated group to monitor the process, evaluate findings as they are presented and track their fate. This would include tracking the relationship of findings and attendant uncertainties that pass up the hierarchy of documents within AR5. Strict rules for expressing uncertainty in findings that are derived from (possibly multiple) other findings are needed (see, e.g., the second example in the Supplementary Material).

It is not the purpose of this note to discuss other, related scientific assessments of climate change knowledge; but, we do note that our preliminary analysis of the U.S. Global Change Research Program Synthesis and Assessment Products suggests a far less systematic application of the guidance supplied to authors of those documents and far less consistent application of the defined terms. We believe that the concerns we have expressed here, and the resulting recommendations, apply more broadly than the IPCC process.
12 more annotation(s)...
- Your headline is of course utter nonsense.
  
  If I say (correctly) that a single roll of a fair die is "likely" (83%) to turn up 1-5, then throwing a 6 does not make my statement "incorrect". If I repeat the statement (and experiment) 100 times, then 16% of my statements will not have been incorrect either.
- -1-James Annan
  
  Thanks for the comment. If the IPCC statements were in reference to die rolls you'd be correct. But they are not.
  
  A finding of the IPCC such as the following will either be right (it will verify) or it will be wrong (it will not), the IPCC handicaps it at 90% certain:
  
  "It is very likely that hot extremes, heat waves and heavy precipitation events will continue to become more frequent."
  
  There is a 10% chance, according to the IPCC, that the statement will not verify.
  
  There are not 100 die rolls here, just one future.
- Why does your analysis not apply to a roll of a die? If it helps you, assume that after the first throw, a magpie swoops down and steals it. The throw is not repeatable. Would you claim I had been incorrect if I had thrown a 6?
  
  The statements you analyse are still probabilistic statements irrespective of repeatability, and as such can be neither absolutely verified nor refuted in isolation by the future. If 10% of their very likely statements turn out to not occur, this would simply mean their assessment was well calibrated.
  
  Therefore your usage of "incorrect" (which interesting does not appear in the paper, only the blog post) is nonsensical.
  
  I think you know this and are playing at rabble-rousing. My eyebrows are raised, but only a little, at this behaviour.
- -4-James Annan
  
  If I ask you to predict the next roll of a die and you predict that it will be in the range 1-5, an it turns up 6, then you would be wrong.
  
  You could hem and haw about 100 rolls and such, but I would explain that I was not asking for a discourse on probabilities but a simple prediction of the next roll.
- we may both wish that the IPCC was more sophisticated in its discussion of probabilities and futures, and that is a subject of our paper.
  
  Operational weather forecasters will no doubt understand this situation quite well. To suggest that there is no such thing as being incorrect, probably does not help the credibility of climate science.
- But if, rather than making a definitive prediction that it *will* be in the range 1-5, I merely state that this is "likely" to occur, will you still say I was wrong if it comes up a 6?
  
  It seems to me that the IPCC was rather more sophisticated in its usage of probability, than you are in understanding it.
- The probabilities themselves are basically untestable, as far as whether they reflect reality, vis a vis the "real" PDF, unless there is a 1 or a 0 in there somewhere.
  
  The typical probabilistic statement is probably too vague to really be able to pin it down and decide what it really means. As always, the real sticking point comes down to the assumptions behind the probabilities.
  
  Of course, there's also the fact that (from what I've seen) many of the probabilities are about parameters, not actual data, so the predictions probably don't mean what most people will think they mean.
- with respect to the IPCC PDFs are not known, and the report is written about the climate future, of which we will have only one.
  
  Following your logic the IPCC can never be evaluated empirically about anything expressed probabilistically. This is silly. Probabilistic statements are made to add information to decision making, not to subtract from accountability.
  
  If a weather forecaster offers an 80% chance of rain the next day and does so for 360 days, then I can evaluate that forecast by converting it into a deterministic prediction that should verify 80% of the time, if it is well calibrated. Similarly, if the IPCC offers up 360 statements expressed with various certainties, then (if its judgments are well-calibrated) we can expect that the scientific community will collectively judge 28% of those statements to be incorrect in the future.
- Actually, the IPCC offers PROJECTIONS, not predictions. Projections are the result of running models based on a certain scenario, or "storyline", up to a certain point in the future: if the storyline turns up to be correct, AND the model actually embodies the climatic implications of the storyline, then the projection would agree with observation. Logically this does not imply that the storyline is correct, and that's the reason why several such storylines (scenarios) are used. To the uncertain storyline one has to add the uncertainties in the models themselves: they could be structurally wrong, or some parameter of the model may have been wrongly quantified (e.g. climate sensitivity).
- Probabilities refer always to collections of events, not to specific events. They make statements about the relative frequency of an outcome among the various possible outcomes, over a large number of observations. They can be falsified by collection of events, not by individual events. The 1/6 probability of getting a six in one throw of a die is not falsified by the next throw not showing a six, or even by several throws failing to show a six. Twenty or fifty (or more) throws without a six are definitely possible. It will not be confirmed by getting a six in the next throw. But a large number of throws (by the Law of Large Numbers) will show 1/6 of the cases were sixes. This hardly applies to climate projections, because only ONE future would be actually observable, and that will happen in the future and is thus no help to evaluate the projections in the present.
- Most of the "likely" or "very likely" assessments in the IPCC reports are based on ensembles of models, each with its particular quirks and peculiar quantification of parameters. If 66% of those model runs show a certain result (e.g. average temperature rise above 2°) this is not translatable into 66% probability of such warming to occur: other scenarios and models, or other quantifications of the models, would yield different results.
  
  One must also distinguish probability and strength of belief, though in some context (especially those using Bayesian approaches) the two are often not distinguished. If a die fails to show a six after fifty throws, our belief in the fairness of the die starts to falter, but perhaps the die is fair after all, and we are also facing a long succession of no-sixes which is entirely compatible with the die being fair.
  
  The likelihood attributed to future events in IPCC reports are more akin to strength of belief (among IPCC authors) than to probabilities.
- I thought the exchange between Annan and Pielke Jr was amusing, because Pielke’s headline – 28% of the WG1 findings are wrong – was tongue in cheek. He doesn’t really believe that (as his text itself reveals), but was just being provocative. In a broader sense, the argument reminds us that “probability” is not an inherent property of a system but a measurement of our ignorance about the system. In the case of climate, as we learn more about particular phenomena, the probability of their occurrence will change, even though our learning process hasn’t had any impact on how the natural world behaves. A future event that is 72% probable now may become 99% probable in the future (or 1% probable) even if we have done nothing to change its behavior.

Jul

2011

Roger Pielke Jr.'s Blog: The Social Construction of Black Swans

What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.

I stop and summarize the triplet: rarity, extreme impact, and retrospective (though not prospective) predictability. A small number of Black Swans explains almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives.

rogerpielkejr.blogspot.com/...nstruction-of-black-swans.html

- Preview

Black Swan Prediction Nuclear Japan

A "black swan" event in the words of Nassim Nicolas Taleb
The Fukushima nuclear disaster certainly would seem to qualify. However, a closer look at why the disaster occurred reveals that it was a black swan of our own making.
2 more annotation(s)...
- A design flaw at Japan's Fukushima Daiichi nuclear plant -- one that senior engineers had known about for years -- caused the cooling systems to fail at four reactors after the March 11 earthquake and tsunami, according to company records and interviews with current and former employees.
  
  Tokyo Electric Power Co., the plant's operator, used two different designs for protecting the backup generators for its cooling systems at 10 reactors in the Fukushima region. The cooling systems at the reactors with the older design failed, causing fuel meltdowns and explosions.
  
  The older Fukushima reactors, dating back to the 1960s and built with General Electric Co.'s Mark I design, housed their electric-switching stations in exterior buildings vulnerable to the tsunami's waves. Newer reactors, meanwhile, house these stations in their sturdy main buildings.
  
  When the waves knocked out the switching stations at the older reactors, they rendered the backup generators useless. "Once water gets in there, the whole thing is kaput," said Katsuya Tomono, a former TEPCO executive vice president.
  
  GE said its reactors are safe and that any design flaws are TEPCO's fault because the company was in charge of design changes. Current and former TEPCO engineers say that in retrospect, they should have done something about the flaw.
- It is well understood that black swan events emerge from the murky unknown. What is less appreciated is that the murky unknown is often a region of our own creation.

Jun

2011

Sam Geall - Lies, damn lies and Chinese science | New Humanist

newhumanist.org.uk/...-damn-lies-and-chinese-science

- Preview

Science Methodology Prediction Forecast Seismology Earthquake China Plagiarism

China’s anti-imperialist May Fourth Movement: the early 20th-century cultural and political uprising that championed critical thought and innovation, guided by two enlightenment concepts famously personified as “Mr Science” and “Mr Democracy”. “Not many people understand the work we are doing,” he said. “Most Chinese people’s attitudes to science are superstitious and fearful.” Things may be even worse at the elite level, he said, where science is encouraged in the abstract, without a grasp of the scientific method. Regarding scientific and critical thinking, Fang added, “Chinese people need a new enlightenment.”
Jia Hepeng, editor of the government-backed magazine Science News Bi-weekly, agreed. At an elite level, he explained: “Science – with a capital ‘S’ – is regarded as a once-and-for-all solution to Chinese problems, and as a result it has enjoyed a higher status than any other discipline in China. Anything that is scientific is equal to good.” An important slogan of the current generation of Chinese leaders is the so-called “scientific view of development”, and the government periodically leads crackdowns against “superstition”. But these have nothing to do with “evidence-based approaches” or the “experimental spirit”, he said. Here is the predicament in today’s China: Mr Science may be good, but independent, critical thinking is bad – or as Fang discovered, even life-threatening. This leaves the science dissenters – the sceptics who understand science not as an ideology, but advocate experimental, evidence-based approaches and dare to criticise malpractice – walking a political tightrope.
4 more annotation(s)...
- This publish-or-perish culture has led to unrealistic targets at Chinese universities – and as a predictable consequence, rampant plagiarism.
- In January, the peer-reviewed international journal Acta Crystallographica Section E announced the retraction of more than 70 papers by Chinese scientists who had falsified data. Three months later, the same publication announced the removal of another 39 articles “as a result of problems with the data sets or incorrect atom assignments”, 37 of which were entirely produced in Chinese universities. The New Jersey-based Centenary College closed its affiliated Chinese business school programme in July after a review “revealed evidence of widespread plagiarism, among other issues, at a level that ordinarily would have resulted in students’ immediate dismissal from the college.” A government study, cited by Nature, found that about one-third of over 6,000 scientists surveyed at six top Chinese institutions had practised “plagiarism, falsification or fabrication”.
- Publication bias – the tendency to privilege the results of studies that show a significant finding, rather than inconclusive results – is notoriously pervasive. One systematic review of acupuncture studies from 1998, published in Controlled Clinical Trials, found that every single clinical trial originating in China was positive – in other words, no trial published in China had found a treatment to be ineffective. Moreover, a nationalistic and defensive approach to discredited methods keeps superstition alive in the academies and government.
- One commentary in Science, titled “Earthquakes Cannot Be Predicted”, concluded that it seemed “unwise to invest heavily in monitoring possible [earthquake] precursors”. An advisory body to the Japanese prime minister – Japan had previously been supportive of earthquake prediction research – argued in 1999 that forecasting was not realistic and that research should instead focus on developing new disaster prevention technologies. But nonetheless the practice remains an article of faith for many Chinese scientists and officials

May

2011

Three Questions for America’s Financial Future - Economic View - NYTimes.com

you have probably not run into many humble economists. By its nature, punditry craves attention, which is easier to attract with certainties than with equivocation.

www.nytimes.com/...08view.html

- Preview

Economics Uncertainty Probability Chance Prediction

certitude reflects bravado more often than true knowledge.
In 1967, Milton Friedman gave an address to the American Economic Association with this simple but profound message: The inflation rate that the economy gets is, in large measure, based on the inflation rate that people expect. When everyone expects high inflation, workers bargain hard for wage increases, and companies push prices higher to keep up with the projected cost increases. When everyone expects inflation to be benign, workers and companies are less aggressive. In short, the perception of inflation — or of the lack of it — creates the reality.
2 more annotation(s)...
- Even if expectations are as important as the conventional canon presumes, it isn’t obvious what determines those expectations. Are people merely backward-looking, extrapolating recent experience into the future? Or are the expectations based on the credibility of policy makers? And if credibility matters, how is it established? Are people making rational judgments, or are they easily overcome by fear and influenced by extraneous events?
- in turbulent times, expectations can become unmoored more easily than they think.