Weiye Loh's Library tagged Confidence → View Popular, Search in Google

Aug

2011

Roger Pielke Jr.'s Blog: More Fun With Uncertainty Guidance

In its AR4 report the IPCC says:
The uncertainty guidance provided for the Fourth Assessment Report draws, for the first time, a careful distinction between levels of confidence in scientific understanding and the likelihoods of specific results. This allows authors to express high confidence that an event is extremely unlikely (e.g., rolling a dice twice and getting a six both times), as well as high confidence that an event is about as likely as not (e.g., a tossed coin coming up heads). Confidence and likelihood as used here are distinct concepts but are often linked in practice.
Here are some specific definitions to help you answer some questions.

A. "high confidence" means "about 8 out of 10 chance of being correct".
B. "extremely unlikely" means "less than 5% probability" of the event or outcome
C. "as likely as not" means "33 to 66% probability" of the event or outcome

So here are your questions:

1. If the IPCC says of a die that it has -- "high confidence that an event is extremely unlikely (e.g., rolling a dice twice and getting a six both times)" -- how should a decision maker interpret this statement in terms of the probability of two sixes being rolled on the next two rolls of the die?

2. If the IPCC says of a die that it has -- "high confidence that an event is about as likely as not (e.g., a tossed coin coming up heads)" -- how should a decision maker interpret this statement in terms of the probability of a head appearing on the next coin flip?

Please provide quantitative answers to 1 and 2, show your work.

rogerpielkejr.blogspot.com/...with-uncertainty-guidance.html

- Preview

Uncertainty Confidence Probability

There is insufficient information to do the calculation rigorously. Actual calculation would require assignment of the rest of the second-order probability (the "confidence") to different probability distributions.

In the absence of that information, a reasonable decision-maker might
(a) decide to ignore the second-order probability and use the existing PDFs ("extremely unlikely" for double sixes and "as likely as not" for the coin toss
(b) hedge against incorrect estimates of the "true" probabilities by assigning additional likelihood to "tail" events. In the coin toss event this doesn't really matter - if you only have two possible outcomes that are approximately equally likely, being somewhat wrong about the PDF won't make much difference for planning. But if you have reason to worry that an event someone called "extremely unlikely" might actually be just "unlikely", you might want to invest extra in hedging against it. (As well, of course, as extra research).

This just returns to my more general claim, that the issue for decision-makers is deciding what PDF to act "as if" is the "true" PDF, even when there really is no "true" PDF.
-1-Paul Baer

Thanks ... I agree with this answer, and have these thoughts about the implications:

(a) ignore the second order stuff (the IPCC generally does) -- bad idea, as this means ignoring potentially relevant info (see #3 below)

You write: "the issue for decision-makers is deciding what PDF to act "as if" is the "true" PDF, even when there really is no "true" PDF"

As I said on the other thread, I get what you are saying, but I have a hard time translating this to practical situations. I think that the decision calculus has to factor in what it means to be "wrong" in a probability judgment as related to the outcomes of a decision based on such judgments. Expressing the view that such judgments cannot be "wrong" is not the way to go.

I return to the notion that context matters in such judgments and flipping a coin has little in common with ratings of mortgage-backed securities.