Charlie Gibbons's Library tagged econometrics →  View Popular

The confidence level is set in advance. The resulting p-value is a random variable. To adjust the level in response to the p-value is an improper ex post move. So if the pre-set level of significance is 95% and the resulting p-value is 0.0001, the proper response is to say "The results are significant at the 95% level (p = 0.0001)." To readjust the claim to "The results are significant at the 99% level (since) p = 0.0001" is simply incorrect.

Thus: there should only be one star per table, at whatever level the analyst sets in advance.

udging by his assertion that surgery may violate parameter stability, it seems that Heckman conflates the DEFINITION of causal effect with the practical means available for TESTING it. Whereas the latter can be crude or inadequate, the former is always pure and incisive.

This conflation is reflected again on page 43, where Heckman states: "If for a given model, the parameters of (16a) or (16b) shift when external variables are manipulated, or if external variables cannot be independently manipulated, causal effects of one internal variable on another cannot be defined within that model (italic in the original)." The first part of the sentence speaks about physical manipulation, while the second speaks about a mathematical definition. The sentence as a whole, implies that the causal effect of Y₂ on Y₁ depends on the technology available to the experimenter — hardly a dependency we would expect from a "definition".

On page 44, Heckman argues again that the definition of causal effects must depend on the physical manipulations available to the experimenter. He describes a situation in which the behavior of agent 2 changes, depending on whether U1 is randomized or chosen naturally . He then says: "At issue is whether such a randomization would recover c₁₂. It [randomizing U₁] might fundamentally alter agent 1's response to Y₂ if that person is randomly assigned as opposed to being selected by the agent. Judging the suitability of an invariance assumption entails a thought experiment — a purely mental act." Again, Heckman confuses the causal effect c₁₂ as encoded in (16), with some perturbed causal effect c'₁₂ that prevails when certain interventions are implemented to measure c₁₂. Obviously, if an experimental intervention modifies c₁₂ or has other side-effects, the modification must be encoded in the model and assessed whether the original c₁₂ can be recovered from data obtained under such imperfect manipulation. But this does not change the fact that the original c₁₂ (prior to intervention) is the parameter of interest and, more importantly, that c12 can be defined within the original unperturbed model Eqs (16a) and (16b) by a mental act of shutting down eq. (16b), fixing every variable on the rhs to a constant, Y₁=y₁, X₁=x₁, X₂=x₂, and computing mathematically the partial derivative (with respect to y₁) of the expected value of Y<!-- ∂ --> und