Matt Jans's List: Total Survey Error Favs

Processing Error

Once a "survey datum" is provided by a respondent (verbally in interviewer-admin surveys, written in paper-and-pencil questionnaires, and typed in web surveys and forms), it must go through processing before it can be analyzed statistically. That might including the interviewer or data entry person entering the response into a computer system, data cleaning and logical skip checking, and moving data files from place to place before a final tangible data set is produced. Any errors in this process can result in mis-representations of the measure taken on the respondent.

Measurement Error

For a given survey question, whether or not the question has high or low validity, a respondent has a true value or score on the construct or information requested. If no measurement error exists, they will report that value accurately. If the question is confusing or vauge, the presentation format is complex or visually unclear, the mode of administration engenders conscious or unconscious "fudging of numbers" (e.g., socially desirable responding and motivated misremembering), measurement error will be introduced.

Construct Validity

When researchers write survey questions they usually have scientific constructs in mind that they want the questions to represent (e.g., attitude toward gun control), or unknown facts or information that they want to gather from the respondent (e.g., the number of doctor visits a person has had in the past year). When a question (i.e., measure) is written that matches the construct well, it is said to have high validity.

Adjustment Error

We use statistical adjustments (e.g., weighting and imputation) to counteract the effects of coverage error, sampling error, and nonresponse error. If those adjustments are not optimal (or are just wrong, including forgetting to adjust data when error sources are present), we can introduce adjustment error. Some adjustments introduce known and measurable error (e.g., the additional variance introduced with multiple imputation).

Nonresponse Error

Nonresponse error occurs when the combination of the response rate (or nonresponse rate) and the difference between respondents and nonrespondents is such that the statistics we calculate from our data are not accurate estimates of the population parameters we hope them to be. Nonresponse error can happen when a sampled unit is never contacted, refuses, or forgets to participate in the survey (unit nonresponse), or when a respondent doesn't or can't answer individual survey questions (item nonresponse).

Sampling Error

Sampling error is easily the most studied and best understood error sources in surveys. Like all survey error sources, errors can be introduced as biased (fixed errors) and variances (variable errors). This list will probably never include ALL the relevant sampling error links, but I'm sharing the resources I've found most useful over time (and new things I find on the topic).

Coverage Error

Books on TSE

Journal Articles on TSE