Workshop on Nonresponse

Tuesday, 22 November 2005

Place: Seminar Room 3031, Social Statistics Research Centre (Building 39), University of Southampton

Programme

Abstracts (with linked papers)

Nonresponse research at Statistics Netherlands, activities and plans: Jelke Bethlehem

A sample selection model for mixed mode surveys: Fannie Cobben

The nonresponse mechanism and its influence on survey outcomes can be modelled by a sample selection model. The estimates from this model can be used to adjust the survey outcome of interest for nonresponse bias. We want to investigate whether this technique is successful in reducing the bias due to nonresponse.

The sample selection model consists of two equations, a participation equation and a regression equation. The dependent variable in the regression equation is only observed when the participation equation has a positive outcome (resulting in a response as opposed to a nonresponse). We are interested in the average expected value of the answer to the survey question among all sampled persons, regardless of whether they participated in the survey or not.

There are some interesting properties of the model that we would like to discuss:

• The identification of the model depends on the specification of the distribution of the error terms from the two equations.
• The model only fits continuous dependent variables; this could be extended to categorical variables.
• Extensions to the model to fit data from mixed-mode surveys and distinguish between the source of nonresponse (no contact, refusal, illness, etc.)

Choosing weighting strata based on minimisation of maximal absolute bias: Barry Schouten

Estimates for population statistics can be seriously biased in case response rates are low and the response to a survey is selective. Methods like poststratification or propensity score weighting are often employed in order to adjust for bias due to nonresponse.

One problem that many adjustment methods have in common is the choice which of the available auxiliary variables to use. In the case of poststratification it must be decided what strata are defined. In the case of propensity score weighting adjustment cells must be formed that have comparable response probabilities.

I show that bounds can be constructed for the bias of the Horvitz-Thompson estimator and conjecture that similar bounds hold for the general regression estimator. These bounds define a bias interval. Without any assumptions about the missing-data mechanism, it holds that the size of the bias is smaller than the width of this interval.

I propose to select strata or adjustment cells based on minimisation of the maximal absolute bias . This strategy simultaneously accounts for the relation with response behaviour and the relation with the important survey questions. There a number of open questions that I would like to discuss.

Nonresponse Research at S3RI, activities and plans: Chris Skinner

Analysis of household survey nonresponse using data from the ONS survey nonresponse census link study: Gabi Durrant (presentation not available)

Over the last decades many surveys have seen a decline in response rates, which is of particular concern for major government surveys. In addition to decreasing response rates, there is indication that the type of non-response may have changed over time, leading to a potential change in non-response bias. It is thought that both factors may affect, potentially severely, the quality of survey data. To increase response rates in surveys and to improve adjustment methods in the presence of non-response, such as imputation and weighting methods, it is crucial to gain a deeper understanding of the nature and causes of non-response. The aim of this paper is to investigate the factors influencing household unit non-response in government surveys. This work aims to contribute to a better understanding of non-response and the improvement of non-response models.

To be able to investigate the reasons for non-response it is important to have reliable data on both the responding and non-responding units. The advantage of this study is that it makes use of a rich set of auxiliary variables, comprising individual and household level census variables, information obtained from interviewers and detailed area information, all available for respondents and non-respondents. The data includes interviewer observations, information about the interaction process between the household and the interviewer at the ‘doorstep’ and information about the experience and attitude of the interviewer. Another advantage is that several major government surveys are included in one study, allowing for comparisons of surveys with different designs and subject matters, rather than focussing on just one survey at a time. The data have been provided by the Office for National Statistics (ONS) and the project is part of the ‘ONS Survey Non-response Census Link Study’.

Household non-response is regarded as a two-stage process, partitioned into non-contact and refusal. The initial results from logistic regression models for refusals (and non-contact) are presented. Since the data contains individual, household, interviewer and area level information it has a cross-classified hierarchical structure. The aim of the project is to investigate (cross-classified) multilevel models to determine the factors of non-response, based on binary and multi-category response data. One aim of the study is the separation of interviewer and area effects through cross-classified multilevel models.

The project aims to investigate implications for survey practice to improve the quality of survey data and estimation. Such approaches include the development of effective methods to reduce non-response rates by modification and improvement of the survey data collection process, the survey design and interviewer training methods. Another important aim is the development and improvement of post-survey adjustment methods, such as weighting, taking into account the hierarchical as well as the two-phase structure of the survey non-response process.

Non-response in surveys with complex designs: the case of England in the PISA cross-national survey of learning achievement: John Micklewright

PISA is a cross-national survey of learning achievement of 15 year olds conducted in some 40 rich industrialised and middle-income countries under the auspices of the OECD. England has a poor track record of response in these types of surveys and the results for England (and the rest of the UK) for the most recent round, 2003, were not published due to concern over non-response bias. The survey has a complex design, with sampling of schools and then of pupils within schools, and response in PISA and other similar surveys is consistently below the international average at both levels. The presentation will outline on-going work at S3RI to analyse the pattern of response at both school and pupil levels in England in PISA 2000 and 2003 and to comment on the appropriateness of the OECD's decision to drop the UK from the 2003 international report.

This work has been carried out with funding from the UK Department for Education and Skills (DfES). A report written for DfES and summary of results are available:

• Response Bias in England in PISA 2003 and 2003 Report
• Response Bias in England in PISA 2003 and 2003 Summary