Statistical Inference for Correlated Data Based on Censored Observations

Abstract

Many physical quantities measured over time and space are often observed with data irregularities, such as truncation (detection limit) or censoring. Practitioners often disregard censored data cases which may result in inefficient estimates. On the other hand, censored data treated as observed values will lead to biased estimates. For instance, the data values collected by a monitoring device may have a specific detection limit and the device records the value with its limit, or a constant exceeding the limit value, when the real value exceeds the limit. We present an attractive remedy for handling censored or truncated data collected over time or space. Our method produces (asymptotically) unbiased estimates that are more efficient than the estimates based on treating censored observations as completely observed. In particular, we introduce an imputation method particularly well suited for fitting statistical models dealing with correlated observations in the presence of censored data. Our proposed imputation method involves generating random samples from the conditional distribution of the censored data given the (completely) observed data and current estimates of the parameters. The parameter estimates are then updated based on imputed and completely observed data until convergence. Under Gaussian processes, such a conditional distribution turns out to be a truncated multivariate normal distribution. We use a Gibbs sampling method to generate samples from such truncated multivariate normal distributions. We demonstrate the effectiveness of the technique for a problem common to many correlated data sets and describe its application to several other frequently encountered situations. First, we discuss the use of an imputation technique for a stationary time series data assuming an autoregressive moving average model. Then, we relax the model assumption and discuss how the imputation method works with a nonparametric estimation of a covariance matrix. The use of the imputation method is not limited to a time series model and can be applied to other types of correlated data such as a spatial data. A lattice model is discussed as another application field of the imputation method. For pedagogic purposes, our illustration of the approach based on a simulation study is limited to some simple models such as a first order autoregressive time series model, first order moving average time series model, and first order simultaneous autoregressive error model, with left or right censoring. However, the method can easily be extended to more complicated models. We also derive the Fisher information matrix for an AR(1) process containing censored observations and explain the effect of the censoring on the efficiency gain of the estimates using the trace of the Fisher Information matrix.