Aberrant Response Detection Using Change-point Analysis

Aberrant responses, which are caused by examinees' unusual behaviors (e.g., carelessness, speededness, item pre-knowledge, warm-up, copying answers from neighbors), are frequently seen in various testing programs. Having aberrant responses may bias item parameter estimates and threaten test validity. Thus proper modeling and detection of aberrant responses is crucial both theoretically and practically in educational and psychological measurement.

Previous work has primarily approached the aberrant responses research in terms of developing person fit statistics and proposing models to capture the aberrant behaviors. This dissertation attempts to incorporate change-point analysis (CPA) to investigate two types of aberrant responses: speededness and warm-up effect; both of which can be viewed as having an abrupt change point. This is an extension work of Shao, Li, and Cheng (in press) which first introduced the CPA method to detect test speededness with known item parameters. Following this line of research, three studies are carried out.

The first study investigates whether CPA can help improve item calibration in the presence of speededness. In this study, the CPA method is used to first detect speeded responses with unknown item parameters. After the detection of speeded examinees as well as their corresponding speeding point (the point at which an examinee starts to speed), a cleansing procedure based upon the detected speeding point is proposed to help improve item parameter estimation. The performance of speededness detection as well as the item parameter recovery before/after utilizing the cleansing procedure are evaluated.

The second study applies the CPA method to warm-up effect detection. This study first compares the impact of warm-up effect on ability estimation and examinees' classification accuracy on two types of tests: Computerized Adaptive Testing and Paper and Pencil (linear) Testing. In the second step, the CPA based detection of warm-up effect is carried out. Two methods of obtaining the critical values are utilized in the study.

Different from the first two studies which both conduct CPA using item response information, the third study applies CPA to response time data in speededness detection. A gradual change log-normal model is proposed to better model the real-life change of response time affected by speededness. Two test statistics are introduced to test for a change point using response time. Moreover, this study explores the possibility of using a fixed critical value across conditions at each nominal level so that practitioners do not have to obtain a critical value through simulation every time they run speededness detection.

The dissertation thesis concludes with the final chapter including a discussion of the results, limitations, practical implications, and future directions.