Some Consequences of Response Time Model Misspecification in Educational Measurement

Doctoral Dissertation

Abstract

Response times (RTs) on test items are a valuable source of information concerning examinees and the items themselves. As such, they have the potential to improve a wide variety of measurement activities. However, researchers have found that empirical RT distributions can exhibit a variety of shapes among the items within a single test. Though a number of semiparametric and “flexible” parametric models are available, no single model can accommodate all plausible shapes of empirical RT distributions. Thus the goal of this research was to study a few of the potential consequences of RT model misspecification in educational measurement. In particular, two promising applications of RT models were of interest: examinee ability estimation and item selection in computerized adaptive testing (CAT).

First, by jointly modeling RTs and item responses, RTs can be used as collateral information in the estimation of examinee ability. This can be accomplished by embedding separate models for RTs and item responses in Level 1 of a hierarchical model and allowing their parameters to correlate in Level 2. If the RT model is misspecified, a potential drawback of this hierarchical structure is that any negative impact on estimates of the RT model parameters may, in turn, negatively impact ability estimates. However, a simulation study found that estimates of the RT model parameters were robust to misspecification of the RT model. In turn, ability estimates were also robust.

Second, by considering the time intensity of items during item selection in CAT, test completion times can be reduced without sacrificing the precision of ability estimates. This can be done by choosing items that maximize the ratio of item information to the examinee’s predicted RT. However, an RT model is needed to make the prediction; if the RT model is misspecified, this method may not perform as intended. A simulation study found that whether or not the correct RT model was used to make the prediction had no bearing on test completion times. Additionally, using a simple, average RT as the prediction was just as effective as model-based prediction in reducing test completion times.

Attributes

Attribute NameValues
URN
  • etd-08112014-044235

Author Jeffrey M. Patton
Advisor Ying Cheng
Contributor Ying Cheng, Committee Chair
Contributor Ke-Hai Yuan, Committee Member
Contributor Zhiyong Zhang, Committee Member
Contributor Scott E. Maxwell, Committee Member
Degree Level Doctoral Dissertation
Degree Discipline Psychology
Degree Name PhD
Defense Date
  • 2014-08-01

Submission Date 2014-08-11
Country
  • United States of America

Subject
  • adaptive testing

  • model misfit

  • item response theory

Publisher
  • University of Notre Dame

Language
  • English

Record Visibility Public
Embargo Release Date
  • 2015-08-11

Content License
  • All rights reserved

Departments and Units

Files

Please Note: You may encounter a delay before a download begins. Large or infrequently accessed files can take several minutes to retrieve from our archival storage system.