Equivalence Testing for Anchor Selection in Differential Item Functioning (DIF) Detection

Anchor selection procedure is a crucial step that should be conducted before DIF analysis can be employed effectively. A DIF analysis that uses a contaminated anchor set suffers from high false alarm rate and low power, making the analysis fails to identify DIF items and incorrectly detects DIF-free items as having DIF. Therefore many methods of anchor selection procedure have been proposed, many of which utilizes Null Hypothesis Test (NHT) to select anchor items. In such methods, one includes an item into an anchor set if the DIF analysis provides a non-significant test. Such procedures are flawed due to two problems. First of all, claiming an item as DIF-free item when the test fails to reject the null hypothesis are based on an incorrect understanding of a non-significant result. Failure to reject the null hypothesis cannot be treated as a piece of evidence in support of the null hypothesis. The second problem is the dependency of anchor selection procedure using NHT on sample size: with sufficiently small sample size, items with large magnitudes of DIF can be identified as DIF-free items, while with sufficiently large sample size, items with negligible DIF magnitudes can be excluded from the anchor set. The current study proposed two anchor selection procedures based on equivalence testing, which is more in line with the goal of identifying DIF-free items as anchor and does not suffer from the two problems mentioned before. The two anchor selection procedures were based on Mantel-Haenszel procedure of detecting DIF items. They were anchor selection procedure employing Mantel-Haenszel chi square-based equivalence testing and Mantel-Haenszel Delta-based equivalence testing. A simulation study was conducted to evaluate the performance of the two procedures. The results showed that the anchor selection procedure employing MH chi square-based equivalence testing was superior compared to the procedures employing MH Delta-based equivalence testing and NHT. The anchor selection procedure employing MH chi square-based equivalence testing provided lower risk of obtaining less than four DIF-free items, lower risk of contamination and lower degree of contamination when anchor contamination did in fact happen. Discussion on the results and limitation of the current study were provided.