twitter email
A comparison of different psychometric approaches to modeling testlet structures

JEMReference. Schroeders, U., Robitzsch, A. & Schipolowski, S. (2014). A comparison of different psychometric approaches to modeling testlet structures: an example with c-tests. Journal of Educational Measurement, 51, 400–418. doi:10.1111/jedm.12054

Abstract. C-tests are a specific variant of cloze tests that are considered time-efficient, valid indicators of general language proficiency. They are commonly analyzed with models of item response theory assuming local item independence. In this article we estimated local interdependencies for 12 C-tests and compared the changes in item difficulties, reliability estimates, and person parameter estimates for different modeling approaches: (a) Rasch, (b) testlet, (c) partial credit, and (d) copula models. The results are complemented with findings of a simulation study in which sample size, number of testlets, and strength of residual correlations between items were systematically manipulated. Results are discussed with regard to the pivotal question whether residual dependencies between items are an artifact or part of the construct.

Example code for the different models and data file

library(TAM)
# specify working directory, read in data
setwd("c:\\temp\\")
dat <- read.table("text308.csv", sep=";", dec=",", header=T, na="" )
testlet <- c(1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 4, 4, 4, 4, 4, 5, 5, 6, 6, 6)

# Rasch model
mod.rasch <- tam(resp = dat[,-1], pid=dat[,1], control=list(maxiter=300, snodes=5000, seed = 99081))

# Testlet model
# testlets are dimensions, assign items to Q-matrix
TT <- length(unique(testlet))
Q <- matrix(0, nrow=ncol(dat[,-1]), ncol= TT + 1)
Q[,1] <- 1 # First dimension constitutes g-factor
for (tt in 1:TT){ Q[testlet == tt, tt+1] <- 1 }

# In a testlet model, all dimensions are uncorrelated among each other, that is, all pairwise correlations are set to 0, which can be accomplished with the "variance.fixed" command
library(combinat)
variance.fixed <- cbind(t(combn(TT+1, 2)), 0)
mod.testlet <- tam(resp = dat[, -1], pid = dat[, 1], Q = Q, control=list(snodes = 5000, maxiter = 300, seed = 99081), variance.fixed = variance.fixed)

# Partial credit model
scores <- list()
testlet.names <- NULL
dat.pcm <- NULL
for (l in 1:max(testlet)) {
    scores[[l]] <- rowSums(dat[ ,-1][, testlet == l, drop=FALSE] )
    dat.pcm <- c(dat.pcm, list(c(scores[[l]])))
    testlet.names <- append(testlet.names, paste("testlet", l, sep=""))
}
dat.pcm <- as.data.frame(dat.pcm)
colnames(dat.pcm) <- testlet.names

mod.pcm <- tam(resp = dat.pcm, control=list(snodes = 5000, maxiter = 300, seed = 99081))

# Copula model
library(sirt)
mod.copula <- rasch.copula2(dat=dat[ ,-1], itemcluster = testlet)