Meta-heuristics in short scale construction

Reference. Schroeders, U., Wilhelm, O., & Olaru, G. (2016). Meta-heuristics in short scale construction: Ant Colony Optimization and Genetic Algorithm. PloS One, 11, e0167110.

Abstract. The advent of large-scale assessment, but also the more frequent use of longitudinal and multivariate approaches to measurement in psychological, educational, and sociological research, caused an increased demand for psychometrically sound short scales. Shortening scales economizes on valuable administration time, but might result in inadequate measures because reducing an item set could: a) change the internal structure of the measure, b) result in poorer reliability and measurement precision, c) deliver measures that cannot effectively discriminate between persons on the intended ability spectrum, and d) reduce test-criterion relations. Different approaches to abbreviate measures fare differently with respect to the above-mentioned problems. Therefore, we compare the quality and efficiency of three item selection strategies to derive short scales from an existing long version: a Stepwise COnfirmatory Factor Analytical approach (SCOFA) that maximizes factor loadings and two metaheuristics, specifically an Ant Colony Optimization (ACO) with a tailored userdefined optimization function and a Genetic Algorithm (GA) with an unspecific cost-reduction function. SCOFA compiled short versions were highly reliable, but had poor validity. In contrast, both metaheuristics outperformed SCOFA and produced efficient and psychometrically sound short versions (unidimensional, reliable, sensitive, and valid). We discuss under which circumstances ACO and GA produce equivalent results and provide recommendations for conditions in which it is advisable to use a metaheuristic with an unspecific out-of-the-box optimization function.

Comment. This is my first Open Access pubclication, funded by the University of Bamberg. With respect to Open Material, all syntax used is published on my GitHub-repository. Finally, in this paper data from the National Educational Panel Study (NEPS): Starting Cohort 4-9th Grade, doi:10.5157/NEPS:SC4:4.0.0. is used, that is, Open Data. Thus, hat trick: Open Access, Open Materials, and Open Data.

Pitfalls and challenges in constructing short forms of cognitive ability measures

Journal of Individual DifferencesReference. Schipolowski, S., Schroeders, U., & Wilhelm, O. (2014). Pitfalls and challenges in constructing short forms of cognitive ability measures. Journal of Individual Differences, 35, 190–200. doi:10.1027/1614-0001/a000134

Abstract. Especially in survey research and large-scale assessment there is a growing interest in short scales for the cost-efficient measurement of psychological constructs. However, only relatively few standardized short forms are available for the measurement of cognitive abilities. In this article we point out pitfalls and challenges typically encountered in the construction of cognitive short forms. First we discuss item selection strategies, the analysis of binary response data, the problem of floor and ceiling effects, and issues related to measurement precision and validity. We subsequently illustrate these challenges and how to deal with them based on an empirical example, the development of short forms for the measurement of crystallized intelligence. Scale shortening had only small effects on associations with covariates. Even for an ultra-short six-item scale, a unidimensional measurement model showed excellent fit and yielded acceptable reliability. However, measurement precision on the individual level was very low and the short forms were more likely to produce skewed score distributions in ability-restricted subpopulations. We conclude that short scales may serve as proxies for cognitive abilities in typical research settings, but their use for decisions on the individual level should be discouraged in most cases.

BEFKI GC-K. Eine Kurzskala zur Messung kristalliner Intelligenz

mdaReference. Schipolowski, S., Wilhelm, O., Schroeders, U., Kovaleva, A., Kemper, C. J. & Rammstedt, B. (2013). BEFKI GC-K: Eine Kurzskala zur Messung kristalliner Intelligenz. methoden, daten, analysen, 7, 153–181. doi:10.12758/mda.2013.010

Abstract. In aktuellen Intelligenzstrukturmodellen gehört kristalline Intelligenz (gc) zu den am besten etablierten Fähigkeitsfaktoren. Dabei spiegelt gc die Einflüsse von Lernen und Akkulturation wider und umfasst somit alles Wissen, das Menschen im Laufe ihres Lebens erwerben und zum Problemlösen einsetzen. In diesem Beitrag beschreiben wir die Entwicklung einer Kurzskala zur Messung kristalliner Intelligenz mit fünfminütiger Bearbeitungszeit, die auf deklarativen Wissensfragen aus den Natur-, Geistes- und Sozialwissenschaften beruht. Aus einem umfangreichen Itempool wurde ein 32 Fragen umfassender Wissenstest zusammengestellt und einer bundesweit repräsentativen Stichprobe von 1.134 Erwachsenen vorgelegt. Anhand psychometrischer Kennwerte und der Beziehungen zu Kovariaten erfolgte eine Auswahl von 12 Items für die Kurzskala. Ein eindimensionales Messmodell für diese Itemauswahl wies eine gute Passung und eine hohe Reliabilität des latenten Faktors auf. In der Zielpopulation der erwachsenen deutschen Bevölkerung wurden keine substanziellen Boden- oder Deckeneffekte beobachtet. Übereinstimmend mit der Langversion zeigten sich für die Kurzskala hohe Beziehungen zum Bildungsabschluss (ISCED-97) und sozioökonomischen Status (ISEI) sowie erwartungskonforme Korrelationen mit selbstberichtetem Wissen und den fünf Hauptdimensionen der Persönlichkeit (Big Five). Die Kurzskala ermöglicht folglich eine effiziente, reliable und valide Erfassung kristalliner Intelligenz im Rahmen der Umfrageforschung.

Der vollständige Beitrag kann von den Seiten der GESIS heruntergeladen werden.