Research

Technology-Based Assessment

In the last decades, the digitalization of educational content, the integration of computers in different educational settings and the opportunity to connect knowledge and people via the Internet has led to fundamental changes in the way we gather, process, and evaluate information. Also, more and more tablet PCs or notebooks are used in schools and—in comparison to traditional sources of information such as text books—the Internet seems to be more appealing, versatile, and accessible. Technology-based assessment has been concerned with questions of comparability of test scores across test media, transferring already existing measurement instruments to digital devices. Nowadays, researchers are more interested in enriching the assessment by using interactive tasks and video material or make the testing more efficient using digital behavior traces.

My research in this area is aligned along this dichotomy of comparability of test data gathered on different devices vs. enrichment of the assessment. For example, colleagues and I try to answer fundamental questions about declarative knowledge with a smartphone App: How many dimensions of knowledge have to be distinguished (e.g., humanities vs. natural sciences)? Does knowledge differentiate with age? Is it possible to predict cheating behavior based on auxiliary data such as reaction times and page defocusing events?

Selection of recent publications

Intelligence Research

Our understanding of intelligence has been — and still is — significantly influenced by the development and application of new testing procedures as well as novel computational and statistical methods. In science, methodological developments typically follow new theoretical ideas. In intelligence research, however, great breakthroughs often followed the reverse order. For instance, the once-novel factor analytic tools preceded and facilitated new theoretical ideas such as the theory of multiple group factors of intelligence. Therefore, the way we assess and analyze intelligent behavior also shapes the way we think about intelligence.

I’m mainly interested in an adequate measurement of crystallized intelligence (gc) or, more precisely, declarative knowledge. Despite its enormous importance and ubiquitous nature, gc is often neglected and has aptly been labeled the “dark matter of intelligence”. This is most likely due to the difficulties of its measurement, because as Cattell (1971, p.143) worded “crystallized ability begins after school to extend into Protean forms and that no single investment such as playing bridge or skill in dentistry can be used as a manifestation by which to test all people”. Colleagues and I try to answer fundamental questions about declarative knowledge using modern technology. In a smartphone-based study with more than 4,000 knowledge items we explore, how many dimensions of knowledge have to be distinguished (e.g., humanities vs. natural sciences)? Or, does knowledge differentiate with age?

Selection of recent publications

Machine Learning and Metaheuristics

Machine learning algorithms have positively influenced research in various scientific disciplines such as astrophysics, genetics, or medicine. Psychology is also trying to exploit the new possibilities of machine learning, although psychologists often deal with comparatively small samples and fuzzy indicators. Thus, there is a thin line between serious research and hype. In this research area, I’m concered with two things: (a) to evaluate the potential of machine learning algorithm to predict real-world outcomes, and (b) to use metaheuristics to optimize psychological measures.

To give an impression of the latter line of research: Ant Colony Optimization, which mimics the foraging behavior of ants, is a popular optimization algorithm in computational sciences. Due to its great flexibility it can also be applied to psychological settings, for example, in the construction of short scales: The advent of large-scale assessment and the more frequent use of longitudinal measurement designs led to an increased demand for psychometrically sound short scales. This is often done by simply removing the items with the lowest item-total correlation from the item pool of the long version. We compared the quality and efficiency of such traditional strategies to construct short scales and demonstrated that metaheuristics outperform traditional strategies of item selection.

Selection of recent publications

Structural Equation Modeling Techniques

Structural Equation Modeling (SEM) is a versatile and powerful statistical tool in the social and behavioral sciences. It is useful in creating and refining psychological measures (i.e., establishing measurement models), but also to evaluate complex theoretical models against empirical data (i.e., testing structural models). My research dealing with SEM evolves along three lines: (a) Measurement invariance testing (MI with categorical indicators, longitudinal MI), (b) Meta-Analytic Structural Equation Modeling (MASEM), and (c) Local Weighted Structural Equation Modeling (LSEM).

Measurement invariance is an important concept of test construction and an essential prerequisite to ensure valid and fair comparisons across cultures, administration modes, language versions, or sociodemographic groups. Often MI is tested in a straightforward procedure of comparing measurement models with increasing parameter restritions across subgroups (e.g., female vs. male). Put simply, if results can be attributed to differences in the construct in question rather than membership to a certain group, researchers speak of an invariant measurement. LSEM provides a flexible extension to model covariance structures in dependence of a continuous context variable. These models turned out to be especially useful in describing the development of skills and abilities, since age can be handled as a continuous variable without artificial categorization into different age groups.

Selection of recent publications

#assessment #CBA #crystallized intelligence #knowledge #intelligence #psychometrics #machine learning #metaheuristics #LSEM