Tagged "crystallized intelligence"

Technology-Based Assessment

In the last decades, the digitalization of educational content, the integration of computers in different educational settings and the opportunity to connect knowledge and people via the Internet has led to fundamental changes in the way we gather, process, and evaluate information. Also, more and more tablet PCs or notebooks are used in schools and—in comparison to traditional sources of information such as text books—the Internet seems to be more appealing, versatile, and accessible. Technology-based assessment has been concerned with questions of comparability of test scores across test media, transferring already existing measurement instruments to digital devices. Nowadays, researchers are more interested in enriching the assessment by using interactive tasks and video material or make the testing more efficient using digital behavior traces.

NOVA - Next-Generation Open Vocabulary Assessment

NOVA (= Next-Generation Open Vocabulary Assessment) are two openly available, parallel vocabulary tests designed to measure the receptive vocabulary of German-speaking adults. Given the scarcity of modern, non-proprietary instruments, NOVA was developed to fill this gap, using Ant Colony Optimization to ensure high reliability, appropriate item difficulty and discrimination, and close parallelism across forms. The tests showed high conditional reliability in the lower ability range, making them well suited for individual assessment in neuropsychological contexts, and correlated strongly with a test of declarative knowledge. The test development, including the construction rationale, and the psychometric prorperties are described in detail in Schroeders and Achaa-Amankwaa (2025). The norms are based on a large, heterogeneous sample of adults (N = 1,052). A Shiny app is available for scoring, allowing users to compute IRT-based norm scores and percentile ranks from individual response patterns. The items are available in the OSF project.

This is the third post in a series on a paper — “Age-related nuances in knowledge assessment” — we recently published in Intelligence. The first post reflected on how knowledge is organized, the second post dealt with psychometric issues. This post is going to be more mathematical (yes, there will be some formulae) and it will be a cautionary note on the use of machine learning algorithms. Machine learning algorithms have positively influenced research in various scientific disciplines such as astrophysics, genetics, or medicine. Also, subdisciplines in psychology such as personality science (e.g., Stachl et al., 2020) or clinical research (Cearns et al., 2019) are adapting the new statistical tools. However, as pointed out in my research statement, every new method initially bears the risk of applying new techniques without the necessary background knowledge. I mainly blame statistical and methodological courses in psychology studies for this. We really have to teach math, stats, and methods more rigorously in university teaching, especially in structured PhD programs.

This is the second post in a series on a recent paper entitled “Age-related nuances in knowledge assessment” that we wrote with Luc Watrin and Oliver Wilhelm. The first post dealt with the way how we conceptualize the organization of knowledge in a hierarchy in a multidimensional knowledge space. The second post reflects on the way we measure or model knowledge. In textbooks knowledge assessments have a special standing, because they can be modeled both from a reflective and a formative perspective.

We published a new paper entitled “Age-related nuances in knowledge assessment” in Intelligence. I really like this paper because it deals with on the way we assess, model, and understand knowledge. And, btw, it employs machine learning methods. Thus, both in terms of content and methodology it hopefully sets a stage for future research avenues that are promising to follow up on. I would like to cover some of the key findings in a series of blog posts. This first post deals with knowledge at different levels of granularity, how they relate to age, and the recurring finding that item sampling plays an important role in test compilation.