Tagged "knowledge"

NOVA - Next-Generation Open Vocabulary Assessment

NOVA (= Next-Generation Open Vocabulary Assessment) are two openly available, parallel vocabulary tests designed to measure the receptive vocabulary of German-speaking adults. Given the scarcity of modern, non-proprietary instruments, NOVA was developed to fill this gap, using Ant Colony Optimization to ensure high reliability, appropriate item difficulty and discrimination, and close parallelism across forms. The tests showed high conditional reliability in the lower ability range, making them well suited for individual assessment in neuropsychological contexts, and correlated strongly with a test of declarative knowledge. The test development, including the construction rationale, and the psychometric prorperties are described in detail in Schroeders and Achaa-Amankwaa (2025). The norms are based on a large, heterogeneous sample of adults (N = 1,052). A Shiny app is available for scoring, allowing users to compute IRT-based norm scores and percentile ranks from individual response patterns. The items are available in the OSF project.

German translation of this site

↗ PINGUIN
↗ Replication Crisis in Machine Learning

Potential Identification in Elementary School

In the project “Potential Identification in Elementary School for Individual Support” (PINGUIN), we are developing a screening tool to objectively and reliably assess students’ cognitive potential and initial learning conditions at school entry. The computer-based assessment of the PINGUIN project consists of four modules: (1) cognitive potential, (2) language skills, (3) early literacy, and (4) basic mathematical competencies. For each module, tasks are selected adaptively from a comprehensive item bank. The study is conducted in small groups at school using tablets. PINGUIN is designed to help identify children’s potential at an early stage, to provide an objective and fair evaluation of their initial learning conditions, and provide individual support. Teachers can use the knowledge of each child’s individual strengths and weaknesses to tailor their teaching.

↗ PINGUIN
↗ Replikationskrise im Machine Learning

Potenzialidentifikation in der Grundschule zur individuellen Förderung

Im Projekt “Potenzialidentifikation IN der GrUndschule zur INdividuellen Förderung”, kurz PINGUIN, entwickeln wir in einem großem Team ein Screening zur objektiven und zuverlässigen Erfassung des kognitiven Potenzials sowie der Lernausgangslage von Schülerinnen und Schülern in der Schuleingangsphase. Das computerbasierte Messinstrument des PINGUIN-Projekts besteht aus vier Modulen: (1) kognitives Potenzial, (2) sprachliche Leistungen, (3) schriftsprachliche und (4) mathematische Basiskompetenzen. Für jedes Modul werden die Aufgaben adaptiv aus einer umfangreichen Aufgabendatenbank gezogen. Die Untersuchung wird mittels Tablets in Kleingruppen in der Schule durchgeführt. PINGUIN soll dazu beitragen, die Potenziale der Kinder frühzeitig zu erkennen und eine faire, datenbasierte Förderung zu ermöglichen. Das Wissen über die individuellen Stärken und Schwächen der einzelnen Kinder kann von Lehrkräften für ihre Unterrichtsgestaltung herangezogen werden.

Technology-Based Assessment

In the last decades, the digitalization of educational content, the integration of computers in different educational settings and the opportunity to connect knowledge and people via the Internet has led to fundamental changes in the way we gather, process, and evaluate information. Also, more and more tablet PCs or notebooks are used in schools and—in comparison to traditional sources of information such as text books—the Internet seems to be more appealing, versatile, and accessible. Technology-based assessment has been concerned with questions of comparability of test scores across test media, transferring already existing measurement instruments to digital devices. Nowadays, researchers are more interested in enriching the assessment by using interactive tasks and video material or make the testing more efficient using digital behavior traces.

This is the third post in a series on a paper — “Age-related nuances in knowledge assessment” — we recently published in Intelligence. The first post reflected on how knowledge is organized, the second post dealt with psychometric issues. This post is going to be more mathematical (yes, there will be some formulae) and it will be a cautionary note on the use of machine learning algorithms. Machine learning algorithms have positively influenced research in various scientific disciplines such as astrophysics, genetics, or medicine. Also, subdisciplines in psychology such as personality science (e.g., Stachl et al., 2020) or clinical research (Cearns et al., 2019) are adapting the new statistical tools. However, as pointed out in my research statement, every new method initially bears the risk of applying new techniques without the necessary background knowledge. I mainly blame statistical and methodological courses in psychology studies for this. We really have to teach math, stats, and methods more rigorously in university teaching, especially in structured PhD programs.

This is the second post in a series on a recent paper entitled “Age-related nuances in knowledge assessment” that we wrote with Luc Watrin and Oliver Wilhelm. The first post dealt with the way how we conceptualize the organization of knowledge in a hierarchy in a multidimensional knowledge space. The second post reflects on the way we measure or model knowledge. In textbooks knowledge assessments have a special standing, because they can be modeled both from a reflective and a formative perspective.

We published a new paper entitled “Age-related nuances in knowledge assessment” in Intelligence. I really like this paper because it deals with on the way we assess, model, and understand knowledge. And, btw, it employs machine learning methods. Thus, both in terms of content and methodology it hopefully sets a stage for future research avenues that are promising to follow up on. I would like to cover some of the key findings in a series of blog posts. This first post deals with knowledge at different levels of granularity, how they relate to age, and the recurring finding that item sampling plays an important role in test compilation.