Introducing Test-Level Analysis in Inspera

This Quicksight Dashboard is part of phase 1 in our strategy for Introducing Test-Level Analysis in Inspera Higher Education Institutions. Further details regarding our strategy and phasing are described in the following sections.

Background

Inspera currently offers advanced analytics exclusively for clients in another market segment using item banks and Item Response Theory (IRT) through Inspera’s GIRT-engine (Generalised Item Response Theory). These clients, primarily awarding bodies, deliver standardised tests within Inspera, with the analytics enabling them to conduct detailed item-level analysis both directly within the Inspera platform and through our Data Discovery Tool, which offers advanced filtering capabilities for in-depth data exploration.

However, this IRT-based approach is not directly applicable to our traditional Higher Education customers. Typically, HE institutions do not utilise item banks in the same way as awarding bodies, as they have different needs regarding content creation, permission control, and metadata services (we are currently working on adapting item banks within Inspera to better cater to these HE requirements). Furthermore, HE institutions tend to rely on Classical Test Theory (CTT) for test analysis because they need to analyse data at the test level or within the context of a single test, rather than a strong focus on individual question level. In contrast, awarding bodies’ assessment model generates individual test versions for each test taker, rendering test-level analysis across versions impractical. This necessitates a focus on analysing individual question performance across all test versions. Therefore, to better serve our HE users, we are expanding our analytics to include CTT-based test-centric analysis, moving beyond the current focus on individual questions.

Below, we outline our roadmap and staged approach to build this new capability, crucially incorporating client feedback at each stage to ensure a product that meets your needs.

Vision for Test-Level Analysis in Inspera

Building upon our existing analytics capabilities, we aim to provide a similarly comprehensive and integrated solution for HE institutions. This means you will have access to:

Visualizations: Embedded directly within the Inspera platform, offering readily accessible insights into test performance using CTT-based metrics. This will build on our existing Item Review Tool and allow you to quickly identify trends, areas of strength and weakness, and potential issues that need to be addressed.

API and Data exploration tools: Most likely, this will be an extension of our existing API for awarding bodies (currently IRT metrics) and Data Discovery Tool, enabling you to perform in-depth analysis of test-level data. This will include advanced filtering capabilities, allowing you to explore specific cohorts, questions, or other relevant criteria.

This approach mirrors our current offering for awarding bodies, ensuring a consistent and user-friendly experience across different user groups. The key difference for HE institutions will be the focus on CTT-based metrics and visualizations tailored to your specific needs.

Rollout plan

Released:

Phase 1: Quicksight Dashboard (Feedback Gathering, Data & Visualisation Access)

Objective: To allow HE clients to interact with test data and for Inspera to gather essential feedback from HE users on the most valuable test statistic graphs.

Implementation: We will deploy a Quicksight dashboard showcasing various CTT-based visualizations. We have chosen Quicksight for this phase as it allows us to quickly and easily share test data and visualizations with you. However, it is important to note that Quicksight is a temporary solution and prototyping tool for this feedback-gathering phase. The prioritized visualizations will ultimately be embedded within the Inspera platform, providing more users with immediate access to critical insights at the exact point of need within their workflows.

Purpose: This allows HE clients to interact with the data and provide direct feedback on graph utility and design. This feedback is critical for informing the development of integrated graphs and tabular data within the Inspera platform.

Next phases:

Phase 2: API Data Export (Enhanced Data Accessibility)

Objective: To enable HE clients to export test statistic data via an API.

Implementation: We will provide an API for exporting CTT-based data.

Purpose: This grants clients the flexibility to analyze the data using their preferred tools and create bespoke visualizations. This also facilitates user-driven exploration and feedback, allowing clients to suggest improvements or new graph designs for integration into Inspera.

Phase 3: Integrated Platform Analytics (Actionable Insights and Workflow Integration)

Objective: To integrate the refined CTT-based test statistic graphs directly into Inspera to deliver actionable insights within key workflows.

Implementation: Based on the feedback gathered in Phases 1 and 2, we will explore and implement features that integrate test statistics into specific tasks within the Inspera platform, providing users with actionable insights at the point of need, such as when setting grade thresholds or identifying non-functioning items prior to marking and grading.

Purpose: To provide seamless access to essential test statistics directly within the Inspera workflow, eliminating the need for external dashboards, and to deliver actionable insights at the point of need, enhancing workflow efficiency and enabling informed decision-making.

These enhancements will empower HE institutions with the tools and insights needed to conduct comprehensive test analysis. This will lead to more effective and reliable assessments, ensuring fairer and more accurate evaluations of student learning. Institutions will be able to identify specific areas where students excel or struggle, enabling data-informed improvements in assessment design. Furthermore, this development will provide a foundation for future integration of AI capabilities, such as automated psychometric summaries and suggested actions, further streamlining analysis and enhancing assessment quality.