Who decides when practice becomes a test?
When AI assessment systems blend learning and testing, the student experience looks and feels different. Learners don’t move from classroom to testing center, #2 pencil and scantron in hand. The line between practice and testing is less clear.
In the Current Skills Validation (CSV) prototype we are building at WGU Labs, students can be anywhere, answering a series of questions until the system has enough evidence about their skill level to remove scaffolding and then attempt to validate whether or not they have mastered a skill.
CSV is an AI-enabled system that evaluates skills people develop through work experience and self-directed learning. For a given job role, it creates a skills framework with scoring rubrics, then gathers evidence through five assessment types (role play, multiple choice, open-ended, short answer, and reflection) that are all scored against that framework. Learners start in a practice stage and move into validation, a test-like stage where feedback stops and learner responses count toward a skill determination, once the system has enough evidence of their skill.
My focus is the learner's side of that design: how learners want to experience the shift, and how much they trust the system's judgment and feedback.
The case for and against learner control
As a user-centered learning designer, I generally advocate for learner autonomy and agency. Self-determination theory treats autonomy as a basic need that strengthens motivation (Ryan & Deci, 2000), so my first instinct was to let learners decide when they are ready to be tested.
But the Dunning-Kruger effect gave me pause. Kruger and Dunning (1999) found that the lowest performers tend to overestimate their knowledge and skills the most, so giving learners control over the shift may negatively impact those learners who would benefit the most from extra practice in the formative stage of the system.
What we tested
We recruited 41 Student Insights Council members enrolled in WGU’s School of Education. Some were early in their program and others were close to graduation. We invited them to test a CSV prototype focused on just one skill: differentiation (tailoring teaching methods to fit individual student needs) and scaffolding (providing temporary support that is gradually removed as students gain independence).
Before accessing the system, we asked students about their understanding and experience with differentiating instruction. Most told us they understood differentiated instruction in theory, but few had applied it in instructional settings.
Students had 35 minutes to engage with the prototype. Everyone started in the practice stage, where the system selects items from a pre-generated question bank that matches the learner's current Bloom's taxonomy level (a scale of thinking complexity that runs from “Remember” up to “Create”).
Multiple-choice questions appear in sets of three, short answers in pairs, and role plays or open-ended items function as interactive AI conversations. To move into validation, learners have to cross a sufficiency gate by earning enough competent ratings across different item formats and Bloom's levels.
We randomly assigned learners to one of three conditions for shifting from practice to validation:
- Recommend (14 learners): the system suggested the move from practice to validation, and the learner could accept or decline.
- Inform (14 learners): the system announced the move.
- Unannounced (13 learners): the system made the move silently, and we told learners after.
What happened
While we set out to learn about how learners experienced the shift from practice to validation, only 18 of 41 learners crossed the sufficiency gate, at a median of 27.7 minutes into the 35-minute session. Only 5 showed any sign of noticing the change.
One learner said, “The whole time I wasn’t really sure what mode I was in.” Part of the reason so few crossed into testing was related to scoring: some AI-written scoring guides expected ideas the question never asked for, so learners kept hearing “not quite” without knowing why.
As another learner said, “I feel like I kept saying things, but it was like, ‘next time do this,’ and then it just kept giving me more questions.” We’re still working out how much evidence the system should need before it moves a learner into validation.
What learners told us about trust and feedback
Immediately following the 35-minute session, students completed a brief survey rating nine statements on a five-point agreement scale. Overall ratings were high. Seven of the nine items earned a median score of 5 out of 5. Ratings were similar across the three conditions, though most learners never shifted to testing, so these ratings mostly reflect the practice experience.
In interviews, learners tied trust to whether the feedback matched how they thought they did: "I felt like when I was reading the feedback, I'm like, ‘oh, that makes complete sense.’" Some wanted a human backstop and said they would trust the feedback, "as long as there's some kind of like, way I can refute it and have, like, a human touch it."
Learners split on control
Once we saw that many learners were not crossing the sufficiency gate from practice into validation when the feedback and scaffolding would be removed, we changed our interview protocol. We added a question asking them whether they would want a button that would let them move into validation whenever they felt ready.
Of those we asked, results split evenly. Eight would want it and use it, but three of those only if they could switch back to practice if they felt they needed it. Eight learners said they would not click that button even if it was available, explaining that they thought knowing they were moving into validation might increase their test anxiety.
Self-determination theory suggests there are benefits to giving learners this control such as increased engagement and motivation (Ryan & Deci, 2000). In our interviews, half wanted it, and the other half preferred to let the system decide, some out of trust in the system and some out of anxiety or doubt about their own judgment. These are stated preferences, though, and learners may choose differently once the button is actually in front of them.
What this means for learning designers
- Feedback is where trust gets built. Design it with as much care as the scoring.
- Pair learner choice with a system recommendation and a way back. Learners were split, and research suggests self-assessment alone is unreliable (Dunning et al., 2004; Kruger & Dunning, 1999).
- Decide on purpose whether learners should see the boundary between practice and testing. If learners have a say in when the shift happens, they need to see it.
What's next
This study included 41 students and one skill. Next, we will calibrate the evidence requirements against expert judgment, compare learner-timed and system-timed shifts, and test the AI evaluator against expert raters.
I still believe learners deserve a real say in how they are assessed, and this work is helping us figure out what that should look like.
Want to support or partner in this work? Contact us at info@wgulabs.org.
