Research InitiativeDo students find the prompts helpful?
A prompt a student experiences as nagging is worse than no prompt. We will ask students directly, in interviews and right after sessions, whether a given prompt helped, annoyed, or was ignored.
Research
What the evidence shows, and what it does not.
Two kinds of result exist so far: development checks on synthetic text, and an offline replay of real student logs. Neither is a learning result. Each number below says which kind it is, and the limitations section says why each claim is phrased the way it is.
Results to date
Two kinds of evidence. Neither is a learning result.
Software checks on synthetic text test whether the system behaves as designed. An offline replay of real student logs (ARIA-Pace) tests the timing policy. No study with students has been run.
Timing model · held-out test52%
Fewer interruptions
Under the quiet cost setting and the 3-correct-in-a-row mastery definition, ARIA-Pace intervened 52% less often than helping after every wrong step, and missed 3.7% of students who went on to get stuck (95% CI 2.2–5.3).
Real students in the data4,163
Students whose logs trained and tested the model
We trained and tested the model using public, de-identified student logs. We did not collect any student data ourselves. We also ran a cross-dataset test, but its results will remain unpublished until the dataset terms are confirmed.
Demo problem bank30
Problems with answers re-derived in code
The problem bank covers algebra, trigonometry, and calculus. A symbolic engine recomputes every answer when the bank loads and excludes any problem that fails the check.
Classroom evidenceNot yet
Learning results
No study with students has been run, so we are not claiming that ARIA improves learning, retention, transfer, or outcomes for any group of learners.
Language classifier · development benchmark
Useful for finding failures. Not evidence of learning.
This benchmark uses synthetic examples without independent human labels. The ARIA-Pace gate does not depend on this classifier.
| Development check | Current result | Plain-language meaning | Evidence level |
|---|---|---|---|
| Same-generator synthetic recognition | 84.6% | About 85 of 100 held-out simulated messages from the same generator matched their designed label; no human ground truth. | Synthetic development test |
| Balanced synthetic score | 0.837 | This score summarizes performance while giving each legacy state equal weight. | Synthetic development test |
| Writing-style stress test | 9.05-point gap | Average accuracy changed when unfamiliar generators wrote the examples. | Cross-generator stress test |
| Independent human labels | Pending | Two trained annotators must label real student language before accuracy claims advance. | No result yet |
What these numbers describe.
They test the language classifier on text written by language models in the same style as its training data. They cannot show that ARIA understands real students. In the current tutor, whether to step in after a wrong step is decided by verified correctness and the timing model, not by this classifier. Its flags (they need an explicit feeling word) can add an affect-based reply or prompt a reply to a words-only message, but never override correctness.
Evidence gates
Each stronger claim needs a stronger kind of evidence.
The protocol separates software correctness, offline replay, educator judgment, real-language validation, feasibility with students, and learning outcomes. A result advances only the claim it tests.
- 01
Done: We evaluated the timing policy once on a locked held-out split of 2009–10 student logs and once on a locked 2012–13 split (those numbers are held until the dataset terms are confirmed, as on the ARIA-Pace page), with the specification committed before modeling.
- 02
Pending: Educators still need to review the problem bank and judge when the tutor should intervene.
- 03
Validate the language flags on real student messages, split by complete student or session, against independent human labels.
- 04
Run a reviewed feasibility pilot before testing learning and transfer against an active control.
Limitations
What the current evidence does not show.
ARIA is an early-stage research prototype. Each limitation below is a reason a claim on this site is phrased the way it is.
- The language classifier is built on synthetic think-aloud examples.
That text was generated to imitate student reasoning. It was not collected from students in classrooms, and it has not been validated against human labels; it was evaluated only on synthetic messages, and accuracy fell when a different generator wrote them. The ARIA-Pace gate and its published results do not depend on it. In the demo its flags (they need an explicit feeling word) can add an affect-based reply or prompt a reply to a words-only message, but never override correctness.
- The timing results come from replaying logs, not from students using ARIA.
They show what the policy would have done on recorded behaviour. They cannot show that helping at that moment works, or that any student learned more. “Unnecessary interruption” is a proxy, since the logs contain no real interventions.
- The cost settings are weights we chose.
Quiet, balanced, and supportive are explicit trade-offs between interruption and delayed help. The cost of an interruption is an assumed weight in an offline simulation, not a measured harm: whether the quiet setting reduces frustration or preserves working memory has not been measured, and that needs a live comparison with students. Nothing shows that any group of learners corresponds to the settings.
- The timing model was fitted on math logs, not calculus.
Its parameters come from ASSISTments skill-builder math logs (grade level not stated). The demo applies them unchanged to algebra, trigonometry and calculus problems; they need re-calibration on the target domain before any deployment. The demo also uses one pooled parameter set rather than the per-skill parameters the evaluated policy used; on validation this changes about 14% of Quiet wrong-step decisions, so the demo approximates the evaluated policy. Response time was tested and did not clear the pre-registered bar.
- The interaction design is not yet validated.
Hint wording and timing are informed judgments. Student interviews and prototype testing are how they get corrected.
- The headline saving depends on how “stuck” is defined, and the checks cover BKT only.
The 52% is under the 3-correct-in-a-row mastery definition, which is the ASSISTments skill-builder’s own completion rule. Cross-validated on training and validation students, the quiet saving is 50.1% (95% CI 48.5 to 51.6) with 6.2% missed help under that rule and 7.1% to 41.6% under the four other definitions (exploratory, validation only). Missed help is about 4 to 6% (3.7% on the locked test). The analysis used BKT as the risk score only; the other risk models were not re-run.
- We are not claiming classroom-validated learning gains.
ARIA has not been shown to improve grades, retention, or transfer, and it is not a substitute for a teacher, a tutor, or an IEP support plan.
- The summit demo works on typed math.
Handwriting and photo input are not supported in the build shown at the summit.
- Real-student evaluation is the next step.
The claims on this site will change as that evidence arrives, including if it contradicts what we expected.
Open questions
Four questions the current evidence cannot answer.
Log replay and software tests cannot answer these. Student interviews and prototype sessions are designed to, and any of them could come back against the design.
Do teachers agree with the policy about when to step in?
The timing rule was tuned to a cost we chose. The test that matters is whether experienced teachers, shown the same student moments, would make the same call, and whether they agree with each other.
Does scaffolding improve persistence?
We are looking at whether students attempt more steps before asking for an answer, and whether they return to a hard problem — not at whether the final answer was right.
What input do students actually prefer?
We will investigate when students prefer text, tappable choices, or voice. We expect preferences to depend on the setting, especially whether students are working near peers.
Background
Ideas this builds on.
These ideas motivated the question. Each one is listed with what ARIA does not test, so none of them should be read as support for a result. Cognitive load (Sweller, 1988) is described in the related-work cards below.
Learning with a more capable other
Vygotsky described the gap between what a learner can do alone and what they can do with a little guidance. It motivates ARIA’s question of how much help is enough.
ARIA does not test it: we measure when a tutor speaks in old logs, not what a student can do with or without help.
Vygotsky (1978), Mind in Society, Harvard University Press (a book; no DOI).
Metacognition
Flavell described people monitoring their own thinking. It motivates the idea that a student should get a chance to notice a wrong step before the tutor speaks.
ARIA does not test it: nothing here measures whether a student monitors their own thinking.
Flavell (1979), American Psychologist 34(10), 906–911. DOI
Productive failure
Kapur studied learners who struggle with a problem before being shown the method. It motivates asking whether early help is always better.
ARIA does not test it: we have no learning outcomes, so we cannot say whether staying quiet helps students learn.
Kapur (2008), Cognition and Instruction 26(3), 379–424. DOI
Related work
Research this work builds on.
ARIA is a student project. These are the lines of learning-sciences research that define the question and the method. The full list, with verification notes, is on the ARIA-Pace page.
The assistance dilemma
Koedinger and Aleven (2007) framed the question directly: when does giving help aid learning, and when does withholding it? ARIA-Pace treats that as a decision with an explicit cost for each kind of mistake.
Wheel-spinning
Beck and Gong (2013) described students who keep attempting a skill without ever mastering it. It is the outcome ARIA-Pace predicts, using the operationalisation of later ASSISTments work, and it can be read directly from logs without hand labelling.
Knowledge tracing
Corbett and Anderson’s (1995) Bayesian Knowledge Tracing estimates mastery from a sequence of right and wrong answers. It is the risk model ARIA-Pace currently runs on; four neural models we trained, two single-skill and two that also see earlier attempts on other skills, did not improve the wheel-spinning ranking.
Cognitive load
Sweller (1988) examined how the demands of solving a problem can use up the working memory that learning needs. It informs our choice to keep each hint to one step, as described on the Method page; it is a design influence, not evidence that ARIA works.
These are influences on our thinking, not endorsements or affiliations. We are not affiliated with the researchers or institutions behind this work, and nothing here should be read as their validation of ARIA. Sources and verification notes.