Research InitiativeResearch note 01 · Updated October 2026
How ARIA plans to earn its claims.
ARIA separates software checks, educator judgment, real-language validation, student feasibility, and learning outcomes. A result advances only the claim it actually tests.
The research question
Can ARIA provide grounded help without taking over the work?
The program tests several different questions: whether tasks are correct, whether responses use the student’s actual reasoning, whether language labels match independent human judgments, and eventually whether students learn and transfer strategies.
Evidence sequence
Five gates, from a working system to a learning claim.
- 01
Validate the task bank.
Two subject-qualified educators independently review answer models, solution paths, misconceptions, hints, and scoring criteria. Disagreements stay visible.
- 02
Blindly compare response quality.
Educators rate paired conditions for problem grounding, student grounding, usefulness, actionability, ownership, answer leakage, and invented student actions.
- 03
Validate language on real student messages.
Two trained annotators label observable reasoning moves with exact evidence spans. Evaluation is split by complete student or session, never random messages.
- 04
Run a reviewed feasibility pilot.
With institutional review, parent permission, and student assent, test whether the tool is understandable, usable, and safe before asking whether it is effective.
- 05
Measure learning against an active control.
Use independent outcome tasks, concealed assignment, blinded scoring, intention-to- treat analysis, and a delayed no-ARIA transfer measure.
Current evidence ledger
Demo problems
A symbolic engine re-derives the answers. Independent educator review is pending.
Answer leaks in scripted tests
The scripted harness replies contained no false mathematical claims. These checks are not evidence from real students.
Completed outcome studies
No classroom learning effectiveness claim is being made.
| Evidence layer | Current status | What remains | Claim allowed today |
|---|---|---|---|
| Task models | 30 symbolic-engine-checked demo problems | Educator correctness review is pending. | Answers are re-derived by a symbolic engine across algebra, trigonometry and calculus. |
| Observable language | Uncertainty, frustration, and help-seeking flags | Independent annotation on real student language is pending. | Language flags shape the tone of help; the math checker determines correctness. |
| Response quality | 0 answer leaks, 0 math errors, 0 crashes in scripted harness runs on the 8 featured problems | Independent human evaluation remains pending. | These results come from a scripted software test on chosen inputs, not a student study. The results are recorded in results/demo_harness_summit_pace.json in the aria.kls repository on the summit-pace-mac branch. |
| Intervention timing | 52% fewer interventions in offline replay (Quiet cost setting, 3-correct-in-a-row mastery definition) | We evaluated the policy using held-out ASSISTments 2009–10 logs. Interruptions were weighted three times as costly as delayed help; reversing those weights reduces the saving to 3%. | The policy missed 3.7% of students who went on to get stuck (95% CI 2.2–5.3%). Results and protocol are on the ARIA-Pace page. Replay does not establish that an intervention works. |
| Synthetic stress test | 84.6% same-generator synthetic accuracy | The balanced score is 0.837; the cross-generator gap is 9.05 points. | These results are useful for debugging, not for establishing learning benefits. |
| Student outcomes | 0 completed controlled studies | Feasibility, learning, retention, and transfer remain untested with students. | We do not claim that ARIA improves learning outcomes. |
Reading the language
Four terms that prevent inflated claims.
- Development benchmark
- A development benchmark tests for software weaknesses. Synthetic benchmark scores are not evidence that students learn more.
- Human ground truth
- Human ground truth consists of labels created independently by trained people using a locked codebook, without seeing ARIA’s prediction.
- Active control
- An active control uses the same tasks, interface, time, and base model, but does not include ARIA’s learner-conditioned pipeline.
- Transfer
- A student uses planning, checking, or self-correction on a new task while ARIA is absent.
Sources behind the design
Evidence for the method, not proof of the product.
These sources support how ARIA should be studied. They do not establish that ARIA is effective.
- Observable self-regulated learning
- Human coding of planning, monitoring, evaluation, and related think-aloud activity.
- Human evaluation of AI tutors
- Mistake identification, guidance, and actionability evaluated with human labels.
- Grounded tutoring dialogue
- Teacher-authored scaffolding and documented risks of incorrect feedback or answer revelation.
- Metacognition guidance
- Planning, monitoring, and evaluation taught explicitly inside subject learning.
- What Works Clearinghouse standards
- Randomization, attrition, baseline equivalence, eligible outcomes, and study confounds.
- Research with children
- Institutional review, parental permission, affirmative assent, and risk requirements.
Limitations
What ARIA does not know yet.
- The 30 demo problems have not yet been approved by independent educators.
- The observable-move baseline has not yet been tested against real human labels.
- The paired response conditions have not yet received locked, blinded ratings.
- Timing has been tested on logs, not on students. Replay does not show that helping at that moment works.
- The cost of an interruption is an assumed weight in an offline simulation, not a measured harm or benefit.
- The 52% is under the ASSISTments 3-correct-in-a-row definition. Cross-validated, the quiet saving is 7.1% to 41.6% under the four other definitions we tested (50.1% under 3-correct-in-a-row, with 6.2% missed help; missed help is about 4 to 6%, 3.7% on the locked test).
- The live demo uses one pooled parameter set rather than the per-skill parameters the evaluated policy used; on validation this changes about 14% of Quiet wrong-step decisions, so the demo approximates the evaluated policy. Response time was tested and did not clear the pre-registered bar.
- The timing model was fitted on ASSISTments math logs (grade level not stated), not calculus, and needs re-calibration before deployment.
- The language analyzer was evaluated only on synthetic messages; the ARIA-Pace gate and its published results do not depend on it. In the demo its flags (they need an explicit feeling word) can add an affect-based reply or prompt a reply to a words-only message, but never override correctness.
- ARIA currently works on typed math, not handwriting or photos.
- No student study has established usability, learning, retention, or transfer.
- Synthetic benchmark performance does not establish real-student understanding.
- ARIA is an educational prototype, not a diagnostic or clinical tool.
Evidence base
Built from learning science and rigorous evaluation standards.
The design draws on observable self-regulated-learning coding, human evaluation of AI tutoring, grounded tutoring dialogue, metacognitive transfer research, and What Works Clearinghouse standards. Those sources justify the study design, not ARIA’s effectiveness.