ARIA is presenting at the Khan Lab School AI in Education SummitOctober 24 · Mountain View
Teacher activity

ARIA-Pace · research project page · October 2026

When should a math tutor stay quiet?

A timing rule for a math tutor, evaluated by replaying the logs of 597 held-out students. An offline replay, not a classroom trial.

fewer skill sequences hinted, Quiet setting, against hinting after every wrong step
52%
missed help: 19 of 518 stuck sequences (95% CI 2.2–5.3%)
3.7%
held-out test students, evaluated once
597

A second locked evaluation has also been run once. Its numbers are withheld until the dataset terms are confirmed.

1 · Summary

The short version.

When a student writes a wrong step, should a tutor speak right away or wait? ARIA-Pace is a timing rule: a knowledge-tracing model estimates the chance the student is getting stuck, and the tutor gives a hint only when that risk passes a threshold set by how costly an interruption is compared with a late hint. On a locked test of 597 students, the Quiet setting hinted 52% fewer skill sequences than hinting after every wrong step, and missed help in 3.7% of stuck sequences (19 of 518). This is an offline replay of logged data, not a classroom trial. It cannot show how a student would have responded to help, and we make no claim about learning.

2 · The question

Interrupt now, or let the student keep working?

ARIA treats hint timing as a just-in-time adaptive intervention (JITAI): at each decision point the system chooses between acting and waiting, based on the state it can observe. Help that arrives too early can cut short useful struggle. Help that arrives too late can leave a student spinning their wheels. Koedinger and Aleven (2007) called this the assistance dilemma.

We make the trade-off explicit. Each unnecessary interruption has a cost, each opportunity of delayed help has a cost, and the three settings are three ratios of those costs. They are cost settings, not types of student.

Table 1. The three cost settings
SettingInterruption : delay costHint threshold
Quiet3:157%
Balanced1:119%
Supportive1:39%

3 · Data

Public ASSISTments logs, split by student.

ASSISTments 2009–10 skill-builder logs, corrected and collapsed: 346,860 rows from 4,217 students, public and de-identified. We collected no student data. Rows with no skill tag (63,755, 18.38%) were dropped, leaving 64,818 sequences, where a sequence is one student’s work on one problem set. The data are math logs; the sources do not state a grade level.

Table 2. Split by student, so no student appears in two splits
SplitStudentsUsed for
Train2,498Fitting the risk models
Validation833Calibration and thresholds
Test832One locked evaluation

The test evaluation covers the 597 test students with at least one in-scope sequence: 3,468 skill sequences, 570 of them labeled stuck.

The “stuck” label

A sequence is stuck (wheel-spinning) if the student has no 3 correct answers in a row within their first 10 tries. The logs record this directly, so nobody labeled anything by hand.

Following Wang et al. and Kai et al., which attribute the idea to Beck and Gong (2013).

4 · Model

A step, a check, a risk, a decision.

  1. 1Student writes a stepOne step of the problem, typed as math.
  2. 2SymPy checks itRight or wrong. A right step gets no reply.
  3. 3Stuck risk updatesBayesian Knowledge Tracing, calibrated.
  4. 4Hint, or waitIf the step is wrong and the risk passes the setting’s threshold, hint.

The risk model is Bayesian Knowledge Tracing (BKT), fitted per skill on the training students with a pooled fallback for skills it has not seen. Stuck risk is the calibrated probability of not mastering the skill by try 10. On validation data, deep-learning models (DKT, SAKT) did not beat BKT at predicting stuck sequences, in single-skill or multi-skill versions (single-skill: aria_pace/results/dkt_val.json:28 and aria_pace/results/sakt_val.json:17; multi-skill: aria_pace/RESULTS_v3.md:8), so BKT stayed.

Thresholds were chosen on validation students only, then frozen. The three settings weight an unnecessary interruption against delayed help at 3:1, 1:1 and 1:3. In the replay, the dataset’s own correctness flag stands in for the SymPy check, which is evaluated separately in the live tutor.

5 · Evaluation protocol

Written down first, tested once.

  • Specification first. Labels, splits, metrics and cost settings were committed before any model was fit. Later changes are numbered amendments with dates. This is a repository commit history, not an external registry.
  • Locked test set, run once. Every threshold was frozen on validation. A lock file makes the evaluation script refuse a second run.
  • Paired bootstrap over students. Confidence intervals come from 1,000 resamples of students.

What the two measures count

Fewer hints counts the first hint in each skill sequence, compared with hinting after every wrong step. Missed help is the share of stuck sequences that had a chance for help but received no hint before try 10.

Hints are considered at tries 3 to 9, at most one counted per sequence.

6 · Results

Quiet roughly halves the hints and misses few stuck students.

We evaluated the policy once on 597 test students (3,468 skill sequences, 570 of them wheel-spinning). All figures use the 3-correct-in-a-row definition of stuck; the robustness section shows what other definitions do.

Trade-off between hints given and missed helpInterventions per student against the share of stuck sequences that were never helped, on the locked test. Always helping gives 2.54 interventions and 0% missed help. ARIA-Pace Quiet gives 1.22 and 3.7%, Balanced 2.15 and 0.2%, Supportive 2.46 and 0%. A tuned two-wrong-in-a-row rule gives 1.28 with 25.9% missed, a matched-rate random rule 1.22 with 33.2%, and an untuned three-wrong rule 0.70 with 56.6%.0.00.51.01.52.02.50%10%20%30%40%50%60%Missed help: stuck sequences never hinted before try 10Hints given per studentbetterARIA-Pace threshold sweep(test, descriptive)Always help: 2.54, 0%Two wrong in a row, tuned25.9% missedRandom, same rate33.2% missedThree wrong, untuned56.6% missedPFA thresholdRisk headSupportive: 2.46, 0%Balanced: 2.15, 0.2%Quiet: 1.22, 3.7%52% fewer hints than always helping
Figure 1. The trade-off. Hints given per student against missed help on the locked test. Down and to the left is better. The three yellow points are ARIA-Pace’s settings; the other markers are baselines scored at the Quiet setting. A tuned two-wrong-in-a-row rule uses about as many hints as Quiet but misses far more stuck sequences.Source: docs/aria_pace/RESULTS.md and aria_pace/results/test_results.json (mirrors aria_pace/figures/frontier_test.png)
Table 3. ARIA-Pace by setting, locked test
SettingHints per studentFewer than always-helpMissed help
Quiet1.2252%3.7% (19 of 518)
Balanced2.1515%0.2% (1 of 518)
Supportive2.463%0% (0 of 518)
Always help2.54–0%

The 95% confidence interval for Quiet missed help is 2.2% to 5.3%, calculated by bootstrapping over students. For Supportive the difference from always-help includes zero: abstaining saves little when delayed help is the costly side.

Table 4. Baselines at the Quiet setting
MethodHints per studentMissed help
ARIA-Pace (BKT threshold)1.223.7% (19/518)
PFA threshold1.442.1% (11/518)
Risk head (logistic)1.302.9% (15/518)
Two-wrong-in-a-row rule, tuned1.2825.9% (134/518)
Three-wrong rule, untuned0.7056.6% (293/518)
Random, matched to ARIA-Pace's rate1.2233.2% (171/518)
Always help after a wrong step2.540.0% (0/518)

ARIA-Pace is not distinguishable in cost from PFA. It is cheaper than always-help, the untuned rule and the matched-rate random rule at Quiet.

One student, six tries, three settingsTries 1 and 2 are right. Tries 3 to 6 are wrong, and the stuck risk is 15, 25, 46 and 70 percent. Supportive (threshold 9%) gives its first hint at try 3, Balanced (19%) at try 4, and Quiet (57%) at try 6.Try 1rightTry 2rightTry 3wrongTry 4wrongTry 5wrongTry 6wrongStuck riskchance of notmastering by try 10Quiet 57%Balanced 19%Supportive 9%Correct steps: no hint needed15%25%46%70%Quietfirst hintBalancedfirst hintSupportivefirst hintopen dot: waitssmall dot: later hint(not counted in results)
Figure 2. The same student under three settings. One student gets tries 1 and 2 right, then writes four wrong answers. Stuck risk climbs from 15% to 70%. Supportive gives its first hint at try 3, Balanced at try 4, Quiet at try 6. Same answers, different timing.Source: docs/DEMO_RUNBOOK.md section 6 (lines 133-150) and the ARIA summit poster; thresholds from aria_pace/results/frozen_val.json

7 · Robustness

The saving depends on what “stuck” means.

These checks use validation students and BKT only. They are labeled exploratory where the source labels them so, and none of them has a locked-test number.

Quiet-setting saving under five mastery definitionsCross-validated, exploratory. Saving versus always helping: 50.1% for three correct in a row within ten tries, 8.3% for four in a row, 26.4% for five in a row within fifteen, 41.6% for five in a row within twenty, and 7.1% for a running-average rule.0%10%20%30%40%50%60%70%Fewer hints than helping after every wrong step (Quiet)Missed help3 in a row, 10 triesthe definition used for the headline50.1%6.2%4 in a row, 10 triesstricter than the headline rule8.3%0.2%5 in a row, 15 triesstricter than the headline rule26.4%4.3%5 in a row, 20 triesstricter than the headline rule41.6%10.4%Running averageaverage ≥ 80% for 3 tries, 10 tries7.1%0.3%
Figure 3. Mastery-definition sensitivity. Quiet-setting saving against always-help under five definitions of stuck, with 95% intervals. Exploratory. Five-fold cross-validated on training and validation students, BKT only. The saving is 50.1% under 3-in-a-row and 7.1–41.6% under the four others.Source: aria_pace/RESULTS_v4.md, Step 2 (amendment A7)

Pooled versus per-skill parameters

The live demo uses one pooled set of BKT parameters; the evaluated policy used per-skill parameters. On validation (exploratory), the Quiet saving is 49.8% [46.5%, 53.0%] per-skill and 43.9% [40.6%, 47.0%] pooled, and 14.4% of Quiet wrong-step decisions flip. Missed help is 4.8% per-skill and 3.4% pooled. So the demo approximates the evaluated policy; it does not reproduce it.

Students who left early

Students who left a skill before try 10 have no label and are dropped from the main analysis. If they are counted as not stuck instead, the best Quiet threshold moves from 0.57 to 0.85 and missed help rises to 12.4% (95% CI 9.4%–15.5%). The 3.7% figure belongs to the primary assumption.

8 · Limitations

What this does not show.

  1. Offline replay. The logs contain no tutor interventions, so nothing here shows how students respond to being helped or interrupted.
  2. “Unnecessary” is a stand-in. It means the sequence was later mastered within 10 tries without logged help. A student may have mastered because of unlogged help, and earlier help might have changed the outcome.
  3. The data are from 2009–10. One public dataset from one platform and one student-level split, with no external replication.
  4. The demo uses pooled parameters, so it approximates the evaluated policy rather than reproducing it.
  5. No classroom evidence yet. We make no claim that ARIA improves learning, and the three settings are cost settings, not groups of students. No clinical or demographic claim is made.

9 · Reproducibility

Every number comes from a file.

Code, specification and result files are in the research repository. The validation-frozen configuration is pinned by a hash, and the test results record the same hash.

Research repository on GitHub

Frozen configuration hash (SHA-256)

057f4389adbc0163a54205607947ec3515be62d8cd6ca09f1bbbb4353a847dce

Dataset file hash (SHA-256)

162ef8d2d28bcbfea6591a282994062bd8d5eaa00636544292a0d268dca6e5da

Rerun

python -m pytest aria_pace/tests
python -m aria_pace.run_baselines
python -m aria_pace.run_policy
python -m aria_pace.freeze_val
python -m aria_pace.report

The test evaluation is not in this list on purpose: a lock file blocks a second run. Setup is in DATA.md.

10 · Cite this work

BibTeX

@misc{saravanan2026ariapace,
  author       = {Saravanan, Naren and Malireddy, Karthik},
  title        = {{ARIA-Pace}: When Should a Math Tutor Stay Quiet? An Offline Replay on ASSISTments 2009--10 Logs},
  year         = {2026},
  howpublished = {\url{https://github.com/NarenS31/aria.kls}},
  note         = {Student-led research project, Marvin Ridge High School. Not peer reviewed. Offline replay of logged data, not a classroom study.}
}