Research InitiativeSynthetic think-aloud data
Generate think-aloud examples to build and test a first version of the system. This shows the pipeline works end to end. It does not show that it works for real students.
Research roadmap
Five phases, in order.
Phase 1Complete Phase 2Current Timing model on real student logs
ARIA-Pace combines a knowledge-tracing model fitted on public student logs with a timing rule. We evaluated it on locked splits from 2009–10 and 2012–13. The second set of results remains unpublished until the dataset terms are confirmed. These evaluations replay historical logs rather than testing the tutor in a classroom.
Read the resultsPhase 3Planned Student interviews
We plan to interview students about where they get stuck in math, what help they have rejected, and how they want to interact. Ethics review and consent must come first.
Phase 4Planned Formative prototype testing
We plan to observe small sessions where students use the prototype on math work. We will use these sessions to identify prompts that misfire, annoy, or get ignored and revise them before making outcome claims.
Phase 5Planned Classroom and teacher-informed evaluation
We plan to design an evaluation with teachers in real instructional settings, using measures they consider meaningful. This stage would allow us to begin investigating learning outcomes.
Planned · Student interviews
From data we generated to students we listen to.
ARIA’s language classifier was built on synthetic think-aloud examples: text generated to imitate how students describe their reasoning. The timing model has since been evaluated on real student logs, but no student has yet used ARIA in a study, and the logs record behaviour, not what students wanted.
The next phase is structured interviews with students who struggle in mathematics, asking what happens when they get stuck and what kind of help has and has not worked. Interviews with minors require ethics review and parental consent, and will not begin until both are in place.
What we learn is meant to change the system, not just the write-up: which situations the policy distinguishes, how hints are worded and chosen, and what the interface asks of a student who is already frustrated.
What we ask students
- Where do you actually get stuck?Setting up the problem, a specific step, or checking whether the answer is right.
- What do you do next?Re-read, guess, ask someone, look up the answer, or stop.
- What help has failed you?Which tutoring tools felt useless, condescending, or too slow to be worth it.
- What would you actually use?Typing, tapping a choice, or speaking, and what you would never use in a classroom.
October242026
Khan Lab School AI in Education Summit
Intentional Innovation: Keeping Learning Human in an AI World
Naren Saravanan and Karthik Malireddy will present ARIA’s design, the ARIA-Pace results, and the limitations, and run a short activity asking teachers when they would step in. The aim is critique of the study design and, if a teacher is willing, a classroom pilot.
- When
- Saturday, October 24 · 8:00 AM to 5:00 PM PT
- Where
- Khan Lab School · Mountain View, California