Why plan a disagreement
Teaching someone else is one of the most reliable ways to learn, and a teachable agent lets every student be the teacher. Comparing different solutions to the same problem can deepen that learning, but disagreement can also backfire: in our studies of student pairs, disagreements left unresolved predicted lower rapport. So this version of Emma doesn’t leave disagreement to chance. Once she understands the student’s solution, she offers a verified alternative, and after the problem is solved, she asks the student to compare the two.
How the conversation flows
I’m designing and building the LLM agent harness behind Emma, in Python. It runs on Azure OpenAI (Azure AI Foundry) for the study, and I serve open-weight models locally on vLLM for prototyping and offline experiments.
- The student starts explaining their solution
- Emma matches it to a verified solution, or asks a clarifying question
- Emma introduces a verified alternative and asks which approach to follow
- Emma goes through the steps, tracking what, how, and why for each
- Emma reflects back both solutions and asks which works better on a slightly different problem
Simplified flow for one problem.
Each LLM call returns one structured JSON object that contains both Emma’s judgment (for example, which solution matched and which parts of a step are covered) and what she says next, so the judgment and the response are generated together.
What made it hard
Two problems took the most work: knowing which solution the student means, and tracking each step.
Uncertainty about the student’s solution. Students’ first explanations are often incomplete, so an early match can be wrong. After Emma commits to one verified solution, I keep running the LLM classifier asynchronously during the first step of that solution, so it adds no latency. If the student’s later explanations fit another verified solution better, Emma switches the match.
Tracking what, how, and why. Students often explain several steps at once, or give the reason for a step before reaching it. So the LLM records what, how, and why for future steps too, not just the current one. When a new step already has all three, Emma simply asks a yes/no question, like “Should we do … next?” This significantly reduced unnecessary back-and-forth.
Running the study
The agent is deployed in an IRB-approved controlled study with 72 participants. I own the study infrastructure end to end: an ASP.NET web application, the Python dialogue service (migrated from rule-based AIML to LLM-based), Azure SQL, and the data pipelines, with participant data kept in an IRB-compliant institutional Azure environment, separate from other research compute.