Resolved is not learned

Two documents about AI tutors in higher education landed six weeks apart, and they sound like they disagree. They measure different things, and what each one can see is the subject of this edition.

In August, LearnWise, a vendor that sells AI assistants to higher education, published its 2026 State of AI-Powered Teaching & Learning report. It covers 191,283 tutor conversations across 56 partner institutions in 11 countries, collected from September 2025 to April 2026, plus 17,937 faculty-feedback actions, which is a serious data set. The headline number is a 99.4 percent resolution rate. The page defines it narrowly: only 0.6 percent of conversations "ended with the AI explicitly stating it could not help." The median time from a student's question to a working answer was three minutes (LearnWise, 20 August 2026). I worked from the report page and haven't seen the full report.

That's a standard operational definition: the tool didn't decline. It isn't a measure of whether the answer was right or whether the student understood, and the page I read reports neither. The full report may.

A University of Maryland working paper, dated 30 September and reported on 2 October, describes a randomized trial of a course-integrated GPT-4o tutor in Fall 2025. Instructors were assigned to offer the tutor or not, within blocks of the same or similar courses: 2,379 undergraduates and 30 instructors, with three assigned instructors who never switched the tutor on dropped from the analysis. Across all classes, students offered the tutor finished about three points lower, which wasn't statistically significant at the usual threshold. Among sections of the same course (1,353 students, 22 instructors), the difference was 4.12 points on the instructors' 0-to-100 scale, or 0.37 standard deviations. Recorded participation in instructor-designed online activities, meaning discussion responses, assignment submissions and quizzes, fell by 0.90 standard deviations, though the share of homework submitted didn't change (Liu, Sweet, Chen et al., EdWorkingPaper 26-1598; SSRN record; reported by EdTech Innovation Hub, 2 October 2026). It hasn't been peer reviewed; the EdWorkingPapers series circulates papers before review. This edition draws on the full text, not only the abstract.

Resolution says the tool didn't decline the question. It takes an instrument outside the tool, a gradebook at minimum, to say whether the work got better.

What the Maryland paper can and can't say

A few more details sharpen it. Only about 15 percent of students in the classes offered the tutor used it at least once, and users averaged about four sessions. Of 3,460 requests, 73.8 percent sought information, explanations or solutions and 11.3 percent involved practice and test preparation. Fewer than 1 in 50 asked for feedback on the student's own work, though an AI classifier did the sorting and was weakest on that category. Instructors could choose a direct-instruction mode or a tutoring mode that guides students toward a correct response, and only two ended the term in tutoring mode.

That's where the tidy story stops. The paper reports that lower final grades "were not confined to students who used" the tutor but were similar among those who didn't. Students in both conditions also had free access to other AI tools. In an unadjusted survey with an 18 percent response rate, and fewer responses from treated classes, 41 percent of treated-class respondents described their instructors as encouraging or embedding generative AI, against 11 percent in control classes. The authors offer two readings, that the tutor may displace engagement with existing learning activities and that its presence may have changed the wider AI environment in the course, and they say their findings concern this tutor as configured, not scaffolded tutoring in general. So I can't tell you the tutor's answers caused the drop, and neither can the paper. A resolved-conversation rate measured on this tutor would have shown none of it, and it couldn't have seen the 85 percent who never opened it, which is the point: that rate describes conversations, and the trial's effects reached beyond them.

Why the dashboard can't see this

A dashboard reports what the tool can observe about itself. It can count messages and resolutions. It can't see whether a discussion thread got read less because a tutor was there, or whether a student read an explanation, nodded, and couldn't do the problem alone the following Tuesday. Whether those things happen, and how often, is measured elsewhere, by instruments the tool doesn't operate: a gradebook, and a retention check weeks later. Course grades are an imperfect instrument too. The Maryland authors say so, since instructors set them and weren't blind to condition.

LearnWise is clear about what it counts. Its page says "Faculty are always the ones editing and approving what a student ultimately sees," and puts study practice at 35 percent of conversations. Those are the right measurements for what's being counted, which is tool use. The trouble starts when a conversation about usage turns into an argument about learning, and nothing on the page I read makes that argument. A buyer can.

A second trial, a different result

A two-year cluster-randomized trial of Khan Academy's Khanmigo in remedial math sessions across 18 Tennessee middle schools found something different. Khanmigo was set to coach rather than give answers. Measured gains were small, 1.3 national percentile ranks per term, and resembled the gains from Khan Academy practice without AI, so the study can't separate the tutor from the practice package it came in. Ninety-six percent of students tried the tool at least once, but the median student messaged it on only about a third of the days they practiced, and in just 17 percent of the exercise sessions where they made a mistake. The authors conclude that engagement appears to be the binding constraint (Oreopoulos and Low, NBER Working Paper 35620, August 2026).

The two trials ran on different ages and different tools, one found small gains from a practice-plus-tutor package and the other found losses, and I'm not claiming they add up to one finding. What they share is a measurement point. In both, you can only say what the tutor did by comparing against students who didn't have it.

What to ask when someone shows you a tutor's dashboard

Four questions I'd want in my pocket the next time someone shows a chart, as one input beside the pilot, privacy and accessibility reviews a purchase already goes through. They're mine, drawn from the research above and not tested in it. Edit them to fit your situation.

• What outcome did you measure, and whose instrument measured it? Resolution rate is the tool measuring itself. A gradebook or a weeks-later retention check is somebody else measuring the student, imperfectly. The second kind is the one that pays rent.
• How much did engagement with existing learning activities change? That data usually lives in your own learning management system, so it's a question for your team as much as for the vendor. A chart without it has answered a question about adoption and left learning untouched.
• Which mode did instructors actually use? A direct-instruction mode and a tutoring mode are different products with the same login, and vendor averages blur them. In the Maryland trial only two instructors ended the term in tutoring mode, which is useful to know before reading any aggregate.
• Whose outcomes moved, and were they the students who used it? In the Maryland trial, first-generation students' estimated grade losses were more than double other students' in both samples, yet they were no less likely to use the tutor. Vendors often don't hold subgroup data, and privacy rules may keep them from it, so the breakdown may have to come from your own records. Whoever holds it, someone should look before the contract.

How this was made. Full disclosure, and I mean it as a courtesy, not a confession. AI tools help me research, check my claims, tidy my grammar, and stay on top of far more reading than one person can. The thinking and the judgment are mine, I edit every word into my own voice by hand, and then I put my name on it. Telling you that is the whole point of keeping it human.

One question I'd like answered, and I don't have a good answer myself. If you've evaluated an AI tutor at your institution, what did you measure that the vendor's own dashboard couldn't show you? Something beyond usage or satisfaction. The specific outcome you looked at, and whose instrument measured it. My guess is the useful answers are scattered across a lot of people solving the same problem alone, and I'd like to hear a few.

Next edition: what AI feedback changes about the way a student revises, and what it leaves exactly as it was. The findings run in two directions.

These editions run alongside Learning to Learn With AI, a longer series working the same ground in order.

Common questions

What does a 99.4 percent resolution rate for an AI tutor mean? In the LearnWise report, it means only 0.6 percent of conversations ended with the AI explicitly stating it could not help. It isn't a measure of whether the answer was right or whether the student understood.

Did the University of Maryland trial show that AI tutors lower grades? Not in general. Students offered one course-integrated GPT-4o tutor finished about three points lower overall, which wasn't statistically significant, and 4.12 points lower in sections of the same course. The authors say their findings concern this tutor as configured, not scaffolded tutoring in general, and the paper hasn't been peer reviewed.

What should I ask when a vendor shows an AI tutor dashboard? Four things: what outcome was measured and whose instrument measured it, how engagement with existing learning activities changed, which mode instructors actually used, and whose outcomes moved.

References

• LearnWise. (2026, 20 August). LearnWise Education Report: The 2026 State of AI-Powered Teaching & Learning. https://www.learnwise.ai/news-insights/learnwise-education-report-the-2026-state-of-ai-powered-teaching-learning (Vendor report page, checked 2026-10-22. Source for 191,283 conversations, 56 partner institutions, 11 countries, the September 2025 to April 2026 window, 17,937 feedback actions, the 99.4 percent figure and its definition, the three-minute median, the 35 percent study-practice share and the faculty quotation. The full report behind the page was not opened.)
• Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J., & King Rice, J. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial. EdWorkingPaper No. 26-1598, version dated 30 September 2026 and listed October 2026. https://edworkingpapers.com/ai26-1598 (Working paper; EdWorkingPapers states its papers circulate before peer review. Full text read 2026-10-22 and the source for every Maryland figure in the body: the full-sample primary analysis (about 3 points, not significant at the usual threshold), the exact-match sample (1,353 students, 22 instructors, 4.12 points, 0.37 standard deviations), the three dropped instructors, the 0.90 standard-deviation participation result and unchanged homework submission, the 15 percent uptake, 3,460 classified requests and their shares, the mode count and the user and non-user comparison. The request shares come from an AI classifier the authors validated against human coding; the paper's Discussion misprints the feedback share as 7.6 percent, and its results section gives 0.7 percent.)
• Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J., & King Rice, J. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7546479
• Thompson, E. (2026, 2 October). Students offered an AI tutor got lower grades and disengaged from course materials, University of Maryland trial finds. EdTech Innovation Hub. https://www.edtechinnovationhub.com/news/students-offered-an-ai-tutor-got-lower-grades-and-disengaged-from-course-materials-university-of-maryland-trial-finds (Trade report, checked 2026-10-22 against the paper. It rounds the grade effect to about four percentage points and does not mention that the effect is in the same-course subsample or that losses were similar among non-users.)
• Oreopoulos, P., & Low, N. (2026, August). One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment. NBER Working Paper 35620. https://www.nber.org/papers/w35620 (Cited previously in editions 8 and 9. Abstract checked 2026-10-22 for the design, the remedial math setting, the 1.3 percentile-rank effect, the 96 percent, one-third and 17 percent figures, the coaching setting and the engagement conclusion.)