Chris Berdik for The Hechinger Report
1 / 1How Well Are Students Engaging in AI?
Merlyn Mind's AI assistant, Origin, was piloted by thousands of teachers nationwide this past school year, including Thompson and three other teachers at the Ron Clark Academy. The South Atlanta private school, where tuition is heavily subsidized for a majority low-income student body, is in a brick warehouse renovated to look like a low-slung Hogwarts, replete with an elaborate clocktower and a winged dragon perched above the main entrance.
As Thompson moved among his students, he wielded a slim remote control with a button-activated microphone he uses to command the AI software. At first, Thompson told the AI to start a three-minute timer that popped up on the smartboard, then he began asking rapid-fire review questions from a previous lesson, such as what causes wind. When students couldn't remember the details, Thompson asked the AI to display an illustration of airflow caused by uneven heating of the Earth's surface.
At one point, he clambered up on a student worktable while discussing the stratosphere, claiming (inaccurately) that it was the atmospheric layer where most weather happens, just to see if any students caught his mistake (several students reminded him that weather happens in the troposphere). Then he conjured a new timer and launched into a lesson on water by asking the AI assistant to find a short educational movie about fresh and saltwater ecosystems. As Thompson moved through the class, he occasionally paused the video and quizzed students about the new content.
Study after study has shown the importance of student engagement for academic success. A strong connection between teachers and students is especially important when learners feel challenged or discouraged, according to Nitta. While AI has many strengths, he said, "it's not very good at motivating you to keep doing something you're not very interested in doing. The elephant in the room with all these chatbots is how long will anyone engage with them?" he said.
The answer for Watson was not long at all, Nitta recalled. In trial runs, some students just ignored Watson's attempts to probe their understanding of a topic, and the engagement level of those who initially did respond to the bot dropped off precipitously. Despite all Watson's knowledge and facility with natural language, students just weren't interested in chatting with it.
At a spring 2023 TED talk shortly after launching Khanmigo, Sal Khan, founder and CEO of Khan Academy, pointed out that tutoring has provided some of the biggest jolts to student performance among studied education interventions. But, there aren't enough human tutors available nor enough money to pay for them, especially in the wake of pandemic-induced learning loss.
Khan envisioned a world where AI tutors filled that gap. "We're at the cusp of using AI for probably the biggest positive transformation that education has ever seen," he declared. "And the way we're going to do that is by giving every student on the planet an artificially intelligent but amazing personal tutor."
One of Khanmigo's architects, Khan Academy's chief learning officer, Kristen DiCerbo, was the vice president of learning research and design for education publisher Pearson in 2016 when it partnered with IBM on the Watson tutor project.
"It was a different technology," said DiCerbo, recalling the laborious task of scripting Watson's responses to students.
Since Watson's heyday, AI has become a lot more engaging. One of the breakthroughs of generative AI powered by LLMs is its ability to give unscripted, human-like responses to user prompts.
To spur engagement, Khanmigo doesn't answer student questions directly, but starts with questions of its own, such as asking if the student has any ideas about how to find an answer. Then it guides them to a solution, step by step, with hints and encouragement (a positive tone is assured by its programmers). Another feature for stoking engagement allows students to ask the bot to assume the identity of historical or literary figures for chats about their life and times. Teachers, meanwhile, can tap the bot for help planning lessons and formulating assessments.
Notwithstanding Khan's expansive vision of "amazing" personal tutors for every student on the planet, DiCerbo assigns Khanmigo a more limited teaching role. When students are working independently on a skill or concept but get hung up or caught in a cognitive rut, she said, "we want to help students get unstuck."
Some 100,000 students and teachers piloted Khanmigo this past academic year in schools nationwide, helping to flag any hallucinations the bot makes and providing tons of student-bot conversations for DiCerbo and her team to analyze.
"We look for things like summarizing, providing hints and encouraging," she explained. "Does [Khanmigo] do the motivational things that human tutors do?"
The degree to which Khanmigo has closed AI's engagement gap is not yet known. Khan Academy plans to release some summary data on student-bot interactions later this summer, according to DiCerbo. Plans for third-party researchers to assess the tutor's impact on learning will take longer.
Nevertheless, many tutoring experts stress the importance of building a strong relationship between tutors and students to achieve significant learning boosts. "If a student is not motivated, or if they don't see themselves as a math person, then they're not going to have a deep conversation with an AI bot," said Brent Milne, the vice president of product research and development at Saga Education, a nonprofit provider of in-person tutoring.
Since 2021, Saga has been a partner in the Personalized Learning Initiative (PLI), run by the University of Chicago's Education Lab, to help scale high-dosage tutoring — generally defined as one-on-one or small group sessions for at least 30 minutes every day. The PLI team sees a big and growing role for AI in tutoring, one that augments but doesn't replicate human efforts.
For instance, Saga has been experimenting with AI feedback to help tutors better engage and motivate students. Working with researchers from the University of Memphis and the University of Colorado, the Saga team fed transcripts of their math tutoring sessions into an AI model trained to recognize when the tutor was prompting students to explain their reasoning, refine their answers or initiate a deeper discussion. The AI analyzed how often each tutor took these steps.
When Saga piloted this AI tool in 2023, the nonprofit provided the feedback to their tutor coaches, who worked with four to eight tutors each. Tracking some 2,300 tutoring sessions over several weeks, they found that tutors whose coaches used the AI feedback peppered their sessions with significantly more of these prompts to encourage student engagement.
While Saga is looking into having AI deliver some feedback directly to tutors, it's doing so cautiously, because, according to Milne, "having a human coach in the loop is really valuable to us."
In addition to using AI to help train tutors, the Saga team wondered if they could offload certain tutor tasks to a machine without compromising the strong relationship between tutors and students. Specifically, they understood that tutoring sessions were typically a mix of teaching concepts and practicing them, according to Milne. A tutor might spend some time explaining the why and how of factoring algebraic equations, for example, and then guide a student through practice problems. But what if the tutor could delegate the latter task to AI, which excels at providing precisely targeted adaptive practice problems and hints?
The Saga team tested the idea in their algebra tutoring sessions during the 2023-24 school year. They found that students who were tutored daily in a group of two had about the same gains in math scores as students who were tutored in a group of four with assistance from ALEKS, an AI-powered learning software by McGraw Hill. In the group of four, two students worked directly with the tutor and two with the AI, switching each day. In other words, the AI assistance effectively doubled the reach of the tutor.
Experts expect that AI's role in education is bound to grow, and its interactions will continue to seem more and more human. Earlier this year, OpenAI and the startup Hume AI separately launched "emotionally intelligent" AI that analyzes tone of voice and facial expressions to infer a user's mood and respond with calibrated "empathy." Nevertheless, even emotionally intelligent AI will likely fall short on the student engagement front, according to Brown University computer science professor Michael Littman, who is also the National Science Foundation's division director for information and intelligent systems.
No matter how humanlike the conversation, he said, students understand at a fundamental level that AI doesn't really care about them, what they have to say in their writing or whether they pass or fail algebra. In turn, students will never really care about the bot and what it thinks. A June study in the journal "Learning and Instruction" found that AI can already provide decent feedback on student essays. What is not clear is whether student writers will put in care and effort — rather than offloading the task to a bot — if AI becomes the primary audience for their work.
"There's incredible value in the human relationship component of learning," Littman said, "and when you just take humans out of the equation, something is lost."
