They are out there, and while the current crop of chatbot tutors takes advantage of what is possible now, they’re… not that good. There’s evidence online math help only works for the 5% of students who use it at the dosage intended. Some of the reasons chatbot tutors aren’t very good yet are evident and I’ll outline them here, along with reasons to believe these barriers will be overcome.
Reading. To interact with a chatbot takes a high degree of literacy. They just flat-out produce lots of text, and decoding all that text can quickly tax a learner’s brain, or disincline a student from engaging altogether. Voice bots are a thing that have struggled with latency because they have to do speech recognition, send that to an LLM, wait for a response, and then generate audio. This isn’t fast enough yet to feel natural, but we’re getting there.
Diagrams. A math tutor can only get so far without drawing a diagram, and learners are impeded from showing their thinking without being able to draw a diagram. Tutoring also often involves working out a picture collaboratively. The text-only chatbots are too limiting to be very useful. But GPT Vision exists and people are going to start building chatbots that can produce visuals and see what a student sketches on a whiteboard.
Reasoning and Getting the Math Right. It takes some engineering, but it’s already possible to hook up a function call to Wolfram Alpha to check the math of an LLM’s output. Also, models that are better at reasoning and planning are coming. I’m super interested to see how they are at getting math right. Anyway, hurdling this barrier is only a matter of time.
Knowledge of Student. Tutoring is better if the tutor knows what the student is good at, what they already know, what they especially struggle with, and what they are interested in. The atomic MVP chatbot tutors we see now are generally operating without this knowledge, but we’re going to see them integrated into systems that can give the tutor some context about the individual student. I would also put “memory of previous chats” in this bucket.
As these barriers come down, chatbot tutors will become more and more of a viable resource. To me the use cases are clear. Ask high school teachers about opportunities for interventions for students who need some help getting ready for a lesson, who missed school, need some understanding shored up, need more time with some idea, or need to approach an idea from a different perspective to understand it. Many of them describe all kinds of opportunities in the school day---helpers pushing into their classes, students being plucked from class for tutoring, students being assigned mandatory after-school or lunchtime tutoring, extra help on math during their AVID period, or the availability of an academic support room they can visit during study hall. The adult helpers and peer tutors and self-directed study in these spaces should have access to all kinds of resources, and we are going to start to see a chatbot tutor as a viable one. It might not work well for all students in all cases, and how many it works well for and under what conditions will certainly be studied. But I really believe that the naysaying we’re hearing now in spring 2024 is short-sighted, because it’s not based on the trajectory of where things are going.
Comments