transcribe

AI Makes Cheating Easy. Here’s How It Can Make Kids Smarter Instead | Sue Khim

Peter Yang · 45m · transcribed Sep 2026
More from Peter Yang Family and Health
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

The Role of AI in Education

How is AI currently impacting education, and what should its role be?

AI is often misused by students for cheating, which detracts from genuine learning. The goal should be to empower students to engage deeply with challenging material rather than providing easy answers.

  • AI should enhance learning, not replace it.
  • Students often feel AI makes them less intelligent.
  • Effective learning requires effort and engagement.
# 9:07

Personalized Learning with AI

What advantages does AI provide in personalized learning environments?

AI can offer personalized reviews and create a comfortable space for students to learn without fear of embarrassment, which is often present in traditional classroom settings.

  • AI tutors can provide individualized attention that teachers may struggle to offer.
  • Students feel more comfortable making mistakes with AI.
  • Personalized learning can lead to better engagement and understanding.
# 18:15

Human-Centered AI Design in Education

How does Brilliant ensure quality in its educational content?

Brilliant combines human-designed lessons with AI implementation to maintain high quality. Human educators design the pedagogy, while AI executes the delivery, ensuring effective learning experiences.

  • Human design is crucial for effective educational content.
  • AI should enhance, not replace, human teaching methods.
  • Quality control is essential for successful learning outcomes.
# 27:23

Onboarding and Learning by Doing

What is the best approach to onboarding new employees in a tech-driven environment?

Onboarding should focus on hands-on experience, allowing new employees to engage with the product and workflows immediately, rather than just reading documentation.

  • Learning by doing is more effective than traditional onboarding methods.
  • Immediate engagement helps new employees understand workflows.
  • AI can streamline the onboarding process by automating documentation.
# 36:31

Data-Driven Improvement in Educational Tools

How does data influence the effectiveness of educational products?

Unique data collected from user interactions can enhance the product over time. However, data from AI tools like ChatGPT may not reflect true learning outcomes, as students often seek quick answers rather than understanding.

  • Unique user data is vital for improving educational tools.
  • Not all data is useful; understanding learning processes is key.
  • AI should facilitate genuine learning rather than just providing answers.

Transcript

0:00 AI today is mostly used by students for cheating. We believe technology can and should be making our kids stronger, not dumber and weaker. >> Feel like AI is making me dumber, too, cuz it's just easier to just ask it to do stuff than to actually learn. >> This is an actual playback of a real tutoring session on Brilliant. What's much more surprising is that we try not to offer explanations either. Explanations feel good, but you're not doing the work yourself. If we truly have super intelligence, we should get learners to stick with something hard and actually use it themselves and get the students to do better thinking.

0:35 >> Hey everyone. Today I'm really excited to welcome Sue, the co-founder and CEO of Brilliant. Sue has been thinking about AI and education for over a decade. And just as a parent, I'm really excited to talk to her about our education system and the best way to use AI to help kids learn without just giving them all the answers. So, welcome Sue. >> Thanks, Peter. It's great to be here. >> Yeah. I I really love how Brilliant is affordable and very approachable. Why don't we start with this? Like I'm curious what your thoughts are on kind of just like the overall US education system. Like where do you think is doing well and where do you think it needs improvement?

1:07 >> Yeah, that's the US education system is really interesting. High highs low lows. I think one thing that's uniquely good about the US system relative to a lot of other places is that there's more freedom for experimentation. So you know you see new home schools and experimental schools everywhere with very talented people working on them. And so we do have the conditions for new solutions. The terrible thing that's happening in the US is this phenomenon of test scores are down but grades are up somehow. Kids can't read or do math anymore. And we hear all the time from colleges that used to have strong STEM degree programs and they're now looking at remedial algebra solutions for college students. I mean, we're talking about middle school level math. And you know, some of the reporting on this is that part of how we've been dealing with worse performance is by lowering standards, meaning we've introduced grade inflation. And so we are changing the standard so that more students can get an A instead of fixing the underlying problem. And obviously if one school does it then of course other schools are also going to feel pressured to do it too cuz you don't want your school to be the only one where a lot of students are getting seized. And so this is sort of happening behind the scenes.

2:24 I think it's pretty opaque to parents and doing this grade inflation works for a while to keep the parents happy but obviously it is the worst way to deal with actual falling performance. So we are in this real you know learning trough and I think it's going to take time to get ourselves out of it. But conversely, I think that the super intelligence that we're building at these AI labs also is a real opportunity. Like we think that if it's used in the right way, AI can make our kids into geniuses. It's like a sea change like how the invention of writing suddenly allowed everyone to have infinite memory.

2:58 >> Mhm. Do you think I feel like that for schools it comes down to incentives and and maybe their funding is based on like is is it based on how many students can get high grades or something or >> I certainly think that you know teachers are measured to a degree on how I mean it's obviously like very different from different schools like charter schools are different public schools are different yeah >> you know private schools are a different beast home schools have their own standards you know I I think that what they are first order responding to is parents being upset and you know because the standardized tests are also getting easier. One very easy way to appease the parents is by also you know lowering the standard at the school to match because they can do that. And so we have this phenomenon where on the standardized tests you have lower performance just on an absolute basis. And that's masked by the fact that you know at the school we've also lowered the standards. And really the only place that this is coming up is internationally graded exams where you can see the delta relative to years past.

4:02 >> I see. And so like personally I went to grade one in China. I grade one in China. And then I remember like that grade one was intense like I was learning tons of math trying to copy like the all the mandering and I remember getting like straight C's or something in in grade one. And then I went to school in Canada. I feel like even in Canada, it was like pretty much easy coasting all the way up to like grade grade seven, >> especially after taking grade one math in China.

4:26 >> Yeah. >> But but but I feel like I said just like trade-offs. I I feel like some of the Asian countries, they kind of teach kids to just memorize stuff and like just grind it out. And I think one good thing about like the schools here is like, you know, you have more free time for better for so so you can explore things your explore your own interests. I'm I'm I'm just curious like have you had kind of like a similar experience? Did you grow up here or >> I did grow up here. I mean, I've also gone to school in Korea and so I've had a few and I moved around a lot as a kid. So, I've had a lot of different educational experiences.

4:58 School is very long and I think the hours are really more than you need for the amount that you are actually learning in that period. And schools serve other functions in society besides teaching kids things. And you know to think that you know I I also shared your experience where you know it was like pretty boring in school a lot of the time and to think that like now we have made all of the tests a lot easier even since then is is pretty scary to think about. You know, I was reading this data that came out recently about the AP English exam and how, you know, it's an AP. So, it used to test undergraduate and graduate level material and then it slowly drifted down to, okay, we're testing high school level reading passages and now we're apparently in the last two years we're testing middle school level passages and it's like we've just given up. This episode is brought to you by linear.

5:47 When engineers use tools like cursor, clock code, and codecs, a lot of work happens invisibly. Someone can go from a bug report in Slack to a shipped fix without creating any record of what happened outside of the code editor. And that's fine for speed, but it makes coordination harder as you scale. Linear integrates with the very best agent coding tools directly like cursor and codecs. That way, anyone can see what an agent is working on and who assigned them to the task. You get the speed of agents without losing visibility across the team. product teams at OpenAI, RAMP, and Block are all using Linear to collaborate with AI agents. And I use Linear myself to run my creator business. So, check it out at linear.app/ aents. That's linear.app/ aents. Now, back to our episode. Yeah, I guess just as an Asian parent, like like a tiger parent. I try not to be a tiger parent, but it's it's it's hard. like I I feel you got to supplement in some ways with like you know just like teaching your kids your yourself, right?

6:48 And we're going to get to the AI tutors soon, but like I'm curious if you have any thoughts about what kind of learning works best in the classroom environment versus like one-on-one tutoring versus like, you know, hands-on stuff. Have you thought about this? Cuz I feel like 25 kids in a classroom, unless I'm doing like a school project with my classmates, it's kind of boring just listening to your teacher draw on about stuff. >> Yeah. >> Yeah.

7:09 >> Yeah. So classrooms are this unique environment. You have a teacher, you have a lot of peers. And I think a really great use of classroom time is peer learning and interaction with a teacher who can motivate people and inspire everyone in the class to learn. You know, enthusi we are humans. Enthusiasm is infectious. The best teachers are creating excellent collaborative learning environments and they are devising ways for peers to learn a lot from each other and that's really powerful. But in most schools that is not how classroom time is used the majority of the time and so you know you're spending most of the time delivering lectures and there's a kind of instructional delivery and practice that happens through lectures and the kid just actually practicing that I think is not the best use of classroom time and is not the best use of a human's special abilities. like people don't typically get into the teaching profession so that they can stand at a chalkboard and write stuff on there and then watch students just stare at their paper. so so there's this like kind of individualized instruction reporting personalization that you know you can get some of from a human tutor who you know personalizes and tells the parent what's going on and a great deal more from software. You know, this is a lot more than what a teacher can provide to every student in a large classroom. And I think software on many axes is going to exceed what you can get with private human one-on-one tutoring. You know, obviously on cost and convenience, software is on a whole different order of magnitude. Human tutors are very, very expensive and so most people are priced out. And then you know for features like if you are working one-on-one with the student in a software environment you get things like real-time graphical interaction you can adapt the behavior on the fly to the student as they're problem solving. You are able to do that based on a huge amount of data and reps on how kids best learn a particular concept. You can you know generate personalized review. And a thing we hear a lot is also like there's no fear of embarrassment or failure. So you're comfortable, you know, trying and trying again. Like most kids would not ever raise their hand in a classroom.

9:23 Like just the thought of that is horrifying. But, you know, with an AI tutor, like kids are very comfortable doing that. >> Interesting. Yeah. Yeah. I I think if you have 25 kids in a classroom, it's just kind of hard to have all the personal contacts on each kid and like give each kid your personal attention, right? >> Yeah. Yeah. It's impossible. I mean, we're already asking teachers to do so much. >> Yeah. So, let's talk about that then.

9:42 So, so the mission of Brilliant is to bring a world-class tutor to every home, right? >> Yeah. >> Do you want to like talk about what that looks like and maybe you can show us how the product works too? >> Yeah. Yeah. You know, when we say worldclass tutor in every home, there's some core philosophies underlying that. So, for example, we believe technology can and should be making our kids stronger not dumber and weaker. And you know, AI today is mostly used by students for cheating. And one of the analogies that we say is using AI as a way to skip the part where you learn is like bringing a robotic arm to the gym to lift weights for you. Like we're really losing the plot of why you went to the gym in the first place. And so what we built our AI tutor Cooji to do is to use its own intelligence to protect the time and effort that the student spends thinking for themselves just like a great teacher would. And maybe what would be helpful is for me to actually just show you. Yeah, I feel like it's not even just the kids. I feel like AI is making me dumber, too, because it's just easier to just just ask it to do stuff.

10:47 >> It's so It's so seductive. It's so seductive. Okay, so let me go ahead and play. So, you know, we this is an actual playback of a real tutoring session on Brilliant. and I think it's just instructive to see, you know, how someone gets stuck and what we're doing there. and so you'll see the user checking the answer. it's not correct and then they're going back and forth with the tutor Cooji. And you know, one of the things that surprises people about Brilliant is I think everyone knows that just giving the answer doesn't work for learning. But what's much more surprising is that we try not to offer explanations either.

11:29 Like explanations feel good, but you're not doing the work yourself. And so what we do is try to put the learner through the harder work of doing the work themselves. And so you know in this interaction the student and Koji are going back via voice. You know for privacy reasons we not share that. but this is a student trying to learn fractions by shading parts of a shape. H they need to make 51 12ths by shading in parts of the shape. and they shaded in five different parts which makes intuitive sense but the thing is that the shapes are worth different amounts and that is the insight that we want them to discover here. So here the learner understands the five but not the 12th. And so not only do we not want to give the answer, we don't want to just tell them h and just give that explanation that I gave you. You know, it's the most natural thing for you to want to do as a teacher or as a parent that's like hovering over your child.

12:28 And it's actually not the most effective tutoring intervention. Like that explanation only makes sense if you kind of already understand it. And you know, one of the things that we have built into the product is that is this idea that cheating and explanations are closer than you'd think. Like they are both a way of letting the learner off the hook. And we want the learner to actually sit there in the struggle of why is this wrong? Figure out the insight themselves and realize how far they can go in thinking through the problem themselves. And you know, if we can get them there, that is such a more powerful learning session than if we had just tell told them, oh, hey, this is why it's wrong.

13:09 >> Yeah, that makes a lot of sense. Yeah. But I I feel like I feel like building this product, you kind of have to balance the teaching and the motivation. >> Totally. Totally. Like cuz if if it's just I I guess a couple things. I was watching my kid you use the software, right? And and like in the beginning it was pretty easy. So she didn't have to talk to the AI at all. She can just like answer all the questions. And then at some point it got harder. And then I I told her Yeah. I told her that there's this avatar on the left that you can talk to and yeah, it kind of walked her through the steps and kind of like it didn't actually explain the answers to her, but it kind of asked her like a bunch of prayerful questions to kind of try to help her get to the answer.

13:42 >> Yeah. >> Yeah. Yeah. Yeah. >> But I think it's kind of like just as a building this product, how do you think about keeping the kids going without like making it too easy for them or making it too hard for them or like you know, how do you think about that? >> Yeah. So, you know, there are both AI and nonI interventions for that. So in the breadand butter building of the product, there are things that we do around pacing and feedback and hints.

14:04 You know, as you noticed in the onboarding, she didn't really need to fall back to the tutor at all. She's just like kind of building up reps, figuring out, okay, how does this all work? You know, what is a lesson? Oh, I, you know, solve these problems and get feedback. And it's getting a little bit harder every time. And, you know, it also helps to keep the learner just actively doing something. you know whether that's manipulating things or making predictions or testing ideas and getting feedback from the UI and you know it's just all these little interactions that then build up to the student saying oh wait I I see it like I get it now and then once you are struggling then on the AI side there is Cooji's personality and pedagogy like how he responds when you're struggling and that is one of our most active areas of work right now so you know a Good tutor is constantly making decisions about the student's next move. So, it's not just can I explain this concept.

14:58 It's what is the student confused about? Are they ready for a hint? Did they make a small mistake or is there, you know, something bigger that they're missing? You know, should I simplify the problem? Should I ask them to draw it? You know, do I just let them struggle for another minute? And you know, as you rightly point out, it really is a balance between the student getting so frustrated that they drop off and then they're act they're not going to learn it at all. and actually doing the thing that is the most effective thing for that student to learn. And that judgment and pedagogy is the trade-off that we are building into the product.

15:33 You know, it's very early. We've got a long way to go, but we ran the beta for over six months and you know we've been live for a few weeks now and we are seeing improvement every month in learners sticking with it more and it's our topmost priority. Like the thing we are asking learners to do on Brilliant is stick with something hard. >> Yeah, >> it is absolutely like worse in the short term for engagement. Like if we just gave people the answer or like a you know great hint that like totally unlocks the problem for them when they're stuck, people would do more and they would probably get way less frustrated. But I think that's like a local maximum that you reach with the product. And if we truly have super intelligence, we should get learners to stick with something hard and actually use it themselves and you know get the students to do better thinking.

16:17 >> Yeah. It's like the same incentives as kind of like the US education system we talked about earlier, right? Like it's probably easier to just like help your students all get A's and like >> easy. Yeah. And as a teacher, it's like, "Oh, man." Just you know, and I have kids, too, and I just want to tell them like that it would be so much easier. Like, they're about to give up. I just want to tell you. and so I really think like so much of the magic of a great human tutor is exactly how you exercise that judgment and keep the kid going and and basically extend the amount of time that they can focus and concentrate.

16:50 So, so let's talk about like how the product works behind the scenes as as builders like so so I so I guess like the default experience is like you have a problem and then you can ask Cooji about the problem, right? Or like you can have a conversation with it. >> so is it just as simple as that or like what what what are some product principles that that you have to kind of make this thing a good experience?

17:12 >> Yeah, so the userfacing UI is very simple. the what we do under the hood. There's a lot and I came with a couple demos to to show you. but I think we can boil down the you know what I'm going to show to three things. one is that AI models are very spiky. And so it's been really important for us to use AI for what it's good at and not what it's bad at. And I can talk a little bit more about that. second thing is the agent interface is just as important as the user interface when you're building products today. and the third is what this means is everyone's job is now to define an agent interface for their area. So I can actually I mean that that all sounds very abstract when I say it like that. So let me let me pull up my screen again and show you. Okay. So here I'm going to show you a specific example. oh, let me okay of how brilliant was built with agentic workflows. So this is an example of a video showing a learning designer who has sketched out a lesson for teaching the coordinate plane. You know when we've tried to use codeex or claude to simply generate this content h even after training it on our entire body of pedagogical interventions and content you know it looks almost right but it is actually nowhere close to the level of quality control on the output that we get when a master teacher designs it.

18:43 And people can tell you know the kid struggles, they get super frustrated by the sequence, like the teaching is a mishmash. And so the core story or pedagogy for every single concept on Brilliant is all designed by humans. And then what you're seeing here is that it's implemented by AI. So the human is designing the lesson and then handing off to the AI to implement the design. And what we're doing here is equipping the LLM with a lot like there's an entire product around it in order to make a good teacher.

19:18 >> So the human design is actually each question and and also the order and that the LM executes. >> Correct. Correct. and that's, you know, taking into account again like what are models good at, what are they bad at? And, you know, one belief that we have is that companies that are selling an experience that is under the hood driven end to end by an LLM are going to disappoint their customers. Like it's really appealing to just let the LLM swallow your entire product.

19:44 Like your users talk to the model, the model goes off and calls other models which then call other models and then they magically solve the user's problem. And I think general solutions like this feel really seductive. But you know the issue is that any job that you're trying to do for the user in our case tutoring is made up of a dozen different tasks. The models are very unevenly shaped. And so they're amazing at some of those things like this implementation task that I just showed you. And they're terrible at others like designing an excellent pedagogical learning experience. And so those tasks that it's bad at are going to ruin the whole experience. It's like and you see this in coding too. The models are excellent at writing working code. We use them a lot for that. They are not so good at architecture and that becomes the bottleneck on how effective they're going to be. And you know what we've seen is that the models for our use case are really bad at creating engaging interactive learning experiences. Like we can't let hallucinate that part. And I really liked what Bology had to say about this which is that AI is a middle-to- solution not the endto-end solution. And we found that this is like a super useful way to think about, you know, integrating AI into building products. Like if you can limit the role of AI to what it really shines at, that's when it feels the most magical.

21:01 >> It's like it's Yeah, that's my experience too. It does really well in the middle 80% execution. >> Yeah. >> But then the the planning you have to be involved and like the verification got to be involved at the either. >> Totally. >> Yeah. Yeah. you have to really constrain the use of what AI does in the product or you know your product experience is going to be governed by the thing that the AI is bad at because that's what people are going to notice and >> maybe that that's where the secret sauce is cuz like you know I I could probably use cloud and generate like you know 200 lessons too right >> yeah exactly >> about coronets and stuff but like you know I think I I think just got to carry enough to actually review his work and make sure it's actually good instead of >> pump slop yeah >> yeah it's so easy to pump out slop. And I think that like the, >> you know, it's like going to get worse before it gets better. There's a lot of edgy slop out there right now.

21:50 >> Yeah. >> and I think that, you know, for anyone building education products, what I would say is that people can tell. You might not think that they can tell, but they can definitely tell. And you see it in the engagement metrics. >> Interesting. >> Yeah. People just get frustrated and drop off when they're having to read slop. >> Yeah. Even Even the kid kids, right? Yeah, can tell. >> Yeah, they can totally tell. >> Yeah. And I feel like if you have enough slop in your product, at some point it becomes like the whole product becomes slop and you don't even know what what's going on anymore. It's like >> which parts are actually human craft and which parts are slop, you know?

22:24 >> Totally. >> So let's let's talk about the second one though. What do you mean by designing for agents? Because the user is still the kid, right? Or like what do you mean by designing product for a agents? >> Yeah. So the user is the kid and you want both an agent UI and a user UI. So I can I can go ahead and show you you know what I mean by agent UI. But before I do that like in brief when you are building an AI native consumer product obviously you want an amazing user interface. You want delight and ease of use and you know visual product interventions. You want it to feel you know delightful and intuitive. And you know we did this across all of our learning apps but equally important now is the agent interface. So these are the tools and actions skills the harness that the model needs to have to match its intelligence to what's important in the product. So let me show you something that is our primitives library. when we are building an interactive app for teaching quadratics we built the UI to be awesome but under the hood for each of these so you can see tons and tons of primitives on the left rail here each of these there is also an API that a model can write to to create a problem to annotate a problem to highlight or ask a sub question and so these primitives can talk to each other and talk to an LLM. And so these were all intentionally designed so that the model isn't hallucinating what to do. And this is how you get really good combination of latency and accuracy and reliability because there is this sort of deterministic grounding. And so, you know, designing this agent interface for our universe of learning applications makes it much easier for a learning designer to, you know, sketch a sequence with an agent. And basically, this library of primitives with the agent interface helps power the agentic workflow that I just showed off previously. like the agent can grab the learning designer sketches, use the graphing module that an engineer built, assemble the lesson following the guidelines from our learning design team and upload it. and you know, you like this kind of speed up is like part of the magic of AI because just to cover algebra 1, we had to build thousands of modules and we had to build an interface for the model to read and write to based on what the user is doing in the UI in real time. And so, you know, this is what the AI is good at, like taking in that multimodal input and reasoning what the right next thing to do is and avoiding the thing that it's bad at.

25:07 >> So, the primitives are kind of like components that that >> Yeah, it's like modular composable components. Yeah. >> Got it. And so, so then they kind of put some guard rails for the AI to make sure to use the components to design lessons with the human, right? Such generate stuff. >> Yeah. And so, you you know, you you're not hallucinating the whole interactive from scratch. You're not hallucinating all of the misconceptions or tutoring interventions from scratch. Instead, you're providing the model with a a schema that it can navigate very quickly to figure out based on like what I've ingested as multimodal input from the user, what is the correct multimodal output, but it doesn't have to hallucinate all of the multimodal output. Like if you did that, it would take a long time and it wouldn't be as accurate.

25:50 >> I see. Okay. Yeah. Cuz these elements like structure day, right? They like like they like little boxes and stuff to stick to. Yeah. >> Yeah. Yeah. If you can if you can provide more structure so that they can do less thinking, it you know has a lot of benefits for your product. >> Okay. Cool. what about the what about the third one that you mentioned like I think some something about the user interface hasn't been invented yet or >> Oh. Oh. So this is about how everyone's job is defining an interface for their area. So this is a way in which like work at Brilliant has really changed.

26:23 in that you know everyone who's designing for users is also in some sense also designing for agents like you have to consider both parties and the way that we think about AI is like the glue or the connective tissue you know we are building an ecosystem of interactive learning tools that AI works really really well with and you know your job is to make it so that any person at Brilliant with an agent can get good information out and make good changes so that you know people with agents can navigate, you know, this whole network of things with a lot more autonomy. So, you know, one goal that we have is to make it so that someone on day one at the job can take advantage of everyone else's expertise because it's already baked into the system. You know, how do we teach? What tools do we already have? What do we use AI for? You know, what do humans do and why? And then you can bring the special skills that you were hired in for and already have a you know sort of comprehensive set of interfaces and documentations for agents to help you navigate the whole network of things that we have.

27:25 >> Okay. So this one is more about like employee productivity I guess. >> Yeah. It's about like how we work you know what are you thinking about day-to-day you know are you building something or are you building the machine that builds the thing? And I think more and more you know the way that people's brains around their work have shifted is you know we are building the machine to build the thing. >> Yeah that that's really interesting because normally before all this AI stuff like when you onboard an employee they have to read a bunch of documentation right they got to read a bunch of wiks they got to have a bunch of one-on- ones to onboard. Yeah. I mean, and and then you have to write the wikis and the one-on- ones. You know, I I think one of the best inventions of innovations of AI is the wiki that writes itself and kind of plugs into your workflow. And, you know, that used to be a huge task to keep all that stuff up to date. And now, you know, we don't have to do that anymore, which is amazing. So, so how do you so, so like when you hire somebody like you know how do you give them like do you just be like hey here's kind of all of our internal skills and workflows and like you know like how how do you kind of onboard them to this >> whole and I think it's different for different domains but I I would say one core thing that's true across all the domains is you learn by doing like we really still to this day try to have technical people ship something right away because that requires you having everything installed and working and like really seeing from end to end like how does something go from idea to executed for the end user. You know similarly if you are joining as a learning designer you know our entire learning design team at this point is quite technical and we are you know asking you to be able to use the tools well enough to actually ship something to the end user. So you know exactly what the onboarding steps are you know what you have to read what you have to learn to do that change dayto day but I think ultimately like we've done a good job if it is efficient for you to make an impact that makes the product for users on day one >> okay so actually ship something on day one >> user facing yeah >> yeah I mean we're we're imperfect but but I think philosophically you know that is what we want everyone to do >> let's talk about like how do you make sure let's say you hired me as a learning designer right Yeah.

29:35 >> And and I make like a math lesson. How do you make sure that like the stuff is up to quality? Like do you have some evals that you run or like you royal to some percent of users first? >> we do have comprehensive eval. So all of those like primitives that I showed you. There are evals that run on them. It's not just eval correctness. It's eval obeying the laws of physics. You know, not having areas of visual overlap, having generous enough tap targets. like we eval across a bunch of different axes but one of the things that we I think will be doing indefinitely is that there are multiple human reviewers for you know everything that we put out and you know learning designers are hired for their you know particularly their craft and judgment and taste on is this good so that's the first order like before we put something out there are humans judging whether or not it's good there's evals that run we also have synthetic students that we put into, you know, whatever a thousand tutoring sessions in that lesson to check the tutoring interventions and see, you know, is this going off the rails, like, you know, are the tutoring interventions like sensical for all these different paths that users can go down. it's nowhere near as good as like real human data. but it is, you know, it's still a useful exercise to do to catch any, you know, serious errors before we ship it to users. And then roll out to users is incremental. So there's a lot of checkpoints along the way.

31:05 >> So you have synthetic users that maybe that have different personalities or something. >> Yeah. Yeah. So we have synthetic users that go down like various different adversarial or unhappy paths and that just you know bark up the wrong tree and then we have the tutor try to save that session and then we see okay how is the tutor working and you know it's like I don't know it's 20% as good as real human data but it's better than nothing.

31:28 >> Interesting. Okay. So maybe you have like a legacy user versus like a like a someone who's too smart or something like skips all the stuff. Yeah. Okay. >> Yeah. Yeah. Yeah. So there's different different personalities. but you know ultimately I would say the like most of the arbitration of quality comes from the team and there's some sanity checks before it goes out the door and then we learn a lot from actually just you know watching the tutoring sessions and watching how it's going and looking at real student data. And there's no there's no replacement for that you know like once you have tens of thousands of students going through a single concept like you are going to get such more comprehensive data on the right way to teach that concept so that people people understand it. How do you like okay you can obviously get qualitative feedback from the real students but how do you measure the thing we talked about which is like this thing is just difficult enough to keep them going like because cuz like for example if you only measure like completion rate or something then you just make it too easy you know how do you you know is there some quantifiable way of >> yeah so we actually we actually don't measure completion rate the only thing that we care about is your performance on the assessment and >> Oh that's right there assessment that's right yeah Yeah. Yeah. So, this is something that we're you know rolling out new features to across different courses all the time and just testing like how do we structure the assessment to sort of get the best most incisive data on whether or not the student has actually learned. But you know when we're building a course we are working backwards from okay you know at the end of this chapter what should the user be able to do like and you know all of the learning sequence is measured against okay did the student actually learn how to do that thing and got it >> and so you know if you can do it in less time with fewer nodes completed that's great like we should be speeding you up like maybe it's better for you to not complete every you know thinly sliced thing and you know if you're really struggling then your goal might not be to complete that course. Your goal might be to go back to some like way earlier thing and do that instead. So I think it's really important to not measure stuff like time spent learning or you know quantity of stuff like ultimately what matters is like can you do the thing and so you know can you do the thing can you do it today can you do it in a week from now can you do it a month from now like can you do it and have you retained it? Yeah, because that that's a really good point because I I remember like when I was going through with my kid like every lesson had a lesson part and an assessment part, right? And then like >> Yeah. Then there's a summitative assessment part at the end too.

34:02 >> Yeah. So So I guess in some ways it's it's kind of like a bunch of mini tests which >> Yeah. >> I guess in the US education system it's not very kosher to have a lot of tests but Yeah. >> Yeah. Yeah. I mean we're Yeah. I mean I guess this is one benefit of being a consumer company is you know it doesn't matter sort of like what's kosher or not. like we can just focus on what works. And you know, we're always testing things like, you know, do you want more assessments closer together or do you want them further apart? You know, how big of a range of stuff do you want to test per assessment? You know, the thing that that's nice about having a ton of usage is that you can come up with ideas and then test them pretty quickly.

34:39 >> That makes sense. That makes sense. Do you have any thoughts on like how to build a I mean Bernie has been around for many years but do you have any thoughts on how to build kind of like a durable AI product with everything changing all the time? >> Yeah, I mean AI products feel pretty easy to copy generally, right? >> Yeah. >> And so the way that we think about it is that the most defensible products that are going to come out of this era are the ones that are generating valuable data in domains that aren't automatically verifiable. So if it's automatically verifiable like coding, you're going to get run over by a frontier lab like anything where the AI model has a verifiable reward is you know I think it's just going to be tough for startups to you know do anything differentiated there that isn't just like that isn't just beaten by the pure scale unless you're in a super niche base that they don't care about but then maybe maybe that business is not the most promising startup. But yeah, our rule of thumb is if the big model companies can RL your domain, they will eat you alive, but there's lots of alpha finding domains that no one else can RL on. And tutoring is one of those because the outcome that you want there is a real human student having actually learned something well and having actually sustained attention on the task. And so you need actual tutoring sessions at scale to generate that data. Like you need real student mistakes and misconceptions. You need to find out what tutoring interventions work best. You need to develop judgment on how to sustain their attention on the task across you know billions of problems solved. And in our case we're doing that in a kind of graphical tutoring environment that is totally novel. And so we are using this data from our tutoring sessions to build better and better tutoring instincts now into the model weights and in doing so create you know a vertically integrated optimized experience for this very specific point of view that we have on learning and how we can make it more efficient and effective. And so, you know, what I would say is just like try to find areas where you have unique data that makes your product stronger over time >> or or maybe like using your product will produce the unique data.

36:51 >> Correct. Yeah. Yeah. You got to build the product, you got to get the data and then that data makes your product better. Like that's a great flywheel. >> just play devil's advocate. Like I'm I'm I'm sure a lot of kids are talking to Chad GBT, right? >> Yeah. >> But but but I guess they're probably not asking this. They're probably not following the same learning process that that you're trying to do. They're probably just asking Chad to do their homework.

37:10 >> Yeah. >> So, so in some ways, OpenAI has a bunch of data, but it's probably not as >> correct. They don't know if you've learned it as we have. >> Yeah. They don't know if you've learned it. And actually, you know, I think every Frontier Lab has run into this with the homework help aspect of their products, which is that students will, you know, slurp in their homework, the answer machine gives them the answers, and then the interaction's over. And so, you know, you don't know if you are chatbt whether or not the student actually understands it. That that's not what the exchange was about. the exchange was, you know, the user gave you their homework and you solved their homework for them. And so, you know, we would say in in our case, like that's not useful data for us.

37:46 >> Yeah. >> Like the only data we got is what was on your homework. Like what are you learning right now? but whether or not you know it's not data that we would feed into making the tutor better. >> Yeah. Yeah. I I I think kids are already smart, right? They're going to find the shortest path to, you know, finish the homework and then they can go off and play. But but yeah, but I think this is the incentives problem, right? Like the US is unique in that we make homework count towards the grade. You know, in other countries, the homework isn't like you don't get graded on your homework.

38:16 You get graded on the test. >> And so, you know, I I think students very quickly understand, oh, cheating on the homework doesn't help me with anything. Like the homework is there to help me >> interesting, >> you know, get reps on the things I need to do to pass the test. And so, you know, I know people are very concerned about homework cheating as a use case of AI, but it's like, well, you know, maybe homework shouldn't have counted towards the grade in the first place and like now it's finally time for us to fix that incentives problem.

38:39 >> Yeah, that's a really good point. And maybe homework can be like more project based. So, like is it actually like what kind of stuff you make as opposed to did you fill this sheet correct correctly? >> Yeah. >> Yeah. Yeah. I mean, I I think like what outcome are you measuring? Like what are you asking the student to actually do? Like kids are smart. If you tell them, hey, do this thing and do this thing so that you can get a grade and that's the purpose like and there's an easy way to do that and get a good grade. Like, of course, they're going to do that. But, you know, that wasn't the point. That wasn't the point of the homework, but we have made it the point by creating this incentive.

39:10 >> How how do you I mean, okay, you're you're like a CEO of an education startup. How do you with your own kids, do you want them to get straight A's or like what are your thoughts on this? What are your thoughts on what they should optimize for? >> You know, my parents were very free range. Like I don't think they ever looked at a report card. >> Wow. >> I would Yeah, >> they just they were very clear.

39:33 School is your job, not my job. You know, I I think like other kids like their parents like were paying a lot of attention and they got a lot of help. and my parents are like if you can't do your homework, I guess you can't do your homework, you know? Like that's not my job. You know, one thing I learned is that they had very high standards. They talked a lot about how hard it is to achieve great things. they were themselves unafraid to do hard things.

39:55 they you know said I need to use my talents to solve really hard problems and you know they did like also buy a lot of biographies of which you know the primary sort of like story in the biography was people spending a really hard time sorry spending a really long time toiling away at super hard problems that no human had ever solved before. you know, just like literally like Marie Cury giving herself, you know, po, you know, radiation poisoning, like these kinds of these kinds of stories.

40:30 And and I think that, you know, for my kids, the thing that I really pay attention to, and they're young still, you know, they're five and three, and the thing that I really pay attention to is just helping them cultivate concentration and attention. And I think that, you know, they're interested in different things. And so like the easiest way to do that is just getting them, you know, creating space for them to be immersed in their own interests.

40:55 And what I really want is for them to get reps of the idea that it should feel hard to learn something new. but also it can be super satisfying. And so however I can help them, you know, learn that it's fun to do hard things, it's satisfying to do hard things, it's also hard to do hard things. And I don't think it has to be math, you know. It could be >> building something. It can be monkey bars, you know. It could be joining a chess club or a lemonade stand or like learning magic tricks. But I think as soon as they start to believe that learning sucks and like you do it to please authority figures and maybe a good way to do that is to have catchup do your homework so that you can get a good like you know at this point you've like lost the plot, right? And it's really really hard to undo. So I spend a lot of time talking to them about you know learning isn't about the answer. It is about strengthening that part of you that can find one that can concentrate and struggle that's unafraid of trying new problems. And so you know maybe this will change as they get a little bit older but right now I'm much more interested in how deep of a rabbit hole can they go down by themselves because they're really interested in something.

42:09 >> Yeah, that's really important. And and it's funny. I guess your parents kind of like passively aggressively made you all those biographies. So >> yeah. Yeah. That's really good. So I think a lot about this too. My my kids a little bit older. They're like five and eight. >> Yeah. >> And and like I I I feel like definitely one principle is like let them try many things and see what they like. Right. Yeah. >> But I think like the whole basis of brilliant is like even if you like the thing, it doesn't mean that it's not going to become hard, you know? Like to actually become good at it, you have to you have to struggle. So, so that I think that's the part that I think is really hard to even as a parent really hard to teach because like you know like for for example let's say she likes like she she likes vibe coding games for example she likes voding games >> but then but then like you know sometimes she runs with bugs and stuff and and then she gets demotivated.

42:54 >> Yeah. >> She gets demotivated and and then she's like oh maybe I'm not very good at vi games. >> So so like you kind of have to be there as a parent to kind of like re reward her in some way shape or form. >> Yeah. Yeah. And I think this is what like learning products will get, you know, better and better at exponentially is creating that correct ramp like just at the edge of your difficulty, like that pacing that makes it so that you stick with it. because every time, you know, like when you play Mario, right, and you're like jumping, there's never a point where the kid is like, "Oh man, I'm just not good at Mario. Like I was not meant for this." You know like it's structured in such a way that like that next jump feels really hard but you feel like if you try 10 more times you can do it.

43:39 >> Yeah. >> And I think that if more of learning were like that where you know students cultivated a belief that you know it is going to get harder and harder and it's going to require more and more effort and my brain's going to have to work harder and harder but you know I can do that. That's like an amazing thing to walk away with from you know the learning that you do in your childhood. >> Yeah. It's like a sense of accomplishment, right? That you can do something that other people might have given up on.

44:03 >> Yeah. And a sense of, you know, willingness to do something hard. >> I think another thing I'll say is like I love how you made Brilliant really approachable and like cuz in the Bay Area a lot of, you know, we talked about this like a lot of private schools are like 50k a year or something. >> Yeah. >> And and I I I love how accessible Britain is. >> Yeah. And your kids are young, you know, they're like way before the age that it's built for. Like this is a product for middle school and high school students. But we we're getting there.

44:28 We're we're working on it. >> Oh, okay. Yeah. Interesting. Okay. She got pretty far on the code coding course, actually. >> Amazing. Amazing. Yeah. Kids are >> hard. Kids are like way capable, more capable than they think. I I I feel like grade one kids can understand like algebra. Like I feel like explain to them. >> Yeah. And you know, we're having fights about, you know, should we allow the teaching of algebra in 8th grade in California?

44:51 >> And it's just, you know, set high standards. Set high standards. And like you'll be very surprised by what kids can do. >> Yeah. All right. So, so where can people find more about Brilliant and find you? >> Yeah. So, we got our website at brilliant.org. you can find Brilliant in the App Store and Play Store. you can also follow us on Twitter where we post updates and preview things that are shipping soon. I'm Sue Kim.

45:18 the account for Brilliant is Brilliant. >> Awesome. Well, well, I have a ton of respect for what you're doing. I'm I'm I'm sure it's way harder to build a education startup than like just build a mobile game or something else, you know, as Peter entertainment, but I I I think I think it could have massive potential on all of our kids. Thank you so much. >> Yeah, it's a labor of love. Thanks so much, Peter.

Summary

AI in education is currently being misused by students for cheating, leading to concerns about its impact on learning. Sue, co-founder of Brilliant, emphasizes the need for AI to enhance educational experiences rather than diminish them, advocating for a focus on genuine understanding and problem-solving rather than simply providing answers.

- The U.S. education system faces challenges with declining test scores and grade inflation, masking underlying issues in student performance.
- AI can be a powerful tool in education if used correctly, promoting deeper learning and critical thinking rather than shortcuts.
- Brilliant's AI tutor, Cooji, is designed to encourage students to struggle through problems, fostering independent thinking rather than simply providing answers.
- Effective classroom learning should prioritize peer interaction and engagement, moving away from traditional lecture-based methods.
- The integration of AI in education should focus on enhancing personalized learning experiences, making them more accessible and effective.
- Quality control in educational content is maintained through human oversight and evaluation, ensuring that AI-generated lessons meet high standards.
- The importance of cultivating a growth mindset in students is emphasized, encouraging them to embrace challenges and learn from difficulties.
- Brilliant aims to provide a world-class tutoring experience that is affordable and accessible, challenging the notion that high-quality education must be expensive.

Questions Answered

How is AI currently impacting education, and what should its role be?

AI is often misused by students for cheating, which detracts from genuine learning. The goal should be to empower students to engage deeply with challenging material rather than providing easy answers.

What advantages does AI provide in personalized learning environments?

AI can offer personalized reviews and create a comfortable space for students to learn without fear of embarrassment, which is often present in traditional classroom settings.

How does Brilliant ensure quality in its educational content?

Brilliant combines human-designed lessons with AI implementation to maintain high quality. Human educators design the pedagogy, while AI executes the delivery, ensuring effective learning experiences.

What is the best approach to onboarding new employees in a tech-driven environment?

Onboarding should focus on hands-on experience, allowing new employees to engage with the product and workflows immediately, rather than just reading documentation.

How does data influence the effectiveness of educational products?

Unique data collected from user interactions can enhance the product over time. However, data from AI tools like ChatGPT may not reflect true learning outcomes, as students often seek quick answers rather than understanding.

© transcribe · For agents Built with care and craft by Gokul Rajaram