transcribe

How to Ace AI Eval Interview Questions (for Engineers & PMs)

Hamel Husain · 21m · transcribed Oct 2026
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Understanding Evals in AI Job Interviews

What is the role of e-vals in AI job interviews?

Evals are increasingly common in AI job interviews, often involving take-home assignments or in-office tasks where candidates analyze datasets. Interviewers assess how candidates handle incomplete or tricky data.

  • Evals are a significant component of AI job applications.
  • Candidates may face challenges with incomplete or misleading data.
  • Demonstrating resourcefulness in data handling is crucial.
# 4:15

Handling Incomplete Data in Evals

How should candidates approach incomplete data during evaluations?

Candidates should show resourcefulness by leveraging external datasets to augment the provided data. This approach can help in addressing gaps and improving the evaluation process.

  • Resourcefulness is key when dealing with incomplete datasets.
  • External data can enhance the evaluation and provide context.
  • Candidates should think critically about how to utilize available resources.
# 8:31

Presenting Experimental Results

What is the best way to present findings from experiments during an eval?

Candidates should create a table summarizing various experiments, including metrics used and results obtained. This helps to clearly communicate their approach and findings.

  • Organizing results in a table format aids clarity.
  • Candidates should include a baseline for comparison.
  • Demonstrating a variety of approaches showcases knowledge and adaptability.
# 12:47

Effective Communication in Interviews

How can candidates differentiate themselves in AI interviews?

Candidates should explain their reasoning clearly and thoughtfully, which can impress interviewers and demonstrate deeper understanding. Providing constructive feedback on the evals can also set candidates apart.

  • Clear communication is essential for standing out in interviews.
  • Providing thoughtful feedback can enhance the interview experience.
  • Candidates should be prepared to justify their decisions and methods.
# 17:03

Next Steps and Product Improvements

What should candidates consider for future improvements after an eval?

Candidates should suggest potential next steps, such as acquiring better data or proposing product improvements. Detailing these suggestions demonstrates critical thinking and a forward-looking mindset.

  • Always consider how to improve data quality and product functionality.
  • Specificity in suggestions shows depth of understanding.
  • Candidates should frame their ideas in a way that aligns with the company's goals.

Transcript

0:00 And an increasing number of jobs have an e-vals component to it. Like when I look at jobs online, around like half of the AI jobs I see nowadays mention e-vals. These are interview tips. So, just to give you a little bit of background, I do consulting around e-vals. A lot of times, if you're asked to do something around e-vals, you get a take-home or you're asked to do something in the office. They give you like 3 or 4 hours and you do an interview. So, the way it usually goes is you get a data set.

0:24 You're asked to do some kind of e-vals. I put this meme here. Just please, for the love of God, don't do this. Don't just ask Cloud to find all the errors. It's very tempting. This is where a lot of the interviews are designed to booby trap you if you are going to do this. A hypothetical example is Tree Buddy. So, let's say there is a Tree Buddy app. The Tree Buddy app allows you to take a picture of a tree and understand if that tree is healthy, how you should care for that tree, etc. The app has like different components. You don't want people to abuse the thing, so you you want to make sure the inputs are actually trees. People can't like do things with your application. Okay. You have the liability and you want to give people like facts about the tree, any health findings, a summary of what they should do, or in like along with recommendations, etc. In this particular example, you might get something that looks like this. So, you get a data set.

1:12 That data set usually has different components to it. You'll get some labeled data. Let's say you'll get a bunch of trees. You'll get some labeled data maybe around the guardrails they want you to build. Examples of bad inputs like where it should pass or fail. Oftentimes, they won't give you any labeled inputs at all. There's going to be a lot of traps in the data. And I'll give you some examples. You'll be asked to potentially do some data extraction. Here we have photos and we're going to do a data extraction of some kind. And data extraction is a favorite thing that people like to do in e-vals. It feels more verifiable cuz you can compare data. They can give you some data that they want you to extract and then compare the data you extracted to some golden set. And then there's unlabeled data. In this case, you might have the summary and recommendation just like text. They may be labeling that.

1:54 They may not be labeling it, but usually there's some form of unlabeled data somewhere. And then a lot of times what you are asked to do as a PM or AI engineer is like build an agent that can perform well. And they'll give you some evals. They'll give you some metrics that they want you to like think about. Okay. So this is what I recommend. We want to produce a structure. This is the way I like to structure it. Like to have like an executive summary up front. I'll go through these sections. Before you talk about anything, like how you built the app, what the results, whatever, talk about the data. You always want to focus on the data. What data exist, what issues there in the data. Then you want to talk about experiments you ran. You want to summarize the results. You then want to talk about what you learned and then next steps. And you always want to then have a little bit disclaimer of any AI tools you used. Depends. It's always good to have that. So the executive summary, I'm not going to spend too much time on this. You want to keep it to a third of a page or less. You want to just say like what you built and the results. So think about like a short paragraph. And there's going to be some repetition in this like artifact that you create. So you want to go directly into the data. And this is like where the meat of it is. A lot of people skip this section. This is like the most important section. This is where it separates people who understand evals and people that don't. I say spend 50 to 70% of your time here. And here are some things you want to look at. You want to look for things like missing or inconsistent labels. You want to look at insufficient label data, like maybe it's not labeled correctly or maybe the labels don't make sense. Never assume a data set you are given in real life and especially in an interview is {quote} correct because the person is trying to simulate your ability to be skeptical and like really look at the data. And that's what they're doing here. And also you want to think about how noisy is the data. And these are all like very related, but you basically want to see what are the issues in this data. And let me show you some examples. So let's say here's example of a data set.

3:36 There's 120 examples and there are some attributes of this data, like some metadata attached to these photos. And they have like species, like what species the tree is. Every example is labeled, but only 96 of those examples have a health finding at all. Maybe only 24 of them have height, maybe only eight of them have age. I just made this up. And then like we talked about guardrails. Let's assume they just give you data set, but they had a guardrail pass. Or is this violating a policy of like what pictures can be input? And they all say false. None of them violate. So the whole data set doesn't have any violations at all. You'll get examples like this. You might be wondering, well how the hell am I supposed to build this if I don't have examples of what is pass and fail? That is exactly the problem you will have to solve. That is the mess that they're going to throw in to here. If you just try to bench max this, you'll get like 100% accuracy on input policy cuz there's not any bad examples. So they're trying to test you. See like how you handle this data where you get like incomplete data. So one thing you might want to do, even before doing synthetic data, is synthetic data might work. What I like to do is be as resourceful as you can. So what I mean by that is if you're given a problem with this tree thing, maybe there is like a trustworthy data set that someone already has for some of these dimensions. If you think about it, like it's kind of hard, right? Like height, age, like where are you going to find that? You're just going to synthetically generate pictures of trees and know what the height and age are.

4:58 It's a little bit tricky. So you want to try to show off your resourcefulness. And what I would do is think about outside data where possible. Not always, but like in the outside data doesn't necessarily help you solve everything, but it can expand the data set and sort of say, can I have like a reference data set that I can test with and use the data set that they gave me as the test set as like the final like readout on how it's going. You're going to get this incomplete data. You might think, oh that's nice animals, trees, whatever.

5:25 The reason I'm like constructing this hypothetical example with the trees, I found that like 80% of take-home exams and examples, you could always leverage outside data. Because they're trying to give you an example that's like amenable to an interview a lot of times. Not always the case, but always think about how you can be resourceful. Is there any other data that you can use to augment what you're doing? And it's like, okay, these are two data sets actually.

5:47 There's this tree Curio tree data on Hugging Face. There's also this research data archive on trees. And for the guardrail we mentioned synthetic data, that's exactly right. I would actually create these negative examples here. And these are not real people and real things. You can see that this is AI generated, but it's better than nothing. Again, you're trying to show off your resourcefulness cuz like you are given limited resources when it comes to data. And you are trying to be creative on how you fill those gaps. Sometimes with synthetic data, maybe sometimes with external data. Whatever may be, you try to choose like what is the best proxy here. But most importantly, you want to actually look at the data. And you want to find things like inconsistent labels.

6:24 How would an inconsistent label look like? The tree on the left, it says species is English oak. And then the tree on the right, it says species pine, period, long needles, orange bark. And then the height, you see, is not normalized. So you have like height 12 m. I don't even know if that's a reasonable height for the tree, honestly. And then there's height 50. 50 what? God knows. I don't know. Okay, if it's a good eval's interview, they're going to give you this to try to frustrate you. You have to think about what to do here. You have to think about either label this yourself or get a LLM to try to do your best job to normalize these labels. The whole idea of the artifact that you give them is like you discuss it. You say like, "Hey, this is the issue that I found in the data. And this is what I tried to do to overcome it." You might not be able to perfectly overcome this. Now, we do have that data set where maybe we can have more trust cuz it is normalized, those external data sets on Hugging Face and the other one where everything is normalized. And you can say, "Hey, like because this data had these problems, I used this data set as a proxy to like try to get things started." So again, like it's a job interview and you want to show that you're paying attention and you're looking into the details and you're understanding the data. And you're not just haphazardly pointing your agent at this. They're trying to simulate like what actual product work is going to look like, what you are going to encounter. And then like you want to also pay attention to noise. It's how I generated this AI, but I'm trying to drive home the point. You want to see these examples, and you'll see there's real trees, and then there's bonsai trees, these like miniature trees. Do we want these bonsai trees in the data set?

7:51 A lot of times there'll be things in the data set that shouldn't be there. You want to call that out when you write up this artifact to say like, "Hey, I took these out of the data set cuz they didn't belong." Or you have to make some product judgment like how you want to handle this, but there's these traps that you want to think about. So, when you look at this data and you see what's going on here, you will find issues. You just want to summarize those issues. You want to like maybe have bulleted list, whatever they look like, what's going on in this data, and you want to use your skills of communication to talk about what the data issues are. Okay, so next moving on to an experiment. So, what are experiments? Experiments are your attempts to make this agent better at solving this problem that they posed to you. You want to frame it like this.

8:30 What is the goal of experiment, what you tried, and then any methods and findings that came out of that experiment. An example goal could be species identification of the trees. And you want to classify the species correctly. What I recommend you do is to have a table of results. They will give you a metric that they want you to express. And that may not be TPR, TNR, whatever, it may be something that they have. Whether you argue with that metric is up to you, we'll get to that in a moment.

8:56 Sometimes you might want to argue with that metric, but only if you think it's wrong. And I'll show you an example of where the metric might be a bad idea. But what you want to do is show a table of different experiments that you ran, and you can organize the table in any way you want. have to look like this. But you want to say, "Hey, I tried this thing." So, for example, in this table, I tried GPT-6 Luna with just a prompt. I tried GPT-6 Luna with this external tree database where like when I look up the answer, I also try to get the most similar tree in the database, and I use that to inform my answer. I'm just making this up. V3 is like, "Hey, I'm doing the same thing as V1. I just changed the model of the soul." V7 here is like using automated prompt tuning like Jeppa or something like that. And the idea is you want to show the different things you tried. Like how did you try to improve this? You want to have a baseline, so you want to make sure you have like a simple baseline.

9:44 That's why I have on this slide prompt only. You want the simplest dumbest thing you can as your baseline. Then you want to try different ideas you might have to improve your agent. And there's no formulaic way to go about this. This has to be informed by your judgment, what things you want to try. This is all an opportunity to showcase your knowledge. And it's a nuanced thing. Like if I had to do one of these today, if Jev just came out, I'm like let me try Jev. It depends. If I'm interviewing at Anthropic, then I mean I'd use Jev cuz I'll just use their stuff. You have to think about like what they want to see. They might give you parameters of things to try or they might tell you use a certain SDK or use this, use that, whatever. Think about having a table of results where you can quickly show things you tried. And then you want some narrative under the table. I tried this, this is what this means. I found this approach to be good. You want to show some thinking of tradeoffs. When you have a table like this, you might have something that has like the highest score, but maybe that's not the one you want because it's slower, it's more expensive, it's more complicated. And what kind of the next level is to like talk about okay, like I actually like this certain example because I think it has the best tradeoff in terms of complexity and performance. And there's also if you notice on this table, there's this validation and test. People won't tell you to do validation test.

10:54 You want to pay attention to overfitting. See this like V6 row here has a very high test score on the validation set, but the test set score drops significant. So something is being overfitted. Like the score is not consistent and we don't like that. So these are the kinds of things you want to discuss. It's not just about score maxing. It's like bring your judgment and talk about oh like this is the kind of tradeoffs I think about and this is what I would like. You want to have your methods underneath the table and you want your findings. So like an example finding would be Soul and Rax seems like the best tradeoff. Astro is overfitting and automated prompt tuning do not help.

11:28 And again, you're trying to show off your judgment, your ability to think about trade-offs, and also showcase that you know a little bit about different approaches to improving the agent that you want to write. Don't let AI do all this stuff for you. You're going to get in trouble. You're going to be like this person in the boat. Don't do that. Now, sometimes this is a kind of insidious thing to do, kind of borderline evil, but it will happen. They will ask you to do things that you shouldn't be doing.

11:52 For example, you have this tree. Never mind that the tree the picture sucks. Can't even see the whole tree. That's a problem. But, okay, like your app produces this summary. The brown leaf edges mean your tree is drought stressed. Give it extra water this week and watch for more browning. It's clear you've been neglecting your tree. Let's say the interviewer gives you this direction. Grade helpfulness and tone on a scale of 1 to 5 and combine the helpfulness and tone score into a composite score from 0 to 100. You want the score to be actionable and interpretable. Like, what the hell is this? This is a little bit trickier. It will come down to this. Now Now, sometimes this is a very insidious question meant to trap you. Sometimes they don't know evals. They're interviewing for evals and they don't know evals. I would always just compute the score, but then also explain, "Hey, this is not the way I would do it. I would think about refactoring this metric to blah blah blah." That always goes over really well. There's two possible outcomes from that. One is you avoided the trap that they set for you.

12:47 You're going to teach them something. They're going to be like, "Oh, you know, that's a very thoughtful comment." If you explain properly like why. You have to be nice about it. You have to explain yourself. Give the reasoning behind that. They will learn something, and that will put you at a completely different level than anyone else doing this interview. And that's what you need. You need to be at a completely different level. The AI interviews nowadays or job interviews in general, they just have way too many people they're considering and way too many applicants. So, like from this specific example, what are some examples of evals that I would create? I would say like, "Okay, is a full-length picture taken?

13:19 Is the watering advice maybe it needs to check soil moisture? I'm not sure like why are they giving that to me." You know, you got to make sure it's grounded. No blame or scolding. Saying you have been neglecting your tree. Why would they have to tell you you're neglecting your tree? That's That's harsh. I would describe why these are better. Because you don't want to be caught in a situation where your interviewer is like, how do you know that the score is good? So, experiments.

13:39 You want to put it all together. You want to have, like I said, the goal. You want your table of experiments. And then you want those methods and findings. Expect to have lots of experiments. You would want to break those experiments down by different features that they want you to tackle. They have like guardrails. They want you to classify the species. You get like health recommendations of the tree. Okay, there's a lot of things going on here.

14:00 And you want different experiments for those different areas. And then it's going to results. So, this one is quick. You want to just synthesize your experiments. So, you're going to have all these experiments that you run. You have many different tables. Probably one experiment per feature. So, you're going to have a lot of information in that experiment section. What you're going to want to do is like condense it to results. You want to say like, okay, what is the winning combination of these experiments? How would you go about it?

14:23 And you want to synthesize and say, okay, these are your recommendation of how you proceed, sort of based on all these experiments. You want to reiterate like what the baseline is, what the final eval scores is across these different evals they gave you. And you want to also think about the cost and latency trade-off. So, remember I talked about trade-offs. It's not just benchmark seeing or getting the highest score you can. It's opportunity for you to demonstrate your product thinking and your judgment. I would go as far to say you should have something in your table of results with a higher score that you decide not It doesn't have to be like so unrealistic, so heavy-handed, it's like expensive, it's overcomplicated. It's really good to have that. So, you can showcase this quality of yours of judgment. Like, no, I'm not going to go for the highest score. That would be bad for my product. This other alternative is better because it's just a little bit lower, but it's so much faster, so much cheaper, so much more stable. And then after results, I would talk about what you learned. And you could There's some repetition in this. And some repetition is good. People don't read. So there's some amount of repetition that you need when you submit these artifacts. And so what you learned kind of should talk a lot about like what worked well, like what experiments you tried, what didn't work. So like one example of what worked well, maybe synthetic data was really effective. Maybe this external data was effective with retrieval. Maybe maybe you caught overfitting. Talk about all the things that you learned while you were building this application. Talk about what didn't work. those experiments like kind of synthesize like oh okay. Yeah, maybe prompt tuning didn't work. Maybe larger models didn't work. Then talk about limitations. So what I do here is like repeat any quote bad things that you were forced to do.

16:01 Maybe they were telling you to like take like 10 generic metrics and combine them into a single score. I have seen this on a real take home exam by very reputable companies of like hey let's take these five 10 metrics. They're super generic and combine them into a single score. In cases where you were asked to compute something from text data and you don't have labels and you can't get labels, do your best. You might even go as far as say hey these LM judge would not validate against human labels because of like this this or other limitations. Try to label yourself if you can. But hey, you might not be able to if they give you something if they give you a data set. Keep in mind that when you encounter interviews, it'll be a mixture of people trying to set a deliberate trap and people who don't understand evals. They're just getting an AI to generate an eval interview. They didn't think through it. That happens a lot.

16:50 You still want to showcase your knowledge here. You want to teach the interviewer something in that case. That's the best outcome. And again, you want to repeat any limitations of the data that you saw or the approach. Okay, so this is like where you exercise some humility. You say okay like this is kind of like where we were limited. And then always like to have a next step section. You want to put your product hat on is like oh okay like if you had more time, based upon what you learned like what are ideas you would have for improving performance. And it is always the case, 100% of the time, that you can have better data. Everyone can have better data. And so, you should always mention, "Oh, like, I would try to get better data on XYZ ABC." And also, I like to talk about product improvements. This is again another opportunity for you to showcase how good you are by thinking outside the box and helping that person envision working with you. You talk about, "Oh, you know what? You gave me this tree app. I would actually redesign the app to do this thing instead or like in you know, have these other functionalities that will make it easier for you to evaluate." But, okay, just make sure if you do this, if you talk about better data or product improvements, you discuss the details.

17:56 So, don't just say better data. Talk about like what exactly do you mean? Like, what's wrong with the data? What kind of data would you get exactly? And then like product improvements, like what what are some product improvements you might try to make? And that's it. That's kind of my synthesis of how you should think about framing the problem if you're going to do an interview like this, if you're going to think about evals. >> A few questions that have been around the experiments that you have been talking, how does it differentiate with AB testing that normally software engineers do?

18:25 >> I would say don't worry about AB testing. Unless it's a data science interview, they're not going to usually do that. They're going to give you a data set and ask you to compute some evals and compute some metrics. >> Will it be helpful to present the results in in kind of like UI? >> Be very careful with that. So, it depends what they ask for. Usually, they ask for something. They're like, "Hey, please give us a report in a PDF. Please give us a GitHub repo." You want to follow that. Be careful about UIs, presenting stuff. Like, you don't want to become a slop cannon. Easiest way to turn someone off in the interview is becoming a slop cannon. The reason I say this is I can't tell you how many slides and stuff I see all day that I can't read. And so, just don't do that to yourself. Write it thoughtfully.

19:04 >> Whatever you have covered so far, it was more of like very much technical topics for a few. So, what is the normal bar for PMs into the eval space? Like interview for PMs. >> So, I would say like it's not as technical as it might sound. So, if it's a PM interview, they're not going to like hammer you on calculate F1 score. They probably won't even ask you for an F1 score, but they will tell you to like look at the data and tell us how you might go about this. They might ask you to write code something. They usually do nowadays. They might give you a data set along with that. And what you should do is try to like look at the data applying some of these principles and showcase like all of the nuanced thinking that you have.

19:45 >> Last question that I can see is many questions I've seen around these mock interviews. Like if there's any platform for mock interviews. Just a question to you from my side like given cloud board skills, cloud skills, cloud recently the eval skills came up and a couple of like codex and other skills. Is this possible to mimic some kind of like okay, what kind of given you have some set of data or traces to mimic what kind of eval process should go around or in the in general the interview process?

20:12 >> I mean, you should use whatever tools you can. It's part of resourcefulness. Use your coding agent. Use the eval skills. Use anything you learn in this class etc. I would like be very careful not turning your brain off and saying like, oh, how can I solve this interview with an agent? Because that is going to cut against you. Because that's what everyone else is doing. And you do not want to look the same as everybody else.

20:31 Like the person needs to read your report or listen to your presentation and you need to hear something different. And the way you do that is this like way that I showed. Real quick, if you're building an AR product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than a thousand hours of working with companies on building AR products. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals.

20:56 You don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. Go to bit.ly/evals-guide and we'll send it to you.

Summary

The discussion focuses on the increasing importance of evaluation (e-vals) in AI job interviews, particularly for roles involving data analysis and product management. The speaker provides insights on how to effectively approach e-vals tasks, emphasizing the significance of data quality, experimentation, and clear communication of findings.

- E-vals are becoming a standard component in AI job applications, often involving take-home assignments or in-office tasks.
- Candidates typically receive a dataset with labeled and unlabeled data, requiring them to identify data quality issues and build effective models.
- A structured approach is recommended, starting with an executive summary, followed by a detailed analysis of the data, experiments conducted, results, and next steps.
- Key data issues to look for include missing labels, inconsistent data, and noise, which can complicate model building.
- Candidates should showcase resourcefulness by leveraging external datasets or synthetic data to fill gaps in the provided data.
- Experimentation is crucial; candidates should document various approaches, their goals, and findings in a clear and organized manner.
- It’s important to communicate trade-offs between model performance, complexity, and cost, demonstrating product thinking.
- Candidates should also prepare to discuss limitations of their approaches and suggest improvements for future iterations.

Questions Answered

What is the role of e-vals in AI job interviews?

Evals are increasingly common in AI job interviews, often involving take-home assignments or in-office tasks where candidates analyze datasets. Interviewers assess how candidates handle incomplete or tricky data.

How should candidates approach incomplete data during evaluations?

Candidates should show resourcefulness by leveraging external datasets to augment the provided data. This approach can help in addressing gaps and improving the evaluation process.

What is the best way to present findings from experiments during an eval?

Candidates should create a table summarizing various experiments, including metrics used and results obtained. This helps to clearly communicate their approach and findings.

How can candidates differentiate themselves in AI interviews?

Candidates should explain their reasoning clearly and thoughtfully, which can impress interviewers and demonstrate deeper understanding. Providing constructive feedback on the evals can also set candidates apart.

What should candidates consider for future improvements after an eval?

Candidates should suggest potential next steps, such as acquiring better data or proposing product improvements. Detailing these suggestions demonstrates critical thinking and a forward-looking mindset.

© transcribe · For agents Built with care and craft by Gokul Rajaram