Section Insights
Building Virtual Employees
What is the approach to building AI for companies?
The focus should be on creating virtual employees that behave like real people, rather than open-ended agents. This allows companies to sell the work itself instead of just the tools.
- Virtual employees can enhance productivity by mimicking human behavior.
- Focusing on specific tasks can lead to more effective AI solutions.
- AI can transform traditional roles by automating routine tasks.
Challenges of Product Management
What are the challenges faced by product managers today?
Product managers often find themselves bogged down by administrative tasks like hiring and performance reviews, which detracts from their primary goal of engaging with users and creating great products.
- Administrative burdens can hinder the effectiveness of product managers.
- There is a need for tools that streamline PM responsibilities.
- Focusing on user engagement is crucial for product success.
AI in Recruitment
How is AI being utilized in the recruitment process?
AI tools like Jim can automate outreach to candidates on platforms like LinkedIn, making the process feel personal while handling tasks that would typically require human intervention.
- AI can enhance recruitment by automating candidate outreach.
- Personalization in AI communication is key to effective engagement.
- AI can reduce the workload on recruiters while maintaining a human touch.
Understanding AI Token Consumption
How can companies analyze their AI usage?
Companies can categorize AI usage into three pillars: corporate AI for internal efficiency, operational AI for customer service, and product AI for feature development. This helps in understanding token consumption and optimizing costs.
- Categorizing AI usage helps in better resource management.
- Understanding costs associated with AI operations is crucial.
- Data-driven insights can lead to more effective AI strategies.
Future of AI in the Workforce
What is the future of AI in the workplace?
The future will see a blending of roles where everyone becomes a builder to some extent, with a significant increase in productivity as more people learn to effectively use AI tools.
- AI will democratize productivity, raising the performance floor and ceiling.
- The workforce will increasingly rely on AI for enhanced efficiency.
- There is still significant potential for growth in AI adoption.
Transcript
0:00 If someone's starting a company today, I think the question is like, why shouldn't it just be you and agents? The best way to go about it is to focus on this idea of building virtual employees versus open-ended agents that could do a million things. The fundamental thing that we spent time on was how to make it behave, look, and feel like a real person. Jim is our open cloud recruiter. It does like effectively three things. It sources people, it filters all inbound, and third, it serves as an analytics layer. It's just like actually behaving like a coworker, which is like coolest thing I think here. You're now able to sell the work itself versus just sell the tool.
0:37 All right. Hey everyone, my guest today is Pedro, co-founder and CEO of Brex. Pedro is one of the most AI pill CEOs that I've ever met. So really excited to talk to him about PMing and also how Brex uses agents internally and also how he personally uses AI. Welcome, sir. >> Thanks for having me. Excited to be here. So, so dude, like a couple months ago while I still have my PM job, you tweeted this, you tweeted that the PM playbook of writing PRDs and aligning stakeholders is dead and and I think many PMs wish this was true, but like >> yeah, >> is this true that can you unpack this a little bit or >> Yeah, I mean look, I think I think we've seen that over and over again the the job of a traditional PM actually taking a step back the way I describe the quality of an engineering in a product organization is how quickly it takes between ingesting some signal out there in the world about your customer about a painoint about an opportunity internalizing that into a product or or a feature or whatever it is and then shipping something that works and addresses the pain point and put it in front of a customer. So what you're trying to do is compress cycles in that loop and and of course a lot of the job of a PM traditionally was spend ingesting tons of information making trade-off decisions and and just like actually prototyping things right and building things and we we bra a lot of the role of PM evolved over the past couple of years where for a long time I would say maybe five years ago I think there was this view that PMs ran the show and PMs were you the ones designing the product. They were thinking about how the trade-offs of how things work.
2:14 They were debating which opportunities were. There was this whole like PM is the mini CEO. and we actually don't believe in that like the way we see the world is like you need you need the craft of engineering and design to be equal footing with PM in order for to make the model work. And so so so a lot of what we've done at Brex is so this is maybe the backdrop and then a lot of what we're seeing with AI is the conflation of these roles. So the lines between what was traditionally engineering or product or design I think is getting more and more blurred and you see PMs writing code you see designers you know writing code and building things you see engineers making decisions that with data and context that would traditionally be in the PM realm so so we're seeing a lot of that and I would say PM specifically at Brex became a lot more about how do you use a lot of the tooling that we've built internally to just ingest context about customers and and and and really what's happening happening on the ground, especially with the sales team. and then based on that information, make trade-off and prioritization decisions that affect what we build. and then of course, this is maybe the sort of the first half of the job. The second half of the job is making sure the things you build are actually great and address the the very pain that you were you're addressing to solve. So a lot of the things that we see PMs doing now is just you know like ingesting I don't know six months years of u gong calls for example making sense of that in a very clean way unearthing insights from unstructured data presenting and validating those hypothesis with very cheap and easy to do prototypes and then whenever we go build something we have so much conviction it's the right thing that you just compress the cycles a lot and you end up shipping the right thing in the beginning.
3:55 >> Got it. Yeah, that makes sense. Yeah, I was talking to a PM on the Codex team and he said, you know, the these days engineers are empowered to just like build their own features and actually talk to customers too. So, yeah, he spends a lot of time with the enterprise folks just like understanding what what they want and it's just trying to manage all the stuff that the engineering actually wants to ship. It's like it's like so much more stuff shipping now, right? Because it's like so much easier to build.
4:15 >> 100%. And and and I think it's funny because I I I actually think because it's so easy to build things, having discernment on what matters I think is like more important than ever. and and I think a lot of what we we we push on the teams is is just this idea of like what is a true lever to make this thing move and and actually deliver value in a unique way and let's go do this one thing even though we could do five or 10. we really believe in this idea of like minimizing the surface area going really deep in solving one thing really well and then sort of building around that. So so for us it's been a it's been a journey of just seeing this over and over again and you know at the end of the day even though you can build a lot more things I think leadership bandwidth is something that concentrated always yields like better outcomes I think. And how do you let me just push a little bit on the stakeholder alignment thing because I I I still think like after a certain size it's kind of hard not to like you don't want you don't want your PM just like ships of themselves, right? There's probably some sort of review process but like at the same time you don't want people to wait like a month to meet with you. So how do you >> Yeah, I mean a lot of it for us was was I would say was was two things. one is I I really believe that the I would say I probably spend half of my job my time just reviewing work. So, so, so this is just the nature of the job because I think the way you set a high quality bar is by, you know, effectively building a case law system inside your company where you have, you know, I would say the Supreme Court, which is when something comes from me to review where you're setting the standard for what great looks like and then you have reviews that teams are doing internally all the way down to like an IC building something themselves and understanding ultimately what the task that this thing needs to needs to needs to sort of, you know, withstand. And I would say I think that the biggest thing is you know you just have to engage with the work directly right I think like one of the mantras that we have at Brex is like you know we manage the work not the people and we manage people around the work versus the other way around. And I think a lot of reviews for us now they you know when we do a review we have super two sort of types of reviews at Brex the things we call problem alignment and solution alignment. And problem alignment is like what is the problem we're solving and there's like a lot of things that we could go about solving in a space. So let's say you see like retention is maybe less strong than we would like on a specific feature or a product. Then there's a like a lot of the 50% of the problem is like which problem is worth solving and let's make sure we're align on that. And and a lot of what AI helped us is is make sense of like immense amounts of unstructured data. So you know when you know when a rep is in front of a customer and you know we may be losing a deal for example like why are we losing the deal? Like what are the features that are mentioned? what are the things that are not mentioned but you know we can understand that the customer cared about based on something else they asked based on an RFP based on like a lot of different signals and and and that is something that PM I think before didn't have access to and and now obviously do so so that that's like a big part and then when you are on a review a lot of the work is just u how do you how do you build a hypothesis around that and of course LLMs help on that but but I think the judgment plays a little bit of role and saying here's how this aligns to what we're trying to do as a company more broadly versus you know doing the sort of paper pushing of understanding and digesting data and making it suit.
7:42 >> Some of the exacts have like skills for themselves to that their PMs can run to get the initial pass. Do you have something like that? >> Yeah, we we do we do we have a we have a repo of like I mean it's it's a it's a we have all the skills that a product or uses in a repo and we try to keep it pretty updated. So me and my head of product, we spend a bunch of time making sure those things are correct. So when we when someone gets a review, they already on that skill. and it's like it's it's it's I would say relatively digested already.
8:11 >> Yeah. >> To a point where the things that really spend spending time are things that are idiosyncratic to that problem versus okay, let me just apply the same algorithm I would apply on anything that shows up in front of me. so absolutely. So, so and like the skill is basically like basic stuff like what your principles are like is the problem clearly stated like that kind of stuff right like PM1 stuff. >> I think I think I think it's like I think it's a little bit of that. I think it's like also how do you codify the ethos of what makes a good product from the perspective of your company and your customer. A lot of it for us is honestly about go to market is like I think like I think PM at Brex like plays less of a role as like the actual craft of building and probably a little bit more of a role as being the bridge of go to market and understanding which problems we're solving and and for example a lot of the times we spend time on you know even understanding how how is this affecting the reality on the ground of a goto market team much more than what's happening inside the product. Got it?
9:17 >> Because you know at the end of the day the winning is winning starts way upstream, right? It's not just something that happens when you're using a product. So, so we spend a bunch of time there. I would say more more specifically we do have a bunch of things that we always ask people right it's like you know how did you think of trade-offs on this thing like you know what what were the alternatives like what's the actual bottleneck limiting the rate of progress of this initiative why why why are we how is this creating more throughput on the bottleneck there are things like that that we ask but but you know we we spend a lot of time just clarifying what are the common patterns that we based on, you know, a bunch of meetings and a bunch of reviews that we have, you know, transcripts on.
10:03 >> This episode is brought to you by Riverside. I've used Riverside for years to record my podcast because it records in 4K resolution each person locally. So, the audio and video still comes through clean, even if a guest's Wi-Fi gets shaky. But the reason I love it now more than ever is what happens after we stop recording. If I go in here that I can use these AI tools to remove pauses, remove filler words and just clean up the recording. And I I can also edit the transcript directly and it will automatically generate clips with captions ready to publish to YouTube, Spotify, and all types of social media platforms all from one place. As a oneperson creator business, that matters a lot. Riverside is the upgrade your content workflow needs. Try it at creators.side.com/peryang riverside.com/peteryang and use code peteryang at checkout to get one month completely free. That's creators.
10:59 Peter Yang. Now, back to our episode. As a PM, I just want to talk to users and like, you know, welcome engineers and craft something great, right? But like there's a lot of other stuff, dude. Like you got to hire PMs, you got to write performance reviews, you got to do all this other crap. And like you actually have some really cool agents or tools maybe you can show us like that does some of this or helps on some of the stuff. Yeah, happy to. So we we spent a lot of time thinking about how to actually go about 10xing the productivity of non-engineering teams of Rex and because I feel like where where most people are in AI now is you have like the sort of token masters 10x engineers that are spending a lot of time in a coding harness and and becoming sort of hyperproductive and then you have you know people that are using you know basically like you know cloud code or codeax X and or SHP with a bunch of MCPS and the question is like what is how do you bridge the gap between the non-engineering teams and the engineers and from a productivity standpoint and and and a lot of the belief that we had Rex is like effectively that the best way to go about it is to focus on this idea of building virtual employees versus like you know sort of open-ended agents that could do a million things and and and and really the I would say the fundamental thing that we spent time on when we when we sort of went down this route was how to make it really easy to make it behave, look, and feel like a real person. so I can show you like an example here of the way this works. So this is a this is Jim. Jim is our openclaw recruiter. So basically it runs I mean he he posted we have a channel called welcome to bra where any new employee posts you know their their their sort of welcome to the company and this is what Jim posted. but anyway Jim is like an actual like it looks and feels like an actual person. So so you can actually like go and talk to Jim you know and and effectively interact with it like like a normal recruiter.
13:04 and and you know it's been running since February actually and it runs on effectively like a three-stage pipeline where you know it first syncs the greenhouse which is our ATS then we have of course you know notes and roles and applications in that system then it ingests candidates and parses resumes and rich LinkedIn profiles and fetches even GitHub data to complete a full picture on the applicant and manager evaluates every candidate against like you know ro specific criteria So, let me see if I can show you this just the way the UI works on Jim. Yeah, while it's loading. but the cool thing is like because it's been running for a while, we can actually just ask questions about it and see what happens. So, like for example, like if I ask like you know Jim, you know, what criteria changes have you made to the you know, senior software engineer AI role based on feedback.
13:59 >> Uhhuh. >> And this is all just open claw running on a harness. And think of it as like a set of skills deployed with like state right so you have like state of candidates that are you know in process you have connections greenhouse but it it's just a collection of skills and a fair amount of infrastructure that we build around eh halls to make sure this is actually behaves correctly. let's see what Jim finds. I don't know what this warning is but certainly interesting.
14:28 >> Yeah because I I really dislike using the greenhouse UI. So anything that that I don't have to use it >> green greenhouse UI is not not something we like. well well while it does that I can just show you the UI so you can get a sense for the way this actually works in practice. This is like basically the web UI that Jim exposes and you can basically see here everyone that needs action and how they found Brex. You can see you can go here and see applicants and and you can also see job posting. So it all starts with a job posting and you can see here like a staff rock designer role or you can go into like let's say an engineering role let me see like this role for example there's people advanced a lot that we rejected and we can see here you know sourcers we can also create a custom criteria for a candidate let me find someone that has a custom criteria here but you can basically come in here set a criteria and then we're going to evaluate candidates it's against this this spec. And if I go here into applicants, I can effectively filter for let me get let me actually source this by confidence score.
15:40 and you know, for example, this person here, you know, it's an exceptional fit for senior front end because they've been, you know, Microsoft for a long time and Coinbase. and we can effectively see, you know, the criteria score, where the career timeline came from. we can see the original resume from the person where they applied and you know this basically just saves a bunch of time from recruiting and the agent is doing a bunch of sourcing and flagging things that recruiting should do. So the idea is you can just create a job posting and say hey Jim go about it and then Jim's going to come back with with candidates to go in and spend and spend time doing it. We we spent a lot of the time just you know we put a little bit of the infrastructure around it but a lot of the idea of this thing was how can we make this like mostly self bootstrapped u so we spend some some time building an infrastructure around you know defining you know what it means to actually score a candidate against against a job posting for example >> but other than that like a lot of this was just like evolves and telling the model and prompting and saying hey like this is the wrong decision you should go and change this and change that and and if you go here into the Slack channel. You can see people interacting with Jim in a pretty clean way. So, here's a recruiter talking to Jim.
16:58 and you can see here like, hey, which role has the highest average candidate score? you know, are there any outliers? And he's like showing us like the roles. Or I can see top 30 applicants for this role. And this is just like giving us like a bunch of context on people. and it also posts like anyone that's like a standout candidate here with like LinkedIn links. So it's just like actually behaving like a coworker which is like kind of the the the coolest thing I think here. It also has like a self improvement loop where you know it just goes through like things that we should be building to automate more of the process and then you can go in and say you know here's a step to go and implement this. So the idea is like you can just you can just do a lot of that by building the right crown jobs and skills and and we do use open claw for this but the harness doesn't really matter as much as just the skills. So you can see how this creates a bunch of leverage for recruiting.
17:51 >> Okay. So recruiting can just like live in Slack instead of green greenhouse. They can just like >> exactly >> ask questions. Yeah. >> Exactly. And these are like people that apply now for example. >> So Jim is basically doing the upper funnel pass like filtering inbound. Does Jim source people too? Jim does source people too. it does it does like effectively three things. It sources people. it filters all inbound. It it started by filtering inbound and finding signal there. and third it serves as an analytics layer.
18:22 So you can just effectively go in and and you know enable autosourcing on the role. So let me just see if I can do that now. So if you go into job posting, so you can just come in here for example and say yeah like enable I'm not going to disable it but you can just like source candidates weekly for this role and this is going to go in and like find people using the job description evaluation criteria and sourcing criteria. So we're going to go run the evaluation criteria against candidates and make sure that we are only reaching out to people that actually feel like a fit for example. So this is just one example and again we're not in the business of building recruiting tools but this is just something that we build that is in our opinion what a virtual employee should look like.
19:05 >> Okay. So Jim will actually DM people on LinkedIn like hey you know uni from page. >> Yes. Yes. and and you know we there there's a bunch of things we do to >> make that email be and feel like a a person. But you know, yes, we we do spend a lot of time >> you know, making sure it can send on respon it's emailing on behalf of a recruiter or sourcer. You know, there's ways to you know, tell who is the sorcerer, who is a recruiter here. So, anyway, there's like there's like ways you can just see how this thing all like compounds and you know, becomes yeah, you can see here metrics as well.
19:40 So, anyway, it's a pretty >> Yeah, this is like a better greenhouse, dude. This is what greenhouse should have built. >> Yeah. Anyway, this is just this is just one example, but I think it shows like the the power of what you can do with AI and like and obviously like you know we have a team of like agent builders that builds things for like a lot of teams in the company but this oh nice >> you know creates creates leverage for one single use case right but but it all you can sort of see it all playing out with with you know it's sending emails it's doing a bunch of things that most people would be like okay this feels like you know it's pretty risky which actually gets us to how we make it secure to deploy this thing because most people would say, "Oh gosh, you have like a thing emailing candidates and like doing things that feel pretty risky on behalf of real people and that's that's like, you know, a lot worse than time of actually crab trap."
20:33 >> I feel like LinkedIn is basically like all AI now anyway. So, it's not big a deal. The gym is really >> Well, it definitely helps. I mean, to be honest, it doesn't actually use LinkedIn as much. It uses a bunch of other data sources. GitHub, Twitter, like is it's it's LinkedIn is like more where we get a profile versus where we go about, you know. >> Yeah. I want to ask you this actually like just to close this hiring thing. I I feel like a lot of people these days are a lot of AI builders I talk to, they're making a bunch of open source contributions on GitHub and stuff.
21:02 >> Yeah. >> Like like when you hire engineers, do you >> because LinkedIn is just like people can make up on LinkedIn, right? Like do you think the open source stuff is very important when you hire? >> I don't think I don't think open source is that important. I I mean I think it can be important but like I would say the presence of it is a signal. The absence of it isn't a signal, right? So it's like so I would say people that don't have it I think we our our interview process is pretty much a builder loop like you just have to go build something when you interview as an engineer. So if you don't know how to use tools and have a good sense for you know how to just be 10x more productive you just cannot pass the interview like you're going to fail because it's just too much work to do manually or do without AI proficiency.
21:42 >> Okay. Okay. So, no more elite code is it's actually building stuff with AI. >> That's right. Yeah. And this is just one example of like a lot of a lot of where we spend time is like how how do you create like measurement between like what humans are doing and the agents doing. So, this is like you know how many like how many missed recruiters like you know recruiters actually passed the candidate and Jim rejected and this is like people that Jim said that were actually standouts that recruiters actually didn't reach out. So you're basically monitoring the agreement rate between both. So anyway, you can just see how this thing compounds. But all these things are things that we built in service of you know creating leverage for internal employees.
22:20 >> Yeah. Yeah. You can make it better over time. Yeah. >> Yeah. >> That that's amazing too. That that that's amazing. why don't we talk about some of the other problems of enterprises? >> Yeah, I was talking about security. >> Yeah. Because that that's actually well there's actually two things we should talk about. One is security, the other is cost controls because this is all you know fun and games until you know you have a security issue and and actually that's where we build crab trap. So the idea of crab trap and and actually I can just show you the the GitHub repo because >> yeah crab trap is open source right so any company can use it. So, so approp's open source and really what it does like is this like proxy that like intercepts requests and and and the idea is like when you when you have an agent in production the only way to secure the agent in our opinion is to actually do something the network boundary because you know you see a lot of like Nemo claw and a lot of these like you know initiatives that happen in in the community around like constraining tools and things like that but at the end of the day if if the if the agent has the capability of running code which we think it should it can always like cross a network boundary and do something wrong. So, so what we've done is like how do you actually create you know an HTTP proxy that sits around the agent and controls all the network traffic and actually that's actually how a gym works. So the way Jim works is we have a crab trap here and I'm just going to show you in in audit trail mode first where there's a bunch of users. You can just like filter for Jim's user and this is actually showing like basically all the network traffic that is going through Jim. And you see here that there's like a bunch of like requests and this is actually live. So you can see this actually happening now. and you can see here like the approver for these transactions like an LM static rule, >> but I can go here and say I just want to see see things that actually went through the actual an actual LM as a gateway and and the LM here what it is doing is actually behaving as a judge and we're screening a request against you know the LM. So basically this is a get request to list emails account and this is consistent with the policy that we created for Jim and and basically what Jim is operating under is this policy. So let me just find Jim here and there is this production policy in place and this policy is actually like how we control the traffic of the agent.
24:43 So on every request we're first applying the set of static rules of here are things that we believe should be always allowed for the agent to go and do and then here are here is the prompt of what everything that is not a static rule should go through this prompt and we're actually running every request against an LLM and actually measuring whether it passes or not. So the way it starts you you don't have to come in and write the policy yourself. What you can do is you can you can just come in here and actually go on Jim's user and say, you know, suggest a policy based on the HTTP traffic. And what it's going to do is actually replay the traffic and and use another LLM to summarize all the network traffic that went through this agent and build a policy based on that. And what you're going to end up with and this is going to take like probably 30 minutes to run. you're going to end up with something like this which is you know it observed the traffic and created all these rules. So this is just effectively based on the network traffic that this agent observed. So you can see how this thing becomes pretty robust as you start to have evolves to start to replay network traffic and block things that are not allowed in a way that you wouldn't be able to do traditionally just by deploying an agent and saying yeah you know the world's your oyster go crazy >> and I'm reading the prompt right now. So is a lot of it just like blocking certain domains or like allow listing.
26:06 yeah and and and you know and for example saying you know here's what you can do in GitHub you can like you can research candidates but you cannot like go in and >> post on any other repo you cannot like push on a repo that is not like the claw right and and traditionally you cannot do this if allow list because that's actually reasoning about the nature of the request >> so that's that's like a lot of the things that we we end up spending a lot of time doing >> you know here's like an example what it cannot do you cannot like delete GitHub repost cannot change octa configurations. It cannot change policies and if it tries to do any of these things are just going to block them.
26:42 >> So if Jim goes to the piratebay.com it will get blocked. >> That's right. >> Yeah. >> Yeah. So for example, it was trying to access granola nodes and granola is blocked. >> So it can go in here and actually go in and allow granola. But for some reason it's not allowed and I don't know why we would allow it or not. But this is something that like you know it ran against sonnet. It took us, you know, two seconds to run it, thousand tokens in, 104 tokens out, and you know, it blocked the request. So, this is what we do to like actually secure agents in production. which is a huge part of what it takes to make this work.
27:16 >> Is is this whole interface also open source? So, I can set this up too. >> Yeah, this is open source. You >> want to? >> Okay, great. >> Yeah, this is great, dude. Yeah. >> Yeah. So, so anyway, this is this is the maybe the second thing we had to go build, which is like a way to secure agents in production. and then the third thing we had to build is a way to track costs and this is funny because tracking costs is part of our business right like Brex is a spend management company and you know fundamentally I would say we think that tokens are a massive well will be even more a massive lever of cost in a company and like many customers are spending a lot of money across you know anthropic openi bedrock talk, cursor, codecs, like you name it, right? Like all the all the sort of harnesses and and and models and and so what we had to do is like we just couldn't really understand how pricing had drifted on tokens based on the use cases that we had. So and and we think this is the wall that every company has when adopting AI. So so really what we did is we ended up building a tool that we call Magpby that actually helps you control token costs. so I'm going to show you what we've done.
28:31 >> What's your general philosophy on this? Like do you think every token maxing engineer is like doing a 10x job or like what's your >> well it depends on what they're doing which is what this helps you do. >> Okay. >> So so so the so the question is like how do you actually go about understanding token consumption in your company? >> So this is a this is like a very this is some of this is demo data. We should figure out how toize this but I'll just give you a sense for for how this thing works. So basically the idea is you can see here like effectively like three pillars of AI usage. you also have a leaderboard. If you love leaderboards yes you can you can go and do that. but the three pillars that we have first one is like corporate AI and this is like how are people using AI internally at Brex to be better at their jobs. Operation AI is how we use AI to better serve customers when it comes to driving more growth or automating processes. And product AI is the actual product we ship and how AI affects like literal features in the product.
29:28 and a lot of what we've done is is actually like think about how to create the ability to drill down really deep into token consumption in a granular way. So for example, let's say I want to go here in operational AI and understand what are different callers that are using models and what are these things are doing. So, for example, if I want to go into like transaction tagging is a very important thing we do, you can see that this thing costs 10 cents a call today and we spend like 28,000 roughly on this thing a month. and then you can go here and disputes and see that this cost like two bucks transactions. It's actually pretty expensive. It's 240x difference. So, you can go in here and try to understand, okay, what are things that actually driving this thing up and down and and you can see here a chart.
30:12 there's another way here I forgot how to do it where you can see like caching which affects like performance and cost pretty aggressively. >> Okay. So so so this is actually tagging customer spend variables right? >> No this is transaction tagging for like underwriting for example in fraud. Okay. >> is it making sense of like you know a very unstructured transaction. but I think that's cool is that like you can see here that like the number of calls that we're making is like we're making 329 calls and each call is costing us like you know roughly 15 bucks or or nine bucks and this cost like really skyrocketed over the past like two weeks for this specific thing. If you just go through like the the cost per call which is this line and you can see here that like if you go through transaction tagging this has been pretty stable over time. So something changed here that is driving this cost to go up. And I can imagine doing this across like everything you have and actually being able to plot and see which services are consuming the most amount of traffic and why.
31:05 >> and and and and and this is like something that we're actually really excited about because this is this is actually something that will evolve into a breast product versus just an internal tools because the thing is like this is just a usage side but then the important side is actually reconciling to the dollars that go out of your company's bank account and we own that data right because we are the card and we are the rail. So when you put both together, the idea is like how to get AI costs that's grounded in the money that actually clears and leaves your account.
31:36 so the idea is like every company that spends on Brex will very soon be able to have this whole thing for their own AI spend. And of course you can still go in here and compare, you know, usage by, you know, model and and employee and all that, which, you know, we think is really cool. You can see who's token maxing and all that, which is great. But I would say at the same time the the real value is just building this like entire layer of analytics where you can see the entire thing together into model use case you know you know can see which models are performing the best and you know and and so on so on in a way that allows you to actually track and do something about cost because we think this is going to be the biggest cost of companies moving forward.
32:17 >> And without this like what kind of stuff can they track? You can just track like spend per model maybe per employee. >> you can track spend per model. You can track spend per employee. You can you can you can track spend by caller. you know so you can see here like every single thing we have that uses AI in the company and what is actually using the most amount. So you can see here you know cloud code is at the top codeex is second cursor and then you have audit agent which is like a product we have transaction tagging in production. you can see like literally everything. and you can go here in product and be like, okay, how much is this thing also costing per customer. So you can see here, you know, at the same time, you know, which customers are the most expensive to serve. and this data is of course highly confidential, but you can see here, you know, a lot of games that take a bunch of time for us to and and dollars for us to go in and and serve.
33:05 >> So so you can just sort of see how this whole thing stacks together. So the problem we're trying to solve is like you know if my if my token costs get out of control I I can actually isolate the root cause and try to like optimize >> yeah you can just have like full token analytics the same way you have spend management you need to have like token spend management and this is really you know what we're doing what we're doing at bra for for us internally but you know at the same time will be very soon a reality that we had to build this for ourselves and then it became you know spend management has to understand AI natively which is what we want to go build with this thing >> what I thought do you give your engineers a limit limited token budget or is there like a cap?
33:41 >> we we do we do give them unlimited token budget. we we have a few caps because some people of course have used the process but we try to be pretty unconstrained if we're actually using it to deliver productivity. >> Yeah. Cuz because I I I feel like I feel like there's a difference between like somebody's like you know like like a Gary Tan or like even my myself like we have like free to tokens, right?
34:04 just just took max >> versus like actually people spending tokens on API and then it gets very expensive if you just like token max. So >> 100%. Yeah, I think I think I think my I think there's a couple things, right? Obviously, you know, not all like I I think one of the learnings that we've had at Brex is you you you really you really probably have to there's a first phase which is how do you create the the ability for people to go pretty deep and and and then you optimize it, right? And I think like if we had tried to optimize it before, it would probably not have really driven the outcomes we want. and I would say we're glad we did that that way. So, >> oh, go very deep. I think just understand the usage overall and then I think >> just just just let people go really deep on AI first and get AI build and then you can always optimize it later. It hasn't spent a lot of time trying to control costs in the beginning and now I think we have the tools to actually go in and very surgically say where does it have lower productivity and go in and do something about it because also a lot of what drives like cost savings is like for example are you using like caching efficiently when you go like have a product using a bunch of tokens if you're using corporate AI and and and you know and and and have like a bunch of adoption are you running everything on like Opus 4.8 eight or maybe table five soon or are you just you know or or or or you know can you use some things that can run on a cheaper model.
35:32 So answering these things is really hard unless you actually have a layer of analytics that can give you these answers which is why we build MacPI for us and we're going to soon have it available for for BEX customers. >> Okay. Yeah, that that was my next question for you. It's probably not smart for a company just be like a anthropic shop or like a open eye shop, right? like you kind of want to have multiple models in play so you can like orchestrate stuff >> and and the thing is like it's there's certainly the multiple models at play but there's also the fact that a lot of products will become metered on a token basis right so in the past you used to have like paying per seat and then over time you're going to start to see more and more products that are like cursor for example that at the end of the day they're really wrapping tokens as the pricing mechanism and so so so even if you're using, you know, anthropic and open AI, you're still going to have a bunch of products that may be indirectly using tokens, and and and and pricing per token. And making sense of usage, in these new patterns of consumption, I think, is something that is is much harder than than people realize. And so, so we think it's like absolutely shouldn't be a single model shop, but most importantly, you're going to have a lot of different things consuming tokens versus just, you know, opening eye onropic.
36:46 >> It's funny. What do you think? Do you think it's going to be like half the budget for a company is going to tokens versus humans? Or I guess it depends on the company. >> I mean, I think I think I think like I think it depends on the company, but the way I think about it is like, you know, if someone's starting a company today, I think the question is like why shouldn't it just be you, right? And and a bunch of and a bunch of and a big token budget and agents. and and and obviously I think there's still you know I'm going to talk about the role that that people should play but you know more and more we think a huge percentage of payroll is going to go to tokens and that's okay as long as you can manage it and make sure it's like the ROI is there.
37:22 >> Yeah. I I keep wondering if I should I mean I have like a very small company but I keep wondering if I should hire someone but now I just figured out how to use codeex to do what whatever I want that person to do. Just just figure out how to do it. Yeah. >> It's hard. >> Cool man. Just just a couple more questions. Why don't we point that thread a little bit? So, do you actually believe that there's like people, you know, these big companies are like laying people off, dude? So, like do you think there's be less people working in tech now or do you think maybe smaller companies will fill the gap? Like, what do you think is going to happen to our employment?
37:47 >> I think you're going to have there's going to be a shift in in in jobs, right? And and I, you know, there's a lot of the sort of, you know, huge unemployment duorisks and all that. And and my sense is you're gonna have some of that, but you're also going to have like a lot more jobs that we may not even know about yet. my my personal thesis is I think I think what's going to happen is you're going to have a much bigger conflation of roles and everybody's going to be a builder to some degree.
38:17 So, and I can show you like the way I use AI personally and show you a little bit of my own setup, but my sense is like I think what you're really going to be in the market for is like 10x employees and I think great people are going to learn how to use AI and and generally I think good people are going to see what great people are doing and replicate some of that. So you're going to have like a dispersion in productivity as you always have in performance like you know your 10x engineer is 10x better than your 1x engineer obviously and I think you're going to have a similar thing with AI but I think you're going to raise the floor and probably raise the se raise the ceiling and raise the floor at the same time. although I think you're going to raise the ceiling a lot more than the floor, you're going to have a general productivity increase. And and the second thing is I think we're so early in AI still, right? So realistically, you know, we there's a really good image that I like which is like and I can show you is like basically splitting the world in 2500 boxes. So like when when someone asks like oh god like what's what's going to happen in AI? I'm like, I just showed them this, which is like each dot in this thing is 3.2 million people, right?
39:29 Every red box is like someone that never used AI in the world. Green box is like free chat users. Orange box is like you're paying 20 bucks a month for AI. >> And this tiny red box is people that are using agents effectively. So, this is still so early, right? And I think like people don't necessarily make sense of that when they when they talk about like, you know, everything is going to go shift. And and the reality is like most businesses, most people, most companies are going to need to go through that transformation and it's not going to happen alone. You're going to need people to go do that. Yeah.
40:00 >> but in general, I think there's just going to be a lot more productivity per person. the same way that you know I I I kind of joke with you know a lot of a lot of founders that I think I I build that you know and you never been so productive and never worked so hard because you just feel so much more now that you can do more and you're just going to go and tackle these things versus you know sort of keep them in the corner and say oh you know I have more time or free time let me go to the beach now you know you're just gonna I think it's just the nature of being a builder you just get energized by the process so >> it's If if you're in that little cording harness box, you can literally be like 100x more productive than someone's not using AI. So, yeah, it's like got to capture the moment >> 100%.
40:42 >> do you want to like quickly show us some of your personal stuff and then we can >> I can Yeah, I mean I I have I have this this system and obviously, you know, we should redact this, but I have a system that I I call autopilot and basically what it does is like it's an open claw harness that basically runs my my life. And the idea is I have like basically two building blocks like people and programs and these are like programs that I care about. These are people that I actually spend a lot of time with on my team. and then the idea is it all starts with like signals and signals are like literally like Slack messages, emails, meeting notes and signals just exist in the context of like everything that I am doing and and they're reading think of it as reading my Slack messages and >> you know you can see here like reading my email and basically finding things that match either a program or a and you can see here this job it effectively like runs this like signal collector that's like collecting my email things on my inbox collecting meetings and collecting slack and then the whole magic is this thing called aggregate where really what it's doing and you can see here like the output of this thing really what it's doing is it is basically going through every person and every program I have and updating these files and and basically you can go here in PR and social for example and just see, you know, this is how I actually prepared for this podcast. So, you can see here that, you know, you know, someone helped me prep for for for this my camera was acting up. So, you can see here there's a few marketing campaigns we're working on. and you know, this is this is the this is the sort of status of the thing. And there's blockers and risks. And then the cool thing here is you can see here like effectively this like list of action items of everything that I have to go do based on based on this program. So I had Kobe on my team write a script for me to demo macpie. He's done that already. So I can actually check that.
42:50 and I have to go do this. I already done this. And these things here that are in yellow are drafts that this that that autopilot automatically created from signaling injection. So, I can click in here and say, you know, there's this DM that someone messaged me asking if I want to go to this conference. It ingested my Slack and this is like literally the DM. So, I was like, okay, like here's the DM. And it just like figured out that I have to go in and make a decision about whether I want to go and join this thing. so you can see how this whole thing compounds into like a pretty sort of robust task system. And I also have an interaction log. So, I can come in here and say, let's see, if I just want to see what happened today, I can come in here and filter. Well, sounds like nothing happened today, but today and yesterday, these are all the things that happened. And I can go in here and be like literally like, hey, here's just a DM. I can go in here and like give a give a like a view on the signal and actually see exactly where that came from. And this this whole thing just like runs in the context of that program. And I have this for like a bunch of other programs and a bunch of other people. So I can do this and have like effectively a download of everything that's happening on this and every task that I have to go track and every interaction log. So it's very easy for me to say okay I want to see everything that happened in one place in one glance and not have to go through slack email meeting notes and etc.
44:11 and also docs it can come in here and summarize all the docs that you know we I had interactive to make this thing work. >> Yeah. Okay. Okay. So, so basically you have a bunch of inputs from like Slack, Granola and everything else. And then and then are are each of these programs like a prompt or something like to kind of extract? >> yeah. Each of these programs is is just a markdown file. So I can go in here and like literally define what it is and then the signal injection is going to go in and take care of like filling it all out. I can this is all markdown. So I can go in and edit each of the files. I can edit the interaction log and then there's a bunch of scripts that just create nice UIs and make this like editable and searchable.
44:48 but it's all markdown file >> and then you have like a database or like a QMD thing or like >> it's all markdown files. >> It's all markdown files and they all get committed to a repo that you run. So it's so it's literally just text files and and a bunch of UI and and sort of >> you know nice things to be able to to go in and have this like job management system here. but you know it's effectively just like a way to consume the entire state of my life into one system.
45:15 >> That's amazing dude. I can't even get open call to work reliably. You have this whole thing set up. >> yeah, I mean Open CL is open C is a learned acquired taste when it comes to making work reliably, but you know I think it's worth it, but definitely takes a bunch of time to to do it well. >> Okay. All right, dude. Well, let me ask you one last question. I feel like the good old days of like SAS we can just charge a subscription for a niche product are are kind of like depends on what kind of SAS it is but it's kind of like dying a little bit because I feel a lot of stuff I can just use skills and like hook up to a bunch of APIs and do the same result.
45:47 >> So like personally even I want to build businesses I want to build products that make money but I feel like it's kind of hard to assassinate anymore. What are your thoughts on this? I mean, I think I think it's like we just changed the currency of things, right? But the reality is you're still paying for solving problems for people, whether it was like go build a SAS to do it or build an agent that is consuming tokens and you're charging a markup on the tokens. Like I think I think that's like fundamentally still true. The question is like I think the bar for how to build good AI products is just a lot higher now because you can just sell the work itself versus just sell the software.
46:23 And and I think like the reality of all good AI products is they're all the same thing which is you know an agentic loop and a measure of tools like that that that's really all it is. It's like you expose the tools to the model you run it on aentic loop and just let it do its thing. and and and fundamentally I think that applies to a lot of different problems and domains and you can go pretty far of that. I think the but I think the fundamental thing is like you're now able to sell the work itself versus just sell you know the the the tool.
46:56 >> Okay. So you have to sell the output of Okay. Got it. So it's like you go do it yourself. You can be like hey this agent will do it for you. >> Yeah. Exactly. >> Okay. All right Pedro. Well thank thanks so much man. This was this is like pretty eye opening. I guess where can people find you online or do or do you want to be found online? >> well go to X. I'm Pedro age 96.
47:15 so yeah you should be able to find me there and I try to keep the world you know updated to some degree on the things that I'm building and a lot of the stuff we're building at bra because you know we at the end of the day the the most valuable thing we can do is help companies better manage their AI spend and you know build great things with AI token >> max responsibly that that's the model.
47:34 >> Exactly. Exactly. Very responsibly. >> Cool. Nice.
Summary
- The traditional PM role is evolving; PMs now focus on using AI tools to understand customer needs and prioritize product features.
- Brex employs AI agents, like Jim, to automate recruiting tasks, including sourcing candidates and filtering applications, which enhances efficiency.
- The integration of AI allows for a more collaborative environment where engineers and PMs share responsibilities, blurring the lines between roles.
- Brex emphasizes the importance of discerning which problems to solve deeply rather than spreading efforts too thin across multiple initiatives.
- Security and cost management are critical when deploying AI agents; Brex developed tools like Crab Trap to monitor and control agent actions.
- The future of work may see a shift towards more productivity per person, with AI augmenting human capabilities rather than replacing jobs.
- Companies need to adapt to new AI-driven business models, focusing on selling the output of work rather than just software tools.
- Brex is developing tools to manage AI costs effectively, recognizing that token consumption will be a significant expense for businesses.
Questions Answered
What is the approach to building AI for companies?
The focus should be on creating virtual employees that behave like real people, rather than open-ended agents. This allows companies to sell the work itself instead of just the tools.
What are the challenges faced by product managers today?
Product managers often find themselves bogged down by administrative tasks like hiring and performance reviews, which detracts from their primary goal of engaging with users and creating great products.
How is AI being utilized in the recruitment process?
AI tools like Jim can automate outreach to candidates on platforms like LinkedIn, making the process feel personal while handling tasks that would typically require human intervention.
How can companies analyze their AI usage?
Companies can categorize AI usage into three pillars: corporate AI for internal efficiency, operational AI for customer service, and product AI for feature development. This helps in understanding token consumption and optimizing costs.
What is the future of AI in the workplace?
The future will see a blending of roles where everyone becomes a builder to some extent, with a significant increase in productivity as more people learn to effectively use AI tools.