The Future of Forward Deployed Engineering | OpenAI, Ramp, Nominal, Dataland

0:02 [music] >> First of all, thank you all for being here. This is an incredible turnout for a Tuesday night. Um, there are a lot of folks here who are starting their own companies and interested in a forward-deployed motion for their company. Um, a lot of folks who are interested in the FD function, um, potentially becoming an FD. So, we will cover a bunch of relevant topics tonight and have an amazing lineup for you. Um, before we get started, I'm Finn. I'm one of the partners at South Park Commons.

0:31 Uh, for those who don't know or are unfamiliar with the community, South Park Commons is a place for builders who are exploring their next uh, adventure, whether that is starting a company, um, doing independent research, working on open source software, to come and pursue technical projects of any sort. Um, we host events like this to bring together folks who are doing interesting frontier work in technology to to learn um, from each other. So, without further ado, we have Calvin, who uh, built the forward-deployed engineering team at Ramp, um, one of the kind of iconic New York startups now. Uh, Jason, who is the founder of Nominal, um, also was earlier in his career kind of at Palantir, uh, spent time um, working uh, across internal platform and forward deployment there. Howard, who is uh, a founder who actually is from kind of the South Park Commons community, started a company called Dataland, and also spent time earlier in his career at Palantir. And Colin, who uh, is here in town from the UK. Uh, we great to kind of have him while he was here in town, who leads forward-deployed engineering at OpenAI.

1:39 Um, so, yeah, thank you all for joining us. >> [applause] >> Rather than trying to give long intros for everyone, I thought a good place to start would be um, I've seen many different definitions of forward-deployed engineering. So, maybe uh, each of you could just share a a quick sentence or two about your company, um, and then go a little bit deeper into what does forward deployed engineering mean in the context of your business. Who wants to start?

2:03 Is this thing on? All right. So, Ramp, hopefully you all know, save time and money. Uh, corporate cards, expense management, bill payments, uh, the financial operations platform that replaces an Amex and Concur setup traditionally. Uh, FDE is, yeah, we we kind of, uh, use the term a little bit broadly. So, FDE at Ramp has a mandate to win enterprise, the the upmarket segment. And the way we do that, we are allowed to get very creative. So, we're not restricted in any way. We build core roadmap features.

2:39 We we do whatever it takes to win enterprise. That's the one-line version of FDE at Ramp. Awesome. Uh, thanks for having us, Finn and South Park Commons. My name is Jason. I'm the CTO at Nominal. Nominal's building, uh, data and AI platform to get hardware engineering data into the hands of hardware engineers. So, think people who are building satellites, nuclear reactors, um, and other next generations of hardware. Uh, forward deployed engineering has been like core to what we do since the beginning. One of our company values is empower their mission, and kind of as Calvin references, it's a broad mandate.

3:14 It's help the customer succeed. Uh, but it more like crucially, I think it's, you know, learn where the bleeding edge of product is, what do our users need to like fully unlock their workflows, and how does that like feed into our longer-term product roadmap. Excited to talk about it more tonight. Hey, I'm Howard. Uh, I'm the, uh, one of the co-founders of Dataland. Um, so Dataland builds, uh, AI for enterprise labor outsourcing. Um, and we work in a bunch of different sectors, like, uh, healthcare, energy, um, consumer electronics, uh, logistics, waste management. Um, so, we work in so many different sectors, and, uh you know, for the point engineering is is really core to our company because we we build, you know, highly heterogeneous agents depending on the company we work with, right? And so, it's it's literally the lifeblood of the company is like we don't we're not we don't we don't we don't we don't try to sell like a some sort of large platform core platform to our customers. Like we we sell them specifically where that that meets like sort of their needs for, you know, their their enterprise needs.

4:16 Um so, uh yeah, happy to talk about it more as we as we go. Hey folks, I'm Gohan. Work for OpenAI. We're trying to develop AGI. And um I guess to do that, uh you require two things. So, one of them is like broad adoption, and the other one is like super capable models. And those are kind of the two mandates of the FDE team at OpenAI. So, one of them is to like find uh I guess problems in the market that we think are repeatable, and embed with customers to build platform, and then decide to either like ship that as a separate product, or build it into something like Codex. That's like the kind of repeatable side. And the other bit is uh we have a a group that focus on like the most challenging industries in the world. So, we work with like semiconductors, life sciences, this kind of thing. Try and find the hardest problems, and then work with them to solve those like whatever it takes, which might be like uh help working with post-training to improve the model, or it might again involve like creating some good kind of application. So, that's kind of how we operate. Like we're the tip of the spear for OpenAI with the enterprise, and we uh try and make things repeatable, and we try and push the bounds of what the models can do.

5:14 Amazing. Uh thank you all for joining. Maybe a good place to start would be uh I read recently that the number of FDE roles has increased 10x uh from where it was last uh last year. So, obviously becoming a lot more uh popular. And it kind of strikes me as counterintuitive because on one hand, to your point, these models are getting a lot more capable. Um software generally is getting a lot better. Um but the importance of this role is kind of only increasing. Um so, I'm curious to just hear maybe a little bit more about kind of why why you think that is. I think Howard and Colin, you're both kind of working uh more with, you know, a lot of kind of agentic uh and native kind of workflows. I'm curious why FDUs becoming more important as these models are also getting more capable.

5:55 Uh yeah, for sure. Um well, I think that the scope of what sort of is is addressable in the in the B2B space is just much, much broader than it was before, right? Like before you, you know, you you could sort of build a SaaS platform that addresses some one workflow, try to sell it to everyone, um and there's only a certain amount of like problems of that shape in the world. Um but you know, the the the amount of uh we the amount of, you know, capital we spend on on on labor is is orders of magnitude more, more than that, right? So and um and I think that if you think about all the different uh jobs to be done in the world, I mean, it's it's dizzy I mean, it's obviously it's like it's so heterogeneous, right?

6:30 And there's not a singular there's not um And I think uh so I think in order and and be- because of the rise of AI, we can actually start to tackle, you know, we can start to tackle the full gamut of those problems. And uh and I think um and and because of that, it's like, "Okay, well, you need engineers who actually really understand the use case." Uh so you need you need to send them in and they they'll they'll almost need to, you know, they almost need to be able to do that job themselves, right? Um and that's really what a forward deployed engineer is. And then and then they but they must marry that with a deep understanding of the technology and of the fundamental, you know, frontier AI platforms like OpenAI uh in order to actually solve the problem.

7:09 Um so I think yeah, that I think that's kind of my take. And you know, I think part of the reason why coding agents have taken off so hard is like everyone is like, you know, every software engineer is naturally a forward deployed engineer in the in the coding space, right? Because like we all code every day. So we deeply understand that use case and that's why like the coding agents are so, so, so good. But um you actually need forward deployed engineers if you want to, you know, work in deep in the energy space to to to go in there and really, really understand that that that the the specificity of that industry and that role, right? In order to to to solve for to solve for with job to be done. I think the models are good enough to do it, but it's like what the unlock is actually the the the need for for FTEs in in my opinion.

7:47 Yeah, and yeah, I definitely agree with that. I think it's like I think if if I would look at one sort of shift over the last year is like maybe when we were building a year ago, when we were building off like the V1 of the agents SDK, like you spent a lot of the your time solving a problem just like writing a bunch of plumbing and building tons of evals. And for every like for every like problem you would have to build five different agents and five different sets of evals. And like the amount of time that it took to actually just like get to the point where you had something that worked took so much time that actually like the difficulty of the problems you could take on was much lower because, you know, you need to show the customer something in like some some kind of length of time. Whereas like what we found in our team probably over like since maybe like January, February of this year is like now with these capable like coding models like that that kind of they can do long horizon tasks. They can actually like you can actually sort of you build a lot less plumbing and you spend a lot more time solving like actual actual use cases. And I think that I mean like I guess like the fundamental thing about FTE is like embedding with specialists and understanding their tasks and then making sure the model can solve those tasks. So I think like that's how our role has changed a lot is like we spend a lot less time just like writing the same infrastructure every time and we spend more time actually like moving up the stack and solving like harder problems. If I use like one example, like we were working with this like semiconductor company for the last like probably 14 months. And the first 10 months all we did was like software engineering acceleration like, you know, deploy agents into their CI pipeline, like make like an auto debug agent and everything. But now we're actually like moving and doing like making agents that can do physical designs of chips and this sort of thing cuz like we're sort of building on top of like a more stable like like set of primitives where like we can kind of trust that Codex is going to do the bottom 50% of tasks pretty well and so we can focus on like the higher level tasks which actually like at the end of the day like give the business value. So that's how I've seen the role change probably over the last like six months. Jason Goldman, I'm curious how that manifests for for you too. Do you find yourself getting a lot more leverage or or kind of encouraging a lot more use of of Codex or coding agents kind of as as you go and deploy with customers today or is the set of problems a bit different?

9:50 Well, I was going to say something that uh just strikes me in the conversation even already is that uh it's not specific to AI necessarily. So, like when I was, you know, I interned at Palantir in 2012, and I think back then they were just the company that was crazy enough to invest that many resources in like a forward-deployed engineering team. And, you know, maybe in 99 out of 100 multiverse paths it would have failed, but it just happened to work. Um and I think what's happened is it's just gotten like cheaper and cheaper to produce software in general such that like to Howard's point like the number of things that are addressable by the software industry, by the tech industry has just like expanded a lot and then like absolutely exploded with AI. Um but in general, like whether or not you're talking about agentic or deploying agents or like deploying any kind of software infrastructure, like you always have to have this kind of a taste around like, "Well, what is the customer's job versus what is something that like our forward-deployed team would do?" Um and some of that will come down to your like business model and like where your kind of like risk boundary is. But, I think where it like really sings is when there is this like shared infrastructure that you watch yourself building at multiple customers and then that gets pulled into your core product. So, it might be in a model improvement or it might be in certain types of like how agents are orchestrated, but I would say at nominal like, "Yes, like we feel like the forward-deployed engineering team can be, you know, themselves like much more productive with agentic coding and and other kind of AI tools." But, uh I think you know, the way I think about forward-deployed engineering is kind of goes back to those other principles.

11:17 In our case, we're deploying to finance teams. Uh so, they are not coding. Some of them, but most most do not. Uh so, this is why some of you might have seen Ramp Labs. We have the Excel agent cuz where are they? They're in Excel. So, I think that's one of the general principles of forward deployed engineering is to meet the customer where they are. Uh what I what I'd add to some of this, uh our motion is a little less grandiose, I would say.

11:44 Uh we are just trying to not drown in enterprise. Like there there's a classic trap that a lot of software companies run into. You start with like a great PLG motion. You you get you get some traction and then and then you look at the enterprise world. There's so much revenue to be had there. And it comes along and it derails your road map. You you you start going for these deals and you just end up building so much stuff that only works for one customer.

12:12 And I think FDE at this point is probably the the classical solution to that problem, where you can have your core teams continue to build your road map and the FDE teams deal with the enterprise customers. So, I know that's a very negative conception of FDE, but >> [laughter] >> that is that is somewhat how it came to be within Ramp. Uh and so as a result, we're mostly just trying to I I I say it's a sword and a shield. We're trying to win the enterprise deals and we're trying to protect the core teams.

12:40 And yeah, so there's there's not quite as much like deploying AI uh as some of these other folks. Uh but that's really exciting. >> [laughter] >> What what was the impetus then for Ramp? Because I I think of it as a a very kind of product-led company, but then at at some point um you know, uh Eric probably came to you and said, "Calvin, we need you to go and and kind of, you know, solve this problem." What what was kind of that that tipping point um and and how did that conversation go? Yeah, when I talk about that trap, we were we were like walking towards it. And uh but luckily we had some foresight and we realized, "Okay, these large projects that the enterprise customers are asking for, this is not what we were planning on building if we didn't have enterprise customers. And therefore, like this is a bit of a problem for us, and we need to solve it.

13:28 And so, the fundamental idea is instead of playing this game of telephone where the customer talks to the account manager, the account manager talks to the product manager, product manager talks to the engineer, the engineer says, "Well, that's that's not good." Uh and then and then back and forth they go. Uh that's what we were doing before we had FD, and it was it was causing, yeah, road map delays uh because, yeah, uh everything I had just said before. And so, the idea behind FD at Ramp at least was let's have the engineer talk directly to the customer. Unlike a pound here, we don't actually visit them all that often.

14:06 Uh so, like once a quarter or so, you will will mostly do it over Zoom. >> [gasps] >> Uh but yeah, we will talk directly to the customer. The engineer will know what the customer needs and the Ramp code base and be able to think of solutions that the account manager and product manager would not be able to think of themselves. Uh and so, having one person, one very smart person, a great engineer with all of the context necessary can produce brilliant solutions that a game of telephone will inevitably miss. And that's how you solve the paradox of like, how do we win the customer but not derail the road map. So, that's that's kind of the the the FD thesis at Ramp.

14:45 So, you all kind of touched on something pretty similar, which is there's uh there's like a a fine line between a FD and maybe a a consultant or someone just doing kind of services. Like, where where where where do you draw that line in your business between what you need to go and build on a custom basis for each customer, one-off versus bake into the platform more broadly? How how do you decide where the the cut-off point is there?

15:10 I think I have a lot of scar tissue here. Uh just I think Palantir went through a few years of what we call the dark ages where uh the FD team essentially revolted and said that the product that had originally been built was completely useless for the customer engagements we were doing at the time and basically started to build entirely new things. Um, so when we started Nominal, we were like really intense about like not letting that happen and one of the things that we did was we took people who had like, you know, built software platforms and uh had been on the receiving end of those like road map telephone games that Calvin was referencing and basically put them uh as they were the first four deployed engineers and so they could like I could have really um high trust conversations with them that they weren't basically going off and doing consulting and that if something came up that was like too big that we'd be like really intentional about it. Like sometimes, like another thing that we haven't touched on yet is that uh four deployed engineering can accelerate sales cycles and sometimes that's just a win-win. You know, I will spend like Nominal points against um you know, instead of waiting 12 months for a large legacy company to get to a certain point in our engagement, like we can get them there faster and it's like worth it for all parties. Um but you want to do that like really carefully, especially as you're growing your team.

16:21 Are there any examples of of kind of a place or or a customer where it really accelerated things or or kind of helped uh move that sales cycle along? I'm just curious how that how that manifests. Yeah, I remember like our um honestly it was like our first large contract that we signed, the technical counterpart there was really pushing me. He was like, "Hey, once I get the data into Nominal, um my engineers love using it to understand like they were doing drone flight testing and so um his goal was get 40 people in his organization looking at data every single flight on average before us it was two and uh you know, the first MVP was the data had to kind of go through these hoops before it was ready to be analyzed in Nominal and he was like, "These are scripts that my uh my engineers need to be running on their laptops. It'd be great if you guys just handled this for us." And he didn't really know what that meant from a software architecture standpoint. And um we had a guy on our team, Ross, he was a four deployed engineer and he and I concocted a plan where he was basically going to build this for them, but do it in a way that was generalizable across other customers, as opposed to just like bespoke to their specific data format.

17:27 So, uh you know, we built the architecture in such a way that they didn't really realize that, you know, it had this like it was a container that you could upload that had the data transformation logic that, you know, Ross wrote it for this first customer to basically prove that the architecture worked and get them to their, you know, mission success. Um but it became this kind of like core thing that we ended up selling across the rest of the fleet.

17:48 And I would tell people like this was on our road map, you know, I I knew we would build this at some point. We just kind of like pulled it left because the opportunity presented itself. Yeah, yeah, sure. Then yeah, I guess just on the like, you know, what your product uh like you know, like what OpenAI offers is like a repeatable thing. I think like we probably evolved this like twice at OpenAI. It's like I I think we started with like actually like far less grandiose kind of goals. It was basically like I think a lot of you probably seen the article. It was like 5% of enterprises actually are like getting ROI on their on their on their like investments in AI. And like we were we always said, "Well, if the other 5% work with us." But but we but we all but our take at the start was like, "Well, we're just going to get like zero to one success is like always the measure of success." Because when we started, we like didn't have a platform at OpenAI.

18:32 It was like just the API and then the customer and then like you were in between. So, we just built a ton of stuff and then we started to find these like kind of like these like these kind of like pockets of product that we thought we were going to build. And probably about 6 months ago, if you'd asked me what the FDE was going to build, it was kind of like because because AI is making it easier to write software, it was like we're going to make like 50 of these little products and maybe we'll make like an ecosystem and other people can make products. And then interestingly, then suddenly Codex got bigger, they got got better. And then and now like the question we ask of every product is like, "Can this just be a Codex extension?" And that probably ate like 80% of the products that we were making. And the two that we had left like uh one like Pearson in the crowd is actually working on uh around like regulatory document authoring, and then there's one on uh that's more of like a kind of workflow automation platform. And for these, it's like for the enterprise where they need a very high level of consistency, there is a product like argument there, but everything else kind of folds into this more PLG type like Codex thing. And that's kind of what we do in OpenAI IFTD now is like for everything it's like, can this be Codex? If not, like does it really have to be a separate product?

19:35 You know, can could the model just like get smarter and be good at this ask? So, this is kind of it. I guess uh I guess to answer the question of like, is it like what's consulting versus like what's like software or something? I Especially as a founder, too. Right? >> Yeah. Yeah, yeah, yeah, for sure. >> to find, you know, your early customers. >> And then we're we're way earlier so I than than than these guys. So, I think um but I you know, I I tend to think of this as just the way of, you know, public markets evaluate like companies, right? It's like you know, and like what what companies have uh like as a financial asset, what makes something a a software company versus a consulting company? The question is, is there recurring value that you're delivering to your customer on on some sort of fixed cost, right? Like at the end of the day like that is sort of the economic model that defines the difference. Um so, I and and I think there's many ways in which this can be true without necessarily being like, well, I have a SaaS platform that I'm trying to sell everyone. Um and uh and I think especially with things like Codex or Cloud Code, as the cost of software creation goes down, it is not um like you know, I as a younger company, I'm not as ideological about like, hey, we need to figure out some some core thing that we are trying to sell we're trying to sell everyone. I think um I think the way we think about it is is uh what can we learn from every single one of our customers uh that allows us to accelerate the process of building these custom agents for the next customer, right? So, how how how can we have this flywheel of learning that, you know, that reduces that initial fixed cost? And when once our once our agent is actively doing an at-scale enterprise workflow, it's like it's it's it has extreme recurring value over time, right? So so for each of our use cases, um there is this sort of fixed-cost recurring value equation, which makes it into, you know, a a sort of a more uh like a the thing you want, which is a you know, software economics. Um I think Yeah, I mean uh this is Yeah, this is something that Palantir thought through for many years as well, which is like, you know, is is FTE a bug or a feature? And uh and I think for um and uh and and and Palantir's um uh and I think for Palantir wasn't sure for many years, but I think like uh I think uh uh even in my time there, the realization was FTE is is is is a huge feature, right? It's it's not a it's not a bug. Um and uh and yeah, so maybe I'll I'll leave it leave it there. Is there a way you guys measure kind of ROI from the FTEs that you have out there in the field? Because I would imagine at some point if you're successful, you probably want to invest less and less time um you know, with uh your FTE spending time with your customers because a lot of what they've built should be, you know, helping that kind of customer scale maybe themselves.

22:10 Um so I'm curious like how how you measure that and then at what point you kind of start to hand off what you've built um you know, for your customers or alongside your customers to them? Yeah, it's really simple at Ramp. We have the revenue from the enterprise customers and the costs of the salaries. >> [laughter] >> Uh so do that division and you get the ROI. But but to to to give something a bit more substantive, uh we we are always trying to inform the road map and or execute on the road map.

22:41 So just just like we were saying here, just pulling things to the left. Um we love it when we can build a road map feature for the customer instead of having to build something custom cuz the custom stuff tends to require more maintenance. Ramp FTEs are operating in the core code base, to be clear, as well. Uh so they have close partnerships with each of the core product pods. And we are always trying to keep our changes minimal if we can so as not to introduce maintenance costs down the line. So, we we have a a pretty restrained team and we try to serve a lot of customers at once. So, one FTE is taking on like five or six when when we can. Uh, doesn't always work out that way, but that's that's what we strive for. Uh, and so that when when you have that set up the RI becomes extremely attractive.

23:34 And I think the the FTE the motion at OpenEye's probably like slightly different because we're we're like because we're we're sort of I guess we're trying to bet on like problems that are going to save customer like 100 million hundreds of millions of dollars, billions of dollars. Like so, if I solve this problem, it will have like a kind of outsized impact and so we generally go deeper with like a fewer amount of customers. Like I think one of our our biggest engagement has like 15 FTEs at this at this semiconductor thing cuz we're trying to like basically change the whole like the whole value chain.

24:03 Interestingly, like the the actual probably the most like long-term profitable engagements are more the like product-based ones which might have like only two to four FTEs, but we're really like like the sorry, taking a step back, I think like the reason why a lot of consultancies do FTE like sort of badly is because they are like the services revenue is like a drug that they just like can't get off of and they just kept keep selling like bigger and bigger custom things. And like the good like probably the good thing about OpenEye's that the product like we are not the power center of the business. The commercial arm is not the power center of the business like product and research arm. So, they're very much pushing us to like build things that they can then that we can like the the the the market can self-serve or like, you know, we can do with a very small amount of effort in the future. So, we're kind of like trying to look across the portfolio and be like, "Okay, what are the verticals that we could solve these massive problems for and maybe get like some kind of recurring revenue for?" And then what are those repeatable problems that we could like ship I I know, this workflow automation thing. Is that going to like make a ton of revenue over time? So, with us it's always like the services line is like a small focus, but I'd say like the bigger focus is like what is the long-term value? What is like the ARR that we're potentially going to unlock for like all the digital native customers that are then going to adopt this thing and like cuz that's really where like the value comes for us.

25:13 It's kind of an interesting um one takeaway from that I guess for for Hard and Jason is that uh I I feel like you know, if you're doing the FTE motion it's like you said the the revenue is kind of a dragon so it's very tempting to you know, like you almost have this mirage maybe of uh product market fit where you know, it feels like you're starting to make a a bunch of money, you're ramping up you know, your kind of revenue, you're not actually really developing kind of anything scalable underneath and and I think we have a lot of founders in the room who are thinking about doing this motion and you know, at what point do you maybe like fall victim to to kind of that feeling of of the revenue growing but don't really have deep product market fit. How have you guys thought about that?

25:51 Yeah, it's funny. I don't know if I think of it as quantitatively in terms of like directly just the revenue and the cost. Like I think that's obviously part of it, but um you can end up in a trap. Like I guess I've seen this in the past where like the customers actually like the customer is addicted to the forward deployed engineers more so than you as the company are addicted to the services revenue and then when you try to pull the forward deployed engineers away they just fire you and that that's that's bad. And so like you want to be kind of intellectually rigorous about like are you you know, are you viewed by the customer as a consultant? I I think there's like a really magical sweet spot which is that by forward deploying and like I do uh want to push you on like the going on site piece because it can lead to this um they they value the product, you teach them about the product and they view you as a thought partner and they'll tell you more candidly, you know, feedback about the product, ideas about the road map um and then when you go and engage with them like you might even be sending, you know, a product team to embed with them and kind of like be a build partner and they're willing to do that because you kind of established this relationship with them. Honestly, I think just to be vulnerable, I think I've had to kind of learn more about traditional sales cuz Palantir didn't do that back in my day. And a lot of like sales teams like this is what they do is they kind of develop that rapport, but part of the magic of forward deployed engineering is just making sure that it's like, you know, there's not a game of telephone. It's like the engineering team is directly developing that rapport.

27:16 Yeah, I think I think the way we think about ROI is is also revenue [laughter] divided by the number of head count. But I I I think I think there is You guys are only two head count. >> Yeah, well well we're we're we're going to be at three soon. We're we're adding >> [laughter] >> and we're you try trying to get to like 10 or something. But I think but yeah, I you know, I I I think that yeah, I think the critical judgment call you have to make is like is the value occurring, right?

27:40 And like if you if you build something, you know, if you just build whatever they ask for, like there's infinite things they'll ask for, right? And and then you know, it might deliver you know, it might it might deliver like, you know, a momentary piece of value, but then that value evaporates. And some sometimes actually tactically you maybe need to do that in order to sort of unlock the, you know, other use case. You have to build trust, you know, you have to you have to unlock sort of the bigger fish, but I do think that ultimately you have to really keep in mind like, hey, is there is there this long-term continuous value value delivery that you're able to do based on your fixed unit of of work, right? So I think that's that's the framework that I that I think about it think about it in.

28:24 Yeah, and I say I say two people half jokingly because you you've obviously gotten to extremely large scale for just two people. So I also think it kind of speaks to maybe kind of the leverage you guys are getting from some of the systems you've built because I don't think that two people would, you know, I don't know if we'll see the the one-person billion-dollar startup soon, but but you know, I I think, you know, the the trend line is definitely heading that way in terms of the amount of leverage people are are kind of getting just in their own work. Yeah, we we are extremely AI piled.

28:55 Like I think you know, so like we we spend a very we actually spend a lot of our time building our our our meta meta agents which which accelerate our own process of building new agents, right? And of maintain like like no agent you build is a static thing in time. It has it's a it's a dynamic actor within the enterprise context of your customer. It has to you know, their their needs for that piece of labor will evolve over time, right? Like their business is going to change, their systems are going to change, their their policies are going to change, they're going to release new product lines that interact with your agent that your agent must know about, etc, right? So it's so how can you you know, put in this fixed cost and get this continuous long-term value, right?

29:37 It's like like you actually must figure out some some sort of mechanism, right? For like to to to actually autonomize the outer loop, right? Of like how do you as as you know, as the business changes like how does your agent keep up with that? How does it sort of have to sort of self self-improvement loop, right? And so we we spend a lot of time thinking about that outer loop as well in order to give us a leverage to be able to you know, we we have multiple many millions of ARR per per per head per head count. So Yeah, I'm I'm curious Khan what that looks like for you all because I think what Har just kind of touched on is you know, there's obviously I feel like product and research are becoming more tightly intertwined and now you know, I think kind of what I'm hearing today is that you know, go to market, product and research are all now three kind of becoming more tightly intertwined when the FDE motion is done well. So I'm curious how you guys think about interfacing not just with product and and the products you build, but also the researchers too in terms of probably a ton of really useful customer data that you can you can get and feed back into improving things. Yeah. Yeah, yeah, for sure. Yeah, and that's kind of like a key part of the FDE model at OpenAI where like you know, outcome first so always like success and then scale and you've got two ways to scale. One is product, the other is through the model and like can we generate e-vowels on this task with like synthetic data that kind of mimic the task and then can we make our model like much much better at this thing and like I guess practically like one one that like the model was really bad at was like slide generation and then we were working with this this like this like Japanese sales team with like these 2000 sales guys and they wanted like this like slide buddy that they could work with. So we so we we basically like first we had to find how do we format the slide so that the model can produce like quick slides that actually look good and it starts off as like okay you got six boxes and you can only place stuff in the boxes and you're like all right that looks [ __ ] terrible and then you're like okay, you know, okay generate HTML and then you can have like these super nice, you know, so you try like 50 things and then you settle on one and then you generate like a bunch of examples and then we take that to the post training team and then in three months another snapshot pops out and then suddenly like the slides are really nice. So that's kind of like the that's like sort of what the flywheel looks like is like I guess this like you know FTEs embed with the customer figure out like how to represent the task to the model and then generate lots of examples of that and then the research team can like understand how to make the model do that thing and it's often like it's like pretty esoteric sometimes so that's like that's like roughly how it works. I think one other example I'd give is like we worked with a big telco to do like voice customer service and like I don't know if you guys have used the real time model but it is like it's like like I'm getting it to like follow you [ __ ] instructions is like really hard and like you know getting it to follow like customer service policies like obviously like pretty important that it follows the instructions. So this is the point that like I think when we pitched them we did a test of 10 to just get it to read back my phone number and it like just would not work and we were like oh my god >> [laughter] >> and then and then like six months later we shipped it and now it's like like something like 70,000 calls a day getting deflected by this like AI real time thing and you know no major jailbreaks so far so and that was like probably like you know 6 months of iteration with the post training team to improve the model and also building as FDEs like a bunch of useful platform on top so that they can also they can create their own evals and like make their own self-improvement loop for like subsequent runs that they don't need the FDEs, which again is like sort of the as these folks said like is the is the dream.

32:57 You guys have any thoughts there? I was just curious, did you put all of McKinsey out of a job with slide buddy? That was That was all I could think about. Yeah, we actually do have some McKinsey guys in the team and they were like, you know, shedding tears for their their former their former [laughter] their former teammates, but it was yeah, yeah. The slides are actually not very good. So, like so so it'll take some it'll take some it'll take some time before McKinsey's out of out of a job, but yeah, we're we're definitely we're trending in the right direction.

33:22 More work to be done. Um, I know I want to leave some time for audience questions, so maybe just a couple of more from from me. I want to kind of ask all of you to I'm curious what you look for in the FDEs that you bring onto the team. You're all obviously all growing your teams pretty significantly. And maybe Jason and Howard we can start start with you because you've seen this over, you know, probably the evolution of the FDE for over a decade now from your time at Palantir. So, I'm curious what what you think makes a great FDE. What do you look for as you're as you're hiring folks?

33:52 What are the the skills that that you want? Uh, yeah, I mean I I I think you you really need that strong you really need to be like very very strong technically, but also sort of really keep up with with all of these frontier model changes like Colin said as like, you know, the number of iterations on just a real time model. >> [laughter] >> Like we you know, we're constantly, you know, keeping up to date with that, right? And and we're constantly delivering that new frontier capabilities to our to our enterprise customers.

34:17 So, so you have to be, you know, really good at traditional software engineering as well because you're building really complex integrations, right? Within your in these enterprise environments. So, so it's not just oh, I'll you know, it's not just the non-deterministic model pieces. And but there's this really you know, the other half of it is sort of the account management and the customer success halves, right? And uh I think you know, FDE really is you know, I think it's really sort of the best training ground to be a future founder cuz you like like in order for you to succeed at this role, it's like you actually you have to like be able to build from zero to one, right? You have to be on the cutting edge of AI, you have to uh you know, be be good at um you have to be good with people, you have to build you have to win trust, right? With with these enterprise um customers. Uh you have to fig- you have to figure out the politics of their organ and map it out and traverse it, right? so I think um and uh yeah, so I think you know, you have to really we're looking for people who can span that full full gamut of skills. And it's it's very hard to find, so if you know, we we'd I'd love to talk to you if you guys think that uh you have that skill set, so.

35:23 Yeah, Howard's doing a good job pitching it cuz I I think it's true that I spent, you know, only one of my five years at Palantir doing forward deployed engineering, but it was like honestly I learned so much in that time and it's just I draw on it again and again. Um even just being like an early stage engineer at some startups later on, um you have to be really scrappy and kind of like be super generalist in the way you solve problems, like curious about all types of things, um and and humble in times when it's like necessary.

35:50 Uh I think another thing I was just thinking about, I don't know if we touched on this, but rotating between forward deployed engineering and what you think of as like core product engineering is really important. Um again, some scar tissue born of like a you know, if you try to grow those organizations separately, uh you end up wanting them to be more conjoined so that you can like more smoothly rotate. So like at nominal, um you know, one of my happiest moments was was like watching a early career engineer who'd been building a product fly to Europe and like try to use it on site and then come back and be like, "Oh my god, it sucks." Like I and it was super motivated to fix a bunch of parts of it.

36:25 Um and then kind of the other direction as well, like someone who's been doing a lot of forward deployed engineering, like them having a really like they have conviction and passion about a certain thing that otherwise wouldn't be on the product road map, and they just eventually are like, "I'm going to go build this myself." Um, to that point, I think like at different times in the journey as a startup founder, you have to be careful about the role can evolve and your needs as a company can change. Um, Ross Fubini, who's this like wonderful investor we work with, he talks about you go through these product expansion and contraction phases as a startup or as a company, and I think at different times in that expansion and contraction motion, you might want different things from a forward deployed engineering team, and you know, again, I think there's a humility in people who are willing to like roll with that uh, change, and also you might actually like change your hiring profile at different times as a result.

37:13 Similar to what was just said, we like to hire former founders. Uh, people who, yeah, care about revenue. That's that's a big thing that, you know, a lot of engineers don't. >> [laughter] >> They just want to go and build something cool, and you know, that's actually really wonderful in many cases. Uh, but yeah, caring about revenue goes a long way. Caring about the success of the business. I I have a phrase that like the FD team is the team that wants to say yes because they care about us winning that customer. A lot of engineers, they might not admit it, but they they want to say no to what the customer's asking for.

37:51 They want to continue building their thing. Uh, which is also admirable in its own way. Uh, but yeah. And then the Yeah, so former founders, early engineers, people who can talk to the customer, also not true of all engineers. Uh, so there's an additional screen on the FD interviews that we do where like it's can you actually communicate? >> [laughter] >> Not a not a hard requirement for for regular engineering. Uh, you don't normally have to put them in front of customers, right? Anyway, and perhaps all of this is obvious, but uh, yeah, the team that wants to say yes, I think is is pretty fun. We were very grateful when the customer appreciates Yeah, they do get a little addicted.

38:34 >> [laughter] >> Yeah, they're they're generally very happy to have someone a real engineer they can talk to. Colin, you mentioned you've gone from like what, two people when you started to to 90 plus just in a couple of years. So, yeah, what about it for Yeah, for sure. And I think like largely I agree with what people have said before. Like I think it's like it's like the relentless pursuit of value is like what makes good and a good forward deployed engineer. And I think so much of the time people love that start to love the form of what they've created more than the function. And when it's like when you just look at the users use it and they don't use it, they're just like, oh, but like we built the thing. Like you got to like the thing. Well, when actually it's like the the best forward deployed engineers are like, right, we'll just like tear it up and let's make something completely different like because that's what they need. And that is like probably the most key thing like because we have folks from like all we've like a fairly diverse set of backgrounds. Like we got some BCG McKinsey folks, we got a ton of volunteer folks, we got a lot of ex-founders, but the common thing is that they're all just like super outcome focused and like they really don't care what the answer is as long as it's like as long as they're using it. And that is like that is kind of the proof. So, that is the That's probably the most important thing that I've that I've seen.

39:38 Um maybe with that I want to open it up to the audience for some questions as well. Who has a question? You want to Should we start over here? >> [clears throat] >> You just stand up and say your name as well. So, yeah. Hey, I'm I'm Varun. I'm a FDE [clears throat] right now. So, this is very interesting to me. Um I think you guys spoke on like a really interesting tension that exists between kind of within FDE orgs, right? Which is you know, customers really get addicted to the FDE motion. Um but FDEs as an individual, right, you really want people who are high customer empathy, want to say yes, you know, really want to own the customer outcome. I guess have you guys you know, built any tactics within your orgs to kind of counterbalance those tensions, right? So like organizationally making sure that like you know, you're able to pull back FDEs once the customer is you know, maybe had their needs satisfied, you know, in a way that your your customer doesn't like fire you and you go from, you know, 10 people in the org and scale it down because you feel their needs are met.

40:36 But also obviously like building a culture of saying yes. I can go really briefly on this. Uh my answer's probably different from the others. As you can probably tell, we stretch our FDEs very thin at Ramp. So any given customer, they only ever had like a third of an FDE. And so if that goes to like a quarter or a sixth, they they they don't notice quite as much. Uh we we've never really had like two FDEs on a single customer. So that is admittedly a departure from the traditional model, but we find that that creates the necessary incentives where the FDE is looking at all of their customers and trying hard to put their attention where it's needed as opposed to like just Yeah, they the customers are all asking for stuff and the FDE has to look at that holistically and think to themselves, what will actually move the business.

41:27 Anything anyone else would add? Yeah, I mean Yeah, if the customer's going to fire you because you reduce the headcount, that means you haven't done the the the value engineering of like, are you delivering continuous value, right? Cuz if you are, then even if you take those people like the the the the people are should should not be the thing that's that's delivering that value. Um and I think Yeah, so uh Yeah, and and but also we're we're the same.

41:48 We have, you know, many accounts per per person. So so they never got addicted. They never had the expectation that I guess there's like, you know, five people show up and um and and you know, we they they kind of what they're paying for is the the man hours of these five people, right? Like we we I think and I think part of the part of the FDE's job is to communicate that that's not what they're buying. Um right? They're they're buying the the outcome or the continuous value delivery. So.

42:14 One final point is I've seen companies that have FDs assigned to customers for 12, 24 months, or even longer. And then I've seen others where it's like 6 weeks as a rule. So, that can be one way to limit it. Other questions? Maybe we do one more up there and then we can come back up here. Hi, I'm Richa, like Rich. My name is Sorry, I have so much Uh I mean Richa A at the end. Uh I'm graduating from MIT very soon and joining as FDE.

42:43 Um So, I'm seeing a lot of roles out there which actually matches my interest and roles and everything. But, FD is definitely unique and that's why I'm here today. So, when it comes to like solution engineering or solution architect, when the pipeline do actually fit and where do you see you come in or you like you make a distance? So, like that's what I want to ask. Yeah. So, so so I guess speaking to OpenAI, like there's there's kind of a delineation between those roles in OpenAI where like um solution engineering is pre-sales for like the scale motion, so like the wider market.

43:18 Solution architecture is the scale motion post-sales for like the wider market. FD is actually like a different business unit um that is like separate from those and we only really work like 10 engagements at a time. Um so, it's like very it's sort of like almost a different motion where like we get qualified in very early if they think this is an FD shaped problem. And then we run the full like pre and post-sale from that point because it's almost like we're like a cross-functional team that land and we're like, "Okay, you know, is this something we want to do?

43:47 Are there milestones? Do we think it's going to convert into a recurring revenue?" We have sort of like a bunch of checks we do as like whether this is something we want to do. And that means like unfortunately, like though we though we do want to say yes to like every problem, we're very like excited by every problem, we have to say like no more times than we have to say yes cuz there's a lot of people with interesting problems, but not every problem is going to like get to scale and to like ARR or like has a chance of doing that. So, um that's like sort of how we how we prioritize in in a in in a open end.

44:14 Well, can we uh maybe come back up here? We had another one right here. Cool. Thank you so much. Um I'm Emmy. I'm also an FDE right now and I'd love to compare notes on the team structure that exists within your companies on the FDE team. So, at Palantir for example, my understanding is there's Echo and Delta. And so, I'm wondering like does that is does that kind of resonate with how you structure your team at your respective companies? And if not, then what does that look like? Like what is the delineation of responsibilities between uh the different individual the different individuals that uh consist of your FDE teams?

44:54 I'll jump in. So, at Nominal we have we call it mission ops, mission development, and then we have um a growing sales team, so account executives. And, you know, mission ops at Nominal are people who are highly technical. They tend to be like former mechanical engineers, electrical engineers. Some of them designed engines before. Um but not necessarily like code native in the way they approach things. Uh increasingly that's changing just like with AI. I think a lot of them are like, you know, they can't help themselves. They're writing a lot of code. Um and you know, the delineation responsibilities kind of falls down to uh you know, if you really need to understand like what is the workflow of the mechanical engineer who's using Nominal. Like it's really helpful if you have that background being part of that organization before and that's why we have the mission ops role. That's kind of like the origins of the Echo role at at Palantir. Like very loosely speaking.

45:41 Like they wouldn't always have the background, but they would, you know, maybe read a book about it. Um they were generally intelligent people who like maybe had consulting backgrounds and were used to kind of like going zero to one in a new space. Um and then increasingly like I think we're like we're a growing company like uh you know, Nominal's around 150 people and growing every day and uh we're trying to like figure out really carefully like well, what is the delineation between like an account executive who might be doing some of the report building like understanding uh you know what is the path to increased value at this company like over the next 5 years versus Palantir like that role didn't exist. So good questions to ask and I would like just kind of a meta point to this in the previous question but anyone who's doing forward deployed engineering or like considering a job in forward deployed engineering because it's such a broad role like ask a lot of questions like this of any company that you're considering working with.

46:27 I I would say for us it's like uh I I think we think of it as yeah so I was also at Palantir and and and I I was actually a echo intern and then I was a delta but I I think um I think for us we actually see this like where I think where AI has has has allowed us to unlock is where this radical ownership model that's now possible where you know the forward deployed engineer like if they have the skill set right should be able to like you know it used to be that hey it was just too much work in a particular deployment for you know you had you needed the deltas to code right and then you needed the the echoes to go and sort of own the customer relationship and figure out new use cases and stuff but if you can dramatically reduce the cost of of of the production of code then then perhaps you can have this radical ownership where one person can actually hold all of that context in their head right and and that's beautiful because they also know you know what what's hard and what's easy and they're not they're not just a you know a pure non-technical person making those value decisions right so I think so that's what we're trying to make happen and we'll we'll see if it's possible if we can find the people with the skill sets.

47:33 Yeah and yeah and I guess like just the the opening eye model is like sort of like I guess maybe similar to the the initial one where it was like like we have this sort of echo and delta concept but I think like I came from like like classic consulting back in the days and I like hated like the amount of roles that there were like there was like specific shapes of people and then like all these different higher like it was like a total nightmare. So when we started the team there was only FTEs and what like there's some of the early FTEs are in the room and like we realized very quickly that we like really needed the the echoes to like take up a lot of the work around the account cuz these were big accounts, there's lots to do.

48:06 So we have we then introduced that role and then do we've just started to add these like experts industry experts where like we're working in semiconductors and stuff. What we're finding is like I was saying earlier like we're now solving harder and harder problems and what that means is like in AI means that you have to be able to write like good tasks and evals for those things. And it means that like when we go up the stack in like chip design, we really need to know like how to design a chip. So we're now introducing within the vertical teams like chip verification engineers or like people in life sciences who work as scientists or whatever. So like there's a small number of them, but again there's they like what we're hoping is that they'll have an outsized impact in terms of like all the generalist FDs will learn a ton from them and like still do their general FD FD stuff.

48:48 Super interesting. Maybe we have time for for one or two more. Why don't we yeah, take one? Yeah, so you all said a version of the like platform and product is very important as well as FD motion need to have a balance of both. I'm curious like how does like the priority and one influence the other? I mean Ramp is a good example it's very much platform product first and FDs like to help adopt it, but for like nominal and data land like how much do you just try a bunch of FD cases and that informs the product versus you have like a very strong product platform vision and then the FD motion is more to like adopt it and like slightly shape it.

49:23 I think it goes back to my earlier answer about like expansion and contraction. So I think it is like actively changing right now. I think that we are finally at a scale where like nominal can build more of a platform and having like an FD motion is like a real test of that, right? Like are they finding it valuable? Are they building on top of it? Like in many ways they will be the first users of something that you claim to be a platform.

49:43 This is something that Palantir did a really good job of, but like the FDs would build on top of the platform first and then eventually it would graduate to be like the users, the customers would build on top of it directly. Um so I think if you're making a platform as a company like it is all the more obvious maybe to grow an FDE team alongside it. Calvin, I'm curious what that looks like for for you all. Yeah.

50:06 It's pretty much exactly what she said. Uh the platform comes first at Ramp. We know what it ought to do. It ought to like take a picture of your receipt and it it gets processed properly. Uh so we we know exactly what we're trying to achieve on the customer side. So the FDEs are more figuring out how to achieve that in the particular cases with the particular constraints of these enterprise customers as opposed to doing something radically new. Now that being said, there is some like cyclical flow that goes on where sometimes a bunch of customers will have the same blocker towards implementation of Ramp and then it's like, all right, that should really be in the platform. Like we're we're tired of dealing with this.

50:50 We're going to build it once and for all and that's that's where an FDE does get to act like a core engineer. Uh and similarly if there's something on the road map that just needs to be pulled forward, we'll often just pull it forward and and ship it ourselves in coordination with the core team there. Awesome. I think we are out of time here. So why don't we move to wrap but um maybe just quickly before we do, you all are hiring um how can people get in touch?

51:15 Calvin@ramp.com. Easy to remember. Jason@nominal.io. Uh howard@dataland.io. >> [laughter] >> Colin.jarvis@opennet.com. All right. So there you have it. You can reach out if you're interested but um yeah, I want to say thank you all for joining. Thank you to our panelists for for being here. Let's give them a round of applause. Thank you all. >> [applause]