The Quiet AI Revolution: Eddie Siegel, Fractional AI CTO on Solving Real Problems Behind the AI Hype

0:00 nothing is more important to the success of a startup than the first 10 Engineers you hire that's why I started first 10 hi I'm Boris Epstein CEO of first 10 and with me is my colleague Alexis merer hi at first 10 we work with early Founders to help build early engineering teams and this is the first 10 podcast where we bring you valuable insights on building early engineering teams from all sides of the equ Engineers Founders

0:32 even investors on today's episode of the first 10 podcast we welcome Eddie seagull the brilliant mind behind fractional AI wow how cool is it to talk to Eddie you he has such a unique vantage point and so immersed into what's actually being built in AI you know what really jumped out is the hype vers substance spectrum that he spoke about right like how much hype is AI

1:03 versus how much substance is actually being built and you know Eddie leading fractional which is on the inside of such a broad range of companies building AI I mean he's way more skewed to the substance scale which is exciting I mean there's some very exciting product and applications and yeah progress that's being made uh with AI and it's really cool to hear Eddie talk about that and validate that yeah I agree I think his team in particular is getting the chance

1:34 to see things from a a vantage point that most probably won't even inside of tech but I think you know we hear a lot about Ai and oh my gosh it's going to do all these things and I think you know right now you hear a lot about like the dark side of it but I think that there is a light side and I do think that you know that they've really identified this Niche sort of area or specialization

1:56 within AI where there's all these companies that have all these things that are super manual and time intensive and they're not in a position to do anything about it themselves so this is where a company like fractional can come in and really solve and work on some really interesting problems in that space and just make things much more efficient and rather than take somebody's job help them do it faster and better I think that's cool yeah plus

2:18 one I that I think of Eddie as a True Tech Visionary some who intricately understands the rhythm of building a thriving company from the ground up Eddie's not just a coding Myro he's a Serial entrepreneur who's navigated the startup world with Incredible success taking companies like wo from fledgling idea to successful acquisition now with fractional AI he's tackling one of the

2:49 biggest challenges companies face today harnessing the power of AI Edd's a leader who deeply understands the importance of a strong engineering culture and he's a mentor who Champions Talent I'm excited to Deep dive into Eddie's world and get Insider perspective on the future of AI and what it means for Founders and Engineers just starting out Eddie seagull welcome to the first 10 podcast thank you so much for having me great for you to be here yeah yeah thrilled to be here let's

3:15 let's start this off on a little bit of a of a weird note oh nice I like weird what's the weirdest question youve ever asked in llm and what you and what answer did you get well there's one circulating right now this one's gone like fairly viral I got did not come up with this myself but people are asking LMS for intuitive insights about themselves you know what have you learned by interacting with me oh I wish

3:38 I could actually remember the answer I found it insightful and immediately forgot about it but um I do think they have this sort of way or like on the on the topic of interviewing sometimes my my style is is to take these very sort of stream of Consciousness style like notes for myself and they're really like not ready to go share with other people the sort of Trends aren't thought out yeah and it's point I was trying to get

4:00 an llm to go through a series of my notes and produce some kinds of summaries that I could put into our internal tracking system and i' gone through a series of these and I got in tune to produce output I liked and then I just asked it to kind of summarize my style so that I could use that in a prompt to a future llm to be able to replicate it and it had this whole dissection of like all of my style all

4:20 the things about me that I found like were really like almost a little too insightful I had to kind of back off and be like uh you know I don't know if I want to know all this by myself that's fine wow like too much of a mirror for me exactly exactly that's funny I'm curious what is something the world doesn't yet know about AI but you think will soon so much real stuff is happening inside of businesses that

4:45 people might otherwise not think of as being super Cutting Edge or super sexy businesses that like is actually moving the needle I think there's a lot of conversation where people are still wondering is this real is it going to be useful is this hype and then if you go look on the insides of you know incumbents in basically any industry you can think of there there are massive needle moving projects happening right now way people don't don't fully

5:06 appreciate I think yeah that's really cool and because I think too like I don't know about you but I hear a lot of this like it's G to replace us and take all of our jobs and I'm but then there's that side of it that I wish people knew more about yeah you know I was concerned about that going into this business that you know implicit in the idea of sort of automating something is that you know

5:26 you might end up eliminating some jobs I think our track record so far is 100% of projects we've done are in production or or currently in development on the path of production there's a lot of these high values or workflow automations going on I don't think we've done a single one that has resulted in anybody's job getting eliminated instead it just looks a lot more like bring people up to do less grunt work and to do higher value things they move on to

5:50 the next the higher leverage thing to do or they're like I wish we could as a team get 10x more done than we're getting done today but we can't 10x the size this team there's like that's just like logistically possible and now they can get to an next time with the same team yeah that's awesome so I guess on the scale of hype or not how much hype is AI versus how far from hype is it I

6:09 mean in a lot of ways the the founding story of the company is like that we're generally Skeptics of these kinds of uh sort of like hype Cycles you know waves of enthusiasm in Silicon Valley not that there's never any real substance behind them but I think you know we have a way of sort of like you know the premise of venture capital is to imagine like if it works how big can it be it's not

6:30 necessarily machine designed for figuring out like How likely is it really to work and so like you know we were sort of like fairly crypto skeptical uh for for example this feels like the first first one of these waves I can remember in my career where seems very real it feels like living through you know the invention of the internet or or something like that where there's like very real change happening now of course like you know the do boom is also

6:54 a thing and like at any time when you have one of these significant shifts there is some level of hype to it and there's going to be some froy like you know use cases that are not grounded in in actual real you know fit with a real market and so you know of course there's going to be some silly things that we look back we like I can't believe we did that during this moment of crazy hype

7:13 but I think I think it's a very real transformational shift that's happening it definitely feels that way to us as well uh feels like kind of being back in mid 90s uh when the internet first came onto the onto the scene yeah I lived through some of that stuff as maybe a user but not as a a builder it feels very cool to be in the middle of actually helping build this stuff yeah it must be super fun from the lens of

7:34 you know getting the chance to build a cool startup so tell us your brief Summerview of your story right so graduated from upen and here you are today building fractional AI yes so I met my co-founders actually on the early team at live ramp so I I originally uh I joined live ramp as an intern when I was still in college uh then came back and joined full-time on the engineering team I'm a software engineer by background

7:53 I've in programming since I was 10 years old wow and joined live rep as a first as an engineering intern then as an engineer then eventually became VP of engineering of this this division that we kind of spun off and and sold and that's where I met who eventually would be my my co-founders in fractional so one co-founder is Chris Taylor our CEO he was the top performing salesperson at live Rapids that's how we how we met and

8:17 then Travis May our third co-founder who he also started an intern but eventually became CEO wow hard to get a better trajectory than that yeah and so we all had kind of intersecting and winding paths from there like I I after live ramp went started my first company Chris shortly thereafter joined and so we worked together on on what eventually became wo which we spent off and sold I think I think by this time this is like

8:40 Chris and I have now worked together like four or five times and it's been pretty continuous since I think like 20 13 or 2012 or something and uh Travis was CEO at live ramp then went on to found this company called datavant which he's since had this you know Mammoth uh you know home run sort of exit from and then it was earlier this year that we started kind of getting back together and talking to each other about

9:03 like you know is there some way to team up and do something else together and I think to your question before about you know hype versus not hype what we kind of realized through those conversations was like we all had the same feeling this is like wow I really believe in this one but then we started thinking about like so if there's going to be this massive kind of amount of value unlocked in the world what's the right

9:22 way for us to kind of participate in that and all three of us come from ventor back product backgrounds right with product people and started trying to think through you know these various product ideas for products we can build to help or to participate this whole movement and the answers were like really not obvious to us like everything we could kind of come up with had some scary combination of like you have to kind of bet on where the technology is

9:46 going you know you have to kind of bet on being a new entrant and somehow beating out incumbents which I think is very very challenging to do in the places where AI is going to win and I think we we really realized a huge pocket of where this value is actually going to be is going to be successful existing companies with strong Market positions that had things that were not automatable you know five years ago that

10:11 are now you know where they're doing lots of manual paperwork or people powered processes that are somehow limiting their growth limiting their ability to scale or they're costing them tons of money and that five years ago that was just the way it worked and then now that's not so true anymore but those projects now that they're sort of those things are automated that doesn't mean they're easy to automate especially right now when there's sort of no no

10:32 playbook for this thing and a lot of those companies are also the companies that don't have the kind of access they want to have to create talent and you know if it takes a really really high caliber engineer to do this you know where where they can sort to get access to that and so that that's how fraction was born and I do believe that like you know let's take for an example like um somewhere in the healthcare World your

10:51 uh revenue cycle management company and your your job is to like appeal Deni insurance claims right so five years ago what does that look like it's a bunch of people writing letters appeals letters to insurance companies actually probably still today that's how it works and you can imagine how AI is gonna make that so much better right like that that whole industry is going to change like AI could be Drafting and writing these letters for you at a bare minimum uh if

11:12 not like maybe this whole process should change right the insurance companies are also going to employ this technology in some way but just because it's going to change a doesn't make it easy and B like who would you rather be in that scenario do you want to be the new YC company coming around being like I'm going to change the world of revenue cycle management or do you want to be the incum that has you know a decade of

11:31 experience and has hospitals as customers and knows how to appeal these these denials and lives in the flow of data you're already integrated and have all the these historical uh you know medical records and things like that I definitely want to be that incumbent I think that's part of why it's really challenging to build an enduring large scale Product Company in this space or at least predict where there's going to be Product Company winners interesting yeah yeah that is yeah super cool and I

11:55 I in talking to other Engineers about fractional and that's kind of something that I I highlight to them I'd love to hear from you though like if you were going to talk to an engineer about like why coming to do AI here at fractional is going to teach you like more than going somewhere else like what what would be that answer like what would you what would you tell them and we see so many very cool real

12:19 interesting valuable projects that AI is at the center of here in a way that very few people do like you get to come here and actually build something valuable in the world that you know ahead of time you're not in this product Market fit seeking exercise of like if I build this does anybody want it you're like a company wants to pay us to automate this painful process for them today but it's really really hard and oftentimes to go

12:41 get those projects first of all you'd usually get to only work on one of them right if you're going to go work for a company that has that problem you'd be working on that problem for the next five years you wouldn't be like get to sample multiple of them but two you usually have to go to a company you don't want to work at like do I want to build the Next Generation AI customer support phone agent thing for United

13:00 Airlines yes do I want to go work at United Airlines definitely and so being able to work on like their coolest hardest problem from a startup environment surrounded by other brilliant people and get to do sort of like a survey of those things over your career I think is really powerful thing and we also like you don't have to already be an expert like most the people that get to work on this stuff you need to have some existing expertise

13:22 this is the place where we do expect you to be a super strong engineer um you know the people on our team are all like ex Founders ex engineering leaders uh people with 10 to 10 to 25 years of experience that they've done really incredible things but most of them came in with no AI experience and they're sort of learning on the job and so it's it's a really powerful way to kind of learn the kinds of Applied AI skills

13:44 that can then be valuable anywhere else yeah that's so cool that's a great opportunity i' I have so many questions about fractional and your world through the lens of Engineers but before we jump into some of those fractional obviously is in your first rodeo right you talked about having built numerous previous startups what keeps you coming back and what what are you kind of wanting to accomplish with fractional it's different than what you've you accomplished before um well one is I

14:07 think there's like an opportunity to build like a hundred billion dollar plus company here and so that's that's definitely part of it is want to build some really massive enduring Unforgettable thing and I don't think I've done I you know I've been in startup land for a while I've had varying degrees of success I haven't had any success at that magnitude and so definitely want to want to take a run at that another is that like I've always

14:29 been very motivated by building the kinds of teams that I want to work on like I really love working at places where I'm like surrounded by people that are brilliant that are impressive that are deeply technical that I get along with they're also nice you know sort of low ego yeah and was fortunate enough to actually have experienced that at live ramp and that sort of taught me how to value that but I think apart from live

14:52 ramp like I found that really hard to find again like a lot of places don't don't have that and trying to build it myself I found very fulfilling an interesting thing about this time around though is that like in the past it's always just been like my own motivation that drove that like I do think that W had a really talented team and we built a talented team because I thought it was important to build a talented team for a

15:10 lot of reasons this is the first time where like actually the business like really really needs that like this this company won't work if we don't build a really talented team and that has been really fun for me like there's like an actual feedback mechanism for that now instead of just my force of will yeah and uh and that's been really nice and I think that also shows in the team like where everybody is working around the

15:31 clock to build the incredible team because we know how important it is to success of the company and that's actually bearing fruit and you can see it in the caliber team yeah so I'm curious the types of companies that you've worked on before obviously so different than what fractional is I think a lot of it's it's just different in the market and I think that's so cool and refreshing what experiences from like those early days at live ramp have

15:53 helped you as you're kind of building the team over at fractional now like what are some of the things let's I I guess that you're drawing on to do that now I think you have to have worked with a really Stellar team in order to like one of the hardest parts about building a stellar team is like keeping your bar while you're going through the process of building it like it's very easy to sit there in a vacuum and be like it's

16:15 important to like build a great place full of great people and then it's very different when you're like interviewing someone that seems pretty good yeah or like you're unsure or you're going through like a interviews are so imperfect right you're like reading tea leaves the whole time and when you're in that process especially when you have aggressive hiring targets and it's the bottleneck for the growth of your business I mean here like the thing that's preventing the business from

16:37 being larger is our highering the side of things right like we have more business than you know the current team can deal with and so of course it' be like very easy and natural for me to be like that person seems good enough I can find a place for them to add value here yeah and I think without having worked on an amazing team with discipline around how to sort of like keep the bar held high it's really hard to know how

16:59 to how to do that in some way like the hardest part of hiring is like it's actually that you know it when you see it and then when you're not seeing it reminding yourself that you're not seeing it it's so hard to do it must be so easy especially being in the business of the more Engineers equals more business right I mean there's a very direct correlation to that right that must be very hard thing to do to

17:18 maintain that bar yeah I mean long term we won't succeed if we like lower the bar right but short term we certainly will like short term if you know we're like oh we could like literally just make money now by saying yes to hiring this person and so yeah you have to stay very in tune to the long term and trying to build build the right thing over time luckily I have very supportive co-founders and partners around me that

17:40 that understand this because sometimes I do feel like you know they have to sort of rely on my technical lenss that I'm looking at people I'm like I think this person does or doesn't have it and they're like all right we'll take your word for it because they're if they're not technical themselves how are they going to see that and so I'm very fortunate to have people around me that also get it and are long-term thinkers

17:58 and have the trusted me to to believe me when I say it and I'm sure everyone will appreciate that dividend right as it you know as it as it plays through oh yeah like it's also yeah one of the biggest sort of ways to get this to work is like now now the team is small but like the caliber sort undeniable and so when you see it actually like working out in practice that's definitely the ammo you

18:18 need I think this would be a very different story if I was like no no no trust me like we we should hold this really high bar then the team around me kind of suck I don't think I would get a lot of trust then so yeah so there's a lot of kind of mission alignment there right our the name of our company is called first 10 because we believe that there's nothing more important to the

18:33 success of a startup than the first 10 Engineers that they hire so tell us about the quality of your team what is this kind of bar that you hold new engineering talent to tell us about that yeah yeah so the folks we're trying to hire I mean really what we're trying to figure out is like if we visualize them being on one of these projects that we do you know if you if you're asked to go

18:53 own end to end this ambiguous challenging kind of you know project that we from our customers like do we think it's going to result in success or not um and that's like a many faceted thing that we're looking at and it's not it's not just one thing especially when you take into consideration the fact that like the best fits are actually going to be sort of lopsided in some way right they're going to have like spiky

19:12 strengths where they're you know a 10 out of 10 on something and people that are nobody's 10 out of 10 on everything and so you're you're trying to figure out like does this configuration of this person's sort of spikes and sort of like you know less strong areas do they do they result in something that I think is going to be success and so we try and design our interview process around just like simulating the job basically you

19:30 know if we just sit down and we build something from scratch together that stretches your understanding of a problem that's kind of hard to think about what do you do like can can you actually like deliver something especially if you have not a ton of guidance you know you're given sort of customer level or product level understanding of what we're trying to accomplish but no one's sitting there telling you like in great detail like here are the exact specs for this thing

19:52 you know can you can you thrive in that environment and uh I most people cannot and so we're trying to kind of observe observe now one of the things that makes that extra tricky is that like a thing that we can't really do much about is that we're also doing it an interview setting yeah and so like you know they're under pressure and we know that and so we're trying to like sort of like give them leeway and be like okay what

20:13 what do we think would happen if they're actually not in an interview setting and on the job there's some things we can do to control for that like we try and let them use their own environment they show up with their own tools their own IDE we let them you know use AI tools and do Google searches and we try and prepare them ahead of time for you know this is not it's not some game with some rubric

20:29 this is like us sitting down trying to write a program together but ultimately they're in an interview setting that makes it very hard to read so that's the sort of high level how I'm thinking about interviews when we're going through it and then at the end I'm like did this person impress me in some way can I picture sitting next to this person every day and then being like wow I'm really proud to going to this office

20:48 and that's like a pretty high bar that means people that are otherwise good might not hit the bar on on that Dimension yeah and I I recognize it's fuzzy uh for me to say like say that and I think it it is fuzzy but we're actually flexible about what that looks like like there's many different ways to be impressive there's in fact I think no two people on our team are like impressive in the same way but every one

21:10 of them I'm like very proud to work with and and I think that showed in some way on the interview process somehow i' I'd love to know a little bit more about the culture at at fractional and kind of you know what's it like there I always struggle with this one because I feel like culture is like how do you define that you feel culture yeah yeah so I think think a Common Thread throughout this business is like we're very

21:31 impacted by the fact that we do we do Project work we're a Professional Services business right we're building custom to spoke software for our customers and every customer is a little different every prodject is a little different and so we end up being pretty uh customer Centric in that we sort of like spontaneously organize around whatever project needs at any given time so like on average our teams are like two engineers and a product person with

21:56 some support from random folks around the org and we're like laser focused on whatever individual project we working on any any given time and so as an engineer you're only ever working on one thing at at the any given time I'm I'm spread across projects there are people on the company spread across projects but they're Engineers are one on one thing at once and so their life tends to be organized around like what client am I serving right now what do they need

22:18 Etc and we tend to even though we're building you know pretty complicated engineering projects and we're thinking a lot about AI all day we're also we take a pretty customer first view of things like like I think I've been at some companies before where like the technical details are what rule here I think what the customer needs is what rules and then we also go down these like sort of nerdy side paths where we kind of geek out about the technical

22:39 stuff because we you know we're using AI this is like never ending nerd snipe territory we could talk about this stuff all day long there a lot of AI talk around the office yeah we do these like weekly project lunches where the idea that is supposed to be like you pick a project a week and then someone goes and deep dives into what their what's going on in their project and they share with the rest of team and then by the end

22:57 it's all us just sort of like nerding out about AI in some way talking about AI philosophy or try some new tool we tried out and a lot of that as well that's fun as you're describing I could I could picture it and I I would imagine that you know what one thing I'm curious about is you talk about being open to Bringing people in that don't necessarily have ai experience How does it go from like day one with no AI

23:19 experience to like what you just described which is like immersed right that's a good question this definitely an area where we've evolved over time like I think the initial kind of set of first principles here was like doesn't make sense to go hire someone with 10 years of gen experience for 10 gen is not 10 years old not not in the form we are thinking of what we talk about it and when we encounter these projects

23:39 we're generally the first people to have thought about what we're doing or close to it and so the the skill set here is much more like how do you operate in ambiguity how do you have a generalist skill set how do you have a broad array of experiences to pattern match a broad set of stuff too I still stand by all that but then when we like started at the very beginning we just like plunge head first to these projects and like

23:58 people were not like equipped the learning curve is steep especially when you're learning to also help like sort of manage a you know thinking about what clients need that's also a separate sort of thing you're learning at the same time so we've evolved a ton on that front so first of all we like have a series of orientations that are like designed to kind of give you the sort of quick boot camp on like the stuff that

24:18 you can learn quickly okay but then what we like to do is when you first join the company we have you do like an apprenticeship project where we put you we pair you with somebody that is already running a project that's well underway and then we don't build a client for your time usually okay and so what this results in is like the great win-win you like are not owning a full thing end to end you I mean you're still

24:39 trying to figure out how to deliver for a client but it's not with the pressure of like great you own this thing you don't understand like I'll come back later and we'll see if it's great yeah instead you get some some set of training wheels you get to sit next to somebody that knows this really well and you get to learn the best practices from them and then the client views that it's just like overd delivering we just gave

24:54 them extra free resources we're not charging for and it lets us tackle the long tale of stuff that we normally wouldn't have gotten to and the project loves it the client loves it it's it can really set you up for success and then you feel really good about like transitioning into actually owning a project from the start after that so that person must feel like a total kid and candy store right just being fully immersed into this world right we also

25:15 invest a lot in making sure there's a lot of communication between projects like part of my job is to see across all these projects and to make sure I'm like like all day I'm like oh cool you're encountering this thing someone else encounters something like that you should make sure you talk to them about this or yeah here I've seen a similar thing on another project and we do these I mentioned the project lals internally we've sort of evolved that practice over

25:35 time and do more and more of them goal being like yeah you need to it's hard to be spread between more than one project and be able to give it all your attention so we do want you to like be on one project at once but then once you're on that project tunnel vision is like the last thing we want and so we gotta facilitate all the sort of cross pollination between things so that you know you can be benefiting from

25:54 everybody else's experience in the team we also have this just company value using AI like our value comes from being on the Forefront of using new tools and having seen projects many times and be able to pattern match between them and so we're like very willing to invest in like buying new tools and giving you time to like get up to speed on them and try them out and you should be doing that enough to the point where like

26:15 sometimes it was a waste of time and that's fine and like the answer was that was the wrong tool for the job it's there's a lot of kind of like room for experimentation playing around that I think people find uh fun and interesting and help support them in their Project work on so something I'm curious about that I think the world is similarly CU about you guys have such an immersive view of what's going on with AI across

26:38 such a broad range of companies right from startups to tech companies to bigger tech companies to non-tech companies right what is going on with AI at these companies what are some misconceptions are is it not on their radar are they dabbling are they like all in like what could we expect to see over the next 12 months yeah I think like earlier this year there was a lot of like people that are like we should demonstrate that we are doing something

27:02 Ai and without a ton of substance behind it sometimes and then now we feel like we're in a different phase like we see substance all day long it's like real companies of all types are figuring out how to get leverage from AI in their business that looks a few different ways but the kinds of things we like to see like the way we think about the core sort of new functionality it's available is mostly that computers can now read

27:28 and make like simple basic decisions across a crazy amount of domains and that unlocks a lot of a lot of things the kinds of things that unlocks is usually like freeing up people from doing those things so places where people are doing basic Reading Writing wrote decisions are now open for technology the way they weren't before and sometimes that looks like grunt work that's happening at companies they don't want to do anymore you know pushing around paperwork and things like that

27:55 sometimes it looks like entirely new workflows that weren't possible before cuz you couldn't scale out large enough to do it other times it looks like new product features that are framing their customers up to do that sort of thing so like you know we did a project recently that was like automating the building of API Integrations and that that looks like a product feature like they're releasing it to their customers but that's because their customers were

28:14 spending lots of time building new API Integrations and it was like sort of a manual process they'd have to Loop engineers in to do it Engineers didn't like doing it now they don't really have to sometimes it looks more like that revenue cycle example or like we did a Content moderation project recently where like there was a team of people that found themselves in this position where they're like they're working for this social media platform and they've

28:32 got to go in and review a bunch of content by hand every day and determine like does this violate our terms does not violate our terms that's a pretty natural language is kind of thing that was like not really available to computers uh you know five years ago at least would have been much much much harder probably not worth the effort and now that unlocks this whole automation opportunity I think there's little pockets that all over the place and one

28:56 thing I've learned in this businesses that you can't always predict ahead of time what what those use cases look like like you know people ask us early on like do you have specific verticals you know you're you're focused on and nowhere on my bingo card was like 150 year old farm equipment manufacturer trying to do this like you know information extraction process over old like oil stained manuals that are on their Factory floor like I didn't know

29:23 that was a thing like I guess I know it's a thing now and it's a lot of that yeah I wouldn't thought that either to be yeah that's awesome out of curiosity like how long like what's the average length for time to like develop a project like end to end like months year shortest one is probably like eight weeks maybe six weeks the longer ones are years like we're we're Midstream on on a multi-year long thing right now or

29:50 year plus long thing right now yeah they really run the gamut also like the longer it gets the more the sort of lines blur between what a project is like I think a good good way to tackle an AI project with a big vision is often to find some small start and sort of prove that it's doable and expand from there you don't start by trying to like do every single American Airlines customer service request you go what is

30:12 the highest volume of requests to require the most like sort of rote effort you use that to sort of prove out a use case it's very tricky to get these things working in a real production environment right it's not just getting a voice bot that can say the right thing sometimes it's got toally say the right thing every time even against an adversarial User it's got to know when escalate to a human it's got to work a

30:29 crazy scale the the amount of edge cases is really large and so starting with something small and being able to sort of like incrementally chip away at the broader problem you know how where would you draw a line around what a project is there so oftentimes the projects we're most interested in are ones that have like transformative potential where like if we succeed you can change the economics of a business at some core level and those projects I think take

30:50 time and they expand over time yeah so I don't know something that you just said I was thinking about so I read in one of the logs that's actually out on fractional AI you talk about focusing on on picking the right model when thinking about AI projects is is a is a mistake I'm curious what makes you what makes you say that and and what should startup Founders or or anybody that's like focused on AI like what should they

31:15 focus on instead yeah so I think model choice is definitely a thing I think it's just happens to be an area where there's a lot of preconceived notions and a lot of sort of misconceptions that make it so that it's harder for someone to start on a project yeah at some base level an AI automation project is just an engineering project yeah you do need to make a decision about what programming language to use it's not the

31:35 most Salient thing about the project and if you design your project the way we recommend which is very first thing you do in the project is you try and get to a place where you have end to end evals in place and what I mean by evals is essentially like an automated test Suite that's measuring how well you're doing at any given time like you need that for all sorts of reasons you need that to be

31:54 able to tell you how well you're doing you need that to be able to sort of guide the development process you know like it'll it will surface anecdotes where your system is performing poorly you can go like oh hey I'm like working on this content moderation project like I got the answers wrong 30% of the time what are the ones I'm getting wrong okay it's getting this type of stuff wrong let me go adjust my system to get better

32:11 at these sorts of things if you build that pipeline out that way and you have evals designed it becomes like fairly easy to test what happens if I change to a different model there's some Nuance there you may like adjust your prompts to adapt to what model you're using you may change your architecture slightly based on the model different models are they're good at different things and they're sort of addressed in different ways sometimes should become fairly

32:33 straightforward to like somewhere along the way you test a bunch of different models and you see which one performs well and like it's very common for us to have a solution where we start off just using open AI models because it's kind of easy and then we like test anthropic models in multiple steps and then like one of the steps of our pipeline uses anthropic for one thing because it seems to perform better there in practice we

32:52 rarely end up using open source models because they I think typically perform a little worse if not maybe the and then you got to worry about deployment and where do you go Host this thing and manage this thing and like there's a whole can of worms you open which are not spending your core time thinking about like how do I actually build a reliable system here how do I get done what I want to get done how do I coax

33:09 this model into producing the right output and so those are much much harder things yeah and I think people start off like sort of misconceptions when someone says like I want to build a system based on this model my first question is why yeah think the why is often like I'm not really sure like Sally boils down to like you know sort of read good thing or a lot of people have beliefs around I have extreme data security requirements

33:34 I have to self-host some model and uh you know I can't let any data be used to train the model they don't realize it like open AI out of the box actually gives you the promise they're not training on your data yeah or like if you really have this requirement you can go like access gla's model through bedrock in your AWS account and it's never leaving your AWS account so the like sort of core reasons not always but

33:53 often they don't hold up when you kind of go through this process and instead like you should be spending like 90 something per of your effort thinking about things that are not about model selection interesting that's really interesting you know you made me think of something as you're talking about models so models are advancing at such a rapid Pace right given the resources that are invested in them what will existing models be able to do you know

34:15 that startups currently do that you know will likely not exist like what AI startups Will Survive and what models will kind of just blow through yeah um it's very hard for me to predict who is going to win and who's going to lose in in startup land I think when you're looking at like there's a sort of Separation here between like tools that are used to build AI Solutions and AI Solutions themselves right so AI Solutions are like something

34:36 verticalized like I want to build the AI powered revenue cycle management company the other category is like I want to build open AI or I want to build Bland they make like a AI phone agent sort of layer that could be used to build other AI phone agents top of them so it's like the sort of Dev tools is pieces of the stack and then there's like sort of verticalized Solutions on the verticalized solutions I find it

34:56 impossible to pick winners like you pick any given industry and it's hard to tell like are the incumbents going to move fast enough to actually be able to win here and do they have a strong enough advantage or you know maybe they won't move fast enough but maybe private Equity will buy them and then private Equity will win or maybe like even that will fail and then like the new start of tion will figure out how to CH away

35:13 their market share I think like it's really hard to tell on the tooling side I think the tooling is just like getting better and better and it enables us to move faster and faster ultimately like just because you have good AI phone agent software does not mean that like you're TurnKey for United Airlines I feel like maybe every time I've given this example I picked a different Airline but whatever uh getting to something that works end to endend for United in

35:37 production is a lot more than picking an off-the-shelf tool but the off-the-shelf tools are getting better and they're important and I think they're going to definitely take a slice of the sort of uh money that's being spent here but beyond that it's very hard for me to actually pick winners and losers here I'd be wary of being an AI company selling AI stuff to other AI companies right now you know we try and limit ourselves like you know of course like

36:00 in this kind of uh I environment like there's going to be certain companies that are like they have budget to demonstrate that they're doing something Ai and like if you really get you know critical under the hood of like is there true user value at the end of this whole chain there's a question mark attached to it we really try and limit the amount of that business we do I mean we we'll do some of it because it's natural it

36:19 helps us sort of grow but the use cases we really like or like company's been doing something for 50 years and now they don't have to do anymore which feels like it has much more enduring value if you are an AI company whose customers are other AI companies you have to assume that over time like some degree of error is going to get let out of the bubble and I would worry about I would worry about that yeah yeah that sucks so

36:39 another reference to the blog seriously anybody listening to this they should go read the blog it's really cool but you you discussed how llms are changing the hiring landscape and curious from my own perspective like as AI tools become more powerful what skills do you think will become even more crucial for engineers to develop up especially in context of fractional too if they wanted to come work for fractional I'd love to hear more about that yeah well so I think a

37:04 thing that's on the top of a lot of people's minds is like where do we put AI into the interview process or not you know we had to think pretty deeply about that when we started the company because on the one hand like you do want to be testing people s raw engineering skills and capabilities but on the other hand like the nature of what it means to be a software engineer these days it's like you get a lot of Leverage out of using

37:22 AI tools and I would view it as pretty problematic if someone joined here and then like you know couldn't get any leverage out of using and you also want the interview process to feel kind of natural and like you want the the candidate to feel comfortable nothing worse and being like you know yelled out because the code you wrote on a whiteboard doesn't compile and this feels like the same thing it's like I'm being asked to like program in this

37:41 constrained environment and like normally what I do is I ask AI yeah the flip side of that is that interview questions are the kinds of questions that llms are disproportionately good at at solving yeah in my day-to-day job when I'm programming I find co-pilot very helpful find ason chat GPT like sometimes helpful but like I probably throw out you know 75% of what it does because it can't actually grock my full big problem or isn't actually good at

38:05 solving the really hard Parts in an interview setting the nature of the Beast is that you're being given a problem that can be described to you in the time window of an interview yeah it's self-contained in some way and that self-contained problem the LMS are disproportionately good at solving and so what we do is we tell candidates like feel free to use AI tools here we ask them to turn off the proactive Auto complete style of AI so like co-pilot or

38:29 things like that but they're allowed to go prompt the AI themselves and the reason for that is like we want to be able to see when you're making an active decision versus when you're making a passive one we're not trying to like sort of handcuff you but like you know we are sort of wondering like oh this thing just wrote all the code for you without you asking for it did you even think about it like did you even think

38:45 about that or did the computer just know it ahead of time yeah so that that is sort of where we decided to draw the line but I think others draw it in other places the flip side of that is that candidates have a pretty varied set of experiences in terms of like how effectively they were able to use these tools and a lot of that boils down to where they worked last right if they just didn't have a job and they were

39:03 like working on side projects at home or something probably using AI for stuff at least I hope so I would suggest if you're out there in that in that situation that you try using AI you know if they just worked at a startup they probably are using AI like I don't know if you worked at like some big Healthcare company that like didn't you know had a policy of like no AI tools or you worked at Google and like you're

39:22 only allowed to use Google's AI tools and like you have this very sort of biased view of like what the tool set is that's out there it's not really your fault and like I would what I'd want to be testing in that case is like how adaptable are you and like do I believe that if you joined here pretty quickly You' learn new tools so I think it presents sort of challenges and opportunities from both the candidate's

39:39 perspective and the interviewer's perspective but it does give certain people superpowers another area where we try and like at least give some thought is like if a candidate worked at an AI company before okay on the one hand that's like demonstrates AI enthusiasm at some way which is nice the other hand they have like a up that probably will like diminish within their first you know few weeks to a month on the job and so like don't want to over index on like

40:03 the experience they had at other AI company yeah and so try and hold them to a slightly higher bar in terms of like you know how familiar we expect them to be with certain tools but uh yeah in our interviews we generally encourage people to like even when they're encountering AI specific questions we're asking we encourage them to just think about it like an engineering problem don't think about it as some some magic AI quiz that

40:22 you know you haven't studied for yeah yeah so continuing along that that thought so we speak with so many different Engineers some that are you know immersed in AI some that you know are on the other end of the spectrum where they're like you know I've heard of this thing but maybe I've asked chat gbt a couple questions right so what advice would you give to the average engineer given your immersive kind of nature in Ai and where you kind of see

40:44 AI going I mean if you want to develop these skills I think my advice is just force yourself to reach for it more and then the rest will happen pretty naturally like you have to build the muscle of like I you need to know to reach for it an easy place to start is like see how little you can Google search like try and make your new default Google to be open Ai and be chat BT another is like turn on all the AI

41:05 features in your IDE uh you know get co-pilot the jet brains one is like not so great but like you could use a cursor or one of the Alternatives and just turn it all on and try and use it and try and challenge yourself to use it more and more those those would be the the the quickest ways I think to like sort of start to build the muscle to start reaching for it more and more and more

41:24 the more you do that I think you'll like eventually bump up against like wow really wish I could use XYZ and then you'll know to sort of Google search for the next pool that could potentially like help you with that thing your recommendation if you're if you're not into it get into it do it um okay yeah I mean it feels to me right now as a developer like if you have the option to use co-pilot and you're not or you have

41:44 the option to use something like cursor or similar and you're not you're like effectively opting to use like it's like choosing to use like Vim with no syntax highlighting and like no plugins nothing is turned on and then like wondering why you're like half as productive whereas like you know there's ways to make them productive not to this is not to trap talk them but like I would not program right now in you know Microsoft Notepad

42:06 I would use some like tooling to make myself faster it it it feels like we're pretty much there with something like co-pilot right cool any anything that we missed asking you anything that you kind of really kind of wish you had the opportunity to to chat about but didn't didn't have that chance nothing really jumps to mind I think we've like covered a lot of different things oh yeah all kinds of topics that's fun yeah I think

42:26 so too this spel pretty pretty productive I felt like we could have you know definitely gone down some more rabbit Hills but it was really cool to hear your view of the world given kind of given what you have seen and see as it relates to AI yeah I mean I feel very fortunate to be able to like actually work on these kinds of problems every day I think there's very few Pockets where you can work on kind of trendy new

42:48 interesting things that are also like deeply valuable in a way that you can understand is valuable yeah like it's a lot less speculative whether this is valuable you we have folks on our team that like very very experienced very seasoned and I think one of the things that appeals to them is they're just like I've done the thing a few times of like you know going to an early stage company and like wondering is there ever

43:05 going to be a there there and can we find our way to building something that people want and there's something nice about being a little more baked into the premise like working on this because someone has expressed that it is valuable and like it's still hard there's still risk here you know we still an early stage startup and I want to declare victory at this stage but uh the kind of like the premise involving that someone already kind of has raised

43:26 their hand that said like this is a thing that actually is valuable to us you can sort of see where that value comes from it's an interesting set of ingredients to mix with like Ai and being on the Forefront being able to play with new things like those two things existing in the same spot are pretty remarkable I feel pretty fortunate to be able to come in and work on them every day ah absolutely that's awesome I love that so Eddie where where

43:45 can people reach you uh who do you want to hear from what should their first message to you be well I mean the first and foremost the most important thing to us is hiring other great team members so if if you are someone you know is a an engineer that's in working on these kinds of things please reach out to me uh you can come to our website fraction. would absolutely love to hear from you in fact that's the only thing

44:05 we're going to plug that's that's what I want I want to meet other smart Engineers that are interested in working on real valuable AI projects and I want to want to talk to you awesome love that love that well thank you again Eddie for joining us uh we really certainly uh you know appreciated you know getting the glimpse of the world from your from your lens and I'm sure our wonderful world of Engineers and founders and investors who

44:27 listened we're appreciating that as well so thanks again and keep building what you're building we're excited to cheer from the sidelines and see how this whole wonderful world of AI turns out awesome thank you so much for having me yeah thanks Eddie and that brings us to the end of this episode do let us know what you thought of our episode on our Twitter or LinkedIn look for the links in the show notes both Alexis and I are not

44:49 professional podcasters as I'm sure you could tell we just love talking about early stage startups and if there's a recruiting related a question you'd like to answered or you could use help with anything hiring related jump on our website it's ww. 10.com that's 1st.com and we'll do our best to get that answered for you thank you so much for staying with us till the end my name is Boris and with me is Alexis monger and we'll see you next time on the

45:15 firsthand podcast bye