Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI

0:12 >> I'm Scott Worley. I'm the CTO of Norm AI and I'm privileged enough to be sharing the stage here with John Stecher who's the CTO of Blackstone. From what I understand, John has basically three full-time jobs. It sounds like you've responsible for all of the technology at Blackstone, also advises on their investments and is as well as serves as a resource for the portfolio companies for all things technology. Did I miss anything? >> Nope, all perfectly mapped.

0:39 >> And so cool job. Yeah, and if you're not familiar with what Blackstone does, then you probably have wandered into the wrong conference. so I'm not going to go into that, but you may not know what Norm does. So really briefly, Norm's an AI company and we operate in the legal and regulatory space. We have an enterprise AI platform that's deployed in some of the largest financial institutions in the world and then we also have an affiliated law firm, Norm Law, and our AI platform also supports that.

1:05 so since the end of last year and certainly all of this year, everyone's been racing to figure out how to deploy AI agents. Now I'd love to hear how you're thinking about this at Blackstone and how you're trying to make sure you get this right. >> Yeah, I mean I think the the race has definitely been on as like the technology has improved mightily since you know really like February, March of last year. Like just in terms of context windows being able to run stuff for significantly longer amounts of time have like resulted in people being able to produce just more value.

1:32 And one of the things that like we've been focused on a lot and I think every company in our portfolio advise on and work with all the time that they need to have top of mind for like being successful with AI agents in general is you have to get your data story right. And so like one of the biggest things, impediments, and tripping points that I see for companies out there when they want they're all in on AI, they want to build as many agents as possible, they really want to kind of extract value. If they don't have a cohesive data strategy that's moving stuff out of transactional systems into something that can be accessed and easily managed and recalled from like an agent perspective, it is going to fail. And so that's been one of the biggest things that we've been focused on you know as a big asset manager we have a ton of data proprietary data public data all woven together. We have to move that into out of systems into something that we can really leverage from an AI perspective.

2:21 I think the second thing that you know has nothing to do with AI but you can leverage AI to actually figure out how to do the best possible way is straight through processing still matters. I'm always shocked at how everybody just talks about like agents as being the biggest differentiator in the world. The thing that I've leveraged myself using factory and a couple other ways to go off and build code inside of the firm is like taking time having agents help to craft something for me understand the workflows out there that people are doing and then make sure that I'm like actually writing code to optimize for straight through processing and then using agents to inject an intelligence where necessary where humans were making decisions in the past for a variety of wrote work things. So that's really like the two key elements that I I spend a ton of time thinking about every single day. And then the biggest thing from an agent perspective is really stepping back and looking at like what is work that we want to build agents to do better to offload like intellectual like cycles from our investment professionals our finance professionals our engineers such that they can do higher value work. And there's a ton of stuff when you think about in the software development life cycle writing test cases functional test cases performance test cases version versioning forward libraries underneath applications. It's the bane of every engineer's existence. I've hated it for 27 years that I've been doing this.

3:37 Having an agent be able to go off and do these things for me and renovate my code and my source code has made all the difference in the world for the velocity that we deliver at. So how are you thinking about it at Norm? >> Yeah so Norm was founded in 2023 so you know that's 3 year I guess 4 years after GPT-1 but still a year before there was like you know chain of thought. So it was like an interesting time in terms of development. I remember when we had our first team get together we were at Stanford's campus for a week and I mean, I'm probably misremembering the exact timeline, but it felt like we went from 4K to 8K to 128K context windows and then we had multimodal and everything just felt like it was kind of coming all at once. And then there was a bit of a time where things did slow down when it was coming from the frontier. And so I think like a lot of companies in like early 2024, you know, we were trying to deploy these models as part of like our workflow and as part of like our production software.

4:26 And so we ended up adding a lot of structure to the system and and and this comes from you know, all different parts of the stack. So whether that was from like how we went and took the content that we knew needed to be like eventually added to context to the models and we had like very specific ways in which we did that or if there was like you know, just the sequencing of the models, if there was like steps we needed to do in terms of looking for consistency and then ultimately like distilling everything down into some useful output. We ended up with like something looked like a large like map reduce problem effectively, right?

4:56 Whereas like we had a lot of scaffolding for the agents and we were very particular about like how these well, I would say it's really just how these models ran. It was much more like a workflow. And what this allowed us to do was get like really, you know, high quality and also with accurate very accurate, but also just very consistent, which is very important for us in the regulatory space outputs. but then one fortuitous thing as part of doing this cuz we wanted to have a some way to regression test this big system.

5:22 We ended up with a really large set of evaluation samples. We had tens of thousands actually of like hand labeled examples by experts. And so as soon as we moved to a place where it was like clear that agents were actually a thing and by agents I simply mean you know, given like an LLM a system or given it a set of tools that it can use, give it some goal and give it some way to verify itself against that goal, allowing it just to go figure out how to do the thing in the best way possible.

5:47 We were able to just take this benchmark set and just immediately run on top of it with some of these more agentic approaches and we saw, you know, I think a lot of people had this moment where you're staring at hundreds of thousands of lines of code your team had really thoughtfully written and realizing most of it was pretty obsolete and was just kind of actually in in And so to update quickly on this, what we did was we we took I got the engineering leadership team. We actually rented a a Sprinter van, and we drove to Upstate New York. It was actually after a big snowstorm. We had an Airbnb, and it was like, "Forget norm for a minute. Let's just say what would we build today as far as a platform, knowing what we know right now if we were going to be starting over?" And then from that, we left with basically a set of specs about like what this platform needed to be able to do to enable two things. One, deliver agentic experiences to our clients. And then also it needed to be amenable to having AIs like co-developers alongside of our own development team. And that's what we've been executing against ever since then. So, I think this Do you have a question?

6:44 >> No, no. I was going to say, I think you hit on like a key point, Scott, which is like the eval side of the fence that you were talking about before. If you're working in financial services, like the the quality bar for what agents have to deliver on is very, very high. And so, I think really thinking through like what is your eval set for how you basically test your agents, what they're actually, you know, whatever work they're performing, that the output makes sense, and it's consistent and repeatable. And I've seen across the board as model versions change underneath the covers, you get different answers. You get skew in terms of the accuracy of information that comes back. So, having good evals so that you can run those consistently and really measure the health of a system is paramount to success. Same thing happens if harnesses change that you're running stuff on top of, you'll get different answers. Data changes, different answers. So, it's it's something where I think, you know, if you're trafficking in financial services, like the duty to be accurate is like >> >> up here, and hitting that is like something like we think about all the time.

7:40 >> Definitely. I think this leads nicely into this concept that we've been talking about just like with the bitter lesson. So, like everyone's familiar with it, and you know, this is this idea that these hand-crafted systems built by, you know, clever people thinking hard about the problem will often times eventually be dominated by systems that scale with compute. Right? And that this is actually I think Rich Sutton originally talked about this in 2019, but this has been since the beginning of neural networks. If you go back to like the '50s and '60s like with Rosenblatt's little perceptron, I think the New York Times ran an article or somebody did saying it was the first first machine that could ever have its own original thoughts, you know? And so and there was actually it's there's actually quite a fascinating history here of how the you know, expert systems and the connectionists and what ended up happening a lot of it actually was highly litigated in in England and a lot of AI research was dismantled, but one funny thing if you go back to the 80s even, there was a lead at IBM's translation lab and he made a comment, which is very much the bitter lesson, every time I fire a linguist my models improve, right? No, the point of having these like linguists in there and trying to think about all the rules of language versus like training a statistical model based on just probabilities. And so we speed up to today and like we see this all the time. We've been touching on lots of points about this and I'm curious when you think about your road maps, when people on your team are bringing you project ideas, like how do you keep this in mind as you're thinking about this for Blackstone?

8:54 >> Yeah, I mean a lot of what we do is like, you know, simple equation is is giving enough compute every problem is solvable by like hardware and software at some point in time in the future. It's Elon talks about this energy-wise, etc. So it's like it's a big map, it's a whole bunch of stuff. How do you bring it back to reality? Like what I spend a lot of my time doing from like my seat working with engineers in my team, with product people, with our SRE function, with like stakeholders is really trying to define like, you know, what is the ROI of what I'm being asked to work on. And so like when I think of building agents, I think about you know, getting data into the right spot, think about building dashboards, trading platforms, whatever it is. It's like if I look at like what's the ROI that's going to be created from the actual ask that's coming in to me from a team perspective and then like what's the what's the true cost of actually building something today? Like how can I balance that equation and then I make a decision of like, you know, is this something where the models are going to get good enough?

9:51 I'm going to have more than enough compute in 6 months that it's not worth trying to tackle the problem today. It's better to delay it and actually address it with one of the new models. We have made that decision before is like let's actually wait for the model to become slightly more sophisticated so that we can solve these problems with it because it will cost less at that time and the need for that application to be built today isn't pressing. There's no like real business value increase that can be like will be created because it's produced today versus 6 months from now.

10:20 So, that's a trade-off that we actually, you know, make regularly and I think about regularly. the other thing that we do is like you don't want to wait too long to basically build something. And so, you can always be waiting for the new model, waiting for the new technology, or whatever else. Like the beautiful thing with like all of the various different software development harnesses today is if you lean into them, it's amazing how quick you can go from like concept, mapping out your planning and concept with like Fable for like, you know, just hey, work with me and let's define this with the product person or with the stakeholder to actually writing code to having it like a workable proof of concept. And when I say workable proof of concept, it's something people could click through, interact with, whatever else it is.

10:58 Doesn't have cybersecurity review, doesn't have all the other things that you need to do in our industry to make sure you like put something into production that scales, but you can get to a POC or a POV very quickly. And so, like that's one big thing I try to encourage all my engineers to do frequently is like just get to POC. Don't worry about PowerPoints, don't worry about anything else. Leverage tokens, which like I think it's a fallacy to be like counting tokens in some way, shape, or form. It's better to think about producing value. Go off, produce value, produce POVs, put it in front of people, iterate on it quickly, and throw it away if you don't need it.

11:29 You will spend far less time and money doing those types of things. And so, that's like a big way that we think about trying to solve it. It doesn't maps into the parallel like the you know, the how do you deal with increasing compute, making it easier to do these things, but like I just I think in a much shorter time frame. >> Yeah, it's so interesting cuz it's like, you know, I mean, you touched on so many things that I think everybody's probably been thinking about, too. And if you go back to 2023 when we were originally like sitting around as a small group talking about what we're building, we were aware of the better lesson then. We talked about it. We said, "How do we not get in this trap?" And I think at this point, it's just really evident to me that it's like it's definitely a gradient, right? It's not this binary thing about you can somehow build the perfect technology to index into all the up you know, everything coming in from the frontier in a a that you don't have raft to throw things away.

12:13 really it gets down to like where did we want to invest heavily and where are we willing to invest conservatively because like you said, we need something today, but at the same time we're just aware that like some of these investments we make particularly around things like we've built, you know, fairly complicated systems around all of our agent primitives. So, the harness, the sandbox, how egress works, you know, all of these things and we we also want to make sure that as like things are coming out either from like a frontier lab or from like a a third party that we're able to index into that. And so, I think a lot of it is just kind of when do you update? And I that's one of the hardest things to do and I like for me a good example was I really updated really hard on cloud code. Not so much that it was suddenly tools that like in my profession that we could use that were really incredible. What it really was for me was just this idea cuz I was a bit skeptical up to that point of how we would get something that was like truly generalizable. It's like, well, what does cloud code give you? Well, you can write code and then it can execute the code and like we're basically running a computer in some verifiable way. At that moment, I was like, well, we have all the ingredients for like a truly generalizable intelligence because if you can go do this at scale, then I mean, you know, of course like you can basically do anything in terms of most knowledge work and actually even outside of knowledge work, which we're seeing some today. And you fast forward to now and like all these tools that people are using these like AI assistants and things, this is why they're so amazing.

13:26 It's cuz you can ask a question that's never seen before fully out of distribution. It can spin up a computer and go do this thing in the background. And for consumer, it's amazing and then we're also seeing this now within the enterprise, too. And so, for me that moment was like, well, where are we going to invest heavily? Of course, we still need to invest in what we're building on the agent side, but going back to what John mentioned, how we do our data capture and distillation and how we make this available as like state to these models.

13:50 And then also just how we do our benchmarks and evaluations. It's And so, with like the law firm for example, we have this really unique situation where we started this law firm, you know, from nothing. So, we we've kind of grown together, they've grown with Anom together. And so, we work directly with lawyers to develop to stand up these benchmarks and evaluations. And effectively, rather than think of a traditional road map, like when when are you going to build an agent that can go do these three things? It's let's have a scorecard that determines when an agent or when a human even is has success is successful at completing these tasks across the various dimensions. And then we want to be able to see this system improve over time. So, as we make general improvements to the system, we should see an uplift to these, you know, these things like an attorney would expect to improve their workflow, but at the same time not not getting kind of going back to the state that we were originally where we had added a lot of structure to it in a way that ultimately we found that we were actually we could get even better performance if we were to remove that up front >> structure. Yeah. I think like another thing that's important to think through is like you can produce, like I was saying, a lot more proof of concepts, proof of value very quickly. I still think like the hard yards for a lot of development out there is getting it to production. And what I mean by that is like infrastructure buildout that actually runs inside of the infrastructure that's available inside of any financial services or enterprise company for that matter out there. Like that is a challenge that a lot of folks like underestimate. If you're selling AI tools, it's that is hard on the other side, just so you know, I deal with it quite a bit.

15:15 The other thing is is just like dealing with like the cybersecurity side of the fence and the supportability side of the fence. And so like one thing I've noticed, you know, the agents are getting better at this now, but it's making sure that the actual software that's being written is truly traceable and it feeds into your SRE platform so that it can be monitorable in production. Those are things still that like I like to have people think about. It's like they haven't started to scale yet with agents. They will, but it's something that from my perspective I think about regularly.

15:44 >> Makes sense. And we our approach to this, I'm sure a lot of other companies and maybe even you guys have taken this approach as well, is there's parts of our code base that we like still require like complete human oversight from when it comes to designing, building, code ownership, even approving any kind of changes to the code base. And these are like our core primitives. And these are things where, you know, basically you can build things on top of these, but the things you build on top of these are not allowed to like change these systems without having like human review. And it's because of things you mentioned is like, you know, data security for us is, you know, is is is like as paramount as delivering quick. It's actually more important. So, and for depends on where you are and if whether it's a consumer product or for us with obviously working with highly sensitive data, like we have to build in so much infrastructure around this and ensure that we're not ever getting out ahead of like, you know, these parts of the stack, which it can be hard cuz maybe somebody brings you and I'm sure you've seen this like a really cool demo and it's really exciting and you're looking at it terrified cuz we're like, "This is the This is very cool, but to get to this point in terms of capabilities, like it would expose so much that we're not able to do yet." So, I think it's just always that trade-off. So, we just, you know, basically we invest heavily in these things and then invest conservatively in places where we realize as the frontier continues to improve, we want to make sure we do we have like conduits. The way I described it actually this morning with the team was like, "Basically throughout our platform we want to have like conduits for what's coming from the frontier. You don't want to have dams where like the intelligence is coming in and it can't quite get into your stack because you built things in such a way."

17:09 Again, it's a gradient. We're going to get some of it wrong, some of it right. Just being very cognizant of it, talking about it a lot, and making sure that our core pieces are are locked down, but then also, you know, being very open about just ripping and replacing. I think that's like a thing that I think everybody's gotten really good at. I I think that's like one of the biggest I mean I like I I've been doing this for 27 years. Sometimes I feel like it's been like 100 years. Other times I feel like it's been five. I think the the thing that's like really made a material difference like when I I look at the ability to write software today versus what I was doing 10 years ago was 10 years ago it took so much time, effort, and money, and people's like intellectual capital to produce a piece of software. It was almost like the ship had sailed. Like by the time you designed it and started writing code, it was out to sea and you were going to ride that baby for like as long as you could until it got to the other side of the ocean. And like what I've become very comfortable with today is like for the most part a lot of like internal applications can be just thrown away.

18:05 It's build them, have them work for a purpose in a point in time, but understand that they're going to disappear at some point in time a lot now much more quickly than ever before, and then just think about like what comes next. And so I I think one of the challenges with like sitting in my seat or any like good engineer seat should be constantly questioning like, you know, should I rewrite this and does it cost me less to rewrite it than it cost me to keep versioning the current version forward? And that is like a question that a decade ago never would have came up in anyone's head. If you went to say I want to rewrite an internal application, like your CFO would have looked at you like you were insane. And I think now it's a completely relevant thing that all of us should be doing cuz net new applications might produce way more business value at the end of the day. So, it's just something that I encourage folks to do. It sounds like you guys are doing quite a bit at Norm.

18:51 >> Yeah, exactly. And I think it's just that concept of what used to be expensive that's now cheap. And if you don't update quickly on that, then it's, you know, it like you it's incredible you said it feels like 27 or 5 years or 100 years or 5 years. It's like that if you over if you over index too hard on something that was in June and now it's October, like, you know, it it feels like in some ways there's been like 10 years of technological advancement. So, Well, great. We only have a minute or so left. Are there any closing remarks you >> I mean, I think like the the one thing I'd like leave the audience on if you if you want like, you know, like good takeaways from this, it's like we talked about a lot of like very interesting things. I think like what makes me like excited to go to work is like it's cool to get back to engineering. One of the things that like I value is like, you know, I was a hardcore low-level engineer for 10 years at IBM and like wrote like, you know, this is way back in the day of using notepad and VI to write like instruction level code for RISC machines. It was like awesome. It's nerdy as all heck. No one does this stuff anymore, but it was like fun for me. I still enjoy those days. As you get more senior in your career, it like the barrier to actually writing code on a daily basis disappears or it becomes more like hard to get through like dealing with a stream like processing stuff off a stream. It's just all these things are like, oh my god, I can't even get started cuz the scaffolding is so hard. With like AI, it's been like reinvigorating to get back and actually build stuff cuz it's like I never lost my knowledge on how to architect things.

20:12 I see this with a lot of engineers out there. They understand because they've been in the market 10, 20, 30 years. They understand the complexity of system design which hasn't changed and now they can actually express that knowledge with these new cool agents. And so I think it's a super exciting time to be like starting a company, working at a company and like building software cuz there's like a whole new world that we can create out there. So that's what like excites me. So well thank you so much. It's been an honor. Yep, thank you guys for having us here today. I appreciate it.