Box CEO Aaron Levie on why businesses aren't ready for autonomous AI agents

0:05 to see you. >> So, one of the things I've been asking you a bunch of times on stage >> Oh, no. Don't talk. You're not going to start with Poom, are you? >> With what? >> Poom. >> No, no, no, no, no. We'll get We'll get there. but you know, start on a happy note. Like one of the things I always feel like >> you all are great at because Box for those who don't know you know a long time ago it's just where I went to like save files and share them with someone but for a long time you have been really at the center of how businesses use AI.

0:37 You're one of the first to try out all the new models. You're always really thoughtful not just about which model is better but also what do businesses need to do to not get stuck. And I feel like it's been enough time since we talked, which can be like a week or two and progress. where are businesses succeeding as this shift to agents? Where are they getting stuck right now? >> Yeah, I I think it's it's always this this funny thing. So, I live in two worlds. So, you know, we're we're based in Silicon Valley and Redwood City. So you get the the complete, you know, kind of incessant innovation curve and you get to, you know, hang out with researchers and sort of see where model progress is going and then you get instantly calibrated because you go into the real world and you talk to, you know, banks and pharma companies and manufacturing companies and and have a good sense of like what are actually the the roadblocks and the the hurdles and there's this general I think you know common conversation we have on on you know these types of conversations or or on you know Twitter you see which is like complete and utter vertical takeoff of things like coding agents where it just works insanely well. We we we know how to deploy these models against these types of tasks. and and you have a complete revolution in terms of how software engineering works. And then you know there's sort of these you know this this histogram of things that are further away from and closer to coding.

2:03 And that kind of tells a little bit of the picture of like where where agents have been adopted well and and and taken off in the enterprise. And so you go into the real world and you look at, you know, where can you deploy these agents that have access to your data that can they can use coding and computer use to be able to automate tasks. you know, we're starting to see, you know, some workflows get re-engineered in in back office operations. we're seeing, you know, the early signs of agents that will be sort of set up against large sets of data to pull out insights and be able to automate workflows around that data. But we are we are still truly in you know the like the moments before the first ending of of of this rollout and it's because there's just a tremendous amount of change management that has to happen in the enterprise. most enterprises don't have their data environment set up in a way that they could easily go and deploy agents. They have access controls and data governance and security challenges with getting the right data to agents. They don't they haven't figured out exactly where should the human be in the loop in the workflow? Do they let the agent do more of the work?

3:05 Do they have to have humans kind of review each of the major steps of that work? you have governance and compliance teams that are trying to figure out what's the liability regime that we're going to be operating under. Can an agent make a decision in that you know, critical financial transaction or that critical healthcare decision? So think about like all of those things that all of us on Twitter never have to think about like you know when you go and see the vibecoded app that that you can go build in you know in an hour that used to take 3 months that it's an incredible sign of the capability progress but has almost no sort of relationship to then actually go and the work it takes to go change the workflow inside of one of these large enterprises. So so this is why I think diffusion is going to take longer than people realize. I don't think that's a bad thing because I think it gives the economy sort of a chance to to really wrestle with where where do we take these these, you know, the the workflows themselves? What what is the future of jobs? How do we roll out AI safely? So, I think it's actually a good thing that this is still going to take a lot of work. And it also means there's a ton of opportunity. So, right now there's a tremendous amount of opportunity at the applied layer of AI.

4:11 The companies that take the AI model breakthroughs and can bridge them into the economy. all the roles that will produce in terms of FTEES and new kinds of of of integration firms. so that's that's where we are. >> So there's two things I heard in there that are big obstacles. One is, >> you know, >> I think I said seven, but >> you said seven, but I'm going to focus on two. One is >> the the role of humans in there. And I think, you know, I sort of think of it like the autonomous cars thing. Like if we were just building autonomous cars to drive by themselves on roads, it'd actually be a much simpler problem.

4:44 but we're trying to build that >> the pesky humans are in the way. Yes. >> And I think that's true in business like you know it's important because that's where the whole economy is but when you're trying to like deploy AI having it fit within humans is a challenge. And then the other piece I heard is agents are actually a challenge because we don't know how much permission to give them. We don't know you know there's this whole like you know initially we were like oh we don't want them running all over the enterprise. No problem.

5:12 we'll match the agents permissions to the human that they're an agent of and that'll be great, right? And it turns out, you know, the CEO may have the right to see everything in the company. We probably don't want the CEO's agent, right? Are companies getting stuck more on one than the other or they're both >> Yeah. >> It's it's interesting. I think I think there's there's there's some parts where the the the self-driving kind of Whimo analogy works and then some parts where where it it it breaks down a little bit. but you know like self-driving cars doesn't have this problem where let's say you wanted to you know get a loan you know somewhere and there was this autonomous system that you know made decisions about loans and you were like well you probably want that autonomous system to self-improve over time but the act of self-improving means it's going to sort of retain information about other people's loans that have come before and that's going to it's going to use that judgment in future decisions. but agents are still kind of sloppy in random areas. So you can't 100% guarantee that that agent's not going to accidentally go and sort of you know leak data downstream in some you know future workflow. So so that's like that's an entire problem just in the realm of agents. It it had nothing to do with humans being involved in the process or not. It has to do with with our privacy laws and and the fact that we actually want these systems to be you know you know quite governed and controlled with the right guard rails.

6:32 That's all infrastructure that has to get built out in most organizations. And it's a it's infrastructure built just for agents. and and and a complete new way to think about, you know, how do you how do you separate data in in in this in this sort of new kind of way? Because me as a person, I could look at lots of people's loans in that workflow and know how to not share information between them over time and like, you know, because I I'm able to compartmentalize that information. The agent, you know, doesn't doesn't exactly 100% know how to do that. You can kind of see this because if you use the memory feature of most AI, you know, chat services, sometimes it'll, you know, kind of like introduce a fact that you said, you know, a month ago in an area where it's sort of like not relevant. Like it's like, no, that's not my health issue anymore. I was asking for a friend. And but like a person wouldn't do that. A person would have the intuition and the judgment to be like, okay, I'm in a I I have a completely new conversation that I'm having with this person and those prior facts are no longer relevant. that is pure you know a mix of AI capability progress as well as just how do we think through you know how do we handle memory how do we handle context how do we decide what to delete and what to retain how how do you deal with the fact that too much information a model goes ary too little information it can't actually execute the workflow so these are these are the kind of just blocking and tackling things you have to go do in a new era of technology and this is the work that's ahead for most enterprises which is you know it's a little bit easier when you're starting from scratch and you no existing customers and liability and responsibility. It's a lot harder obviously if you're JP Morgan or Eli Liy and everybody else is sort of somewhere in between that journey but that's the the work ahead for us >> and I think you also we you know there's so many issues that it's easy to skip over a few but in that loan example it's also basing off the history of humans making loans which has all sorts of bias in there. So, you talked about the privacy, some of the other things.

8:23 I've talked for a long time, lots of people have, you know, the risk that we're sort of automating bias and in systems where it's going to be really hard to detect. >> Yeah, I I think that's true. I I I don't know where the equilibrium is on this one because the exact opposite could be true is in a in a much more centralized way, we could reduce bias with the right level of intention. so, so it's I I don't know which one that swings to, with the right level of thoughtfulness.

8:49 >> So, that one's we've known about for a little while as an issue to at least watch. The one that I think is coming up more and more is the accountability piece. Yeah. >> and in certain cases, it's clear, you mentioned regulated industries. A lot of the laws that we have in certain specific areas apply whether they a human makes the decision or an AI. In those cases, I think we're reasonably well covered because it's based on the outcome, not the intent.

9:17 >> Yeah. Although, although now the more that you hand off to the agent, we're going to be in a a funny situation of who's actually then accountable. >> Well, that was within that regime. >> That was my next thing is is how do you hold an AI accountable? >> Yeah. And so, >> well, andropic and opening, I don't want you to at least. So, >> well, they don't want to be held. Let's be clear. They don't care if you hold the agent accountable. They care if you hold them accountable. Yes.

9:40 >> historically the software industry has had this amazing way of solving all these issues. You click a little I agree when you load the software and they're off the hook. But >> yeah, just that little check box and we're we're solved. >> As AI is obviously the biggest tech revolution of my lifetime. >> I still remain really concerned that we're moving to a world where AI is making decisions. Humans are insufficiently in the loop. So maybe there's a human in the loop, but >> you get to a point where you can't you you're just clicking okay. Again, >> I I think this is a very unsolved area of AI from a case law standpoint and what the industry we we we can't have it both ways. We can't have super intelligence that you rely on for many of these critical decisions, but also no liability in the entire stack of of it making those decisions. So somebody at some point has to take on that liability and the ultimate you know like the way it would work today is obviously the the the service provider that you've contracted with as a consumer or as a customer is going to take that liability. The question is why would that company take that liability in a world where they're relying on the super intelligence for ultimately making the call. totally hazy area right now.

10:52 that that the only reason actually we haven't really tested it is because AI is still very early in in these mission critical use cases. so over the next 10 years this will all be you know kind of have to be figured out. The great the I mean the great thing for capitalism is like you know we will have all new kinds of insurance that gets created. We'll have new risk you know ways of of of basically you know sort of financializing that risk in some ways.

11:17 so the economy will will figure this out somehow. >> Okay. Well, let's assume that's going to be solved. Let's get to my next really thorny challenge. not going to quite get to PDM, but it's related. which is help me understand. I asked Richard this question, too. So, we're building intelligence that's going to be way smarter than humans. We're like the progress has been pretty incredible. I I don't really imagine it slowing down, but we're still like writing the guard rails. humans kind of using their best effort, but we're already seeing agents kind of doing their own thing. Yeah.

11:53 >> Which I don't love when it's my 13-year-old, and I really don't love it when it's millions of agents on the web. Why do we think a bunch of admittedly smart humans, but we're pretty simple creatures are going to be able to write a set of rules that a super intelligent system is going to follow in the way that we want it to follow? >> Yeah. what what are your thoughts? >> Oh yeah. So I think first of all this will be one of the defining you know questions of AI whether AI works or not will come down to this question which is and it's broadly the alignment question. It's the AI safety question.

12:30 It's can we build an AI system that we can trust to execute tasks on our our behalf in a way that is aligned with with us you know as a species and not you know do the the the risks that have been kind of postulated by by by some of the the more kind of you know you know researchoriented safetyoriented folks that that are certainly possible which is you tell an AI system I think Richard just you know shared the example you give it a task it thinks it's trying to execute that task but but along the way there's a bunch sort of interrelated tasks that it can solve in a hacky way to get to its end end goal that are very bad for us as a as a society or economy or species or or or individual companies. So this is like that's the multi- trillion dollar question is can we solve AI alignment and safety. In the meantime, we certainly have to we have to do more tractable issues which is can you secure your systems in a way that that sort of doesn't have to know the answer to that question. can you secure your data? you know we spend a lot of time thinking about the problem of you have crown jewels in the form of your your contracts and your intellectual property and your you know advanced research that you're doing and your your customer data. how do you defend that information protect that information from the equivalent of the hugging face you know agent swarm trying to find those kinds of resources? How do you get alerted on on you know strange anomalous activity? How do you sort of raise levels of defense when when you start to see kind of patterns that that that exhibit those those kinds of behaviors? So we have to build a lot of infrastructure you know at at box to solve this. you know I think the Cisco and IBMs and Palo Alto networks of the world have to do the same thing. The Salesforces the Service Now. So like all of the places where the data goes today and the workflows are that it's just going to be a massive effort to make sure that these systems are secure.

14:16 which which is like it's one word that has about 50 subp parts. how do we build the next generation of of threat detection and intelligence systems to to defend against these issues while at the same time the labs themselves obviously have to be working on AI model progress in a in a way that that you know has safety and alignment at the center of it. So, we're almost out of time and I want to give you a chance to talk about the incredible progress because again, whenever a new model comes out, I think of you all. You all tell me it's this greatest thing since sliced bread. It's more efficient.

14:46 >> It literally is true every time. >> So, which lab has surprised you the most? Like what what surprised you about the progress and which >> which of the players has surprised you the most? Well, I I mean I think the I like but like on the surprise quotient I would say actually meta may be the biggest surprise of of the year. So so it went from like not we we couldn't really understand what where the llama of progress was going to go.

15:13 obviously massive organizational shift and change over there and then boom you know muse and all the you know 1 1.2 and then 1.3 you know clearly at near frontier. so that's probably like on like the the the actual definition of surprise, you know, kind of, you know, what what I would say. But between Meta you know deep mind anthropic open AI and SpaceX I think you've got a really interesting for the US five player race between those th those players all effectively pushing the frontier you know sort of pacing pacing messaging and and you know kind of real real work aside they are all constantly pushing progress on the future of models. We are all benefiting from that u because models are getting cheaper. they're getting vastly more capable. they can solve much deeper problems in in real kind of pragmatic contexts in in the enterprise on legal and life sciences and healthcare and financial services. So we're all benefiting from the the incredible race that that's happening. Clearly, we need to make sure that that race is being done in a way that is is safe and and controlled and there's, you know, good best practices being shared. I think the the the the accords last week were actually a good strong update because you had you had really strong alignment from the major AI labs on how we are going to build AI safely. I think they will create shared accountability around that with the with the ability to have a rip cord from the government at any time which is like yeah we can just we can put this model on an export list and not let you release it. So so we have the kinds of tools that are emerging to build safe AI. We have alignment across the labs. and I think we have an incredible competitive race between them to bring down the cost of AI while bringing up the capability.

16:52 >> Well, I think you all can see why this is the other guy wanted on stage. Yeah. Okay. >> so Richard says, "Poom zero. Are you going above or less?" >> did he go full zero? >> He went full zero. >> I I'm willing to add like a like like some, you know, decimal integer, you know, in there. So maybe I' I'd say but but very much sub 2 or 3%. and mostly because I have bio friends that get pretty freaked out. So so that that that that's always interesting. And then if you if you go hang out in the wrong parts of the internet right now, the math people are very afraid of of cryptography. That kind of scares me.

17:34 But but you know at the same time we have also incredibly smart people working on the right levels of defense and resilience of these systems. so so I think we actually are also balancing this out with a lot of really strong investment in how to build you know safe and and and and strong AI as well. >> So I want you to stay seated. We're going to have to leave it there but I want to thank all of you for coming. I want to thank Richard and Aaron and Jonathan and Naveen and yeah, we're going to have some light appetizers and stuff. So stay around and mingle. we have more events coming up including as I mentioned our AI Plus Summit December 3rd here in San Francisco. Next year we'll have them back again in SFDC in New York. So wherever you are and you can always join us on the internet thanks to the people who tuned in and we'll be on axios.com all the time. Thanks.