How Agents Can Work Together — With Vijoy Pandey

0:00 What will multi-agent systems look like when they go from training to production? Let's talk about it with Vijay Pande, the SVP and GM of Outshift by Cisco, in a conversation brought to you by Outshift by Cisco. Vijay, great to see you. Thank you for being here. >> It's a pleasure to be here, Alex. >> So, Outshift by Cisco has been on top of this idea that agents are going to work together to get things done for a long time. We spoke with your colleague Guillaume a few months back talking a little bit about the Internet of Cognition and how your division within Cisco really has been doing the research and looking at how multi-agent systems will collaborate and get things done. Then a crazy thing happened, right? the world started to become exposed to the fact that agents can work together. Of course, when we saw what happened with OpenAI and Hugging Face hack, and it led to lots of questions about what this is going to look like when it goes towards productive uses as opposed to what we saw in the past, which was a destructive use. So, we have the perfect person in the right conversation here to go into what happens when agents work together because you've actually worked on doing this in practice. So, I'd love to hear a little bit first about what you've built when it comes to multi-agent systems.

1:15 >> Alex, agents have gone through three phases pretty quickly. They've gone from isolated agents to islands of agents, and these are agents within a single harness. So, whether it's OpenAI harness or Anthropic harness or your harness of choice, to then moving towards connected agents. So, isolated islands to connected. And what we've been seeing is a whole host of use cases around islands of agents solving for certain things.

1:47 As Cisco, what we've been saying seeing in the enterprise is agents from different vendors sitting in different harnesses on different clouds belonging to different organizations and teams, all trying to come together to collaborate to solve for a business goal. And so, connectivity between these agents, isolated oil islands, is actually what we've been working on for the past 18 months or so. And we've launched a couple of open-source projects. Agency is something that we launched about a year ago with Google and Dell and Oracle and a whole bunch of other players to solve for this problem that looks at discovery of agents.

2:30 Can you figure out agents with particular needs and capabilities? Then identity, which is giving them the access controls for that what they need to do for only the duration that they need to do that task. Then communication, this is where we work with the A2A project, the MCP project, all within the AIF within the Linux Foundation. And that's the communication layer. And then finally observability, which is are they really doing what they're supposed to do? So, those are the four pillars that between Agency, A2A, and MCP, they solve it in an open and interoperable manner. And what we've seen is a whole host of use cases in the enterprise across many verticals.

3:14 Network configuration, which we are familiar with. Cloud platform engineering, healthcare, video surveillance. There's a whole host of applications. We work with ServiceNow and SoftServe and and WebEx and Digital Realty and a whole bunch of partners to build these playbooks across many of these verticals that you can go and read about and and model your use cases around that. >> Okay, Vijoy. So, this is interesting because what we saw with OpenAI, for instance, was an example of a bunch of agents really from the same company and the same origin working together. And so, what you're what you've seen is a bunch of agents from across the internet and different services finding ways to collaborate and get things done. So, you know, I think there's been a lot of fear about what could happen just with the first instance. You know, some people might say this is even scarier.

4:09 But, I'm curious what you you know, you say that this could be productive. So, since you've been studying this and studying and working on it, can you tell us a little bit about what you've learned after doing it? >> First and foremost, agents are no not dissimilar from human teams. So, you can think about business goal, a business outcome, that a team comes together on. Because we're all subject matter experts, we come together as subject matter experts to solve for a business need.

4:38 And so, the fact that we want agents to work together is going to be the de facto model of deployment for agents in any large enterprise. You will get agents from Salesforce and Cisco and Microsoft and ServiceNow. And by the way, these are all the partners that we work with to then come together and solve for a business need that, for example, in this case Cisco might have. We might be using agents from all sorts of vendors because they all come from for for with different expertise and and and and sub-subject matter expertise.

5:10 So, what we've been seeing is that connectivity was just the first layer. So, discovering these agents, plumbing them together, to then solve for a business outcome, as I described using agency in A2A and MCP, was just the first layer. And that was the easy part. As we started deploying them in production across all of these verticals I talked about, we figured out that collective behavior itself starts breaking down. So, not just individual agents making mistakes, not islands of agents making mistakes, yes, those things happen, but also collective behavior started breaking down.

5:48 And again, when I speak through this, think of the mapping with human teams where agents started misaligning on intent. What is it that we are really trying to solve? Because every agent, just like every human, comes in with a certain set of incentives. So Salesforce agent, a Cisco agent, even personal agents, they come in with certain intent and incentives. But what we want them to solve for is a global intent. There's something that the organization wants solved. So misalignment of intent was a problem.

6:19 They started doing local optimizations without anybody else challenging them. So that was a problem. They had some sort of coordinated activity, but the outcomes were heading in the wrong direction. So this led us to a more basic question, which is what happens when you connect agents, but there are silent cognitive failures that accumulate over time inside a multi-agent system? So what we started measuring these outcomes, and we found out that 100% of multi-agency teams, whether they are in islands like the hugging face example that we had, or whether they are connected agents, 100% of multi-agent systems had at least one cognitive failure.

7:06 Misaligned intent, non-coordination, doing something that you could not verify towards a goal. Pick a cognitive failure, but they had at least one cognitive failure. Second thing, they ended up with 98% accuracy. With two cognitive failures, they dropped down to about 80%. >> That's where it compounds. >> It compounds, and with three it dropped down to 16. So with just three cognitive failures, and by the way, we saw many many many cognitive failures. >> Right. >> But with just three, you are down to as good as picking a random number in your outcome.

7:41 >> So what is going wrong in these systems and that are causing these failures? Is it inherent to an LLM or is there something that you can pick out and potentially fix? >> What's happening is that bad assumptions are spreading like wildfire across these systems. So, an agent introduces an assumption, whether it's an assumption towards intent or whether it's an assumption towards coordination. The next agent takes that takes that assumption takes this as context and this just compounds that bad assumption.

8:14 And it treats this as evidence. So, that's what's happening. And so, either other agents challenge or they just take take it as evidence and then move move forward. And by the time you're done and you end up with a verifier, that verifier could be an agent or it could be a human that is supposed to verify the outcome, the system might be looking consistent externally, but the entire system is cognitively in a very bad place. And what's happening is you have all of these agents with local incentives, they are leading to local optimization that is reinforcing this bad assumption.

8:50 These assumptions are propagating, verifiers are also inheriting the same premise, and it's just becoming more and more consistent in the wrong direction. So, connected agents can still have different goals, different contexts, different assumptions, different constraints, different beliefs, and they're exchanging perfect messages. So, if you're looking at observability, message of message are flowing completely well, but you're going through and you're going towards completely wrong shared understanding.

9:24 And so, from a dashboard perspective, your dashboards are all green, but your shared understanding and your cognitive understanding is completely red. >> You know, Vijay, there's a a debate that's going on right now, which is do you even need agent multi-agent collaboration or can you use just one agent? Let me just get give you an example. you know, a multi-agent system may have an agent that does the action and the agent that checks the work. Right?

9:50 The argument to have a single agent would be, okay, you just have the agent instead of having one that does the work and one that checks, it just does the work and when it's done it checks. And so, you know, the failures and the compounding failures that you're describing might suggest to me or others that, you know what? It might not make sense to have multi-agent collaboration at all and maybe just go single-threaded. So, what do you think about that?

10:14 >> That's a super interesting thought there, Alex. And I think the way I would phrase it is having diversity in your environment, especially as we're dealing with distributed systems, has been a design principle that we've been using for decades, even before agents and AI came into the picture. I come from a hyperscaler and we used to build infrastructure that where which did not share a single line of code, which did not share a single infrastructure element because failures then might happen in similar ways.

10:49 And you're building these resilient systems that should be resilient against similar failures. And so, design principles like this first and foremost should be the de facto way you design your systems. So, regardless of agents. So, you should be building checkers and creators and verifiers and security agents and scale agents using different stacks because they don't fail similarly. So, that is design principle number one.

11:19 The thing that the second thing I would say is you don't have a choice. So, in a large enterprise, you will deal with agents coming from, like I said, a Salesforce or a Microsoft or a Cisco or or a or a ServiceNow, or or SAP. I mean, take your pick. We all use these systems. They're all moving towards agentic systems. And to set up something simple, so I'll give you a very simple example that an enterprise deals with day-to-day.

11:49 And that is a large deal flow. Or even a small deal flow, it doesn't matter. Like, let's say there is a $2 million deal flowing through your system. You will have a sales agent. You will have a legal agent from Harvey or somebody else. You'll have a finance agent. You'll have developers and SREs and and security agents that are primarily what people are familiar with when they think about agentic systems, but these are one kind of agents. You have a marketing agent.

12:17 So, you have all of these agents working on that deal flow. And a simple thing like, "When is this feature ready for this deal?" The entire engineering team, SRE, security, code, they might say, "It's ready on date X." That is not really ready. That is when the engineering team is ready. So, ready means different things for different people. By the time that flows through the system and the sales team can commit to the customer that the product is ready and we can actually sign the contract, that's a very different date from what the engineering team said. And this happens every day.

13:00 So, this entire problem and this pipeline is owned by many agents, but come from Harvey, Salesforce, from coding agents that might be Claude and Codex. And so, you see this diversity that is going to live in your enterprise environment all the time. We've seen this in the past even with coding, but it's actually broader. So, the answer is not fewer agents. The answer is better machinery, better infrastructure that allows agents, just like human beings, to then collaborate as subject matter experts to solve for a business outcome.

13:33 >> So, what has been missing then in terms of making this all work together? Is it what you say, which is like a better infrastructure that they can use in order to collaborate? You tell me. >> Connection has been the first missing piece, which we've been solving so far. So, we've gone from isolated islands to connected agents. This is what I talked about earlier, agency to agency. We've solved the connection piece. And I'm the I'm a futurist. I look at My group actually looks at what's emerging tech. So, we think of the problem being solved even though we do need to propagate it into every enterprise and let them and and and look at use cases across the board. But, connection turned out to be turned out to be easy part.

14:18 Agents can connect together without thinking together today. And so, what is missing is I would draw parallels to how human intelligence evolved, where humans became smarter and smarter. Our brains became bigger for hundreds of thousands of years, but the innovations we did remained isolated within ourselves or within small teams, islands. And about 70,000 years ago, language was invented. And language caused what is now known as the cognitive revolution, where humans moved from isolated individual intelligence to collective intelligence. So, we could go and say, "Let's build this village together. Let's cap Let's capture that animal together." So, we could do that as together, not individuals.

15:08 And that's how civilizations formed. And so, what humans did was did three languages allowed humans to do three things. First, share intent and goals. And coordinate towards those goals. Second, build shared context. So, this is the libraries and the modern manifestations of those SharePoint and Google Docs. It's the same thing. So, sharing context and build upon that context. And third, because of those two things, we could then reason together. So, that is the missing infrastructure layer even in agents where we need to have protocols to share intent, coordinate towards those intents, then have a shared context layer, shared knowledge and belief system.

15:53 That's an emergent behavior. And then finally, go towards collective reasoning as a multi-agent system team. >> But Vijo, didn't that shared language also make humanity more manipulable? So, how do you do this without allowing that manipulation to come in? >> The amazing thing about agents is that even though they are They They passed the human the Turing test a long time ago. I mean, there's a probable paper that was written about 2 years ago where we we stopped distinguishing between humans and agents.

16:23 But, the amazing thing about agents is that that's still software. And you can codify a whole bunch of things for agents that you could not codify easily for humans. And so, you can actually we can actually build infrastructure to prevent against some of the things that you're pointing pointing out. And so, the way we're looking at the system, Alex, is that we're building it in a couple of layers. So, first and foremost, >> >> and and the analogy to think about as I walk through this is not a self-driving car because that is completely agentic, but an automated like half self-driving car that has been around for a while, which is things like lane correction, things like making sure that the car is braking when you are approaching a car in front of you. So, this semi-automated driving has been around for a while.

17:19 So, what we've built is a base layer of observability and explainability at the cognition layer, at the semantic layer. So, it's observing semantic exchanges and it's explaining those semantic exchanges between these agents. What do these agents mean when they say something X or Y? And so, we've built this basic observability layer at the at the semantic layer. And it's looking at whether you're driving the car in your lane or you're starting to veer off your driving lane.

17:56 So, when you see drift in intent or when you start seeing drift in coordination, then we kick in the corrective action. And the corrective action are protocols that come in and say, "Wait, wait, wait. You're not aligning on intent." So, there's a protocol that we're calling Accord and it's an acronym and I'm not going to attempt explaining that acronym here. But it's Excuse me, it's basically a protocol which then aligns these agents back into the global intent.

18:32 So, it kicks when you start veering on the intent. When you start coordinating and you see the coordination is veering off course, then we kick on another protocol called Concord and then you go back to aligning on coordination. And if nothing is wrong, then we let agents just do what they're doing. And as they do this and as they're kicking these protocols into gear, we also have something called cognition engines and these are accelerators or guardrails. So, you're to your earlier point, yes, they can be manipulative. They have local incentives.

19:09 So, agents can act in weird ways either non-maliciously or maliciously. But we have these guardrails and cognition accelerators that then shape where they are going because we have a good grasp, semantically, of what they're trying to achieve. >> Okay, but Vijay, I've had a lot of smart people come on this channel and tell me it's only a matter of time until these AIs get smart enough that they're able to coordinate with themselves and break containment in a way that actually matters. You don't seem that worried about it.

19:40 >> I'm not worried at all because I'm a big believer in you create technologies to solve for the problems that other technologies have created for you. And we've done that forever. Let's start with fire. This is a canonical example. I mean, you can burn the planet down, but we know how to contain fire. And all of the narrative that we're seeing out there around, yes, you can break harnesses and and so on and so forth, and you can go and poison the internet, and there's a lot of that happening, and these are all agentic problems that we've created, technological problems that we've created. I see those things as opportunities because these are problems to go ahead and solve through other pieces of technology. I mean, let's take the deterministic world. A few maybe even a decade or a little less than that, people used to write virus software, and they still do.

20:37 And viruses used to be pieces of code that go and break containment of some sorts and and infect computers and infect the internet in some ways. And then you used to write software as an opportunity to then go ahead and contain such software. This is no different, and the fact that agentic capability is becoming or moving towards ASI or AGI, pick your term, but it's getting better and better over time, those same pieces of tech can be used to then contain what's happening between agents as well.

21:11 So, as we're building out these COGS and GATS, which are cognitive accelerators and guardrail technologies, they will be using the best models out there to guardrail things that are happening. Similarly, the protocols, as we're listening to these semantic exchanges and trying to understand the meaning of what these agents are exchanging, they are leveraging the best out there in terms of agentic technology to then align these agents, whether it's intent or coordination. So, the same piece of tech is helping you produce as well as guardrail.

21:49 >> Right. So, how does this fit into computing more broadly then? >> Computing itself is actually going through a pretty profound foundational shift. And when you think about computing for the last 70 plus years, we've lived with one rule, which is determinism. And we've been familiar with this paradigm as the same input, the same output every single time. That fundamental computing rule of determinism is now shattered. That axiom is dead.

22:24 And we are moving towards what's now known as probabilistic engines. And the workload actually moved first. So, that's what you're seeing in agentic computing, where agents are workloads that are probabilistic engines. The inputs that you're providing are non-deterministic inputs. You say, "Give me the best drug." Or, "Give me the best material for this use case." Or, "Give me the best logistics plan for for for some transport company." So, these are probabilistic inputs because best is not a binary input or a question. It is an optimization problem.

23:03 And so, what comes back is also a probabilistic response, which is this is the best drug given the constraints that were given to me, which is this is the data you have, this is the time you have, this is the cost you have. This is the best drug for the time being. Any of these agents or anything that we do, we know that that drug is only going to improve over time. There's not the foundationally binary best drug forever that we're going to achieve, right? So, it it is a it is a probabilistic outcome. So, we are moving towards a probabilistic engine in the workload, and that move has happened.

23:40 What has not happened is the substrate. The substrate on which these probabilistic engines are sitting are still deterministic. You have CPUs and GPUs and TPUs that are computing binary. And they're still very, very deterministic and binary, and we're approximating the probabilistic world by matrix multiplications and pseudo random number generation and all of that stuff, but it's an approximation. So, what we are going to see is a pretty massive shift in the compute substrate as well, where the computing engine itself is going to move away from binary deterministic computing to different kinds of computing.

24:23 And we at Cisco have a pretty simple philosophy here. All the things that we're looking at is as you scale computing, whether it's at the workload layer or the infrastructure layer, there are three ways you can scale computing. You can scale it algorithmically, make better and better algorithms. You can scale it vertically, make bigger and bigger nodes, whether they're LLMs or agents or servers or GPUs, or you can scale horizontally, which is distributed computing, where you can connect a bunch of these things together through a network and make them behave as a singular entity.

25:01 And that's been happening for a while as well in the deterministic world. So, as we look at workload shifting towards the agentic or the compute shifting towards non-binary, we come in and look at scaling horizontally. And we are building technologies to then scale these technologies horizontally. >> And where else could you see a pattern similar to the one that you're seeing with multi-agent? >> So, the first place where we see this is in quantum computing. And that's the first >> Yeah, I mean actually that is something that there is a pretty tight loop here, Alex.

25:34 And >> Are you are you a quantum believer cuz I still I don't know where to stand on on quantum. >> Let me change your belief. >> Okay. >> In this in this in in this answer. And where I'll start with is those three axes. Whenever whenever you want to scale piece of computing, you look at algorithmic improvement, vertical scaling, horizontal scaling. So, let's start from where we are and what does quantum do. So, quantum fundamentally is a different way of computing. For the listeners out there, very simply put, binary computing looks at ones and zeros. So, your your result is either a one or a zero.

26:16 In quantum, you look at superposition of bits or information where you have a qubit that is a one and a zero at the same time. This is your canonical Schrodinger's cat where the cat within the box is dead and alive till you open the box and you realize, the cat is dead or yeah, the cat is alive. Right? So, then you go binary. So, the fact that it's dead and alive allows you to explore these really vast spaces like drug discovery and materials discovery in a very efficient manner. So, you can solve for some of these large problems very efficiently and effective effectively. So, that is the promise.

27:00 The reality used to be that we needed about 100,000 cubits to a million cubits physical to solve for some of these problems. That used to be the case. Today, in practice, what we are seeing is a dramatic jump in just that vertical scaling, where we have technologies that have gone from 100 physical cubits to now about 1,000 physical cubits. So, you already have single nodes that can do 1,000 cubits. And within 2 years, every public roadmap out there from some of these companies shows 10,000 cubits.

27:40 So, we're seeing 1,000 to 10,000 happen in the next 2 years. But, VJ, you're still nowhere close to 100,000 cubits or million cubits. Yes. Let's look at the other two axes. Algorithmically, it used to take about a million cubits to solve for what's known as Shor's algorithm, that, by the way, is the algorithm that is going to compromise all of security on the internet. This is your RSA encryption or Diffie-Hellman.

28:13 Shor's algorithm is the one that's going to compromise. It used to require a million cubits. It's been dropping over time. Just a few months ago, there was an algorithm that says Shor's now requires 10,000 cubits. So, now we're down from a million to 10,000 cubits. That's algorithmic scale. So, 10,000 cubits, we already have 1,000 cubit computers. And then we enter. So, we are building the quantum network, where you can take 10 of these 1,000 cubit computers or 100 of these thousand-qubit computers, connect them in the quantum domain through a quantum network that we're building, and you will get to 100,000 qubits through the network. So, that's horizontal scale. So, given those three axes, I'm convinced that Q-Day, which is the day that encryption is broken, is 2029.

29:10 So, it's 2 years away. And that is very high confidence. I actually think that if you're not looking at scale where everybody's deploying these quantum nodes, if you're looking at 10, 15, 20 companies deploying these nodes, I'm convinced it's not 2029, it's 2028 or earlier. >> what happens to the world when encryption is broken? Sounds like a lot of people are going to get into some trouble. And maybe maybe Bitcoin becomes worthless. What do you What do you think's going to happen?

29:39 >> Well, I would say paranoia should kick in because this is not science fiction, this is happening just like the ChatGPT moment that happened, and everybody woke up to wow, transformers are real. They've been around for some time, but it's consumable, and it's real, and I see it, and I can play with it, and I can do something with it. There's going to be a moment like that in quantum, but unlike AI, which is a small matter of software or programming, this is fundamentally new tech all the way from the bit level all the way up. So, qubits all the way up. So, this is a wake-up call for everyone.

30:21 But also, I would share the paranoia a little bit and say, A, don't be worried about every traffic that's flowing because the fact that somebody can break encryption on Vijay buying socks on Amazon does not matter. >> I would be very embarrassed if my socks buying leaked. >> Yeah, I I don't want people to look at my red socks, but that's okay. I mean, they they they can I mean, embarrassing behavior is not the most important thing somebody can do.

30:47 But, it's also ephemeral. So, you have a connection that's built up, you drop the connection once the purchase is done. So, these are very ephemeral connections. It's not important. But, what is important is services that are important, that you need to protect. I think people should start identifying those services today. We have a service that we offer through Cisco that allows you to just observe all the services, software services, or machines in your environment.

31:17 Identify the endpoints that you should be paranoid about. These are longer-lived, sensitive endpoints. And then, for those endpoints, you should be deploying a software fix that is post-quantum cryptography. Think of it as a new version of TLS that leverages different encryption schemes, lattice-based, instead of factorization-based that RSA and others are based on. So, this has already been standardized by NIST.

31:48 You should be leveraging those and deploying them starting now. Because the other thing that people are not aware of, maybe, is this class of attacks known as harvest now and decrypt later, which is somebody might be tapping into your network. >> I just heard what you the way that you call it, and given what you've just told us, I'm freaked out by this. But, go ahead, please explain it. >> It's very simply put, somebody might be tapping into your network either physically or through a routing hack, doesn't matter, or through a backdoor somewhere in your stack.

32:23 They might be storing all of your encrypted traffic. Today, they might be all encrypted. But, they might be storing all of their encrypted traffic, and the moment they have access to an algorithmically, vertically, and horizontally scaled quantum node, all of that time >> underlying encryption >> is plain text. >> Yes. >> So, get paranoid, folks. >> Okay. I I certainly am after hearing that. All right, Vjay, you know, since you are the expert, why don't we just talk a little bit about what networking is going to look like over the next decade, what it has to become given all the changes that we're seeing.

32:57 >> So, we've been talking about that a little bit over the course of this conversation, Alex, but primarily the computing stack is foundationally changing. It's going to change dramatically in the next few years, decade, and so on and so forth. And dramatically to the point where the last 70 years is going to look like a joke. And so, we are going to see new kinds of compute emerge all the way from applications to software infra to hardware.

33:28 And as that computing stack changes dramatically, foundationally, Cisco is here to horizontally scale that stack. So, that is our value proposition. That is what we're focused on. Now, what does that mean? What it means is for the past, again, 40-plus years, since networks became a thing, Cisco has been in existence for roughly 40-plus years. Networks maybe 50-plus years. For that duration of time, networks have scaled data. So, they've been built to scale data across tens of billions of endpoints and billions of humans.

34:11 That's the internet as we know of. But the endpoints themselves are changing dramatically. The endpoints are beginning to think agents. They're beginning to entangle quantum. They're beginning to move embodied and physical AI. And they're beginning to fluctuate. And this is something that we didn't even talk about, which is a new kind of compute that is emerging that is not leveraging just quantum physics, but leveraging good old classical physics.

34:42 Things like thermodynamics and oscillations and things like that. So, the endpoints are beginning to think and tangle, fluctuate, and move. They are no longer data endpoints. So, we come in and say, "How can you horizontally scale endpoints of this kind, these kinds?" Whereas others in the ecosystem will improve the algorithmic and vertical scaling aspects of all of these endpoints. So, that is what's going to happen in the next 10, 20, 30 years.

35:14 And one thing I will end with also here, Alex, is when people ask me, "Is computer science dead? Do I really need to learn CS?" Yes. Because the world is dramatically changing, and whatever you will learn in theory books, in school, use that as foundational knowledge, but use it to solve for some of these new problems that are going to emerge in the next 20, 30, 40 years. >> So, you're hiring entry-level computer engineers.

35:45 >> 1,000%. >> Okay. All right. Bijo, great speaking with you. Thank you for all the insights. If people want to learn more about Alif by Cisco, where should they go? >> We are at alif.com, and we are an incubator, we are the incubator for Cisco. So, we are always looking at emerging technologies, so a big part of what we do is educational, because we are looking beyond the horizons in some way. And so, a big part of what we do is educate people around some of the technologies that we just discussed.

36:15 a big part is community building on open source. So, you can go to alif.com and learn a lot more, participate in what we do, and help us move the ball forward. I also talk about a lot of what we do in my Substack for your information and on LinkedIn. And I talk to a whole bunch of researchers who are pushing the field forward. So, there are many, many avenues to go and learn and participate because we need all of us to tackle this brave new world.

36:45 >> B Joy Pande, good to speak with you. >> Same here. Thank you, Alex.