Transcript
0:09 really my pleasure to be here and happy to share with you what what we are working on to build safe and intelligent autonomy and the unique uniqueness of the work we do at my lab is to bridge what we call the rigorous methods and machine learning and I in this talk tell you what I mean by rigorous method doesn't mean machine learning is not rigorous it it can be and what I mean by bridging them so the motivation of what we're doing is as engineers in general not not only roboticism but general engineers designing autonomous systems like any type of autonomous systems we have to decide between a cluster of different tools or methodologies to design our system and especially nowadays when we're looking at the very complex modern autonomous system including modern robotic systems we are like we we definitely take a lot of classes on these what do you call the classical approaches include not include but not limited to like control theory formal methods optimization design and many different approaches versus a pure datadriven approach when I talk about datadriven approach like machine learning, reinforcement learning, large models. by no means saying like they are now still clear separated. People are definitely working in the mixture of these ideas. But when I kind of like look at the fundamentals of these different two different classical approaches, they are there are tradeoffs and when we are deciding how to use these methods, we are most ca most most of the time looking at what to trade off for example when we are looking at safety stability and I would argue that these very critical properties they are typically by design using the classical method we're given a a model and we know what we are optimizing or what we are controlling the system against and by design by the theory behind them we get these properties by default when we use these approaches.
2:23 However, when it comes to a pure datadriven approach a lot of these very strict rigorous guarantees they are largely empirical. Well, in many cases, empirical is fine, but for critical systems like robot interacting with human, maybe empirical is not enough. And on and on the on the other side when it comes to the model, I just said like classical approaches typically require you to provide a model. So we start with a model. Well, we all know that models all models are wrong and they are up to certain fidelity, but they are distance to to the real world.
3:03 And a very big appealing part of machine learning is that model is a a physical model for machine learning to work on is not always needed and a lot of the information is learned from data itself. And when it comes to scalability, I guess that's one of the primary reason people are looking at machine learning approaches. The scalability is obvious bottleneck for the classical approaches in terms of the scale of the complexity of the system people can work on.
3:34 However, when I look back at reasoning, so this reasoning is not the reasoning mode of your large model. Although I do acknowledge that is reasoning. Although that reasoning is more of the datadriven based reasoning where we don't really know what they're reasoning about there they show you something but you if we use the classical approaches we have a way of understanding which components is supposed to do what what's the bottleneck when a problem arises there's a way for us to track now where these issues happen so I would say reasoning is tractable in classical methods but it's an emerging topic in machine learning and still people are defining what he means by reasoning and when it comes to being able to provide proofs and guarantees they are often there in the classical approaches however in datadriven approaches we see that it's really rare that people are talking about proofs and guarantee they are harness but what I'm talking about this mathematically rigorous objects describing the guarantees we can have so the what we are looking we're looking at is this critical gap how do we provide build pro safety at scale in the sense like we don't want to compromise between the two. I don't want to just pick one over another when I say can I have a way to bring the strength of both in one framework. So we have my my lab where my students work on each of these topic looking at how to bridge in this different unique strength and in today's talk due to time limit I will show you two approaches two complimentary path we are taking to build safe autonomy and there will be a bunch of other work unfortunately I won't have time to go over today but happy to talk about those if you have interest you can just contact me so the two complimentary path we're taking is from two views of looking at how we ensure safety and ensure guarantees in a safety critical autonomous system. The first is to ensure safety and mission satisfaction by design. Second is to to understand when and where it fail by testing. These are both necessary and I will tell you I will as I I proceed in this talk I will hopefully convince you that both are needed to build safe autonomy. So at a very high level for safety and mission satisfaction by design, we want to learn policies that inherently have these constraints satisfied during the learning process and we build the requirements directly into the machine learning process. While in the other approach where you know when we build things by design there are always just limited consideration. We have the real world can be wild. No matter how hard we do there will be failures. So we want to generate cases to reveal when they fails why it fails. Is it by because of a design or because of some consideration we're missing or because of some things that we just cannot handle and we want to learn what to test the system to generate the evidence about the trust for the system. So in short the the rigorous the provable safe at scale is done by rigorous design and evidence from testing al together. Now let me start the first part what I mean by safety and successful by safe safe and successful by design. So we are looking at these design methodologies where people abstract the safety satisfaction of mission as mathematical objects and use them in the design process of autonomous system versus in datadriven approaches. We don't know how to insert this knowledge but we do get the scalability and model free model free power of the datadriven approach and the way we are looking to bridge these two approaches together is to enforce hard constraints. So we no longer compromise to say your constraints will be penalized they will be hard hard constraints and I ask them to be satisfied during the learning process and also to be able to handle very complex tasks. we decompose very this very complex t in the very automatic way but we can argue and reason why this decomposed version will satisfy your original mission now to be able to do that I need a it's not really a model but I need a mathematical description of the problem we are solving so this problem we are looking at at a high level it is a constrained optimal control problem let's say and the mathematical description can can can actually capture a lot of interesting robotics fix design problem but he says we want to find a policy that maximize our objective it can be minimize your cost it doesn't matter and subject to task specifications like safety constraints etc and am I there there will be a robot will follow the system dynamics although the dynamics is not known to me when I'm designing these things so how would I how would I solve this problem using what I talked about enforcing part constraints and decomposing in the complex tasks. I gave you an example of something like that fall under this description. Again, that's just an example, but I personally find is a pretty interesting example to describe what we probably want for a future warehouse. So, it's a warehouse delivery task where I have two types of robots. One is a aerial robots. They can fly very fast and they want to revisit the supplier, grab the package, deliver the package. And I have the ground supplier that always just like go to the into the warehouse and re get the packages on their their quadripads. They get the packages on their back. They meet the area robots on the some corridor, let them pick up the package.
9:36 And the safety constraints are a lot. We want them to avoid collisions. We want them we want that the ground suppliers actually they are quite limited to where they can you can travel thinking about like a huge ground robot like or a bus they only can travel on some restricted area. However, the warehouse is not some place we want the drones to fly in. So there's a no-fly zone which is the warehouse and the aerial robots are not supposed to get into it. Now what I'm enforcing here is not a one-time constraint. It's a infinite. So this as long as they are up and running of course I can also put like a battery constraint when they are running low on battery they should get charged or we should fine replace it with another robot that is fully charged they can also be insert here but as long as the warehouse is up and in operation I want all the constraints to be satisfied. That means that my ground supplier will need to constantly revisit the warehouse because there will always be packages coming in. they will have to carry the packages, grab them outside and the air robot whenever there is a delivery need they will have to find the ground supplier pick the package and deliver them. Okay, so that is my example. I will come back to this example and show you how like our approach can give you a learning based policy but strictly satisfy the mission objective delivery and also strictly satisfy all the safety constraints.
11:02 Now coming back to my mathematical abstraction, there are basically two hard problems I need to resolve before I can run some sort of like either classical methods or the learning based approach. The first is how do I enforce the safety and stability as hard constraints in the process and second is how do I handle this complex objective or mission goal in the one I just described like if if I have to say like whenever it comes something there there isn't really a reward function that's at hand for me to directly say this constraint this delivery task and random everything will be enforced forever okay and let's start with the first part how do I enforce this part constraints. Now when I further I further translate this into a more mathematical version of a description and bear me of the symbols they are the the the depth the deepest math I'm going to talk about and after that we'll come back to more higher level description. So if I further translate this into a multi- aent version because eventually we want all these tasks to be scalable and in my toy example is already including multiple robots. They are hierarch hurtogenous they can have different types and each have to accomplish their own mission. And my belief is the way to go to scale up is to go decentralized.
12:26 So we need them to learn a distributed policy. the distributed policies for each agent but all of them together will have to accomplish one goal. So they have to jointly optimize one cost function or maximize one objective function or reward function. They are equivalent here subject to safety constraints. So the safety constraint here I say I have a way of saying all the agents will only have their local observations and when using their local observation there's a function to describe what it mean by safe and here I still assume that safety can be described as some sort of a function over observation although that function can be really flexible it can be like even a a human labeling what you mean by safe or unsafe and then there are system dynamics here it is unknown. Now if I were a RL person solving this problem, the straightforward way to go is to flip my hard constraints as soft constraints to be penalized or using these larger engineering methods like lender duality methods. And the consequence of this is it will change your hard constraint to a soft constraint and you are also ended up with a objective function or reward cost function with multiple pieces. And if you ever tuned a RL method, you will find a difficult time balancing these different coefficients. Some may work, some set of coefficients may work for one thing, but when something changed during the environment, it will not work. So this is a very tedious and there's not really good way to balance the different objectives, especially when there are safety considerations because in many cases safety is strictly conflicting with mission. The best way to be safe is to not move. That's particularly true for the self-driving cars. If you penalize a lot of the collisions, they will just stay still or drive very slowly. Well, as as a result, your performance of getting there quickly or finishing the task will be compromised. So, we don't want to use this soft constraint idea or using this lender duality idea. How would we enforce this hard constraint here? So we have shown that there is a particular con value function called constraint value function. In this particular case this value function is to take the minimum of h over infinite horizon time.
14:53 It is a control barrier function for your system. And for the distributed multi- aent case it is actually a graph control barrier function for your partially observable multi- aent system. It is theoretically proved. If you have never heard about control error function, don't worry. In one minute, I will be an pro of control error function after I tell you what it is. Now, if I really visualize what is this value of the constraint value function. Now, here because it's a value function, we can find a balma equation for it. There are clear ways of solving that Bman equation through either if you have a model the traditional optimal control approach or you don't have a model RL approach to do the value iteration. So if you have this value function if you plot it you can see that either for a multi- aent case for a single agent case there are actually the ground the the the background is a contour of some function that tell you where the robot should move to. So what is a control barrier function then? control barrier function is from control theoretical approach. They are a function that taking a high dimensional input could be our state including the environment and output a scalar value. So in this case it's like a super super simplified case where you want to control this helicopter to avoid the unsafe area which is a volcano area. Then the control barrier function will take in all the state information and output a scalar value that is positive for safe and negative for unsafe values. And the reason it is called a control barrier function is because it actually inform us how to pick the input. In this case, assume that I can control the velocity of this helicopter. Then whatever below this red line is what the CBF, the control barrier function will be able to compute and tell us as long as your velocity is within this range and then you can accomplish your task, then you're guaranteed to be safe or there exist a safe action to keep you safe.
16:50 And of course this is not a fixed function because when environment changes when the environment kind of like have a new area that's windy you want it to get into you need to have another contribarial function that do the same thing in form safe versus unsafe and being able to tell you what is allowed input. Now this is you know now you should be a pro of CBF and what I just said is you don't have to construct this CBF or come up with a CBF. The constraint value function has been theoretically proved to be a CBF and for a multi- aent case it is a graph version of it. Meaning each agent can only will need only to read the surroundings using the local observation and together your entire multi- aent system will have the global contribution for safety. So as long as we are using or enforcing this constraint value function, safety is already embedded with a mathematical object that we can understand. Now let's say my task is to have multiple drones. They are tiny drones but they need to carry a pretty heavy payload. So they cannot do it by themsel but they have to work together.
17:57 Now these things the the safety considerations would be they'll be able to move things around. So avoid obstacles and also all of these drones they cannot be too far away from each other. They cannot be too close to each other. They have to maintain a safe distance and also trying to avoid the wind disturbance caused by each other. And if I enforce the constraint value function in this process and I if I compute the control value the constraint value function from the plot here this on the right it is the value of the constraint value function which is the GCBF for the agent with a star. You can see that it is oh this is a flipped but it's like it can clearly satisfy safe from unsafe and we know why this policy is not chosen to in work in this way to accomplish automation and also keep everything safe because we designed everything to be in a decentralized way it can generalize. So in training we only did three, four, five agent transportation of the same object. It can work seamlessly on six agents with like a different object.
19:01 They can still move things around in a safe manner. It can really scale up because theoretically there's no limit. We in simulation we have tried to simulated this delivery task with thousands of agents only training with eight of them and they never saw the obstacles in training because they are they have their local observation they can use camera or they can use lighter to see around and because of the GCBF that's embedded in the process training with eight agents theoretically you can go up to arbitrary large number of them but here we tested up to like a thousand of them and you can see we remain the state-of-the-art safety and task successful rate. This can have a can be applied to higher dimensional cases where we have here there's like the inspection robot where two of them have to work together to make sure the target is always being observed within a certain inspection region. bottom here we have a two F-16 they are high fidelity the full scale F-16 airplanes trying to accomplish different tasks in this case if you ever watched the top gun one it is the one there you Maverick was supposed to follow their their teammate but because they got too close there was an accident so there's a safe region to follow it but do not get too close to the jet or snow we don't have a version the top gun too I don't have the video here and all different type of tasks Because of the safety barrier function as a abstraction of robustness, it can tolerate a lot of gap or inaccuracy in the model. So we actually train everything in simulation.
20:42 We never collect real data to do training. Everything was trained in simulation. Of course, the model will not be accurate. But when we deploy them on hardware, the robustness that we're able to build will be able to handle like the wind disturbance or other adversarials here there these robots will be able to navigate in the environment taking off and landing on the on the moving platform etc. accomplishing all these tasks. So that is the first first piece. we have talked about how to enforce safety stability at hard constraints. The way we do it is to figure out the right ab representation of a certificate and knowing how it exist in the machine learning process. So you can now you can guide your machine learning process make use of all the powerful RL approaches but being able to enforce all the hard constraints while you actually as a product this value function you can have them at hand as a safety certificate for your system. You can use it for another system if you need. But I haven't really talked about the complex objects. I kind of like by default said there's a way of describing the mission goal. But if we want to think about our original delivery task, it's still not really solved. How to how do we understand these very complex objectives like do we use classical control? They there in classical control there are languages we can use to describe these tasks like temporal logic. But we know that these classical approaches cannot go with high dimensional case. Do we use directly standard RL or MPC that require a lot of reward or cost tuning which means a lot of dollar amount you have to spend on the tuning and it may po pose some misalignment in terms of what you really want to do as with respect to your what your reward function is describing.
22:35 Well, I know a lot of you work on VAS or large models. Do we use large models? Yes, you could, but you need data, right? Like a lot of the missions, I don't even have data. What if you don't have data at all? How should I start? Then I at least need have something to generate the data for me. So our idea is continuing this idea of understanding what is the mathematical object we are trying to achieve in this whole training process or learning process. It is a value. So can we decompose this value here? Now, now coming back to my toy example, I actually describe this this whole mission as this thing called temporal logic. If you never heard about it, it's okay. It's a systematic symbolic language to describe a lot of different missions that can be infinite horizon that can have timing, temporal and logical dependencies. It can describe recurrent things like things that have to happen forever like your whenever there are package it has to be delivered things like this. And this can precisely describe my tasks like the agent need to revisit the locations, the ground suppliers need to revisit the supply, go to the warehouse, there are safety, there's no fly zone, etc. And everything is here to be precisely described as temporal logic. Now you may wonder, I don't know how to write it down. Well, luckily you don't have to.
23:55 GPT6 can do a perfect job for you to give you a a perfect temporal logic. three years ago I have a work on like how to use larger language models to translate it from English to temporal logic and because at that time JPT 3.5 was not able to do a perfect job but at that time I already knew that that that word probably wouldn't have any long-term meaning because the newer generation of the model would be more powerful and now I think you can pretty confidently ask large language model to do the translation for you now assume that we have this tempor ologic description of my mission already. What do we do? The way we approach is called value decomposition. It's making use of the fact that the temporal logic language is a rigorous system language with its clear grammar. It has syntactic it has syntactic syntactic and semantic meanings and they they follow all the grammar rules. So we can look at all the grammar that is used to construct this temporal logic and tell you how to decompose it. So we have proved that any temporal logic tasks can be basically either like we can of course the decomposition will happen recursively up to some atomic proposition but they were eventually some combinations of multiple ritual avoids or multiple recurrent tasks.
25:18 Eventually they will be decomposed to this fundamental tasks. So rich avoid meaning you want to reach some goal while avoiding certain things. Recurren means something will have to repeatedly happen and eventually any temporal logic task will be able to be decomposed in that way. Let me give you an example. Let's say I want you to reach a reach b never hit the wall. Now from the original description there's a using the value decomposition rule we can translate this into a disjunction of two because they are disjunction. If anything is disjunction one of them happening is fine. So you can do a parallel computation for both. Of course they are not the end and each of these child node will have to according use the rule to translate them into the the more bottom level of the the descriptions. But eventually every temporal logic is going to be a graph of certain rules like reach avoid until avoid or globally or globally until this repeated roots. Basically every value eventually the child node are atomic HJ values with nonboman equations. That means every child of them can be learned or can be searched depending on your scenario but every one of them have a bioman equation. You can use any of your favorite RL algorithm to solve them. And together all of these atomic values will form a decomposed value graph. Now once we have those graph the node the nodes we can actually learn them simultaneously because they are all completely in parallel. Now we can learn them simultaneously and the implicit value relationship are bootstrapped across the different nodes. In execution they will just read the graph and execute the policy and there we have the guarantee that your reach A reach B always avoid will be always forever satisfied. Using this way we'll be able to handle very interesting tasks in simulation and that can scale pretty well. Here we have the agents trying to visit a number of tasks, number of targets and they can go up to like a hundred with very high success rates across many different tasks. They remain a very high success rate. Now let me give you two examples I personally find very interesting. We have talked about delivery. I will show how it looks. But there is also a hering thing.
27:45 Thinking about our quadripad as real dogs to do hering example. So we have the sheep in this case. They are still drones. We we cannot have the real sheep in our lab. They are not controlled. All they do is run away from dogs. And we have the bigger dogs. One of them is a ground robot. Is a slower dog. One of them is a faster drone. It's a faster dog. Their goal is to just push the sheep all the way through the gates and get back to their pen and stay there forever. So we can see the the the active constraint here are highlighted.
28:20 You can see how they are actually working seamlessly together. You need a distributed policy here to squish everyone back in the pen which is the this circle here. And because they are looking at from different direction they keep the ship perfectly within the pen and you can repeat the test many times. They're all very successful. Now coming to my hardware task. we have the bigger ground robot here going into the warehouse area and we have the smaller drones. They have to constantly meet the bigger guy, land and pick things up and take off from there, deliver packages to their goals and this will continue forever. And we also you know cross test across many different configurations and parameters and show this is very robust to the changes in the environment. Another example I would like to mention is self-driving because it has been proved that every single traffic rule can be perfectly described by a temporal logic specification. There are papers telling you how to translate every single traffic rule into a temporal logic formula. Now now we can what we can do is we can actually train these self-driving algorithms together with the hard constraints. These are traffic rules. We can just you know use one car train them on different road segments. As a result they really learn all of them learn to follow the traffic rule. There will be stop sign there will be traffic lights and believe it or not stop sign is actually a pretty complicated rule to enforce. You have to stop. have to look at all your surrounding and decide when you are safe to pass. But you enforce this rule on each of them. They collaboratively know how to drive. We can give different level of you know different behaviors.
30:06 It can model aggressive behaviors, conservative behaviors or behaviors that you know other surrounding agents that do not strictly follow the same rule. But our agents will always follow the rule and they learn and they know which rule to look up to when it comes to different traffic scenarios. Something interesting is they can actually adapt to things they have never seen before. We never use a traffic circle to train these agents but they learned because of the traffic rule of rear end collision and some sort of like the stop sign they learned how to cooperate around traffic circles.
30:42 So that is the conclusion of like the the the last this is the last piece of my first part. So we talked about how to enforce all these mission specifications, mission requirements, safety, stability as hard constraints and formulate them as clear value functions and how to handle them as very complex value functions to decompose as something everything that is solvable because we know every one of them exists a a known biomman value function. So as a result we are able to finish the task with a guarantee and satisfy safety with a guarantee with a clear mathematical object to look up to. Now then you may ask is that the end of the story? in terms of providing some sort of guarantee we can have let's say control barrier functions leapon functions temporal logic satisfaction to show they do satisfy the satisfy whatever we want them to do. However, it's not the end of the story because no matter how hard we try, failures will happen. I don't think there is a system that never fail. And our approach to to look at the to look at how to build a safe autonomy or guarantee safe autonomy from another perspective is to understand when they fail, what has caused the failure. So how to formulate this problem? Now if I need a if I need use another mathematical object to describe the problem we're given this system it could be a subsystem it could be a system of subsystem doesn't matter you could have a simulator you could have a hardware but eventually what the system is do is they take in two set of parameters design parameter these are the parameters we actually we can tune we can change the value and we design them for example using the approach that I just talked about and The goal is to m the goal is to find the design parameter to minimize your cost. And there's another set of parameters called exogenous parameters or external parameters. These are things like the teran, the collecting, the weather, other human who do not want to listen to you. They can be wild. We can enforce all the constraints into the system when we know how to describe them. But there's no way to force safety with respect to someone who wouldn't listen to your safety rules. Like if someone want to break my all the specification there's no way for me to ensure it. So these are the extortionous parameters.
33:10 So my system will be have like I would assume that there exist a j which is my cost function. It it could be really broad. It could be my goal, my cost, my constraint, the limits etc. Then the design is to find the design parameter to minimize this cost while testing is the problem of finding the exogenous parameters to break the cost or have a high cost. So this high cost could be violating a safety constraint or breaking a certain limit not finishing your goal. And here the the two perspective is they are really looking at two set of parameters. When we design, we assume what could happen at the exogenous parameter. But our assumption will never be perfect. And the testing is looking at given a design, how to find the parameters that I ignore that could cause a failure.
34:03 Okay, I hope that that is clear. Now the second part is about testing. There are many approaches to do testing. Other people call it verification. So VMV, if you ever heard about name, it's called verification and validation. TE is testing and evaluation. These are key methods engineer use to understand the performance of their system. And when I look at the literature of the broad VMV and TNE problem, they are really trading a different considerations robustness, flexibility and efficiency. If we are looking at the traditional formal methods like the reachability the optimization based approaches they are very robust robust in the sense that they do give you a precise over approximation of the behavior of the system. So if we can use these approaches to prove our systems safe, they're pretty safe robustly. However, they are not very flexible because they only can be used on a limited set of system typically like a small scale polomial dynamics. They cannot work on blackbox systems and they all often computitionally very expensive. And when I go to blackbox learning and testing, this is normally what the large model people like do testing in the blackbox way trying to use RL to figure out adversarial behavior or blackbox optimization etc. So they are on the other side they are very flexible because it's not making any assumption on the system but in terms of robustness it is less because these are best effort approaches meaning if I use approach to find a failure or find the case that doesn't work I may be lucky but if I don't find anything there's not too much I can say about the robustness of my own system and there are of are domain specific tools. I call it white box learning or testing because they use domain specific simulators like people develop all different type of simulators. A lot of them are made to be differentiable. So we can deploy a lot of differentiable approaches to accelerate the process. They are making a good tradeoff but our goal is here our goal is to improve flexibility. We want to generate the right text test case for very complex blackbox systems without model assumption of it. I said a quote unquote blackbox is is because eventually if you have a simulator it's not really a black box. And we want to improve efficiency in terms of generating high coverage test causes test cases at a low cost. And we want to improve robustness in terms of being able to say we can quantify the risk. we can diagonal diagonalize the failures instead of saying I found something I'm lucky if I don't I would just keep trying. Well, if you don't have a robustness, you ended up driving a self-driving car forever on the road hoping that you will see a a tricky case. And now with that objective, there are different levels of failure that we have looked at so far. The first level is called distributional. We want to answer the question of where does it fail? So that produced a failure distribution. I will talk about a few approaches we have taken to answer the distributional question. The second is level is reality. It failed in simulation or it failed on this lower fidelity system. Does it really fail in real because there are simulation seem to real gap or sim to real discrepancy.
37:41 The third level is what we call contextual failures. It answer the question of what counts as failure because it's actually a pretty hard problem if your robot is not performing as you thought like what do you mean by failure to some people failure means they're not finishing their task other people said failure means they bump to something they're not supposed to so what call that failure what is a failure criteria and the next level is looking at a group everyone have their own understanding of what they want what they don't want they how would we calibrate who is valued to decide on what should be tested, what should be the performance of your system. It's called a normative level of failure. And finally, it's called retrospective failure because no matter what we do, believe it or not, your system will fail. No matter what we do, when failure happened, we want to understand what has caused it. What is the hidden factor that I can monitor next time before an actual failure happen again? I can prevent it or maybe I can use this idea this hidden factor to generate more test cases and due to time limit I will be not be able to tell you everything we have done but I I hope I can show you a little bit of what we have done to understand the distributional failure what is the contextual failure and the retrospective failure and for others again if you are interested in it I'd be happy to spend more time offline the first is where does a system fail now Coming back to my mathematical description, I pretty much answer the question where does the system fail? We find the exogenous parameter Y that maximize my cost. That's how they fail.
39:23 Then the natural thought is can I just solve this optimization problem to find a failure. So why is testing not simply solving an optimization problem? The fundamental reason is it hard it is very difficult to find the different failure modes. Failure is a multiode case. failure could be caused by many different reasons and they they can appear in different ways and if I look at this optimization problem to generate test cases I'm really having a non-convex like a super non-convex with multiple optima if I use an optimization based approach even if there are hyper heristics to help you get rid of the getting stuck at local optima you will eventually get this what we call the region of attraction meaning where you start to test will always land to that particular ular failure and you will be bland about the other failures that could happen. And we propose that maybe we can reccast this failure discovery problem or testing problem from an optimization problem to a sampling problem because sampling has a unique strength of not trapping at the local optimal. So if I look at the same toy example where if I use the sampling based approach the region of attraction is really messed up because wherever you start with a certain probability you will land at one of the optima. So that's the hope. What do I mean by testing as a probabilistic evidence generation process or sampling process.
40:50 We want to reccast the search for failure as a sampling from a posterior distribution. So what is this probability distribution? is a posterior distribution that identify the region of the parameter where failure most likely going to happen. And if you work on inference, you know that posterior depend on prior and likelihood. So what is a prior? Prior represent the designer's initial belief of what regions are to search whether they are safe or risky. And likelihood is the s is the data collection process. we can quantify how likely a specific set of parameters is going to cause a system failure. Now so going from there where what should we do? Let's say that suppose that we do have a clear cost function that J remember that J describe my task goal my mission my physical limits my constraints and etc. If I do have a J then I can already approximate that likelihood. So that likelihood using a very standard approach is can be approximated as this exponential function using the the J. So in that way what we're doing is starting from a prior trying to go through a sequence of MCMC sampling to get the higher probability and the higher cost failures of a posterior. We don't just want to rely on the belief original belief but being able to sample from the high cost of failure. And this can be done easily using MCMC sampling where you if you have a good simulator with gradient information, there are versions to accelerate the process using the gradient. If there's no gradient information, there are also versions of the sampling of the model free versions we can use to do the sample. So if we do this approach, we have tested across a cluster of different applications with so X is the design parameter, Y is the exogenous parameter, they can go up to like thousands or sometimes close to a quarter million dimensions and able to see that our method can achieve a 10 times lower cost compared to other approaches to generate the diverse failure cases. And if I if we repair the system using the failures we have generated, it can give you a five time improvement in terms of the system robustness. If you give you a more concrete example here, so one testing question would be let's suppose you are designing a racing car. This is a racing car using a sampling based controller.
43:31 So you know that sampling based controller there are a stoastic controller. There's no way to get the gradient information back propagation to do any type of optimization based sampling. But we want to ask the question of where does a obstacle will cause a collision of your well-designed sampling based controller. And using this approach, we not only tell you the region the corresponding probability, we can also design the different cases. For example, in this right case, there are two obstacles. And because the car was trying to avoid the first obstacle, it caused an in unavoidable clash to the wreck for the second obstacle. And each of these position now comes with a probability of failure. And overall, we can use this to quantify whatever the risk you care about the racing algorithm. However, is this the end of the story? A lot of people will immediately say that well, I don't know a J, right? There are a lot of problem there, a lot of issues.
44:32 So far we have assumed that we are solving this problem through sampling through the posterior. There are already a lot of problems. We don't know the prior. What if the prior is not accurate? what if I don't have a complete knowledge of the graph in the sense that there's a significant same to real gap and collecting failure sample itself is a very expensive process for physical systems and more importantly I don't even know how to write down the cost function that's a very typical case we met when we work with real world like real world problems or working with engineers it would be like there the system design is a very very complex document there's not a single J function I can optimize or even sample against for example for a mobile manipulator what balance the task success the safety clearance the handling asking for help all these things how do I even define a failure out of it and not to say every every person who participate in this approach may have their own priority in terms of what they want to see and the reason we want to turn this test case generation from a search process to inference process is because classical testing assume that we already know what constitute as a failure meaning we have a clear description of J and it will ask for what we call an contra example or witness what you're ending up with is a fixed set of things like today's regular racing process it give you a fixed set of cases you have to run and if you pass you say the system is robust but we know passing those tests does not mean that you are guaranteed to be Instead an inference process or inference based testing is going to treat the search of failure as a sampling from a posterior distribution where the sampling itself should condition on what you know about the system what you know about the failure.
46:28 It will output instead of witnesses it will output a risk landscape and using the risk landscape we have a better idea of where the failures concentrate what have cost them. So to address this challenge what we are proposing is an adaptive scenario generation framework where this really system is working in the loop to actively and sequentially generate what we call the maximum informative test cases. So we have a system and you can query it. It it's a simulator. It doesn't have to be a robot. It can be a large model. I can show you examples of like just testing the large language model. And when once you send a query or simulate the system, you get an outcome. It could be a trajectory, it could be an answer, it could be anything. And we'll be able to provide assessment and this assessment is how you evaluate the outcome. It can be objective, it can be subjective.
47:23 People can express their own opinions about what I mean, what I like about safe, what I mean by risk. And then this assessment will be used to fit a failure what we call failure prediction model probability of the of the risk given this particular exogenous parameter combination. The risk prediction model will be used to generate the next most informative test case and this approach is called active inference. Now more detail because we don't want to be constrained to a prior.
48:00 It's very likely that we don't have a good prior to overcome that we can use variational inference. I won't go have time to go deep into the varational inference but you can look it up. It's a very powerful approach to get rid of this approximation of a prior and to generate the cases what we need is to use our risk model to design a right acquisition function. And this acquisition function should come with two part. One is called information gain. The second part is called user utility. And I will show you why you need both. Information gain is to reduce what you call uncertainty about understanding the failure of the system and user utility is what as user they want to see and what they want guide as failure cases. If I summarize, this is really a loop where we do test, assess, learn, and generate in the loop until we get what we want to see where the next test is always selected to optimize the information gain and mission utility with a goal of turning from very limited evidence to deployable evidence. The limited evidence could be your simulator, your historical failure, your limited hardware trials and or your objective subjective assessment to deployable evidence of answering where can the system fail? Does it fail in reality? Does it fail persist in reality? What count as a failure?
49:29 Whose value? And what has caused the failure? So this is a loop to answer these questions and I will give you maybe a few example to show why and that is the contextual failure. So as I said failure in many cases are very very subjective and we don't even know how to clearly write down an function to describe if I feed in this case it can tell me if it is a failure or not. for collisions is clear but for example do I optimize securely grasping something or have the potential of damaging that object. So these type of failures are very difficult to anal analytically write down because the reasons could be multiple and the knowledge of where it rise could be from many different reasons. To overcome this actually we use the same remember our loop right in now we have the testers or the engineers as the one who provide assessment and this in reality could be a person and could be large models. Actually in all of our work we replace that human with large models because we think they are pretty good at providing assessment about if this is a failure they want to see or if you there are particular things about the failure they want to talk about. Using this this evaluation or assessment, we'll be able to train surrogate models for the different failure modes. And when we design, what we do is to optimize the coverage of all possible failure modes across different severity level. Right now, we want to generate cases failure cases not only caused by a single failure, but could be from multiple failure sources or causes. and each of them could contribute to it in a different way and this will be give us more experiments or more data used for further evaluation. That's a whole process. Using this we can generate cases about detection like detection failure cause some collisions but we can get to know maybe this is a lighting condition or the noise or the misinterpretation due to distance caused at the perception level. We can consistently generate test cases across these different modes and could be combining different modes or self-driving case. I can give you a few examples that are I find interesting. So we can learn from historical failures.
51:58 So all of these failures they are real failures caused by different reasons and we can actually learn and generate a case that combine the information from five actual failures but they were they never happened in real world. they will fail most of the self-driving algorithms that that available today and like this is a generated test case that we learned from historical case. This can help us understand the real cost of the system and more importantly we can do more than just than the robotics testing. So here we take a look at large language model. In this particular case, we are looking at the problem where large models are suggesting actions after a disaster. So they will read tweets, they will read satellite images and suggest what you should do to mitigate the disaster because there are limited resources or limited things you can do as a disaster response. And here we actually do not really call it a failure. We call that difficult. When can we trust a LM output? We can we not trust it. And the utility here is a reported difficulty with the external assessment of hardness. For example, we can generate hard decisions and easy decisions. Easy decisions will be the low difficult case with high confidence.
53:20 This is because these type of tweets provide a lot of relevant information and LLM is very confident in terms of getting to output the actions to get mid of the disaster. And we also have the difficult case where it's highly difficult, low confidence because the information show that rescue actions is really needed but it's not clear which is the actionable thing that should be done. We can read the pre- disaster and post- disaster satellite images and figure out both low difficult and high confidence cases. This is particularly from the hurricane Florence case where the pre and post disaster images can provide a high confidence on action suggestions which are the suggestions in purple and we can have the opposite and this case could be selected from a real data set or we can artificially generate these cases. We can generate pre and post satellite images to tell you this will really stress test your LLM and in this case you shouldn't trust what LLM is going to say. We can use this to understand the different modality different actions. For example, we see that like GPT 5 I think 5.2 actually as the more advanced model they're not really trustworthy because the actions can be very sensitive to the input. they spread out everywhere versus GPT 4.1 meaning you can see their their suggestions are more have a lower variance and they consistently provide the same type of suggestions given the different type of input and I think I'm really running short on time I just use one minute to finish this up when it comes to retrospective failure we are looking at the cases where you we don't have any access to run the system but there are failures for example something I feel most of us experience that and get suffered with is flight cancellation delays like for no reason my flight just got delayed and for a very long time I don't even know who made the decision to cancel this flight. So we were looking at this particular case of 2022 Southwest scheduling crisis where if you were here in the US you probably heard more than thousands of flight got cancelled during the Christmas week. two million people got stranded at the airport and it all started from a snow winter storm happened at Denver but somehow within three four days Southwest has to cancel every single part of their their their flight across the US and there's as of today no one was claiming responsibility southwest would say it's the winter storm we were delayed by deicing but it's it can't be true because only Denver had a winter storm other where else it was completely okay the swapa there are their pilot association have other complaints about their leadership where CNN will complain about network so what is the root cause here a we are actually deploying exactly the same process but here the difficult is we don't really have a model we don't even have a simulator we only have data I mean maybe I I won't have time to go to the details but maybe you're curious about what actually has caused the southwest constellation we identify that the one important hidden factor here is the airplane reserve.
56:45 It's a number of airplane each airport reserve overnight for their flight next day. And we can see that although the winter storm indeed disrupted Denver, this is a Denver airplane reserve short like shortage compared to the required and by the way all the value here are infer because they were never public information. We can see that the airplane reserve actually recovered pretty quickly after the winter storm and Denver. However, other headquarters, a big airports like Las Vegas and Dallas, they were not infected by the winter storm, but their airplane reserve continues to get worse over the week.
57:24 And that is a very obvious at least factor we can keep observing. So here we can actually plot over that week what's the relative airplane reserve and the amount of cancellations that happening during the during the week across different cities and clearly you can see Denver got recovered pretty quickly but Las Vegas and Dallas got pretty significantly impacted by by the whole scheduling process. So to summarize, we talked about building safe and intelligent autonomy from two complimentary aspects. Safe and mission satisfaction by design versus generating evidence through active testing. And they are looking at different set of the parameters of your system and the environment. And I think that they together can bring some of the rigorous mathematical methodology from our classical approach into using datadriven approaches and enable us to to build systems at large scale that are safe by design, scalable by architecture and reliable by proof. And that's all I want to talk about today. I know I'm running a little longer and I be I will stay around for the after seminar event. If you want to discuss anything I'm happy to thank you for the talk. Maybe we have make time for one or two questions quick questions. Yeah.
58:53 >> Yeah. Thank you very much. >> Thank you very much for the great talk. I just want to ask that you mentioned that safety constraints cannot be like compromised like it should not be soft or should be hard but sometimes like under like extreme cases for example like like we may these kind of safety constraints may be contradictory to each other >> like for example like in the for example in the driving scenario like one car may just go to your lane so that you may need to go to the opposite lane order to avoid the car that is of course like not hold to the safety constraint but you need to do that in order to avoid the car. So I just wonder like under this kind of cases like the safety constraints cannot be like hold all the all of them and we have to sacrifice some of them like how would you deal with this kind of situations? Yeah, that's a very good question about like you know there are there are cases first of all when we say the safety are hard constraints there are hard constraints on whatever you define as safety right like as I said why the reason we need testing is there are always cases we cannot imagine at the design phase so we have to understand when they fail because certain certain assumptions we make on the design phase got break broke so when we enforce a hard constraint they can only enforce the ones that we kind of like describe as hard constraints they will be satisfied but in your in your case it will be the adversarial that will just cause you know for collision avoidance a simple case would be you have adversarial traveling at the light speed to to hit you there's no way for you to get rid of it so that would be I would say like that's why we were also looking at the other perspective like what we want to fight figure out that adversary maybe they exist maybe they not but what like There will be domains we can constrain on but like in the particular case for example if your safety constraint is strictly collision avoidance then collision avoidance will be respected but if you didn't if you did put stay in the len as a strictly safe constraint it will be there and in this particular case collision is not avoidable at all. Well, maybe through testing we figure out this could be a case. We could, you know, relax some some of it like staying in the land as a secondary safety consideration that can be relaxed when they're strictly not possible to satisfy both and that we can encourage some you know behavior that that may avoid violate certain constraints to satisfy other more higher priority constraints. So those are I guess design choices but mathematically if you put it there enforce it there they will be they will be kind of like yielding our approach will be kind of like you know encouraged and to some extent enforced during the dur in the policy and you know for those adversarials that cannot be avoided they cannot be avoided.