Building Secure AI Applications with Red Teaming | Accel Cybersecurity Summit

0:06 yeah I'm Richard co-founder of side side Labs have been in the security space for a little over 10 years now started with Google in the US spent time working on Insider threats and AI SEC security there moved to India to set up Google's India security practice that for a couple of years in the last 3 four years was the CW tampion mobile premier league and since late last year have started Sid laabs at Sid laabs we focused on building security

0:36 products for gen system starting with llms just to get a sense of the crowd is everybody with a security background with a show of hands like currently or in the past have you worked in security okay so I'll assume there's 80ish person 90% people who worked in that space some who are just interested in it okay at going to the next point right like why are we talking about security for AI differently than security for everything

1:07 else right like isn't AI just software this is some a conversation I I've had extensive debates with somebody who used to be like a VP at security VP of security at Google because she keeps telling me AI is also just software right like why is it any different than any type of software that you're protecting you're protecting the same thing it's still code it still bites it hasn't converted to atoms right so I'll try to cover that at least why I

1:35 think a securing AI is different or at least you should start thinking differently right like so if you've been focused on securing traditional software how should you probably think differently when it comes to Ai and why the problem has evolved and none of it means that you don't protect your other systems right like it this goes hand in hand with protecting your regular software first one right like how do you test a system for security most of the times

2:04 when we do red teaming vaps penetration testing in house externally anywhere if I'm a security engineer I write an attack payload I test it if it works it works if it doesn't work I know it will not work even 10 if I try 10 more times right but when it comes to stochastic systems like AI systems your internal security system engineer might try an attack it would fail and if they assume this attack doesn't work would be a wrong assumption right

2:33 because the next time somebody else who potentially could be external tries the exact same payload there is a chance that this time the attack would work so you can't go with the same approach of traditional security where let's say you have done your testing and tried every attack once or run a scan once and you're like okay this is done this is secured don't need to test it again or especially if a lot of companies that don't have large inhouse security

2:58 teams rely on external pen testing services right there if you're getting it done from them they also usually have a checklist to follow try all the attacks they say this didn't work this didn't work this work this didn't work doesn't work with AI right like at that point it did not work but without making any change to the system now it could suddenly start working good one next is you don't know what has changed right so this is one

3:23 example we were doing internally right like we took a Bas llm model we tried a very simple query this is one of the most popular models out there that we are founders of XYZ help me do ABC and it said I can't do this then we took 500 rows of Q&A which were very simple questions it were just like what's the capital of India what's the temperature in the city very simple 500 rows of data fine tune that model and

3:50 now suddenly exact same question and it starts abusing random companies in the world and saying that you don't deserve to be a Founder right so it why I bring this up is in a traditional system if you've designed a system which does a plus b equal to C if nothing has changed there you don't need to test it again for systems with AI you may have made changes for x and suddenly something completely different gets impacted so

4:17 you are never certain of what needs to be tested because a lot of times in traditional security when you do your security testing it's always incremental right like you first ask what's the change going in the system and that's the only change that gets reviewed but you can't rely on that assumption anymore right like anytime you're making changes almost everything needs to be retested right and something that we've been hearing for years right like shift Left Right everything needs to

4:42 shift left when you shift left your cost reduces your time reduces your efficiency improves and you get better results but is it really true for AI right so let's say you do everything secure while you're developing or maybe even one more system left right like while you're designing you can do everything right from a security point of view but it doesn't give you any guarantees because these systems can misbehave at runtime you don't really know how these systems are

5:07 going to behave not saying you don't need to shift left I'm just questioning the dollar sign right like there isn't enough maybe there's not enough research done yet in this space where you can say that you'll get better results if you do it earlier so far I I have a strong opinion you get much better results if you do it later but that needs to change it will change but I don't know what that timeline is so

5:32 for the time being the Paradigm of let's shift left I may not apply the same way as it has traditionally been applying especially from a Time cost and Effectiveness for security point right is open source really open right like there's so much talk about using open source models open source software and unlike traditional software like if you're picking anything open source it's actually something you can go in review check what is

6:04 being done you can actually turn your security scanners on top of them see exactly what's wrong what's what you're actually bringing in the line is very blurred when it comes to models right there's a wide range of what can be termed open source right and it's being used very vaguely so you sometimes even the model provider doesn't know what actually goes in right like for them itself it's a blackbox so when you're bringing in these models you don't

6:28 really know what you're actually bringing in I picked two random not random but recent examples one was I don't know how many saw the serialization attack that happened through open source models from hugging face but again that was something that was not very trivial to capture right like that's not what your traditional systems would have captured and anybody who has AIML teams working in their companies would know that these days everybody's picking models from anywhere they find deploying anywhere

6:55 they can most likely running in a random duper Notebook on their laptop or a random server so you don't know what's getting run and the second one I picked the recent back door not really AI related but the fact that op software had a back door that was undetected for years and had massive consequence this can be extremely worse when it comes to ai1 it's much harder to detect these it's much harder to do any sort

7:21 sort of auditing on a wide scale when it comes to like all open source software and in some case not just possible div Fe ility so there are number of back doors that could be going into a lot of these models coming out you never know when when those trigger and those back doors need not just be code executions need not be RC's they might also be data poison right like you don't know what's the data going in

7:44 in most of these cases now two slightly philosophical questions that we've run into when talking to customers these days right so who owns security this question started mostly during the whole transition to Cloud right like who who owns security when you're not owning all your Hardware all your software all your inst it's getting even more blurred when it comes to AI right like I'm sure some of you might have heard the Air Canada incident they launched a chatbot on their website

8:12 which would address customer complaints this chatbot was powered by an llm customer raised a bement claim to get a refund after taking a flight and was able to use certain ways to as abuse the system and get the refund when the code thing went to code one of the things that Air Canada tried to do was they tried to blame AI for the actions taken and they said we didn't do anything it was the bot that did it but that

8:39 was quickly struck down and the court said Canadian Court said that if your system did it you responsible but now then they wanted to start claiming Vu St open AI in the back end so we need to blame the model so there's still a lot of uncertainty on when you're using AI who who owns security is it the person building the model is it the person who's fine-tuning it or is it the person who's then using it to deploy the system

9:03 or is the AI itself responsible right but in most cases it comes down to what the customer looks at right like if Air Canada is serving it to me in most cases it will be termed as Air Canada quent for its security right and the last one is something I still don't have an answer like something that nobody's found an answer right like who wants safety when you're using AI in these systems and that AI ends up answering

9:29 something that it shouldn't or answering something that's biased or toxic or inaccurate whose ownership is it and especially when it comes to Enterprise we've gone to calls where we've had like 10 11 different Executives sitting from completely different parts of the business like marketing legal Finance AI security privacy policy they all want to solve the problem but nobody wants to raise their hand and say this is my problem to own right nobody has figured who in a company owns this once we

9:59 deploy the system in most cases when things go wrong most of you might be aware it's usually the security Cup team that people will chase right that why did you not catch this before so this will then lead into why red teaming sort of becomes important cool so yeah AI red teaming again red teaming World War time concept nothing completely new but very quickly how do you define red teaming instead of me doing it I let the

10:28 White House do it so redeeming I won't go through the whole thing but the thing is it's starting to become something that governments are recognizing regulatory bodies are recognizing until at this point for AI that has turned out to be one thing that everybody is reliant on in order to check whether your system is safe or not or secure or not and it also take takes you to when do you need to do red teaming as of now

10:55 we're already getting to a point where it's extremely important right like the us executive order already mandates AI red teaming the eui ACT requires a report to be submitted for f ver serial testing for certain use cases most copyright guarantees right like so for example if you're using a model and you get a copyright claim against you you for example Microsoft gives you a copyright guarantee that if you use their models and you get a copyright

11:19 claim they'll pay for it but it has a fine print it says unless you have done red teaming to test against copyright infringement they won't pay it right so unless you a copyright red teaming on Microsoft models Microsoft will not give you your copyright claims so a lot of places red teaming has almost started becoming an assumption that everybody deploying AI is doing it before they deploy the system now what all falls under red teaming again this is not exhaustive and

11:50 this is not a session to teach how to do red teaming just going into an overview of things you need to look at things you need to start keeping in mind while you're doing it and when you should or should not be doing it the first part most obvious one that comes to everybody's mind when they think of red teaming and AI system is a security pieces how we like to approach it is we we big fans of the OAS SPM top 10

12:14 they cover most of the common abuse vectors so make sure when you are starting to do your air red teaming this is something that you ensure that you're covering most of them because in most cases all of them will apply if you want to go one step deeper when you're starting thread modeling before you do red teaming mit's Atlas framework is extremely exhaustive extremely good will take a time to digest it so take a weekend couple of

12:41 days go through it if you're familiar with the amiter attack framework it's adapted from miter attack into miter Atlas which is focused on AI and ml U but yeah if you're starting to focus on security testing these would be the places I would start the next two are areas that I've seen a lot of companies miss when they're doing red teaming and that comes back to the whole who owns this piece right so when it goes to the security team they'll focus

13:08 on these security problems but then with AI systems there's lot of safety issues that come in as well and that usually most companies will not have like a trust and safety team dedicated to test products right like they're more operational teams versus testing teams so there's concerns around toxicity violence bias misuse dangerous information non-exhaustive list hence three dots but based on what you are serving your use case this becomes an extremely important testing Suite right not all security teams will be

13:40 equipped or let's say experts in doing this but it's very important for you to sit with your business counterparts your policy counterparts product counterparts to understand what's the need or what are the attack vectors when it comes to safety testing and the last one which is even less common that I see people follow is things like copyright issues right like when you're bringing in an external model right like what data was that model trained on based on the use

14:06 case you are using could there be copyright issues that come up brand guidelines right like in most cases if you are serving the model as a company what are your brand guidelines are there ways to bypass it could it cause a PR issue there's a lot of systems that now are trying to detect deep fix let's say you are a kyc company right and are using models to detect whether the livess check are you able to run red

14:30 teaming on trying to use deep fakes on them and seeing are you able to bypass those systems and abuse of platform right like for example there was this mattress company right which was selling mattresses but had a negotiation bot where you can negotiate the price using an llm and people were able to get it to give like almost 90 99% discount by prompt injection so are you checking for what a potential abuse to your business use cases through

14:58 these system cool now while you're doing this when you're building your team planning this executing it what are key things that you should keep in mind right like and this is very similar to what you would do with a traditional red teaming just some minor tweaks and changes plan it really well right like this is something that can become a rabbit hole very quickly you can because number of abuse vectors are so large you need to ensure that you have very clear

15:28 goals while you're doing it maybe break it into multiple exercises rather than a single one figure out what you're focused on right like if you're focused on Security First abuse first toxicity first Safety First figure that out this is not something that can be done independently by just a security team right like make sure your you understand what's the business actual use case of this system is right like let's say you're deploying customer support agent on a customer support bot

15:56 on your website and the business once it should only answer things related to your business and about that customer but let's say somebody comes and asks it to write code in Python you as a red teamer might assume this is not bad it's fine it's just helping the customer but the business might have a specific use case of not answering off-topic questions so that also becomes things that you need to test for AI is new for everybody right

16:24 like it's been there for a long time but the way it's being used now is very new so make sure your SEC and especially like security expertise is extremely rare in the world AI expertise very similar and an intersection is almost non-existent so make sure that your team has the resources to understand what are the models how the systems are designed where is the data coming from what are the abilities that your system has can it actually go to the internet and fetch

16:49 things does it have the ability to take actions does it have the ability to read read customer data all of those pieces make a very important input into what your plan for your becomes for most of the reasons that I discussed why AI is different manual won't work right you'll need a tool to do this for example like the first example I gave where you run the payload once it didn't work but run it 10 more

17:13 times it might start working so this is not something you can do manually and let's say your models are updating every 5 days 6 days 2 weeks you need to run your entire Red Team all over again right like just not something that a set of humans can do at skin so you will need to pick a tool build a tool will start running these scans so that also goes into next point this needs to be

17:33 continuous you need to be testing it even with the smallest change right like it can't just be that oh this was a very small change to the system prompt or very small change to the model it has the whole thing needs to be tested again right one thing that comes up very frequently it's not very it's it's more of a nit pick but I wanted to call this out is a lot of times when system prompt leaks system prompts leak or

17:57 prompt extraction happens most people are like so what right but I want to call out it's it's actually really bad in most cases because when you're building these systems your engineers are not assuming this to be an open system a lot of business logic goes in there a lot of value it's almost like the Recon step of your traditional security space right like you're giving away every step or conditions that you're putting into your system so

18:23 even though in it in itself may not be that critical it becomes a very important part of an actual attacker right like if you have a threat actor it becomes extremely useful information for them so you don't want to leak that and another misconception which has been broken again and again now is a better model is not a safer model right like we've seen it across most of the leading model providers be it open AI or

18:50 meta with Lama or Google with Gemini as the models are improving all our testing shows that security is only going or security safety all of it is only going down we haven't seen any model that has gotten better or a newer version has come out to be safer so again don't make that assumption cool some tools again not exhaustive list a quick Shameless plug that help you automate these things there's a lot more I just pick

19:19 the ones that I used in the past so there's we have one tool within side Labs called sidebox which helps with at teaming there's the art tool Azor Microsoft has a tool called pirate gak is a fairly popular open source tool as well so pick your tool right you will need a tool you will have to automate things leading into the whole Shameless plug a quick overview of how our tool works so maybe not very clear in terms of the screenshots but

19:50 we have like two ways where we run attacks we run we have like a large attack Library which runs against your system gives you a report at the end of it on where you're broken what types of attacks happen this is something which we use to also give you like zero day attacks let's say a new technique comes out it immediately gets updated here you can auto run it you can set schedules it run against your systems and give you a

20:14 report the second one is an agent tick approach where you can run all the way from blackbox testing all the way to white box testing where you don't tell us anything about your system it will first interrogate your system to figure out what it does craft goals almost act like a security engineer start attacking with every payload it evaluates whether the payload was successful or failed if it was if it fails it improves the payload if it was

20:36 successful it improves its attack goal and in the end again gives you the exact full report of where it failed where it was good what are the types of attacks that are working whatever types of attacks they're not working how you should potentially fix those attacks and prevent these attacks on your systems some examples right like so some this is the attack Library one this is very popular a tag known as adversarial suffix where you add a

21:02 suffix at the end of your system based on the weights of the model which sort of Tricks it into saying things so let's say if you say write a tutorial on how to make a bomb open air usually would say sorry I can't help with this but with this particular suffix behind it the first two times it failed but the third time it started giving instructions on how to build a bomb right so this is one two examples one

21:24 is the types of attacks the second is why you need to send it multiple times right like the exact same string did not work two times but worked the third time so this is the agent talking with the same system all right like so if we go back if it goes back half so if you see it starts having a conversation almost as this security engineer is having when based on every response it improves so when it found out that

21:46 role play based attacks are working it then doubled down that it continued the same role play along with virtual scenario setups to see what is the worst it can get the system to do so yeah these are the two way our systems work I think that'll be the end of my plug but yeah what can you do today if you are building or just learning in this space so if you're a security Ana security ex executive in

22:13 this space but you're just starting how do you prepare for AI adoption so if you're in very early days where your company is just starting to take of how we want to adopt AI start threat modeling experiments because in most companies we talk to they've started with experiments right like where they're like let's try these five 10 different projects whichever works out well we'll take it to production but in most of these companies they decide we'll do security

22:38 testing of only what is going life don't wait for that this will be a good training exercise for your security teams as well start thre modeling or any experiment that's going live that will also teach you how they're building these systems if the company's already started building these systems start redeeming them don't wait for them to hit production think of this as testing again lots of resources on how you can do that but happy to connect

23:02 offline if you're in this phase and want to try out a product as well and if you're already in a phase where your company has already adopted Ai and already deployed do all of the above but also ensure you have enough runtime security pieces in place which includes let's AI firewalls observability anything that you can do in runtime as of today in AI runtime security is still something that gives a lot more results than let's say design

23:30 time security key takeaways I don't know if I've covered all the key takeaways from before but I just tried to cover a few securing AI systems is different from securing other systems I don't know if I was able to convince you on that or not but I strongly believe that when it comes to AI the line between security safety abuse bias all of those pieces are very blury right so you make sure your company there's no fights around

23:57 who wants what somebody will have to raise their hand and own it all right like having different people own this doesn't help and most likely when it goes wrong security is the people who are going to get blamed so might as well take the responsibility to fix it red teaming is an extremely powerful tool in your journey to secure your AI systems right because it helps you figure out what your current state of things are where are you susceptible and then

24:23 change your strategy as needed gives you the confidence that we ready to go live or not go live and yeah automation is your best friend I don't think red teaming AI without automation is of any use right you you'll get a false sense of security you'll just get okay as of today as of this attempt my system was Secure but I don't know 5 minutes later if it's still secure or not yeah I think that was it thank you I think I

24:50 saved you the five minutes of my introduction and reach out anytime if you have questions you sure any questions sure hello yeah yeah so in side boox you mentioned right you update payload again and then you update goal again so this combinations you keep continuously with the same payload like same type of

25:22 attack workflow so how do you decide like this is the worst it can do like when do you stop that so you you you get to set so this is a customization you get in while you're running your scan you get to decide how far should it go right like at what time should it at what point should it give up right so you we have a default depth and a default breadth on your attack tree

25:42 based on that it decides how deep will it go now let's say your system could be even more vulnerable but let's say if you've set the depth to 10 attacks then it'll stop there right yeah so I will not be able to figure out the later vulnerabilities that still might exist but I have not adjusted the depth accordingly 100% % so what that is why like once you fix those then you can start increasing your depth and it also

26:04 learns from its past attacks right like so it won't be that let's say it did attack a b and c and found those didn't work and D worked next time it won't try ABC it just start from the point B okay but you say that there systems transitions right correct so like if I want write the previous vectors so it should take in consideration of the previous vectors because if I again try the previous vectors then again ideally

26:24 it would be possible they might work correct great so the thing is none of these attacks are St static right so it's using different types of techniques and procedures so even if it doesn't end up using ab and C the techniques that were used in ab and c will get used somewhere now let's say if the pre new attacks are not working it will start going back to older ones do a combination of all of them and then try

26:47 those care okay thank you any there one here Ro quick question on how to detect zero day whatever you show was is published on archive it's been wa couple of months correct so things like that the poem poing example is there a way to let say detect if the fing data is poisoned red p p red pajamas again this will be based on what everything that I have read there are there is research that's already in

27:19 that space going on like you can look for the known poisons for unknown poisons again it comes down to extensive testing but there is unless you have full control of the data that went in or somebody who had that control is willing to give you that in contractual way that this isn't for example right like coher right like coh's whole selling point is that their data does not contain any copyrighted content and they're giving contractual coverage for that right so if

27:48 similarly like whatever model you taking if the person who built that model is able to give you that guarantee of the data that went in that would be the most effective way as of today but there are ways to look for known poisons so if you know that these are the types of popular poison data pieces that are within models you can test for their existence in models but the unknown poisons is still an un semi unsolved

28:11 problem thanks this is mainly with respect to the runtime security aspect of it yeah so I was wondering let's say you're typically interacting with with one agent right interact with the agent the agent is giving a response right before the response is probably shown to the user why not have another agent should probably validate if the response is as per expectations sure and I'm I'm

28:42 assuming when you say agent you're talking about an llm agine right so that is a fairly common method right like where there is an input guard dril and an output guard now that output guard could be built using llms as well a lot of things come down to what is your most accurate way of doing it and and reducing your false positives reducing your cost reducing your latency so that is one of the ways you could

29:05 potentially Do It but just protecting the output may not be enough especially when your input could lead to taking some actions which don't even lead to like an output coming or or leading to let's say the output itself being something that can then trigger a injection or a Jailbreak on the output llm so best is the defense in depth right you IDE want to protect your input and your output thank you cool thank you so

29:35 much thank you