Building an AI pentesting platform, with Philippe Dourassov of Aikido

0:00 So if you want to find bugs in your app, you can scan the code using static analysis, but it's also really good to test the app itself, whatever is deployed and staging a production, because there's many reasons vulnerability might be in the app, but can't be found by just looking at the code. For example, you could be deploying like a different version of the app or the vulnerability could be introduced by a dependency or a configuration that's outside of the repo you're scanning.

0:25 So we have ways to scan what's actually deployed. And Philippe here has been working on some of that. Hi there. Yeah, indeed, I've been working on AI Pentest for a while now. I started I think around a year and and a few months already. Joined Aikido to continue there for I've been there for eight months now. I've been with my team as AI Pentest leads, mostly working on the technical side of it and making the agents do the hacking.

0:52 Very cool. And so you're going to talk about what you've been working on. And I think it's very interesting because it's all like new and evolving. But before we talk about AI pentesting, we want to talk about a little bit of the state of the industry that led to AI pentesting. So first, traditionally, if you wanted to test the web app, you use DAST or dynamic application security testing. And those are basically web app scanners and crawlers.

1:17 So you have like OWASP's ZAP or it's not OWASP's app, it's Checkmark's ZAP now. Burp Suite has a scanner, Tenable has one. I think Aikido, you mentioned, had a DAS tool as well. And I want to cover a little bit about some of the advantages and some of the challenges of those so that we better understand what advantages and challenges AI pentesting has in contrast. Hm. Yeah, Yeah, so I think a among the pros, the best one, I think right now is is the the ratio between the amount, the volume that you can scan and the the price of it, right?

1:50 Like it's basically running codes that's going to be run the same way that you can run twenty four seven on all of your websites. And it's just HTP requests that are sent, right? The cost is electric like once you have the license for DAST it's about the electricity that you use and the maybe network traffic that is being generated, right? It's it's very much scalable in terms of the amounts that you can scan. the second one I think is also about determinism.

2:15 Right? If you do the scan once and you do a second time, you should know what to expect, right? It's something that's that is predictable and you're not going to have surprises or maybe issues that are only found 50% of the time. It's either you find it or it's not or well it's not detectable by the DAST, right? And I think these two are are the ones that's really allowed it to go very far so far and what allows companies to very easily just set it up and and get going, quite easily.

2:44 Yeah, I completely agree. are great points. And for example, Zap is free. So you can even deploy it as a GitHub action in a repo and run on a schedule. And that's it. It takes a short amount of time to deploy. The basic version, you can run it forever without paying anything other than the GitHub action runner cost. So these are great points. Now there are some pretty big cons to those. I think there's a reason DAS tools are quite hated in general.

3:11 They don't have a good reputation. that that's fair absolutely I mean they're they're limited by by whatever code is is is present right in the end DAST is just doing its best efforts with the capabilities it has to find the vulnerabilities but the best for efforts are are really just a very very big checklist of like attempts to do right so you can provide some context to the DAST right you can help it adapt to some situation for example If you go on the websites, the login panel can be different in in different websites.

3:42 It's actually very often different in every website that it encounters. the DAS will be able to dynamically see the fields that are here but and and and test them, right? But the issue is that it can only it can only detect maybe the forms if it uses the HTML form tag, right? And maybe it's actually another HTML tag that is used and it suddenly breaks because it's it's not able to log in anymore and if a DAST is not able to login, well it it's stuck at the very first step and s it cannot test any of the features that are actually interesting, right?

4:10 Maybe you can have some headers as well, but if the header actually is has a refresh token that it has to use to refresh a JWT every ten minutes or so, well again after ten minutes the DAST is broken and And you cannot use it anymore and you cannot find the issues, right? And basically you can you can always try to find some circ like some workarounds to make it work on the setups you have. But you're always going to be limited but by whatever is fundamentally supported by the DAST and sometimes that's not enough for some situations.

4:39 Yeah, definitely. that's, think, a challenge I've seen with many companies. They're like, oh, let's set up DAS. And some of them, they just set it up without authentication. They're just hitting the login page and like, yeah, we have DAS. We're scanning. It's like, well, you have to figure out how to get it to authenticate. That's a big part of it giving you value. Yeah, yeah. And the second one I think is also besides the the the adapting to the authentication part is also adapting to what issues are you looking for, right?

5:05 a a DAST will always try the XSS and SQLite and LFI payloads at the same place every time. those are basically the same payloads can be used on ninety percent of the cases. So like that's fine. These issues can be usually found by DAST and like that's where they excel actually, right? maybe they can they still cannot are not able to find a very complex that are second order and require multi-step multi-steps for it. But generally speaking it's it's it's good at finding these technical issues.

5:33 Now if you look at the logic issues, the ones that are more very specific to the business, for example by passing a paywall or being able to escalate the privileges of your account suddenly it breaks because it doesn't understand what's what's an admin account is versus a member account it is. Doesn't understand what tenant isolation is or sensitive data among the user information. So all of those are unfortunately not really let's say predictable in terms of what type of attacks you should do and what is the criteria that defines whether the attack was successful or not.

6:04 Yeah, that's a really good point. mean, that's also a challenge even for like Pentester as a human. Like if someone gives you access to the app and you want to try like cross account or like cross role, like authorization, like violations. And it's not always easy to determine like, am I supposed to be doing this or not? Sometimes you just got to ask the, the build, like the developers, cause like you don't have enough context to tell if this is an action that you're supposed to be able to take or not.

6:30 And so obviously DAS will not have that context and wouldn't be able to tell if there's a violation of permissions. And I think the last one was like if it was just high false positive rate, they have a tendency to flag a lot of things that like you don't really care about. That's indeed I think the the the deal breaker for many is that's you end up being drowned and you have to search for the one issue that's actually relevant.

6:51 You don't even know if there's one, so you maybe just are looking into one hundred different issues that are in the end just going to make you waste the precious time that you have. Yeah, so that's, you you're trying to automate those tests. Now, on the other hand, you can hire actual pen testers and here like you're hiring a company outsourced or in-house it could be, right? But it's a human actually coming in and testing your app, right?

7:15 So here, I mean, it's a human, so they have context, at least more context than an automated scanner. Mm-hmm. Yeah, indeed usually you will have some interaction, maybe a scoping call between the customer and the pen tester. And at the end like you get the information about what the target is, what's what's maybe is the resources that you get. Maybe it's a white box pen test and you get the code base as well, right? And you can actually explore it and do this mix between looking at what the codebase contains, for maybe leads of potential features to test, maybe vulnerabilities, suspicions that you want to test on the application.

7:51 And from there do the dynamic the validation of checking hey is this working or is this maybe blocked by some upstream proxy that's that blocks these requests so it's always always interesting to have this combination because instead of blindly testing every every attacks the human can actually make sense of where the application is most susceptible to be vulnerable Those are really good. Yeah, very good points. then the cons, mean, pentesting has a lot of cons. Like it's usually related to the price.

8:22 I mean, if I can have an unlimited ongoing pentest continuously running all the time, that would be amazing. But because of the price you have compared to a pentast that runs twenty-four seven indeed. And the the the thing is that the price is is is mostly because of the amount of time spent because of the press we actually also limits how often we run it right that's I think a big issue is that's going for a pen test means that if you do it only every two every two months you are going to have well the code that is shipped the day after that is going to be left unchecked for two entire months.

8:56 Maybe that's fine for some but for many organizations ideally they would make sure that's they're safe from day one and not from day 60. Mm-hmm. Mm-hmm. Yeah, these are great. Yeah. So this gives us some context to go into the thing you've been working on, which is AI pentesting. And so you've been building a whole platform for that, and you recently gave a talk around that as well. Yeah, that's correct. I was in Vilnius great great conference called CyberwiseCon.

9:24 they invited me to talk about the challenges we faced while building the A agents. And you know, well we're a commercial company, we were like a at first we didn't want to share those, but it's already been months and like now we see this development where you can see this interest in a agents on how can you use them to secure your code, how do you use them to have these agents do the hacking for for for you and test your features.

9:47 And I think these problems are technically very interesting for many people, not just the ones that are creating these platforms. So that's why I presented this talk is because I wanted to share this this challenges that we faced, how we approached the issues we had and potentially help other other companies to to have these agents get secure and maybe understand as well. better what's the hype part of it, what's the actual efficiency part of it, why it works, why it doesn't sometimes.

10:11 So that's that's what's the goal of this talk here. So yeah, the talk was mostly going through how the agents that are doing the A pen test are working, how they're going to hack the websites, how they behave during it, and the f challenges we faced while building them. So yeah, the first part is about how the agents are actually working on these these targets. And the the very first step for these agents is to understand what do they need to attack, right?

10:35 What's what are the different places that they need to try pedals on and and try to find vulnerabilities in, right? And well You have two paths, right? Either you do the white box part. White box is when the customer provides the codes, the codebase to to help the agents figure out what are the best ways to do the hacking. sometimes codebases have three million lines of codes, that's the example of open claw here. It's going to be hard if you just give the whole code base to cloud code and ask it to find all of the vulnerabilities, right?

11:03 because if you want to go in depth in every feature that you want to test, you actually need to spawn multiple agents that are going to work in parallel so they can each go in depth for testing their feature, right? So we need a dispatching strategy and then give each of these agents that you dispatch the classic usual tools for agentic well exploration of the code, right? Allowing them to understand the codes by reading the files, listing the file, grab blob.

11:29 All of those are very classic agent tools. You can see them in in in different IDEs or agentic coding tooling. now for black box it's a bit different because you Don't have the code that allows you to see all of the features that are present in the in the code, right? Black box is different because you only have the targets, the URL of the target, but you don't know what API endpoints are. Maybe maybe the customer provided the file, in which case you can do a similar strategy as before of dispatching.

11:54 But if the customer doesn't let you know what are the different endpoints and only gives you this targets link that's then the agents have to hack. you need to have a different approach if you want to have this dispatching strategy. And mean, that's also a real challenge. There's organizations that have web apps deployed and they don't know what they are. You know, it's like, we only have this URL. We don't actually have any more context other than just the URL on, on, on what, what's supposed to be, how it's supposed to work.

12:23 so that is a real problem. Ideally, like, like they'll just give you like more context. And I think like, that's an exercise that, that like everyone should, should try. It's like, let's actually try to have a. a markdown file listing all of the context that I would give an agent to describe what this web app is, but sometimes that's just impossible I think. Yeah, yeah, indeed. It it depends off of the security maturity of the companies as well, right?

12:46 Maybe enterprises have a better attack surface management, but the ones that just want to go fast and and ship without without being slowed down by security, maybe they didn't do these steps, which is understandable honestly, right? In the end it creates a lot of friction when you have to go through all of those. So i black box is sometimes easier because you In theory don't really have to take care of those. performances will take a hit unfortunately, right?

13:10 the less context you provide to those, the least like it's both both for the agents and both for the human pen testers. More context equals more targeted attacks and less just brute force, brute forcing every potential attack is possible, right? But it it's sometimes still relevant to the black box. It's easier, it's going to be faster. Maybe sometimes you don't have to go through legal arrangements to do it as well for the big enterprises. many reasons why black box is still used today and we understand it we don't want to fight against it.

13:37 We still want to secure the ones that cannot allow themselves to go inbox. But yeah for the agents to map the the the this this features and do this dispatching we actually provide them with a browser right a browser where they can click and interact with all of the features. And if you give these agents capabilities to interact with all of the features and capture the network while they're interacting, fitting every form, clicking every buttons, you actually get a list of all of the requests that were sent.

14:03 And once you have get this list you can actually map what are all of the endpoints that were accessed during the testing, right? Or what are of all of the passes and then you can get an idea of what's the above is. now, this is kind of like just like having a web crawler just like click every link and try to like interact with every endpoint, right? Like for now, this is like nothing new, I guess.

14:24 I mean the the the the the good part about the Agentic allow is is allowing them to also do the multi-step harder well fitting the forms that are like require, maybe A valid a valid password that has to be released at the right time or like filling a to a fa form as well. That's that's a common one that we see. I think that's allowing them to give more capabilities than just clicking every buttons, but also maybe checking their email for the password results, email that they receive that they want to interact with.

14:50 That allows them to do much more than just clicking every buttons. then the second part. The fun one about exploitation is it's going to be once you have all of the lists of what you want to hack, you can just dispatch agents on them, right? You have one agent or multiple agents work and go in depth on every endpoint that you saw or maybe every sources like in the code as well you can see all of the endpoints as well the sources where where can you actually affect the application with your user input, right?

15:19 And once you've mapped all of those you can have agents be dispatched on all of those and and go check in depth for that. And this flow right here is very similar to how you would do a real pentest. Because on the first day of a pentest, you're just clicking around, you're just exploring the app, you're going through the password reset flow, checking your email, right? So this is actually how a pentest works. The first day clicking around, then you start actually trying to exploit what you found in your burplog history or whatever.

15:47 yeah yeah. No that that's that's indeed like there's this philosophy that we had while building a Pentestis usually. if you had an army of humans that you could s dispatch on a target, how would you do it? What would be the most efficient way of doing it? And usually this how will the human will do it and that's something that's helped us a lot during the development of this this product. But that's going that's something we can actually see later on when we're going to talk about the challenges and how we solve them, this this aspect is going to come back.

16:13 So yeah, the next step after the sorry, the next step, the exploitation part. you might be wondering, okay, what are the agents looking for? There's so many issues types that they can find, IDOR SQL RCE all of those. And yeah, like how do you how do you have them find those, right? Well you can actually separate them in in different type of issues, right? you have the the technical issues, that's the ones that we mentioned where the DAST already perform pretty well.

16:37 Everything that can be like Done in within like the same kind of payloads, which is SQL injection, you do the single quote or sleep on and sleep. for maybe LFI, you do the dot slash dot dot slash etc password. All of those technical issues they end up being verifiable because they're the same on all of the web applications. If you can read a file with an LFI on an application, well you can read the ETC password, and if you do it on another Application, well you can also read it password, right?

17:03 So like there's there's way to verify whether these are real or not, and it's always the same ones. Now the more logic issues, I mean you could call it differently I think, but that's that's like something that's I consider when I'm talking about the more business specific issues, application specific issues, that are not universal. the ones that are going to be specific to how the login page of the application works because every login pages are different again right a good understanding of the app itself to be able to determine those.

17:30 Exactly, yeah, you have to first reason about it and then you can actually exploit it. that can also be about paywalls, right? So if you can bypass the paywall or if you have different privileges and you have maybe a low privilege account, the agent can check, hey, can I make it admin? And then well that's again something that's very application specific. Now the hardening checks I think that's more of a security he hygiene, right? I'm making sure that you use the right security header, that you have a CSP in place.

17:57 And these are just like you can you can have it already on the previous like that's not AI pen test specific, but that's also part of what a pen test has, right? I mean just the normal DAST scanners usually they mostly find these and then nothing else like like 90 % of their findings are like like oh old TLS or It can yeah like that that's the one that's like code is good at it because y you make sure that these are always checked for and then you can have them always reported.

18:25 So I think that's that's that's something we integrated not from the AA part of it, but just because it's good to have it in the report. Yeah, yeah, mean, they're for compliance, like they always really want you to find those. So yeah. it's it's always going to be like lows or mediums, maybe informative as well, right? But it it's still nice to have them. Now there's another interesting type of issue that we often notice is the fixes bypasses.

18:47 oftentimes when a user wants to well patch an issue and well ship it's well it's always good to retest whether the patch is actually working or not. And well the agents can just test the same attack, see if it's fast patched a more interesting approach is also have them try to bypass the new fix, right? See if you can actually maybe think of a way to go around what was supposed to be the mitigation and actually go back to what's to exploiting the same issue by finding another path maybe to it, right?

19:18 so that's also something that's the the agents can find quite often. And now well once you have these issue types, well you Your goal is to have as many agents as possible to try every potential attacks on every feature, right? ideally with the code it's even more targeted than black box, but especially for black box it's important that you have enough agents that can actually try and run their own scripts, their own attacks on every features that they can find.

19:42 And then Again it's like four hundred agents. The goal is that they can actually interact with the application, that they can run this export scripts. so we can give them every every single adjust. We can also give them their own machine, their own sandbox, and that allows them to actually do the dynamic parts, the pen testing part of the pen test, right? To do the exploitation by running scripts, so like payloads via bash or python or via the browser directly if it's an XSS, you can just use the browser to see if you can get the next executed, right?

20:08 And the first challenge I think about here is like, this is basically approaching the realm of fuzzing. it's like the big challenges is really the cost because like I can parallelize like 1 billion agents with like 1 billion computed computing years of like continuously testing the app. But then that's the first limitation. But I did think like, do you think this is something that like local models? like models that are locally that are cheap to run are good at?

20:37 Or do you really benefit way more from the frontier models? Well in this specific example, this is a very basic attack, right? You're going to try different payloads on a cister certain parameter that was used on the static thing. That's almost something that's the DAS can already do by itself, right? here you can just see that's the agent adapted with the right content type for some reason. I'm not sure what was the issue here, but they had to use the right content type for the specific specific endpoints and try different payloads, right?

21:06 But that that's very easy, right? Every every models that are in the last two years that are not that that are f like 12 frontier at some point they can do it quite easily. now the good part about it is that well by doing these dynamic validation parts you can actually again see the results themselves, right? If you can read a file, well read it and see if it's get if you get the outputs and actually validate the issue, right?

21:28 now this this testing is also again like quite basic and desk and also validates the issue. So like that that's really the basic checks. And now the interesting part as well is that the once once once the issues were found, like and you validate them, you can actually mix the the formal validation and and the Well, agentic testing part, right? if you can read the file and you get the output, you can actually see whether the output is really here or not.

21:50 And that's like all of the technical bugs, no hallucination. That's like very like much easier to do. now there's going to be another part for the validation. Right, I'm going to go back here. Now for the logic issues, it's going to be more interesting, right? Because previously we saw some stuff that DAST can accomplish and and this is like pretty basic, but we already saw that the flow of the DAST is that it cannot find the logic issues that are very business specific.

22:15 So how does AI pen test and most importantly how do we make sure that these are valid and not hallucinations, right? Well first we have to kind of c categorize and separate these technical issues, right? Technical issues, if we can validate them, let's reduce the noise as much as we can already. Now we don't want to restrict it to the technical issues. We also want the logic issues whenever you can. And that's going to be about the missing authorization, the sensitive data exposure.

22:39 And that's actually the majority of the issues that we're we're finding is is going to be this these checks that are very business specific. So the the good part is that they can actually verify it by testing its live, right? and it's not going to just check, for example, if there's a 4.3 and authorized when you put another ID, right? Ideally the full reproduction of whether an order is real is can you read the data of other users?

23:03 But at the same time you don't want the agents to read the customer data as well, right? So what an agent could do here is to have two different accounts and we actually recommend to provide two different credential sets. Have one account creates maybe in that case it was a note, so it creates a private note. And then the second account checks whether it can access this note that was created, right? So you actually verify dynamically pentest would do that.

23:25 You would have multiple accounts and you would test with them. Nice, nice. And it's the same, it's not just about crossed accounts, but ideally like the perfect case scenario is that we have four accounts. Two on each in two different tenants and these two accounts also have different roles as well, right? And like that's the perfect is because you can s test for cross tenants issues, but also maybe inside the tenants you can still have issues where a member can become admin or a member can read the data of other member while it shouldn't, right?

23:51 So these are this the perfect scenario is is really to reproduce the environment that allows the agents to test everything by themselves and and confirm it, basically. So now that we have this base of how you create the this this approach for an A pen test, you realize that when you put this this product that you worked so hard to create in a production environment, you're going to cry because there is nothing that w goes well, right?

24:17 have to work on their setup right and I'm going to go over the main ones I think that were were most interesting to go over. So the first one it's the authentication. So we mentioned that DAST have some issues in authentication, right? It's always going to be a different login page. And the agents themselves, well, they also have to work on those cases, right? If you do a curl of accounts.google.com, well the agents cannot from the curl figure out how it would need to log in, right?

24:42 It will be a very, very complex multi-step approach that it will need to take here. and authentication is important, right? Because usually you have three fourths of the features that you want to test that are behind the authentication rule. So yeah, we have to find a way to support all of these different authentication, right? You may have email password, you may have the HTTP basic auth as well, you may I you may have like login via third party as well.

25:03 so the philosophy here again is what a human will do, right? And like it's pretty simple, the human will just use their own browser and login because like it's it's the obvious thing to do, right? Well let's give a browser let's give a browser to the adjusts then, right? And see how they well they perform. And it actually worked quite well, right? We're surprisingly we're positively surprised how efficient an agent was at using a browser. Now it's a bit more popular already, but at first it was like a year ago, we were like, man don't know how good it was, right?

25:30 And now it's even better because it doesn't do misclicks anymore, right? It just fills the form and clicks the right buttons all the time. And it's super impressive to watch it live. Now you may be wondering Okay, we have we have four hundred agents that need to log in and like they have a browser call, they can input username, password. But what happens when you have four hundred agents try to log in on the same account at the same time, right?

25:48 So there's potential different potential scenarios that could happen from here, right? And I'm going to give four of them. Multiple ones are correct, maybe a few are are wrong. so the first scenario would be every agents are able to log in successfully, right? They enter user username, password, that that's that's what we want, right? the second one is the customer complains because you have like an email that's being sent every time there's a new login authentic login attempt that is being done on the account, right?

26:11 And then they complain because they have four hundred emails or more because some agents also logged out because like they need it sometimes. And they have like five hundred emails at the end of the pen test and like they're wondering what what the fuck is happening, why are you doing this to me? the the third one is well yeah, I IP banned after the tense attempt, right? agents like they try to log in and like they get IP banned because they use the same IP.

26:30 And the first one is the account is stocked out because too many authentication attempts and the server doesn't like it. And like it it never goes as expected. it's there's always always something going wrong with that, right? It you just it just doesn't doesn't work. so from there, yeah. DDoS as a service. Yeah, like you're DDoSing some of these authentication servers. But also, the interesting point here is I assume you're running all of these from your own infra.

27:04 And ideally, the client is allowed listing all of your infra for this pen test, which is also like a... an important thing to do during real pen tests. Yeah, I mean like that that's again the this this approach of what a human will do and again yeah it's giving the IPs to be waitlisted. It's not it's not just for the account, right? It's sometimes about the different captcha or turn still systems they have. It's it can be also about the the Cloudflare WAF, that's a very common one, they always have to wait list for the Cloudflare WAF, so we still get to find the issues because usually like even if there's a WAF, it's always good to fix the root cause issue and not just the symptom.

27:38 right. And then outside of WAF some applications do their own rate limiting. Mm-hmm that's true, that's true, yeah. So yeah, we give both like this user agent header of course, but ideally the IP because user agent can always be spoofed. the IP is much harder for that. And so wha what will be the solution for this, right? Because you want all of the agents to use the same accounts and don't want to get them banned. And again, let's just think what would the human do?

28:03 If you had four hundred humans and they all want to do the spent tests and they have the same account, but they know there's like a locking mechanism that's likely to happen. Well I think as a human I will just have one person log in and then share the cookie or the local storage to the other. So like basically just allow cross agent communication and collaboration, for them to well work together and avoid these type of situations, right?

28:25 Mm-hmm. Mm-hmm. That's a good point. Yeah, I mean like that's something we encountered a lot. and yeah there's another one that we encountered is is especially for enterprise application. Yeah. I'm curious, if you have multiple agents running in a sandbox, how do you get them to talk to each other? Hmm. So the way it works is that we have we have separation, right? We have this sandbox that the agents have, that they own the machine, right?

28:50 They can do whatever they want, like it's RC by design, like it's providing a VPS to the agents or the user if they want to do fun stuff on it as well, right? and then this VPS is isolated from the rest of the intra except the API. The API is the only thing they can access, they have their own token for each. to access the API and from there the API can actually well receive requests of like hey this agent wants to communicate this to other agents, right?

29:13 And then like their have their own unique well token to like say, hey that's that's you or that's you and then like you can actually have them communicate together. Nice, nice, nice. And the API is like the API of your own platform that you built. Yeah, yeah, okay. great it's it's like in the end our our info is is a lot of orchestration around the agent. Like the agent itself, of course we have to give it the right tools, but like in the end like the the model itself is is what matters a lot and then it's going to be the classic agentic tool to give it the right capabilities.

29:43 but I think what's very important here is is the whole infrastructure and allowing them to work together efficiently and like that's that's also the part where we have them to be able to communicate with each other as well. That's the that's the same thing. It's not it's not just about coordinating for this specific issue but like the other that can be reused for other issues right if there's an agent that wants to test the password resets well the password is different now right so you want the other agents to know that's hey now you should use this password instead and it's important to communicate those because otherwise then the agents end up being locked out and they cannot continue testing anymore.

30:17 Now, internal application that's also something we encountered for enterprises. That's that's like something that's gated behind their own network, it's not connected to the internet. But they still want it to test. But our services on the cloud, it's not just a software that you can run on your own laptop, right? It's it requires a lot of GPU, it requires a lot of resources as well because you have four hundred machines. to install their VPN client on every single sandbox you're running?

30:41 Yeah, I mean like in in theory that that's it's not impossible because we'll see it later. But we can like the network is going to be centralized in one place. So you could you we we are actually implementing it for two customers that we have. But like not like even then not everyone has a ready open VPN connection that they can give to to us, right? so we we have this VPN support, but we also have like a simpler approach where we can just install a Docker container and this Docker container is actually going to connect back directly to our services, right?

31:11 It's not it's not it's not something they have to expose, it's us that are exposed via a public endpoint that they can connect to with their own token and then it creates this bridge between this this Aikido and and and their own application. Yeah, which is also something on real pen tests. Sometimes if you want to test their internal network, you just send them like a Raspberry Pi or like a small box that they deploy and now you have access their environment.

31:32 So that's like a... that's that's really close. In end like much of the most of the things we do is like very close to what an army of Pentas would have done in terms of a hiking rights. And and then besides that there's like many capabilities that we have to support to be more universal, right? We had customers that ask for two FA for connection, other that ask for an email, maybe it's like magic link, or if you want to test the reset password as well, right?

31:53 Rez password is a very interesting feature to test for. And all of those are the required email for each different agents their own email or each account their own email. So that's something you implemented? Each agent has an inbox? Wow. each agent their own inbox, but sometimes, you know, when you create an account that the agents are sharing together, then it's a shared mailbox as well that the agents can can check together. And that allows really to have basically universal setup support for every application.

32:21 That's what we aim for at least, right? Something that works on every application without having case pair case ba basis. It's like giving capabilities instead of giving implementations directly, right? there's also phone number support, right? it can be two FA like SMS verification when you log in. it can be hosting external payloads as well, right? If you want to test for log4fell, which is a very big one, log4fell. well you need to expose some kind of of like you need to expose the payloads that you're hosting to be able to expose this this this RCE.

32:49 And if you don't have this feature, well you're missing a critical ten on ten C V E that's It usually is like something you don't want to miss, right? And yeah, file upload support also like that's that's also part of the browser. When when you give a browser to the agents, you have to think that well they don't have a whole laptop, they have a browser, right? And this file upload feature is like a mix between a browser and the laptop itself, right?

33:12 that's like a bridge between the file system and the browser itself. so we had to also implement this bridge itself because we like maybe eventually we'll give a whole computer to the agents. But right now like it's it's like a VPS that's headless with a browser and like we still have to give these capabilities for them to interact with what's usually well used in these applications. And this screenshot reminded me, does each agent have its own persona with theoretical PII basically?

33:39 Would you upload a passport or a driver's license for each agent? Well that that example is is very specific to to what we had. It's it's not real from a customer, it's from Google Image, right? But our customer did ask the agents to do the authentication by uploading an image of the f of their face actually, right? So every agent had this image dot like face dot png that the y customer uploaded. that's the that in their turn the agents could then well upload to log into the service.

34:04 So that's that like again, instead of doing a very specific implementation of uploading a face.png at this endpoint, we instead give the capability to the agents to do it themselves, right? so like it this approach is really what allows us to like grow bigger and bigger to what we can test for. And right now like we got to a point where ninety nine percent of the well ninety five percent of the pen test like it's self-served and you don't even need some like engineering help there.

34:28 Mm. And yeah, like there's much more as well. I think like I would need five slides if I wanted to list all of the ones that we had to implement in the end. but again like we always want to give like maximum capabilities to the agents, right? Now in terms of the what's the customer expects from the product. right now we're selling them something that's basically says, hey We have an ha AI hacker that's extremely good and better than whatever you can find out there.

34:52 and if you pay us, we're going to let it lose on your on your platform, right? And like it's very scary of course, right? So we actually had put some safeguards to make sure that they are not damaging things that they shouldn't. the very first one is that Every every connection from the agent that is being done goes through a forward proxy that we have, something that will check the destination of the request and is going to check is it part of the scope or is it out of scope?

35:15 Are you attacking the targets? Are you are you accessing a static file? Well those are fine. But if you're attacking stripe.com because there's a stripe page on the the platform, well that's bad because suddenly like the agents are are are doing stuff that they shouldn't. Same for like a staging application, right? Like if they stay on the staging environment, that's fine. If suddenly they start to go on the production one. Well, usually it's still fine, but like you still want to avoid this case and taking this risk that the agents do the hacking on the production.

35:42 Maybe other other users could see the payouts that are being used and and whatnot, right? So like that's something that's really is restricted and Not at the prompt level, not at the trust of the agents level, but really at the programmatic level to make sure that it's not just we ask them not to interact with those, but we prevent them and don't give them the capabilities to interact with what they shouldn't interact with. And yeah, if for example cal.com that's another example.

36:06 If there's a calendar to take booking on the page, well it's going to prevent access to the cal.com domain, right? Now another issue that we faced as well is the fact that there's this training gap, this knowledge gap between when the model was trained, the data of of the model and the present, right? The model is very good as knowing what was the maybe last twenty years techniques, but we're talking about the very, very recent and modern kind of techniques that were discovered.

36:33 it's not in the training data that was used to train the model. in this example is Gemini Gemini 3.1 Pro but the 3.5 flash that's released in like May it's also like the same twenty twenty-five January. Like that means that there's one year and a half of data that the agent just doesn't have and doesn't know about. so we still want them to know about the latest techniques and attacks, right? especially because they're usually the rising threats that the customer really care about.

37:00 an example of that is the all of the L LM attacks, right? Prompt injection, making sure that the tooling is allowed. And that's something that can be solved with some more manual work where like the human in the loop right now in our team we have researchers, we have many hackers in the team actually. most of us are hackers actually. and well we have this knowledge ourselves that we can provide to the agents. It's basically distilling our own data to the agents to have them be the best and know about every techniques that are out there, right?

37:25 and skill is also something that's like a good practice in terms of the context. we don't want to give maybe this techniques to all of the agents because maybe there's going to be only one percent of the agents that's one that are testing the LLM features. but with the skill you can actually allow them allow only the one percent to have the context and not Pollutes the context of the ninety nine other persons. Nice. Now an interesting point is also checking how pen test is being launched, right?

37:50 And there's going to be multiple steps. The first one is obviously what is the target, right? the customer wants to provide the lists that are going to be used by the by the agents and then ideally provide more context than just the URL. As we saw before, black box is hard, you have to find yourself all of the features if you provide the codes directly while the agents know what to look for, what attacks to try.

38:10 And it's going to be much easier for them to find the issues as well, right? And then you can just launch them and observe do the hacking. That's also pretty fun as well because you can see all of them working in parallel and at the end you get the full report of the issues, right? And if you look at this in practice, well we can actually check what the final product looks like. So in terms of how it looks like on the final side, right, once we have implemented all of those.

38:31 well we have this part here where we have to provide the domain that we want to attack themselves, right? Usually it's going to be maybe front-end and API. We often see it separated, sometimes it's just one domain, in that case it is just one. and then you can actually have the interesting part which is having the users, right? If you have a username, password for example, we can just have the agents go on this page here and like Just give it a as natural language text as well, right?

38:54 It's an LLM, it then understands those instructions. You can just say it go on this page, enter username, ASD, SD, ASD, passwords one, two, three. And then well this context that you provided is going to well give to the LLM to the agents that has its own browser and it's going to read it, go on the page here. So it's now navigating to the page, you can actually have the traces of photos they use. You can also have the screenshot of the browser itself.

39:18 You can see that now it will press login automatically, enter the username, password fields as well. Press login. It it goes pretty fast so we didn't see everything. But like you can see that it was able to log in using the browser. And once once it completed well you can actually well confirm that it was being done, right? And you can also just see all the request logs that are sent. That's also something we want to make sure that the user can always see what's happening and Make sure that see what happened as well in the past.

39:44 Now that's the part where you have the safeguards, what's attackable, what's accessible, everything that's outside of this list is blocked. And then like that's the the repositories where you have to where you can list what you want in terms of the the repositories. Sometimes it's one monorepo for the web application, sometimes it can be ten microservices, sometimes it's front end, back end. It's very it's very depending on on what customers there are, right? And just curious, know Aikido has a code scanning tools, Here, are you basically handing this off to the code scanning tools and agents or are you doing like your own analysis here independently?

40:21 So when you import the project, it's going to automatically do the basic checks, right? It's going to use the SAST and like going to have the issues listed here. But the pen test itself is completely different and it's going to be fully agentic from A to A to Z. The reason for that is it's the same as DAST, right? It's really going to be cheaper to do these deterministic and fast checks, but in the end nothing beats as as anything that's as close as possible to the model itself, which in this case is just giving the codes to the agents to figure it out.

40:47 Mm-hmm, mm-hmm. And now it's going to be another another one which is the throttle. If you want to not break your application, you can put it a bit less than hundred and fifty requests per second maximum. We added this, I I think there was one time like we did hundreds of pen tests by by this time, right? But one time we actually had the agents s send too many requests. So since then we actually implemented this cursor and avoids the agents sent more than let's say hundred requests per second as well.

41:14 That's also something we can often see like in pen test as well, I think. That's again a practice that's that is very common that we want to integrate here. And from there you can just do the schedule, continue, have the price depend on the size of the codebase as well. bigger codebase equals more agents as well. And then you can just start it. And we do ask for staging environments. we know that many many customers cannot provide a staging environment, so we ask it but don't force a staging environment.

41:40 And we allow the agents to go on there. we cannot guarantee that nothing will be let's say yeah broken but like we're trying our best and so far like again it did break one time when there was too many requests that were sent. So we still ask for a staging environment, but I think right now the agents are good, like are pretty good and it's now quite rare that stuff breaks. We still recommend staging because sometimes on production it creates a lot of data, right?

42:04 You have all of these payloads that are sent. and you have to clean them up if it's on production otherwise you have maybe one hundred tickets on support that was that were created or fifty registration requests that were created. it's going to take to to create some noise not on the results themselves but on the application itself. so staging is always let's say easier but production is also like we got to a point where the models know that they shouldn't do risky actions and we also ask them to do a very minimum minimum impact proof first, right?

42:31 If they have a SQL injection, they don't check the data of other users, they check either their own or just do like even a slip to see whether they can actually do it. And then they report the issue. They we don't allow them to check for other customers' data or try to see if they can do a reversal on the server itself. Just like a small like ID as an execution to see that it works. That's that's enough for the for the adjust to stop and report the issue.

42:53 Now that we have the whole product that is ready an important part of it is just testing it, making sure that what you're doing is working, that your new improvements are actually improving the performances and not worsening them as well. And that's something we worked quite a bit on. The very first thing that we wanted to see is like are we even comparable to human pen testers, right? is it is it it doesn't make sense to go on for an A pen test versus a human pen test?

43:18 And six months ago we already had like smaller models, but we still did the test already. in terms of time, like obviously a computer. is is faster than like a human, especially when you have 400 agents working in parallel. So yeah, the agents just do all of the features in parallel, it's just much much faster. But time is not the main criteria, right? Usually what people want is can you find all of the issues. And that's the depth part of it.

43:40 one thing I'm curious about in comparison of time, I think time is a weird metric to pick just because it's a measure of level of effort. so, well, the first thing is you would compare maybe one agent to one human, but then also how many agent tokens are equivalent. to one human day of like eight hours, right? Like those are metrics that are kind of, but I guess do they even matter because the end user will get the results of what they see is like parallel agents working at the same time over longer periods.

44:17 But then I guess the comparison would be, to make the comparison relevant, it would be cost. Mm-hmm. guess it would be like if I had to hire a full-time pen tester for one day, that amount of money, put that in agents and now I can do a valid comparison. But then even cost is variable for human pen testers. So I guess my point is the comparison is a bit hard to make here because there's so many variables including like variable pen test rates.

44:44 Mm-hmm. No, I I mean I agree with you. I I think like right now we mostly focused on what we see people are looking for, right? And what we see is that sometimes well they have one week to get a pen test being done, right? And if you want to give this to well a traditional firm, well You would get either worst rough to work that they didn't have time to complete or maybe like extreme fees of like urgency because they have to drop other projects and or post them.

45:11 So like that that's still something that we think is is relevant is how fast can they get done with with this with this pen test. Sometimes it's much faster than what they expected and that's something that's still relevant for them. Like I think Yeah, yeah. I mean that that's one of the criteria of course for further, I agree, like I think it's major. I think right now like a pen test we just made it pretty close to human pentest.

45:33 I think even even a bit cheaper actually honestly. But like we wanted to prioritize performances over like over cost efficiency because in the end like the the teams already have their budgets, they just want to have the best performances with the budget they have, right? and then depth is like what actually matters, right? It's how many issues you can find. Like again, like n in average like the agents were able to find more issues that are like the ones that are leading to RCs and others.

45:59 Now in terms of like the complaints, we still like so humans see more of the like bads say patterns that's are important for the compliance side of it's like the security security hygiene that comes back here, right? And I mean that's something we worked on that like about six months ago, so we improved since like based on the failures. Like it's always good to compare, like see what are the differences, and then you can always improve it space on like what are the differences.

46:24 And then like okay, we got to a point where like we can we can match or outperform the like traditional level. what about the other A agents that do this, right? And like to test it we actually created the benchmark application, right? We just put 80 vulnerabilities in this benchmark app, like an actual big application. and then we went over like all of the pen test equivalent that we could find, right? So like strict, double security agents, Shannon, all of those.

46:48 And we ran on all of them to see and compare like issues found but also false positives, right? And it costed a bit because we each had their own model, right? Like OPS 4.6 for Clued Code and Shannon to go over like a 200 n point application that's like It's it's not exactly cheap, right? But it's like comparison that was like we wanted to do and see like where we still have like gaps that we can improve on, right?

47:11 And I'm curious, was this over like one scan or an average of findings over multiple scans? Hm that's an interesting one because it that's true that they're not deterministic, right? So there could be some different results here. we did one scan because like we didn't spend w want to spend an additional twenty K on it. But but like it it's just it was just a general idea like to get on on on how we were performing, right?

47:31 And well I mean again like we we we found like quite some issues and like false positive as well, not a lot. It was very interesting to see that cloud code which is like very bare bone, right? Like it's agents, you give it the tools And like it can it can do like the basic action. It's just one agent, right? Well it's still outperform like some repos like Strix, which is like very, very much engineered around it.

47:51 So like that's something we see where like giving guard reels and like trying to guide the agents and holding its hands, it's really going to be counterproductive and you end up restricting the performances you could obtain. if on the other hand you really use this this concept of like leaving its as much as possible close to all of its capabilities that it can have and maybe like adding new capabilities that it didn't have previously and maybe also managing them to make sure that they can work in parallel on different features.

48:18 Not restricting them to the feature but dispatching them for having a different focus, initial focus, right? well that's that's essentially what allows to get better performances and it's much better than trying to have deterministic workflows, I think. Very interesting, very interesting. I am curious though, it is expensive and hard to compare agents because of non-terrorism, but also just the token cost is expensive. So I was curious, are there any plans to make this benchmark app public so that other people can use their own and then have like community sourced comparisons?

48:54 I mean we didn't have much incentives to do it open source yet and to help our competitors but I mean if there's that's something that's like that can be Let's say creates more proof because like that's like essentially right now it's like this is a trusty benchmark, right? This is not something that people can reproduce themselves. So like it's not ideal to to prove the performances. so in that sense it could make sense to open source it so like we can have a fair comparison.

49:16 Maybe someone else other than us do the comparison. I think that's something that could be helpful as well, right? That could be interesting. like just benchmarking is a whole other conversation we can go into a lot, but like I always think it's interesting because a lot of vendors pick, for example, OWASP Juice Shop as an example of like an app to test. But then I'm always conflicted around like, is this actually a realistic? Like it's kind of realistic, but it's a bit too contrived because it's a bit too vulnerable.

49:42 But some people like set it to the like harder levels or like the harder security level and then try to test. So it's just like the whole topic of a benchmark is very interesting to me because it's actually not easy. Like it's hard to find something that would be a fair comparison to everyone. But I guess that's just part of real life, know? Also, some agents or the way they're made might be better in some scenarios rather than others.

50:03 So there's never going to be something that's consistently good over everything. And and like that that that's also something that's I saw recently like on a message from someone like on the like AI hacking community on Discord. he mentioned that he tested the Xbo benchmark on the most recent models and he actually noticed that the models themselves now had data poisoning because the latest ones actually ingested the benchmark itself as part of the training data. So like na now even in the Xbox you cannot test them anymore because they know about the code content.

50:36 exactly, exactly, yeah, good point, yeah. And like that's even that's even more the case for the more like the the older you get in the to the benchmarks, the more usually it's like presence and and s strengthened in doing data, right? So like like maybe like a bot that auto generates a new app to test and then that gets tested and like every time it's like a different app or something like, yeah. that's actually that's actually like what what we we did later.

51:01 and and like so going going back to like comparing with with the others, like all of those is like what we could access, right? The open source ones or like the proprietary butt that's our that everyone can just self-sign up, right? now the thing is that there are others that we cannot sign up ourselves because like there's sales sales process and like they wouldn't allow the competitor to go through it. and the other point is also that's until now we compared with our own benchmark with our own setup.

51:28 what about like but the realistic comparison is having an external party judge themselves right and so like that's that's what we essentially did right we asked a third party a third party that was very like independent from ourselves and the things that we wanted to benchmark against right and then they selected two applications that they tested on and they had different criteria right they wanted to have it complex enough to be realistic so like making sure that there's different type of authentication, multiple privileges But other multiple languages as well, right?

51:58 And overall like open source applications that are like very much production like, right? real life and like real life like and very close to what customers would use and have as targets on a real pen test, right? And so they tested these two repositories on both both like the product of IKido and Xbo. I think Xbo is like the one that's like made some noise in twenty twenty five and we wanted to compare it against to see what's the grand status and well there's different criteria that they use like previously we used time and depth they used all the other ones they used the setup process so like self-serving versus sales process for example the speed well the speed we already had testing process is an interesting one right that's the things we mentioned before it's about will the agents talk their own accounts out right will they clean up before their own payloads once they're done testing.

52:48 And that's like the the the things that's making sure that's hey the pen test is going well and doesn't break mid pen test. But also that's after the pen test you have everything that's like fine and you don't have to work to clean up behind them for example, right? that's the depth itself, like it's finding more or less issues. le the false positive I think like we did a good job both of us, right? Like actually done Xbo one and two false positives.

53:10 Like that's that's pretty nice. Yeah yeah. Like there's there's this thing where like Like people still have like automated equals DAST equals high false positives, but with the verification parts, well we get to a point where like we we have pretty good pretty good numbers now I think. And then the severity as and I think that's that's an interesting one because it's a bit arbitrary, right? It's like s severity, whether you use CV like even if you use like C VSS which is standardized format, like it's it's always going to be different opinions for for different people.

53:38 Definitely, definitely. And then the reporting itself it's about like the like okay, you get the issues, how actionable they are, right? So the the source where where's the issue, how do you fix it, etc etc. And yeah, like I mean again we were able to like well get the numbers that we wanted in terms of the performances, but this is like only the issues we saw, right? And there's this thing where like when you do a pen test you don't know what you missed like it's pretty hard because you most likely are missing issues, right?

54:07 And you would want want to know which one they were, right? And well there's this C V dataset, well every every month there are like thousands of C Vs that go out, right? Like vulnerabilities in open source software. Sometimes not open source, but many of them are open source. And this there's like thousands of them every month, right? So like even if you want to have The ones that are like after the training data of the model.

54:29 So like in maybe in the last six months, we still have like six thousand C V E' that you can use for for like testing stuff on, right? And like the goal here is like can we find hundred percent of the C V E' again? Can the adjust without the context of the C V E and in the vulnerable version find the C V E again? And try to improve it and see when it when it can and make sure that next time it tries, well it's able to.

54:52 And the one issue with that is that there's still this issue where like you don't have hundred percent of the issues there, right? CVEs are only the issues that were found by other people and reported by other people. but maybe until now there's like this fifty percent issues that's no one ever found and we don't want to optimize to only find the fifty first percent, we also want the fifty percent that was never found by anyone, right?

55:12 Yeah, I don't know. like, will an agent ever reach the ability of finding 100 % of all possible? I don't even know if that's possible. I mean, maybe if you get it's the same with fuzzing. If you give it like unlimited computing, like 10 billion light years of compute, like maybe eventually you find you converge close to 100%. But like the typewriters, you know, if you give enough monkeys. Yeah yeah. Yeah, it's kind of like that.

55:40 I feel like it's impossible to get a hundred percent. But that it it's like like I think the for for like formal proof verification would be the only way, but like that's that's not applicable in real life. I feel like it's it's it's right, exactly. It's like not realistic, not practical. So I yeah, I I think again we can get c as close as possible to the hundred percent, but hundred like actual hundred percent is maybe idealistic, I think.

56:01 Yeah, I think so. but yeah like anyway the the the next goal is like to find this fifty percent no one found until now, right? And well we actually well first have this dataset of like not only the open source application but the one the customers that do the pen test themselves, right? Like we're able to like see what the agents are finding on the pen test themselves and well quite quite a few times we we found issues that previous pen tests missed, so like that was also interesting to see.

56:24 It's that good to compare beyond what was known as issues. And then there's like in open source as well where we found quite some issues. there was like spelt kits, for example, qualify, length fuse, an A10, Astro, Storybook, and like basically every time we scan the repository, we always found like a new issue on CVE that we reported. We often do blog posts about them as well. But like the problem is that now there's this imbalance because every time we scan a new repository, we always find like some issues that we want to report.

56:51 We only have a few researchers in the team that can report all of those. on the other hand there's also the maintainers that s receive many reports from our sides and have to deal with through them because there's not only the IKED reports, there's other also like other other ones that report the CVEs. it it's hard for them to like do all of this work to defend and like I guess AI can also be used for creating the patch, but that doesn't mean that like it's it removes like all of the work, right?

57:17 You still need like the humans to review it, the maintainers still have to go through them and make sure that it's not like slop, right? And like it is it's it's a lot of work so it's it's still good if we can help the defenders like by providing them guidance as well on like the root cause, trying to be as detailed as possible. one thing that's that's is also often useful is providing a PoC so like you like some a script you can run and verify that the issue is real and you don't have to like go yourself through it and try to understand like you just have a script that's that shows it.

57:44 like AI being able to provide a POC because sometimes the developers will argue with you, like, I don't think this is exploitable, but then like here, here's how you reproduce it, like 100 % confidence, like just click or like, you know, run this command and you'll see that it's exploitable. Like that really helps convince people that it's worth fixing. I think that's a huge advantage. they receive a lot of false positive as well, right? And like they don't want to waste time on it.

58:04 So providing a Pok is like instantly triaged, right, basically. And yeah, also like providing help on like how to fix the issues is also often appreciated. Having a human go through it as well to verify it's as a second layer it's also something that can like give more reassurance. we don't want to like automate the issue creation. That's that's like it's maybe too risky in terms of like how much noise it can create so we still have like like people that will create a more detailed approach, maybe screenshots as well, video reproduction to help the maintainer have a high quality report that's like actually worked on for a bit.

58:39 but the issue is that's like we find more issues that we have the capacity that's to report them and like it's i it's pretty hard still to like secure everything when when we have a a small team, honestly. But we're trying our best. Like we basically aim for like the big repositories, right? The one that have more than fifty case that impact as much possible as pos as much people as possible. I think that's like it makes sense because if you have limited resources might as well have the biggest impact with those.

59:03 in the end like that's that's like something we we notice like okay now we're in a time where like issues are much easier to find with AI and like at a scale and it's not just like creativity or non-creativity from the AI. Like as of now it finds more issues even the multi-step complex ones. that's like pretty like it it's like right now we're in a period where like large scale hacking is not unrealistic and it's it's good to have the AI also on the defender side, not just the attacker side.

59:33 Yeah, definitely. Yeah. mean, all of the system you described to find bugs, like eventually people will build similar systems to help patch bugs, right? The same way you can generate a POC from that, you can, you know, kind of like trace through the code and see which part of the code the vulnerability exists in and have a working patch for that and validate that. So I mean, it is also a lot of work, but I imagine eventually we'll get there as well.

59:59 Yeah, I mean like that's that's also what we wanted to do here, right? Like in terms of of of helping the the user is like well we we have AI find the issue, like as you mentioned, you can also find a way to fix the issue, right? It's like that only works for white box, however. For black box you cannot provide a code patch because there's no code to auto patch here. But for white box, there's like this view auto fix button, you can click it and like yeah.

60:21 you can create a PR quite easily. And like that's the private note without check that we saw earlier. It's like adding a check. And like that's a logic issue and not a technical issue. So like it's actually relevant that it's AI and not something that's like autocomplete, change the function for another one. It's something where it's like it makes sense to understand what the issue is and actually fix it properly. Okay, so that was an awesome presentation.

60:43 I think you covered a lot of like super, super good insight. So I wanted just to finish it up by adding some of the context here, comparing AI pentesting with traditional pentesting and just normal DAST. So I think that the obvious one is it has, I'm going to call context aware, but you can give it like real context that a human would understand. Yeah, I mean that that's that's exactly it, right? Like it's it's natural language instead of like programmatic checks, right?

61:12 So you can give them for example instructions on how to log in, you can give them more like if you give it the code it will actually go and understand the code, not just like look for patterns as well. and that allows them to like really interact with everything and test all of the things that other people would consider as being issues, it would also understand this like kind of threat model in the end, right? And and with this model aspect it can like go and find these complex authorization issues that previous DAST couldn't find.

61:40 The other thing is that because you have that many agents in parallel You can really have them work in depth each of them. Sometimes multiple agents on the same aspect as well. We we see that sometimes it helps to have multiple multiple pairs of eyes. And you're you're leaving basically nothing unchecked with this amount. Like if we see enough agents, it's it's not just like the quantity is actually becoming quality, right? It's because you you have everything being checked in depth and you actually can can really basically with smart enough models of course finds all of the issues that's well you would be able to find by reading the code manually.

62:11 Mm-hmm. Definitely, yeah, yeah. Yeah, and then downsides definitely cost. downside there. Like the the cost is going to be more expensive than the DAST for sure. Like it's going to be agents that are like four hundred agents that are running for like many steps. So it's going to like the more the step increase, the more it's there's going to be a bigger context and more tokens that are going to be used. In the end like we still aimed for something that's a bit under the existing pen test budget, so like you don't have to increase it to get seats to work.

62:37 And you can reuse this budget for a pen test, but it's still something that's more expensive than DAST, that's for sure. Yeah, usually people will just pay as much as they pay for a test. I mean, but not necessarily though, depends on each company. So I'm just going to leave that out. And then, so the non-determinism, that's maybe compared to ADAS, but I would say that a normal pen test also has unpredictability. So I think this is not unique to like a normal pen test, it depends on the pen tester you get.

63:07 You know, and like, they having a good day? they, you know, like, maybe. don't want a junior to pen test the bank for example, right? But also yeah, I as as you're saying, it's like also depending on like like we're only human, we cannot be perfect. Like that that's fine, it's just that like it implies that that there's like indeed s sometimes some stuff that is missed. Yeah. But yeah indeed like it's it's the same for a for a Penth, right?

63:30 The agents sometimes are going to go into rabbit holes that they shouldn't or sometimes if they go into rabbit holes that end up being an issue. but like it's it's it's going to not be different. there's going to be sometimes different as well. And that's something as only noticed. Like another reason we're doing all of these agents is because we can kind of fix this like not fix but mitigates the unpredictability by having multiple agents look at different things, right?

63:51 So if one is go is like misunderstanding something while the other is technically to understand it anyway, right? Like the more you parallelize the more you increase coverage and predictability over it Like if you have infinite agents, like probably it's gonna find close to everything but not actually everything eh Interesting cool anything else Yeah, I mean in in terms of the cons, like i it does require more setup than like a a DAS, right? Like you might need to configure different accounts, you mean?

64:21 you say that though, but it seemed pretty simple, like maybe slightly more than DAS, but in a good way because DAS people just like under configure it, you know, or they just don't set up auth, you know, it's like, it's hitting the login page. like, would say the same setup as a pen test. Yeah. No, that that's fair enough. That's true, that's true. More setup than Dast, but similar to Pentest setup. But things like, yeah, like setting up an inbox for each agent.

64:47 I mean, it depends. If your platform handles it, then it's not a cost to me. It's a cost to the platform. But if you automate have a way to framework to automate that, then it's not too bad. Yeah, yeah, yeah, for sure. Cool. Yeah, this was awesome. I feel like now I have a better idea of why, how to, to, or of what AI pentesting is. Glad to hear that. I'm happy I could help with that.

65:10 Yeah, I mean thank thanks for the invitation, honestly. I loved sharing all of these these these challenges and and maybe talk with other people about them because I I felt I felt very sad facing all of those and and being depressed by myself. But no, I I I I I'm joking, I'm joking. Honestly, like it's it's awesome problems to work on and like it's it's great. Like we have a great team at Aikido. By the way, we're looking for more people if you wanna join Aikido.

65:32 and yeah, like overall Like these these are awesome problem we're basically being able to give value by finding more issues to to real companies and like now there's like a lot of a lot of panthers going on that are fully automated. So it is just great to see that what you're building is like better than whatever was being done before. You secure them more and there's actually a lot of use that's and users that are happy about this, right?

65:55 And seeing like all of these messages like, we we love like find the critical after twenty minutes, that's awesome. Like it's always great to see those messages, you know. Definitely yeah, yeah exciting time to be working in Cool, I'll link your Twitter account and things below Thanks for again for the invitation. That's great that's a great chat.