Build Structure - Don't Chase Tools | Ep. 18

0:00 Welcome to Agentic AI execution. I am Lena. Today's episode is about making agentic AI actually ship in production. 95% of agents never get there. what a genetic architecture looks like inside an engineering or and the mental shift that separates teams who are winning with AI from teams who are still just experimenting. Let's get into it.

0:31 Michael, thank you for joining us. >> Thanks for having me. >> Tell us about your journey. What are you doing with AI in your current role right now? >> I was a product manager. I was running a CT office and then I built a TPM team from scratch to this roughly the last 10 years and one thing that I applied to all of it is I'm a little bit obsessed with structure. So I think like when the early terms of AI came up I mean you can simplify that like for instance like with Python scripts or spinning up a machine learning workbench and all of that now it came in a in a full wave of it. So I think many people like have been on on the same journey of automating everything, saving bill hours and then ending up in a pile of tools that never had a system. And I think AI helped all of us to perceive to be faster. But I think there's a few nuances to that and in all of that I'm also writing about that specifically for TPMS on my substack and looking forward to the discussion today.

1:34 >> I have read your subst articles. It's so insightful and helps me in my TPM journey as well. It acts on the agentic side of the AI where I feel right now we are living in about creating more and more agents to automate the task and making that agent end to end so that others can attach their workloads to the existing agent. I'm wondering in your dayto-day how does an AI practice looks? What do you do? You manage a team of TPMS as well. What does AI practice looks on your role? Yeah, I similar here. Everyone is at the moment going through certain phases no matter like which role you are like if you're a product manager, team product design, engine manager engineers themselves and you had this huge phase I think an initial phase of what I would call like experimentation. Everyone was going down what could help him or herself in in the daily life and different funies have access to different AI tools and then you create sort of silos or access to to to certain things because when you work like in an in an enterprise context and then you also need to be very aware on security, governance, policy even companies nowadays give you token limits. I think this makes it more more tangible. So when you when you start to to experiment then everyone is is running towards a goal that potentially is unknown but everyone knows that this might lead to the end goal of having high AI adoption or having high speed or better lead time and if you if you then don't find the moment to switch into building a system underneath then this is getting a little bit out of hands.

3:20 And one thing that we did also at the beginning like we did two long phase of experimentation and then everyone also within let's say the TPM team or the broader engineering team was optimizing for their local things. So that means engineers build front end agents DBMS with program agents and so forth. And then what we did on early to to mitigate that was to startling to to apply to apply an what what we call like an achantic architecture. So in a in a simplified version what you what you have in every company is a sort of is a head of signals. So this could be anything like from Jira data dog and all tool Salesforce where you can bring all of that together. Then in every company there exists a sort of goals or constraints. So what is like your annual goals? What is your monthly goals? what do you want to release? And then at the end of the day, everyone has an has an sort of agent harness. So where are the agents running at the end of the day? So they execute something planning and so forth. And then what you need is sort of some models behind it like that is also coming along like or in the earlier times or the new hype around fable. And then underneath you have also like a sort of a skill library or documents that are serving as an input and then everyone is generating a sort of an output coded like prototype from PL to env prototype or even deploying it further. This is where you also see what is like how do the agents act together and then underneath like you have all of this governance cost privacy and those kind of things in place and one I think like the failure in summary is that you and this I think not only applies to AI and you need to also apply systems to everything that you do like when you think of program management and you always like have a standardized program charter and if you think that in an agent world why not and I wrote about that as well like on on the subsequ like why not put a templates into data and then also fill it with the program documents because at that moment you make it agent readable. This is the way you also provide the context to it and I think many companies also struggle with having this context layer in place. So documents are getting pasted from A to B markdown files being generated and I think this is where everyone is at the moment a little bit there. But what I admired most on the journey was how much people went back to coding like getting hands- on code and also we run a lot of like hackathons and this was clearly our angle to potentially also convince TPMS and the team or others who are not firm with AI to start building or start doing with it or even starting to create their first pulled request in life. It's quite amazing to see. You are spot on. That is pull request going through that whole life cycle. So exciting to see. I was reading a Reddit article which says 95% of the agents are not making into production still. I'm curious how what are some of the things or reason can a DPM try to do in their role to advocate for getting more agents in production and help build a better artificial intelligence layer? I was reading this year is a agentic layer and next year is going to be that artificial intelligence where all the thinking aspect will be done by the agent.

6:32 >> Yeah, I think to to answer the question I would make a differentiation between an agent, a skill and a workflow. So when you think of an agent then it is some sort of an autonomous decision maker. Normally you need skill as well as well which is either it's a model bundle of instructions checklist or scripts you know this very famous like skill file and this is like teaching an agent how to do a specific task and then a workflow is a structured and repeatable sequence of steps that includes also agent invocation or conditional logic and so forth. And I I think like why not many make it at the end of the day is that many forget like to go back really to the basics. So when you when they say okay this process is not the best anymore and then you put AI on top of it and then this is all getting apart because if you just deploy or an agent on top of it then the agent is making assumptions because it's missing the context and by context even like when you think like of a program or an engineering project or in feature development then you need like all of this context that the agent is making the right decisions but you need to also give the right guard rate. So very simply every company has some sort of branding guides or designs and this is what you can very clearly give an agent as instructions in an in a very agent readable format to say okay this is like how we do how the front end should look like from button look and feel pixel size and whatever you do and this is then I think many of us like went too far in applying agents to something very simple and maybe comes back to the the automation question. One thing that we that we did for us is to make an increased adoption of of agents and of skills and workflows. We every skill and every agent in the first step becomes a candidate and this wonders like into a skill library which is called like skills candidates. Then you have the skills evas and like all of that running around and only when two others are using that agent that skill or agent or the skill then it becomes like a real skill which is then baked into the plug-in that you can use in cursor or include and I think this helps a lot like that people start to to think on like what is the best description of a skill like how should an agent act on on top of it and this is how it's you you harden it by giving people positive constraints I would say and the context that they they can get. So it's simply like when you give the context of each program what everyone has been working on in engineing as well as either the code bases in one GitHub repository input all your context into it or in a Google drive folder and this makes it like very seamless that more and more agents like really make it to to production and one thing ne next big wave that is coming is next to the system also like the infrastructure where agents are running and they also like have the classic things reliability performance scalability availability because As human like you always had this limit like of available 8 to 10 hours a day an agent is like always available that's why everyone like you see out there is doing starting with a morning breathing like what is up in my day for today and then or open task. So this makes it very seamless but you need to also give the agent a lot of instruction not that it's making certain assumptions. So for in simple example like when you say want to have a morning briefing then the agent can sum up what you need to do today. But if you give the agent access to your calendar then the agent is starting to say okay you have only meetings today or you have your interview day as a manager then you know that when you have the task you know do this >> I like the idea where you are saying use AI as a tool and then make it a teammate measure it on basis of adoption. If two people start using the skills then make an agent. Have you seen the success with that positive amplifications on adoption of agent? Is it helping or creating more pull mechanism? I'm coming from a place where I know like companies Amazon, Meta, they've created that leadership board where whoever uses most tokens gets rewards, better performance rewards. And I'm curious, how do you see the AI agents and humans succeeding together? And how are we progressing the society in a right direction? Yeah, I think one one thing like you said every role when you look out there on LinkedIn as well as being called tech and many perceive like being lost in AI because what you said like they treat it as a tools race and this leaderboard on token maximization I think is the wrong way to do it and I mean from my perspective anchor everything like on on one thing that doesn't change and this is normally the structure and what you what everyone I think like could benefit on from let's say a little bit of mental model is don't chase tools build structure and this is what we have seen in the last 12 to 18 months like tools change every month the architecture memory policy gates is like durable and no matter like if you build software like all of this let's say let's take a very simple thing like in data privacy policy is very simple like don't share anything like from the company with the outside world like this is a very durable policy and then if you anchor what persists the tool churn stops mattering. So it doesn't matter one day and let's say like you you want to launch a product for sure you can improve the process can make it faster but the core to the essence is that you want to launch features that customers love still the same thing and the second one on the mental model is doesn't change your your job it potential potentially raises your altitude a little bit so you're at the end I sound also a little bit offensive not becoming a prompt engineer is you are deciding what the agent reads what canonical, what needs a human, what never ships unreed. And this applies like to every job because when you're a product manager, you clearly can decide like on how the right coded product or the hardcoded prototype looks, but it's still up to your product taste. And as an TPM can help you a lot like in order to go to go from a tracker to orchestrator to an AI architect where you then can like more deep dive on strategic topics like where you can more deep dive on architectural assessments where you can deep dive more on what are the real risks that AI cannot judge yet versus human in the loop. And the third one is as you said I I think like treat AI as a teammate not as a tool is a tool is something you operate. teammate is something that you delegate to, you verify and hold accountable. So you need to also hold AI accountable. So the more you don't bluntly take the output of AI and start to judge it, the better it gets. And the moment the question shifts a little bit from how do I use it to how do I lead it is maybe the moment you stop being lost. And I think this helps a lot in in in turning a little bit the mental model for every role out there in tech to to a positive thing. And also as in summary like embrace what you have access to because it's just great.

13:34 >> I love your framing. Just don't use AI for using it but really make sure you are enabling the teams to be AI forward. I'm just curious as a leader in TPM with AI enablement. What do you see separates a team who is succeeding with AI versus teams who are not succeeding with some of the AI development? Is it more about creating more process structure or defining the why why we need certain agents? >> I think as a when when you're leading a team then so my my job is as a leader is to get people to use AI. I mean they already want to and one thing that you also see is that potentially the private or how people use it in private life is way more freely than they can use it in a a corporate environment. So they want to use AI. everyone wants to and as a leader it's to build the structure that makes their usage add up to something and I would say like individually they are saving hours and together only architecture makes the compound and if you if you're summing that up at at the end of the day is as a leader like it's there to remove barriers so you shouldn't run like an motivation campaign and if the blockers like were organizationally like integration policy tool motivation was and is never the problem but as a leader like you need to find ways like on to work with this constraints and then also I think you need to design the trust as the last one is aim AI at judgment not just for for busy work so the s teams I think automate like summaries and and status and and so forth and I mean in a TPM world when you want to like really thrive then you can move up to risk dependencies technical little judgment and I think this is exactly like where you could amplify as a leader and one thing I think that AI also unlocked is that as a leader you can also lead by example and for example I mean really theory is that you can also put the direction and write code along build the system along with that and this is like where you create all of those kind of trust that is needed that people also start trusting AI and also like getting on board with it >> if Everybody is leading a team of engineers, front- end developers. I'm curious what is the gap? H how can they close the gap between the AI adoption usage versus architecture gap? What is the first move they should be making?

16:09 >> Sorry. I think that the first thing is what you could do with the team is to give them a green field. And as mentioned before, I I love this hackathon approach and in best case you can do it on site because this gives people the total freedom to do whatever they want. And one followup that you then need to do, I mean you can repeat it multiple times with not moving the needle potentially. I think one one thing that you could do is like make the gap visible to everyone. and and you can layer it then maybe something around the context so that you're missing like memory or that something is broken in your processes or you have certain gates or something is slowing slowing you down and then what next step is you make it very simple like start with one problem and then name an owner who who can do that and then also start like to potentially take the person who has the most AI I savvy in the team and then give the person like what could be the architecture around it like how you you make that happen. do you have something that you that you need to do like for instance when you what I mentioned also before is when every company has its brand guidelines has how front end works and what makes the the front end up then you can codify that and this also has the upstream effect that for instance a product manager who is then building prototypes with AI and then also and without knowing how the front end texture or the front end pieces work together. just with the agent read from that repository and the prototypes are getting like very consistent because you know that everything is company colors like as in our cases our buttons are blue for certain actions or a cancel button is always like red and they don't know that but you can use that and this is like also which makes it makes it quite easy and one thing is and then you can start like sequencing those things but you don't big bang it similar to the TPM rule is like oh can we apply it to the full portfolio like apply to like one program. Can you a simple question like could an agent run a program autonomously? Very simple question but very hard to achieve. But can we go from a from a feature request to a deployment?

18:34 I mean one example that we also did is when you think every product that you have out there has like certain error messages where end users are getting frustrated. when you think of that approach and then could you go to the code base get all of these error messages out and then pattern them and then also like blend it with support tickets and then it's very easy to with what I mentioned before with the constraints then to improve the error message and open an PR or you even can review the PR if you have an agent but these are very simple things that where AI brings you like very far where in previous times like some engineer needs to open the ticket look at it, triage it, and so forth. It makes it like very seamless because you have like all of that in place. And this is a few examples where you could start, but you start with a little bit a little bit smaller.

19:23 >> and then also like celebrate that this things have happened. >> Yeah. Building the first agent and then celebrating the success of the agent >> is critical to keep going and creating more agents in future. I'm curious what we have not covered and you want to share with the listeners as final thing before our closing. >> So I think I mean for me like there are three things like one is don't run away from AI embrace it. and I think this helps a lot in in order to to potentially level up yourself like an example is setting up your private GitHub repository. try to do a few things that that you said I can never do them but maybe AI brings you that far and that might help you also to see what is possible but also to to bring you like on a potentially personal development journey and I think it was never so cheap to iterate also That's what I mean like by you know by embracing it and in a nutshell like on that point by embracing it like adoption is like very individual and meaning like success is architectural.

20:49 So as mentioned like don't chase the tools but build certain structure. and then maybe the second thing is pick one thing that you can build this week. This could be something in your corporate context. This could be something you you potentially have for each project or program that you do a program chart or a template and it might be fascinating if you codify that template and then take all your program master documents from the portfolio and chain it against you will detect a lot of things that are missing in other programs that you have overlooked. This gives you a first thing or also explore in your private life what you could do.

21:28 For instance, one thing I recently built is so my my daughter always like wants to know like how the weather is outside. Like this is one of like either when you leave the house earlier and then you can like build like you know a kid-friendly weather board that sends her a message every morning and says like hey it's raining today please wear your jacket because everyone knows that you don't want to have an let's say a discussion on kids are wearing the right shoes and those kind of fun facts. So you you could also like embrace it and use it. then you can also deploy that somewhere. It's like getting getting so easy. And the third thing is really really go deeper on things that you didn't have time because you might be fascinated how quickly you could get up to speed on topics that you never have thought of. For instance, everyone is talking let's take an example like you you have a lot of like discussions with platform engineers and everyone is talking about like Kubernetes but you never had time for that. One thing that you could do is you can just like open a notebook alm do a research and make an podcast out of it that you can listen to in your 20 minute like ride to the office. So it became like very information became like also very accessible and also in different fashion. So you can also be challenging your creativity with all of that.

22:49 >> Wow, those are such a practical day-to-day tips you have given us listeners around embracing AI, building with AI and making it so easy to go deeper and using it on a daily basis. I'm definitely going to try it and use Notebook LLM to do my own research while drive to office. So thank you. Thank you, Michael. Thank you for joining us. It was a pleasure having you. >> Thank you for having me.

23:19 >> That was Michael. You can find his newsletter, TPM Breakdowns on Substack. Links in the show notes. If this episode was useful, send it to someone on your team who is still in the experimentation phase. Subscribe so you don't miss the next one. Until next time, stay curious, keep building, and bring someone along for the