Transcript
0:06 Wow. It's so amazing to be in a room with so many awesome product leaders. I think this is the first time I've been together with so many product people in one room. and so many of you I recognize and know it's it's really amazing. And with many of you, I've had conversations over the last year or two about what is our job and what is the role of a PM. So that's what I'm here to talk to you about. What are we actually seeing practically on the ground? If you remember back a year or two ago, if we can all remember that far back, everyone was pointing at everyone else's craft and saying, "Your job is going to go away." PMS were telling engineers their job was going to go away. PMs, designers were telling PMs, everyone was saying that everyone's job is going away. And everyone was actually really nervous that their job was going away.
0:57 Thankfully, we are not hearing that anymore. What are we hearing today? Well, each job is overlapping. In fact, and there's a new era of the AI builder. If you listen to Lenny Lenny's podcast, if you talk to people at startups, if you read tweets on X, you will see that a lot of people are talking about the AI builder. And the AI builder job is real. There are startups that are not hiring PMs or not hiring designers. They are hiring builders. There's job descriptions. There's builders in small companies. There's builders in large companies. And if you think about it, this has always been the case for founders. Founders have always when they started a company been the builder and really worn lots of hats. So I do believe that this is something that is very real that's happening and also very exciting. But what does this look like in a company with over 10,000 people?
1:52 I'm sure many of you work in large companies that are not five person startups or 10 person startups and you listen to these podcasts and you kind of scratch your head and you say that would be really cool to work like that. I don't have that reality. So what I'm going to do is I'm going to talk to you about what it looks like at Atlassian. I'm not going to go into what the engineering teams are doing. Of course they're doing a lot. They're using coding agents. They're refactoring their code bases. There's a lot of work going on the on the engineering side. I'm going to focus on the PM and what's happening there. As we said, the roles are overlapping, but they are also expanding. So that's what we're seeing is the people everyone can do so much more than they that they could do before and it's exciting. So the job of a PM is really the same as it always was. What is your job? It's to find product market fit. It's to build products that people love. It's also to make sure you're building a business that people will actually pay for your products. None of that has really changed. It's how we do it that's changed. And what has changed?
2:59 Obviously, AI tools have gotten much better and continue to get better at a rapid rate. And one of the questions that I get a lot from some of the people in this room is like, how do you stay on top of it all? And the answer is you don't. So, I don't think anybody could. But the tip is that everything that you read on X, don't believe. You have to figure out what's actually working for your customers. So focus on the customer and what they can get out of the AI tools. The other thing that is changing is context. You've been hearing all about context graphs. You've been hearing about how you need to have an enterprise brain. Well, this is very real. You now can see through lots of different tools the context of your organization. And at lastian we built what we call the teamwork graph that you can see the con full context of your organization. So what does this mean for the PM? Well the PM used to be the person taking notes in meetings, following up on action items, communicating with go to market when things are going to launch. They don't have to do that anymore. We have tools that can do that. The other day, last week, Mike, our CEO, DM'd me and one of the product leads and says, "When when is some feature launching?" Five minutes later, he sent a screenshot of Rovo, which is Atlassian's AI product, and said, "This is what Robo said. Is it right?" We're like, "Yeah." And he said, "Sorry, I wasted your time." And so, what we are seeing is people have access to this information, but we also have to help them along so that they actually use it. There's change management involved, but PMS have a front row seat into how all of this is changing. And one of the things that PMs can do is focus on acceleration. We've all been hearing how things are moving so much faster, but the PM can really help accelerate what's happening in their team. At Alaskan, we think about it as a ship that is going. So you have a ship that is going through to get to your destination. How are you going to get to your destination faster? One of the things is intelligence plus context. So the intelligence of the models plus the context of your organization, how can you put that together to move faster?
5:13 Now going back to the ship metaphor, are you as a PM, are you rowing or are you steering? What is the best way for you to make the ship go faster? So we've done a bunch of experiments. We've organized teams in different ways, measured outputs, and the answer that we have come to is it depends. It depends what type of product it is. It depends on what phase in the product it is. So what I want to do now is actually talk through three concrete examples and what the role of the PM was in each one. Now one is a new feature in an existing codebase. Another is a 0ero to1 product and another is enhancements in a very large existing codebase. All the examples are obviously using Atlassian products because that's what we use internally. But of course you can use all of this is true with whatever products your company uses. So we're going to start with Confluence. So Confluence recently launched two features called remix with robo and confluence slides. So remix is you highlight a piece of text, you remix it into multiple different formats. You choose the format and slides is you take a page and you click a button and it converts it into beautiful slides.
6:23 Obviously these are you heavily based on AI. When the team started, they didn't just say, "Here's what I want to build." They took a challenge of how do I build it much faster and how do I significantly change how I work? And again, in a big company, this is different than you have a clean slate and you're just starting. So, I'm going to go through what are some things that changed for the confluence team. So, first PM started checking in code. I'm sure if I asked you to raise your hand, many of you are checking in code as well. This PM Ya had never written any code, had never even used a terminal before, but she sat down with the engineering partner who actually built a harness for her to use that made it easier for her to check in code in the front end. And so sometimes you have to partner with somebody else to figure out how you can leverage the tools the best.
7:15 She ended up checking in 26 PRs in a month, which was more than most of the engineers on the team. I asked her why. Why did she decide she was going to check in code? and she said there were so many things we needed to fix in the UX and we didn't have enough engineers to do it and I wanted the engineers to focus on higher level activities and this is something I could do. So she felt that this would accelerate the project. Next is EVELS.
7:40 So I assume I hope most of you understand that EVELS is a significant part of your job now and that you're training yourselves or taking classes on how to do EVELs. different teams. Sometimes the PMs do eval, sometimes the engineers do evals. Well, in this instance, the evals for something like confluence slides was so important to get right. That the PMs had done them traditionally and they worked with the engineers to make sure that they relayed how they wanted the team to eval this product and then they got a 2x throughput in eval. The PMS in turn started using the LLM platform. We use arise at Atlassian for debugging. So they were using the platform to find issues with the prompts and then send it to the engineers again faster. I think this one is my favorite. They automated the fixing of design bugs. How many times have you seen the code and it didn't match the design? So they used Figma MCP and mapped the designs from Figma to the actual code and then automated the fixing with a coding agent. And so they could fix something like 14 bugs in an hour. So this again made the product much better. The last one was test creation. This the engineers went from creating a test in a half day to 10 minutes. Just huge throughput improvements. So in this example, the remix launched in 6 weeks, Confluence launched in 8 weeks. Now if you're at a startup, you might be like, "Oh my god, that's still a long time."
9:04 But remember, this is enterprise software that has to be compliant, has to be secure, and sometimes doing this stuff takes the longest to get done to make sure your enterprise customers can use it. So what would have taken before AI about 6 months now took 6 weeks with much tighter iteration loops. So the iteration loops with the customers, with the the beta customers, and internally just fix things much faster. So taking a step back here, what was the role of the PM? The PM was actually rowing and checking in code. And this is what she decided was the fastest thing to make the ship go faster. The engineering manager gave her a clean isolated repository, which is very important. So nothing got messed up in the larger repository. And the team up front had a very intentional contribution model of who was going to do what and they blurred the roles by design and they said we're going to question existing processes. So this is one example.
10:10 Next is a project called Roclaw. Now Robo is our product. Not to be confused. This was not a feature on Robo. This was a zero to one pro project from the ground up written. Now it started with a PM Josh and a designer Kevin. They vibe coded the whole thing. They from soup to nuts got to a working alpha that they got customers using internally. And once they saw they were onto something, then they added engineers to the project. So then we got the PMs and the designers were still checking in code and the engineers were working as well. But Josh started to see that the engineers started veering off track because he was busy writing code and he one day is like wait a minute should I be spending my time coding? Is this actually the highest leverage activity that I could be doing? So he decided to step back from the coding and to focus on what a PM usually does setting direction prioritizing and unlocking. And then things started moving much faster. The fact that he had started out by coding significantly helped him because he could understand what the blockers were.
11:21 He could understand what the challenges were and it made for a much better environment of how he could steer the team. And then on the flip side, he also used AI to help him not do other things. I hope most of you are not writing weekly reports by hand anymore. And if you are, you probably don't want to admit it, but please go and write build an agent to do it. So Josh used Rovoclaw and built an agent in the product he was developing to give the weekly updates on itself which were super amusing and so he no longer writes weekly updates. So in this instance the PM started out rowing and then shifted to steering because it was a different phase of the project and that's what he needed to do to make the project go faster. So he was unfocused on unblocking the team whatever he needed to do at that point in time and he had the ability to contribute code which really helped him. So he focused on his highest leverage activities. Okay, last example, Jira. Most of you have probably heard of Jira. Many of you have used it.
12:29 Jira has been around for over 20 years. It's a huge complicated codebase. We have things like it started on data center and then moved to the cloud. It it supports isolated cloud. It supports Fed ramp. It supports the customizations. and you have we have hundreds of thousands of customers that you can't mess up. So this instance of PM checking in code can be kind of dangerous. So we didn't want to do that. But we also had to bring Jira into the AI age. And our goal was to make it AI first. And so the PMs and the engineers were building AI features in Jira to help the customers use AI to do their software development, which was really fun. like how are you reinventing the software development life cycle with the tools that you're building and how do you make sure you can assign to agents how do you make sure you can seam seamlessly use agents for everything in Jira so this was a big project for the team and from in progress to shipped their throughput was about 3x what it normally was and they shipped in about 10 weeks 22 userfacing features how did they do it what were the changes they made so let's start with prototyping.
13:43 Again, everybody here is prototyping that. I'm 100% sure. But what did they do to make the prototyping more effective? So, first they started with Loom. And in a Loom video, you could record the UI that you wanted to change or you could record here the Figma designs with a talk track or you could just brainstorm that goes into a recording and then Loom has the ability from the recording to automatically create work items. And then from the work item the PM can move it to in progress and then that automatically kicks off a coding agent. So this whole thing was very streamlined and the coding agent wrote code in the actual front-end repository and used a managed agent in the cloud.
14:29 So PMS didn't have to deal with their local environment. So then you got code that was already compliant, accessible and in the Atlassian design language. So when they were done, if they wanted to implement the prototype, it was much more seamless and the handoff to developers was much faster. So having the right environment for prototyping is key of getting it faster into the engineers hands. So this was a big unlock. Next was triaging feedback. As you can imagine, if you're use if you're making changes in Jira at a company like Atlassian, everybody in the company is using it and you get tons of feedback.
15:08 So, what did we do? We took the feedback and the bugs that came in through Slack and we used the Jira agent in Slack to triage them and then automatically send them to the coding agent for fixes. Again, saving a huge amount of time for engineering. And then we also did a lot of user studies for customer insights. So here we used Jira service management to take and review all of the videos from the customer insights. We got over 900 pieces of feedback. We use used a robo agent to then triage and categorize them. And we were very happily surprised with the quality that we got of the triage. So then we knew what to send to engineering to fix. All of these things together in this instance the PM was clearly steering. The PM had to focus on unblocking the engineers in order to make the project go faster. That included processing feedback internally, externally more effectively, and building prototypes that were set for reuse. And as I said in the beginning, PMS did not check in code to the production environment. So, if you put all this together, what are things that PMs are doing now that they weren't doing before? They're obviously prototyping and testing. They're writing evals. They're going deeper on customer feedback. They're consuming information at a much higher rate rate. And all of this works if all the AI tools understand the context of your organization. And that is built on top of at Atlassian the teamwork graph. And then what are they not doing? They are not doing manual updates. They're not compiling research by hand. They're not creating SLI slides from scratch. I would love to say that there are no synchronous meetings, but that would be an exaggeration. but there are I believe fewer synchronous meetings.
17:00 Well, how did we do this? What was our playbook at Atlassian? When I talk to people at startups, they tell me hire people that are AI native. Well, we have a lot of PMs already who are amazing PMs at Atlassian. So, we want them to become AI native. We want to help them be able to learn the skills that they need to learn in order to build products for our customers. So we introduced something that we call the AI fluency index and this is to help people understand where they need to go and we have six capabilities in the fluency index. Things like do you understand how to use the tools? Do you understand how to write emails? Do you understand how to automate data insights? Can you prototype? And do you have the level of technical literacy that you need? And then we have an index from one to five.
17:49 One is curious, five is pioneering, three is capable, and we do not expect every PM to be a five at everything. And this is not a ladder. This is for development. So that you have a northstar on the skills that you need to be developing because of course this changes all the time. The ladder is about outcomes and what you produce for your customer. This is helping PMS understand the skills that they should be attaining. We do expect people to get to an L3 in all of these over time and you should understand what your team needs. You should understand what is the most important point for your team and that you focus on getting to an L5. How do we train PM? So we want them to become AI fluent. we've tried a bunch of different things and the thing that stuck the most was our AI builder weeks.
18:40 So, our AI builder week is once a quarter. We take a week where you're not doing your regular work and we train PMs and designers. It starts with bringing in external speakers. Shout out to Claire who came to our last builder week. And then we have PMs internally in the organization who are more advanced in certain skills training other PMs. And then we have people do a project. So it depending on what it is, they can work with their teams, they can work with other people that but they do a project to use the skills they're learning. This has been incredibly successful. We've had over a thousand people go through our builder weeks.
19:20 We've built over 120 new workflows that people are using after the builder weeks. So what are the focus and we focus on one thing each time. So the first one was prototyping. The second one was eval. Third one was how do you build agents? And then the the most recent one was checking in code. We have talked to customers a lot about this and we've actually built an AI builder week in a box. You can find it in our website where it shows you kind of the agendas that we've had and how we've utilized them. Okay. So now we've told the PMS where what they're going to achieve.
19:52 We've trained them. And what is a question I get from customers all the time is how are you measuring outcomes? I'll be honest, we haven't figured it out. I don't think anyone has figured it out. I talked to a lot of people about this and everyone is I mean it's the age-old how do you measure developer velocity. we are measuring PRs deployed not PRs written but PR is deployed to production. We're measuring features delivered. It's a little tricky to figure out when the feature started but from idea to delivery in the customer's hands and usage. Obviously you want to have the right OKRs. You want to make sure that your OKRs being met and like overall throughput.
20:30 Sometimes it's easier to measure the throughput of an individual team versus an organization. So you want to do both. You want to do make sure teams are working faster and organizations are working faster. So if you have ideas of how you're measuring, let me know. But we are working actively each quarter. We have experiments that teams are running in different ways and we look at what we can measure on the outcome. So bringing this all together as I said the roles are expanding each role is expanding and I think the way I look at it PMS can do more to get their jobs done more effectively and better and they can do more to deliver more to their customers.
21:10 I think this is the best time in the world to be a PM. I think it's super exciting. I am an optimist and I think we'll all be here 10 years from now. But I really think that this you guys are all so lucky to be here today. So that's it. Thank you.