Agile in Action with Bill Raymond
Agile in Action with Bill Raymond
Using metrics to drive value delivery
[00:00:00] Intro spot
---
Bill Raymond: I was talking to someone who was frustrated with SCRUM and agile, and I said, why? And they said, we're working on these very advanced algorithms and we just can't seem to get them out,
And I said, well, show me how you're measuring the success of the project. And it was around velocity.
Yeah. And I would call those metrics that matter less, right?
Dave Witkin: That takes away time that we could be spending on things that matter more. And I think that's a key part of the challenge.
Bill Raymond: Bill, we have no illusion, there's no such thing as a perfect metric. this is why, for every metric, I thinkcontext does matter.
Dave Witkin: That's that. I'm going to give you some. I'm not going to cop out here and give you the old consulting line that, it depends.
[00:00:44] Intro
---
Speaker: Welcome to the Agile in Action Podcast with Bill Raymond. Bill will explore how business disruptors are adopting agile techniques to gain a competitive advantage in this fast-paced technology driven market.
Bill Raymond: Hi and welcome to the podcast. Today, I'm joined by Dave Witkin, founder and managing principal at Packaged Agile.
Hi Dave. How are you today?
Dave Witkin: Hey Bill. Great. And thanks for having me.
[00:01:15] Today's Topic - Using Metrics to Drive Value Delivery
---
Bill Raymond: I'm looking forward to our conversation. Today, we're going to be talking about Using Metrics to Drive Value Delivery. I think this is going to be a good topic. Before we get started, could you introduce yourself?
Dave Witkin: Yeah. And if it's okay, I'll do that by way of a, a short story about history of my career. I actually started in IT back in the late 1980s. And I remember I was involved in software development, generally giving requirements and we were developing an inventory management system, was one of the big ones we worked on.
And I recall it because one of the things that we did was I would sit side by side with the developer. He was building the tool. I would give him immediate feedback, he'd ask me questions. And within a day, he would build something and I'd give him immediate feedback, so on and so forth. When a cycle worked, great. Then in the mid 1990s I went to work for one of the Big 5 consulting firms. I don't know if there's five anymore, but one of those big consulting firms still around today. And I started working on really big systems. Sometimes these required 50, 100, even 500 people to build. And this is still largely what I focus on. But in the 90s I figured out that at least for these big 5 consulting firms, they built software very differently than I had in the 80s.
So instead of sitting side by side with developers, we created these huge documents, months, even years, strategy documents, requirements documents, design documents, test approach documents, you name it. And I would guess looking back, probably 80, maybe even 90% of our effort went into building documents and perfecting them, right?
Dotting the Is, crossing the Ts, making sure the formatting was right. And each document got handed off to somebody else. And when we did that, massive information was lost in the process. So, as most of us agilists know, humans don't communicate very well through documents. Right? But I thought, I'm just a junior person, this must be the right way to build software. Here's this Big 5 firm, they're making billions of dollars every year. They hire people from these top-notch universities, smarter than me. But you know, when I got my sea legs and started getting more confident, I realized we failed much more often than we succeeded.
And when I say failure, I mean it could be complete failure where we delivered absolutely nothing. More often than not though, we would miss things by years or deliver half of what was originally envisioned. And then there it was riddled with bugs.
So anyway, I knew there had to be a better way and that's when I came to agile. And I read something in the early 2000s by Ken Schwaber, one of the co-creators of SCRUM. I'm guessing many of your listeners probably know that. And I immediately saw how SCRUM could change the game completely, and I loved it.
So 2004, almost 20 years ago now, I left the Big 5 firm. I created a new company Packaged Agile, and over time we focused more and more on agile methods.
So bringing us to where we are now, we are 80-90% government-focused, US government focused and we really pride ourselves on, as I say, real agile, avoiding the zombie agile, trying to get the real quote value that most people feel they were promised when they were led to believe agile was a better way. So we try to make that a reality.
[00:04:30] What does Agile mean to you?
---
Bill Raymond: That's great. Thank you. And I guess this is interesting. This leads into my next conversation, which I ask everyone, what does agile mean to you? You already shared a bit of a story there, but I'mcurious from your perspective, if you're sharing with someone what agile means, how would you define it?
And also, follow up question to that, I don't think anyone's used the term zombie agile on the podcast,shockingly in 80+ podcasts, and you're the first to use it. So maybe what you could do is explain what agile means to you and what zombie agile is.
Dave Witkin: Sure. Well, so that's actually, it's interesting. Part of how I was going to propose the answer was let me tell you what it's not for me, and then I'll move into what it is. Because you've probably got a lot of affirmative answers and I'll give you that as well. But let me start with zombie agile, which is a term, I think there's a group called, The Liberator.
For your audience, they were worth reading, they put out some great content. You could probably search for them, The Liberator's Agile, and it's a term I first saw with them, but you know, even work with SCRUM and Dr. Sutherland, those folks, *we talk about fake agile or really, it's just agile that delivers none of the intended value.*
*Sometimes it's worse than waterfall. Sometimes you get mild improvements, but nothing like the types of results that we know we get when we apply the values and principles effectively. So any time you are in, let's take SCRUM, you're insprint planning, you're in daily standups, daily SCRUMs and it feels like a waste of time.*
*People are not communicating, they're just going through the motions, they don't want to be there, zombie agile, not what I consider what we're really about. That's not agile that's going to deliver value. *
So from a more affirmative way, I'd say agile is really first and foremost, figure out what the customer really needs and wants and force ranking it.
We've got some studies showing, there is a tremendous amount of waste in what we build because we, at least in the software development world building those types of products, we often build things that are rarely or never used. Huge problem. So what do our customers really see value in? And we force rank them.
So everything is not a top priority, often a problem. We deliver things in increments, right? Working products quickly or at least partial working products, hours to days, not weeks to months. We get fast feedback on those products from people who will actually use them. We don't give them to testers who have neverused the actual application, will never use them in production.
You can put things by testers as well, don't get me wrong, but that's not the number one litmus test. It's people who will use them. And then we inspect, we improve both the product and the process in short cycles. SCRUM is very big on that, as is LEAN. And finally I'd say, I've given a lot in there, but maybe one of the last ones, *it's not specific to agile, but I think it's a key component of success. We create an environment that fosters a high performing team. And when I think of something like SCRUM, to me that's really what the intention is. It doesn't have to be about software. If you look at it, the framework itself for SCRUM, it's very little about software. It's about creating a structure where you get a team of people and how do we overcome their biases and get them to work effectively together.*
*And I think that's really the heart of it.*
Bill Raymond: Thank you. And if anyone's listening to this podcast right now for the first time and you're just jumping into this world of agile and you want to know what SCRUM is, we do have a number of podcasts specifically talking about SCRUM. We have a season of SCRUM through 2022 and 2023. And the best one to look up is our intro to SCRUM, which is Predictive Versus Adaptive Project Management and,I don't even want to use the word product project management because we are often talking about products and not projects. But that would be a good way to get started.
[00:08:26] Defining the term "value"
---
Bill Raymond: And I do think though, that you've mentioned the word value quite a few times now, and I think it would probably be good if you kind of give a definition of what that means because the rest of this podcast, we're going to be talking about measuring agile.
Dave Witkin: Yeah. And Bill, I would say, this is where unfortunately, I think a lot of people go off base from the start is that we don't define it. And that causes lots of downstream problems. So if you'll bear it with me, I'll define it through a little bit of a story, this is actually a true story of one of our clients.
I won't give the actual names to protect well-meaning people. But this has actually happened and I'll mention at the end, I think it happens a ton in one form or another, but so there was a senior company executive we worked with a few years ago. Let's call her Tasha. And Tasha really wanted to be a hero at her company.
She didn't use that term, but you could tell she was very driven. She really wanted to improve the company. And she thought the best way to do that was, hey, to find a big juicy problem and be one of the people to solve it. And her leadership saw a lot of value in that. One of those problems they asked her to solve, there was a software application,I think universally, people said they hated it.
It was just super old, it might have even been old green screen, mainframe kind of stuff, it's hard to remember. But it wasn't easy to sustain. So she campaigned, she did some estimates, campaigned, got a hundred million dollars, basically budget to address this problem over the next couple years.
She hires consultants, she engages internal employees and they start work on rebuilding this existing application piece by piece.
So it already existed, they didn't have to start from scratch, but basically re-platforming and designing. And Tasha did believe in really good design. She got some great feedback on that. The team made sure every screen was beautiful. It was rebuilt, it was tested, it looked great, it was modern technology and she got a lot of advice.
And she also understood a little bit about agile development from where she was before. So every part of the application was rebuilt and tested in short cycles. I think they used three weeks. So our team's super efficient, right? They got all the efficiencies that you would hope from an agile approach.
And they actually, they had three years. They finished in about two and a half years. Okay? So they're six months ahead of schedule, at least their schedule. Company's proud, everyone's excited. And then they release the software. And that's always where the rubber hits the road, isn't it? And it wasn't horrible, but the reaction was very mixed.
And what I want to get across, I guess is the mix, right, was based on different roles. Who used the software? So we had internal users of the software. Internal users who looked at it and they loved these new screens. Much more efficient, they were easy to use, you could get from tab from field to field.
It worked well. And they had a pretty good set of, we call them edits to make sure that the data that they entered was clean. But there were even more external users, people outside the company. And the screens were still pretty, right? They were consistent. It really didn't solve the problems with the old system.
And the problems with the old system were poor data quality, checks right at the point of entry. So what would happen is you'd enter some data and then it would be 10 steps later where you would figure out that you didn't qualify for something. And thenyou had to go back and basically start again.
You lost some of the state, it wasn't well saved. So because with sort of a different product line or something, if you didn'tqualify. Her team was also pretty smart. They built some, your, your listeners may have heard of this telemetry. If you don't know, telemetry is sort of how do we look at the actual application, get some data on it to figure out how it's used, what's the uptime? Is it performing well? Right? When you hit save, how quickly do you get control back?
And one of the pieces of telemetry that they used helped them evaluate what's called the abandonment rate. And the abandonment rate basically says, for every person who starts filling out this loan application, how many people actually complete the application?
And you want low abandonment rates, right? Low is good. You want people who start the process to finish. Because if they don't, it tells you you know, there's some problems here. The abandonment rate on the new application was over 70%. So seven out of 10 people in this brand new, beautiful system who started filling out a loan application quit before they submit it.
And worse, at least for one other stakeholder group, it turned out, some of that telemetry told them that 75% of the functions the team rebuilt, really replatformed in the old application, were either never used or almost never used. So some of the executives took that to mean, and it's reasonable, that they put a hundred million dollars into this, literally 75% of those million dollars were completely wasted, forgetting about the external users who were not happy and were abandoning it.
The end result basically was the company's competition, they continued to take market share. Tasha was smart enough to find another job before she could be let go.
And so, to sum it up, I'll say the moral of this story in terms of value is that there, I think you can see there isn't just one view of value on most, especially large initiatives. We had the internal user's view of value, which was how quickly can I complete activities I need to do my day-to-day job?
Right? So that's one. And frankly, they were pretty well satisfied. The external customers or potential customers saw value in some things that were not delivered. Good edits, making sure data was clean, early identification of when they were not eligible for certain products, and maybe the ability if you weren't eligible for one product to save the data and then move it to another product.
So they were unhappy, a very different view of value. And then finally, and this gets overlooked a lot, not everywhere, but we have the senior executives' view of value. And what was their value, right? Their value was, I don't want to spend one dime more than is necessary to solve the problems with the application.
And so they saw 75% of these functions were rarely or never used. So they took that to mean out of their a hundred million dollars, Tasha was partially responsible for wasting 75% of them. And this is a story, this is one client, butI've probably been at, who knows, 30, 40 clients at this point in my career.
And I will tell you, I thinkand maybe your listeners can confirm this, at most organizations, something like this maybe less dramatically, happens pretty much every day when we're building software products. And the results are mixed at best because there is not one view of value. And I think, maybe we can get into this later, value sometimes is also defined, not by what we say, but by what some of our leaders do and what they pay attention to.
Bill Raymond: This was a great story because there are different levels of value, if you will. Different stakeholders had different concepts as to how they were defining the success of this particular effort. What are some of the other challenges that you might see when people are trying to use metrics to drive success?
Dave Witkin: One I would say is, and there's a phrase in metrics you hear a lot, which is, you get what you measure. And we do a lot of government work as I mentioned, and government, and this happens I think in the private sector too still, often, especially for large programs, they want baseline project schedule.
They get uncomfortable when those schedules change, which is what the baselining is all about. It's a subtle way of saying you should really know where you're going from the start of the project before you go spend my money, right? So as soon as you say that, sticking to that detailed schedule is a requirement, that you've baselined it, you need this change control process to change it. The focus of many of the people, at least partially becomes how do we stick to the schedule?
Bill Raymond: And so you start getting people adapting what they do, how they report status, lots of things just to comply with the schedule.
Dave Witkin: And I'm not saying schedules aren't important, right? Schedules may be very important. What I want to get across though is unintentionally for some leaders, and I'll give an example of a program I'm on right now where the leader has now made schedule adherence, the most important metric on the program.
The most important, and I would say, this is sort of an insidious, often overlooked reason why projects run into problems, maybe even sources of failure. Because we start paying attention to things that matter less. Again, I'm not saying schedules don't matter at all. I think they do, but they only matter to the extent that whatever your building delivers value for the consumers that it's for.
Anyway, so I think part of the challenge here is we finish these large projects and very rarely you got lessons learned after big phases or maybe after a large waterfall program. And very rarely do you hear anybody say, you know what the root cause of our problems was? It was the fact we focused so much on sticking to the schedule.
People don't say that, but at the same time I watch and I see the machinations, I see everything they try to do to stick to that schedule. The little, maybe bending of the truth a little bit about, we started this and we finished it on time even though it wasn't really validated, right? What does finished mean? So what's happened is we have just unknowingly picked our top metric.
So you get what you measure for better, for worse. And as I mentioned, this is happening on one large program I'm on today and it's as simple as the person at the top that the hippo, sometimes called, the Highest Paid Person's Opinion.
That person makes the most important meeting. Well, one of the meetings she does not miss and she makes clear everyone needs to attend, is where she gets a status update that is largely focused on the schedule. Are we adhering to the schedule and some other risks and red, yellow, green? But frankly,those risks are often overlooked.
So I think what people will do then is they're going to pay attention to what their leaders actually do, not what they say the key metrics are. So one of the big challenges is when we do define some value-based metrics, we need to help make sure our leaders believe in them and what they do supports and shows everybody else on the program that's really what matters.
[00:19:25] Similar challenges in agile
---
Bill Raymond: And that example was a waterfall example but we have some of the similar challenges in agile too, don't we?
Dave Witkin: 100% right. Yeah. Do you want me to give a couple examples?
Bill Raymond: Sure.
[00:19:38]
---
Dave Witkin: that's that. I'm going to give you some. I'm not going to cop out here and give you the old consulting line that, it depends. Let me actually start with, I think the best one it's actually a class of metrics, and you may have had some people on to talk about this already. But something called objectives and key results.
And for the listeners who don't know, it's often called an OKR. It got popularized in the 1990s by Intel, the chip company. And they wanted a way to keep people focused on those things that matter more. So they said, here's what we'll do. We'll kind of build a framework, and that framework has three pieces.
The number one piece is, what's our big rock objective, right? Little Stephen Covey is in there. What are we trying to achieve? And so for Tasha's program we might say, what we were trying to achieve, at least from the external user perspective, was to reduce the abandonment rate. Okay? So our big level goal is how do we reduce the abandonment rate? And how are we going to measure if we've done that?
So then you get to what are those key results? How do I measure if we're successful? And so we might break that down into a couple different key results. One might simply be okay, we have succeeded if at the end of a certain timeframe, let's say six months, we have reduced the abandonment rate by 30%.
Now you have a key result. You have something specific, it's measurable, we might say it's binary, right? There's not a lot of wiggle room either you did decrease that rate by 30% or you didn't. It's very business-focused, right? Very customer value focused. It makes you really think about what they want and need.
So often you want to talk to them and run this by them. After that, so now you've got your objective, you've broken that down into key results, you take that one step further and now you build initiatives. And those initiatives are designed to address those key results.
So if we go back to the story with Tasha, she didn't do this at all. She completely overlooked this. She said, I know what the problem is and I'm just going to start working to solve it, because of a lot of, there's a ton of anecdotal data. And look, maybe she looked at the system and said, this is terrible, she tried it, green screens and it was bad. But there were multiple stakeholder groups, there were multiple customers, all with different views of value.
So if you're going to build an application, one of the beauties of something like an OKR, OKRs is you can define OKRs for each of your key constituent groups. Could be, you might have an OKR even for executives about how much of what we're going to rebuild, going back to the Tasha story, is actually used on 80% of business days and how much is only used less than 20% of the time.
And we want to focus on those items that provide more value to more people. And that might even become an OKR for them. So it really helps avoid the types of problems that we talked about in Tasha's story. So that's number one. Again, it's a little cheating, it's a classof things, but the reality is you do need to, if you're going to talk about business value, you have to tailor it to the context.
Bill Raymond: That's an excellent example. And are there any others?
[00:22:48] Net Promoter Score
---
Dave Witkin: There are. So, we got time? I'll hit maybe two more.
One I like, again, imperfect, but there's a lot of positives about it. It's called Net Promoter Score. And you may have seen this or your listeners may, without even knowing it. You ever go to a website Bill, where at the end, you're just about to leave, and at the bottom pops up this little scale. It says, would you recommend this software product to your friends or family? And it's a scale, it says 0 to 10, that's all it is. You just pick one of those items from 0 to 10.
Bill Raymond: The 0 to 10 scale, anyway, got popularized and it's part of what's, I can't remember if it was Boston Consulting Group who popularized it, but Net Promoter Score it's very tough, meaning that if the people who score you 0 to 6 they're considered detractors, counts against you, right?
Dave Witkin: Then the people who score 7 or 8, they're neutrals. They don't count for you or against you. And then only two of the scores, 9 and 10 are considered your promoters. These are people who are likely to sing your praises across the globe. They're your tribe, if you will, they're going to help you probably get more customers. It's difficult to get good scores by design and the way it's designed, it's usually measured on a negative 100 to positive 100 scale, where it's considered at least pretty good if you get anything above zero, right? Because again, 70% of the scores are just under 70, counts against you.
It's tough. Part of what I like about it, it's a little bit of a broad sword, right? It doesn't give you an incision on exactly where the problem is. But ultimately, we are trying to keep audiences happy, right? Keep our customers happy. Now we need to dig in and figure out if they give us a 3 on a scale 0 to 10, they're detractors. Why do they give us a three? But just getting that Net Promoter Score, it's sort of a very broad thing and you can measure it over time. What's our trend? So a trend of that Net Promoter Score, very important. We can even use that during longer development cycles, which I don't promote.
But if you have to for your stakeholders to say, look, are we happy with how this is going? Are we getting enough feedback? And you can change the question. So we often do this for stakeholders, right? Maybe those executives, we do it for what we term consumers, people who are actually day-to-day using the products we're building.
And then for team members, which often surprises people. And by team members, I mean people on the agile team who are building the product. And people will say, well, why do you get their Net Promoter Score? Well, many of us coaches,and again, some of the, I don't want to say elite, but even I know some of the signers of the Agile Manifesto are big proponents that people who are working on software products really want to do a good job. And they are often your first line to be able to tell you, look, we're not really headed in the right direction. So people see the team member, sometimes called employee or ENPS, as possibly a leading indicator of problems that may come up later.
Bill Raymond: So you have a number of different approaches to measuring value and delivering value, and I think these are great examples, and I know you have a few more. But I think it would be nice if we could wrap up by just talking a little bit about how, if you're not tracking these metrics, what might it look like if you started to focus on them?
What would happen if you switched gears and said, we're not focusing as much on performance, which is still important, but we're focusing on value and looking at that instead and we're going to really shift directions. What might it look like if you do that?
Dave Witkin: Well first I'd say, one of the things that we want to look at is starting small. So great for us agilists, right? We want these small batches, small things. And again, because if we try to focus on too much, we get lost. Sometimes we'll talk aboutWIP or work in progress. And there's lots of evidence that trying to do too many things at once, really not only hurts you productivity-wise, but hurts you quality-wise. So pick one or two, right? And you can probably find a good coach, there's plenty of people who just focus on OKRs out there and good literature.
And maybe brainstorming it with a team and say, look, if we were going to pick one OKR that we might put in place, just one to start, what would it be? Where would we want to focus? And maybe that even leads to a survey or some conversations or some focus groups with different customers, which are things, again, as great agilists we know we want to be doing anyway. So it kind of forces you to do that, if you will, to do it well.
Once you do that, then I'd say, assuming you do not have the authority to put this measure in place yourself. So let's assume that's the case at least to start, I would say talk to your colleagues. Do you have a community of practice? Or maybe it's your whole team that you have all agreed you want to do this and you have a good rationale why.
Because these leaders, what I often see is somebody, a very passionate agilist, they go to a leader and they say, I want to do this because of X, Y, Z. And there's nothing wrong with that, but there's power in groups. These leaders, if you come with 10 people saying the same thing, we've talked about it. There was a lot of brain power that went into that, right? And they are far more likely to listen to you if you come with a little bit more ammo, if you will, meaning a lot of other people who have agreed, hey, this is a pretty good idea to try. So once you do that and maybe even before you come, you're going to try to put some specific steps in place.
So for example, with Net Promoter Score, and let's say you are going to go outside the organization and to your customers. Well, to do that, what I've run into before is we've got to go to the legal department before we were allowed to measure those external customers or to marketing or to some other stakeholders.
So you need to come up with some of those, what are my steps to actually get there? And then for NPSs, as a great example, we talked about something popping up on the screen. You need to think about those logistics. Do you have a tool, a widget already available to you that's going to integrate with what you do and allow you to collect the data? It doesn't have to be hard, right? So for us with the EMPS or team that promoter score, we simply use Google Forms or Microsoft Forms, it does fine.It doesn't have to be complicated. And I'd say to keep it relatively simple, that's generally enough to get started.
That's a good idea. Just get started internally. You can always look at this external element. It takes baby steps sometimes, I agree, especially when you have to work with legal, but you can get it done. Yes, agreed. And there's plenty of widgets that'll do that, right? For web-based software that you just put and it'll put it on there. Or again, it doesn't have to be that complicated. Google Forms, Microsoft Documents, SurveyMonkey, you know, and you work with some of your constituents to say, hey, is it okay if I send this to a hundred people? Would you mind? And then you need to get people to respond. Would you mind telling them this is important and asking them to take 10 minutes to do it? And I will tell you, at least on some of the larger programs, people say, I don't want to fill out a survey. There are pieces of the constituency, especially if the leaders say, I'd really like you to do this, can you please take five minutes? Put it on your calendar. We sometimes get 50, 70, even 80% response rates. Not all the time, but occasionally because these leaders make it important to respond.
Bill Raymond: I think there's some great ideas right there.
Dave Witkin, it's been a pleasure talking to you today. Thank you so much for sharing your thoughts and ideas. Is there any way someone can reach you?
Dave Witkin: Yeah. I'm passionate about this, so please feel free to reach out. The best way I think Bill, is through Linkedin. There's only a couple Dave Witkins on there, so if you search for Dave Witkin and maybe agile, you are likely to find me.
Bill Raymond: Yeah, that's great. And you have so many things that you've shared with me over the last few days and I really appreciate the time that you've spent to share these stories and kind of bring it home with some case studies. And yeah, absolutely, we'll make sure that your Linkedin address is on the https://agileinaction.com website.
Just look up the Dave Witkin interview, and also if you're listening to this in a podcast app, just scroll down to the show notes, the description, and you'll see the links there as well. Thank you so much for your time today, Dave.
Dave Witkin: Wonderful. Again, appreciate you having me.
[00:30:55] Outro
---
Bill Raymond: Thank you for listening to the Agile and Action Podcast with Bill Raymond. Subscribe now to stay current on the latest trends in team, organization, and agile techniques. Please take a moment to rate and comment to help us grow our community. This podcast is produced in affiliation with Cambermast LLC, and our executive producer is Reama Dagasan.
Speaker: If there is a topic you would like Bill to cover, contact him directly at bill.Raymond@agileinaction.com.
