Fork me on GitHub
Eric Jacobson17:09:44

<!here> does the DORA metric for Deployment Frequency apply to teams, products, or an entire Engineering department? (hoping this is an appropriate place to ask)

Dave Stanke - DORA.dev18:09:18

Hi Eric, it applies to teams (with the general correlation that one team spends most of their time on one service). The data collection is based on โ€œwhat application or service do you most frequently work on?โ€

Eric Jacobson18:09:12

I should measure each dev team?

Eric Jacobson18:09:30

So if I have Team1, Team2, and Team3 each building a microservice for the same application, I still measure each team, right?

๐Ÿ‘ 2
Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and WoW18:09:48

Yep. One might only have the capability to deploy once a week, another might be able to deploy on demand.

Dave Stanke - DORA.dev18:09:03

Yup. Definitely measure each team, and compare themโ€ฆ but donโ€™t make it an antagonistic thing.

๐Ÿ‘ 2
Dave Stanke - DORA.dev18:09:17

Look for opportunities for high-performing teams to teach lower-performing teams.

Eric Jacobson18:09:29

so tempting to gamify it

Dave Stanke - DORA.dev18:09:09

IMHO, gamificationโ€™s fine. But cooking the numbers out of fear is bad. So itโ€™s important to stress that the entire organization (all teams) are looking to improve, together.

๐Ÿ‘ 1
Eric Jacobson18:09:11

I have some teams doing daily deploys and some doing 2-week deploys

Dave Stanke - DORA.dev18:09:35

FYI, the โ€œquick checkโ€ was recently updated and has some additional insights embedded. Worth a look, even if youโ€™ve already done it:

Eric Jacobson18:09:09

great. thank you!

Eric Jacobson18:09:44

"For the primary application or service you work on, what is your lead time"...they mean "you" as in, you are a developer on a dev team?

Manuel Pais, speaker, co-author Team Topologies18:09:11

imho even gamifying can lead to attempts to "look better" by gaming the system

Manuel Pais, speaker, co-author Team Topologies18:09:17

A team delivering every 2 weeks can be bad or can be great, depending on context. Is the cadence driven by customers that do not want more frequent releases? Or is it due to the team working on large batches?

๐Ÿ‘ 1
Manuel Pais, speaker, co-author Team Topologies18:09:23

So, aim to compare apples to apples, that means each team comparing to their past selves, not to other teams.

Eric Jacobson19:09:23

@me1208 how about I frame it as each team is different, compare to your past self, but here are some comps in your area? ...home buying analogy.

Eric Jacobson19:09:21

I really think seeing what your fellow dev teams are doing is such an important reference point. So motivational. Without it, teams seem to assume they are amazing.

Eric Jacobson19:09:47 my experience

โœ”๏ธ 1
Manuel Pais, speaker, co-author Team Topologies20:09:32

That's an interesting point. In a physical office setting the "peer pressure" of seeing other teams' improvements would surface naturally. In the current remote-first setting, however, that might be more difficult. I would say making sure the teams metrics are visible to everyone in the organization (for example via the Team API approach that we talk about in Team Topologies) should be enough. You can/should also have regular presentations by teams explaining how they've been trying to improve their metrics, what helped and what didn't, in a learning/collaboration setting rather than a gaming/competition setting.

Bryan Finster - Walmart (Speaker)23:09:21

I've some experience with gamification of delivery metrics. It's effective.

Bryan Finster - Walmart (Speaker)23:09:45

You DO need to be careful to make sure that managers cannot use the numbers for evil.

๐Ÿ‘ 5
Bryan Finster - Walmart (Speaker)23:09:28

You also need to run the game in a way that teams TRY to game the numbers, but can only win the game if they are moving towards the goals you want.

Bryan Finster - Walmart (Speaker)23:09:41

To keep things fair, I also gamify leadership metrics and compare them to each other and State of DevOps "good".

Bryan Finster - Walmart (Speaker)23:09:40

On the teams trying to "look better". If they get there by deploying faster with higher quality (measure in groups), everyone wins.

Steve Pereira - Co-Author of Flow Engineering16:09:01

One of the best parts of the 4KM are that half protect against the other half, so you can safely gamify them all and maintain balance. If you focus too much on dep freq for instance, MTTR and CF% will likely suffer, and even lead time could be affected. If you track all four you can protect against negative gamifaction effects.

Steve Pereira - Co-Author of Flow Engineering16:09:49

Just make sure you gamify burnout prevention as well ๐Ÿ˜…

Craig Cook - IBM22:09:57

We took a different approach. We measure deployments per service. A team owns one or more services. Those metrics are published, intended only for that teams use though. Teams can view those metrics and decide what they are going to improve next. Management has been trained not to discuss any of those metrics with any of the teams.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)14:09:35

Simple answer: The entire company should probably release more often than individual teams. What is "good enough" and "alarming" really depends whether the team is releasing a mobile app, an internal API or something on a mainframe.

๐Ÿ‘ 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)14:09:30

In a financial organisation, compliance has a significant effect. The regulated software needs jumps through hoops and separation of concerns to be built in somehow and within the same walls it is possible to go really wild, fast and unsafe with some unregulated services for communication between the real estate agent and their customer. So, knowing that it is essentially comparing apples and oranges helps to be humane about it.

Steve Pereira - Co-Author of Flow Engineering15:09:38

@ferrix how do you get the org releasing more often than teams? Don't the org level releases depend on team releases?

Bryan Finster - Walmart (Speaker)15:09:40

Proper pipeline automation fully satisfies compliance and separation of concerns. Pushing on the deploy frequency is a forcing function for waste identification. Any team can deliver releasable changes daily. There may be constraints on delivery, like the app store for instance, but tracking at a staging environment still drives out waste in those instances.

๐Ÿ’ฏ 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:32

@steveelsewhere Many products and teams do not have to release in the same batch. "Something goes out every day" does not mean that something has to be released in every team every day.

Steve Pereira - Co-Author of Flow Engineering15:09:45

Oh yes agreed, team A could be delivering one day and B the next ๐Ÿ‘

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:04

@bryan.finster The pipeline helps in the automated part. However the time taken to run the code through four eyes instead of firing and forgetting does take some extra time. Yes, optimally the non-regulated teams would also care for quality etc but the amount of corners that can be cut when push comes to shove is very different.

Bryan Finster - Walmart (Speaker)15:09:46

Manual compliance really isn't. If there are rules for compliance, then no human can execute those rules as well as a robot.

๐Ÿ‘ 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:32

So, leaving production access open for some developer is not best practice but in other environments it could cost millions in fines and in other places you can push your MVP any way you can cough it up in an afternoon.

Bryan Finster - Walmart (Speaker)15:09:08

No developer should ever have access to production.

Bryan Finster - Walmart (Speaker)15:09:34

The pipeline's job is to inject quality, security, and compliance gates to prevent the delivery of poor quality.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:41

Yeah, exactly what I thought in my last startup. We didn't follow suit and we still ran out of runway.

Bryan Finster - Walmart (Speaker)15:09:43

Two separate delivery problems. Different contexts

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:29

Sure. What I am saying, the feasible target for an arbitrary measurement depends on both where you start and what you are doing. The leaps you can take are somewhat different whether the unit on your error budget is HTTP error codes or deaths.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:38

Pushing deployment frequency to a week is not unlike magic in most organizations with monthly, quarterly or worse schedules. After that, there will be a backlog of other things worth their while, so it makes sense to ignore some of the slowness for a while until it becomes a bottleneck again.

Bryan Finster - Walmart (Speaker)15:09:59

Honestly, orgs where errors will kill people are more likely to kill people with long delivery cycles. Measuring frequency and quality of delivery and focusing on making them better improves upstream quality processes. It drives down complexity, drives out uncertainty, and makes things safer. There is no way any quality or compliance process can be effective with those giant releases. It's just theater at that point. Where I work, I see every delivery problem, even those that can kill people. In the end, the solution is the same: "why can't we deliver daily?" and then daily fix one thing that's preventing it.

๐Ÿ’ฏ 2
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:09

Agree with you there. It's just which measurements you transform first.

Bryan Finster - Walmart (Speaker)16:09:18

We use the cycle time from "started" to "done" on development, how frequently code integrated to trunk, and defect rates to start with.

Bryan Finster - Walmart (Speaker)16:09:08

We're pretty effective at moving the needle on both speed and quality using those.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:07

Okay. The case I am mostly referring to is "started" to "done" in production. There's all kinds of due process from way back when that needs to change before any of the individual teams will be able to do better than deploy to test.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:02

The new way does include compliance before deploying to staging, so there will be a big drop

Bryan Finster - Walmart (Speaker)16:09:43

I'd always measure in groups. However, tracking that metric along with the costs associated and the quality at the end and making that number available to leadership can be a powerful motivator. Also, if the org wants to actually survive, they either need no competition or they need to look at their value stream map hard and see if those processes and handoffs are delivering the expected value.

๐Ÿ‘ 2
Bryan Finster - Walmart (Speaker)16:09:09

If it's manual compliance, it's either arbitrary or we are paying people to pretend to be robots. How do we highlight that issue to executives to get it fixed?

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:15

Yeah, we did the teams*members*salary*weeks in queue math for this group. Management understands now.

Bryan Finster - Walmart (Speaker)16:09:57

I've always had the attitude that if they want CD then anything preventing CD is wrong and they need to help me fix it. ๐Ÿ™‚

Bryan Finster - Walmart (Speaker)16:09:36

If they don't want CD, I don't want to work there. They don't care about their bottom line or the people that enable that bottom line.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:51

Then there is the other cohort of "CD without CI". So, not enough people contributing to the automated tests.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:56

I know. I am also setting people right about that.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)16:09:16

@bryan.finster I think one of the problems that the robots to do the robot jobs are not available yet and the benefits of going down to a weekly release schedule are significant enough while those robots are magically appearing in all the necessary parts of the organisation.

Bryan Finster - Walmart (Speaker)22:09:21

Takes time and leadership who understand.

Bryan Finster - Walmart (Speaker)13:09:44

It always takes investment and training. ๐Ÿ™‚

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)15:09:08

In some teams, they are just waiting for the permission, but that's exceptional