Hi Eric, it applies to teams (with the general correlation that one team spends most of their time on one service). The data collection is based on “what application or service do you most frequently work on?”

I should measure each dev team?

So if I have Team1, Team2, and Team3 each building a microservice for the same application, I still measure each team, right?

Yep. One might only have the capability to deploy once a week, another might be able to deploy on demand.

Yup. Definitely measure each team, and compare them… but don’t make it an antagonistic thing.

Look for opportunities for high-performing teams to teach lower-performing teams.

so tempting to gamify it

IMHO, gamification’s fine. But cooking the numbers out of fear is bad. So it’s important to stress that the entire organization (all teams) are looking to improve, together.

I have some teams doing daily deploys and some doing 2-week deploys

FYI, the “quick check” was recently updated and has some additional insights embedded. Worth a look, even if you’ve already done it: https://www.devops-research.com/quickcheck.html

great. thank you!

"For the primary application or service you work on, what is your lead time"...they mean "you" as in, you are a developer on a dev team?

imho even gamifying can lead to attempts to "look better" by gaming the system

A team delivering every 2 weeks can be bad or can be great, depending on context. Is the cadence driven by customers that do not want more frequent releases? Or is it due to the team working on large batches?

So, aim to compare apples to apples, that means each team comparing to their past selves, not to other teams.

@me1208 how about I frame it as each team is different, compare to your past self, but here are some comps in your area? ...home buying analogy.

I really think seeing what your fellow dev teams are doing is such an important reference point. So motivational. Without it, teams seem to assume they are amazing.

...in my experience

Hi

That's an interesting point. In a physical office setting the "peer pressure" of seeing other teams' improvements would surface naturally. In the current remote-first setting, however, that might be more difficult. I would say making sure the teams metrics are visible to everyone in the organization (for example via the Team API approach that we talk about in Team Topologies) should be enough. You can/should also have regular presentations by teams explaining how they've been trying to improve their metrics, what helped and what didn't, in a learning/collaboration setting rather than a gaming/competition setting.

I've some experience with gamification of delivery metrics. It's effective.

You DO need to be careful to make sure that managers cannot use the numbers for evil.

You also need to run the game in a way that teams TRY to game the numbers, but can only win the game if they are moving towards the goals you want.

It's a fun problem.

To keep things fair, I also gamify leadership metrics and compare them to each other and State of DevOps "good".

On the teams trying to "look better". If they get there by deploying faster with higher quality (measure in groups), everyone wins.

Did someone say gamification: https://dream.devops.games/?utm_source=devops.games_website&utm_content=orangebtn_cta

One of the best parts of the 4KM are that half protect against the other half, so you can safely gamify them all and maintain balance. If you focus too much on dep freq for instance, MTTR and CF% will likely suffer, and even lead time could be affected. If you track all four you can protect against negative gamifaction effects.

Just make sure you gamify burnout prevention as well 😅

Indeed.

We took a different approach. We measure deployments per service. A team owns one or more services. Those metrics are published, intended only for that teams use though. Teams can view those metrics and decide what they are going to improve next. Management has been trained not to discuss any of those metrics with any of the teams.

Simple answer: The entire company should probably release more often than individual teams. What is "good enough" and "alarming" really depends whether the team is releasing a mobile app, an internal API or something on a mainframe.

In a financial organisation, compliance has a significant effect. The regulated software needs jumps through hoops and separation of concerns to be built in somehow and within the same walls it is possible to go really wild, fast and unsafe with some unregulated services for communication between the real estate agent and their customer. So, knowing that it is essentially comparing apples and oranges helps to be humane about it.

@ferrix how do you get the org releasing more often than teams? Don't the org level releases depend on team releases?

Proper pipeline automation fully satisfies compliance and separation of concerns. Pushing on the deploy frequency is a forcing function for waste identification. Any team can deliver releasable changes daily. There may be constraints on delivery, like the app store for instance, but tracking at a staging environment still drives out waste in those instances.

@steveelsewhere Many products and teams do not have to release in the same batch. "Something goes out every day" does not mean that something has to be released in every team every day.

Oh yes agreed, team A could be delivering one day and B the next 👍

@bryan.finster The pipeline helps in the automated part. However the time taken to run the code through four eyes instead of firing and forgetting does take some extra time. Yes, optimally the non-regulated teams would also care for quality etc but the amount of corners that can be cut when push comes to shove is very different.

Manual compliance really isn't. If there are rules for compliance, then no human can execute those rules as well as a robot.

So, leaving production access open for some developer is not best practice but in other environments it could cost millions in fines and in other places you can push your MVP any way you can cough it up in an afternoon.

No developer should ever have access to production.

The pipeline's job is to inject quality, security, and compliance gates to prevent the delivery of poor quality.

Yeah, exactly what I thought in my last startup. We didn't follow suit and we still ran out of runway.

Two separate delivery problems. Different contexts

Sure. What I am saying, the feasible target for an arbitrary measurement depends on both where you start and what you are doing. The leaps you can take are somewhat different whether the unit on your error budget is HTTP error codes or deaths.

Pushing deployment frequency to a week is not unlike magic in most organizations with monthly, quarterly or worse schedules. After that, there will be a backlog of other things worth their while, so it makes sense to ignore some of the slowness for a while until it becomes a bottleneck again.

Honestly, orgs where errors will kill people are more likely to kill people with long delivery cycles. Measuring frequency and quality of delivery and focusing on making them better improves upstream quality processes. It drives down complexity, drives out uncertainty, and makes things safer. There is no way any quality or compliance process can be effective with those giant releases. It's just theater at that point. Where I work, I see every delivery problem, even those that can kill people. In the end, the solution is the same: "why can't we deliver daily?" and then daily fix one thing that's preventing it.

Agree with you there. It's just which measurements you transform first.

We use the cycle time from "started" to "done" on development, how frequently code integrated to trunk, and defect rates to start with.

We're pretty effective at moving the needle on both speed and quality using those.

Okay. The case I am mostly referring to is "started" to "done" in production. There's all kinds of due process from way back when that needs to change before any of the individual teams will be able to do better than deploy to test.

The new way does include compliance before deploying to staging, so there will be a big drop

I'd always measure in groups. However, tracking that metric along with the costs associated and the quality at the end and making that number available to leadership can be a powerful motivator. Also, if the org wants to actually survive, they either need no competition or they need to look at their value stream map hard and see if those processes and handoffs are delivering the expected value.

If it's manual compliance, it's either arbitrary or we are paying people to pretend to be robots. How do we highlight that issue to executives to get it fixed?

Yeah, we did the teams*members*salary*weeks in queue math for this group. Management understands now.

Nice

I've always had the attitude that if they want CD then anything preventing CD is wrong and they need to help me fix it. 🙂

If they don't want CD, I don't want to work there. They don't care about their bottom line or the people that enable that bottom line.

Then there is the other cohort of "CD without CI". So, not enough people contributing to the automated tests.

No, that's not CD

I know. I am also setting people right about that.

Dave Farley has been doing us a great service here https://www.youtube.com/channel/UCCfqyGl3nq_V0bo64CjZh8g/videos

@bryan.finster I think one of the problems that the robots to do the robot jobs are not available yet and the benefits of going down to a weekly release schedule are significant enough while those robots are magically appearing in all the necessary parts of the organisation.

Takes time and leadership who understand.

And sometimes investment and training.

It always takes investment and training. 🙂

In some teams, they are just waiting for the permission, but that's exceptional