Fork me on GitHub

Reminder: Day 2 is starting now – opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.

Ann Perry - IT Revolution16:10:03

And now, let's welcome @stephan.stapel here today to present, Measuring for DevOps Success

Stephan Stapel16:10:26

Hope you enjoy :)

Andy Nelson16:10:29

👋 @stephan.stapel

Jon Smart [Sooner Safer Happier]16:10:23

Agile Principle 8: promote sustainable development. to maintain a constant pace indefinitely 👍

Jon Smart [Sooner Safer Happier]16:10:18

"shine a light on the system of work", nice

Andy Nelson16:10:40

"Make transparent, share in the pain together" - create shared understanding in the system of work

Glenn Wilson, Author of DevSecOps16:10:48

“Find a pace you can work with and don’t go faster” - absolutely agree with this. I see so many teams complain that security slows them down. So I assert that we should work on making security more efficient, not bypass it

💯 1
Stephan Stapel16:10:51

For most of the time I agree. But sometimes it's really hard to understand the(ir) requirements :)

Stephan Stapel16:10:41

If the "two month rolling median" should be hard to understand: the median for 01/21 is the median of all features we finished in 12/20 + 01/21 etc.

Glenn Wilson, Author of DevSecOps16:10:29

@stephan.stapel - have you worked out the standard deviation to understand your statistical control boundaries before and after the start of the migration?

Stephan Stapel16:10:14

yes, we did. It actually didn't change much, almost constant.

👍 1
Andy Nelson16:10:52

@jonathansmart1 his comment there about a horse reminds me of your quote in SSH about putting a 1950s tractor engine in a racecar (I believe when talking about doing a cloud migration just for the sake of doing a cloud migration)

💯 1
Stephan Stapel16:10:16

Haha. I missed the quote in the book. I had this one in mind:

👍 1
Jon Smart [Sooner Safer Happier]16:10:08

Cloud migration: like putting an F1 engine in the 1950s tractor of an organisation and throttling it back to 5 mph

❤️ 1
Andy Nelson16:10:20

thanks - I had it backwards - I should have grabbed the book from my desk 1st...

Stephan Stapel16:10:40

@isuanelson: backwards = putting a car in front of a horse?

Jon Smart [Sooner Safer Happier]16:10:43

Equipping the horse with wheels and an engine and then going at horse pace 🙂

Jon Smart [Sooner Safer Happier]16:10:08

(because of GRC, because of horse levels of bureacracy)

😂 1
Glenn Wilson, Author of DevSecOps16:10:26

Also reminds me of another quote (although I cannot remember where I heard / read it), if you’d asked someone in 1900 how we could travel faster, they’d answer “faster horses”

👍 1
Andy Nelson16:10:53

I had the quote backwards - tractor engine into a racecar vs. a racecar engine into a 1950s tractor - In the book Jon didnt mention the brand of tractor - it may have been a ford....

Jon Smart [Sooner Safer Happier]16:10:44

Might have been a John Deere 🙂

😆 1
Andy Nelson16:10:14

Being a former John Deere employee... Im pretty sure it was a Ford... 😎

Jon Smart [Sooner Safer Happier]16:10:41

"with more mature solutions, teams concentrate on value: frequency drops". This is gold. It's not all about velocity! Making the wrong thing faster makes you wronger.

💯 3
❤️ 2
👍 3
Jon Smart [Sooner Safer Happier]16:10:07

"Learning: record the before state"

💯 1
Andy Nelson16:10:15

In order to measure the improvement of the system - you need the Key Result of improve outcome from X to Y... OKNOTOKOKR

🎉 1
Nicole Forsythe16:10:07

And also stories are a form of data! “Here’s our state now, told as a story and a reflection of current tooling. Here’s what we’re going for…”

👍 1
Nicole Forsythe16:10:08

@stephan.stapel - you mention the conversations about happiness matter. This slide mentions how your analyses allow you to connect with the teams. Are you saying metrics over conversation or conversation before metrics? I have a really exceptional engineering leader struggling with 50ish teams, and I’m nervous about only connecting him with teams through metrics, especially since our agile coaches intimately know the problems these 50ish teams face. Advice?

Andy Nelson16:10:27

what metrics @nicole.forsythe?

Stephan Stapel16:10:23

That is a good question. I'm responsible for/ serving 12 teams and cannot be everywhere and be in close contact to everybody at the same time. The metrics help my management time and me to understand the trends and then to find the times that really need support and dedication. For those teams that are doing well, the metrics help me to acknowledge that they are doing well.

Stephan Stapel16:10:25

Does that help?

Nicole Forsythe18:10:00

It does. Thank you!

Glenn Wilson, Author of DevSecOps16:10:09

Statistical analysis is such an underrated discipline. There’s so much we can learn about our organisations from the data that sits in our systems.

💯 1
Stephan Stapel16:10:09

I can only agree. Speaking only about myself, I find statistics to be really hard. Average, median and standard deviation is fine but there is so much more that's possible. I'm so glad we have good data engineers and data scientists in our company that can help out!

👍 1
Jon Smart [Sooner Safer Happier]16:10:14

Danke schon @stephan.stapel, sehr gut!

🎉 1
👏 2
Stephan Stapel16:10:30

sehr gerne (you are welcome)!

😀 1
Andy Nelson16:10:45

👏 @stephan.stapel Excellent presentation!

❤️ 1
Glenn Wilson, Author of DevSecOps16:10:05

Thank you @stephan.stapel - that was very insightful

👍 1
Ann Perry - IT Revolution16:10:12

More from Discover with @prajakta.yerpude95 and @shivanianand, here to present Cloud Data Journey: Story Of Adopting Cloud Technology and Modernization of Data Pipeline

👏 1
Andy Nelson16:10:28

👋 @prajakta.yerpude95 and @shivanianand - looking forward to it

❤️ 2
Shivani anand16:10:07

Hello Andy, Happy to share our journey with this group!

Andrew Machen17:10:50

We're starting to hear the term "DataOps" a lot more these days, especially in the analytics space. We could use a regular user any exist?

👍 1
Sujay Solomon17:10:38

how do you determine which data should remain on-prem and which data are good candidates to move to the cloud?


I am interested in this as well

Prajakta Yerpude17:10:56

Hey Sujay, it depends on the business use cases and their requirements wrt technologies.

👍 1

where there any factors that drove decisions in your case Prajakta?

Sujay Solomon17:10:18

i think you mentioned PII/sensitive data remaining on-prem but im wondering if even that may get moved to the cloud in the future

👍 2

We are starting to move data to Cloud/ADLS and wanted to understand the reasons why some data was left On Prem

Malcolm McAlpin17:10:22

Thank you!!

❤️ 1
Christina Tan, Strategy at Blameless, Speaker DOES2118:10:40

Excited for @jmrichardson1’s workshop after yesterday's inspiring plenary session!

thumbsup_all 1
Marco Coulter (Speaker DOES21)19:10:30

G’day folks! Warming up the #ask-the-speaker-track-2 slack channel for my session “Avoiding Goodhart’s law - Use SLO’s as Tools not Cudgels”. Loooooove questions so ask them here!

Ann Perry - IT Revolution19:10:00

📣 And now we welcome @marco.coulter who will present, Avoiding Goodhart’s law - Use SLO’s as Tools not Cudgels 📣

Michael Winslow19:10:53

I love the talk about Goodhearts Law! Definitely cheating and gaming the system!

👏 1
Michael Winslow19:10:07

When researching it, I found another similar Law (or Effect) that was really helpful @marco.coulter! I'll wait to type it just in case it spoils something you are about to mention! 🙂

Marco Coulter (Speaker DOES21)19:10:34

spoilers are allowed!

😂 1
Michael Winslow19:10:09

Hawthorne Effect!

Seth Tager - Salesforce19:10:20

We've had many internal discussions with references to Goodharts law, but usually it ends with a conclusion that goes something like "how do we prevent execs from misusing this measure as a target?" - I feel like that's an unsophisticated treatment of the problem.

Seth Tager - Salesforce19:10:01

mostly talking about metrics related to agile outcomes, rather than feature SLOs

Marco Coulter (Speaker DOES21)19:10:54

I look for ways of adding an alternate measure (e.g. that will maintain quality) as a balancing measure.

Seth Tager - Salesforce19:10:27

alternate, meaning additional, not alternative, correct?

Marco Coulter (Speaker DOES21)19:10:35

You got it. Additional but on a different frame of reference

Seth Tager - Salesforce19:10:24

I'd love to see a talk about creating effective orthogonal metrics for this purpose

Marco Coulter (Speaker DOES21)19:10:35

hmm - thats a great idea!!!

Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM19:10:00

tension metrics. love them. The DORA metrics are that. On a service desk, balancing NPS vs call handle time.

Marco Coulter (Speaker DOES21)19:10:44

Look for a session from me on ‘effective orthogonal tension metrics’ at the next DOES! 😉

❤️ 1
David Van Couvering - Senior Principal - eBay19:10:45

One thing I've found is that when I set an SLA (what to do when budget runs out), everyone agrees until the budget is exceeded, then business pressures show up. For example, product is not willing to pause on feature dev to address breaking an error budget. Any tips?

Marco Coulter (Speaker DOES21)19:10:59

googling hawthorne - ooohh nice!

Marco Coulter (Speaker DOES21)19:10:06

@dvancouvering Immediate reaction is to change the SLA outcome to being budget contribution by business! (easier said than done

David Van Couvering - Senior Principal - eBay19:10:35

"budget contribution by business"? not sure what this means?

Marco Coulter (Speaker DOES21)19:10:31

basically that when the error budget is burned up, and they are pushing for more change, the SLA agrees they have to show up with more money to fund things. Or am I misunderstanding the challenge?

Marco Coulter (Speaker DOES21)19:10:34

wait, the budget running out is the error budget - doh! Just caught on.

Marco Coulter (Speaker DOES21)19:10:53

(I was thinking financial budget)

David Van Couvering - Senior Principal - eBay20:10:26

Hey, my daughter needed a ride (kids) so I had to step out... Yes I meant the error budget.

David Van Couvering - Senior Principal - eBay20:10:45

The challenge is you say "ok, we have exceeded our budget so we need to take <action>" and they say "yea, no"

David Van Couvering - Senior Principal - eBay20:10:54

Usually it's like "yea, I see that's really important, but we committed to get Feature X done and you're asking us to take two resources out to fix this problem, but it's just not as high a priority"

David Van Couvering - Senior Principal - eBay20:10:55

It's the same problem I see as setting a WIP limit, which is like another budget if you think of it. When you actually try to enforce the limit, unless you have solid commitment and support from leadership, it gets overridden

Marco Coulter (Speaker DOES21)20:10:11

Toughest conversation to have - it is why I learned to ensure the ‘signoff’ on the SLA (even if just an email). Dependent on personality you can try the ‘We agreed to this. We set this budget because going beyond it could put our entire business/organization at risk. Are you saying this feature is so critical that you will be responsible for risking our entire business/organization to get it in before the error budget is reset?’

David Van Couvering - Senior Principal - eBay20:10:40

Haha that's quite a threat, but yea I see what you mean

Marco Coulter (Speaker DOES21)20:10:54

you are right about the commitment needed - this is where the negotiation skills come in!

David Van Couvering - Senior Principal - eBay20:10:09

Yea, which is why having SLOs related to actual business needs is so important

David Van Couvering - Senior Principal - eBay20:10:18

Rather than some internal technical measure

Marco Coulter (Speaker DOES21)20:10:17

Thanks for this chat! We all learn something today.

Marco Coulter (Speaker DOES21)20:10:50

(the other thing is the set the expectation as the error budget depletes - do not let it be a surprise)

David Van Couvering - Senior Principal - eBay20:10:34

Rats, I was going to connect with you on LinkedIn but I guess I am not allowed. Not sure how that works

Marco Coulter (Speaker DOES21)14:10:05

@dvancouvering - not sure why LI blocked you, just tried to connect to you - let’s see if that works.

Michael Winslow19:10:54

Love it! SLAs have been around for so long and pretty ineffective by themselves. The way you describe SLIs and SLOs allow you to have these conversations much earlier in the process than the concrete SLAs.

❤️ 2
👏 1
Craig Larsen - he/him - Solution Design Group Mpls19:10:36

Love that! It doesn’t need to be “as fast as possible”, just “as fast as necessary”.

❤️ 2
🎉 1
Marco Coulter (Speaker DOES21)19:10:39

@michael_winslow yes! Basically, how soon can you start talking about SLI’s…

Marco Coulter (Speaker DOES21)19:10:53

I found bringing a leader from another part of the org as a facilitator made me some great supporters!

Michael Winslow19:10:09

So simple, yet I've never thought of that! Bring in someone from another part of the company! What a way to remove some bias and be inclusive.

Marco Coulter (Speaker DOES21)19:10:55

So glad I did this session. I am getting so much from the feedback!!

Christina Tan, Strategy at Blameless, Speaker DOES2119:10:31

negotiation skills is good for life and devops - love it!

Randy Shoup (Chief Architect eBay; he/him)19:10:33

Very interesting and actionable, thanks!

Christina Tan, Strategy at Blameless, Speaker DOES2119:10:45

Ah missed your twitter, could you add it here again, @marco.coulter?

Dave Owings19:10:22

thank you @marco.coulter

Michael Winslow19:10:32

Excellent @marco.coulter! Thanks so much!

Marco Coulter (Speaker DOES21)19:10:58

Thanks to all for the active conversation and feeback!!!

🙌 2
David Van Couvering - Senior Principal - eBay19:10:46

I had to step out, but thank you @marco.coulter for the very interesting talk and discussion


Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees.