Fork me on GitHub
#ask-the-speaker-plenary
<
2021-10-07
>
Gene Kim, ITREV, Program Chair13:10:18

Good morning, everyone! Looking forward to another fantastic day here at DevOps Enterprise Summit!

👏 5
☀️ 2
1
Slackbot13:10:06

Reminder: Remember all those talks you attended the first two days of the Summit? Please submit your feedback for those! It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://events.itrevolution.com/virtual-agenda/ https://devopsenterprise.slack.com/files/UATE4LJ94/F02GHSEB604/feedback-does21us.png

Courtney Kissler13:10:50

Learning learning learning!! 🙂

10
Gene Kim, ITREV, Program Chair13:10:53

Yes, I’m hoping you are having lots of mutually exothermic interactions!!! 🎉

❤️ 4
🔥 2
Topo Pal - Programming Committee Member13:10:13

I have a new folder with screen shots of slides!!!

upvotepartyparrot 1
Gene Kim, ITREV, Program Chair13:10:18

I made the mistake of not putting all my TODOs (links, books) in my notes — I need to go thru all the Slack history to find them all! 🙂 I plan on writing some tools to make this a little easier, so anyone can do the same.

Randy Shoup (Chief Architect eBay; he/him)13:10:54

Learning from last year, I have a Google doc open the whole time where I type notes, etc.

Ann Perry - IT Revolution13:10:35

:satellite_antenna: Kicking off our final day is none other than @michael_winslow, here to present Building Confidence in Your SRE Team :satellite_antenna:

1
🙌 2
Gene Kim, ITREV, Program Chair13:10:01

Please welcome @michael_winslow Senior Director of Development and Engineering at Comcast!

Luke Rettig - Target13:10:02

MICHAEL!!! buckle up everyone!

👏 2
Courtney Kissler13:10:02

YAY @michael_winslow

Michael Winslow13:10:43

Miss you Courtney!

❤️ 1
Courtney Kissler13:10:04

Miss you too!! We should do a video catch up soon!

Michael Winslow14:10:33

I'm on it!

❤️ 1
Michael Winslow13:10:34

Hey EVERYONE!!!

👋 3
🎉 1
🎧 1
Gene Kim, ITREV, Program Chair13:10:51

AND NEWS FLASH! Last night, it was announced that @michael_winslow was promoted to Distinguished Engineer!!!! 🎉 🎉

🎉 13
1
👏 2
🍻 1
Topo Pal - Programming Committee Member13:10:56

Congratulations @michael_winslow

Gene Kim, ITREV, Program Chair13:10:25

(Goodbye management tasks, right @michael_winslow? 😆 )

Chris Kernaghan13:10:28

Congratulations @michael_winslow - I remember speaking together at the DevOps Summit

thankyou 1
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:41

Congrats @michael_winslow!!

thankyou 1
🎉 1
Topo Pal - Programming Committee Member13:10:44

What are management tasks @genek @michael_winslow?

Michael Winslow14:10:05

Budget Analysis Comparison was my most recent beast! Comparing our Eng team's list of people + spend against what Finance was seeing. "and believe me, there were some discrepancies!" :rolling_on_the_floor_laughing:

Gene Kim, ITREV, Program Chair13:10:56

(Go ahead and uninstall Microsoft Office. Except for Visio, of course. 😆 )

❤️ 2
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:59

Love it! Coding as part of management tasks!!

👍 1
jeff.gallimore (CTIO - Excella, he/him)13:10:13

@michael_winslow you’re the most famous michael winslow in THIS community

Scott Prugh (CTO - CSG) DOES Prog Committee13:10:20

@michael_winslow I still code too!!

👏 4
David - BBY13:10:54

Congratulations on the new title distinction @michael_winslow - very cool!

thankyou 1
Topo Pal - Programming Committee Member13:10:29

What is a technology company if Comcast is not?

Randy Shoup (Chief Architect eBay; he/him)13:10:56

Technical Maestro @michael_winslow

thankyou 2
Chris Kernaghan13:10:03

Interesting that Sky is in that diagram - did not know that

Michael Winslow13:10:46

Yes, we acquired SKY in 2018. I've visited their offices in UK ... so awesome!

👍 2
Chris Kernaghan13:10:14

They do some great stuff, I have worked there on their SAP system and the SAP team is quite jealous of other teams

Michael Winslow14:10:03

Love working with the new colleagues there!

Michael Winslow14:10:37

I did another talk for DOES last year where we talked about using the Dojo format to work together across the pond!

Chris Kernaghan14:10:54

I shall have a look and watch it - it will be useful for our new distributed development teams in terms of asynchronous teams

Randy Shoup (Chief Architect eBay; he/him)13:10:50

Oh no, SRE as operators ?!

😬 1
😡 2
1
Luke Rettig - Target13:10:04

SRE Operators… oxymoron?!?

Peter K.13:10:26

@michael_winslow what was your organizational structure looking like @ xfinityMobile? Were you set up as a "flat organization structure" where you embed SREs to engineering teams or more of the traditional top to bottom structure?

Michael Winslow13:10:30

We were VERY LEAN. We called what we did DevOps. It was not as concrete as SRE.

thankyou 1
Michael Winslow13:10:45

We ran like a startup at XMOBILE at that time.

🎉 1
Gene Kim, ITREV, Program Chair13:10:29

I’m so delighted that we had @jpetoff and @cleng share the theory and practice of SRE at Google yesterday, so we can point it at people who say “we just operate the code”!!

👍 3
👏 1
Peter K.13:10:28

I like repeating this mantra: "We write software to make software reliable"

Michael Winslow13:10:14

I was so excited watching their talk!

jeff.gallimore (CTIO - Excella, he/him)13:10:24

@michael_winslow i love your curiosity and suspension of skepticism at the beginning with the team.

❤️ 3
Topo Pal - Programming Committee Member13:10:09

Some one said SRE implements DevOps in OOO terms

3
Sean D. Mack13:10:07

That's the definition we've been going with!

Michael Winslow13:10:11

What else implements DEVOPS then? 🙂

Chris13:10:25

Like Google “SRE implements class Devops”? 😉

1
Michael Winslow13:10:28

I believe IaC and CI also implement DEVOPS!

BMK-BNZ-ValueStreamArchitect13:10:35

class sre implements devops

Scott Jaffa (Principal Engineer, ValidaTek)13:10:46

I believe I’ve heard that quote from google.

BMK-BNZ-ValueStreamArchitect13:10:47

DevOps is interface 🙂

Sean D. Mack13:10:04

SREs are the class of humans that implement DevOps.... there are lots of tools too

Michael Winslow13:10:38

If you don't have Culture, Automation, Measurement and Sharing in your SRE team, your code won't compile!!! :rolling_on_the_floor_laughing:

jeff.gallimore (CTIO - Excella, he/him)13:10:42

…and @lizf will be speaking with @shelby later today!

❤️ 2
Liz Fong-Jones16:10:26

oh hi yes 👋

👋 2
Michael Winslow16:10:46

Wow! Look at those closers!!! I'll be in attendance for sure!

💯 2
Gene Kim, ITREV, Program Chair13:10:43

Devs: “who are you to tell us that we should define SLOs and error budgets for our code?!?” 😆 [go pound sand]

4
😀 1
Steve Farley - VP Infrastructure Operations, Nationwide Insurance13:10:58

I saw PRODUCT at the top of the SRE pyramid...what is your take on that as a foundational need for success

👀 1
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM13:10:21

Product centric operating model is a huge movement right now...

Luke Rettig - Target13:10:07

100% agree. Product thinking in SRE or any Infra/Ops function is so important

Luke Rettig - Target13:10:12

and such a paradigm shift

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:13

Agree as this is the next big evolution at Nationwide to really embrace Product orientation

Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:21

Long thread on this yesterday. The problem is that there's a gap in product skills in infra/ops. You can either try to upskill, or parachute in. Both have issues.

Chris Kernaghan13:10:01

Doesn't stop companies from trying that though

Gene Kim, ITREV, Program Chair13:10:21

“I hung this up” to fortify the mission! 🙂

Randy Shoup (Chief Architect eBay; he/him)13:10:24

"Automate away the 100s of routine tasks which create the fog that impedes our vision"

👏 2
Chris13:10:25

Brent s back…

Brian Smith13:10:54

Brent is my favorite character.

Peter K.13:10:57

Brent also needs a raise 😆

Luke Rettig - Target13:10:01

Brent doesnt have time to document!

3
😆 1
Gene Kim, ITREV, Program Chair13:10:12

“Brents often don’t enjoy documentation” 😆

1
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:12

Brent seems to be everywhere. That guy multiplies!

1
Chris Kernaghan13:10:21

I love brent - I also don't enjoy documentation

Gene Kim, ITREV, Program Chair13:10:22

Brent doesn’t need documentation!

🛂 1
Chris Kernaghan13:10:37

Brent needs to automation his documentation process!!

Gene Kim, ITREV, Program Chair13:10:46

Confluence history: “his documentation was actually only five steps!” 😆

😂 2
😄 1
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:01

Real Brents don't use documentation!

Brian Gallop13:10:07

He wanted to work iteratively?

Gene Kim, ITREV, Program Chair13:10:09

• start the work • do the work • validate the work

4
😂 1
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:22

Just like real developers don't use version control!

1
🤯 2
Gene Kim, ITREV, Program Chair13:10:28

“okay, next deploy, the junior developer with do the deploy”

BMK-BNZ-ValueStreamArchitect13:10:43

That confluence page looks like "Interface" now in OOO terms (nothing concrete, all abstract) 😆

🙂 1
jeff.gallimore (CTIO - Excella, he/him)13:10:21

this is a great pattern for increasing collective knowledge

👍 2
David - BBY13:10:24

Cool methodology

2
thankyou 1
jeff.gallimore (CTIO - Excella, he/him)13:10:42

Brent is not allowed to touch the keyboard!

👍 1
David - BBY13:10:49

self-guided Gemba walk

👍 1
Gitty Rosenfeld13:10:15

@michael_winslow Love the team learning

1
Topo Pal - Programming Committee Member13:10:18

Person who can code cant deploy! 😀

Randy Shoup (Chief Architect eBay; he/him)13:10:41

This is such an awesome way to approach this problem

Gene Kim, ITREV, Program Chair13:10:41

Watch the documentation grow! Watch the conditions and branches being discovered!

Luke Rettig - Target13:10:43

anyone else find themselves in a Confluence jungle of documents?

👍 6
🎉 1
🌴 1
Luke Rettig - Target14:10:52

i “strongly encourage” my team to use github markdown pages for documentation in the same repos where the code lives. version control and discoverability are real needs!

Chris Gallivan (Tasktop)14:10:00

with very limited search ability

Craig Cook - IBM13:10:50

Documentation is the first step to automation

BMK-BNZ-ValueStreamArchitect13:10:58

@michael_winslow - your confluence page reminds me - my 2014 deployment nightmare 😞

Peter K.13:10:59

When I was a junior engineer... "I still don't have access" was a mourning routine

😆 1
Brian Smith13:10:05

Two things I took away from Brent: 1. I told our Ops folks that our Real Brent was not allowed to touch the keyboard 2. We made our Brent#2 put his thoughts into our Models (MBSE). When our Brent#2 left the company, thank God.

Eric Hardisty, Technical Director - USG13:10:10

That button is the best thing ever!

1
💯 3
BMK-BNZ-ValueStreamArchitect13:10:22

You can automate only when you know what to automate, so you need documentation

2
👍 1
👆 1
Scott Prugh (CTO - CSG) DOES Prog Committee13:10:25

Great pattern! Note that the 2021 SODR found that documentation is a high performance capability! Also note that @jmrichardson1 and @emily356 discussed the importance of "if you think you understand something: write it down!" https://cloud.google.com/devops/state-of-devops/

👏 6
Gene Kim, ITREV, Program Chair13:10:53

“Delivering our software became a non-event”

👍 3
😍 1
Ryan Taylor - Senior Geospatial Developer, GISinc14:10:06

To achieve this would be an absolute dream.

☝️ 1
BMK-BNZ-ValueStreamArchitect14:10:02

Delivering our Software became a non-event

😎 1
Peter K.14:10:26

best type of deployment

Nicole Forsythe14:10:17

@steve773 @genek - in this quote, I see both fast and slow thinking on behalf of a product team. Thoughts?

jeff.gallimore (CTIO - Excella, he/him)14:10:21

@michael_winslow did you actually capture the numbers for toil vs engineering work?

⬆️ 2
Michael Winslow14:10:29

I captured the amount of time saved per deployment and kept a running calculation for every time the deployments were run to the staging environment and above.

👍 2
Courtney Kissler14:10:00

oh my goodness - key rotation

❤️ 2
Gene Kim, ITREV, Program Chair14:10:32

(suddenly an important thing to do quickly, cloud credentials…)

❤️ 2
Chris Kernaghan14:10:34

@michael_winslow - got to ask a stupid question, is Toil an acronymor just hard work?

Michael Winslow14:10:06

nope ... Just means rote work

jeff.gallimore (CTIO - Excella, he/him)14:10:35

i’m loving this animation/slide progression. it tells a wonderful story!

💯 4
thankyou 1
👍 1
BMK-BNZ-ValueStreamArchitect14:10:40

@michael_winslow - Who is on on-call rotations?

Michael Winslow14:10:11

The existing team was already great at scheduling on-call through pager duty. The main change we did there was to collect MUCH MORE information from every incident in order to create engineering tasks.

❤️ 1
Dave Burnison14:10:24

That was a FANTATIC Illustration of this:

🙏 2
Gene Kim, ITREV, Program Chair14:10:00

Ops was no longer the dumping ground for Devs. 🙂

❤️ 3
Michael Winslow14:10:49

Did you hear when @jpetoff said "You can't build a wall and expect Dev Teams not to throw code over it?" I love that soooooo much!!!

❤️ 1
Ryan Taylor - Senior Geospatial Developer, GISinc14:10:02

How do you measure toil? I have a sense of how much I experience (and it's prob high) but being able to measure and show reduction would seem to be beneficial. Is this the DORA metrics?

👍 2
Ryan Taylor - Senior Geospatial Developer, GISinc14:10:49

Or, do measurements need to be more specific (x time writing release notes)

Scott Jaffa (Principal Engineer, ValidaTek)14:10:36

If you have a work tracking tool, use that. If you don’t, define it and ask your people. They’ll tell you.

👍 1
Ryan Taylor - Senior Geospatial Developer, GISinc14:10:28

We use Jira... fwiw. Almost everything is a ticket. Sometimes def of done is clear, sometimes less so. Sometimes abused, sometimes not.

Craig Cook - IBM14:10:35

Nice, dumping/passing off on the SRE team is not a right, it needs to be earned.

❤️ 4
Topo Pal - Programming Committee Member14:10:35

I find meetings to be the most painful toil

🙏 1
💯 2
Luke Rettig - Target14:10:48

“make it the easy and obvious choice” - Michael implementing product management mindset to SRE offerings

Gene Kim, ITREV, Program Chair14:10:56

Extent to which devs are using certain standardized platforms increases likelihood of getting SRE support

👏 4
Gene Kim, ITREV, Program Chair14:10:50

“find joy and do real engineering work!”

👏 2
Scott Nasello - Nike - Director of Digital Support14:10:15

@michael_winslow - my wife just said "I recognize that voice"... Me - yeah that's DJ Boo Boo Wife - YES!

😂 4
Scott Nasello - Nike - Director of Digital Support14:10:15

@michael_winslow - my wife just said "I recognize that voice"... Me - yeah that's DJ Boo Boo Wife - YES!

😂 4
Scott Nasello - Nike - Director of Digital Support14:10:49

The family that watches DOES talks together stays together

😆 1
Margueritte Kim (CEO, IT Revolution, she/her)14:10:07

Oh haiiiii Nasello fam! Miss you.

💯 1
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:20

proposed: the essential distinction between SRE and traditional ops is the political power to refuse handover.

Dian Hansen14:10:22

@michael_winslow how big was your team when you started?

Chris14:10:44

and eventually serving how many products/teams?

👀 1
Michael Winslow14:10:06

Start: 4 people / Now: 11 people / Serving 5 products (~12 variations)

Michael Winslow14:10:14

Product => Xfinity Stream on (Android, iOS, Roku, Web, Flex) + Chromecast, FireTC, etc

Chris14:10:01

sounds like large product teams

Chris Kernaghan14:10:32

How did you manage to be selective about your workload in a corporate environment @michael_winslow

👀 3
BMK-BNZ-ValueStreamArchitect14:10:51

May be one talk title "How not to attend meetings - Devs how to find & ignore your productivity enemy :)" cc - @topo.pal

5
jeff.gallimore (CTIO - Excella, he/him)14:10:01

@michael_winslow did you institute a “hand back” to the team if a system/app gets too unstable?

👀 2
Michael Winslow14:10:39

Not exactly 🥺

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:13

Error Budgets have not taken hold yet at Nationwide. Fairley new SRE team....leadership buy in is always the challenge

Michael Winslow14:10:50

Look forward to continuing to share stories!

Courtney Kissler14:10:14

Amazing @michael_winslow

2
🙌 6
❤️ 1
Malcolm McAlpin14:10:21

Thank you, wonderful insights!

💯 1
Andy Nortrup - Director of PM at Tanium14:10:26

I've been trying to hard to get error budgets adopted right now. I'm having some success getting buy in from the field support team, who has to deal with customer issues when they come in. If we can use an error budget to tell them that something is going wrong before the customer they like to see that.

Andy Nortrup - Director of PM at Tanium14:10:42

Also saw them become very popular with our engineers when they had started alerting on something bad (error counts) and were feeling the pain of that very noisy alerting while holding a pager.

Daniel Cahill - Engineer - Ontario Systems14:10:29

Is there a different type of error budget or metric the field support team uses? That has been a problem I hit when I was trying to get them to use something I would eventually want as an error budget.

Andy Nortrup - Director of PM at Tanium14:10:24

I'm trying to pick SLIs that are customer facing, so our account teams or module SMEs want to know if the query success rate is below our targets because it means something has happened in that environment that they can go fix to support their customer.

Andy Nortrup - Director of PM at Tanium14:10:04

I'm kind of lucky that our technical account managers lean so far forward to support customers.

Michael Winslow14:10:00

this is ver interesting

Andy Nortrup - Director of PM at Tanium15:10:17

Haven't succeeded yet. Still preaching the good word and looking for converts.

Andy Nortrup - Director of PM at Tanium15:10:52

Comcast is a customer, so it'd be real easy to get together and talk more if you are interested. I think I'd learn more from the conversation though.

Roberto Hernandez14:10:28

Great talk!

❤️ 1
Graham McGregor14:10:30

👏 Great practical insights!

🙏 1
BMK-BNZ-ValueStreamArchitect14:10:32

Excellent one @michael_winslow - Love it

1
❤️ 1
Gene Kim, ITREV, Program Chair14:10:35

“our products always exceeded their defined SLOs” — ah, yes, I loved how @cleng described some of the tough love here, and how SREs can leave the product at any time. A truly self-balancing system. 🙂

❤️ 2
Michael Winslow14:10:32

SRE teams leaving was a profound thought!

James T Snell14:10:42

Great talk Michael! Thank you. 🙂

2
thankyou 1
Amri Abuseman - Flatiron Health, Director of Quality Engineering14:10:48

@michael_winslow - we have implemented error budget (for real) on teams that reach incident threshold (by quarter)

Michael Winslow14:10:42

I can't wait to talk to you!

👍 1
Amri Abuseman - Flatiron Health, Director of Quality Engineering14:10:05

We focus on the outcomes for customers in phrasing the "whys" for the error budget (versus assigning blame/promoting risk aversion). Once a team exceeds the error budget, it will be on code freeze (duration is left up to the team). During this time, team is asked to come up with actions to de-risk new changesets (usually action items from post-mortem).

Ann Perry - IT Revolution14:10:52

Next, we're honored to welcome @farleys, who will share Putting the Ops in DevOps – An Infrastructure Story

👍 1
Justin Reock - Field CTO/Chief Evangelist - Gradle, Inc14:10:09

Very interesting. Thanks @michael_winslow! BTW we use Test/Failure analytics as opposed to an error budget so that we can enforce across all business units

Gene Kim, ITREV, Program Chair14:10:34

Please welcome @farleys!!! (and one wonders what he did either really great, or really awful, that led to this new role!! 😆 😆

Ganga Narayanan14:10:39

Thank you @michael_winslow,this strongly resonates! Toil has been the talk of town for us as well! Love how you stood up an empowered SRE team!

Joe E Drury14:10:52

@michael_winslow Thank you very much for this fantastic presentation/discussion. Bravo!!!

Gene Kim, ITREV, Program Chair14:10:17

Some very similar themes here to the fantastic talk from Capital One from @girija.rao @denee.ferguson and @jennifer.miles!

❤️ 4
Courtney Kissler14:10:18

Welcome @farleys

Marco Coulter (Speaker DOES21)14:10:39

@michael_winslow Great session, mate. TIL.

🙌 1
upvotepartyparrot 1
Gene Kim, ITREV, Program Chair14:10:46

“how we may have left infrastructure behind”

❤️ 2
Dipesh Bhatia14:10:50

@michael_winslow Leadership was sold on error budget. We have a monthly meeting where go over error budget and a OKR to reduce the max errors that we were seeing . The SRE helped us with analysis and going to architecture

Gene Kim, ITREV, Program Chair14:10:57

(Reminder: add your title and org to your Slack name — it helps everyone! 🙂

🎉 1
👍 1
Gene Kim, ITREV, Program Chair14:10:16

I love that @farleys has insurance designation!

Scott Nasello - Nike - Director of Digital Support14:10:33

@michael_winslow Great talk! I'm going to be spinning up a new SRE org at Nike and your talk will be very helpful for our future

❤️ 6
Stuart R Ainsworth14:10:59

@michael_winslow great presentation. I've struggled with the concept of the "premium" offering; I have a bunch of "helpers" on my team. I'd love to hear more about how you drew the line.

2
Chris Kernaghan14:10:22

@nasello.scott - Good luck on that, sounds like a lot of fun and rewarding

❤️ 2
Gene Kim, ITREV, Program Chair14:10:09

Nationwide: $46B revenue; #1 in numerous categories; 8400 technologists; $1.3B IT spend, 2700 applications

👏 1
BMK-BNZ-ValueStreamArchitect14:10:56

Stephen 30 Years with Nationwide, wow 🙂 (he was referring his family working for Nationwide)

Gene Kim, ITREV, Program Chair14:10:18

Infrastructure now supports 60 product teams — “this is a new construct for us, took us awhile to be able to determine this”

Nick Eggleston, IT Architect (open to offers)14:10:01

Definitely interested in the follow-up discussion about the help @michael_winslow asked for with error budgets. Love to join a discussion with , @jpetoff, @cleng and others...

👍 5
jeff.gallimore (CTIO - Excella, he/him)14:10:36

@cleng will be hosting a birds of a feather session on sre today!

👀 3
☝️ 2
Amri Abuseman - Flatiron Health, Director of Quality Engineering14:10:43

I can share our experience with implementing error budgets for teams that reach incident threshold (quarterly)

❤️ 2
BMK-BNZ-ValueStreamArchitect14:10:00

This is where our conf value is and especially in-person all these hallway conversations - learning so much from our DevOps Enterprise community

💯 1
Chris Kernaghan14:10:21

New term to research - scrumfall, feels familiar in SAP

Gene Kim, ITREV, Program Chair14:10:23

“I drew a pink circle, not red as it’s in trouble, but just to focus our attention and infuse transformation into infrastructure” — @farleys

👏 3
Nick Eggleston, IT Architect (open to offers)14:10:03

@farleys what did you use for benchmarking against peers?

Gene Kim, ITREV, Program Chair14:10:24

“I had a guilty feeling — I was so indexed on the software [dev] side of the side, that I never paid much attention to infrastructure and ops” — @farleys 😆 A familiar journey!!!

Gene Kim, ITREV, Program Chair14:10:54

@farleys Was that code for “I used to complain about Infrastructure and Ops all the time?”

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:04

We actually did financial benchmarking using key vendors for those benchmarks in Infrastructure cost....peer comparisons on financials. We were way off the mark at first look so needed to rationalize whyf

👏 2
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:06

speaking from the infrastructure/ops side of the house.... sometimes it's hard to understand how to apply agile to our world!!

💯 1
Chris Kernaghan14:10:08

We have started treating it that way in my firm - it's not an easy road as lots of folks don't quite get it

Gene Kim, ITREV, Program Chair14:10:58

“Product approach to I&O made sense: we found product orientation to make sense; we found that those teams that built and ran their own services to have better lead times, better employee engagement”

❤️ 1
Gene Kim, ITREV, Program Chair14:10:31

Big mystery: “given global nature of infrastructure, could we actually achieve value stream independence?” A GREAT QUESTION!

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:36

Service as a Product vs the other flavors of product orientations always interesting conversation. We try to treat them like Product teams with Tech Services Roadmaps etc. but in reality they support our real product teams that Build/Ship/Run real infrastructure and software

👏 1
Luke Rettig - Target14:10:51

this is so awesome @farleys - driving product thinking into Infra/Ops is a big thing at Target at the moment!

👍 3
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:56

Wow this is such a cutting edge conversation - so many folks talking about this

Nick Eggleston, IT Architect (open to offers)14:10:58

Very interesting question... how do you charge for it?

Luke Rettig - Target14:10:07

sweet - we should chat 🙂

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:44

Long time ago but did work in City Center office downtown MN. Lived in Plymouth.

Dipesh Bhatia14:10:23

@farleys Budget is in the components hand . We did sub-account for all of our components

Steve Fowler - American Airlines - Architect14:10:24

Very interested about this as we organized infra into products earlier this year

BMK-BNZ-ValueStreamArchitect14:10:04

What is product for infra team:

❤️ 1
Nicole Forsythe14:10:05

We talk a lot about how product can be defined and I’ve personally settled on company by company what matters is only that we’re aligned on what the products are FOR US.

💯 1
Craig Cook - IBM14:10:34

Charge back for services is hard. My org, Toolbox@IBM, is in the charge back business to internal teams.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:16

^^^^ THIS. I have worked in both worlds... I vastly prefer not having to deal with chargeback models

Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:56

@farleys did you train engineers into product mgmt role?

Gene Kim, ITREV, Program Chair14:10:16

“You have to own all elements of products; you have to own unit costs, SLAs, lifecycle, and understand what drives unit costs.” Holy cow, I don’t recall this being taught in HP OpenView certification course @farleys 😆

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:27

Yes stay tuned Charlie on the new roles we needed for product management.....

jeff.gallimore (CTIO - Excella, he/him)14:10:40

“driven by user empathy” for an infrastructure team! that’s awesome!

🎉 1
Luke Rettig - Target14:10:48

if you assume your users have other choices and your job is to make something that those users love, yet work for your business. good things will happen

💯 1
Gene Kim, ITREV, Program Chair14:10:42

FINSIM: Financial Simplification: created in concert with Finance.

Chris Kernaghan14:10:52

Holy Moly - @farleys - you are articulating my current challenges and giving me so many great ideas on who to progress

Gene Kim, ITREV, Program Chair14:10:54

44 products supported by 26 teams!

Nicole Forsythe14:10:54

What we’ve been using at Best Buy

👏 1
❤️ 4
Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:40

That 44 is really just Platform product.....meaning we unitized and charge for. Those role into Product Suites which are departmentalized

👍 2
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:16

This is the question of Team:Product cardinality. Default assumption of 1:1 is often not valid.

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:35

Absolutely true http://Charlie.tg one should not assume a 1:1 relationship of team to product. We have large products supported by multi-team and small products where one team support multiple products. When I say that I mean products that are unitized and have their own backlog of work. We try to create product team alignment that makes sense but often a single team might own a couple products and vice versa.

Gene Kim, ITREV, Program Chair14:10:45

All columns on right are charged for, unitized. @farleys Is there an easy definition for “unitized?”

Gene Kim, ITREV, Program Chair14:10:16

As in “we turned it into a chargeable unit?” @farleys

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:33

@farleys I don't see underlying things like circuits or routers/switches in play here... how are those costs accounted for?

1
1
BMK-BNZ-ValueStreamArchitect14:10:18

Interesting @farleys, so you consider every vendor product also as a product, example Tomcat? referring to this picture

Daniel Cahill - Engineer - Ontario Systems14:10:50

Was there something that you thought would fit the category of a service team at first but needed to change later?

Amy Cheng - TELUS14:10:54

I love the in depth look at products. Thanks @farleys!!

❤️ 2
Craig Cook - IBM14:10:22

Wow. Breaking down costs to ID's is hard.

👍 2
Gene Kim, ITREV, Program Chair14:10:25

IAM unitization — high res screenshot here from his slides • Ping Cloud Director: $0.59 / user • internal IAM: $269

👏 2
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:39

The semantics of "Product" vs "Capability" are very interesting. 15 years ago at Best Buy I was a "Product Capability Manager" - the richness of that concept has only dawned on me in recent years. @nicole.forsythe is thta title still used?

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:18

Sometime yes on the specific product question like Tomcat or Oracle. But not always unitized at that low level. But you will see the Database Product Suite would have those low level sub-products that we must support and help dev teams with from an infrastructure perspective

🙏 1
Gerard Drossart14:10:27

My company has also been more focused on the development side for Agile and DevOps (and still is). How does this structure really help the development teams delivery faster/better?

Marc Price - Nationwide Building Society (Speaker)14:10:58

what's the difference between tech delivery lead and tech delivery professional?

Khan, Humayoun at TELUS14:10:05

I like the sound of Technology Delivery Leader!

❤️ 4
Nicole Forsythe14:10:26

This comes directly from my mentors at https://www.humanizingwork.com/, but I often use the definition of Product Ownership as “Great product ownership is learning and communicating what to build (including who and why) in collaboration with a cross-functional team and other stakeholders in order to maximize the flow of value through that team.”

❤️ 4
Nick Eggleston, IT Architect (open to offers)14:10:58

How does this relate to what Gartner is calling Digital Humanism (if you know)?

Nicole Forsythe18:10:10

IDK about the Gartner stuff, so can’t say.

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:38

They help the software teams first by having product teams infuse self service automation, help software teams with monitoring infrastructure components, and provides feedback mechanisms for software teams

1
Gene Kim, ITREV, Program Chair14:10:39

I love all your focus on helping everyone learn thru this transition, @farleys — and your obvious standard for excellence and clarity!!! I told @mik how blown away I was by how you tackled this!

👏 3
Gene Kim, ITREV, Program Chair14:10:13

@farleys What was your biggest surprise in this journey?

👀 1
Denver Martin, Dir DevSecOps, he/him14:10:31

This is making me want to look at moving my insurance.... so much great thoughts and actions...

❤️ 3
💯 1
Dave Burnison14:10:50

Great presentation. There is soooo much here that it could be unpacked into a multi-day conference.

💯 7
2
Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:46

Just surprised how Infrastructure engineers were not influence by the DevOps movement over the years and how it could apply to them. Software teams were all over it at Nationwide. Thus the surprise to me when I took this new role.

❤️ 5
😂 1
Chris14:10:22

You re not alone, still hearing we don t need devops for infra… However including them in the clients for our platform, probably even the first to work with to add self service, automation to infra services, as they re building blocks for all other products and if we don t get this velocity first, it will be difficult to serve the business products

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:10

totally agree and personally I have landed on infrastructure needs to infuse with Software Teams and also embrace DevOps for what they personally Build/Ship/Run in engineering. If infrastructure is not thinking about continuous flow, automation in all we do, along with all forms of monitoring that both Software teams and Infrastructure product teams can leverage.... then how are we really helping the DevOps movement.

👍 1
BMK-BNZ-ValueStreamArchitect14:10:03

The slide pack has so much to takeaway - probably I need to playback multiple times and unpack this. Blown away by the effort behind the project to product transformation. Awesome @farleys

👍 1
💯 4
3
Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:14

A great book on internal costing/economics is https://www.amazon.com/Internal-Market-Economics-resource-governance-principles/dp/1892606313 - Dean has kind of been a "voice in the wilderness" on this

🔖 1
👏 1
Gene Kim, ITREV, Program Chair14:10:29

“I had passion for how we respond to product feedback: feedback doesn’t mean a meeting — people don’t feel heard.” So good. As @farleys suggests, no amount of meetings make people love a product!

David - BBY14:10:33

Does CI's mean Continuous Improvements in this context?

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:31

Yes Continuous Improvement Culture and discipline in our backlogs

👏 2
Craig Cook - IBM14:10:57

Is there also a focus on value?

David - BBY14:10:04

Very cool - love the talk so far @farleys

Gene Kim, ITREV, Program Chair14:10:06

$25MM in identified savings that we’ll capture in the next couple of years

Daniel Cahill - Engineer - Ontario Systems14:10:25

When do you talk about savings in terms of "This is what we will save this month" to "This is what we will save in the next couple of years" I try to sell my cost saving efforts but it hasn't gained as much traction.

jeff.gallimore (CTIO - Excella, he/him)14:10:20

$25m savings… 💰💰💰 that should get some attention.

👏 1
BMK-BNZ-ValueStreamArchitect14:10:00

@farleys - Love the framework and the ideas behind the project to product transformation. Know I realized I was undermining the "Platform as Product" in a complex enterprise. This is real eyeopener. Probably opensource all these ideas or probably an addition/reference implementation to @mik book.

Gene Kim, ITREV, Program Chair14:10:34

($25MM is 2% of entire $1.3B tech spend. Wild.)

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:14

Working directly with Mik on next evolutions of Product adoption at Nationwide in how we associate value streams to our products

❤️ 1
David - BBY14:10:20

Are you guys making heavy use of Tasktop?

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:49

We are a tasktop customer for integration of our DevOps tool chain. I am personally working with Mik on his new value stream tools and how that might help us with our Product Orientation and thinking as we try to map more at the enterprise level

David - BBY14:10:22

ok, very cool; thanks @farleys - and good luck on the Next Gen tooling!

Mik Kersten (Project to Product, Tasktop)15:10:07

Amazing work on this @farleys! One of the best examples of product orientation for operations that I have come across in large enterprises!

Steve Farley - VP Infrastructure Operations, Nationwide Insurance18:10:03

Thanks Mik and can't wait to get working with you on how we can partner in the evolution with Nationwide. Gave you a few Kudo's along the way today

Gene Kim, ITREV, Program Chair14:10:27

Container platform based on Rancher is a growth product — “we’re investing in it, expecting more customers; not focusing on savings.”

❤️ 2
👏 1
jeff.gallimore (CTIO - Excella, he/him)14:10:35

i love the distinction between “efficiency” products versus “growth” products. they should be treated differently (three horizons/zone to win).

Charlie Betz - Forrester Research - Principal Analyst, DevOps/ESM14:10:01

@farleys this is terrific. I'm talking to multiple large organizations attempting this. Where did you find the product mgmt talent for the engineering offerings, primarily? Engineers upskilling, or product mgrs learning the platforms?

💯 1
Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:14

Not sure if you can read....but hot of the press this week on what our CTO's love about our transformation, CTO means CIO at Nationwide

❤️ 1
BMK-BNZ-ValueStreamArchitect14:10:14

One question on "Vendor lock-in" in most of the enterprises, how did you or dealing with that? @farleys

Gene Kim, ITREV, Program Chair14:10:20

“Previously, people don’t know what they’re paying for: mainframe? storage? Now they know: databases, compute, storage, mainframe MIPS, IDs.” Wow.

💰 1
👍 2
Alex Broderick-Forster, IT Revolution, Event Staff14:10:59

will be up after the morning plenary talks finish airing!

👏 1
❤️ 1
Dave Burnison14:10:26

Ah, ok. :thumbsup:

Brian W. Spolarich - Cal Poly14:10:53

Is IT ops now funded entirely via cost recovery models?

👀 1
Steve Farley - VP Infrastructure Operations, Nationwide Insurance15:10:08

Yes Brian IT Ops is recovered either thru Product Units or some still allocated across business unit. The Unitized model is driving transparency and alignment in what is being consumed and actually driving consumption reduction.

Joe E Drury14:10:03

@farleys Wow, thanks for sharing so much about your transformation to a product centic model...pure gold!!! Great job!

Gene Kim, ITREV, Program Chair14:10:22

“I&O group now has top quarter Gallup engagement scores; people now know what’s expected of them; they’re accountable for plan/build/run and to delight your customers” ❤️ ❤️ ❤️

❤️ 4
Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:30

Vendor lock is real and needs regular working especially CLOUD lock-in, but also platform lock-in, and even SaaS lock-in or big purchased App lock in. We struggle with it.

💯 2
thankyou 1
Scott Nasello - Nike - Director of Digital Support14:10:52

@farleys - did you apply anything like Wardley Mapping to think through some of these challenges?

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:18

We did not Scott but will look at it.....thanks for the tip

Scott Nasello - Nike - Director of Digital Support14:10:30

@farleys Steep learning curve with Warley Mapping for sure, but can be helpful to identify duplication, in appropriate methods, technical debt, etc. Here's a good thread as an overview: https://twitter.com/swardley/status/971026844664332289?s=20

BMK-BNZ-ValueStreamArchitect14:10:08

I guess that is same with multi-cloud as well?

👍 1
Ganga Narayanan14:10:19

Loved the presentation, @farleys! Did you have a team of internal champions for training / Dojo/ consulting etc?

❤️ 3
Anderson Tran14:10:40

Your verticality in a provider determines your lock-in. Accenture has an amazing readout of services comparisons and lock-in potential in their "cloud canvas"

Gene Kim, ITREV, Program Chair14:10:57

…I love that these teams eventually became sherpa into the cloud, @farleys

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:06

Yes we created a Product Champions Model and Guilds to support

Dave Burnison14:10:10

Thank you @farleys for a FANTASTIC session!

💯 1
Gene Kim, ITREV, Program Chair14:10:13

Thank you so much, @farleys!!!

💯 3
❤️ 1
jeff.gallimore (CTIO - Excella, he/him)14:10:19

“hopefully that’s an interesting infrastructure story.” ummm… yeah. SO MUCH!!!

👍 4
BMK-BNZ-ValueStreamArchitect14:10:21

One thing I could tell you after seeing @farleys presentation, most of the enterprise will reboot their platform as product transformation work

👍 1
Joe Arrowood14:10:34

Awesome Talk, Thank you @farleys

Khan, Humayoun at TELUS14:10:35

Thanks @farleys for an amazing insights !

Ann Perry - IT Revolution14:10:40

🌟 And now, we welcome @kboth_does @christina and @aalvare2, here to present "She’s Not Dead Yet, Jim": Vulnerability and Retrospectives in Emergency Medicine 🌟

👏 2
Nick Eggleston, IT Architect (open to offers)14:10:50

@farleys what kinds of help are you looking for from the community?

Christina Tan, Strategy at Blameless, Speaker DOES2114:10:19

@kboth_does, @aalvare2, and I are here for live discussions. Feel free to ask any questions!

👍 3
BMK-BNZ-ValueStreamArchitect14:10:58

Looking forward to it...

Steve Farley - VP Infrastructure Operations, Nationwide Insurance14:10:21

Very proud to be a part of this community...we learn so much from each other so humbled by your comments and I have learned much from everyone at the conference this year.

👏 6
❤️ 5
Gene Kim, ITREV, Program Chair14:10:38

Please welcome Dr. @aalvare2 Clinical Assistant Professor, Emergency Medicine, Stanford Medicine, who is responsible for Director for Well-Being and Human potential, on implementing blamelessness in the ER. Co-presenting with @kboth_does and @christina

👏 2
👋 3
1
Nicole Forsythe14:10:53

I just realized @genek’s books are organized by color. As a former Librarian, I both appreciate and shudder. 😜

👏 2
😆 3
🌈 1
Gene Kim, ITREV, Program Chair14:10:25

That marvelous achievement was done by my boss, @mvk842 . I was so proud I had enough great books in all parts of the color spectrum!!! 😆

❤️ 3
Margueritte Kim (CEO, IT Revolution, she/her)14:10:12

@nicole.forsythe - I know…the Dewey Decimal System weeps, but the visually pleasing outcome cannot be denied. 😬😂

Nicole Forsythe14:10:47

I was a college librarian so Library of Congress FTW! I organize mine by my own conceptual linkages…but it’s definitely something I think about a lot.

Gene Kim, ITREV, Program Chair14:10:32

“what can we learn from the high tempo, very high consequence world of emergency departments”

Kurt A, Blameless, SRE Architect14:10:39

For people who want some additional decoder rings, here's the resource list from the end of our presentation (with all the book citations we mention): Chatter - Ethan Kross Awaken Compassion in the Workplace - Monica Worline, Jane Dutton The Checklist Manifesto - Atul Gawande Retrospectives for Humans - Courtney Eckhardt The Centre for Compassion and Altruism Research and Education (CCARE, Stanford) Dare to Lead - Bréne Brown Incident Metrics for SRE - Štěpán Davidovič Behind Human Error - Woods, Dekker, Cook, et al. Just Culture: Who are We Really Afraid of? - Steven Shorrock Just Culture - Sidney Dekker Tribute to Dr. McCoy - https://youtu.be/IAsaHZ-m3k0

🔖 3
👍 2
❤️ 2
Gene Kim, ITREV, Program Chair14:10:23

“I am the Director of Well-Being at Stanford emergency medicine clinically. I’m an emergency physician at Stanford. I serve as the associate residency program director focused on quality patient safety and process improvement. I also serve as the co-chair of the physician wellness forum at Stanford. By the way are my friends here. And so please feel free to call me by my first name.” — @aalvare2

👏 1
❤️ 1
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:36

haha, yes, Al'ai is very down to earth

1
❤️ 3
Daniel Cahill - Engineer - Ontario Systems14:10:47

The presentation from Dr. Chris Strear, Idealcast with Trent Green, healthcare chapter of HVE by Dr. Spear, and discussion of the new case study in DevOps Handbook v2 have me so excited for this talk. I've started reading books about improving healthcare it is so interesting. So great to be able to learn things from them too!

❤️ 2
Gene Kim, ITREV, Program Chair14:10:01

15 minutes for each medical debrief

Gene Kim, ITREV, Program Chair14:10:42

focus on individual, team, and then environment — individual focus helps avoid finger-pointing.

💯 2
jeff.gallimore (CTIO - Excella, he/him)14:10:48

bringing people back to the shared goal of saving peoples’ lives. yesssss. excellent!

💯 5
Ganga Narayanan14:10:51

Very interesting @kboth_does! Thanks for the references as well! Been looking forward to reading Chatter!

C14:10:18

model failure!

❤️ 3
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:30

We also have a resources slide at the end of the slide deck, Chatter is on there

❤️ 1
Gene Kim, ITREV, Program Chair14:10:42

leaders set tone by sharing their own mistake first; “I was nervous when patient first arrived and went into cardiac arrest; my focus caused void in leadership” (!!)

❤️ 10
Nick Eggleston, IT Architect (open to offers)14:10:25

Important that the shared mistake is non-trivial, right?

jeff.gallimore (CTIO - Excella, he/him)14:10:26

one important technique promoted by dr. amy edmondson for promoting psychological safety

Chris Kernaghan14:10:39

MrsK is amazing at doing this in her clinical skills retrospectives - fascinating in seeing the difference between education leaders and management

Amy Cheng - TELUS14:10:39

Practice practice practice sharing vulnerably!

Christina Tan, Strategy at Blameless, Speaker DOES2114:10:19

@nickeggleston No, if it 's immediately in the moment post procedure debrief, it's okay to be small. Everyone's stressed and tense, even sharing a small mistake can make a big difference.

Christina Tan, Strategy at Blameless, Speaker DOES2114:10:18

"what's the 1% change I can do to better the outcome?"

Gene Kim, ITREV, Program Chair14:10:33

focus on 1%: what is that small change that could change/deflect the outcome; especially for devastating events; that we can actually achieve

Nicole Forsythe14:10:43

A mentor of mine recently shared the concept of moral foundations theory as a way to create a blameless environment that’s required for the psychology safety. He recommended the book https://dividedwefall.com/2018/07/15/the-righteous-mind-moral-foundations-theory/#:~:text=In%20his%20groundbreaking%20book%2C%20%E2%80%9CThe,%2C%20Fairness%2FCheating%2C%20Loyalty%2F.

👏 2
BMK-BNZ-ValueStreamArchitect14:10:03

Interesting talk - excellent - @christina;I love @aalvare2 this slide and his explanation around this :

Gene Kim, ITREV, Program Chair14:10:05

inspire ownership: “Dr. Christina did a great job; how about we have her lead the rest of the debrief” (decision made before-hand)

❤️ 2
Nick Eggleston, IT Architect (open to offers)14:10:22

Do you record the debriefs for future analysis?

Nicole Forsythe14:10:02

We personally don’t because it would impact the psychological safety of the room, but we did debate it. (Not that we are in the same industry and not that you asked me.)

Kurt A, Blameless, SRE Architect14:10:38

Generally not - there are not facilities to do that in the hallway 🙂 And it could undermine the safety of the conversation.

Daniel Cahill - Engineer - Ontario Systems15:10:00

I find a lot of value in pattern matching problems but it seems like you are saying iterative learning is more important. Thats a new way of thinking of things for me.

Kurt A, Blameless, SRE Architect15:10:26

continuous improvement vs. big bang 😄

Christina Tan, Strategy at Blameless, Speaker DOES2114:10:23

"The goal of debriefing is to unpack and create a shared mental model."

👏 3
❤️ 1
Chris Kernaghan14:10:28

Shared mental models are so powerful

💯 3
Ganga Narayanan14:10:44

We use the term project post-mortem so frequently that we forget that the term comes from medicine! This is so awesome!

👍 2
1
Amy Cheng - TELUS14:10:18

Post Implementation Review, After Action Reviews!! yep

Al'ai Alvarez, MD14:10:50

I’m sure we learned it from the business world, as well.

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere14:10:00

This is a good point. Noting that this talk uses the term "debriefs" and not "post-mortems" because of the very real correlation in emergency medicine. Curious your take @aalvare2.

❤️ 2
👏 1
Al'ai Alvarez, MD14:10:10

After Action Review is a better term, in my opinion. It takes away the stigma from the word, “Debrief”

👍 2
Al'ai Alvarez, MD14:10:11

post mortems are fine for physicians, although again, it implies someone died. We use AAR for celebrations, too, when things have gone well and we want to highlight positive deviations in practice.

Kurt A, Blameless, SRE Architect14:10:40

@ckissler mentioned that she has been calling them learning reviews

👏 1
Nicole Forsythe14:10:25

It’s interesting how every industry has verbiage. From Captain Marquet’s Turn the Ship Around, they got away from “debrief” is it was laden with meaning and tradition. I often prefer “retrospective” but a team inoculated against Scrum by bad experience that doesn’t work.

Al'ai Alvarez, MD14:10:28

I like Learning Reviews. It does, however, also imply we have to learn something (we do… though in the heat of the moment, it may feel gaslighting for those who’s already learned something, ie, they were directly involved in a medical harm, etc).

Al'ai Alvarez, MD14:10:57

AAR is agnostic to whether it was a positive or negative outcome. It’s just a review.

Kurt A, Blameless, SRE Architect15:10:49

debriefs are the quick, immediate recaps; case review is the later, bigger discussion

❤️ 1
jeff.gallimore (CTIO - Excella, he/him)14:10:04

there are so many parallels here to the technology world — multiple, different perspectives: nurses, doctors, social workers, …

💯 2
👏 1
Chris Kernaghan14:10:52

Technology has so much to learn from other industries - instead of trying to reinvent stuff.

💯 1
Gene Kim, ITREV, Program Chair14:10:28

overriding goal: ensure shared mental model — we didn’t all see/feel the same thing; give people heads up beforehand, so they’re ready, vs. processing/surprised in meeting, feeling defensive (so much pre-work for the leader!)

👏 1
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:25

The pre-meeting heads up make a big difference in the level of psychological safety the team feels

❤️ 3
Frotz Faatuai (Cisco IT - he/him)14:10:49

“Shoulder-to-Shoulder” (ref: @jmrichardson1)

👏 2
BMK-BNZ-ValueStreamArchitect14:10:53

In this space - you do not waste time like "Tech" in meetings, because it is all about "Saving Lives" - The one thing what we could learn (Legacy Enterprises or old mindset) from this.

Gene Kim, ITREV, Program Chair14:10:13

a tough case, where patient died; lots of mixed emotions; after resuscitation; surgeon not happy; why? shared perspective, but I want to hear from you b/c no time to brief senior, I stood next to surgeon, to ensure he/she not alone; I never expected trauma surgeon to say “I screwed up”, because so different than norm. It was huge, b/c we had obviously created safe space for professionals to share mistakes.

Marc Price - Nationwide Building Society (Speaker)14:10:32

psychological safety is so important

💯 5
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:24

"Compassion allows people to move past self blame and into a mode of curiosity - what can I do better?"

BMK-BNZ-ValueStreamArchitect14:10:28

@aalvare2 referring to Mindfulness 🙂 Oh yes - not just in Med and but everywhere

❤️ 1
Gene Kim, ITREV, Program Chair14:10:07

how do you coach people on self-blame? I think people need to experience the negative emotions, like grief, but not feel shame; don’t try to cheer people up, b/c it sets up internal conflict; you’ll have many mistakes, and you’ll feel crappy, but this is where compassion is so important. Can’t do coulda/shoulda/woulda game. Be kind to yourself.

❤️ 6
Kurt A, Blameless, SRE Architect15:10:08

Also fits with the points that @jmrichardson1 and @emily356 made in their workshop yesterday about supportive connections 🪶 and just like me

Pete Nuwayser - IBM14:10:09

Have to start with compassion for yourself

❤️ 1
Virginia Laurenzano NSA/MARFORCYBER14:10:32

"Just like me" - came up with @jmrichardson1 and @emily356 and now here

❤️ 4
🪄 1
Al'ai Alvarez, MD14:10:53

it does highlight the commonality in our humanity.

Nick Eggleston, IT Architect (open to offers)14:10:33

How does legal risk management feel about admissions that someone screwed up? Often they are very concerned about potential lawsuits..

👀 2
Al'ai Alvarez, MD15:10:07

great point. There’s a lot of work about talking about mistakes and even admitting it to patients and how this minimizes litigation.

Gene Kim, ITREV, Program Chair14:10:21

high consequence ===> someone died, with all sorts of ramifications to individuals, teams, and orgs.

Scott Nasello - Nike - Director of Digital Support14:10:31

@aalvare2 - I'm interested, how did you get introduced to the DevOps / SRE community? What are you most surprised about when working with this community?

2
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:27

Al'ai and I met through a year-long compassion cultivation mindfulness program at Stanford, we also share that program in the resources slide.

❤️ 7
Al'ai Alvarez, MD15:10:13

Yes, I’m honored to be here, and grateful for the opportunity to work with @christina at Stanford Center for Compassion and Altruism Research and Education via the Applied Compassion Training.

❤️ 1
Al'ai Alvarez, MD15:10:38

In essence, this is a manifestation of our training… applying compassion to each of our workspace.

Scott Nasello - Nike - Director of Digital Support15:10:59

I've always loved hearing from the Physicians in our community such as Dr Richard Cook, thank you for being here and your presentation !

❤️ 1
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:34

Compassion and movement away from blame is a form of empowerment - so people can shift the focus to how one can do better, 1% better

Daniel Cahill - Engineer - Ontario Systems14:10:55

If you think of something the next day that you could've done better, do you find ways to discuss that with the team again?

Gene Kim, ITREV, Program Chair14:10:55

debriefs can also be about celebrations: after incredible resusciatation: let’s pause and celebrate: “this is what it feels like to save a life” (goosebumps too!)

🎉 1
Christina Tan, Strategy at Blameless, Speaker DOES2114:10:04

"This is what it's life to save a life." LIke @emily356 said yesterday, focus on what's wroking well

❤️ 2
Nicole Forsythe14:10:05

@jmrichardson1 yesterday talked about the importance of recognizing the good stuff, too! Very similar thematically.

💯 2
BMK-BNZ-ValueStreamArchitect14:10:08

Book mark the positive feelings

❤️ 3
Margueritte Kim (CEO, IT Revolution, she/her)15:10:33

Saying this out loud actually makes me tear up.

2
Al'ai Alvarez, MD15:10:13

ahh. yes. i get goosebumps, and it’s a positive reinforcing action.

❤️ 1
BMK-BNZ-ValueStreamArchitect15:10:55

Well done @aalvare2 - I am not surprised, you will be keynoting now most of the Tech Conferences 🙂

❤️ 1
BMK-BNZ-ValueStreamArchitect15:10:11

Welcome on-board Doc @aalvare2

❤️ 1
Margueritte Kim (CEO, IT Revolution, she/her)15:10:14

@aalvare2 - I am going to say this to my kids every single day from now on! (In addition to my usual, “make good choices!” as they get out of the car. LOL. Asian mom style.)

❤️ 1
😆 1
Pete Nuwayser - IBM14:10:14

We can also have a whole conversation about responses to self-blame • do acknowledge the feelings without trying to cheer the person up or negate their feelings in the moment • do know if it's appropriate to respond with relatability or not be relatable in response

Pete Nuwayser - IBM14:10:25

This should be a Lean Coffee topic

💯 3
Courtney Kissler15:10:56

@annp - let's capture this for the retro and input for London

Pete Nuwayser - IBM15:10:30

Thanks @ckissler!!

Al'ai Alvarez, MD15:10:49

oooooh! whatever this is, it sounds fun

Pete Nuwayser - IBM15:10:26

@aalvare2 small groups that crowdsource and then discuss subtopics on a theme

❤️ 1
Pete Nuwayser - IBM15:10:31

it's been a centerpiece of the DOES afternoon agenda for years

Pete Nuwayser - IBM15:10:47

(and I love it) 🙂

1
BMK-BNZ-ValueStreamArchitect15:10:33

(I would love the Incident Tech Managers are like ) @aalvare2 - Love your work

❤️ 1
Al'ai Alvarez, MD15:10:33

thank you. To be clear, this takes an entire team for after action reviews to work effectively. It takes training and practice.

👍 4
BMK-BNZ-ValueStreamArchitect15:10:04

IMHO - What I would say - our Incident managers in Enterprise IT miss.

Gene Kim, ITREV, Program Chair15:10:53

to leaders: in emergency medicine, people can die, or have long term complications; you can’t say that it’s ok. but also recognize that there’s only so much can do; what are systems issues vs. inherent to disease itself massive heart attack: not right: how can medical in top flight institution prevent a heart attack? maybe interruptions <--- in our control ! or maybe it’s because of the pandemic, when we knew so little about pandemic <--- that is out of our control

❤️ 1
Daniel Cahill - Engineer - Ontario Systems15:10:21

What was the book he just recommended?

Christina Tan, Strategy at Blameless, Speaker DOES2115:10:42

All the book recommendations will be in the resources slide

👍 1
Graham McGregor15:10:24

💯 recommend The Checklist Manifesto

🔖 2
Gene Kim, ITREV, Program Chair15:10:49

designing for well-being: how do leaders use language to create learning and psych safe org

Chris Kernaghan15:10:54

Language matters - lots of people forget this and it often causes issues in retrospectives

❤️ 1
Denver Martin, Dir DevSecOps, he/him15:10:55

Love Checklist Manifesto - I give it to all of my teams leads to help with documentation of work...

👏 1
Justin Reock - Field CTO/Chief Evangelist - Gradle, Inc15:10:20

This is an incredibly valuable session and perspective, thank you

❤️ 2
Gene Kim, ITREV, Program Chair15:10:36

At Stanford: “peer review” ---> “case review” ; it’s nota bout the peer who already feels shame; it’s about dissecting the case, understanding how we get to the outcome; and we protect physicians and frontline workers.

❤️ 4
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:54

"Just like you, I've made many mistakes. Just like me, you'll make mistakes, too"

❤️ 2
Denver Martin, Dir DevSecOps, he/him15:10:54

Before, would get pages and pages of procedures and not quick actionable checklists... this book helped them see the difference.

BMK-BNZ-ValueStreamArchitect15:10:56

I have to do what @genek does - Holy Cow. So much in this talk

2
❤️ 1
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:17

Hard to keep myself from smiling during the call, b/c the recording felt so magical when everything we've prepared for came together

❤️ 2
👍 1
Gene Kim, ITREV, Program Chair15:10:16

Kurt: don’t use word “why” — use “what” or “help me understand”; “why” asks a judgement or agentive: who can we hang/blame for this?

💯 2
❤️ 1
Pete Nuwayser - IBM15:10:32

There are direct lines to be drawn between our response to self-blame and the three components of occupational burnout: 1. increasingly frequent feelings of emotional and/or physical exhaustion 2. increasingly frequent feelings of inefficacy 3. increasingly frequent feelings of cynicism / disengagement

👍 3
1
Kurt A, Blameless, SRE Architect15:10:21

You're telegraphing a topic later in the talk!

1
Nick Eggleston, IT Architect (open to offers)15:10:18

be factual, not counter-factual

💯 1
Denver Martin, Dir DevSecOps, he/him15:10:21

Applied Dekker --- amazing to hear and see in action...

💯 1
Marc Price - Nationwide Building Society (Speaker)15:10:26

did you do everything you could have done in the moment - yes

Jeffrey Fredrick, Author-Agile Conversations15:10:28

@kboth_does: avoiding Why questions seems similar to how @allspaw has moved incident inquiry away from Five Whys.

Jeffrey Fredrick, Author-Agile Conversations15:10:36

I’m going to guess she quotes Chris Voss, Never Split the Difference? That’s where I first heard “why is an attack in every language”.

Kurt A, Blameless, SRE Architect15:10:43

I'm not sure - would have to rewatch it, but yes, that's definitely part of her talk. She also covers lots of other language nuance.

Gene Kim, ITREV, Program Chair15:10:39

Kurt on the danger of counterfactuals: cannot play would have / could have / should have; the goal is to understand how it happened;

👍 2
Jeffrey Fredrick, Author-Agile Conversations15:10:12

“language is a powerful tool for shifting culture” < change your conversations, change your culture 🙂

❤️ 10
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:41

M&Ms -> A&As, amazing and awesome! let's celebrate wins!

❤️ 1
Al'ai Alvarez, MD15:10:24

another reason why using the term “debriefs “alone do not capture the positives. After Action Review highlights positive deviants, as well.

Chris Kernaghan15:10:54

that makes perfect sense

Gene Kim, ITREV, Program Chair15:10:54

We brought “Save of the Month” during pandemic; Morbidity and Mortality rounds: in old days: get in front of room, humiliating; “Save of Month” only highlighted one person; “A&A: Amazing and Awesome” highlight positive variance for everyone!!!! (Profound lens to view these rituals)

jeff.gallimore (CTIO - Excella, he/him)15:10:57

“highlight positive variance”. let’s spotlight some wins!

❤️ 1
👏 1
1
BMK-BNZ-ValueStreamArchitect15:10:10

DOES friends, I will be not surprised, we will see @aalvare2 now keynoting in our Tech conference circuit. Doc got that - Whatever it is.

😂 2
❤️ 7
BMK-BNZ-ValueStreamArchitect15:10:50

@aalvare2 - It will be really good to listen to you and our influence @allspaw together - This will be great session.

❤️ 1
Pete Nuwayser - IBM15:10:11

I would also add that, at DevOpsDays NYC 2020, we had a wonderful Open Space discussion on Burnout that was attended by a trauma / ER doctor who wanted to learn about solutions to burnout. This was held the first week of March, just before lockdown. She attended DOD not because she wanted to learn more about DevOps but because of the common themes.

❤️ 1
Pete Nuwayser - IBM15:10:42

I can look her up on Twitter

BMK-BNZ-ValueStreamArchitect15:10:57

PLEASE DO< Thank you @nuwayser

Pete Nuwayser - IBM15:10:30

cool - will report back on this thread

Al'ai Alvarez, MD15:10:18

Dr. Shannon McNamara is gold. She’s one of our faculty for the High Performance Resuscitation Teams Summit we’re hosting in May 2022. https://ce.mayo.edu/emergency-medicine/content/2021-high-performance-resuscitation-teams-summit-postponing-may-2022-canceled#group-tabs-node-course-default1

👏 1
John Allspaw15:10:09

Shannon is a friend of mine as well - indeed one of the deepest thinkers in her domain and doing lots to bridge across domains ❤️

❤️ 1
👏 1
1
Pete Nuwayser - IBM15:10:58

John, I think YOU were at that Open Space, too!

Andy Nortrup - Director of PM at Tanium15:10:16

How are those A&As and M&Ms put together in time? Do they happen at the same time, are they different events?

Jeffrey Fredrick, Author-Agile Conversations15:10:20

A&A over M&M reminds me of @esh saying “feed that which you wish to see grow”

❤️ 5
1
Catie Martin_Design Manager_BlueCrossBlueShield of SC15:10:13

I love the idea of changing the vernacular to shift culture - I would love to hear how others have not only changed the words but followed up with actions too. What did that journey look like? What challenges did you find and then solve? What successes did you have and how?

❤️ 1
Al'ai Alvarez, MD15:10:07

Here’s a recent Tweetorial we did at Stanford Emergency Medicine as we try to use simulation to practice running after action reviews. https://twitter.com/alvarezzzy/status/1442899032674746368?s=20

❤️ 1
Gene Kim, ITREV, Program Chair15:10:20

@aalvare2 So amazing; what role does “senior” signifiy? Thank you @christina and @kboth_does for introducing Dr. @aalvare2 to this community!

❤️ 7
1
👍 2
Al'ai Alvarez, MD15:10:04

In medicine, there’s the attending (faculty), there’s the senior resident (the most advanced learner), the junior resident (interns and mid-level training), then the medical students at the earlier stages.

Al'ai Alvarez, MD15:10:32

There’s also nursing, pharmacy, respiratory therapist, others plus in trauma, the surgeons and their levels as above.

Scott Nasello - Nike - Director of Digital Support15:10:00

Sometimes we hear about Fellows, are they faculty as well?

Christina Tan, Strategy at Blameless, Speaker DOES2115:10:27

The components of building a culture of learning & psychological safety: language, rituals, and role creation.

❤️ 1
🙌 1
Gene Kim, ITREV, Program Chair15:10:48

Director of [Staff] Well Being: help train future emergency physicians; I created role for myself: training, support and case reviews; I still get palpitations when I get those emails of events Kudos to you @aalvare2 and to Stanford Medical!

❤️ 1
1
Daniel Cahill - Engineer - Ontario Systems15:10:06

With working in a fast paced environment, have you had problems where people wanted to just move on to the next patient instead of stopping to learn? Is there a way you have brought them along for them to get excited?

👀 1
Nick Eggleston, IT Architect (open to offers)15:10:20

@aalvare2 what is your approach/plan to learning about developments in the areas of focus you have shared?

Al'ai Alvarez, MD15:10:53

Can you please elaborate? You mean, how do we operationalize this? https://twitter.com/alvarezzzy/status/1442899032674746368?s=20

Gene Kim, ITREV, Program Chair15:10:02

retention/recruiting: $500K to $2MM per doctor who leaves due to burnout; burnout: leads to medical suicides; 400 physicians die of suicide per year. that’s what motivates my work. med students have lower burnout than their peers outside profession; then at every stage, then 2x burnout than their peers outside of medicine.

😲 1
Craig Cook - IBM15:10:23

Great to hear more about Mental Health issues. We need to talk about this a lot more.

❤️ 6
Al'ai Alvarez, MD15:10:20

yes, and I suspect this is not just a problem in medicine

Brian W. Spolarich - Cal Poly15:10:23

High achievers tend to be perfectionists, which can be dangerous.

💯 1
Craig Cook - IBM15:10:07

There is large wellness community inside IBM.

❤️ 1
Brian W. Spolarich - Cal Poly15:10:11

We need to understand the need for self-compassion, self care, and recognize mental health challenges in ourselves and those we lead.

❤️ 1
Virginia Laurenzano NSA/MARFORCYBER15:10:30

OMG on the burnout. family members in their first years of being attendee physicians are going to be sent this talk...STAT ❤️

❤️ 5
Gene Kim, ITREV, Program Chair15:10:17

aviation safety: so thoroughly studied by Dr. @ronwestrum who spoke yesterday

Gene Kim, ITREV, Program Chair15:10:31

fascinated by the seemingly intrinsic tie between quality outcomes and staff well-being

❤️ 1
BMK-BNZ-ValueStreamArchitect15:10:38

@genek - THIS IS GOLD, PURE GOLD - Well done @aalvare2 @christina  @kboth_does!

❤️ 4
thankyou 1
Al'ai Alvarez, MD15:10:50

thank you. It’s definitely an inspired talk thanks to @genek and team. We wanted to highlight similarities in our work, our shared human experiences/common humanity, through learning in adverse outcomes. Kudos to @christina for leading this effort, and also to @kboth_does.

4
👏 2
❤️ 4
BMK-BNZ-ValueStreamArchitect15:10:56

So you are a musician 🙂

Gene Kim, ITREV, Program Chair15:10:05

real talk: what are the architectures in tech orgs condemn technology workers to burnout and horrendous outcomes for their customers. what are you as the leader willing (and responsible) to do to fix it.

BMK-BNZ-ValueStreamArchitect15:10:58

Excellent closing about Vaccination and frontline workers - well done

❤️ 7
1
Gene Kim, ITREV, Program Chair15:10:04

From @aalvare2 “Please support initiatives that support mental health of medical and frontline healthcare professionals; and thank you for getting vaccinated: you protect yourself, and everyone else around your (and frontline workers)”

❤️ 5
1
thankyou 1
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:04

Yes, ask enough whys, architecture issues can be traced back to leadership decisions as well

💯 1
Anderson Tran15:10:11

@aalvare2 Thank you so much for the insight from medicine!

❤️ 1
John Roesler - Sr Engineer (Gap Inc)15:10:31

Really excellent presentation on the importance of these retros and some good how to tips

Marc Price - Nationwide Building Society (Speaker)15:10:34

what a session! really puts things into perspective! @christina @kboth_does @aalvare2 Thank you!

❤️ 3
👏 1
thankyou 1
Gene Kim, ITREV, Program Chair15:10:45

@christina and @aalvare2 met through Stanford Mindfulness Community <----- the importance of Connections.

❤️ 1
🪶 1
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:56

Close your eyes ----- for a break

❤️ 1
Catie Martin_Design Manager_BlueCrossBlueShield of SC15:10:58

Love the mindfulness - taking a moment to reflect is a gift we rarely give ourselves.

👏 2
💯 1
❤️ 1
BMK-BNZ-ValueStreamArchitect15:10:22

Now - we need to include Mindfulness meditation end of every session or during break hours

💯 9
Nick Eggleston, IT Architect (open to offers)15:10:40

@aalvare2 can you expand on what you said at the end of the talk about pausing to consider the other's perspective when your gut reaction is to completely disagree with what the other person said?

Al'ai Alvarez, MD15:10:07

Generous interpretation is important. Using open ended questions, appreciative inquiry… I’ve learned so much doing this because not everyone will have the same frame.

Klaudia Breslavets - Vanguard15:10:01

This was great...I really enjoyed hearing about specific behaviors that contribute to a learning organization. It's rooted in psychological safety & feeling comfortable learning from incidents on the job, being in an environment where the org can capitalize from the learning without emotional toil.

❤️ 2
Gene Kim, ITREV, Program Chair15:10:18

Thank you Dr. @aalvare2 for teaching us about all your amazing work, and improving the systems that support patients and healthcare workers at Stanford Medical. 🙏 And thank you to @christina and @kboth_does for introducing him to us. ❤️ ❤️ ❤️

❤️ 12
👏 6
1
1
Pedro Jordan15:10:05

thank you !!!

❤️ 1
Malcolm McAlpin15:10:08

Thank you!!

❤️ 1
Jennifer Collings15:10:08

Great talk! Thank you!

❤️ 1
Brian Smith15:10:09

Thank you

❤️ 1
Andy Nortrup - Director of PM at Tanium15:10:14

I could use more guided meditation at the end of meetings.

2
😊 1
❤️ 4
👍 3
Nick Eggleston, IT Architect (open to offers)15:10:16

I wonder how to handle the apparent conflict between psychological safety and cognitive dissonance... :thinking_face:

Christina Tan, Strategy at Blameless, Speaker DOES2115:10:23

Thank you everyone! Such an honour to be here!

❤️ 6
Tashfeen Mahmood15:10:36

Nice! Thank you!

Chrystina Nguyen, Rhythmic Technologies15:10:52

can we get that mantra in writing please.. May i be happy… may i be..

💯 1
Christina Tan, Strategy at Blameless, Speaker DOES2115:10:17

May I be happy.. May I be peaceful... May I be free from suffering

❤️ 3
Chrystina Nguyen, Rhythmic Technologies15:10:52

Dropping that in my discord.. Thank you for this moment!

❤️ 1
Dian Hansen15:10:58

@aalvare2 @christina@kboth_does this was an amazing and rewarding talk - thank you!!!

❤️ 4
David - BBY15:10:04

thank you for that moment of mental reset!

❤️ 3
👍 2
1
Ann Perry - IT Revolution15:10:11

It's a privilege to now welcome @amandas from Microsoft who will be discussing Leadership and Remote Work

❤️ 1
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:16

@christina leading a two-minute mental meditation to end that phenomenal talk. WELL DONE - THAT WAS AWESOME!

❤️ 8
☝️ 2
1
Gene Kim, ITREV, Program Chair15:10:39

I am so so excited to introduce the next speaker, @amandas, CVP of Product for Developer Tools at Microsoft, who I’ve admired for years. I am so excited that I finally got to talk with her a couple of weeks ago, after I heard an interview of her on Scott Hanselman’s podcast!

👏 7
upvotepartyparrot 1
Nick Eggleston, IT Architect (open to offers)15:10:10

Definitely need to go listen to that

Istvan Bathazi15:10:53

@aalvare2 @kboth_does @christina Great session. Thank you!

❤️ 2
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:26

Woohoo! @amandas in the house! So excited for this talk!

❤️ 6
Gene Kim, ITREV, Program Chair15:10:21

@mik and I have spent years marveling at the decades of achievements of the famous Developer Division at Microsoft. I can’t overstate how excited I was to finally talk with her.

👏 5
Craig Cook - IBM15:10:20

Bootcamp? Is that similar to a Dojo?

Gene Kim, ITREV, Program Chair15:10:27

Leader attributes: Awareness, Curiosity, and Courage.

Gene Kim, ITREV, Program Chair15:10:10

Bootcamp refers to the astounding 2 week onboarding process in DevDiv — did I get that right, @amandas? (So delighted and honored that you’re here!)

Amanda Silver15:10:11

Our Bootcamp is essentially the initiation class to our division. It's a 2 week course that new employees take within the first 3 months of their job in the Developer Division.

❤️ 4
Gene Kim, ITREV, Program Chair15:10:50

When was Bootcamp created? Was there a compelling event that caused it to be created?

Nick Eggleston, IT Architect (open to offers)15:10:37

"Dont want decisions to be made by the Highest-Paid Person in the Room" (HPPR) [want them made based on data]

Gene Kim, ITREV, Program Chair15:10:48

“The HiPPO: highest paid person’s in the office” 🙂

👏 2
1
Kurt A, Blameless, SRE Architect15:10:33

or Highest Paid Person's Opinion

Amanda Silver15:10:55

The first week is an intro Microsoft, the developer business, engineering practices, etc. The second week is a "Customer Driven Workshop". Everyone gets to write down their hypotheses and interview a few customers (real actual people who use our products or who we want to use our products) to test their hypotheses.

💡 2
❤️ 5
Nick Eggleston, IT Architect (open to offers)15:10:20

Hehe can we take the bootcamp even though we don't work for Microsoft? :rolling_on_the_floor_laughing:

Marc Price - Nationwide Building Society (Speaker)15:10:14

Highest Paid Person's Opinion now we all work from home?

2
Amanda Silver15:10:48

The Bootcamp was created in 2017, I believe. In part because we had TERRIBLE "net promoter scores" for new employees.

❤️ 2
1
Gene Kim, ITREV, Program Chair15:10:20

This is so amazing, @amandas! I’m sure others can resonate with this, or have lived it!!!

💯 2
Gene Kim, ITREV, Program Chair15:10:49

“we need our people to become the experts, and need to show that they can make decisions, that we trust them”

❤️ 2
BMK-BNZ-ValueStreamArchitect15:10:21

This statement also reminds me Netflix "No Rules Rules" Book by Reed

💯 1
Chris15:10:25

‘Act in the company s best interest’ (underpined, we trust you to determine this best interest). Reminds me as well of a talk with J Willink and Sacha Labourey from Cloudbees, roughly saying that if you want ppl to take leadership, you have to give them, even if as a leader you have 95% of the solution and guy comes with 80% let him go with it as the energy he’ll put in his own plan will exceed the difference

❤️ 1
Craig Cook - IBM15:10:21

IBM has a program we call "Jumpstart" ~2 yrs long. New developers attend it. Various stages, design thinking, development, patents, public speaking, etc.

👍 2
Nick Eggleston, IT Architect (open to offers)15:10:20

Two years for Jumpstart? How much time does the long version take?

Craig Cook - IBM15:10:32

Anyone at IBM < 2 years is considered a "New Hire".

Gene Kim, ITREV, Program Chair15:10:59

“for any given segment of market we can compete in, they now have first-hand experiences of the people we’re trying to serve” — “senior leaders provide global context; teams operate independently with as much local context as possible”. So beautiful, @amandas!

jeff.gallimore (CTIO - Excella, he/him)15:10:12

yes to this. facilitation and meeting management needs to change when not everyone is in the same physical space.

💯 2
❤️ 1
Nick Eggleston, IT Architect (open to offers)15:10:12

Need to create pauses and poll the room for contributions -- so true! as the quick talkers often dominate the conversation

💯 7
Brian W. Spolarich - Cal Poly15:10:12

OMG this is my life.

💯 2
👍 2
Craig Larsen - he/him - Solution Design Group Mpls15:10:40

I laughed … because I can relate to this myself. : )

Brian W. Spolarich - Cal Poly15:10:58

I'm doing this as a leader!

Gene Kim, ITREV, Program Chair15:10:40

One of the many things I admired at Etsy visiting @allspaw in 2013 was their huge investments in conference rooms in AV. Optimizing for remote participants. It was unreal. One of the best sound systems I had ever seen.

Andy Nortrup - Director of PM at Tanium15:10:20

So much video conference fatigue.

👍 2
👏 2
1
Frotz Faatuai (Cisco IT - he/him)15:10:30

“…you have a lot less meetings” - Something that so many companies should learn…

1
Nick Eggleston, IT Architect (open to offers)15:10:32

Much more asynchronous communication and it needs to be snackable

💥 7
❤️ 1
👏 1
Nick Eggleston, IT Architect (open to offers)15:10:32

Much more asynchronous communication and it needs to be snackable

💥 7
❤️ 1
👏 1
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:20

Yes THIS! The "snackable" / bite size content still feels like a problem for our folks.

Gene Kim, ITREV, Program Chair15:10:27

I’m marveling that @amandas is modeling the behavior of Admiral John Richardson (@jmrichardson1) and Captain @emily356 —  Leaders pause to think about the systems that we work within — rethinking the use of meetings in remote only, moving from meetings in Old World to deliberate shift to async communications.

❤️ 2
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:31

"being more intentional about bringing people together... and knowing what we want to accomplish" speaking my language!!!

👍 2
Dr. Ivan Kronkvist15:10:01

BC = Before Covid

😂 4
2
C15:10:13

Gather has been eye-opening for me, very much has the feeling of a team room and hallway conversations

💯 2
🙌 2
Ryan Taylor - Senior Geospatial Developer, GISinc15:10:14

How do you limit async conversations via messaging from becoming long running sync conversations?

C15:10:18

I think you need to be deliberate about outcomes

Nick Eggleston, IT Architect (open to offers)15:10:14

@amandas who do you look up to as leaders (companies/people) for the new model of fully asynchronous, distributed, remote-only ways of working?

⬆️ 2
💡 1
Stijn Claes - Nike15:10:36

Learning by Osmosis.. sounds like how we used to do onboarding pre covid

😅 2
3
Daniel Cahill - Engineer - Ontario Systems15:10:17

Is there a way I can help leaders want to have engineers talk to customers? I have asked every manager and director I have had but after 4.5 years, I still haven't talked to a customer.

👀 1
Gene Kim, ITREV, Program Chair15:10:20

(To explain my reaction, @amandas — when I was at Tripwire from 1997 to 2010, every customer interaction was so fraught with consequence. Sales/account rep orchestrated and controlled everything, and developers talking with one directly was… unthinkable!!! 😆

💯 4
Dr. Ivan Kronkvist15:10:51

Many companies like this

Gene Kim, ITREV, Program Chair15:10:51

“what if the developer SAYS something?!?”

11
Ben Link15:10:20

Did that ever change, Gene? If so, what catalyzed it?

👍 1
Gene Kim, ITREV, Program Chair15:10:15

(Story does end well: in 2005, I helped lead the UX initiative at Tripwire, to atone for all the terrible things we built, and made it common practice to talk directly with customers. Although it did lead to horrifying horrifying discoveries. 🙂

🍿 2
❤️ 2
Amanda Silver15:10:33

I'm not sure who I look up to per say... Always open to recommendations! I have a ton of colleagues, peers, and folks on my team who are constantly teaching me. Monty Hammontree on my team has been absolutely pivotal in terms of bringing outside ideas into the team through reading books like the Culture Code, etc.

👏 3
Nick Eggleston, IT Architect (open to offers)15:10:27

Do you blog about this or publish your consolidated learnings and references?

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:44

This is great! Always have those people who are bringing new ideas or questions in from outside our own (team/dept/company/social) echo chambers.

Amanda Silver15:10:59

I do occasionally, Monty Hammontree, Travis Lowdermilk, Jessica Rich are all prolific writers that often post on LinkedIn and Medium about our process.

🙏 1
Nick Eggleston, IT Architect (open to offers)15:10:26

@amandas ... followed you, monty, travis on twitter... but haven't been able to find jessica. is she on there?

Gene Kim, ITREV, Program Chair15:10:00

The notion of video conferencing fatigue is real: the fact that you could see how leaders in the GitHub study from Dr. @nicolefv last year was startling, showing how leaders are having to work longer, doing more coordination, due to remote work.

👍 6
😩 2
Dr. Ivan Kronkvist15:10:49

It takes discipline to end the workday and stash your laptop.

Scott Jaffa (Principal Engineer, ValidaTek)15:10:10

Seeing a tendency for people to schedule video calls that could be an email because the video call is such low friction vs. scheduling a conference room, locations, etc.

💯 2
BMK-BNZ-ValueStreamArchitect15:10:13

We need to do a video conferencing about how video conference fatigue is real

👍 1
Nick Eggleston, IT Architect (open to offers)15:10:45

In the new remote-only world, we have to be very intentional to discover and create experiences and learnings that just happen by osmosis for those who are in close proximity

2
👏 1
💯 1
C15:10:47

I've had to re-think our "incremental planning" offsite and split remote time to a 3hr, 5hr, 2hr event over 10 days

Anderson Tran15:10:59

The invasive 5 minute and 15 minute "asking a question" meetings 😞

Dave Owings15:10:12

I implemented an optional 30 minute meeting for my staff. I call it "Funday Fridays". The requirement is you have to have your camera on and we do not talk about work. I have been doing this for a year now and the staff have grown closer.

❤️ 2
C15:10:26

to make the 5 hr work, I did a 1 hr ask a question about how this is going to work 'pre-meeting'

Gene Kim, ITREV, Program Chair15:10:52

I totally can relate to what @amandas just said: there was a remote meeting for our kids’ school, and after a long work day, it took SO MUCH WORK to look engaged, and finally turned off camera, and it was such a relief! I’ve never felt anything quite like that.

💯 7
C15:10:53

the 5 hr and two hour on day 2 and 3 were a set of 75 min drills

Daniel Cahill - Engineer - Ontario Systems15:10:57

Its good for me to hear how burnout is also affecting others in meetings and its not just me. Thank you for this.

💯 10
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:37

You are in A LOT of good company @dacahill7. 😄

Nick Eggleston, IT Architect (open to offers)15:10:56

Interesting to me how Gather feels very different from being on Zoom meetings... anyone else have the same sense?

Dian Hansen15:10:49

I'm still struggling with Gather, in person I'm generally fine, but this was hard for me

C15:10:52

I do feel like Gather is better than a zoom, but I still don't care for camera time

C15:10:45

A weird Gather experience I just had tho, was someone who had their mic unmuted, just walked past me and I got a "feedback" double video from his on-camera that really confused me

C15:10:58

'audio' in other screen

Nick Eggleston, IT Architect (open to offers)15:10:18

Gather has lots of room for improvement, yet I think the gamer world and community has a lot to teach us about creating and sustaining enriching distributed, remote experiences

🎯 1
💯 5
Gene Kim, ITREV, Program Chair15:10:02

@amandas You seem to have thought deeply about how people collaborate and work, and revisited the assumptions behind them. Do you allocate time to think about things like this? Where does this category of thinking fall in your roles/responsibilities?

❤️ 4
Amanda Silver15:10:11

Software is a team sport. At their core, our products are designed for teams to collaborate on, deploy together, take accountability for together in operations, and to continually improve. So, from my vantage point, studying how digital product creation teams function is core and essential to what our products need to do. I'm lucky that my job is very meta and I get to have my own team "eat our own dogfood".

Gene Kim, ITREV, Program Chair15:10:13

Kudos to all the incredible decisions coming out of all this incredible and impressive cerebral work — it’s so admirable! 🙏 I suspect that these practices will be rapidly adopted!

👏 2
Ganga Narayanan15:10:09

I've found trying replicate all in-person meetings with an exact replacement with a virtual alternative of the same length to be like replacing a meat dish with a meat-like replacement. It's not going to be the same. 🙂 . This is the opportunity to try something different. We've switched to shorter workshops, offline work etc.., asynch coordination is so true!

❤️ 1
Dr. Ivan Kronkvist15:10:20

Not all meetings have to be camera on, or at least not all the time for sure. Otherwise it feels like having multiple one-on-ones simultaneously

Lochan Vasudev15:10:20

Realistically in lots of remote meetings 90% of the attendees are either not closely involved in the discussion and do other things in parallel compared to face to face meetings. It is one of the realistic picture as per my experience with remote meetings.

💯 1
Christopher Rueber15:10:33

I actually feel like this is one of the most ideal parts of having these remote meetings. Because it allows for more parallel work.

Christopher Rueber15:10:54

Though it does make it necessary sometimes to say “Please pay close attention to this content”

Tashfeen Mahmood15:10:36

I do not know about that, @christopher.rueber. That is the most context switching way of working and therefore not ideal IMO.

Lochan Vasudev15:10:16

Not as productive as face to face meeting. Due to these remote channels, the number of meetings have increased manifolds compared to pre covid time. Focus and productivity is impacted for those who work in complex projects because every couple of minutes there are remote meetings scheduled even for small things.

Christopher Rueber15:10:30

It really depends on how well the company adheres to busting Parkinsons Law. If you have meetings that are an hour, and always expand to that time, it’s nice when you can keep rolling things forward while listening to the important bits of the meeting. Most meetings have “high points” that I want to capture, but that I don’t necessarily need to be part of all the conversation to get there.

Christopher Rueber15:10:05

Ideally if meetings didn’t expand to fill the time allotted, then you really wouldn’t have people that feel the need to multitask. In my experience that is pretty rare.

Nick Eggleston, IT Architect (open to offers)15:10:40

Many people used to multitask on their laptops during in-person meetings as well -- except when the person who called the meeting insisted on people putting all the electronics away

1
Ganga Narayanan15:10:26

Yes to writing!

💯 2
BMK-BNZ-ValueStreamArchitect15:10:16

Connect to human on personal level

❤️ 2
Brian W. Spolarich - Cal Poly15:10:18

Does this imply that the strategy session before was focused on not only what and what needed to happen, but the how, and the "leaner" version is much more about the why and what?

Dian Hansen15:10:43

agree with the concern of "less connected to colleagues"

Stijn Claes - Nike15:10:23

Internal podcasts, love that idea!

👏 4
Gene Kim, ITREV, Program Chair15:10:54

“Podcasts by team members teach people, but also exposes the person behind the content, and their careers behind the story” <--- enables Connections. (Seems so important, given how many times this has come up in this conference… cc @jtf @abd3721)

👏 2
Christina Biangslev15:10:46

Just going to say that during this non-camera-on-conference, I've baked 35 rhubarb pies (from own garden), made mulled wine essence for all of December, prepared christmas cookie dough for the freezer, made chili con carne and potato leek soup. YAY for consuming awesome knowledge without cam on. ❤️

❤️ 5
3
🥧 1
Christina Biangslev15:10:28

I could basically host a watch party on pies at this time. The geek factor would be even higher if I'd made them with Raspberries.... 😉

BMK-BNZ-ValueStreamArchitect15:10:48

Great point about the "Human" @amandas. In most of the Enterprise the people leaders/manager they do not even know their own team members - Family, background, social/economical background, birth day or anniversary days. They expect just clock the time and "Delivery" and of course meetings

Sujay Solomon15:10:53

Influencing decisions has been difficult in the remote world. I think Amanda's point about data-oriented writing is key. Would love to learn more about that.

3
Denver Martin, Dir DevSecOps, he/him15:10:37

This is a great way of managing remote and creating a solid culture,

BMK-BNZ-ValueStreamArchitect15:10:42

Satya's book "Hit Refresh" is a great read (cc - @amandas)

💯 1
❤️ 1
Gene Kim, ITREV, Program Chair15:10:53

Satya Nadella on his role: shift resources and be the cultural ambassador

❤️ 5
Craig Cook - IBM15:10:56

Speaking of bring folks into the industry, it's great to see Women leaders presenting at conferences. Great role models.

🎉 1
💯 3
Lucas Melo (American Airlines Architect)15:10:58

"Customer obsessed, like they are oxygen or sun light!" ❤️

❤️ 1
Gene Kim, ITREV, Program Chair15:10:11

I think it’s so exciting and so beautiful that DevDiv was on the vanguard of the Microsoft transformation, and the extent that it’s spread beyond DevDiv, @amandas

👏 5
BMK-BNZ-ValueStreamArchitect15:10:04

Role of manager - be a role model, be authentic, what are the values of our teams and how we live our values every day

💯 1
💥 1
Gene Kim, ITREV, Program Chair15:10:08

“Everyone, including managers and leaders, needs to grow and mentors; so part of the role of the manager is enable that for your teams. If you’re not coaching people and helping everyone on a learning journey, then you’re missing something very important.” — @amandas

💯 4
👏 1
jeff.gallimore (CTIO - Excella, he/him)15:10:40

for sure: the same boundaries between personal and work don’t exist anymore.

🎯 3
Craig Cook - IBM15:10:28

Nice. Some senior leaders at IBM are also talking about bringing your personal life/feelings to work.

jeff.gallimore (CTIO - Excella, he/him)15:10:54

that is one positive that has come out of all this — recognize the WHOLE person. not just the “work” person.

💯 3
BMK-BNZ-ValueStreamArchitect15:10:58

Great morning session day-3 at DOES; Why does DOES matter? All these slack messages are evident;

🎉 1
Gene Kim, ITREV, Program Chair15:10:07

The extent to which so many orgs here are hiring developers is astounding: e.g., Vanguard has 500+ openings. Almost every talk has ended with “We’re hiring!”

👍 2
🙂 1
Kurt A, Blameless, SRE Architect15:10:51

We didn't put it in our talk, but we're hiring too 😄

Ganga Narayanan15:10:58

There should be a DOES job fair! 🙂

Nick Eggleston, IT Architect (open to offers)15:10:03

DevX = Developer Experience? (or did I miss someting)

Brian W. Spolarich - Cal Poly15:10:39

A friend of mine wrote about the gift of "full human presence" when being a mentor: http://www-personal.umich.edu/~bcoppola/publications/42A.%20Human.pdf

Bryan Finster - Defense Unicorns (Speaker)15:10:51

“Reduce the friction required to get the work done”

👏 7
1
C15:10:53

I think we need to rethink what the minimal set of work needs to be. Just because you can solve a problem with software, doesn't necessarily mean you should, we need to focus on the Why

Virginia Laurenzano NSA/MARFORCYBER15:10:25

Also, on the "should we do it"? Good to Great has awesome insights into focusing on what makes your org special. If it's not what makes your org special, buy it!

C15:10:36

Simon Sinek - Start With Why, Infinite Game, Leaders Eat Last - some of my absolute favorites

❤️ 1
Gene Kim, ITREV, Program Chair15:10:05

…and @amandas is connecting the dots from ^^^ to ensuring the leadership systems and other systems are in place to ensure that leads to achieving business outcomes. Awesome.

❤️ 1
BMK-BNZ-ValueStreamArchitect15:10:10

Amazing @amandas - Love your thoughts and ideas; A great reminder to all of us (rest of us) what to look for in our work environment, our mentors and human connections

👏 2
❤️ 2
Amanda Silver15:10:33

IDC reported last week that we expect a >4M shortage of developers world wide by 2025.

😲 5
🤯 1
😵 2
Ryan Taylor - Senior Geospatial Developer, GISinc15:10:47

I have worked remote only for 10 years and the things I miss the most are those impromptu breaks with other coworkers in the hall, in the lunch room, etc. 10 years ago that was especially hard but infra and tools have improved dramatically. Can that be reasonably replicated virtually now?

1
Ryan Taylor - Senior Geospatial Developer, GISinc15:10:05

Gather looks like a very interesting approach to this.

1
Marc Price - Nationwide Building Society (Speaker)15:10:41

I've found it really hard to replicate those side of desk conversations

👍 1
C15:10:30

Even in cubicles, hard to replicate. Team rooms by far my favorite. Gather has feeling of a team room.

C15:10:22

difficult tho to get the concentration time to do 'Deep Work' tho.

C16:10:06

Deep Work by Cal Newport describes the 'perfect office' where there is collaboration space but also offices with door space. I interviewed at a company that had nearly replicated this with sliding glass doors to close off cubicle sized spaces with a "conference" table and whiteboard walls/glass all around

👍 1
Nick Eggleston, IT Architect (open to offers)16:10:41

I love team rooms -- I hated the cube farm and constant source of distraction those represented

👍 1
C16:10:34

@nickeggleston would you say you are an introvert, extrovert or ambivert

C16:10:02

I self assign as ambivert - but need my 'deep work' alone time to recharge - effectively playing an extrovert at work.

Nick Eggleston, IT Architect (open to offers)16:10:48

Naturally introvert, but can play extrovert with enough focus and energy

1
Virginia Laurenzano NSA/MARFORCYBER15:10:13

the working on other things during a meeting is a rare luxury for me - we don't have any personal electronics at work and are mostly in the office

1
👀 2
Bryan Finster - Defense Unicorns (Speaker)15:10:15

Almost surprised it’s not “paper only”

Virginia Laurenzano NSA/MARFORCYBER15:10:47

I've mastered the art of taking notes in a meeting I'm hosting on a page/email and insist on having the keyboard turned over to me in nearly every meeting...and have now started challenging others to learn how to listen & type

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear15:10:53

I’m still struggling to get my head around how ya’ll work without access to the internet. Not being able to easily google something just 🤯

Bryan Finster - Defense Unicorns (Speaker)15:10:53

@vmshook I can barely type, so…

😂 1
Virginia Laurenzano NSA/MARFORCYBER15:10:58

@joe.waid - I'll add that to my list of "can I talk about this at a future event?"

❤️ 1
Virginia Laurenzano NSA/MARFORCYBER15:10:39

@bryan.finster486 - who knew learning to type on a TYPEWRITER would continue to come in so handy?

Bryan Finster - Defense Unicorns (Speaker)16:10:18

I’m 6-6. Means I have large hands. Keyboards are too small for touch typing. i never learned. 😄

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear16:10:25

typing without looking is a real super power. I’m an oddball who learned dvorak so I had no choice but to master touch typing. 😄

👏 1
Virginia Laurenzano NSA/MARFORCYBER16:10:30

excuses. I'm 5-0 and literally had to get the smallest mouse ordered. Good news: no one uses my desk at work

Virginia Laurenzano NSA/MARFORCYBER15:10:30

love hearing about others' experiences in this space

Gene Kim, ITREV, Program Chair15:10:39

Thank you thank you @amandas so much for teaching us about leadership, the role of the leader, and rethinking deeply held notions of what work should look like! Help she is looking for: dev productivity: to what degree are teams truly supported? time to first contribution! Would love stories of culture changes to enact that to happen!

❤️ 10
👏 1
Gene Kim, ITREV, Program Chair15:10:17

…and that next generation of who are creating software are diverse and representative of humanity!!!! Thank you @amandas!!!!

❤️ 3
Derek Bissinger15:10:26

Very relevant, Thank you!

Amanda Silver15:10:32

Thanks for having me, @genek!

❤️ 2
👏 5
BMK-BNZ-ValueStreamArchitect15:10:43

Thank you @genek brining the best from industry to our summit. DOES standard is now so high. In-person or virtual does not matter.

Nick Eggleston, IT Architect (open to offers)15:10:36

Great interview @amandas and @genek... gonna have to re-watch it multiple times

☝️ 2
❤️ 1
Dave Owings15:10:45

This has been my favorite sessions so far. Thank you @amandas

❤️ 4
2
Sujay Solomon15:10:55

I feel inspired and refreshed after listening to that. Much needed booster shot (no pun intended!) :)

❤️ 5
Ganga Narayanan15:10:58

One thing I can say about inclusion is that, as a hard of hearing person myself, the live-captions on my browser in Zoom sessions, the chats, slack etc. have given me super powers I did not know I had!

❤️ 7
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear15:10:46

Thank you for sharing so much great information @amandas!

Catie Martin_Design Manager_BlueCrossBlueShield of SC15:10:38

So much awesomeness shared in that talk - thank you @amandas!!!!

Nick Eggleston, IT Architect (open to offers)16:10:51

I just heard @jeff.gallimore mention the wrap of the conference and felt the pang of anxiety that things will be ending far too soon

❤️ 8
Nick Eggleston, IT Architect (open to offers)16:10:06

Please tell me these aren't at the same time! 😂

😳 1
😬 1
Slackbot16:10:07

Reminder: The action has moved to the breakouts! Join the following channels to interact with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4

Jill Mead18:10:00

Hello! @erickramlinger and I are excited to be here!

👏 2
Alyssa Levicky - Senior Manager - Hyland19:10:37

Such a fun presentation, I really enjoyed it!

❤️ 1
Jill Mead19:10:01

Thanks @alyssa.levicky We love being creative with these recordings!

David - BBY19:10:45

@paul_littlefair and @lbmkrishna - really liked the presentation! Can I ask, what underlying systems have you wired up to Tasktop Viz at this point?

BMK-BNZ-ValueStreamArchitect19:10:34

We have connected our JIRA (onprem), SNOW in-progress

David - BBY19:10:57

Ok, interesting...we're on the same kind of adoption journey (with help from @dominica herself!)...same two major tools, and are in the debating phase about other / smaller spots where work happens

👏 1
Liz Fong-Jones21:10:40

eee looking forward to you all seeing the talk!

1
1
🙌 2
1
👍 2
BMK-BNZ-ValueStreamArchitect21:10:11

Great to see you here @lizf @shelby

BMK-BNZ-ValueStreamArchitect21:10:22

Your talks, contribution impact on folks like me and other, Enterprises are - tremendous

❤️ 2
Liz Fong-Jones21:10:01

also pre-talk hi-five @shelby!

🙌 2
1
1
1
1
Shelby Spees, SRE at Equinix Metal21:10:32

I'm so stoked to be here

👋 2
Shelby Spees, SRE at Equinix Metal21:10:14

looking forward to everyones questions :]

Slackbot21:10:23

Reminder: The final plenary sessions are starting again in 5 minutes. Start making your way back to your browser. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png

🎉 1
Leah Brown - IT Revolution21:10:00

Last Chance! Grab your free digital copy of these two amazing books before copies run out. Agile Conversations: Change Your Conversations, Change Your Culture (sponsored by Split) Sooner Safer Happier: Antipatterns and Patterns for Business Agility (sponsored by LaunchDarkly) Get yours here! https://members.itrevolution.com/free-ebooks

Ann Perry - IT Revolution21:10:01

Starting the wind-down this afternoon will be the dynamic duo of @shelby and @lizf presenting, Shifting Left on Production Excellence with Observability

❤️ 2
upvotepartyparrot 2
BMK-BNZ-ValueStreamArchitect21:10:33

All pictures shows the Production on the right; But @lizf @shelby talk on "Shifting Left on Production Excellence with Observability"

Topo Pal - Programming Committee Member21:10:30

42% time spent on bad code!!!!

😭 2
Craig Cook - IBM21:10:35

<start flame war> yes, developers should be on-call for their own services. Yes, they should not be setup to fail because if it though, they need coaching (SRE) help to run their services.

Liz Fong-Jones21:10:02

don't hand people ticking time bombs and run away :)

😀 3
💣 1
1
🕐 1
Justin Abrahms (eBay)21:10:07

haha. Joined a little late and heard "Many things can break and we don't necessarily know what it is up front..." and my immediate thought was "Huh. Sounds like honeycomb". so... branding++ 😉

😂 1
BMK-BNZ-ValueStreamArchitect21:10:27

Team do not have time for Knowledge Transfer 🙂

Kurt A, Blameless, SRE Architect21:10:49

Those who do not transfer knowledge are cursed to have to rediscover it 🙂

😲 1
BMK-BNZ-ValueStreamArchitect21:10:23

IMHO: In most complex Enterprises - folks do not know how many services are running in PRODUCTION; You get different data from different catalogues LOL

Kurt A, Blameless, SRE Architect21:10:19

Can you define a service unambiguously?

BMK-BNZ-ValueStreamArchitect21:10:14

@lizf - @shelby does what you do 🙂 glass of water 🙂

🚰 1
❤️ 1
Liz Fong-Jones21:10:31

we're about to get there with how to discover what your services are 🙂

❤️ 2
BMK-BNZ-ValueStreamArchitect21:10:21

Technical decisions are business decisions 🙂 Love That

💯 2
Liz Fong-Jones21:10:28

I was wondering why my voice sounded so ragged but then I remembered that I was still recovering from a cold when we recorded this

🌡️ 4
😄 1
Liz Fong-Jones21:10:25

(I'm feeling better now!)

Liz Fong-Jones21:10:29

and it wasn't covid

👍 1
👏 1
😷 1
BMK-BNZ-ValueStreamArchitect21:10:39

I heard @damon say "Operation is the BUSINESS", I will add this to the list now by @shelby "Technical decisions are business decisions"

❤️ 4
Luke Rettig - Target21:10:36

“our customers shop our stores, i am your colleague” ~ @bryan.finster486

💯 3
jeff.gallimore (CTIO - Excella, he/him)21:10:46

operations is also the group closest to the actual customer

☝️ 1
Gene Kim, ITREV, Program Chair21:10:18

OMG. had to reboot laptop after it crashed! Thank you so much for presenting, @shelby @lizf!!!

👏 2
1
Craig Cook - IBM21:10:20

It's good it crashed now and not during your final remarks 😀

Chris Kernaghan21:10:47

@lizf - when you say event, is that a measured metric event or a tripped metric threshold?

Liz Fong-Jones21:10:08

it's the data for EACH EVENT

Liz Fong-Jones21:10:18

e.g. one request with 350 milliseconds counts as bad if the threshold is 300

Liz Fong-Jones21:10:24

but one request with 250ms counts as good

Liz Fong-Jones21:10:32

and then you sum up ALL the requests over a month or a quarter

Liz Fong-Jones21:10:53

we encourage you to think about SLOs as being per transaction as opposed to per time window

Chris Kernaghan21:10:37

OK with you - and trying to map this to an ERP software package architecture

Liz Fong-Jones21:10:41

it is much more dangerous for your service to break on Black Friday if you're in retail than at 3am on a random Sunday in March.

👏 1
Liz Fong-Jones21:10:55

not every time window is equal. thus count number of transactions succeeded vs failed

Denver Martin, Dir DevSecOps, he/him21:10:21

That thought also brings to mind that if you have scalability you may want to use that same schedule to show you are ramped up already to support that more important window.

🎉 1
BMK-BNZ-ValueStreamArchitect21:10:49

These 9's are crazy numbers - How some architecture document state infinite nines, but no tools to measure that and impact of that;

9️⃣ 1
♾️ 1
Liz Fong-Jones21:10:11

and then you can't do anything because you're violated your SLO have fun

Gene Kim, ITREV, Program Chair21:10:44

I love this observation of mobile devices over cellular networks shaping SLOs, and how 99% is often enough, because mobile reliability is so poor. The famous 2014 Ben Treynor-Sloss talk referenced this, too!

👏 2
1
1
Kurt A, Blameless, SRE Architect21:10:09

There are also some stats on the reliability targets for pacemakers that put "9"s approaches in sobering light.

Kurt A, Blameless, SRE Architect21:10:23

Something like pacemakers are in the low 90s

BMK-BNZ-ValueStreamArchitect21:10:48

Question @lizf - have you seen architects care about this 9's now in the observability world or just stop with Visio diagrams?

Gene Kim, ITREV, Program Chair21:10:21

I love @lizf notion of monitoring SLO budgets — “RELEASE THE CHOAS MONKEYS! UNLEASH MORE DANGER!!!” 😆

1
Liz Fong-Jones21:10:21

they have to, or else you build a 4 9s service on a 2 9s foundational service

Liz Fong-Jones21:10:28

and then it's impossible to achieve the targets

Craig Cook - IBM21:10:29

I was shocked when I first read that google intentionally injects failures to get back to the SLO if your are running over it. It makes sense once you pause and think about it though.

👍 1
Topo Pal - Programming Committee Member21:10:41

Not sure if I understand the calculation - 1 cell phone vs 1 mil cell phones

Liz Fong-Jones21:10:41

making your service down 1 less nanosecond per year doesn't matter if your users in practice experience minutes or hours of downtime per year due to being out of cell signal

Gene Kim, ITREV, Program Chair21:10:43

I think it’s the probability of any transaction failing/erroring – independent of # of events.

Gene Kim, ITREV, Program Chair21:10:51

at sufficient scale, service goes to 0%. 🙂

Topo Pal - Programming Committee Member21:10:30

interesting discussion for a later time 🙂

Liz Fong-Jones21:10:08

and also your service doesn't have to fail all or nothing

Liz Fong-Jones21:10:15

you could be serving 1% errors to 1% of users

Liz Fong-Jones21:10:17

and probably be fine

Liz Fong-Jones21:10:26

as long as you didn't keep serving errors to the same user repeatedly

Liz Fong-Jones21:10:42

if you are not thinking reliability as a critical functional requirement, then you are doing a poor job as an architect

BMK-BNZ-ValueStreamArchitect21:10:00

Without proper engineering what remains is only "CHAOS" :rolling_on_the_floor_laughing:

👏 1
Andrew Davis - Copado - DevOps and Testing for Low Code21:10:07

@lizf observability tools monitoring the actual state of our systems:

Shelby Spees, SRE at Equinix Metal21:10:08

release the chaos monkeys = learn about the dark corners of your system

❤️ 2
BMK-BNZ-ValueStreamArchitect21:10:37

hmm, some just release monkey, forget about learning, what to do?

Shelby Spees, SRE at Equinix Metal21:10:18

I find it helpful to emphasize that it's an experiment and to reference the scientific method. what do you expect to happen? how are you going to measure it? (it's why observability is criticial)

❤️ 1
jeff.gallimore (CTIO - Excella, he/him)21:10:13

i love the “investment” perspective. you can over- and under-invest in this with real business consequences.

👏 3
Topo Pal - Programming Committee Member21:10:33

1 cell phone may be 99% but all together it makes avail req very high?

Shelby Spees, SRE at Equinix Metal21:10:56

also, going way over your SLO target means users get used to that ultra-high reliability and start depending on your service for things it wasn't meant for

Liz Fong-Jones21:10:00

users are used to things failing on their cellphone

Liz Fong-Jones21:10:07

they're more likely to blame the wifi than your service

jeff.gallimore (CTIO - Excella, he/him)21:10:50

a new standard customer support response: “it was probably your wifi.” :rolling_on_the_floor_laughing:

Liz Fong-Jones21:10:12

they do it already 😛

Liz Fong-Jones21:10:18

"have you tried turning it off then on again"

2
Ryan Taylor - Senior Geospatial Developer, GISinc21:10:08

I 💗 this > alert on error budget burndown rate, not every potential cause of alert

1
Liz Fong-Jones21:10:10

they'll just click refresh

Topo Pal - Programming Committee Member21:10:11

Prob of all cell phone not becoming available is very low and hence back end service avail must be very high

Liz Fong-Jones21:10:34

(now if you fail after refresh, then you have a problem)

Shelby Spees, SRE at Equinix Metal21:10:44

alerting on burndown rate is my #1 favorite thing about SLOs

❤️ 4
Gene Kim, ITREV, Program Chair21:10:28

(or worse, if the refresh is what causes the service to fall over. 😆

Liz Fong-Jones21:10:58

fun story once upon a time a change I worked on brought the entire South Korean mobile network down

😲 2
🤯 2
😳 1
ssh 1
Liz Fong-Jones21:10:21

because I disconnected all the clients with hanging GETs to http://google.com/android/services at once, and they all reconnected at once.

🤯 2
Liz Fong-Jones21:10:02

a hanging get takes no attention from a cell tower. a FIN and then new TCP handshake takes a ton of resources.

Craig Cook - IBM21:10:04

Did you get rewarded for it?

Craig Cook - IBM21:10:28

or were you punished?

Gene Kim, ITREV, Program Chair21:10:52

Mind literally blown. That’s one of the wildest things I’ve ever heard. Thank you!

Nick Eggleston, IT Architect (open to offers)21:10:30

@lizf how long did it take to get back to recover?

Liz Fong-Jones21:10:33

I think a few hours

Liz Fong-Jones21:10:38

(and then we fixed it so this could never happen again)

Liz Fong-Jones21:10:44

we stopped sending FIN on server shutdown

Liz Fong-Jones21:10:00

and just let the client send a request to hanging GET at a random time, get a RST, and retry

Nick Eggleston, IT Architect (open to offers)21:10:06

it's like inrush current when servers are turned on. So even if the rack is running fine at steady state, right after a power outage they all try to boot and poof blown breaker - and we are dark

☝️ 1
Nick Eggleston, IT Architect (open to offers)21:10:58

Very cool... or persist the TCP state so other cluster nodes can take over the already active connection and operational state.

BMK-BNZ-ValueStreamArchitect21:10:41

I will not commit code without unit test and also not commit without instrumentation - by @lizf

💯 2
Topo Pal - Programming Committee Member21:10:58

Yea, my wifi is not reliable at all 😃

1
Shelby Spees, SRE at Equinix Metal21:10:07

instrumentation coverage is as important as test coverage

💯 1
Chris Kernaghan21:10:10

I really want to introduce a set of artifacts to be delivered before something goes like - observability, documentation, change documentation, decommissioning

Chris Kernaghan21:10:39

The idea of building instrumentation along with the code is a cool idea

Shelby Spees, SRE at Equinix Metal21:10:43

Honeycomb's PR template includes a question: "How are you going to observe this change as it goes out?"

👏 2
Chris Kernaghan21:10:46

I might not be nuts

👍 1
Shelby Spees, SRE at Equinix Metal21:10:57

doesn't have to be a lot of overhead. just like, look where you're going

BMK-BNZ-ValueStreamArchitect21:10:08

What bothers me a lot in this space is - the SLOs remain just on the paper, some do not know how to measure that; What is involved in measuring that?

Liz Fong-Jones21:10:28

something something I maaay have built a solution for that having been dissatisfied with what was in the field

Liz Fong-Jones21:10:32

but you can also check out nobl9 or blameless

thankyou 1
Chris Kernaghan21:10:34

People default back to SLAs, because that's what middle managers are actuall ymeasured upon

Liz Fong-Jones21:10:37

(psst http://honeycomb.io/slo now that I have tooted the horn of a couple of other players in the space as well)

👏 1
Liz Fong-Jones21:10:44

but also OpenSLO is a thing, you'll see it in a sec

❤️ 2
Shelby Spees, SRE at Equinix Metal21:10:30

also, the upcoming O'Reilly book Observability Engineering discusses event-based SLOs in detail https://www.oreilly.com/library/view/observability-engineering/9781492076438/

🔖 1
👏 1
👀 1
Liz Fong-Jones21:10:26

free copy of the book: https://info.honeycomb.io/observability-engineering-oreilly-book-preview (but you can also get it through Safari if your org has a membership)

Daniel Cahill - Engineer - Ontario Systems21:10:11

This practice also seems like it will help improve confidence in stakeholders if I start my project with instrumentation.

4
BMK-BNZ-ValueStreamArchitect21:10:19

What happened to all the RTO and RPOs now? @lizf

Liz Fong-Jones21:10:01

RTO/RPO is for catastrophic failures

Liz Fong-Jones21:10:07

but you should also be measuring and fixing smaller failures too

Liz Fong-Jones21:10:18

both big and small failures count against your error budget

Liz Fong-Jones21:10:30

it doesn't matter if your RTO/RPO is 20 min if you have 5 outages in a week 😉

✔️ 1
Liz Fong-Jones21:10:41

so instead track your SLO for the past month or quarter to maximize user happiness

BMK-BNZ-ValueStreamArchitect21:10:01

Got you, so those measures are still required

Denver Martin, Dir DevSecOps, he/him21:10:20

I think RTO/RPO is more for DR and that SLO would me more inline for this item..

👏 2
Liz Fong-Jones21:10:18

right. RTO/RPO/TTR are useful metrics for bounding how many outages you can have

Liz Fong-Jones21:10:34

but also if you have very few outages because you've made all the small stuff not be outages

Liz Fong-Jones21:10:41

then of course your RTO/RPO/TTR will be high

Liz Fong-Jones21:10:43

but that's okay!

Liz Fong-Jones21:10:51

because you've squished the fast things, only the slow things are left

Denver Martin, Dir DevSecOps, he/him21:10:40

would you look at the Constraint on telemetry and that should be first?

Liz Fong-Jones21:10:58

if you don't have high quality telemetry, well, garbage in, garbage out

Denver Martin, Dir DevSecOps, he/him21:10:36

I see so you still need to focus on what is being measured and how before you can really look at telemetry and find the right constraints.

Liz Fong-Jones21:10:13

SLIs should be set on meaningful things

Liz Fong-Jones21:10:19

and telemetry needs to be sent about real user traffic

Liz Fong-Jones21:10:28

if in doubt measure from the ingress point inwards.

BMK-BNZ-ValueStreamArchitect21:10:27

Some who lives and breaths DevOps mindset - LOVE IT

Gene Kim, ITREV, Program Chair21:10:24

(thinking: “why can’t I take down entire cell phone network of a country? why does @lizf get to do all the fun things?” Sorry. Still recovering from that story!)

Liz Fong-Jones21:10:03

(oops we were scheduled AFTER Christina)

👍 3
1
Liz Fong-Jones21:10:06

(but her talk was great)

💯 2
👍 2
Christina Yakomin21:10:01

Likewise, Liz!! Your talks simply never miss.

❤️ 1
Shelby Spees, SRE at Equinix Metal21:10:12

you can watch it again in the library!

☝️ 3
BMK-BNZ-ValueStreamArchitect21:10:20

Importantly do not rename OPS or Sys Admins as SRE - seems like the SRE is all together different skill set; OLD WAYS WON'T OPEN NEW DOORS

💯 2
Liz Fong-Jones22:10:55

But also do retain and train, don't try to fire and replace with outside people. The skills are teachable

Gene Kim, ITREV, Program Chair21:10:32

H/t to @christina_yakomin and @rdaitzman at Vanguard.

Topo Pal - Programming Committee Member21:10:50

I need to rewind this talk - full of VERY useful information!

💯 4
BMK-BNZ-ValueStreamArchitect21:10:25

We cannot clone Brent 🙂 and Christina

Dipesh Bhatia21:10:25

but if i am testing code in prod, then the dev has to be on call. what am i missing ?

😂 1
Dipesh Bhatia21:10:25

but if i am testing code in prod, then the dev has to be on call. what am i missing ?

😂 1
Kurt A, Blameless, SRE Architect21:10:04

nothing - that's exactly the point

Liz Fong-Jones21:10:04

that's exactly the point

Dipesh Bhatia21:10:23

Duh…I need to view this talk again . Thanks

Liz Fong-Jones21:10:37

our talk is about how to make it so your devs don't hate being oncall

😄 2
Liz Fong-Jones21:10:45

by making it simpler for them to understand what they do have to do

👍 1
Gene Kim, ITREV, Program Chair21:10:26

@shelby explores where/how are we going to get all the SRE/Ops skills needed for what tens of developers will be writing?

Chris Kernaghan21:10:33

@topo.pal I need to rewind the last 3 days

👍 1
❤️ 1
Chris Kernaghan21:10:20

CNCF slack - I need to look at this

1
Dian Hansen21:10:26

@shelby great point for new generation!

💯 1
Topo Pal - Programming Committee Member21:10:44

Thank you so much @lizf and @shelby

☝️ 2
1
👏 2
Chris Kernaghan21:10:54

Dammit another great talk - great job @lizf @shelby

1
❤️ 4
Gene Kim, ITREV, Program Chair21:10:02

Thank you @shelby and @lizf!!

🙌 4
❤️ 1
Ann Perry - IT Revolution21:10:11

👏:skin-tone-3: And now, to bring us to a close is none other than @allspaw, presenting: Learning Effectively From Incidents: The Messy Details 👏:skin-tone-3:

🎉 4
Gene Kim, ITREV, Program Chair21:10:24

And last speaker for this conference: @allspaw!

Dave Mangot - DevOps transformation professional21:10:28

Allspaw? Who? Never heard of him.

😁 1
BMK-BNZ-ValueStreamArchitect21:10:30

Brilliant as usual - both @lizf and @shelby - I am delighted that Liz you spoke at my "Continuous Learning Series" - BNZ Thank you

🎉 1
Gene Kim, ITREV, Program Chair21:10:50

I loved your close and call to action, @shelby — and thanks for proposing this talk!!!

❤️ 4
Shelby Spees, SRE at Equinix Metal21:10:29

thanks for making it happen!

Liz Fong-Jones21:10:55

thanks for having us, and for a keynote even!

Denver Martin, Dir DevSecOps, he/him21:10:02

I talk about 10 Deploys a day often... helping my teams understand that is possible...

Shelby Spees, SRE at Equinix Metal21:10:07

@allspaw giving us the BLUF (bottom line up front) 💯

😆 6
👋 1
Gene Kim, ITREV, Program Chair21:10:55

“learning requires remembering” (oh)

Dave Mangot - DevOps transformation professional21:10:31

Don’t give John any ideas @nickeggleston

Gene Kim, ITREV, Program Chair21:10:57

@allspaw I loved this talk from this morning from @aalvare2 from Stanford Medical and @christina @kboth_does describing the realities and consequences of this. Was such a beautiful talk! https://videos.itrevolution.com/watch/625427503/

👀 1
❤️ 4
1
jeff.gallimore (CTIO - Excella, he/him)21:10:28

@allspaw let me guess… you fired that engineer, right? just a bad apple, right?

😂 6
🏹 1
🔥 2
🍎 1
Chris Kernaghan21:10:51

It might have been John

1
Ferrix Hovi - Head of DevOps - Siili21:10:24

@jeff.gallimore Like you should and what is right 😄

💯 2
1
Scott Prugh (CTO - CSG) DOES Prog Committee21:10:40

Yes!! @allspaw "Rich stories are valuable."

👍 2
Jeffrey Fredrick, Author-Agile Conversations21:10:41

Stories are unit of idea transmission.

❤️ 4
🤯 1
John Allspaw21:10:17

So: what do you remember about the initial story I told?

😮 2
Gene Kim, ITREV, Program Chair21:10:39

I love this presentation — @allspaw describes the performance improvement plan he put an engineer on after a disruption in a revenue generating service.

❤️ 1
Gene Kim, ITREV, Program Chair21:10:48

…and a template that we can all use… 🙂

Denver Martin, Dir DevSecOps, he/him21:10:02

Wonder if there is a difference in remembering the exact items or remembering where to find the details is the same - same type of learning?

John Allspaw21:10:04

I see what you did there

😀 4
🎉 1
Ferrix Hovi - Head of DevOps - Siili21:10:55

Quoting me from earlier today: "Ops stories are cooler. Like the time I held all company data on my knee because someone had not installed the rack rails..."

👍 1
Gene Kim, ITREV, Program Chair21:10:03

I love this observation from @allspaw on what is required for an event to be remembered, which is nearly a perquisite for the right learning to occur.

4
Topo Pal - Programming Committee Member21:10:05

I am seeing myself telling the stories when I mistakenly......

👍 1
🙂 2
John Allspaw21:10:27

There aren’t “ops” stories or “dev” stories…there are just stories.

❤️ 2
👍 2
1
Ferrix Hovi - Head of DevOps - Siili21:10:50

I agree. I think my sentiment was that stories about something exposed to customers failing makes a more dramatic story.

John Allspaw22:10:52

They can, but I think if you were to ask hands-on folks about non-customer-impacting incidents, they’ll have stories that are worth listening to!

Ferrix Hovi - Head of DevOps - Siili00:10:39

I didn't say impacting customer but systems with customer facing and live. Near misses are good drama. Whatever happens in pre-production environment doesn't have that tight rope even if the stakes could be high in other ways.

Ferrix Hovi - Head of DevOps - Siili00:10:28

Also, drama does not equal educational. It's just naturally tends to circulate better.

Denver Martin, Dir DevSecOps, he/him21:10:47

Different people will also remember things differently even on the same event...

👍 1
Daniel Cahill - Engineer - Ontario Systems21:10:03

If there is someone wanting to do incident analysis or fill out a ticket before taking the steps to actually resolve the incident, how do you help them want to act more urgently? They are trying to make sure we go slow enough to be safe, but I don't know how to ever build their trust enough to get decision rights. (I just got called during these keynotes because of this problem.)

John Allspaw21:10:11

@denver.martin YES and that is where the gold is.

🙂 1
Brady Robinson - Director, SRE - Cypress.io21:10:12

YES! Adjacent to this, I love the way this was articulated in your 2017 presentation “How Your Systems Keep Running Day After Day”. Different people have different assumptions and perceptions about how the system works. Capturing those as part of the post-mortem is always fascinating. https://youtu.be/xA5U85LSk0M

John Allspaw22:10:37

Oh thanks, @bradysbytes - appreciate that you noticed that bit!

Brady Robinson - Director, SRE - Cypress.io22:10:36

Big fan of your work @allspaw. Appreciate you taking the time to share your work with the community!

thankyou 1
Gene Kim, ITREV, Program Chair21:10:33

(actually, I would agree that Ops stories tend to be wilder and more dramatic than Dev stories, which are mostly about tabs vs spaces. 😆

👆 3
1
Bryan Finster - Defense Unicorns (Speaker)21:10:37

I’ve a few impactful (positive) dev stories. My ops stories are hair raising.

Ferrix Hovi - Head of DevOps - Siili21:10:07

Well... there is the story about writing the bootloader for an embedded platform where we had the wrong datasheet for the NAND chip...

😆 1
Bryan Finster - Defense Unicorns (Speaker)21:10:38

Dropping the inventory table at a Publix DC

Ferrix Hovi - Head of DevOps - Siili21:10:11

... cutting the bootloader code in half... twice

Nick Eggleston, IT Architect (open to offers)21:10:31

Unless the dev story (about something like... not validating input?) results in the exfiltration of huge amounts of sensitive customer information...

Bryan Finster - Defense Unicorns (Speaker)21:10:34

The time I deleted the application for calculating tax in Argentina and it wasn’t discovered for a month.

Kurt A, Blameless, SRE Architect21:10:46

Just don't mix up tabs and spaces in the config 😄

Ferrix Hovi - Head of DevOps - Siili21:10:08

If it's YAML, it will crash with tabs

Chris Kernaghan21:10:46

Productive 11th grader

😂 2
Patrick S. Kelso21:10:51

Just ask a cop. Eyewitnesses are the worst.

Scott Prugh (CTO - CSG) DOES Prog Committee21:10:05

The official writeup has to be in 4th order normal ITSM form. @allspaw

😂 1
❤️ 1
Dave Mangot - DevOps transformation professional21:10:29

Wait, is this because Allspaw looked like Broderick when they were both young?

😆 3
Shelby Spees, SRE at Equinix Metal21:10:50

yeah people would always confuse them and he never recovered

😂 1
Geri O’Toole21:10:38

This reference. lol

Virginia Laurenzano NSA/MARFORCYBER21:10:05

@allspaw you are a hero of mine and the inspiration behind the postmortem writeups I've published at work

❤️ 4
John Allspaw21:10:07

Oh gosh! Thank you!

Virginia Laurenzano NSA/MARFORCYBER21:10:25

at my end of the keyboard - OMG Allspaw recognized my comment! Seriously, though. I'm still recommending folks watch your talk on incidents at DOES from 2018...can't find the link - had the diagrams about what we know (wee bit) and what we don't (massive)

😂 1
Brady Robinson - Director, SRE - Cypress.io22:10:46

@vmshook - This one is fantastic too if you haven’t seen it previously -> https://youtu.be/xA5U85LSk0M. (Great talk Tues btw!)

Virginia Laurenzano NSA/MARFORCYBER22:10:36

that's the talk! I LOVE IT ALL THE DAYS.

💯 1
Shelby Spees, SRE at Equinix Metal21:10:07

there's so much value in good storytelling. narrative

❤️ 1
Kurt A, Blameless, SRE Architect21:10:09

anemic:

a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network

😍 1
🥶 2
Justin Abrahms (eBay)21:10:14

https://github.com/JDHarris007/coe - This is the postmortem format I often point to. Really excited to hear where this goes. 🙂

👍 1
Gene Kim, ITREV, Program Chair21:10:42

“I can’t get a shell” 😱 😱 😱 😱

Gene Kim, ITREV, Program Chair21:10:51

“no ping responses” 😱 😱 😱

Gene Kim, ITREV, Program Chair21:10:07

“we have to manually power cycle” 😱 😱 😱 😱

Topo Pal - Programming Committee Member21:10:13

this is not story... this is epic

😉 3
Gene Kim, ITREV, Program Chair21:10:24

(today was a bad day to stop smoking…)

😀 1
Chris Kernaghan21:10:47

It was a bad day to stop sniffing glue

Chris Kernaghan21:10:12

Totally - awesome movie

Chris Kernaghan21:10:25

That is just my nightmare in Azure

Justin Abrahms (eBay)21:10:43

I'm excited to hear that this was an innocuous change (typo fix or similar)

Gene Kim, ITREV, Program Chair21:10:23

“we started a reboot fest. dividing up machines to reboot”

2
😆 1
Chris Glover21:10:23

The PTSD is kicking in here...

❤️ 1
John Allspaw21:10:00

(this is a different story of the same incident I mentioned at the beginning, BTW)

Denver Martin, Dir DevSecOps, he/him21:10:04

Sad part is at most companies this would be a Resume generating event - instead of using it to learn from and get an ROI on this event...

Gene Kim, ITREV, Program Chair21:10:11

“I’ve got a bad feeling about this…”

Scott Prugh (CTO - CSG) DOES Prog Committee21:10:26

IE is not Blameless!

🙂 3
💥 2
🧨 2
Justin Abrahms (eBay)21:10:56

ooh. fun. recursive 404s. 😄

🔥 3
Gene Kim, ITREV, Program Chair22:10:08

…what happens when 404 page generation fail?

♾️ 4
Dave Mangot - DevOps transformation professional22:10:16

This is why I cringe when clients tell me they have “less testing” for “minor changes”. You don’t know what a minor change is until AFTER you deploy!

💯 1
Denver Martin, Dir DevSecOps, he/him22:10:33

look at all of those devops loops.... nice

2
Jeffrey Fredrick, Author-Agile Conversations22:10:49

🍩 🍩 🍩 🍩 < don’t be a donut!

🥐 3
Daniel Cahill - Engineer - Ontario Systems22:10:18

John is in the Halloween spirit with these horror stories.

😄 4
Gene Kim, ITREV, Program Chair22:10:33

“css issue forced entire fleet of web servers required a power cycle”

😂 2
Gene Kim, ITREV, Program Chair22:10:47

The birth of the Three Armed Sweater Award!!!!

Pedro Jordan22:10:48

a css change cause all that... omg

Shelby Spees, SRE at Equinix Metal22:10:57

never underestimate what's possible with css

☝️ 3
😂 2
😅 1
1
Scott Prugh (CTO - CSG) DOES Prog Committee22:10:13

never underestimate what's impossible with css too

Gene Kim, ITREV, Program Chair22:10:04

This might be most profound statement ever, @shelby

😅 1
Kurt A, Blameless, SRE Architect22:10:32

you keep using that word...

1
Chris Kernaghan22:10:16

I wonder how FAcebook will handle their messy details this week

🧹 1
1
🎻 1
Ferrix Hovi - Head of DevOps - Siili22:10:00

Solution: maximum recurse depth of 404 pages! 😄

Virginia Laurenzano NSA/MARFORCYBER22:10:03

loving how you say richest understanding, not most complete. b/c there is always more

👏 4
1
James T Snell22:10:34

I’m getting so much from this…

Dave Mangot - DevOps transformation professional22:10:30

There are a lot of PDFs that John has for you to read. 🙂

John Allspaw22:10:53

@vmshook YES! There never a comprehensive “done” analysis.

❤️ 1
Shelby Spees, SRE at Equinix Metal22:10:57

"One True Story:tm:" is its own fallacy

Shelby Spees, SRE at Equinix Metal22:10:54

this is the value of studying the humanities and social sciences

Justin Abrahms (eBay)22:10:10

I picked up a habit from Amazon to combat hindsight bias, which is to type out all of my thoughts in chat to make reconstruction way easier.

jeff.gallimore (CTIO - Excella, he/him)22:10:03

externalizing your thinking. 🙌

👍 1
1
Shelby Spees, SRE at Equinix Metal22:10:40

also super valuable for helping your teammates learn from you

👍 1
Justin Abrahms (eBay)22:10:33

I saw $x at $y which makes me think it's $z. Going to look at that next.

👍 4
Chris Kernaghan22:10:11

I do that when I am looking at a new research thingy

James T Snell22:10:45

ansible -m shell -a "touch /var/html/ie-whatever.css" EtsyProdServers

1
Bryce Miller22:10:25

it was already too late tho

James T Snell22:10:37

Well, there could have been a call to stop apache before the touch, then a call to start it again after.. It’s not really a fix, but it might have reduced the down time from ~70mins to maybe 15mins or something and then a real fix could roll out. It’s a hatchet toss

John Allspaw22:10:59

Was “1 hour and 10 minutes” the part of the story you’ll remember? The part of the story you want people to hear?

1
👏 3
1
Gene Kim, ITREV, Program Chair22:10:01

“I am pushing my CSS change, because the worst that could happen is that the text won’t be centered anymore” cc @justin124 😆

10
Topo Pal - Programming Committee Member22:10:02

def qualifies for a standard low impact chg order

Ferrix Hovi - Head of DevOps - Siili22:10:26

Ha. Earlier today I argued that you don't want to issue a security audit for a change of moving something 3 pixels north through means of CSS. 😄

😀 1
Jeffrey Fredrick, Author-Agile Conversations22:10:43

but what Protocol could we use to read our HyperText?

1
Geri O’Toole22:10:16

Support the readers.

John Allspaw22:10:30

@gerijotoole YES YES!

Patrick S. Kelso22:10:32

One place I worked had a field for the change request "What is the worst possible outcome of this change" I always wrote "The heat death of the universe" it took about a dozen changes before someone pulled me up on it.

4
Dave Mangot - DevOps transformation professional22:10:23

Tracks well with The Big Bang being the root cause. 😉

😂 2
Patrick S. Kelso22:10:14

You try writing a valid test to see if you're going to end life as we know it.

Shelby Spees, SRE at Equinix Metal22:10:36

we're starting to see these changes in my org

1
👍 2
Patrick S. Kelso22:10:29

I get the point of the question, but if no one really considers the impact. I certainly wouldn't have believed a small CSS change could take down hundreds/thousands of servers until I heard this story.

Craig Cook - IBM22:10:31

I spent 20+ hours on my last incident review. 13 Interviews with a total of 21 individuals. I got most of these things in it 😀. Certainly could do a lot more to craft it into a story though.

Shelby Spees, SRE at Equinix Metal22:10:03

better writeups and review meetings have also encouraged more people to speak up during an incident when they have a hunch

Daniel Cahill - Engineer - Ontario Systems22:10:30

I'm excited to see if the new VOID (Verica Open Incident Database) will help make it easier to learn from incidents across the community and make it more readable.

❤️ 1
John Allspaw22:10:59

I certainly hope it generates discussion!

John Allspaw22:10:23

We’ll likely learn more about what orgs are willing to publish about their incidents.

Shelby Spees, SRE at Equinix Metal22:10:33

psychological safety comes from everywhere

Gene Kim, ITREV, Program Chair22:10:35

I love that the Etsy Three Arm Sweater award was in the DevOps Handbook: https://codeascraft.com/2012/05/22/blameless-postmortems/

Dave Mangot - DevOps transformation professional22:10:38

A year long how it started how it’s going! 🤯

Ferrix Hovi - Head of DevOps - Siili22:10:44

A certain company has a "Major Incident Meeting" process... they want zero MIMs this year. Suggested solution: don't invite the meetings. 😄

🙈 2
2
Justin Abrahms (eBay)22:10:26

major incident meeting

😲 1
thankyou 1
Ferrix Hovi - Head of DevOps - Siili22:10:49

I think the name of the process is ... unfortunate

Kurt A, Blameless, SRE Architect22:10:03

sounds like we need a bof-messy-details channel 😄

☝️ 2
1
✔️ 2
👆 1
👍 1
‼️ 1
James T Snell22:10:06

Awesome talk, thank you John!

3
Joshua Barton22:10:20

Awesome talk @allspaw! Thank you!

Daniel Cahill - Engineer - Ontario Systems22:10:35

Time for everyone to get on the same zoom like DOES Europe?

Amri Abuseman - Flatiron Health, Director of Quality Engineering22:10:36

👏👏 thank for your enlightening us @allspaw -- I'm on a mission to get richer stories on post-mortem reports

👍 1
1
John Allspaw22:10:51

Thanks so much for giving your time and attention for my talk!

👏 10
1
1
Daniel Cahill - Engineer - Ontario Systems22:10:07

Thanks for speaking! I don't think its an exageration to say that I think of your talk every incident I respond to. It's really helped me practice stepping back and looking for more than one root cause. Even this morning, there was an incident I avoided the down call for and I was able to come up with 4 contributing factors that would've helped us avoid the outage.

John Allspaw22:10:44

Excellent! A great exercise might be to ask others who responded to see what they come up with, without mentioning yours, and see if and how they overlap. 🙂

Sonal Bais22:10:56

Great talk John Allspaw

Jennifer Collings22:10:15

This was great! It was my first DOES - thank you!!

Ganga Narayanan22:10:16

Thanks @allspaw ! Very inspiring!

Martin Chesbrough - Everest Engineering22:10:18

Thanks for the talk @allspaw - sorry no great stories except for my working culture is one of NOT sharing but providing the bare minimum of info

Ben Grinnell - North Highland and DOES PC22:10:22

I'll remember that talk @allspaw

👍 1
Shelby Spees, SRE at Equinix Metal22:10:24

my first DOES so obviously it was the best one ever

3
Patrick S. Kelso22:10:27

Yes, and the best virtual conference I've ever attended.

Nick Eggleston, IT Architect (open to offers)22:10:32

Loved the talk, @allspaw! I was reading a defense of detailed case studies as an important learning tool (vs dismissing it as an "anecdote") and was going to link here. but of course now that I am looking I can't find it.

Dian Hansen22:10:38

first I've been able to attend but mind-blowing!

❤️ 2
Martin Chesbrough - Everest Engineering22:10:39

My first DOES so yes it was the best

❤️ 2
Joshua Barton22:10:40

This is the first that I've been to, but I really enjoyed it! So much value! Thank you!

❤️ 1
Juan Fernandez22:10:48

Yes, Awesome!!!👏

Jude Wellington22:10:55

👏:skin-tone-4:👏:skin-tone-4:👏:skin-tone-4:👏:skin-tone-4:👏:skin-tone-4:

Ryan Taylor - Senior Geospatial Developer, GISinc22:10:03

My first DOES and I absolutely want to attend next year. Everyone was awesome.

❤️ 4
Scott Prugh (CTO - CSG) DOES Prog Committee22:10:11

Note that Facebook outage this week brought back PTSD due to the "strange loop" similarities with our CSG outage that is in the DevOps Handbook v2 @allspaw @genek

😂 5
😱 1
Nick Eggleston, IT Architect (open to offers)22:10:21

Keep an eye on slack in the coming weeks and months so we can keep the conversations going!! Don't disengage!

Gene Kim, ITREV, Program Chair22:10:25

Thank you ITREV team!!! @mvk842 @mollyc @annp @kearav @alex @annan @leahb!!!

🎉 3
👏 10
🙏 6
2
❤️ 4
Denver Martin, Dir DevSecOps, he/him22:10:27

It is up to Canada Standards a 11 star service

James T Snell22:10:31

ovation

❤️ 7
Frotz Faatuai (Cisco IT - he/him)22:10:01

Thank you @mvk842 for sharing @genek with us via the conference!

3
❤️ 3
🙏 2
😂 2
2
‼️ 2
Drew Boyer22:10:09

My first DOES - loved the talks and the community that gathered here.

❤️ 4
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere22:10:11

Gather might be my favorite newest edition to this virtual format. Overall really great conference yet again (4th year attendee). @genek @mvk842

👍 3
❤️ 2
Geri O’Toole22:10:13

Does it have to be over? Will be following and looking for many of you after today. Thanks to all. I just love this community

❤️ 1
Margueritte Kim (CEO, IT Revolution, she/her)22:10:19

Thank you @genek :unicorn_face:❤️

❤️ 11
5
‼️ 1
Nick Eggleston, IT Architect (open to offers)22:10:46

Yes, Yes, Yes keep the party going 🎉

Rosalind22:10:49

Thank you everyone

Jude Wellington22:10:49

@mollyc and Team...YOU ROCK!!! Can't wait to see everyone in person...soon!❤️

💯 3
💖 2
💓 1
Javier Magaña - Walmart22:10:11

@genek Do you have a high quality picture of the books behind you? Or maybe a list somewhere?

Dian Hansen22:10:25

@allspaw I have a failure story where the documented post mortem never reflected the true activities/issues, only found the details from the individual stories people told 1-1

John Allspaw22:10:00

@dian.hansen this is incredibly common! I can count on one hand the organizations where the write-ups and verbal stories have a lot in common.

👍 1
Nick Eggleston, IT Architect (open to offers)22:10:41

Don't abandon virtual when in-person starts again!

2
👍 5
☝️ 1
Nick Eggleston, IT Architect (open to offers)22:10:11

I've heard the same from many people...

Nick Eggleston, IT Architect (open to offers)22:10:06

Experimenting with hybrid (physical + virtual at the same time) vs physical then virtual (or vise-versa) would be super interesting for A/B testing

Scott Jaffa (Principal Engineer, ValidaTek)22:10:09

Yes, I’m LOVING the virtual format. As an introvert it got really tough as DOES got bigger, and the virtual format made it much easier to control interactions and still get to talk to people / meet a few new people.

2
Scott Jaffa (Principal Engineer, ValidaTek)22:10:14

Virtual and then physical would be pretty cool…meet/chat people online and then get a chance to talk in person.

Margueritte Kim (CEO, IT Revolution, she/her)23:10:22

Here’s what our plan is right now, to always have one virtual conference and at least one in person conference (for sure DOES US, possibly another continent). Because we built our own conference platform (+/- of being founded by software developer, we are very fussy about how things work), it would be a shame to just let it go the way of the Dodo. 😆 We also love that a virtual format makes DOES content and community so much more accessible both to attendees and to us as organizers - we’re able to invite speakers who would not normally have time to travel to a venue and deliver a talk at our conference. All of this is to say that DOES - Virtual isn’t going anywhere!

👍 3
2
Sara Gramling22:10:48

@genek Thanks! Can we keep Gather.town open for conference attendees for a few weeks?

Bryce Miller22:10:52

Such a great group of people here. Loved all the 🥐 conversations. Now I need to rewatch half a dozen of the talks and take notes.

💯 5
🇫🇷 3
Nick Eggleston, IT Architect (open to offers)22:10:29

So many videos to watch (or re-watch).... so little time...

☝️ 3
Gene Kim, ITREV, Program Chair22:10:49

I’ll find a more complete picture later — sorting by color achieved by @mvk842. 😆

❤️ 10
👍 3
1
😀 2
Bryce Miller22:10:51

Very much appreciated the absence of velvet ropes, though it was also rather unnerving.

❤️ 5
Slackbot22:10:15

Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://events.itrevolution.com/virtual-agenda/ https://devopsenterprise.slack.com/files/UATE4LJ94/F02GHSEB604/feedback-does21us.png

Joe Arrowood22:10:49

Thank you Jeff and Gene!

Gene Kim, ITREV, Program Chair22:10:51

That’s a wrap! Thank you, all!

🎉 11
👏 28
🙌 1
🎯 2
1
upvotepartyparrot 1
Denver Martin, Dir DevSecOps, he/him22:10:09

I would like to have 2 options - Remote and in-person maybe 2 different conferences ... thoughts

9
❤️ 2
Joshua Barton22:10:21

Thank you, Gene! And everyone who presented!

Ganga Narayanan22:10:29

Thank you so much, IT Revolution and everyone!!

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear22:10:55

Thank you all! This was my first DOES conference and I enjoyed it immensely!

Chris22:10:29

Thanks IT rev team for this edition (my fourth or fifth? But only one in person, not the easiest one to follow… in Beijing)

🌟 1
thankyou 2
Chris22:10:10

And hopefully, we may have some story for a future edition if things start rolling out at scale, in our traditional business, in a not moving very fast country… 😆

🙌 1
Sonal Bais22:10:43

@allspaw Part of the problem, many times, is the Incident Resolution templates being too long and data dependent(I've seen some with about 5-6 separate date and time fields)...one has little interest in documenting the details that really matter.

Denver Martin, Dir DevSecOps, he/him22:10:55

Where I worked, we asked only fill in those items if it mattered or it is quick and easy to fill in. The Story of what happened, how it was resolved and any other things that can speed up response and resolution is required.

👍 1
John Allspaw22:10:00

When there’s a template, what gets written is often what follows the template. As a result, people don’t read them because frankly…they’re not interesting, because templates != stories.

💯 2
👍 1
John Allspaw22:10:51

When the focus is on fixing, not much time is spent on describing. Rich descriptions pull people in. If the understanding of the incident isn’t very good, fixes that come from it aren’t likely to be any good either. 🙂

👍 1
Denver Martin, Dir DevSecOps, he/him22:10:52

This is an amazing conference Thank You everyone that made it possible ❤️💌💚💙💜🤎🖤🤍🧡💛💗

❤️ 5
thankyou 3
3
Denver Martin, Dir DevSecOps, he/him22:10:53

I have to run some errands, will check in later...