Fork me on GitHub
Kimberley Wilson11:10:25

“For years, our retrospectives didn’t feel like they were working; we worked to make it more blameless.  No more people standing up in front of room, explaining how things didn’t go right;  We’ve taken shame and humiliation out of the process, so they can tell stories and make sure it doesn’t happen again.”  — @kimberly_h_johnson How have you managed the balance between improving psychological safety whilst also holding people responsible for their work? It sometimes feels like role accountabilities and the desire to make sure we don't fail our members gives people a justification to continue with their non-psychologically safe behaviours.

❤️ 4
Kimberley Wilson11:10:25

“For years, our retrospectives didn’t feel like they were working; we worked to make it more blameless.  No more people standing up in front of room, explaining how things didn’t go right;  We’ve taken shame and humiliation out of the process, so they can tell stories and make sure it doesn’t happen again.”  — @kimberly_h_johnson How have you managed the balance between improving psychological safety whilst also holding people responsible for their work? It sometimes feels like role accountabilities and the desire to make sure we don't fail our members gives people a justification to continue with their non-psychologically safe behaviours.

❤️ 4
Geri O’Toole12:10:14

For me, I think we have to care and we have to demonstrate that too. We can't make others care, hence we can't make them be responsible. they have to do that for themselves. Everything we do should be a pull system anyway. Just my 2cents.

👍 1
Ferrix Hovi - Head of DevOps - Siili12:10:07

Asking questions and thinking out loud.

👍 1
Courtney Kissler12:10:16

Leading by example - I share my failures and learning too in as public of a way as I can. I also think words matter. I say "learn fast" vs. "fail fast" and have shifted the organization away from using post mortem to using learning reviews. It takes time to build that trust. I also like to talk about "honoring reality" and try to encourage senior leaders to honor reality vs. judging it.

👍 1
💯 3
John Allspaw13:10:31

By flipping the meaning of accountability to mean “able to give a detailed account of their perspective in/around an event, without fear of reprisal” rather than “falling on a sword”, it places the focus on how rich and detailed the account can be. Ironically, this can fulfill the desire you mention, which is the commitment to stakeholders. Public “lashings” don’t demonstrate commitment to genuinely learn from an event, it only implies that the org/leadership believe that all mistakes come from individual motivation, and that shame is an effective deterrent to making future mistakes.

👍 2
💯 1
Jon Smart [Sooner Safer Happier]13:10:37

@ckissler "have shifted the organization away from using post mortem to using learning reviews". I like that. Learning Reviews. I will be stealing that! 🙂

❤️ 1
👍 1
Gene Kim, ITREV, Program Chair13:10:33

@kimberley.wilson2 We have a fantastic talk from the Suncor team later this morning, who will talk about their safety culture, important to them because of the hazardous nature of so much of their work — from @jroa @lideluca and John Hill. (I forgot to ask them the obvious question about whether that emphasis on physical safety encourages psychological safety — it seems implicit, but let’s find out for sure from them!)

👍 1
Amri Abuseman - Flatiron Health, Director of Quality Engineering13:10:25

Good morning and ready for day 2️⃣ !!!

🌞 1
Gene Kim, ITREV, Program Chair13:10:24

Good morning, everyone! I’m excited about the talks today this morning!!

Nick Eggleston, IT Architect (DMs welcome)13:10:30

Love the spotlight on goals and community engagement throughout the year 😊

Jeffrey Fredrick, Author-Agile Conversations13:10:36

Read this last night and thought of this community @genek : If in your working hours you make the work your end, you will presently find yourself all unawares inside the only circle in your profession that really matters. You will be one of the sound craftsmen, and other sound craftsmen will know it. This group of craftsmen will by no means coincide with the Inner Ring or the Important People or the People in the Know. It will not shape that professional policy or work up that professional influence which fights for the profession as a whole against the public: nor will it lead to those periodic scandals and crises which the Inner Ring produces. But it will do those things which that profession exists to do and will in the long run be responsible for all the respect which that profession in fact enjoys and which the speeches and advertisements cannot maintain.

❤️ 3
Gene Kim, ITREV, Program Chair13:10:26

CS Lewis!! Thank you, Jeffrey! 🙂

Mark Peters13:10:26

One of the questions I always like, is do you understand what the next level of leadership is working on, and what they see? Sometimes things that seem odd at lower level make sense from that next perspective. Emphasizes the need for full team in retros

Joey Roa13:10:41


Joey Roa13:10:14

hah! Evening @ferrix

Gene Kim, ITREV, Program Chair13:10:53

So delighted you’re here, Joey! Looking forward to your talk later shortly!

Ferrix Hovi - Head of DevOps - Siili13:10:37

@genek I'm delighted that you're here 😄

Michael Winslow13:10:01

@jason.cox is a champion for the people!

👍 1
😂 1
Jeffrey Fredrick, Author-Agile Conversations13:10:46

“wrong side of the rope” fits perfectly with that Inner Ring essay ^ 🙂

BJ Preece13:10:14

can't wait for Dr Westrum later. Has anyone else listened to Gene's podcast with him at least 4 times :rolling_on_the_floor_laughing:

👍 3
❤️ 2
Jeffrey Fredrick, Author-Agile Conversations13:10:13

I was thinking this morning that I need to reply to the tweet asking about favorite Idealcast episode with Westrum as my choice.

BJ Preece13:10:21

It's so rich with insight, from both guest and host.

Nick Eggleston, IT Architect (DMs welcome)13:10:51

Yes i listened to the Idealcasts @ronwestrum did with @genek many times, pausing frequently to try and think over what I’d heard

Jeffrey Fredrick, Author-Agile Conversations13:10:25

And what a huge difference to put effort into removing that rope, instead of trying to keep a barrier of exclusivity!

👍 3
Gene Kim, ITREV, Program Chair13:10:46

(I listened to the 2 hours episode with @jtf for the 4th time two days ago. 🙂

😄 1
👍 1
Gene Kim, ITREV, Program Chair13:10:20

I forgot to thank @sophie.weston129 on Twitter, which is what prompted it.

👍 2
Nick Eggleston, IT Architect (DMs welcome)13:10:20

I too find myself getting new insight from talks on 3rd or 4th hearing 👍

Nick Eggleston, IT Architect (DMs welcome)13:10:53

How did you connect with people (like other speakers and leadership team) at the other conferences @genek ? Superrrr curious

Gene Kim, ITREV, Program Chair13:10:48

That’s a longer conversation — but happy to share!!

Gene Kim, ITREV, Program Chair13:10:50

Worst case, if we don’t get to talk this week, email me and let’s talk! (Maybe during one of the networking sessions?)

Nick Eggleston, IT Architect (DMs welcome)13:10:21

Yes please! I’m sure others would benefit as well! I’ll follow up with you

Margueritte Kim (CEO, IT Revolution, she/her)13:10:20

@nickeggleston - I’ve also found that, people tend to find Gene…be it a a city park, grocery store, remote beach, crowded bar. 😆

Nick Eggleston, IT Architect (DMs welcome)11:10:10

So a quick trip to the store can end up taking 5 hours…

🎯 1
Gene Kim, ITREV, Program Chair13:10:08

Wanna meetup in Gather during networking session this morning? I can share some of the stories, but MVK is right — it’s about 50/50 me finding them, and them finding me. After Phoenix Project came out, it became much easier, if you can imagine. Hope you’re having a great week here!

Joe Arrowood13:10:07

I think that I will be using the phrase "Wrong side of the rope" from now on.

Amy Cheng - TELUS13:10:39

Thanks for the org passes! 🙂

Michael Winslow13:10:43

Not sure I knew about the Org pass. Please make sure I know how to do this.

Michael Winslow13:10:26

thanks @scott.prugh!

Joe Moretti - Synapse Wireless13:10:52

Excited to hear about future live events.. while I am grateful for the virtual option, I am very much looking forward to getting back together in person hopefully next year!

❤️ 1
⬆️ 1
Gene Kim, ITREV, Program Chair13:10:54

So many of the new amazing great case studies came from this community!

👏 3
Mark Peters13:10:58

excited about the new version release, to bad we couldn't get a different color cover

Ann Perry - IT Revolution13:10:03

☀️ Starting us off this morning is the team from Suncor – @jroa @lideluca and @johnhill presenting, Continuous Delivery at Suncor's Digital Bay; Mining Maintenance Meets DevOps ☀️

👍 4
Ferrix Hovi - Head of DevOps - Siili13:10:49

I am happy that I managed to adjust my day rhythm enough to make yesterday possible. Tomorrow is going to be hard because of a dinner with the team.

Gene Kim, ITREV, Program Chair13:10:55

Thank you for presenting, @jroa @lideluca and @johnhill, to tell us about the amazing Suncor culture and story!

Joey Roa13:10:04

Thank you @genek

Lindsey Deluca13:10:36

Thank you for having us @genek !

Geri O’Toole13:10:54

This really speaks to me. Customers I serve at my org work in Nuclear Power Plants!

Mark Peters13:10:44

Safety - Sidney Dekker - Field guide to human understanding

👍 3
❤️ 2
Gene Kim, ITREV, Program Chair13:10:57

I forgot to ask you a question, @jroa @lideluca @johnhill Does the cultural emphasis on physical safety make it easier to achieve psychological safety? And if so, how? THANK YOU!

Nick Eggleston, IT Architect (DMs welcome)13:10:55

How and how often do you (suncor) measure psychological safety?

Joey Roa13:10:28

Great question, @genek I would suggest yes but it took time... Yes, in that the teams embrace the safety aspects as a common and important goal.

🙏 1
👍 1
Joey Roa13:10:30

It took time b/c as we went through our safety journey, there was pressure within layers of management that were still performanced on throughput. That created a tension that made it difficult at times

Nick Eggleston, IT Architect (DMs welcome)13:10:33

Huge logistical problems not visible until people stop and and questions

👍 1
Lindsey Deluca13:10:07

I agree. Over time - yes. At first it can make people feel uncomfortable as it has been a shift in culture. But overtime. The principles of looking out for one another create phycological safety.

🙏 1
❤️ 1
Gene Kim, ITREV, Program Chair13:10:26

Many have observed that the famous Toyota Andon cord requires a tremendous amount of psychological safety — when installed at GM plants, people were famously yelled at and punished b/c it jeopardized production targets.

👍 7
Geri O’Toole13:10:36

This really speaks to me. The people I serve at my org work in Nuclear Power Plants

👏 1
Gene Kim, ITREV, Program Chair13:10:28

Thank you @lideluca and @jroa!

Virginia Laurenzano NSA/MARFORCYBER13:10:39

key is "how we show up as leaders" - @lideluca

👏 3
Joe Arrowood13:10:53

@jroa, I think that your response about moving safety to a priority position is true for any culture change. In my experience moving Agile principals to a priority position meets with the exact same resistances. Thank you

👏 2
Lindsey Deluca13:10:19

While we don’t measure it per say… I think metrics like “near miss” reporting are a sign of phycological safety. People feel comfortable to step forward and say. “Something almost happened and I want to learn from it”

❤️ 5
Garrin Ball - DevOps Leader - DDMI13:10:31

Having spent most of my career in ecommerce, where people’s lives aren’t typically on the line, this is a great story to share. It’s reflective of other types of culture transformation we are all apart of.

👍 1
Gene Kim, ITREV, Program Chair13:10:05

(For people interested in this topic, you may like the wonderful section in Dr. Amy Edmondson’s book The Fearless Organization of a South African mining company, and a similar journey to zero on the job accidents.)

🔖 4
Larry Davis13:10:10

Awesome about safety culture! How do you maintain safety while moving forward purposefully without 'analysis paralysis'?

Joey Roa13:10:50

@nickeggleston Nice question! We have a few different measures... organizationally, we look to measures such as "Great Places to Work" surveys and similar trust indices. At the Team level, our Agile COE will use a Team Pulse check (similar to spotify's) that measures psych safety, team direction/vision, and how the team inter-operates. We look to the scrum master/team lead and agile coach to work through the check (anonymous) and come up with actions as a result

❤️ 5
Gene Kim, ITREV, Program Chair13:10:57

(It occurs to me that much of Dr. Sidney Dekker’s work also involves dangerous mining and resource extraction companies — showing how dangerous work in those industries are.)

💯 1
Joey Roa13:10:14

Very happy with the results when teams go through the effort (it is optional).

Joey Roa13:10:55

@jarrowood Spot on! Couldn't agree more!

Mark Peters13:10:16

I think one challenge is to make sure that safety doesn't impact development for non-safety issues. Like the DoD manadating operational testing, after users involved in design and fielding, Need to make sure safety acceptance is in team, and not a seperate silo

Gene Kim, ITREV, Program Chair14:10:02

ultra-trucks: 400 ton vehicles used in mining operations.

Virginia Laurenzano NSA/MARFORCYBER14:10:22

I wish we could do a touch-a-truck event with a mining truck.

😀 2
❤️ 4
Joey Roa14:10:50

@lwdavis There is always a risk of becoming to embroiled in all the "what ifs". However, generally speaking, the mix within a team will create a balance of safety and urgency to keep moving forward. We also continue to have progress/performance reports that can be used to highlight areas that might need a nudge

👍 1
Kurt A, Blameless, SRE Architect14:10:53

"increased wrench time" and "uptime of trucks" - a very different POV

Gene Kim, ITREV, Program Chair14:10:28

…when critical mission goals can’t be done with existing COTS software…

Joey Roa14:10:44

@tiny.mpetersii Good call out. Another lens is that when you apply safety across the board, it creates a muscle memory that becomes a reflex (you just do it). Then, the teams help temper the "right" amount of safety given the context of the situation. As always (so important), context matters!

Kurt A, Blameless, SRE Architect14:10:54

sounds like echos of The Checklist Manifesto

Jeffrey Fredrick, Author-Agile Conversations14:10:29

I’m hearing digital andon cord and standardized work. I was expecting Alcoa and I’m hearing Toyota. 🙂

🙂 1
❤️ 2
Joey Roa14:10:48

@kboth_does Checklists are great and serve a very useful purpose. When they are blindly followed, people miss the intent behind the checklist. We've suffered from that checklist mentality and it is something to watch for. I've personally seen IT teams look to get the checkmark instead of understanding and accomplishing what the checkmark actually represents. 😞

Kurt A, Blameless, SRE Architect14:10:07

Good point - I've heard this expressed as the difference between a check off list (going through the motions) vs. a list of things to check (applying thinking and care)

Jennifer Velasquez14:10:56

How do you address the training with such disparate view points of technology?

John Roesler - Sr Engineer (Gap Inc)14:10:08

"How do we implement a build into a culture that is predominantly buy?" Going through this right now

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere14:10:35

We recently went through this as well. Be careful of the pendulum swinging too far the other way. Now we're having conversations about, "Why are we building this now commodity software instead of buy it and invest our people in more differentiating software?"

John Roesler - Sr Engineer (Gap Inc)14:10:11

Completely agree with you there! I have worked in places that were too far down the buy route. We constantly weigh things as we discuss how to bush more build mindset to make sure they're things that differentiate us!

John Roesler - Sr Engineer (Gap Inc)14:10:32

and of course, if we build, we'd ideally like to open source and give back to the community

👍 1
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere14:10:01

Now we're trying to talk more about where our systems fall in Geoffrey's Moore's Core/Context mode:

John Roesler - Sr Engineer (Gap Inc)14:10:31

Thanks for the link - i'll take a look through!

👍 1
Gene Kim, ITREV, Program Chair14:10:09

The nature of the current way of working reminds me of how air tanker refueling scheduling was done in USAF, before Kessel Run Project Jigsaw. Lots of whiteboards for scheduling and tracking. Such an exciting project, @lideluca!

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere14:10:43

@leahb @alex @mvk842 - I notice that this talk is not in the Video Library (yet). Can we expect this to appear later today? Are you aware of any talks that will not be part of the Video Library (due to NDA agreements)? Thinking of @jason.cox's awesome Disney talks from a few years back.

Gene Kim, ITREV, Program Chair14:10:59

To my knowledge, all talks will be published.

Michael Winslow14:10:37

Loved the @jason.cox talks that could not be published! So happy I was there to hear about the "Grumpy" server and how "Darth Vader" is a Pathological Leader!!! lol!

❤️ 2
Alex Broderick-Forster, IT Revolution, Event Staff14:10:29

@mring we’ll be publishing this talk after the AM plenary block is over. I have my finger ready on the publish button.

👍 1
Alex Broderick-Forster, IT Revolution, Event Staff14:10:43

And yes, to my knowledge, all talks will be published.

Gene Kim, ITREV, Program Chair14:10:12

@mring Plenary talks are published at the end of the morning/afternoon. 🙂

👍 1
Gitty Rosenfeld14:10:44

@jroa you mentioned affecting culture change at Suncor, do you have tips on how to effect culture change?

Joey Roa14:10:00

@jen Training is tricky. It costs $ and takes capacity out of the system but it also drives productivity, engagement, etc. What we're trying (jury is out) is having digital fluency/literacy target courses at the org level (e.g. all of IT and functions). This includes areas such as Cloud, DevOps, agile, etc. Specific needs are meant to be identified at the team / project level. Then, a decision about how to best close the gap is made. It coudl be mentoring, buying (course or talent) or alternate.

👍 1
Denver Martin, Dir DevSecOps, he/him14:10:17

Wonder if a talk about small teams vs large teams would be good to have in future DOES - like how to empower the small team, also there is a need for large team, so how to make the large more effective, so really a compare and contrast and then followed by when to try each...

👏 1
Mark Peters15:10:31

Is it really a small vs large or a single team vs a collection of teams. Seems like if the team is too large, you start losing efficiency

Denver Martin, Dir DevSecOps, he/him17:10:05

I think it speaks to where there is spare capacity.. Larger teams seem to have more gray area and you can start with adding smaller items easier, but small teams you may have just look at starting a new team that can overlay on an existing..

Gene Kim, ITREV, Program Chair14:10:36

Thank you, @johnhill @lideluca and @jroa!!

👏 4
❤️ 1
Joe Arrowood14:10:40

Good Talk! Thank you

👏 1
❤️ 1
Ann Perry - IT Revolution14:10:48

google And now, we welcome @jpetoff and @cleng from Google, here to present, How Google SRE and Developers Work Together google

❤️ 2
Lindsey Deluca14:10:39

Thanks for having us

Gene Kim, ITREV, Program Chair14:10:42

Please welcome two phenomenal experts in SRE in theory and practice, Dr. @jpetoff and Dr. @cleng!

❤️ 2
upvotepartyparrot 1
Nick Eggleston, IT Architect (DMs welcome)14:10:16

@jroa how did your teams get introduced to DevOps and obtain buy in from the top?

Joey Roa14:10:05

@gitty.rosenfeld It's cliche but culture change is tough. I think you need to have support from leadership (we do, a la John Hill and others) and a motivated front line (leaders need to convey the why). From there, i sincerely believe it occurs at the team level. Coaches, mentors and leaders need to reinforce the culture we are moving to and sharing stories of successes. When old culture behaviors come forward, it's best to safely (no pun intended) provide that feedback in a non-threatening fashion. It's important for leaders to realize the change takes time. An org may have spent decades entrenching current behaviors. You can't flip that over night. It takes persistence and reinforcement.

❤️ 2
Gene Kim, ITREV, Program Chair14:10:13

(Dr. @cleng Ph.D. thesis was on distributed systems. 🙂 How fitting that he’s now doing SRE — like how some of the best firefighters can think like arsons. 😆

🔥 2
Gene Kim, ITREV, Program Chair14:10:38

“Google: at 2B+ LoC: might be the most complex integrated systems humanity ever created”

Jennifer Petoff14:10:44

Great to be here! Feel free to ask any questions you may have


Hello @jpetoff, @cleng - Looking forward to this

Christof Leng14:10:55

Not super relevant to this talk, but you can read my PhD thesis at

🔖 1
👀 1
Jennifer Petoff14:10:02

@lbmkrishna Nice to see you!

🙏 1
kristin valters14:10:09

@jroa it appears Suncor's DevOps journey started with one business segment versus the whole technology org transforming simultaneously - is this accurate or was it big bang?

Laura Henry - American Airlines [she/her]14:10:23

Love the quote - very empowering for the software engineers

Gene Kim, ITREV, Program Chair14:10:36

“BubbleStorm” — this sounds like one of those wonderful projects that torture people trying to preserve CAP theorem objectives, @cleng!!

🙏 1
Christof Leng14:10:49

Being tortured by CAP constraints is one of my favorite hobbies!

Gene Kim, ITREV, Program Chair14:10:50

“We now have 3000 SREs” — all reporting to VP 24/7 Engineering, Ben Treynor-Sloss.

😲 4
Joey Roa14:10:04

@nickeggleston We're still early in our maturity for DevOps. Our DevOps COE (small team initially), defined the capabilities that make up DevOps at Suncor and then prioritized which ones we felt we should emphasize. From there, we began the change and comms work to educate teams and leaders about the why . For the leaders, the messaging was a reinforcement of quality (repeatability via automation), resilience (being able to restore or deploy in a rapid fashion) and speed (doing 1000's of tests in minutes via tech). This would allow for shorter feedback loops with the customers to help the time to value equation. HTH. If not, hit me up and we can talk more.

Jeffrey Fredrick, Author-Agile Conversations14:10:09

the chemistry background for @jpetoff makes sense to me. a discipline that created the “things I won’t work with” series seems good background for SRE

Kurt A, Blameless, SRE Architect14:10:24

We will be publishing a podcast with @jpetoff later on this month in case you're interested

😊 1
👍 1
Jennifer Petoff15:10:40

@kboth_does can't wait to hear how the podcast turned out!

Jennifer Petoff14:10:26

BTW @genek did the talk start early? I dialed in at X;12 and we were already a few slides in...

Ann Perry - IT Revolution14:10:57

We did – running shockingly ahead this morning. so sorry!!

Gene Kim, ITREV, Program Chair14:10:07

I’m so sorry! Not sure — @annp would know. But all good, we’re all so happy that you’re here! (@annp, can you send the link to the complete video? 🙏 )

Jennifer Petoff14:10:49

All good. Just wanted to make sure I wasn't late accidentally 🙂

Jennifer Petoff14:10:36

Missed your intro :'-(

Gene Kim, ITREV, Program Chair14:10:23

The functional nature of SRE at Google is so interesting to me — and I’m so excited that @jpetoff and @cleng are sharing some of the “engagement models” between the product and SRE orgs, including the economics.

👏 2
🙏 1
Jennifer Petoff14:10:28

@jtf STEM subjects like chemistry are a great foundation for SRE: it's like applying the scientific method in a pressure cooker 🙂

❤️ 9
😆 1
👏 1
Jeffrey Fredrick, Author-Agile Conversations14:10:35

I completely agree! physics-chemistry double major myself. went from an interest in computational physics/chemistry into software.

Jennifer Petoff15:10:36

Very nice! I did synthetic chemistry myself so software is a bit of a departure, but I love it!

👍 1
Joey Roa14:10:34

@kristin.valters TBH, it's neither. We have targeted work going on across all the business units/areas. The messaging and vision has been communicated to everyone (org wide) with an emphasis on digital (some groups don't do anything tech so the message gets lost with them). Then, based on pull (or leader push, on occasion), we focus on a # of small teams in that area. Allows for tailoring of technologies used and business practices employed. Early in the agile transformation work I was leading, I quickly discovered you get the most "bang for the dollar" by going to the teams and areas that want you vs. selling to the groups that don't. The laggards will come along eventually. HTH

Gene Kim, ITREV, Program Chair14:10:40

“Both sides must agree to start the relationship; either side can end it.” (I love the way @jpetoff describes this.)

😊 2
Andrew Davis - Copado - DevOps and Testing for Low Code14:10:55

“SRE is a scarce resource by design” - is that to ensure that Devs retain some level of ownership?

Jennifer Petoff14:10:52

Yes. and also to ensure that we aren't taken for granted and only working on the highest value added things.

Craig Cook - IBM14:10:57

Are "developers" on-call at google?

Jennifer Petoff14:10:52

Indeed. For SRE supported services, Devs may also share a portion of the oncall load and many teams that don't have SRE support handle their own oncall.

❤️ 2
Craig Cook - IBM14:10:40

Is the team that owns the service the first one that gets alerts for it, or are first level alerts sent to a different team?

Jennifer Petoff15:10:06

Yes, the team that owns the service typically gets paged. There is no general triage queue.

❤️ 1
Daniel Cahill - Engineer - Ontario Systems14:10:18

How does the decision to hand a project back to a product team go? Do the SREs collectively decide that or the product team or someone in management?

Gene Kim, ITREV, Program Chair14:10:35

“Work must be challenging to SRE teams. It must improve the reliability of systems thru engineering.” — @jpetoff

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:40

Excellent description of how SRE works with ops!

👍 1
Nick Eggleston, IT Architect (DMs welcome)14:10:59

What is the decision process around funding SREs for those who hold that budget line? @jpetoff

Jennifer Petoff14:10:38

Paging @cleng to weigh in on how this works in practice.

Christof Leng16:10:29

I posted my response in the main chat earlier, but reposting here to make it easier to find: The budget for SRE headcount comes from the Dev org. They "pay" SRE via HC, but once transferred, SRE is in complete control of the HC (until the engagement is ended by either side).

Amy Cheng - TELUS14:10:02

Love it, "Fix it and fix it once and for all!"

Angel Diaz14:10:05

Hello everyone!!

Gene Kim, ITREV, Program Chair14:10:23

So glad you’re here, and thank you for presenting!

Bryan Finster - Defense Unicorns (Speaker)14:10:14

If I’m not accountable for my quality decisions, quality is a fantasy.

👍 1
Michael Winslow14:10:33

@genek told me I was going to love this one! This is so amazing @jpetoff nd @cleng!a

🙏 2

SRE is not Ops Team - Love that; - @jpetoff

Ganga Narayanan14:10:57

"SRE is not an ops team"! 🙂 Yes

Joe Arrowood14:10:11

@bryan.finster486 Well said

❤️ 1
Jon Smart [Sooner Safer Happier]14:10:13

Is there anything that prevent the product teams from thinking, "I'll just leave that for SRE to fix, as I know that they are there to catch things for me, so I don't need to think about it as much as I would have done if SRE didn't exist"?

👀 2
Jennifer Petoff14:10:33

That sounds like a 'throw it over the fence' mindset to me which I'd consider an antipattern. @cleng anything to add?

Jon Smart [Sooner Safer Happier]14:10:23

Yes, definitely an antipattern. What behaviour do you see at Google? Is there anything intentional which prevents that antipattern?

Jennifer Petoff14:10:42

I think there is an educational and communication component to it. SRE leadership and engineers on the ground communicating about the key SRE principles and best practices and how teams should work together for maximum impact. This is something that @cleng is actively working on. My team is also working to share reliability best practices more broadly. Reliability needs to be everyone's concern, not just SRE.

👍 1
Christof Leng16:10:52

There are a number of remedies to this challenge: • Write down who's responsible for what and who has authority over what. If SRE can block bad launches, Dev will try to work something out (However, the job of SRE is NOT to block, it's to advise!) • Work together. Have Devs work a little bit on infrastructure and ops and have SREs work a little bit on product - not too much, because it waters down the role specialization, but enough to maintain a mutual understanding. • Establish ultimate accountability for the product's reliability with the product team. SRE's job is to help them achieve that, but at the end of the day, they remain accountable. • If the situation spirals out of control, declare a "production freeze" (only stability fixes get deployed) and/or "code yellow" (reliability work trumps all other project work until the exit criteria are met). You need support from senior Dev leadership for either. If you can't get support from anyone in the reporting chain on the Dev side, you should find a better Dev org to work with.

👍 1
Jon Smart [Sooner Safer Happier]16:10:01

Thanks @cleng. "Establish ultimate accountability for the product's reliability with the product team. SRE's job is to help them achieve that, but at the end of the day, they remain accountable.", this resonates for me as to how to avoid the human tendency to think that it's someone else's problem.

Denver Martin, Dir DevSecOps, he/him14:10:21

The last 3 companies where I helped bring DevOps to life, I had to spear head getting Ops involved in the DevOps.. This can be challenging as most of these teams have been covered up in firefighting and are staffed to min staffing levels...

Gene Kim, ITREV, Program Chair14:10:22

What’s so remarkable of this talk is that everything is grounded in economics — deliberate surfacing that funding SREs is at expense of product devs; that SREs can’t be “bought” to do non-novel work, etc.

👍 2
Andy Nortrup - Director of PM at Tanium14:10:04

This sounds very much like when you bring a manufacturing engineering into a physical production process in order to address quality or throughput issues. My Dad did this at Pratt and Whitney to go in and help other teams or contractors figure out how to build a part correctly when they were having trouble. And they were an expensive resource to bring in.

👍 1

or for that matter - not Sys Admins renamed as "SRE" 🙂

🎉 1
Jennifer Velasquez14:10:48

Are you seeing the need to address the measurement or incentives for individuals in org? If so, how are you messaging that?

Christof Leng14:10:25

@nickeggleston The budget for SRE headcount comes from the Dev org. They "pay" SRE via HC, but once transferred, SRE is in complete control of the HC (until the engagement is ended by either side).

Denver Martin, Dir DevSecOps, he/him14:10:27

I see SRE as the breakout group that can be the champion in both Dev and Ops... they bridge the gap in many ways...

👍 2

@jpetoff - question, I guess the SRE comes into picture only for web scale systems at Google, correct? I see some refer about SRE for every IT Service/Systems. Just wanted to stand corrected.

Jennifer Petoff14:10:59

SRE support is typically limited to the most mission critical services. It's part of the cost benefit equation.


Love that "Mission Critical Services" - most IT systems owners think that all their services are "Critical" 🙂 in a typical Enterprise (the talk about availability and reliability) but not about the cost to achieve that

👍 1
Maria Luisa Polo14:10:00

@jpetoff what are the main responsabilities of the SRE Education Director, why this role is neccesary in your Organization?

👍 1
Jennifer Petoff14:10:02

Thanks for asking! I lead the learning and development function for SRE. My team is responsible for onboarding, getting folks ready to go oncall and ongoing education opportunities. We also bring reliability-focused education to all of engineering. Reliability needs to be a priority for everyone, not just SRE. This may sound fluffy, but we also foster a strong oral tradition and passdown of the SRE organizational culture through storytelling in our classes.

👏 1
❤️ 1

"Throw it over the wall mentality" 🙂

Gene Kim, ITREV, Program Chair14:10:33

“highly customized infrastructure make it difficult for SREs”, especially in situations when SREs handle multiple services” ==> drives/encourages standardization.

Andrew Davis - Copado - DevOps and Testing for Low Code14:10:36

“You can’t build a wall and then complain about a ‘throw it over the wall’ mentality”

❤️ 6
Jeffrey Fredrick, Author-Agile Conversations14:10:44

standardization again. relevant in truck maintenance and software environments. under appreciated I think.

💯 4
Jon Smart [Sooner Safer Happier]14:10:38

For the knowable, or meta level patterns in context, I would suggest 🙂 A headwind for the unknowable (treating the unknowable as if it's one size fits all)

Vaidik Kapoor (Speaker) - Technology Consultant14:10:49

What are some of the ways that are used at google to “teach how to fish”, especially when something is on fire?

👍 1
Christof Leng14:10:44

• Sharing ops work to some extent • Escalating during incidents, debugging togehter

🙏 1
Christof Leng14:10:22

• reviewing postmortems together • co-design sessions • ops/architecture training sessions for Dev

Denver Martin, Dir DevSecOps, he/him14:10:54

We have the SRE team lead the BPM (Blameless Post Mortems) they are in a great place to look at issues deeper and can then help those involved get to the root cause easier and see the action going as far as possible with in Development or in better Operations.. I know I am not at Google but thought I would chime in... hope that is okay.

❤️ 2
Jennifer Petoff14:10:00

Thanks for chiming in @mr.denver.martin always interested in how others do it.

Christof Leng14:10:43

@mr.denver.martin That's actually very similar to how many Google teams organize postmortem reviews.

Denver Martin, Dir DevSecOps, he/him14:10:04

I can also see SRE building the tools and process for Fishing, not just teaching other to Fish.. but they are not Fisher People... 🙂

Nitin Kulkarni14:10:56

@jpetoff I wonder how you maintain guard rails though and not create a chaos of tools and strategies if SRE is an optional engagement?

Jennifer Petoff14:10:42

I think @cleng is best placed to tackle this one.

Christof Leng16:10:51

It's complicated. The preferred strategy is to make the tools and strategies you want the teams to use the most attractive ones (incentives). They can be most accessible, easy-to-use, best supported, most feature complete. Don't try to force a solution on your engineers that isn't working for them. They'll find ways around it. The second approach is on the relationship level. Just because a Dev team doesn't have (full) SRE support, doesn't mean that they're not exposed to SRE. There can be consulting, SRE Love, training programs, tech talks, etc. When SRE has a reputation to be helpful, the devs will ask you for advice and follow your best practices. SRE is production evangelism. The third approach is "the stick": Policies about what you can/cannot use, automated compliance measurement/enforcement, nice-or-naughty dashboards for senior leadership, etc. I would generally advise against these, but there are corner cases when they are necessary. Typically, when you can convince 80-90% of the org with the other strategies, it can be sufficient to show that number to non-compliant team (and/or their bosses). Then again, not everything always needs to be uniform. If you don't have to deal with it, let them do what they feel is right. If you always follow the one-size-fits-all approach, you stifle innovation. Listen to why they chose a different path. Maybe they have good reasons.

Sean D. Mack14:10:04

@jpetoff @cleng Amazing talk! Does Google have any sort of 24 x 7 monitoring team/Operations or is that all delegated to the dev teams? If it is responsibility for the development teams how do you handle legacy applications which may not be under active development?

👀 3
Gene Kim, ITREV, Program Chair14:10:12

What’s so bad about discussing SLOs after software is written? What could go wrong? 😆 😆 😆 (“Overengineering something at the expense of valuable features”)

😆 2

for the scale, pace of questions, comments for this subject - we need to engage SRE - cc @jpetoff (How are you scrolling all these comments)

👍 1
Jennifer Petoff14:10:57

Ha! working to get through the Qs as quickly as possible. Also listening to the talk at the same time which is distracting me. I'm terrible at multi-tasking 🙂

😆 1
Andrew Davis - Copado - DevOps and Testing for Low Code14:10:23

“Important to not see SRE as a human abstraction layer over production. That’s an invitation for complexity to flourish” - @jpetoff

👍 4
☝️ 3
Andy Nortrup - Director of PM at Tanium14:10:14

The explicit funding model of the engagement process described reminds me of the economic principal of optionality that Gene has talked about in the most recent few episodes of the Ideal Cast. You pay for an SRE to come over if you think that is going to increase your overall value more than spending that money on something else (like another Dev, designer, PM, whatever).

Gene Kim, ITREV, Program Chair14:10:18

“For products in new business units, you could get away with lots of Baseline engagements”

Scott Kellerman (DevEx Product Owner, Vanguard)14:10:54

hey @cleng, can you tell us more about how you measure team maturity?

👍 1
Christof Leng14:10:16

Service maturity: • SLO quality (user-oriented) + compliance • Ops workload (tickets + incidents) • Data integrity processes • Capacity planning / efficiency • postmortem processes and hygiene • Release automation Team maturity: • OKR planning processes • Staffing/attrition • ops workload • SRE/Dev relationship

Scott Kellerman (DevEx Product Owner, Vanguard)14:10:28

thanks @cleng! really insightful, love how staffing attrition is a factor in team maturity, often overlooked but it's a key leading indicator to predict the quality that a team might deliver

Christof Leng16:10:44

Unfortunately attrition can be a trailing indicator, because it takes time to build up enough frustration that the engineers actually leave. Also, be careful to measure this only for organizations big enough that you get a meaningful signal. When you look at a 5 people team and 2 of them leave for reasons completely unrelated to the team health, you get a huge spike. A noisy signal is not useful.

👍 1
Gene Kim, ITREV, Program Chair14:10:20

I love the benefit of being able to call experienced SREs when things go wildly wrong in a major incident!

Nick Eggleston, IT Architect (DMs welcome)14:10:46

Get that project some “SRE ❤️

❤️ 2
Gene Kim, ITREV, Program Chair14:10:15

“SRE Love” — when devs write proposals / requests for SRE help, to aid in knowledge transfer, mentoring, skill upleveling. “Helps build relationships between devs and SREs.”

❤️ 1
Amy Cheng - TELUS14:10:47

The SRE Office hours and Continuous Learning in mentoring the devs is awesome!

❤️ 1
Brian W. Spolarich - Cal Poly14:10:47

Fascinating how this feels like a "service you can buy based on your goals and funding" all grounded in economics.

💯 1
José Chanto14:10:15

I don't get yet how SRE is different to an operation team

Jennifer Petoff14:10:13

I think a big difference is the focus on automating away toil so that the team can scale sublinearly to the size of the service rather than throwing more bodies at the problem.

❤️ 1
Scott Jaffa (Principal Engineer, ValidaTek)14:10:28

Think of it as SREs provide operational (production) expertise. They’ll help fix an operational issue, but the goal is to understand the issue such that they can engineer away the problem from happening again.

❤️ 1
Scott Jaffa (Principal Engineer, ValidaTek)14:10:35

Or the actual expert answered 😄

Jon Smart [Sooner Safer Happier]14:10:54

i.e. writing better fire fighting equipment, so that don't need to spend time fire fighting?

Denver Martin, Dir DevSecOps, he/him14:10:58

Hi @jchanto17 I see SRE as the bridge between Ops and Dev, these are people that are pulled out of or not part of Dev or Ops, but they have skills and knowledge to be able to get to root cause and then can look at 1. fix temporary with work arounds 2. figure out how to respond and restore faster 3. fix long term how to keep it from happening again.

Jennifer Petoff14:10:13

well stated @scott.jaffa!

Pete Nuwayser - IBM14:10:53

Might an SRE also look for risks before they become issues? Is that inherent to eliminating toil?

Jennifer Petoff14:10:28

Yes @nuwayser, SREs are empowered to make tomorrow better than today by focusing on the things that will have the most impact on improving reliability of the services they support.

👍 1
Sujay Solomon14:10:12

SREs make more sense in places where dev teams actually own dev & ops - more common when your apps are cloud-native.

Sujay Solomon14:10:38

in traditional datacenter based apps, SREs may play the role of a bridge between dev & ops

Sujay Solomon14:10:41

i'm involved in some research projects on whether SREs fit into the hybrid world of cloud-native + legacy datacenter apps. It's been interesting and challenging

José Chanto14:10:23

Ok thanks all for the comments, now it makes sense to me. And what I think is that I can take this type of firefighting work out of the Dev team, because in my case Dev team use to tackle this type of problems and slowdown the value delivery

👍 1
Pete Nuwayser - IBM14:10:36

> Yes @nuwayser, SREs are empowered to make tomorrow better than today by focusing on the things that will have the most impact on improving reliability of the services they support. Thank you @jpetoff :) @jchanto17 to your question - as someone who has worked in traditional ops before, an SRE is someone with my mindset + developer skillset, and who is also empowered / expected / measured on their ability to • participate in incident response to close issues • proactively mitigate risks • leverage error budget to find problems make the service more resilient By developer skillset I mean more than python/bash/perl scripting: it's someone who has the skills and license to get into the app code and improve it proactively. @jonathansmart1 to your earlier point re: measurement alignment between SREs and Product owners - curious to know if you think the two roles are responsible for the same outcomes or different ones.

👍 1
Jon Smart [Sooner Safer Happier]17:10:52

Hi @nuwayser, not sure, keen to understand the case study point of view (SREs and Product Teams, not only POs). I guess that it will depend on org by org, as to how that is done. With more aligned incentives or less aligned incentives. Keen to understand and how risk is mitigated.

thankyou 1
Daniel Cahill - Engineer - Ontario Systems14:10:27

Are there any examples of SRE Love projects that we could use as a example?

Christof Leng14:10:35

Typical examples: • Helping a Dev team to set up their monitoring/alerting • Analyzing SLOs / refining them • Architecture review • Picking the right tools / infrastructure for a new "greenfield" service • A migration to new infrastructure (e.g. database) • Improving ops processes

❤️ 1
Jon Smart [Sooner Safer Happier]14:10:28

How does incentivisation (performance appraisals, pay, promotion, rewards) work for SREs? How are SRE incentives and Product Team incentives kept in line, to avoid the antipatterns which would easy to occur (e.g. SREs incentivisation not inline with Product Team org incentives, e.g. lighting fires in order to put them out, as an extreme example)? Thanks!

👀 3
Christof Leng14:10:50

Core aspects of SRE performance are impact (measurable improvements for users/Dev/SRE/budget) and simplicity (standardization, deprecations, cleaner architectures, less dependencies). Heroics can be rewarded temporarily but are a generally not going to get you a promo. Specifically, the focus for performance evaluation is on designs and landing engineering projects not "keeping the lights on".

👍 1
Jennifer Petoff14:10:09

Service Level Objectives are nominally the tool that allow SREs and Devs (and other business stakeholders) to speak the same language and align on incentives

👍 1

@genek - we need a slack plugin to move all these "Gold nuggets" to a ever living doc (kind of shortform or notion) - Goodness me - there is so much to read thru

💯 10
🎉 1
❤️ 1
Graham McGregor14:10:44

Are dev-teams on call overnight? Is there any pattern of follow-the-sun rotation handing off to another team for overnight?

Jennifer Petoff14:10:00

Response time would typically depend on the criticality of the service.

Graham McGregor14:10:22

So are the dev teams are on-call?

Graham McGregor14:10:15

I'm pretty sure that's how it is, but I was hoping for confirmation. There's folks in our org who don't agree with that.

Christof Leng14:10:50

Yes, Dev teams are oncall, some of them 24x7 when they don't have multiple sites across the globe. That's not what we would do for business-critical services though. Being paged at 3am when the rest of your team is soundly sleeping is not a recipe for success. However, SRE escalation support via baseline can be helpful in these scenarios.

Graham McGregor14:10:39

Thank you for the response! What would the size of those teams be? How many people around the globe? We have some teams of 5-7 that are responsible for critical services and currently hand off to operations overnight because they don't want to be woken at 3am, but the operations team is becoming overloaded.

Christof Leng16:10:52

For an SRE team, we recommend at least 6 engineers in each of the two sites. That's a significant investment, so each SRE team is typically working on a large set of services (or a few very big/critical ones). Dev being oncall during business hours and SRE being oncall during off-hours is also a model that is working well for Google SRE teams that are using it. It increases the coordination overhead a bit (more cross-team handoffs), but makes dealing with faulty releases easier (releases should be done when the devs are oncall - they're more familiar with the changes). It also exposes Dev to production, but with a safety net - there's always an SRE familiar with the service you can escalate to.

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear14:10:16

The clearly defined different engagement models is really interesting to me. Having that so well defined to give teams the ability to decide what they can afford and not just what they would ideally want. It seems really useful to keep the smaller and less important projects from swamping the SRE org.

👍 6
Gene Kim, ITREV, Program Chair14:10:09

Haunted Graveyards!!!!

😂 4
🪦 1
👻 4
Javier Magaña - Walmart14:10:24

Hmmm... seems like an SRE role aligns a lot with my interests. Is there any good resources to start learning and grow this muscle? I assume the SRE O'reilly book is a good place to start?

Jennifer Petoff14:10:01

@nepobunceno There are lots of resources to check out at including our various books. There are some good large system desigjn exercises and shorter form articles too.

thankyou 1

There are great books by Google (Free) and also there are number of courses in LinkedIn and Youtube available. The one thing I will recommend strongly the micro learning videos on this subject by Google Seth Vargo and our own @lizf

Daniel Cahill - Engineer - Ontario Systems14:10:06

@jpetoff Where can I find the system design exercises? I'm scrolling through the site and recognize books and articles I've grown from and helped my org change processes to reflect.

Gene Kim, ITREV, Program Chair14:10:41

I’m fascinated by the path one is required to go through to get to Full SRE Support — and how products going thru hypergrowth will likely need it!!'


I have to confess here (A general statement): All these good work by various organizations - SRE, DevOps, Product Management, Funding, Transformation, Team composition, Organizational design - all these get over complicated in their own way in many Enterprises and a common statement comes to answer all the time - WE ARE NOT GOOGLE - cc - @genek, @jpetoff . Reminds @jonathansmart1 book - antipatterns

💯 4
Christof Leng14:10:12

Google is an enterprise with its own challenges. With 140k employees and >20 years of history there are many complications on the ground. Individual SRE PAs find their own solutions on how to adapt to their space. Please don't take Google SRE as a blueprint to be applied verbatim to your org. It's a case study. We keep evolving the way we do things. Because we have to adapt all the time.


^^^^ THIS - Love this @cleng (the same applies to all the models right - Spotify, Etsy, Google, Amazon)

Christof Leng16:10:06

I definitely don't want to become a preacher for a cargo cult. 😉

🙏 1
Jeffrey Fredrick, Author-Agile Conversations14:10:16

I like that the SREs can vote themselves off the island

❤️ 3
😆 1
Nick Eggleston, IT Architect (DMs welcome)14:10:22

How are SREs on a project evaluated for performance/promotions? How frequently and by whom? @jpetoff @cleng

Christof Leng14:10:42

SRE's performance is evaluated by SREs (up to a certain level of seniority), but SRE managers and promo committees expect positive peer feedback from Dev peers as critical support for high ratings.

Jeffrey Fredrick, Author-Agile Conversations14:10:32

that autonomy is a really useful source of information

Gene Kim, ITREV, Program Chair14:10:38

“When SREs have all left/abandoned a [problematic service], well, the developers are left wearing the pager anyway.” 🙂 I love these “tough love” statements from @cleng.

❤️ 6
Jennifer Velasquez14:10:48

“smart engineering” not “brute force”. love it

❤️ 5
Matt Wheeler14:10:24

So many gems in this talk. Definitely one to rewatch and share.

💯 3
Gene Kim, ITREV, Program Chair14:10:35

“you never fully understand a system until you see it burst into flames” — @cleng 😆

👏 7
😆 2
🙌 1
Christof Leng14:10:02

@genek "Some of the best firefighters can think like arsons."

Gene Kim, ITREV, Program Chair14:10:36

😆 😆 Like a connoisseur of disasters, if you will. 🙂

😋 1
Gene Kim, ITREV, Program Chair14:10:17

I’ve always admired the way that Google has talked about how they’ve defined SRE as a career path, as a set of skills, so much pioneered by @jpetoff.

❤️ 2
Jennifer Petoff14:10:50

Thank you! PSA that all the books are available online for free at

Nick Eggleston, IT Architect (DMs welcome)14:10:44

Google got their own TLD? I didn’t know that!

🤯 1
🙂 1
Kurt A, Blameless, SRE Architect14:10:48

They also have .prod 😄

👍 1
Christof Leng14:10:48

Try 😉

Nick Eggleston, IT Architect (DMs welcome)11:10:12

When can I get “Nick.”? :rolling_on_the_floor_laughing:

Gene Kim, ITREV, Program Chair14:10:40

Thank you so much @jpetoff and @cleng for giving us a glimpse of how SRE works inside of Google!! This is something I’ve wanted to better understand for nearly a decade! 🙏

❤️ 7
Jennifer Petoff14:10:49

Thanks so much for inviting. us, @genek. Will continue to work our way through the backlog of Qs!

Christof Leng16:10:07

It's been an honor and a pleasure. Fantastic community at DOES! Great questions, many things to learn from others too!

Jeffrey Fredrick, Author-Agile Conversations14:10:47

Conferences are for conferring 🙂

👏 3
🙂 1

Challenge to Enterprise Technology Leaders - "WE ARE NOT GOOGLE" - WE WILL NEVER BE; 🙂

🙂 1
Jennifer Collings14:10:29

Thank you!!

👍 1
Ganga Narayanan14:10:31

This is amazing and very timely with some of the work we've begun doing! Thank you @jpetoff and @cleng! Reminder to self to read up on your SRE books!

👍 1
Khan, Humayoun at TELUS14:10:32

What was the SRE site again

Khan, Humayoun at TELUS14:10:19

thanks i also googled it

Jennifer Petoff14:10:42

Nice...I see what you did there 🙂

🙂 1
Glenn Wilson, Author of DevSecOps14:10:37

Wow - so much to take in. I’ll be watching this one again

❤️ 1
Jennifer Petoff14:10:42

@eleonravinez @jeff.gallimore There is a lot of unpack there. I think the best way to sum it up is through this conference talk that I gave at FailoverConf last year: "Swim Don't Sink: Why Training Matters to an SRE Practice"

👍 2
thankyou 1
Virginia Laurenzano NSA/MARFORCYBER14:10:45

When I get the "we're not Google lecture" I try to find ways to say "but we can aspire to be"

👏 4
❤️ 6
🙌 2
Jennifer Petoff14:10:39

@vmshook Many of the foundational principles of SRE can be applied no matter what your size (e.g., SLOs and error budgets and a 'vanquish toil' mindset)

👏 1
Nick Eggleston, IT Architect (DMs welcome)14:10:42

I love that the whole conference is focused on these topics at the same time. It’s creates a great shared community context.

Javier Magaña - Walmart14:10:05

Thanks a lot @jpetoff and @cleng. Very informative.

👍 2
🙏 1

thank you @jpetoff, @cleng - Brilliant session (I can manage without a coffee now for a while)

🙂 2
💯 2
Maria Luisa Polo14:10:24

Thanks for sharing!! very usefull, I’ll be watching this one again

✔️ 1
👍 2
🎉 1

When working with an SRE, how do you decide whether to go with an incremental approach over redesigning a whole system to meet reliability goals?

Jennifer Petoff14:10:26

Another good one for @cleng to tackle!

Christof Leng14:10:46

When you have been working closely with the system and know all the hidden timebombs: An incremental approach is generally preferred, because a clean-slate rewrite is costly and high risk. However, sometimes you have explored all incremental options and they either don't get you to an acceptable level or are even costlier/riskier than a rewrite. Then it's time to sharpen your design pencils. When you're new to the system: A thorough architecture review and a regular ops review to assess the long- and short-term viability of the system should get you close to the above.

❤️ 1
Jason Cox - Disney14:10:49

@jpetoff and @cleng - Thank you! Can you post a snapshot of the SRE in a nutshell slide here?

Jennifer Petoff14:10:06

Here you go. Also lots of good material at

Jason Cox - Disney14:10:42

Thank you! Great talk.

❤️ 1

AS @genek says - DOES is like a recharging battery station. I managed to charge by (Inspiration) and Battery over last 6 Years - JUST by DOES

💯 2
🙏 4
Christina Biangslev14:10:28

@cleng @jpetoff Were there ever a valid counter argument for putting SRE inside the product teams? (If this is covered in books/papers I haven't read, please share! this is the single hottest topic where I'm at right now.)

👏 2
👀 1
Jennifer Petoff14:10:02

@christina.biangslev This is not something that has been considered at Google AFAIK. Keeping SRE and Dev reporting lines ensures that reliability is a first class feature. If everyone reported up to Dev, it's possible that the orgs leaders might be tempted to trade off reliability for feature velocity or to take on more tech debt than is wise. I know other companies have approached SRE differently though and have found success with a more embedded or consulting approach.

👍 1
👆 1
Javier Magaña - Walmart14:10:31

I agree. What I see often is compromising quality because of the pull from business to deliver new features. I think this would also be something that would make it a second class citizen.

Christof Leng15:10:40

See for a good argument. Summary by Ben Sloss: "So for that reason SRE has to be its own team. It's my basic thesis. If you don't have a team who views their mission in life as making sure that the product works, you will ignore availability and reliability until you're in real trouble."

🙏 1
👍 1
Christina Biangslev15:10:49

I completely acknowledge this risk, that Feature Fetisch might taking over the SRE agenda. I'm struggling to fit the ideal of the truly independent product team with the experience of our SRE pioneers. I'm sure there's a place for a variety of setups; I'm wondering what the deciding factors would be to consider the truly independent model.

Christof Leng16:10:00

There's a whole website for the many organizational options you have:

🎯 1
Nick Eggleston, IT Architect (DMs welcome)14:10:26

Speaking of talks happening at the same time… if we go back later to listen to one we missed, which channel is best for the follow-up discussion? (since we don’t have one channel per talk or (set of) speaker(s))

🙏 1
Denver Martin, Dir DevSecOps, he/him14:10:06

Maybe do DM to the presenter and maybe they would still be watching the Slack Workspace... just a thought.. but then you miss on others that may have knowledge besides the speaker... hmm.

Nick Eggleston, IT Architect (DMs welcome)14:10:26

It’s tough to know. I love staying on one channel throughout the day for the lively engagement and community, but it would be super cool if the thread discussion for a given talk were moved to a dedicated channel for continuing discussion… @genek

Ann Perry - IT Revolution14:10:20

👏:skin-tone-2: Let's get ready to welcome the team from Capital One – @girija.rao, @denee.ferguson @jennifer.miles, presenting Productizing the Network: Square Peg, Round Hole?:clap::skin-tone-2:

👏 2
Brian W. Spolarich - Cal Poly14:10:45

I love to see this panel!

❤️ 3
👏 1
Gene Kim, ITREV, Program Chair14:10:35

I’m so excited that @girija.rao @denee.ferguson and @jennifer.miles will be talking about they brought DevOps principles for core networking that enables (all?) major bank operations!

Nick Eggleston, IT Architect (DMs welcome)14:10:48

This reminds me of the talk BMW did about applying DevOps principles to their core IT support org

Gene Kim, ITREV, Program Chair14:10:14

“My favorite: Wireless LAN”. from @denee.ferguson (I’m getting stressed out hearing about all these mission critical services. “It’s always the firewall or network.“)

❤️ 1
😆 1
Jeffrey Fredrick, Author-Agile Conversations14:10:09

timely after facebook’s experience this week…

Brian W. Spolarich - Cal Poly14:10:10

Actually, its always DNS.

😆 2
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:46

lol... @genek I like to say "bring it on"... most of the time, it isn't!


so DevNetOps is a thing, right? cc - @denee.ferguson

👍 1
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:19

It is!!!!! Incorporating the development/automation has been key to getting out of the firedrill mode, and bringing sanity to our pace of delivery

❤️ 3
Scott Prugh (DOES Prog Committee)14:10:43

Ah.. @girija.rao is talking about creating Build/Run teams for Network!

Matt Wheeler14:10:29


Gene Kim, ITREV, Program Chair14:10:30

“Previously, we had Engineering and Run reporting to two different executives” — @girija.rao cc @scott.prugh

👏 1
Scott Prugh (DOES Prog Committee)14:10:28

This is awesome. Parallels to some other stories:

👍 1
Scott Heaberlin16:10:16

@scott.prugh may I ask what that’s from?

Craig Cook - IBM14:10:38

I heard this org change referred to as "reverse conways law". If you don't like your architecture, change your organization to reflect what you want your architecture to be.

❤️ 7
Vaidik Kapoor (Speaker) - Technology Consultant14:10:23

align it to business outcomes and the architecture gets fixed.

Gene Kim, ITREV, Program Chair14:10:25

(More difficult to rearchitect your core switches, which are global in nature, than most software. 🙂

Vaidik Kapoor (Speaker) - Technology Consultant14:10:46

but sometimes it doesnt. sometimes figuring out the right archtiecture is also really hard

Pete Nuwayser - IBM15:10:06

@cncook001 We should call it The Law of One Foot

Kurt A, Blameless, SRE Architect15:10:11

Team Topologies in action

✔️ 1
Meghan Glass - PrdMgr Best Buy14:10:05

Network Engineering + Network Operations separate teams => Eng + Ops team but this is still IT focused, correct? These product teams don't include non-IT operations providing value to customer?

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:52

The org itself is IT focused... But the org also has non-IT staff members that are integral to our success.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)14:10:12

@cncook001 Indeed!!! Over 160 people changed managers as part of this transformation....

❤️ 2
Pete Nuwayser - IBM14:10:44

"Shoulder Tapping" - these words are giving me a mild anxiety attack.

💯 2
😂 3
Pete Nuwayser - IBM15:10:29

Yet another reason why open floor plans can kill productivity. At my previous studio job, somebody created paper indicators that you cut out to put on your monitor: • Green: I can be interrupted • Yellow: I can be interrupted but we can't talk about your cat • Red: DND


@cncook001 - This reminds me Martin Fowler quote "You change the organization or change your organization" 🙂

👍 2
Charlie Betz14:10:56

Oooh. "Shoulder-tapping" as A Thing. Stealing that.

Gene Kim, ITREV, Program Chair14:10:32

The scale of this talk blows me away: 14,000 devices, 185k carrier assets (!!)

👍 1
🤯 1
Gene Kim, ITREV, Program Chair14:10:48

Is that 14K networking devices, @denee.ferguson?

Brian Smith15:10:24

I have a manager who loves drive by's . I developed PTSD from every time I heard his office door open.

😆 2
😢 1
Matt Wheeler15:10:54

I had the same, started working in the cafeteria.


Most of the "Productive" Engineers prefer to work from home (as they do not need to see these managers and other distractions)

👆 1
Jennifer Miles15:10:11

You would think remote working would provide some speed bumps for those drive bys but not always the case. Where there is a will there is a way!

💯 1
Brian Smith15:10:45

Now that we are remote, life is much better.

👆 1
Virginia Laurenzano NSA/MARFORCYBER15:10:44

mine is back in force w/no telework options and a move back to an open office. less time to think

Brian Smith15:10:10

I wondered that yesterday about your workforce.

Virginia Laurenzano NSA/MARFORCYBER15:10:28

some jobs have that option. my last one did. my current does not.

Virginia Laurenzano NSA/MARFORCYBER15:10:58

it's definitely something IC/DoD is needing to rethink

Charlie Betz15:10:47

The biggest problem I am hearing in infra product team transitions (aka platform teams) is that 1) engineers don't easily skill into product manager roles and 2) have an antibody reaction if you parachute in someone with product skills who is not an engineer.

👍 4
👀 1
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:40

The biggest lesson is the product managers NEED to understand the technology...

Charlie Betz15:10:11

That can be a purple squirrel quest...

👍 1
Jennifer Miles15:10:12

Completely agree with @denee.ferguson


This leads to me to question - The PdM's do they need to be technical or Business 🙂 or both?

Pete Nuwayser - IBM15:10:24

I ❤️this thread and it's barely gotten started.

Jennifer Miles15:10:46

They are both in our org or at least that is the desired state

Pete Nuwayser - IBM15:10:04

At my previous job we had motion in both directions (engineering->product crafts and vice versa)

❤️ 1
Chris Gallivan (Tasktop-Planview)15:10:28

My attempts at product teams with Infra were underwhelming at first, but I found several months later they came back to me with an appreciation for what they learned. It just took longer to click

❤️ 3
Pete Nuwayser - IBM15:10:54

I think what we found is that engineering->product manager was better than the other way around

Geri O’Toole15:10:56

@char re the purple squirrel quest, one of my challenges is to "train" the infrastructure "POs" at the moment. Im just getting started.

Pete Nuwayser - IBM15:10:36

common theme: I was a developer and kept building features that nobody used or wanted. I became a PM to prevent that from happening

👍 1

In some org (Enterprises) - The Product Management is (abused). The PdM's act as Service Delivery Managers (approving people work)

Charlie Betz15:10:50

Has anyone tried a "two in a box" strategy for platform teams?

Pete Nuwayser - IBM15:10:31 like PM+Engineering Lead?

Geri O’Toole15:10:35

@chris.gallivan421 Id love to hear more about your successes with infra POs. what did they come back to you for?

💡 1
Brian Gallop15:10:38

Can you expound on that?

Jason Trent15:10:50

@char,two in a box??

Chris Gallivan (Tasktop-Planview)15:10:59

@gerijotoole we did a dojo with them (Joel Tosi and I)

Chris Gallivan (Tasktop-Planview)15:10:42

They said they liked mobbing, rather than working in individual silos

Pete Nuwayser - IBM15:10:50

hey @ckissler this needs to be a Lean Coffee topic

❤️ 1
Charlie Betz15:10:58

Many product teams are matrices with consensus leadership shared by two "pyramid" reps

Chris Gallivan (Tasktop-Planview)15:10:09

they also said, they felt like a real team for the first and only time in the dojo

❤️ 1
Charlie Betz15:10:22

eg a product "pyramid" and an engineering "pyramid"

❤️ 1
Geri O’Toole15:10:30

ah.. I have experience with those and have even talked about the quick learnings, but at this time, they are barely "walking," not to mention leadership buy in

Geri O’Toole15:10:11

@chris.gallivan421 those comments have been ones ive witnessed as well. it's a beatiful thing, isn't it?

👍 1
Jeffrey Fredrick, Author-Agile Conversations15:10:12

engineers don’t easily skill into product manager roles < I’m wondering what specific problems you’ve seen with people who wanted to make the transition.

👀 3
Chris Gallivan (Tasktop-Planview)15:10:22

at Stellantis we favored the 2 in a box, but it was mostly because we needed a place to put the managers 🙂

Geri O’Toole15:10:58

@jtf same. I am just getting started, week 3 at a new org, with this setup

Charlie Betz15:10:21

@jtf These are field reports I am hearing. Just went through this in detail yesterday with a large Australian investment brokerage.

Chris Gallivan (Tasktop-Planview)15:10:22

@bryan.finster486 has some good takes on 2 in a box

✔️ 1
Chris Gallivan (Tasktop-Planview)15:10:42

on app side this led to typing pools rather than product teams

Geri O’Toole15:10:50

@bryan.finster486 has some good takes on lots of things. thanks for the tip

Charlie Betz15:10:02

@jtf have heard similar at least 3 or 4 other times.

Bryan Finster - Defense Unicorns (Speaker)15:10:02

I mean, I have OPINIONS.

😂 4
❤️ 1
Bryan Finster - Defense Unicorns (Speaker)15:10:13

I’d cross check them if I were you.

Chris Gallivan (Tasktop-Planview)15:10:21

so much so, we have renamed TPO from Technical Product owner to Typing Pool Owner

Pete Nuwayser - IBM15:10:36

Opinions are free, "good takes" are a fee

Bryan Finster - Defense Unicorns (Speaker)15:10:54

Nah, I open source anything I can.

👍 1
🆒 1
👏 2
Charlie Betz15:10:25

@jtf here is a quote from JP Morgan Chase

👍 3
Brian Gallop15:10:07

@charlie TRUTH!

Jeffrey Fredrick, Author-Agile Conversations15:10:19

@char: I complete agree it happens. I think it can be a problem to move from engineer --> product manager for many different reasons. I’ve also seen problems for people moving from business analyst --> product manager or scrum manager --> product manager. I’m not clear there’s anything uniquely challenging about engineer --> product manager. When I’ve seen problems in transition the biggest issue is the lack of mentoring in the transition and/or a weak (or unhealthy) product management culture in the organization. Might that apply in these cases?

👍 2
Jennifer Miles15:10:49

One of the issues I have seen with engineering moving to product is that they sometimes can have a hard time not solutioning.

Chris Gallivan (Tasktop-Planview)15:10:53

A lot of the people I have seen in these roles are long term employees who used to code. Over the years they have built up a lot of domain knowledge, more so than on the business side

Scott Jaffa (Principal Engineer, ValidaTek)15:10:39

Is there any effect on how the org is structured? Ie. is someone moving from engineering to product because that’s where growth is, vs. because they specifically want to be in product? Curious if organizational dynamics are a major driver of seeing that transition succeed.

Chris Gallivan (Tasktop-Planview)15:10:46

That was in an org that did a lot of outsourcing

Jeffrey Fredrick, Author-Agile Conversations15:10:56

That failure mode for engineers makes sense to me @char. One thought that came to mind for me is that we try to have our engineers oriented to client needs in any case; the product owner acts as a customer proxy and tries to talk in client language to the teams (and sometimes the engineers join client meetings). I suspect that might make an easier transition compared to an environment where the product owner acts as a translation layer and client language doesn’t make it into the engineering team.

❤️ 1
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:57

@scott.jaffa so far we have not had any engineers shift over to product management...

Geri O’Toole15:10:58

I plan to use customer empathy to begin to help these folks, since who we serve work daily in an environment that demands extreme safety. I just got here 3 weeks go, so thinking to start there.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:00

@jtf Having the engineers start paying attention to the user experience that their products provided has been key... having the former engineering team members be on call also helped them get religion (quickly!) over the importance of details

❤️ 4
👏 1
Charlie Betz15:10:10

@jtf These are all good hypotheses, reflective of growth mindset. I would also, as a counterbalance, inquire as to whether there is fixed mindset operating in a way that's not easily disrupted, even with the best training available.

✔️ 1
👍 1
Charlie Betz15:10:53

@denee.ferguson This is exactly what an SRE at a FAANG told me yesterday.

Craig Cook - IBM15:10:45

When IBM Marketplace start 5+ years ago, we adopted "3 in a box" model. Development Manager, Product Owner and Technical lead. We called it "intentional tension in the system". Those 3 had to agree on what to work on next. It worked really well.

❤️ 3
Geri O’Toole15:10:48

@char my group may benefit from the shift of fixed to growth, simply by nature of our business where we serve teams who's lives can be in jeopardy if we don't shift. thanks for the reminder of fixed v growth. that's a great way to bring in customer empathy with them

Charlie Betz15:10:03

@jtf I do believe that one of the major constraints in the whole transition to product-centric operating model is workforce lack of product managers. I mean, there is not even a well accepted educational/training pipeline. (Not to dismiss the efforts and valuable offerings of the boutique sector. But we need more scale IMO.)

Bryan Finster - Defense Unicorns (Speaker)15:10:13

How can the engineers become product experts? I was in distribution centers as often as possible.

👍 1
Scott Jaffa (Principal Engineer, ValidaTek)15:10:14

I’ve been sending all the engineers to the customer product demos, user trainings, etc. Too new to know if it is having success, but that team did just get an amazing kudos from one of the users in a training about the application quality…

👍 2
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:20

@bryan.finster486 We haven't tried cracking that nut yet... the people leader for the engineering teams is the product owner... so we focus heavily on enhancing the user centricity of the product owner. The engineers get there kinda by osmosis.... We do have tight relationships with other internal teams that help us understand specific use cases among our customer base (e.g., call center agents, traders, executives, etc.)

Jeffrey Fredrick, Author-Agile Conversations15:10:37

@char: reminds me of the early days in agile, where lack of people with experience in agile was a bottleneck to widespread adoption. There was much more demand in 2005 than there were people to satisfy it. It seems something similar is happening now with product management. Everyone can see this is a better way to operate, but precisely because it is new there’s a dearth of experienced practitioners. (Mind the Product is great, but only gets you so far.)

💯 2
Charlie Betz15:10:24

@denee.ferguson Are engineers reporting to the product managers in terms of career pathing etc?

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:40

@char Engineers report to product owner; product owners report to me. Product managers report into our Agile org, led by @jennifer.miles While product management could be a potential path for the engineers, I'm not seeing much interest from the engineers in making that move. Most want to remain deeply technical, while a few have aspirations to become product owners/people leaders.


Interesting take. Question - then the PdM and PO spend more time on people management activities than the "Product" management? cc - @denee.ferguson

Chris Gallivan (Tasktop-Planview)15:10:59

While they may not be plentiful, I have met some amazing developers who are passionate about customers, products and clean code

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:09

@lbmkrishna product managers are not necessarily people leaders. Product owners are. We have found that limiting number of direct reports a product owner has is key. For one of my teams, there are 8 engineers (soon to be more as we onboard new hires). One of the engineers that wanted to become a people leader will take over product owners responsibilities for one of the products this team owns, and will become the manager of 2-3 engineers on the team. This will reduce the direct report load on the current product owner for that team. Generally, 6-7 is the max number of direct reports a product owner should have in our org.

👍 1

Thank you @denee.ferguson - Thank you for providing the context, much appreciated.

Gene Kim, ITREV, Program Chair15:10:11

PS: something I learned earlier this year: load balancers and all those Nginx systems shield developers from having to implement all sorts of new endpoints and protocols, such as HTTP/2, QUIC, SSL, and all sorts of other things I’ve only heard of. I gained a whole new appreciation for all those “commodity” devices! (Without them, we’d have to change every app to do things like handle devices roaming between WiFi and cell connections!!)

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:16

Load balancers definitely make technical implementations simpler!!! Now if I could just get MACs to roam better!!!

Gene Kim, ITREV, Program Chair15:10:33

Ha!! When I learned what load balancers did for devs, my jaw dropped in awe. I had no idea!!!

Gene Kim, ITREV, Program Chair15:10:22

Did I get that right, @denee.ferguson?

Craig Cook - IBM15:10:09

oh, triggered... Days without critical incident is not a good metric.

Craig Cook - IBM15:10:04

This can lead to teams not reporting incidents.

👍 1
Javier Magaña - Walmart15:10:33

It depends on the culture. Needs to be a constant reminder for transparency, vulnerability and blamelessness

Jeffrey Fredrick, Author-Agile Conversations15:10:44

I disagree, I think it is a good metric… but only if you understand that good metrics allow you to ask questions. They can’t give you answers.

👍 2
Matt Wheeler15:10:49

Is there a metric that can't be used for bad reasons? good to be aware of the risks and build the right conversations around them.

Girija Rao15:10:55

We have blameless postmortems as part of our overall culture

❤️ 1
Javier Magaña - Walmart15:10:48

A metric by itself can definitely be abused. That is why it is important to identify multiple metrics and track them in tandem to be used as guard rails.

Jeffrey Fredrick, Author-Agile Conversations15:10:31

Is there a metric that can’t be used for bad reasons? < I have a friend who says “if it can’t be used for evil it isn’t a superpower.”

❤️ 2
Craig Cook - IBM15:10:56

Yes, it does depend on culture.

Nathan Kampwerth15:10:16

What were the new Infra Product teams oriented to? Was it more outcome focused or technology focused?

Girija Rao15:10:55

outcome based - oriented around the product evolution and customer experience

👏 1
Jennifer Miles15:10:28


Brian W. Spolarich - Cal Poly15:10:21

Girija - what are a couple of examples of team focuses?

Nathan Kampwerth15:10:50

@girija.rao - So many questions. Would that mean you had a branch focus team that would include WAN design as well? Would FW access requests be part of datacenter connectivity?

Girija Rao15:10:35

@brianspo for e.g. improving the wifi experience, understanding usage painpoints and wish lists through empath interviews and surveys, and determining technology enhancements to drive those outcomes

🙏 1
Girija Rao15:10:24

@nkampwerth we have a core security team and perimeter security team and each one focuses on the related use cases

Nicole Forsythe15:10:16

Can you say more about using velocity and story points - do you standardize how teams assign story points?

👏 1
Nicole Forsythe15:10:09

Would that invalidate how the velocity metric rolls up?

Jennifer Miles15:10:38

no, teams use similar methods to story size

Jennifer Miles15:10:45

the real benefit we found is measuring at the team level but we are able to roll up to see trends

John Awesome Rowe - Best Buy15:10:56

What is the purpose of your story pointing then? I've always stuck to the matra that story points mean absolutely nothing outside of a product team. Since the numbers are all made up and unique to the team, I use it to ensure consistency and predictability of the teams and use other, more real, metrics to see how teams are performing over time

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:03

@john.rowe It helps the team understand how much work they can take on each sprint... Each team knows how many story points they have been able to successfully deliver on average during a sprint.... We do include stories for things like training, vacation, PI Planning meetings etc so that our process is consistent from sprint to sprint. As I understand it (@jennifer.miles is the agile guru), story points isn't a metric that should be compared across teams.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:18

Q4 tends to be a lot lower... change freeze after black friday

👍 1
Marc Price - Nationwide Building Society (Speaker)15:10:26

so your days without incident metric in Q4 is not used? or do you track trends year on year?

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:43

@marc.price the recording was made in mid-September, so we didn't have full Q3 stats at that point, much less Q4. We do track trends year on year

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:26

@denee.ferguson I completely empathize with this challenge of having infra / network teams adopt Scrum and apply it to building out hardware and network configurations. Big aha moment for me when working with teams in my org. It was contextual by team, but we eventually de-emphasized the importance of Scrum ceremonies and transitioned more teams toward Scrumban or Kanban. Provided the benefits of lean/agile but without as much "overhead" that Scrum introduced.

👏 1
Nicole Forsythe15:10:33

Yeah scrum was never intended to be used in this way. We see parts of Scrum as a “starting point” less a “end state”

👍 1
Scott Jaffa (Principal Engineer, ValidaTek)15:10:17

Yeah, Kanban and flow at the speed of the work for infra / hardware. Physical constraints are better handled via kanban and blocker states and you move on when you can.

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:13

Regarding metrics for our teams, we paid less attention to velocity and more on lean metrics like Cumulative Flow Diagram (CFD), WIP and throughput. Flow Metrics (from @mik) would work here too. We also spent time in value stream mapping to identify quality issues (low % C/A) and delays (in scheduling or hand-offs). These were where we got the most "bang for buck" with our infra teams.

👏 2
❤️ 1
👍 1
Charlie Betz15:10:00

Scrum always raises a red flag for me with anything operational as it's not suited to interrupt-driven work.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:08

@char That was part of the square peg, round hole challenge for us... honestly I have found that using scrum ended up improving delivery (quantity, quality... still working on predictability). But we had to teach the engineers how to chunk up their work into bite-size pieces.... It used to be that they might have a single story that would hang out on their kanban board for months (something like "deploy X")... and there was no visibility into how that effort was going, where the team was getting stuck... We have found 2 week sprints to be best fit for my team... The 2 week interval allows us to incorporate unplanned work without disrupting the current sprint, but keep whoever is escalating the need to do X happy.

❤️ 1
Charlie Betz15:10:06

@denee.ferguson It sounds like you have some flexibility then with the Scrum masters accepting emergent tasks that aren't specified at the start of the sprint - some purists (not me)might raise an eyebrow :face_with_rolling_eyes:

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:28

@char yes. As much as I would like to say we will not take on stories mid-sprint, that's not our business reality. My team in particular isn't fully staffed (we're hiring!); hoping that as reqs are filled this will be less of a problem because we will be able to accomplish more each sprint with more engineers.

Brian Smith15:10:03

I am wondering if making more frequent changes in smaller batches reduce the number of severe incidents?

👍 3
Marc Price - Nationwide Building Society (Speaker)15:10:06

I guess it can cause more incidents if you try to change too frequently. faster doesn't always mean better

Scott Prugh (DOES Prog Committee)15:10:39

Smaller batches that reduce the blast area are key. In our loadbalancer space we reduced failure group size first by adding more load balancers enabled via automation.

👆 1
Scott Prugh (DOES Prog Committee)15:10:09

So then the failure of any change affected a much smaller area.

Scott Prugh (DOES Prog Committee)15:10:53

This also greatly helps scheduling dependencies since dependencies increase risk exponentially.

👍 1
Mark Peters15:10:20

Depends on how you define an incident. This is where the A/B and canary deployments can help greatly. Have lately been converted to feature flagging as a way to rapidly fix deployed incidents, as long as you minimize embedding

👍 1
Scott Prugh (DOES Prog Committee)15:10:07

@tiny.mpetersii Agreed. Feature flags are a great technique to 1) reduce blast area, 2) reduce dependencies and risk, 3) inject operational thinking upstream to dev(shift left on ops concerns)

👍 1
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:44

@brian.m.smith There's a balance... The smaller the number of devices in a window, the more windows you need. Since our windows are normally at night, wear and tear on the team is a consideration... So... I personally strive to do enough changes (pilots) that we have the kinks worked out... then ramp up the devices per window. Some of our technologies (e.g., SDWAN, Wireless) it is generally a big bang deployment... There are some ways of limiting the blast radius, but you can only do that so much. In those situations thorough lab testing and bug scrubs are key... Unfortunately, vendors tend to have a lot of bugs.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:09

@brian.m.smith frequency of execution is actually positive... we get to be a well-oiled machine.... but once we get to that point, we're finding that we need to reduce the total number of windows to say... upgrade all devices in the network... so we can get it done in a shorter time period.

Gene Kim, ITREV, Program Chair15:10:51

(I am always in awe of how in this domain, even the smallest changes can have catastrophic global impact: I.e., global outage. firewalls, core switches, etc.)

Dave Fugleberg15:10:55

as in, Facebook's BGP issue this week...

Nick Eggleston, IT Architect (DMs welcome)15:10:24

It’s amazing how one “little” routing change or firewall statement can have unintended and hard-to-backtrace consequences.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:33

yup....... not the first time I've seen BGP route change have significant impact. fortunately not caused by my teams 😉


Dynamic routing is a double edged sword. Once a dynamic routing change propagates, you can't just reverse it as quickly as a "no" statement.

Gene Kim, ITREV, Program Chair15:10:44

“the wrong JIRA structure”

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:56

@brian.m.smith The #devices in scope for a change isn't generally the driver of likelihood of incidents.. Attention to detail in testing, design, adherence to best practice.

👍 1
Raghu Tumuluri15:10:44

Great presentation. QQ please. How did you measure the business outcomes and NPS?

Gene Kim, ITREV, Program Chair15:10:31

A lively thread on the impact of projects like this on infrastructure teams…

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:45

@raghu.tumuluri614 we haven't measured NPS for our products.... we did try to go there for wireless, but honestly found too many cases were users complaints really involved things that were not wireless......

Raghu Tumuluri15:10:56

Thank you Denee.

Raghu Tumuluri15:10:08

did you try to get the pulse surveys at the end of every PI? just to get the business and technology to provide feedback and maybe help to drive NPS in a desired way?


There is a thread growing (a breakout QA thread full of gold)

Marc Price - Nationwide Building Society (Speaker)15:10:34

how did you role out your agile training? lead by internal champions?

Andrew Machen15:10:42

@jennifer.miles What is your approach for avoiding "ivory towers" when teams want to opt out of product model with less than complete data points as to why it won't work?

Jennifer Miles15:10:41

@andrew.machen continued exposure to the benefits of the model, basically a wear them down approach. We have one team who is still resistant but have adopted some of the processes. It is an iterative process with them.

Andrew Machen15:10:27

Our experience is similar. Certainly great metrics and more energy from engineers in making things better as the months and years roll by. Great talk from you and your team!

Gene Kim, ITREV, Program Chair15:10:44

> I’m incredibly pleased at the transformation we achieved with our product-oriented agile-driven restructuring - it enabled us to establish a unified mission and sense of identity, full visibility and prioritization of work, improved execution and delivery, and clear accountability internally and with our stakeholders. This structure also allowed us to easily incorporate several new functions over the past two years. It’s an ongoing journey as we continue to iterate upon this foundation to best meet the evolving needs of our dynamic organization and the services we provide. > Girija Rao Vice President > > The unification of efforts and ownership across the architecture, engineering, and operational aspects of product teams, in concert with the ability to effectively manage priorities has enabled us to transform our technical capabilities while maintaining stable business operations in a more focused and optimized manner. > Vince Gutosky > Senior Director &amp; Chief Network Architect

❤️ 1
Gene Kim, ITREV, Program Chair15:10:56

Thank you, @girija.rao @denee.ferguson @jennifer.miles!!!

upvotepartyparrot 1
Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:09

Thank you all for listening!!! We are hiring!!! look on the hiring tab for an opening on my team...

👍 2
👏 1
❤️ 1
Marc Price - Nationwide Building Society (Speaker)15:10:34

great session, thanks for sharing!

❤️ 1
Charlie Betz15:10:39

Very interesting! Exactly where I am focusing current research.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:08

@char happy to answer questions via linkedin

❤️ 1
Ann Perry - IT Revolution15:10:10

🌟 Up next, we welcome @angeldiazrodriguez and @sheilalodhia from Discover, sharing their presentation, How Discover Financial Services Puts Engineering “Craftsmanship” at the Center of Our Digital Transformation. Joining us for questions will be @kevinjosephallen 🌟

👏 1
👍 1
❤️ 1
Khan, Humayoun at TELUS15:10:00

I recall Capital One was or is a pioneer in SAFe implementation. Is CO still managing this framework @jennifer.miles

👀 1
☝️ 1
Jennifer Miles15:10:46

@humayoun.khan we still use SAFe although in the Network space we have adopted a hybrid approach at this point, using a product based teams in a SAFe like environment.

👍 1
Gene Kim, ITREV, Program Chair15:10:01

Up next: Dr. @angeldiaz and @sheilalodhia!

👏 3
❤️ 2

@genek I ❤️ Apple Pay.

👍 1
❤️ 2
Gene Kim, ITREV, Program Chair15:10:00

(Yes, me, too! I had shared my amazement of how Discover powers Apply Pay with several folks lately — several had said something like, “oh, yeah, I saw their name when I accepted the TOS when I set up my credit cards, and now I know why!“)

❤️ 2
Mark Peters15:10:29

Had a chat with Discover the other day, their fraud prevention folks didn't know about some of the changes to improve the system, like apple pay. So the multiple small charges in a day were getting flagged as fraud, and turned down, when marketing was advocating make those small charges for extra points

Mark Peters15:10:58

Runway stops when you take off... An 80k runway means more time on the ground

Angel Diaz15:10:45

It’s all about reaching escape velocity

👍 1
Roland Krocin15:10:22

@tiny.mpetersii Runway is a way for us to brand the transformation. There are different altitudes that we’re iterating through, but the core point is to get off the ground quickly and keep climbing consistently.

❤️ 3
👍 1
Mark Peters15:10:57

understood, too much Air Force time...

😁 1
Chris Gallivan (Tasktop-Planview)15:10:39

4 week...we always used 6 weeks

Chris Gallivan (Tasktop-Planview)15:10:39

4 week...we always used 6 weeks

Gene Kim, ITREV, Program Chair15:10:35

How did you converge upon 6 weeks?


Everything takes six weeks

Gene Kim, ITREV, Program Chair15:10:05

Nothing takes 6 weeks. 🙂 😆

Chris Gallivan (Tasktop-Planview)15:10:15

We worked with the authors of Creating Your Dojo: Upskill Your Organization for Digital Evolution - Joel Tosi and Dion Stewart who started Dojos at Target. They recommended 6 weeks,

Chris Gallivan (Tasktop-Planview)15:10:34

Over time we saw that it took about 6 weeks for learning to stick

👍 1
Chris Gallivan (Tasktop-Planview)15:10:49

assuming at least 30 hours together a week

Chris Gallivan (Tasktop-Planview)15:10:58

we experimented with other time frames

Chris Gallivan (Tasktop-Planview)15:10:46

based on a lot of data across numerous teams


sorry for formatting.

👍 1

let me edit, I'm new to this slack thing

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear15:10:10

I’d like to learn more about the concept of a Dojo, is that book the place to start?


week 1 - A man, a plan, a canal; Panama During week 1 you make and begin executing a plan Week 2 - No plan survives contact with the enemy If you are tracking to a two week sprint, you realize you didn't plan well enough, and your plan isn't executable. Week 3 - A plan for a real plan Take your lumps from failing during the first two weeks, Week 4 - Get the work done This is the week where the real work happens, because the team finally knows and understands which direction its going Week 5 - If you find yourself going through hell, keep going Week 6 - Celebrate

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:58

Same approach at Principal Financial Group (prior company). Six week dojo challenges. Influenced by talking with folks at Target and the Dojo Consortium.


@genek - everything* takes six weeks, you just need to scope to ensure that

Chris Gallivan (Tasktop-Planview)15:10:46

it doesnt start to hockey stick until week 4 or 5

Chris Gallivan (Tasktop-Planview)15:10:54

at least that is what i observed

🎉 1
Chris Gallivan (Tasktop-Planview)15:10:07

week 2 or 3 is often despair

❤️ 1

@chris.gallivan421 I've boiled it down to: week 1 - let's do this (excitement) week 2 - oh this is bad (overcommitment) week 3 - oh this is really bad (we have no idea what is going on week 4 - TADA! week 5 - I knew we were fine! week 6 - wrong rock

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:11

Also this: "It takes anywhere from 18 to 254 day... to form a new habit and an average of 66 days for the new behavior to become automatic."

Chris Gallivan (Tasktop-Planview)15:10:48

along with Ebbinghaus forgetting curve

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:50

5 days a week x 6 weeks is only 30 work days of habit forming. So even six weeks has a risk of teams leaving a dojo and new habits not sticking.

👏 2

Why 6 weeks? Why are 6 week boundaries attractive? Well because math. People are really bad at predicting the future, and therefore we have to develop means and mechanisms to understand how to break up time, which is a human construct as well. People have been dividing their time into smaller parts since the beginning, of well, time.


@mring - which is why quarters are useful - form - 6 weeks storm, 6 weeks storm/norm, 1 week retro, repeat

Chris Gallivan (Tasktop-Planview)15:10:40

I agree with @mring - ideally it would be more than that amount of time - that was our minimum

👏 2
Chris Gallivan (Tasktop-Planview)15:10:43

we also saw cases of unlearning, where 20 hours spent in dojos, 20 hours spent at desk unlearning

💯 2

and repetition

Chris Gallivan (Tasktop-Planview)15:10:37

I love this topic - I’ll be around anytime to discuss this more


#same - its my philosophical rule num (#) 2


esp in the gov.

Roland Krocin16:10:52

Discover’s dojos are typically 4 to 6 weeks, with a hands-on and interactive experience that covers real-life scenarios which you wouldn't get from classroom-based training. The time line comes from the selection of learning modules that has been crafted to meet the product team. Dojos give product teams a thorough understanding of core concepts while leveraging their existing product backlogs for pairing and upskilling, creating an experiential learning environment that accelerates adoption in the team's backlog immediately. Its the combination of workshops in the morning, and pairing in the afternoon. Old habits take time to break as they practice new patterns to form new habits.

Chris Gallivan (Tasktop-Planview)16:10:38

we tried 4 weeks, but we found teams felt more pressure to deliver something. 6 weeks gave them some more room to breathe and learn


I feel the same way with a 2 week sprint - lots of time spent churning, esp early on.

Chris Gallivan (Tasktop-Planview)16:10:49

software development is like repairing a tractor. sometimes you need to take a few bolts off to understand the problem

Gene Kim, ITREV, Program Chair15:10:37

I particularly enjoyed @bryan.finster486’s talk on weaponizing DORA metrics yesterday — was absolutely fascinating!

👏 2
Gene Kim, ITREV, Program Chair15:10:41

Stories from three teams across some of the most important business units at Discover: • Priya Gupta, Sr Manager, Customer & Account Data • Joe Mathew, Sr Manager, Line Increase Request • Lakshmi Rupanagunta, Manager, Authorized User

👍 1
Angel Diaz15:10:03

Curious as to how do other foster in-open source approach to Dojo’s

Virginia Laurenzano NSA/MARFORCYBER15:10:07

"coding in a closet" resonates with me

👍 2
Bryan Finster - Defense Unicorns (Speaker)15:10:16

@sheilalodhia we also put heavy focus on CI metrics at the WM Dojo. Code merge frequency and dev cycle time. Any thoughts?

Angel Diaz15:10:17

Hi - we do have a bunch of team metrics we use in addition to the ones we highlighted today. Code merge, refactor, re-use etc. included

👍 2

The metrics I presented were the minimums for all teams. The teams are free to measure with other metrics that help them solve their individual problems.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)15:10:06

@marc.price Domenica Degrandis (Tasktop) did some internal training for us; we also had product owner training and to some extent an internal agile coach.... I personally think we should have done more, ensuring more role specific discussions about how this would work on day in day out basis. Having an agile coach working directly with each team would have cut the learning curve substantially.

👍 4
Marc Price - Nationwide Building Society (Speaker)15:10:41

we have a central team of enablement specialists that help with training and helping teams adopt new ways of working, sounds like we are heading in the right direction, thank you for sharing

Bryan Finster - Defense Unicorns (Speaker)15:10:27

We started using this to horizontally scale the knowledge of the goals.

Bryan Finster - Defense Unicorns (Speaker)15:10:49

Trying to push knowledge to the edge.

👍 1
Gene Kim, ITREV, Program Chair15:10:52

Thank you for getting these testimonials from your teams, @angeldiaz — these were great stories from the actual teams solving their actual problems, as opposed to executives talking “Powerpoint to Powerpoint”. 🙂

❤️ 3
👍 2
Gene Kim, ITREV, Program Chair15:10:45

and thank you for those testimonials, @sheilalodhia!

👍 2
Luke Rettig - Target15:10:46

theme of this Summit: Connection > PtP

❤️ 4
Jon Tarrant15:10:56

PtP? PowerPoint or Point To Point?

Luke Rettig - Target15:10:07

powerpoint to powerpoint

👍 2

I love the Discover team presentation with all the micro interviews embedded in their talk - Love it cc - @angeldiaz @sheilalodhia

👍 4
❤️ 2
👏 1
Angel Diaz15:10:12

Thank you Our team is so excited and proud to share with everyone!

Roland Krocin15:10:25

We learn, we share! We get better together! That is a core Discover behavior!


Really appreciate your comment! Love sharing our journey.

Gene Kim, ITREV, Program Chair15:10:04

I love the notion of “what objective evidence is there that a certain person is actually good at X, Y, or Z.” 🙂

👍 2
John Allspaw15:10:35

If I may submit a patch, Gene: “what objective evidence is there that a certain person is perceived by others at being good actually good at X, Y, or Z.”

Nick Eggleston, IT Architect (DMs welcome)15:10:53

Those are two different but highly related questions

Gene Kim, ITREV, Program Chair16:10:20

😆 Nice. Just wanted to have a test case that detects when someone is not actually good at something. (Or anything. 🙂 😆

Nick Eggleston, IT Architect (DMs welcome)12:10:09

Exactly… dunning-Kruger effect is very real, and there are folks who are amazing at expressing confidence while actually creating problems due to lack of competence in ways much harder to detect quickly

Geri O’Toole15:10:02

Leaders and Product folks who get the chance to see and understand the value of dojos are the dream.

👍 2
❤️ 1
Olivier Jacques15:10:03

:thinking_face: I wonder if some here started to leverage Nicole Forsgren's (GitHub / Microsoft), Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler "SPACE" framework on measuring Developer Productivity. Including during and after Dojos.

Jon Tarrant15:10:31

Thanks for the reference. Looks very interesting.

👍 1
Mark Peters15:10:13

Converting innovation to invention - good thought - need to make sure not measured as innovation not resulting in invention is bad. 80% of new ideas tend to fail

👍 1
Gene Kim, ITREV, Program Chair15:10:40

Thank you so much, Dr. @angeldiaz and @sheilalodhia Lohdia!

❤️ 5
👍 2
👏 1
Topo pal15:10:54

Awesome @angeldiaz

❤️ 1
👍 2
Angel Diaz15:10:14

Thanks Topo!!

Swarup Panja15:10:56

I loved presentation from Discover team!!


Thank you for taking out time to listen our story!


Is the Discover Academy open to anyone.. It would be great to see ;o)

Roland Krocin15:10:34

Not yet, but stay tuned! ;)

👏 1
Catie Martin_Design Manager_BlueCrossBlueShield of SC15:10:01

Would love to hear more abut the academy!

👍 1
Roland Krocin15:10:09

You’ll see us here next year talking lots more about it. And we’ll be sure to engage with this community as we talk more about the Discover Tech Academy in the interim.

Ann Perry - IT Revolution15:10:02

:unicorn_face: Coming up next, our very own @genek and @steve773 with The Four Characteristics of Structure Needed to Get Great Dynamics bowtie

🙌 1
Gene Kim, ITREV, Program Chair15:10:04

Up next: Dr. @steve773 and, umm, me! 🙂

upvotepartyparrot 3
🎉 1
❤️ 1
Jennifer Collings15:10:04

Thank you that was great!

❤️ 1
Roland Krocin15:10:10

Fantastic presentation!

❤️ 1
Raghu Tumuluri15:10:37

great presentation. thank you.

Bryce Miller15:10:59

Really enjoyed the recent podcast episode you did together

❤️ 1
Steve Spear15:10:01

Hoping this next speaker is worth listening too… :-)

🙏 1
😄 6
Alyssa Lundgren - Centil - Product Owner15:10:20

YES!! Dr. @steve773 ! Get ready for a fun presentation, all!!

Gene Kim, ITREV, Program Chair15:10:21

This presentation represents how @steve773 and I are trying to prove that four characteristics of structure predict high- vs. low performance.

Ferrix Hovi - Head of DevOps - Siili15:10:39

I'm afraid that @steve773 is going to help us drink from a firehose again.

😄 6
🧑‍🚒 2
🌊 1
Craig Larsen - he/him - Solution Design Group Mpls15:10:51

Ooh, sounds like this will also be a good talk

Tashfeen Mahmood15:10:57

As my 4-year old says: This is going to be fun!


Second consecutive DOES without bow tie for Dr @steve773 The begining of a new era} 😂

bowtie 4
😭 3
Steve Spear15:10:31

Blame that on covid. I’m lucky if I remember to put on big boy pants.

Klaudia Breslavets - Vanguard15:10:48

@angeldiaz @sheilalodhia thank you for sharing the Discover story! I've observed that learning culture is equal parts enablement with curated resources like internal academies but equally encouraging new behaviors for leaders & employees. Do you agree? What kinds of experiences has Discover had with the latter?

👍 3
Roland Krocin15:10:17

@kxbres agreed. It's the means and the ways. We equally coach on the adoption of the curated resources, alongside our core Discover behaviors that enable community, curiosity and innovation (maybe there’s an upcoming DOES talk on those ;)). Neither one can stand alone.


Great question! When changing how we work we tend to focus on the people within the teams and the steps they need to take….and never get to the leaders. This time as we rolled out Runway- we started with leaders in making the changes to the product model to seed the ‘why’ and problem we are solving mindset. Additionally, DTA has created learning journeys for leaders to help them understand the concepts the teams are learning as they get up skilled on DEVOPS.

Gene Kim, ITREV, Program Chair15:10:49

> What are the structures and dynamics necessary to unleash the distributed and collective human creativity and potential to compete and win, in an age being tumultuously disrupted by scientific and technological innovation, market transformations, and political and societal realignments. > > How and why over the last 150 years are some organization able to generate and deliver better ideas, quicker, faster, and more reliably. > > How do they create this magical dynamic that high-performing organizations use to unleash and empower everyone’s innate human creativity and intelligence to advance business and societal needs.

👍 1
Gene Kim, ITREV, Program Chair15:10:42

This is the classic Dev vs. Ops dynamic. But it’s also Merchant vs. Ops dynamic described yesterday by @lucas.rettig. And for that matter, Team of Teams.

Gene Kim, ITREV, Program Chair15:10:08

Did I get that right, @lucas.rettig Merchants vs. ___ ???

👍 1

These two model of communication in the organization make me think of an email from Elon Musk to tesla,..

Nick Eggleston, IT Architect (DMs welcome)15:10:27

Wow this Intro is so cognitively dense!! @genek

jeff.gallimore (CTIO - Excella, he/him)15:10:30

configure vs run for the role of the leader. that’s a powerful lens.

Gene Kim, ITREV, Program Chair15:10:23

The suggestion is that the leader can look at the structure of the system, simulate it in their head, and predict how it will behave. LIke how @jtf looked at MIT Beer Game, and immediately said, “That’ll never work! One way communications, slow feedback!”

🍻 3
🍞 1
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:45

@genek you reference the MIT beer game in your podcast and elsewhere frequently. Do you have a good source you like to reference for those who are new to the concept or want to see it in action?

Gene Kim, ITREV, Program Chair15:10:31

email me at <|> — I’ll send you some show notes in the Idealcast (or just look there).

Gene Kim, ITREV, Program Chair15:10:04

“queues” – what @scott.prugh hates most.

Steve Spear15:10:03

Who feels like that dude in the middle of the system? ideas, requests, inputs coming in from everywhere and outputs expected everywhere else. Holy 😱 Batman.

👍 1
🙌 1
☝️ 5
🙈 2

Can’t find vishnu or ganesha multiarms emoji 😆

Gene Kim, ITREV, Program Chair15:10:09

This is what was described in an Idealcast episode on modularity — the Eppinger Design Structure Matrix, where every node is connected to every other node. A completely full adjacency matrix.

👍 3
jeff.gallimore (CTIO - Excella, he/him)15:10:46

@steve773 @genek simplification, standardization, stabilization, synchronization — is it more useful to start with one of these? or treat them as all inter-related and use them all at once?

Gene Kim, ITREV, Program Chair15:10:04

In a non-simple system, imagine every box full.

Ferrix Hovi - Head of DevOps - Siili15:10:11

The dude in the middle is not feeling Focus, Flow and Joy, I guess.

😄 1
😩 2
Steve Spear15:10:26

Thanks @jeff.gallimore In that order of simplify, standardize, stabilize, and synchronize.

👍 1
Daniel Cahill - Engineer - Ontario Systems15:10:35

I relate a lot to that middle guy and this is really helpful.

Geri O’Toole15:10:37

"liberate brain space for doing good work."

Steve Spear15:10:59

@jeff.gallimore by creating a simpler architecture, there are fewer standards to even create.

❤️ 2
Jeffrey Fredrick, Author-Agile Conversations15:10:10

here we are with standardized work again. 🙂

Nick Eggleston, IT Architect (DMs welcome)15:10:12

Cognitive load is quite a limiting factor for cognitive beings

👏 1
🙌 2
🙏 1
Ferrix Hovi - Head of DevOps - Siili15:10:26

I am very cognitive, it seems.

Ferrix Hovi - Head of DevOps - Siili15:10:22

Last year we were talking a lot about cape fatigue that superheroes get. I think the social structures and setting them into a more working condition is not simple enough. So, the people transforming burn out.

👏 1
Steve Spear15:10:25

@jeff.gallimore without standards, it’s harder to recognize that you even have an aberration that demands stabilization.

jeff.gallimore (CTIO - Excella, he/him)15:10:58

i’m hearing echoes of shewhart and statistical control…

Jeffrey Fredrick, Author-Agile Conversations15:10:55

and normalization of deviance is much easier when there isn’t explicit standards

🤯 2
Jennifer Velasquez15:10:30

Standard…. “wisdom of the ages”!

Gene Kim, ITREV, Program Chair15:10:49

“how does the new person do their job, without any guidance on what is best known way to do that work?” (Reminds me of Kelsey Hightower story of being put on pager duty for an app that he had no idea what it was, how it ran, no access to repo, etc. So familiar to so many Ops people.)

Catie Martin_Design Manager_BlueCrossBlueShield of SC15:10:31

We are having this conversation in my team do provide guidance, encourage autonomy and enable creativity?

Dave Burnison15:10:13

I prefer the term "Guidelines" vs. "Standard". It implies more flexibility to adapt to your team/situation vs. "One size fits all"


For me @steve773 talk is like - "You need to be this tall to ride" - He challenges the conventional thinking - Our Enterprise leaders to take a course with Steve 🙂

👍 2
Dipesh Bhatia15:10:36

So we saw a huge social glitch this week 🙂

😂 1
Steve Spear15:10:36

@jtf to argue against standards is to argue against choreography or musical scores. i.e., life will always be like going to a 3rd grade musical all the time every where.


Communities of Practice?

Ganga Narayanan15:10:00

Sometimes standardization tends to get frowned upon in the agile community. "We're agile, we don't do standards" 🙂



👏 2
Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere15:10:25

Counter-argument. "Here are all of the Agile Best Practices." #maximumPossibleCompliance

Steve Spear15:10:26

@ganga.narayanan I cook. I don’t follow recipes. 😀

😃 2
🙌 1
Gene Kim, ITREV, Program Chair15:10:35

When problems can’t be contained and solved locally

Nicole Forsythe15:10:36

@steve773 What are the smiley face nodes on your diagram?

Nicole Forsythe15:10:58

People, teams, systems?

Steve Spear15:10:28

people getting increasingly aggravated…

👏 1
Nicole Forsythe15:10:13

I get the emotion piece, think the visual was making me think about org structure

Steve Spear15:10:54

@jeff.thomas communities of practice --> wisdom of the ages.

👏 1
Craig Cook - IBM15:10:11

Stabilization -> Coaching

❤️ 2
👍 1
Steve Spear15:10:42

coaching and aid.

❤️ 2
👍 1
John Allspaw15:10:17

Thinking about a distinction between a procedure and a standard in this context.

❤️ 2
Jeffrey Fredrick, Author-Agile Conversations15:10:45

stabilization explains the electron andon cord from the mining truck repair center earlier.

Ferrix Hovi - Head of DevOps - Siili15:10:45

So, the system needs to be simple, standardised, stable, synchronized... So creating that system sounds like not in the obvious domain.

💯 1
Nicole Forsythe15:10:41

Probably why his language is all about “complexity”

Jeffrey Fredrick, Author-Agile Conversations15:10:57

without standardization people can still do Simple domain badly. 🙂

Nicole Forsythe15:10:14

I think Snowden has changed the names - Simple is now Clear and Disorder is now confused

👍 1
Jon Smart [Sooner Safer Happier]15:10:27

The approach to the work needs to match the domain of the work. Which itself is not static.

❤️ 2
Ganga Narayanan15:10:52

Reminds me of Jon Smart's BVSSH: "Software is an agile created box on a lean conveyor belt." P32 🙂

👍 2
Jon Smart [Sooner Safer Happier]15:10:17

Yes! Agile and Lean. Complex & Complicated.

❤️ 1
👍 3
Gene Kim, ITREV, Program Chair15:10:30

I think the claim would be the application of these 4 characteristics are would be a complex, adaptive system — a learning organization that can operate in a CAS.

Kurt A, Blameless, SRE Architect15:10:19

When attuned to a dynamic environment rather than frozen in concrete

Mark Peters16:10:32

Want a complicated, adaptive system. A complex adaptive system means you don't always know what strings will be pulled when adaptation happens . Goal with observability is to move from complex to complicated so fixes can happen. I always like the ball of yarn analogy, complicated means you can pull a string and trace it but difficult, complex means the strings don't have a forecastable result.😀

Kurt A, Blameless, SRE Architect16:10:26

Yes, I understand the desire, but VUCA environments tend not to cooperate with such desires. Ref the situation in Team of Teams for example.

Mark Peters16:10:13

Yep, for sure, but everyone argues for complex and adaptabilitiy and I'm not sure that is always achievable. Used to work with intelligence in the Air force so one of the keys is getting those VUCA areas into the funnel of understandable possibility.

Steve Spear15:10:03

@ferrix “simple” in terms of the degree of higher order interactions. Simple is 1st and 2nd, complex gets you into 3rd 4th 5th … nth order systems.

Ferrix Hovi - Head of DevOps - Siili15:10:56

I think the model of operation needs to be as simple as it can be not to add to the complexity of the surrounding CAS. That takes cognitive energy to simplify the model.

Brian W. Spolarich - Cal Poly15:10:57

Is this not Radical Delegation?

Gene Kim, ITREV, Program Chair15:10:08

In our previous talks, we divide work domains into: • Slow: cognitive, creative, planning, organizing, assessing • Fast: operations, muscle memory, flow, practiced routines In tech, teams love to work in Fast mode, in Flow — and with absence of right strutures, teams are forced to work in Slow mode (meetings, escalations, etc.)

❤️ 8
Daniel Cahill - Engineer - Ontario Systems18:10:15

Rewatching during the break and I'm detecting some Kahneman influence. 😄 More emphasis right now on Thinking Fast and Slow than the Noise research. Now, I'm trying to get my head around what it means to be locally Fast. I was in a meeting this morning where given the same data, I can use Fast thinking but it would be Slow thinking for someone else on the same team. I think this means I need to make that so clear within my "partition" that someone higher in the organization receiving the input can reduce their noise and confusion about decision making.

Kurt A, Blameless, SRE Architect21:10:23

That's part of the aspect of expertise (in a Gary Klein sense) - moving explicit conscious thought patterns to a level of instinctual response.

Steve Spear15:10:23

Ditto to @genek point above to @ferrix picture. We’re building complex adaptive systems. We increase the adaptability by protecting the intellectual horsepower to do useful problem solving by the architectural simplification and the dynamic stabilization.

❤️ 3
👍 2
Gene Kim, ITREV, Program Chair15:10:31

We assert: leaders should be primarily in Slow mode; teams should be in Fast mode.

😬 1
❤️ 3
🍻 1
Kurt A, Blameless, SRE Architect15:10:58

That certainly fits with the paradigm of IC <--> manager being a state change, not a promotion per se

Nicole Forsythe15:10:52

I’m thinking of the old Starfish and the Spider - what about design sprints at the team level? Why can’t teams be in slow mode to discover directly from their customers?

Ryan Taylor - Senior Geospatial Developer, GISinc15:10:47

Does this go against any of the science of motivation by Dan Pink, that is autonomy, mastery, and purpose? It seems like fast mode could remove some sense of autonomy?

jeff.gallimore (CTIO - Excella, he/him)15:10:49

i believe deming would agree with you. from “out of the crisis”: > “The workers are handicapped by the system, and the system belongs to the management.”

jeff.gallimore (CTIO - Excella, he/him)15:10:33

there is working on the system (slow thinking) and working in the system (fast thinking).

👍 1
Nicole Forsythe15:10:21

Don’t we want product teams to engage their cognition, creativity, planning, organizing, and assessing their work?

👆 1
Nicole Forsythe15:10:50

@genek just didn’t want this thread to get lost

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere16:10:27

I like the debate going on in this thread and the nuance about Improvements in the Daily Work (slow) vs. the Daily Work (fast). Also about the difference between Product Discovery and Product Delivery. This is where I'd love to see more discussion and debate between the DevOps and Product communities about some of these concepts. Product Discovery folks are trying to figure out how to do Discovery faster and validate good ideas from bad as early as possible. But discovery work itself feels like it might fall in the "Slow" domain.

💯 2
Nicole Forsythe18:10:15

Exactly. 100% I’d love to hear @genek or @steve773 join in on that, and perhaps it is my startup days and product bias here.

Steve Spear18:10:13

@mring What’d I’d offer on this is fast feedback to inform slow (deliberative) thinking. Worst is slow feedback that triggers rushed thinking (because of little lead time) that is fast (habits and bias because no time to actually be deliberative). @nicole.forsythe

thankyou 2
💯 1
Steve Spear18:10:56

@mring @nicole.forsythe If this is something you’d have time to pick up, would be delighted to schedule a time. This has become a BIG deal in our discussions with @genek

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere19:10:36

@steve773 would absolutely be willing to chat more on this topic. I'll DM you my contact info and we can go from there.

Steve Spear10:10:01

@mring @annp @nicole.forsythe @genek Maybe we can find a time to speak once Gene and Ann have had chance to enjoy their Miller Time. over the weekend. Steve

🍻 2
Daniel Cahill - Engineer - Ontario Systems20:10:36

I'd love to be a fly on the wall in that discussion when that gets scheduled!


I'm definitely slow ;o)

👍 1
Ferrix Hovi - Head of DevOps - Siili15:10:57

When thinking the meaning of words agile, lean and fast, it does need structure. Without a bone structure a cheetah will be a blob of dotted soft tissue... not very fast 😄

Gene Kim, ITREV, Program Chair15:10:09

I think Dr. Cal Newport addresses this in his book “World Without Email” — he says leaders are not creating enough structure and process flows, forcing teams to live too much in Slack, email. (Haven’t finished it yet, but two people gushed endlessly about this book after I shared this concept with them.)

👍 1
❤️ 1
Nicole Forsythe15:10:35

Love Cal’s work! This is the only book of his I haven’t read.

Nicole Forsythe15:10:55

I coach my coaches to be (as Steve Martin said) “so good they can’t ignore you”

❤️ 1
Steve Spear15:10:10

How did @genek say in under 30 seconds what took me ten minutes. Damn!

❤️ 1
👏 2
Gene Kim, ITREV, Program Chair15:10:14

Example 1: Netflix and Chaos Monkey (GOOD)

Ferrix Hovi - Head of DevOps - Siili15:10:22

@genek That reminds me of the three-legged chair that a good leader stands on: one leg is personality that inspires, second leg is excel (or facts and numbers) and the third leg is structures that support reaching the goals with lower states of energy. Courtesy of a friend and former colleague.

👍 2
Nick Eggleston, IT Architect (DMs welcome)15:10:36

@genek love the netflix chaos example… do we have any speakers from Netflix?

Steve Spear15:10:58

…or facebook?… for the counter example?

😱 2
Kurt A, Blameless, SRE Architect15:10:27

They are FAANGs - not horses 😄

Javier Magaña - Walmart15:10:03

We had google a moment ago... :thinking_face:

Nick Eggleston, IT Architect (DMs welcome)15:10:04

So sad, so true. @jwillis was just talking about this in another channel

Gene Kim, ITREV, Program Chair15:10:49

Bookmark: several folks here are from FAANGs or FAANG-like. cc @annp

Nick Eggleston, IT Architect (DMs welcome)11:10:38

I specifically mentioned Netflix because “chaos monkey” and because Dave Hahn at least superficially saying they don’t do DevOps (

Gene Kim, ITREV, Program Chair15:10:53

Example 2: Bad Tightly Coupled Apps (LESS GOOD) — I so much appreciate in this community I can (mostly) get away with shorthands like Havens, Yakomin, things mentioned in Idealcast. 🙂

👍 2
Kurt A, Blameless, SRE Architect15:10:11

shorthand is a characteristic of a scenius

Jon Smart [Sooner Safer Happier]15:10:03

High cohesion, low coupling needed: interesting to see this example of tight coupling

Gene Kim, ITREV, Program Chair15:10:09

@christina_yakomin — you’re now shorthand of a bunch of concepts. 🙂

❤️ 1
Bryce Miller15:10:45

Loved that Scott Havens episode. Blew my mind several times.

Steve Spear15:10:14

There is no difference between a local problem and a system problem. The local will spread to become system and even if you catch the problem, you have to shut down the whole system anyway to fix it.

❤️ 2
Ferrix Hovi - Head of DevOps - Siili15:10:20

I've loved the stories by @steve773 but I think this is my personal favourite.

Gene Kim, ITREV, Program Chair15:10:37

Example 3: CI/CD pipelines (GOOD)

Steve Spear15:10:34

the technical system and the organizational system as an overlay…where I have latitude and where I have dependencies….

Glenn Wilson, Author of DevSecOps15:10:37

Stop the pipeline and swarm the problem = the famous Andon cord (I guess?)

👍 1
Jeremy Sechler15:10:38

@genek @steve773 Nice clean graphics useful for illustration. Typo for next time?

Steve Spear15:10:29

I’m using that slide tomorrow. Thanks for the catch!

Virginia Laurenzano NSA/MARFORCYBER15:10:02

this is what I love about DOES - we thank people for catching or sharing mistakes

👍 1
❤️ 3
Steve Spear16:10:10

@vmshook Well, I do my best to feed that dynamc…

Gene Kim, ITREV, Program Chair15:10:54

Example 4: terrible manual, infrequent deployments (NOT GOOD)

Gene Kim, ITREV, Program Chair15:10:26

“can’t undo the database schema change” (hahaha)

Jeffrey Fredrick, Author-Agile Conversations15:10:00

“tightly coupled, loosely controlled” < nice summary that can be used lots of places

👍 2
Ferrix Hovi - Head of DevOps - Siili15:10:41

I think that still beats tightly coupled and tightly controlled 😄

😄 1
Jon Smart [Sooner Safer Happier]15:10:18

Computer Science 101: High cohesion and low coupling. Bounded contexts 🙂

Ferrix Hovi - Head of DevOps - Siili16:10:07

@jonathansmart1 How about control in that? Maybe a lot? 😛

Jon Smart [Sooner Safer Happier]17:10:34

Low coupling requires federated, delegated controls. Control Objective is common and not implementation specific. How the Control is implemented can vary thousands of ways. And signal detection (which doesn't impede flow) to spot deviations from the Minimal Viable Compliance

👏 1
Gene Kim, ITREV, Program Chair15:10:01

in Phoenix Project: “we’re 1 hour into a 9 hour database conversion, that we thought would take 15m” 😆

😂 2
😄 2
😱 2
😀 1
Gene Kim, ITREV, Program Chair15:10:18

Example 5: Toyota Production System (GOOD)

Steve Spear15:10:20

the absence of agility even in the presence of functionality is a fatal flaw

❤️ 2
💡 1
Gene Kim, ITREV, Program Chair15:10:42

^^ @steve773 When cost of change of change is unacceptably high

👍 2
Geri O’Toole15:10:56

the absence of agility even in the presence of functionality.. @steve773 please repeat that?

Ferrix Hovi - Head of DevOps - Siili15:10:03

I think the Andon is rare. I have not seen zero bug policy in a lot of places or even a threshold for stopping and fixing.

❤️ 2
Adam Bestic16:10:27

I’ve seen the policy, but not the pysch safety to pull the cord

Ferrix Hovi - Head of DevOps - Siili16:10:12

Yeah, pull the cord to invite top level micro management.

Nick Eggleston, IT Architect (DMs welcome)16:10:44

Not even pharma has a zero bug policy :thinking_face: (and those bugs can result in serious harm)

Jeffrey Fredrick, Author-Agile Conversations15:10:04

Like to focus on the human element here. The relationship between people.

👍 1
🎉 3
Steve Spear15:10:45

@gerijotoole We can create the super duper greatest system that has fantastic functionality…today. Then, in the next moment, the environment changes. If the system is agile it can adapt and adjust and maintain its relevance. Without, it quickly loses its relevance because it’s less and less well tuned towards doing something useful.

👍 2
❤️ 5
Brian Smith16:10:18

I get the sense there are a bunch of anti- SAFE people here, so I might cause a firestorm, but my intent for asking this question is to gain understanding. Does the existance of Agile Train provide synchronization, by creating dependable sync points?

Nicole Forsythe16:10:03

Is it an agile train or a release train?

Nicole Forsythe16:10:45

I think it would depend on how resilient, learning, and “agile” your train is

Jon Smart [Sooner Safer Happier]16:10:58

I would say that yes it does. Where that is needed. i.e. there is a high degree of coupling. Ideally, people break the dependencies, have low coupling and don't need to synchronise teams, products and releeases

😍 1
Jon Smart [Sooner Safer Happier]16:10:52

e.g. it might be a good pattern for building a satellite. It is quite likely not a good pattern for a large organisation with thousands of unique contexts

👍 1
Brian Smith16:10:48

@nicole.forsythe, I was referring to the Agile Release Train.

Jon Smart [Sooner Safer Happier]16:10:59

#allframeworks not #noframeworks. If it works in context, use it. And treat it as a departure point, not a destination

👏 1
Brian Smith16:10:11

What does the reference #allframeworks not #noframeworks mean?

Nicole Forsythe16:10:15

Yeah, I think that the word “release” is the key word there…in my experience the process and structure of releasing IS the problem, and like Smart says, “Ideally, we break the dependencies,” if the train is the endpoint, it’s not enough.

Jon Smart [Sooner Safer Happier]16:10:45

i.e. use any framework you want to, if it helps and if it suits context. Not a case of "all frameworks are bad"

Jon Smart [Sooner Safer Happier]16:10:02

Also not a case of "one size fits all"

Brian Smith16:10:57

oh, okay. Where I really need to grow is to understand the conditions when SAFe is good for an organization and when it might not be good. As person who wants to help AF orgs grow in their lean-agile transformations, I want to be able to better advise orgs in their next step, not just sell them training.

💯 1
Brian Smith16:10:55

I want to read more and more of the "Heart", "DNA", "Source" info, so that I can help people solve their transformation problems.

Brian Smith16:10:26

@nicole.forsythe, your book is on my Kindle and Audible. Thanks for the research based help. I am going to read through it a third time. @jonathansmart1, would your book help in this journey.

Jon Smart [Sooner Safer Happier]16:10:48

@brian.m.smith yes. Generally, it's a collection of patterns and antipatterns. In chapter 3, specifically, there is a comparison of frameworks

Nicole Forsythe18:10:15

Brian I’m not Nicole Forsgren, tho I do admire her work. I’m unpublished, so take it all for what you paid for it! (Forsythe and Forsgren are VERY close.)

Gene Kim, ITREV, Program Chair16:10:19

“centralized control system could not keep up with the pace of reality” — consequence is that entire system eventually shuts down.

👍 4
Steve Spear16:10:26

The perversion that is the characteristic of such management systems…

Geri O’Toole16:10:42

I wrote it down. lol

Larry Davis16:10:15

"Disrespectful and perverse all at the same time."

Craig Cook - IBM16:10:59

Has anyone experimented with the Haier org model?

Nick Eggleston, IT Architect (DMs welcome)16:10:17

Centralized systems can not keep up with the pace of the real world

👍 1
Gene Kim, ITREV, Program Chair16:10:53

I had to cut these slides:

👏 2
Jeremy Sechler16:10:14

Was wondering about that for centralization

Ferrix Hovi - Head of DevOps - Siili16:10:02

The good part of Soviet planned economy in 5-year terms would be that we could have been fine with 5 years of Twitter and then none.

😄 1
👍 1
Virginia Laurenzano NSA/MARFORCYBER16:10:48

"why nations fail" is a great book about those two slides, @genek

thankyou 2
🔖 1
👍 2
Andy Nortrup - Director of PM at Tanium16:10:09

I love the idea of synchronizing on goals/outcomes rather than time (which is one of the first things I think about when I think of sync).

Gene Kim, ITREV, Program Chair16:10:04

everyone was trapped in their functional silo, unable to work with their peers in other functional silos — structure wouldn’t enable it.

Steve Spear16:10:33

@vmshook in terms of a society not tuned to getting feedback there’s the example of the Soviet Navy’s experience with reactor propulsion vs that of the USN. Attaching chapter 5 of my book which has case about Naval Reactors program.

thankyou 3
❤️ 5
jeff.gallimore (CTIO - Excella, he/him)16:10:56

all the chapters of his book are excellent.

Virginia Laurenzano NSA/MARFORCYBER16:10:00

adding it to my reading's always insane after DOES

Steve Spear16:10:42

Yes, let us know what you think!

Ganga Narayanan16:10:45

Aah the Facebook structure! 🙂 Reminds me of what just happened with Facebook!

Gene Kim, ITREV, Program Chair16:10:49

Thank you so much, @steve773!!!!

🙏 5
🙌 2
👏 1
Olivier Jacques16:10:18

What a great talk and exchange! Thank you.

Tashfeen Mahmood16:10:58

This is one of those presentations that I will need to listen to a few more times

👍 3
💯 3
Ferrix Hovi - Head of DevOps - Siili16:10:07

Okay. 5 minutes to cool the brain in a bucket.

😁 1
👍 2
Steve Spear16:10:09

@genek you are the man.

👏 2
Ganga Narayanan16:10:12

Thank you @steve773!

Steve Spear16:10:54

You bet. Let’s talk 1x1 about what resonated and why. Would love to hear what you’ve got in mind.

🙏 1
Ganga Narayanan16:10:56

Thank you so much!

Denver Martin, Dir DevSecOps, he/him16:10:29

Seems like this aligns with Dekker's thoughts on Safety Culture where you empower people to do the things you have hired them to do, without fear.

Scott Jaffa (Principal Engineer, ValidaTek)16:10:30

Have to say I’m loving the virtual format and live Q&A with the speakers.

👍 1
Frances Paulisch16:10:41

Thanks so much - great interaction to the common goal of a great presentation

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere16:10:41

Hey @jeff.gallimore - Nobody cuts @steve773 off! (Or maybe that was just my feed). 😆

👍 1
jeff.gallimore (CTIO - Excella, he/him)16:10:13

heaven forbid! let’s blame it on the tech 🙂

😆 1
Nick Eggleston, IT Architect (DMs welcome)16:10:49

My feed too! A very densely edited piece. Hard to pack so much into such a short time.

jeff.gallimore (CTIO - Excella, he/him)16:10:19

imagine all the great stuff they left on the cutting room floor!

Nick Eggleston, IT Architect (DMs welcome)11:10:27

Long version with Gene break-ins like Idealcast for a #watch-party win!

Tashfeen Mahmood16:10:52

Thank you, @genek

👏 1
Virginia Laurenzano NSA/MARFORCYBER16:10:58

I really love getting to see questions, answers and notes from others

❤️ 3
Nick Eggleston, IT Architect (DMs welcome)16:10:07

Transitions between talks are so fast!! My poor cognitive gets overload :rolling_on_the_floor_laughing:

Tashfeen Mahmood16:10:08

and Thank you @steve773 Great Talk!

Glenn Wilson, Author of DevSecOps16:10:08

I could listen to @genek and @steve773 all day. So many aha moments

💯 4
👏 7
Steve Spear16:10:39

Thanks Glenn. Would be delighted to touch base you me Gene to see what ideas resonated, why, etc.

👍 1
Nick Eggleston, IT Architect (DMs welcome)16:10:13

A new spin on all day devops. I’d love to join that!

👍 1
Glenn Wilson, Author of DevSecOps16:10:04

for sure. I face so many challenges with teams that don’t understand DevOps (and DevSecOps for that matter). Being able to give them real examples within a model / framework that shows what’s good and what’s bad is key.

Leah Brown - IT Revolution16:10:10

You can get the new paper @suzette.johnson5 is talking about in the Fall 2021 DevOps Enterprise Journal (sponsored by LaunchDarkly). It's free to read here:

🙏 1
Steve Spear16:10:46

@mr.denver.martin That’s right. In absence of safety, people can’t call out issues, so no stabilization, so problems persist and escape.

Steve Spear16:10:59

Rock on @jeff.gallimore.

🙌 1

Many thanks Dr @steve773 and @genek


@genek - Do we extract these slack conversation to some kind of system to produce some reference materials for our delegates? This is becoming high priority feature request for our conference now.

👍 2
Steve Spear16:10:34

@christian.lefevre You bet. Looking forward to learning what resonated and why and where. Let’s find time with Gene to talk with more bandwidth. Cheers!

Denver Martin, Dir DevSecOps, he/him16:10:45

I am looking at making Safety Culture one of the Keys of Dissertation on Implementation of DevOps in organizations, trying to narrow down my topic still. Open to input @steve773

Steve Spear16:10:18

Let’s find time to talk. Would be delighted to learn what you’ve got in mind. Steve

Denver Martin, Dir DevSecOps, he/him16:10:49

sure, I know you are very busy, I can work around your schedule... I am a First year of a 3 year program.. So I have idea but still have to do a lot of work.

Kurt A, Blameless, SRE Architect16:10:16

@mr.denver.martin are you doing the Lund program?

Denver Martin, Dir DevSecOps, he/him16:10:21

I am attending Crummer Graduate School, Rollins College. EDBA program..

Nick Eggleston, IT Architect (DMs welcome)16:10:43

Can u meet in Gather so interested folks can listen in?

Denver Martin, Dir DevSecOps, he/him16:10:46

I am open to meeting in Gather - I think we just have to work out a time to meet...

Drew Boyer16:10:22

@steve773 - your comment - its about connecting people not work centers REALLY resonated.

👍 3
Nick Eggleston, IT Architect (DMs welcome)16:10:47

@steve773 have u studied the incentives that seem to work against safety where decisions are made to leave defects in place because the cost of settling the lawsuits is less than cost to fix with respect to profits over the considered time period?

Kurt A, Blameless, SRE Architect16:10:43

That seems, in considering things from an SRE perspective, to be similar to the "appropriate level of reliability" - 100% is never the right level

Steve Spear16:10:03

@drew.boyer I do recommend a read of Taichi Ohno’s ’88 book on The Toyota Production System. One would think it’s about material through machines. It’s really a philosophical treatise on creating a positive environment for people.

🔖 2
thankyou 6
❤️ 1
👏 1
Steve Spear16:10:23

Thought this note would be of interest…vis a vis designing human oriented systems…

🔖 1
Nick Eggleston, IT Architect (DMs welcome)16:10:44

@ojacques2 are you collecting bookmarks and notes for this conference like the last one? (

🔖 1
Olivier Jacques16:10:16

I have not, no. But opened to get them!

Kurt A, Blameless, SRE Architect16:10:16

I've been marking citations with 🔖 if it will help


Reminder: The action has moved to the breakouts! Join the following channels to interact with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4

Joe Arrowood16:10:11

@sanjeev.sharma It is refreshing to hear someone else saying these things. Silos work to protect themselves way too well.

👍 2
Denver Martin, Dir DevSecOps, he/him16:10:42

@sanjeev.sharma and @john.comas I have worked with rolling out DevOps at both large and small companies, the hard part is finding the space to set things in motion... the way to find the space is different in both...

Joe Arrowood16:10:23

I have called it "Continuous Everything"

👏 2
Joe Arrowood16:10:15

@john.comas I have told my peers to use what we have before including some shiny new external tool. I so much agree with your statement on the presentation.

Denver Martin, Dir DevSecOps, he/him16:10:46

Yes, we often see it as having an operation while still running a marathon.

John Awesome Rowe - Best Buy16:10:43

I'm curious, what guardrails, if any, are there to prevent a team from driving up infrastructure costs? Are those costs reflected back somehow or is it kept in a centralized bucket of money?

David Roth16:10:50

Working on chargeback and show back models. Based on a person's role that makes the request

David Roth16:10:56

Good tagging strategies help enable it

Joe Arrowood16:10:21

@sanjeev.sharma I completely understand Test Data Management for Integration tests, Performance Tests, Smoke Tests, etc. But, can you explain the need for Unit Test Data Management? In my understanding, Unit Tests should not be updating data in the first place. Input data for Unit Tests facilitate testing the path through the code and the interaction between components.

David Roth16:10:08

Your understanding of unit tests matches mine. The unit tests by nature should be self encapsulated to the atomic units of function we are testing

👍 1
Sanjeev Sharma16:10:44

Agreed. If I said TDM for unit tests in the talk, I mis-spoke.

Joe Arrowood16:10:13

Whew! I just knew that I was missing something. Thank you...

Denver Martin, Dir DevSecOps, he/him16:10:33

@john.rowe we have used tagging in AWS and a tool call Cloud Custodian that will turn down systems based on tagging so they will not get left around...

👍 1
Ganga Narayanan16:10:18

Thank you @sanjeev.sharma @john.comas! Loved the presentation! btw about #3 - I found it a combination of mandates + "build it and they will come" .. very relatable!

👏 1
John Awesome Rowe - Best Buy16:10:33

Per your question on mandates, I've always been of the mind where you need to build something people want and find useful so you don't mandate it. Now, once you've proven out the value and shown success, you can talk about mandating. I never like to see mandates in absence of proven success though

❤️ 2
Sanjeev Sharma16:10:14

Good point and wholeheartedly agree

Charlie Betz17:10:58

Leading companies use an interesting blend of "choose to use" and peer pressure. "You're going to reinvent that? Really?"

Anil Atri16:10:47

Sanjeev, Have you invested in automating the Platform installation/setup using DevOps practices and have you introduced self service in this space yet.

Sanjeev Sharma16:10:13

Still early there @anil.atri. We are piloting a portal as we speak. And yes, we will deliver the platform as a private PaaS

Anil Atri16:10:58

Thanks Sanjeev.

Nicole Forsythe17:10:13

hey @jeremy.castle.lvh0 and @ryan.chambers.rse9 did you use the shopping/cashier analogy internally when you were pitching this work?

Jeremy Castle17:10:06

We kind of stumbled upon that later. I secretly funded the work within my suite first. This is how we are selling it to the org now.

❤️ 1
Kevin Carrier18:10:44

the it crowd haha!!

Anderson Tran18:10:48

This questions are so real, I feel it 😅

Eric Minick - CodeLogic19:10:35

I love that "team 3" who were DevOps tool pro's but didn't bother to give the Ops team a heads' up on a major release.

Dave Owings19:10:58

therapy, love it

Andrew Machen19:10:44

@rajat.sud Can you give us a high-level idea of what a typical "low score" leadership investigation looks like, especially in cases where course correction flies in the face of priorities?

Rajat Sud (DevOps Evangelist - SBPASC, an affiliate of CareFirst) (Speaker)19:10:21

We struggle with this - but that 'buy in' is important. Also, as the CoE, one of our most important mandate is to educate our Business Community on the value-add (be it CI/CD, test automation, Securitization) that DevSecOps adds to their product

Andrew Machen19:10:17

Asked another way - Is product ownership generally in favor of prioritizing these score corrections?

Randy Shoup21:10:01

Hi, all. Looking forward to presenting about eBay's Velocity Initiative in a few moments ...

👍 1

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser.

Ann Perry - IT Revolution21:10:22

Let's get ready to welcome @rshoup and @markwei from eBay, here to present Driving a Tech-led Reimagination of eBay Through DevOps

👏 3
🚀 1
👀 1
Mark Weinberg21:10:29

hi everyone. looking fwd to the session. exciting stuff happening at ebay.

👍 1
Courtney Kissler21:10:35

Extremely excited to hear this talk!!

❤️ 2
Gene Kim, ITREV, Program Chair21:10:11

Me, too, @chawklady!!!

🙌 1
Gene Kim, ITREV, Program Chair21:10:14

Please welcome the team from eBay! VP Engineering and Chief Architect @rshoup and VP of Core Product Engineering @markwei! They’re loading up the film into the projector — will be starting any second now! 🙂

Randy Shoup21:10:52

Still blushing from that intro, Gene 😉

Randy Shoup21:10:50

... and away we go

👏 1
Gene Kim, ITREV, Program Chair21:10:16

(If you don’t see this talk talk playing, please refresh your browser! 🙂

Alex Broderick-Forster, IT Revolution, Event Staff21:10:30

I’ll kick this off with a very important question. What’s the best strategy for sniping an auction on eBay at the very last second? Technologically-speaking, of course.

😆 2
Randy Shoup21:10:07

Set what you are willing to pay up front, and the system will 'snipe' for you up to your limit

Randy Shoup21:10:41

Everything we have done and learned, we learned from this community, and it's such an honor to present at my very favorite conference!

❤️ 5
👍 1
Gene Kim, ITREV, Program Chair21:10:08

eBay: $10.3B, 159MM active buyers; 19MM sellers, 13K employees!

Gene Kim, ITREV, Program Chair21:10:54

“together, we can help teams unblock each other, permit teams to take more risks than they otherwise would… and also set mandates, when that’s required.” 😆@rshoup

Gene Kim, ITREV, Program Chair21:10:08

(revealing the carrot and…)

⚔️ 1
Randy Shoup21:10:21

Value Stream Mapping was a tremendous tool in showing us the bottlenecks in the overall system. No substitute for seeing the "whole board"

🎉 6
Javier Magaña - Walmart21:10:08

How long did your VSM took? how big was it? Any particular challenges?

Randy Shoup21:10:06

Quick and targeted with 3 individual teams as a cross-section. We already started hearing the same things.

Chris Gallivan (Tasktop-Planview)21:10:11

what percent of the develop was wait?

Randy Shoup21:10:30

Don't know that yet. Stay tuned as we deploy Flow Framework.

Chris Gallivan (Tasktop-Planview)21:10:10

I have seen many cases where crazy percentages are spent in wait >50%

☝️ 1
Randy Shoup21:10:16

So many issues across every phase of the product lifecycle. Can't do it all at once. Where to start?

Mark Weinberg21:10:31

find the bottlenecks and break them. everything gets better when u do that.

Gene Kim, ITREV, Program Chair21:10:53

I loved hearing hints about the stack at eBay — I meant to ask, what are the most used languages used these days at eBay these days? (And I loved hearing about the improvements in build and startup times!) @rshoup

Randy Shoup21:10:48

Mostly Java, some node, a smattering of others

Gene Kim, ITREV, Program Chair21:10:32

The improvements in startup times must have been so appreciated and exhilerating!

Mark Weinberg21:10:42

swift and kotlin on mobile (but still some objective c and java as well)

Randy Shoup22:10:33

The dev teams are particularly excited about the build time improvements, so we rapidly went from pilot to full rollout everywhre.

Jeffrey Fredrick, Author-Agile Conversations21:10:04

“dysfunctional experimentation” < it wasn’t for lack of tools. I’m guessing lack of mindset?

Randy Shoup21:10:20

all of the above

Mark Weinberg21:10:27

huge dependencies among teams. when teams r stuck in wait states, what do they do… find something else to work on. that creates more wip. more wip == bad.

😫 1
💯 2
Jeffrey Fredrick, Author-Agile Conversations21:10:38

DORA metrics — drink!

🍻 3
👍 1
Gene Kim, ITREV, Program Chair21:10:50

“we are focusing the the delivery pipeline, because if we improve this, everything else becomes easier.” (Seriously, I love how @rshoup talks about thinking in systems!)

❤️ 1
Randy Shoup21:10:57

Theory of Constraints, baby! Any improvement not at the bottleneck is an illusion, as someone said (@genek)

👏 2
Gene Kim, ITREV, Program Chair21:10:16

“10% of eBay applications are in our pilot; focusing on short-term wins and long-term capabilities” @markwei

Randy Shoup21:10:51

We kept asking ourselves how we could take a massive problem and make it a targeted, actionable problem

👍 2
Jon Smart [Sooner Safer Happier]21:10:52

Think Big, Start Small, Learn Fast: nice

❤️ 5
Mark Weinberg21:10:55

easy to get paralyzed by the number of problems. just start.

💯 1
Jon Smart [Sooner Safer Happier]21:10:27

A bias to thoughtful action

Jon Smart [Sooner Safer Happier]21:10:25

The 6 words above (TB, SS, LF) are words I use several times a week 🙂

Gene Kim, ITREV, Program Chair21:10:02

@rshoup That was definitely from Dr. Eliyahu Goldratt, but that quote was surprisingly hard to find: it was in the Beyond The Goal audiobook.

🔖 1
Randy Shoup21:10:15

... as quoted by ...

Steve Farley - VP Infrastructure Operations, Nationwide Insurance21:10:43

I like hearing about the architecture community taking a lead role in driving Developer experience and velocity. Great broader conversation of the role or architecture as a catalyst for DevOps and developer best practices.

Randy Shoup21:10:07

As Chief Architect, that's my bailiwick. And also, I had to recognize that unblocking software delivery was the only way we were going to get to architecture improvements. How can we change the architecture if we can't deploy code reliably?

👍 1
Steve Farley - VP Infrastructure Operations, Nationwide Insurance21:10:35

I think that is great alignment....I need to take that angle at Nationwide with my architecture friends

Gene Kim, ITREV, Program Chair21:10:45

Wow. @markwei are you saying that in payments, it’s actually even more important to deploy quickly? (e.g., respond to fraud, etc?)

Randy Shoup21:10:14

balancing metrics 🙂

Mark Weinberg21:10:05

very important to deploy fast here and with small changes.

Gene Kim, ITREV, Program Chair21:10:38

I’ve never heard that before — that’s so cool. Usually I hear, “it’s PAYMENTS. We have to be super careful!”

Mark Weinberg21:10:07

ha ha… more frequent deployments with smaller batch sizes == “super careful”

🎉 1
Daniel Cahill - Engineer - Ontario Systems21:10:28

Ha! That "Its payments!!" is a daily conversation I have :)

Randy Shoup21:10:58

Similar to @bryan.finster486’s talk yesterday, asking "why can't we deploy today", we kept asking the same: if we were to ask you to deploy every day, why wouldn't that work. Now we have a list of impediments to remove ...

👏 3
👍 3
upvotepartyparrot 1
Bryan Finster - Defense Unicorns (Speaker)21:10:45

Used that on a team today. Found out they didn’t know about feature flags.

Jeffrey Fredrick, Author-Agile Conversations21:10:15

unblocking dependencies < I love to hear those words

💯 2
❤️ 1
Randy Shoup21:10:45

We say "removing impediments"; potayto, potahto

Bryan Finster - Defense Unicorns (Speaker)21:10:15

You mean they aren’t goals? 👏

Randy Shoup21:10:27

Measuring the DORA metrics was a huge unlock

Randy Shoup21:10:34

They are goals, but I hadn't seen your talk yet 😉

Bryan Finster - Defense Unicorns (Speaker)21:10:21

Gamification really helps, if the culture is good.

Randy Shoup21:10:52

Want to learn more about that. Your talk was great.

Bryan Finster - Defense Unicorns (Speaker)21:10:22

My pal @justin124 joined you guys. Awesome guy. My last roll at Walmart was leading the DevOps Dojo. It sounds like ya’ll are trying something similar. I’ve a bunch of learnings from that if you’re interested.

Justin Abrahms (eBay)21:10:11

We're interested. 🙂

Bryan Finster - Defense Unicorns (Speaker)21:10:55


Bryan Finster - Defense Unicorns (Speaker)21:10:57

There’s something I did at Walmart to help spread the knowledge of the goals in a scaleable way. Happy to talk about that strategy as well.

Gene Kim, ITREV, Program Chair21:10:48

“how long to deploy to first and last machine”;

Gene Kim, ITREV, Program Chair21:10:07

“P95 build times; P95 startup times”

Jon Smart [Sooner Safer Happier]21:10:24

Is there a measure of colleague engagement too? (Happier)

Jon Smart [Sooner Safer Happier]21:10:59

IMO, this is an important gap in the DORA metrics.

Randy Shoup21:10:27

The impediments the product engineering teams identified gave us the roadmap for the platform and infra teams

👍 1
Jeffrey Fredrick, Author-Agile Conversations21:10:34

“remove impediments to flow” : :thumbsup:

💯 1
❤️ 1
Philip Day21:10:26

What's the difference between 'Do' and 'Act'?

Philip Day21:10:49

Would 'Observe, Think, Act' be a better cycle?

Randy Shoup22:10:51

Do == experiment; Act == scale it everywhere

👍 1
Gene Kim, ITREV, Program Chair21:10:15

BTW, listening to NPR Morning Marketplace, I now always hear ads from eBay Technology, recruiting engineers. Wild.

Mark Weinberg21:10:15

this has created a great internal “velocity community”

jeff.gallimore (CTIO - Excella, he/him)21:10:29

i love the approach of getting explicit about what exactly is in the way.

👍 1
Randy Shoup21:10:35

Training and education are key to this. Most of the engineers have never experienced a CD situation.

❤️ 2
Randy Shoup21:10:04

This really got the attention of the execs -- we can double the productivity of the existing teams with no new resources

💰 1
Justin Abrahms (eBay)21:10:08

eBay is an interesting place. I've never been somewhere with people who've been at the company for 10-20 years. And there's a LOT of them.

Bryan Finster - Defense Unicorns (Speaker)21:10:24

Wait, didn’t you come from Walmart?

Justin Abrahms (eBay)21:10:50

Yeah, but there are still a lot more than I encountered at Walmart, but that might be west coast bias.

Bryan Finster - Defense Unicorns (Speaker)21:10:10

Yeah, I was a “kid” I was only there for 19 years.

Mark Weinberg21:10:20

doubling productivity! legit!

🎩 1
📈 1
Gene Kim, ITREV, Program Chair21:10:23

“the same teams are now delivering TWO TIMES MORE features and defect fixes”. Huge. What an amazing finding and conclusion! 🙏 🎉 Thx @rshoup

Dr. Ivan Kronkvist21:10:34

When you say you embbed sr architects, can you talk more abbout that?

Randy Shoup21:10:56

Several are here at the conference. We assign them to a team, and they figure out with the team how to help.

Randy Shoup21:10:04

@justin124 @dvancouvering

Justin Abrahms (eBay)21:10:08

I'm one of those! 🙂

Dr. Ivan Kronkvist21:10:38

Do you help with the coding or more drawing up the diagrams?

Dr. Ivan Kronkvist21:10:03

Maybe a mix if needed?

Justin Abrahms (eBay)21:10:04

I've embedded with our Payments/Identity group. I'm mostly operate in service to those teams to help them however they need. I also serve as a buzzing mosquito to remind them to push the envelope and "we'd like to get to a place where..." etc

👍 2
David Van Couvering - Senior Principal - eBay21:10:22

Haha I only do diagrams - kidding. Definitely involved in coding. It very quickly helps me see where the issues are, and helps me establish a trust relationship with the team

👍 2
Justin Abrahms (eBay)21:10:12

In the past 4 months, I've worked on: • hey, here's a more-efficient way to write integration tests • Our config is confusing and leads to errors. Here's a framework to test it. • We seem to have a gap around feature flagging. Let me go investigate how to fix that. • Why are we running sprints like this? Let's talk about that! etc

David Van Couvering - Senior Principal - eBay21:10:31

But I have also been involved with management and QE suggesting new approaches and pushing the envelope.

Dr. Ivan Kronkvist21:10:49

Great info. Thank you for sharing.

John Awesome Rowe - Best Buy21:10:02

This is really great, I'll go sign our architects up for a coding bootcamp 🙂

Dr. Ivan Kronkvist21:10:15

Are you assigned to one team at a time, or maybe supporting a few?

David Van Couvering - Senior Principal - eBay21:10:40

Here are some of the things I've helped with: • Teaching TDD and coding practices • How to improve/automate the checks we do as part of a deploy, such as localization and accessibility testing • Improving how we do code reviews • Introducing trunk-based development

👍 1
Justin Abrahms (eBay)21:10:55

There's someone who is a representative to the velocity program for Payment/Identity. He pointed me to a few teams who could use my help, so I'm there to help them.

David Van Couvering - Senior Principal - eBay21:10:56

I'm assigned to one team, but I also drop in to meetings with other teams when they have questions

Justin Abrahms (eBay)21:10:58

I work with 2 teams.

John Awesome Rowe - Best Buy21:10:14

That sounds like a role we're trying to create, we just call it a Principal Software Engineer

David Van Couvering - Senior Principal - eBay21:10:22

By "team" I actually mean I work with an overall group - I'm working with the View Item team

David Van Couvering - Senior Principal - eBay21:10:31

Yea, that's basically the same thing different name

Justin Abrahms (eBay)21:10:33

That's close to my title, John. 🙂

David Van Couvering - Senior Principal - eBay21:10:39

All of us architects/principals also meet multiple times a week and share what we're doing and discuss approaches

Dr. Ivan Kronkvist21:10:00

I like what I am hearing.

Jeffrey Fredrick, Author-Agile Conversations21:10:35

Hearing the focus on PDCA it make me think of the whole effort in terms of Toyota Kata. Was that a point of reference?

Craig Larsen - he/him - Solution Design Group Mpls21:10:43

@rshoup @markwei Have you ever run into “we can’t change that much because we don’t want to confuse the customer with all of these changes” type of thing? Is this a problem?

Randy Shoup21:10:30

A bit. Our seller community is conservative. But mostly no.

👍 1
Mark Weinberg21:10:30

different problem from velocity but we do run into that some. but we have been working to decouple deployments from that. we can deploy quickly behind feature flags and rollout as desired.

👍 1
Craig Larsen - he/him - Solution Design Group Mpls21:10:36

I tend to think that the answer to this is “we want to be able to deploy fixes and minor changes quickly, but let’s not deploy a major refactoring of the UI every day”.

Graham McGregor21:10:58

Curious how folks capture data to track change failure rate? Is it manual entry when there's an incident or bug? If automated, how?

Randy Shoup21:10:48

We use rollbacks today. Adding in "P1 bugs" soon.

thankyou 1
Randy Shoup21:10:26

Fully automated

Gene Kim, ITREV, Program Chair21:10:00

“I was shocked that my build time was 1 hour. We worked at it, and now it’s 4 minutes.” 🎉 “1 hour 45m to validate PR; now down to 10 minutes.” So great, @markwei!!

Craig Cook - IBM21:10:36

Is some of the tooling open source?

Randy Shoup21:10:00

We use open source tooling, and hope to open source some of our internal tools over time.

❤️ 1
Rob Fraley21:10:34

@rshoup - did some of the improvement in speed and efficiency include moving folks from older build and deploy processes and tools to the open source tools you were referencing above?

Randy Shoup22:10:56

It wasn't about older tools vs newer tools, but adopting the tools we already had, and improving them. Hope that makes sense.

Gene Kim, ITREV, Program Chair21:10:52

“aspiration: one-click deploys for all our applications” @markwei

🚀 1
Daniel Cahill - Engineer - Ontario Systems21:10:02

What is involved in a partner signoff?

Randy Shoup21:10:31

I run my tests when my dependency changes. Contract testing is the solve ultimately.

Javier Magaña - Walmart21:10:04

Definition of Done is a great accelerator in my experience

Virginia Laurenzano NSA/MARFORCYBER21:10:41

the Marines and I talk about the meaning of 'done' all the time

Hayden Smith (Anchore)21:10:28

AH! Yes. The definition of done could actually be a great talk for next year at DOES!

Gene Kim, ITREV, Program Chair21:10:46

“weekly mobile releases have vastly improved our quality, because there’s not a rush to jam features in.” I love this phenomenon!

🚀 1
🙌 1
🎆 1
Randy Shoup21:10:20

And this was much easier than anyone expected, including and especially my mobile release team 😉

jeff.gallimore (CTIO - Excella, he/him)21:10:26

smaller batch sizes with more frequent delivery… lean FTW!

Randy Shoup21:10:33

They got to weekly 5 months ahead of when we thought was a stretch goal

Gene Kim, ITREV, Program Chair21:10:15

“Excitement and Fun”. “People not in the pilot want in!” What a cool testament, @markwei!

❤️ 2
🎉 1
Craig Larsen - he/him - Solution Design Group Mpls21:10:38

Good thing there’s two of you! You have a lot of questions (which is great)! upvotepartyparrot

Craig Larsen - he/him - Solution Design Group Mpls21:10:38

Good thing there’s two of you! You have a lot of questions (which is great)! upvotepartyparrot

Randy Shoup21:10:53

load balancing

😂 1
Mark Weinberg21:10:15

this partnership has made all the difference in the world. teams see our alignment and follow.

👍 1
Denver Martin, Dir DevSecOps, he/him21:10:41

i would love more about how others deal with Hot-Fix and impact on next sprint...

Bryan Finster - Defense Unicorns (Speaker)21:10:24

“CD” is always my answer. 😄 We design for emergencies and use it to deliver features.

Denver Martin, Dir DevSecOps, he/him21:10:01

yeah, we have issue where the team that should be doing grooming for the next sprint is the ones that get pulled into hotfix so the grooming does not happen as well as it should... So we end up with more issue at deploy then have more hotfix, so it is a never ending spiral.

Bryan Finster - Defense Unicorns (Speaker)21:10:44

Happy to talk a little about that at the bar later, if you’re free.

Denver Martin, Dir DevSecOps, he/him21:10:17

yes, should have time after the closing remarks, But need to drop at 6:45 pm EDT...

Denver Martin, Dir DevSecOps, he/him22:10:15

sound good heading to the bar area

Jeffrey Fredrick, Author-Agile Conversations21:10:44

Culture is a second order effect < focus on being better at performance, on learning, and culture follows. That’s been my experience.

👍 1
Jeffrey Fredrick, Author-Agile Conversations21:10:08

which is good! if you had to fix culture first that would be really really hard. 🙂

🔥 2
Randy Shoup21:10:30

per Jez in Accelerate: change culture by changing behavior, not the other way around

Ron Westrum21:10:04

Culture doesn't come from performance. It comes from leader signals

❤️ 1
Randy Shoup21:10:02

Culture comes from the top, as I've experienced very directly in every place I've ever worked 😉

Jon Smart [Sooner Safer Happier]21:10:12

A colleague NPS score might be interesting? "I would recommend our ways of working to a colleague or friend". Interesting to see the trend over time. It's usually a Duning-Kruger curve (up, down, up).

Jon Smart [Sooner Safer Happier]21:10:00

Also, it's possible to improve on the DORA metrics by working people harder (unhappier, unsustainable, burnout)

Randy Shoup21:10:57

Good idea. We have done a developer survey and plan to do more. NPS is one of the questions.

👍 1
Jon Smart [Sooner Safer Happier]21:10:00

Idea: measure quality (better), value, sooner (flow), safer (compliance, GRC), happier 🙂 #BVSSH

Brad Vandehey22:10:52

👆:thinking_face:.....yeah, makes sense:+1:

Randy Shoup21:10:25

No pressure. There are only two initiatives that are reviewed every month by the exec team -- our massive multiyear long move to mediated payments, and this work

🔥 3
Gene Kim, ITREV, Program Chair21:10:43

“Our CEO Jamie Iannone is a huge supporter of our work, constantly mentions our work in the Town Hall… and tells us to go faster.” 😆

🚀 4
🐿️ 1
Daniel Cahill - Engineer - Ontario Systems21:10:44

QE stands for Quality Engineering?

Randy Shoup21:10:21

Goodhart's Law in action

💯 3
Javier Magaña - Walmart21:10:34

We have always encouraged Cross Training between QE and SE. Some people think they are too good to cross train

Justin Abrahms (eBay)21:10:06

Google had a few programs like this where you did a 'tour of duty' in a different role. Was generally well received.

Javier Magaña - Walmart21:10:51

Interesting. Any blog I can reference? Is an uphill battle most of the times

jeff.gallimore (CTIO - Excella, he/him)21:10:36

focusing too much on the metrics, gaming the system, …

👍 1
Bryan Finster - Defense Unicorns (Speaker)21:10:54

How to get them gaming it in a way that helps the goals? 😄

☝️ 2
Bryan Finster - Defense Unicorns (Speaker)21:10:39

As @nasello.scott says, “hacking the biggest undocumented API”

💥 1
Craig Larsen - he/him - Solution Design Group Mpls21:10:58

Positive metrics showing that we aren’t taking chances.

Brian Smith21:10:26

Are you measuring the benefit hypothesis of features in production?

Randy Shoup21:10:03

Say more about what you mean? We are laser-focused on delivery at the moment, and next year we will introduce Flow Framework for a broader perspective.

Brian Smith21:10:22

sorry, I responded with a new post.

Gene Kim, ITREV, Program Chair21:10:44

“Hey, Randy, I know YOU think you’re doing the right thing, but…“. 😆

Philip Day21:10:12

How did eBay get to 2020 without launching a program like this? Or did they try & abort previous attempts?

Randy Shoup21:10:28

Many failed agile transformations over the previous decade

Randy Shoup21:10:34

We were world class in 2006 😉

🔥 1
Mark Weinberg21:10:46

tech debt… just keep bolting stuff on.

Justin Abrahms (eBay)21:10:46

Goal posts keep moving to be "world class". 🙂

Craig Cook - IBM21:10:04

Why is this effort different?

Philip Day21:10:17

were the failed attempts capital-A, capital-T by any chance?

Randy Shoup21:10:36

Different in all of the things we mention here

❤️ 1
Mark Weinberg21:10:09

support from executives and strong collaboration between teams.

❤️ 1
Philip Day21:10:08

Was this time a bigger resource investment? Or just smarter?

David Van Couvering - Senior Principal - eBay21:10:08

My theory - we have leadership in place now who "get it" - Mark, Randy and Jamie

❤️ 2
Justin Abrahms (eBay)21:10:37

It's not like we have a ton of folks working on this. There are maybe ~6 of us working solely on this? Echoing DVC, I think having platform + product engineering 'joined at the hip', is hugely helpful in aligning incentives.

David Van Couvering - Senior Principal - eBay21:10:10

It takes vision, commitment and courage to implement an effort like this

Randy Shoup22:10:55

Smaller resource investment, but people who had done it. Work smarter, not harder 🙂

👍 1
Jeffrey Fredrick, Author-Agile Conversations21:10:18

“What’s our North Star?” < echo of Toyota Kata again

❤️ 3
Randy Shoup21:10:13

Please help us figure out how to scale. Really interested in experiences around going big and wide.

Virginia Laurenzano NSA/MARFORCYBER21:10:32

there is the middle manager challenge again

👍 1
Brian Smith21:10:47

On you slide two slides ago you said something like Improving the Ebay outcome. Could you restate that again here in the chat?

Randy Shoup22:10:21

Not sure I know exactly what you are referring to, but you can download the slides at

Randy Shoup21:10:23

I love working for Jamie, and I hope you all can see why

👍 1
jeff.gallimore (CTIO - Excella, he/him)21:10:58

A CEO who “gets it”. Yeah, I can see.

Scott Nasello - eBay - Sr Director, Engineering Productivity21:10:39

Boom, CEO vid

💥 3
👏 5
❤️ 1
🚀 1
☝️ 1
Courtney Kissler21:10:11

@genek another Chris O'Malley???

Guillaume Bladier (He/Him)21:10:01

yeah, great surprise ❤️

Gene Kim, ITREV, Program Chair21:10:06

My heartiest congratulations to@rshoup @markwei for pulling this off! 🎉

🍻 4
💯 1
James T Snell21:10:13

Very stimulating talk Randy and Mark, thank you. I’ve added it to my list for rewatching and sharing later. Fun Coincidence: The used Apple Watch I ordered via ebay just arrived

💯 2
Ann Perry - IT Revolution21:10:22

💡 Bringing us to a close today is none other than @ronwestrum, here to share more about Information Flow Cultures 💡

Gene Kim, ITREV, Program Chair21:10:23

“I’ve asked them to scale this initiative even further” 😆

👏 1
Justin Abrahms (eBay)21:10:39

No pressure, but make it happen. 😛

Ganga Narayanan21:10:02

You can do it @markwei, @rshoup ! 😄 Inspiring presentation!

Gene Kim, ITREV, Program Chair21:10:40

THANK YOU! That was awesome, @rshoup @markwei!

💯 2
👏 2
Ron Westrum21:10:46

Nice to be here!

👋 3
👏 2
Brian Smith21:10:47

If you start features with a benefit hypothesis, "If we do this feature, we expect this outcome", then determine how you can measure that benefit in production. One example is A/B testing. There are experts here who could explain the practice better.

Justin Abrahms (eBay)21:10:17

We have a pretty large experimentation effort here. Not evenly applied, but the capability is there.

Randy Shoup22:10:43

For the velocity work, we do have a hypothesis about each of the platform improvements, and each of them have success metrics (e.g., build time, startup time, etc.). Justin is right about customer experimentation; it's uneven.

Scott Nasello - eBay - Sr Director, Engineering Productivity21:10:57

Hey @rshoup @markwei - you forgot to say if you are hiring

😎 2
😆 2
Randy Shoup21:10:10

We are! Come work with us!


thanks @rshoup and @markwei!


Hoping to get the same type of support for a similar initiative ongoing here 😉

Randy Shoup22:10:03


Steve Farley - VP Infrastructure Operations, Nationwide Insurance21:10:35

Thanks ebay team....took away some great action....already chatted our VP of Architecture

👍 1
Randy Shoup21:10:51

Happy to talk more, thanks

Gene Kim, ITREV, Program Chair21:10:44

Watch out Randy! Steve used to manage 200+ dev teams at Nationwide, and then he was asked to take over all of I&O at Nationwide. 😆 He’s speaking tomorrow — amazing amazing story.

😀 1
Gene Kim, ITREV, Program Chair21:10:55

Up next: Please welcome Dr. @ronwestrum, whose work has influenced the work of so many of us!!

❤️ 2
🎉 1
Courtney Kissler21:10:05

I've referenced the Westrum model twice today already 🙂

❤️ 4
Chris Gallivan (Tasktop-Planview)21:10:52

Hey Ron! I'm halfway through Sidewinder. Awesome and Inspiring.

🔖 1
Dave Mangot - DevOps transformation professional21:10:53

I think you generated the phonemes when naming the model @ckissler

Brian Smith21:10:59

OMG goodness!! I am so excited!

Ron Westrum21:10:16

My pleasure Chris!

👍 1
BJ Preece21:10:20

best idealcast episodes ever. I wore the tape out re-listening.

Gene Kim, ITREV, Program Chair21:10:33

Incidentally, during the @allspaw recording session, he and I were arguing about which @ronwestrum paper was the best. Requisite Imagination vs. Information Flows: A Personal Journey.

❤️ 2
🔖 2
Brian Smith21:10:45

I reference Westrum every week in class. I am so happy to hear you present this directly.

Jeffrey Fredrick, Author-Agile Conversations21:10:52

Emphasis on the Mission < key key key

Randy Shoup21:10:58


Denver Martin, Dir DevSecOps, he/him21:10:01

I actually ask about flow of information when I interview at new companies... and then follow up with are they looking to change that flow....

👍 3
Ron Westrum21:10:22

that's the way!

Pete Nuwayser - IBM21:10:24

so. pumped. for. this. talk.

💯 2
Gene Kim, ITREV, Program Chair21:10:45

“Communication is optimized for the receiver” (profound)

💯 3
jeff.gallimore (CTIO - Excella, he/him)21:10:25

so often communication is done from the perspective of what works best for the communicator.

Courtney Kissler21:10:07

we should capture NPS on communication/information flow

👍 2
💯 3
Pete Nuwayser - IBM21:10:43

I love this example.

Randy Shoup21:10:20

I love the idea of the Technical Maestro. So evocative.

Gene Kim, ITREV, Program Chair21:10:27

We heard from Captain @emily356 in the amazing workshop this afternoon: leaders can reward these behaviors: “when I make mistakes, I tell everyone, because that’s who I am”, reinforcing these desired norms.

👏 3
Gene Kim, ITREV, Program Chair21:10:58

Not mentioned here, but so profound, when combined with Technical Maestro: Rabinow’s Law: “if the boss is a dope, you will have, or soon will have dops all the way down.”

☝️ 1
💯 3
😭 2
👏 1
Pete Nuwayser - IBM21:10:37

@genek I'm still waiting for someone to define the Inverse Rabinow's Law.

Jeffrey Fredrick, Author-Agile Conversations22:10:00

but if the boss isn’t a dope but people think he is… what then?

Steve Farley - VP Infrastructure Operations, Nationwide Insurance22:10:31

Gene...I shared the Dope story you gave me with my team that day. Lots of fun out of that. But more important the traits of a leader who isn't a dope which they really loved.

Gene Kim, ITREV, Program Chair21:10:24

The two in combination have tremendous explanatory power: can explain great and horrible orgs, and why

jeff.gallimore (CTIO - Excella, he/him)21:10:27

“requisite imagination” 🤯

👍 1
Ron Westrum21:10:55

the fine art of imagining what will go wrong

💡 1
Scott Jaffa (Principal Engineer, ValidaTek)22:10:28

and you just put a pair of words to an attribute that I’ve been trying to describe. That pretty much covered the cost of the conference in one line. Thanks!

👍 1
jeff.gallimore (CTIO - Excella, he/him)21:10:42

i can see how important being a “technical maestro” must be to the ability to have “requisite imagination”

Pete Nuwayser - IBM21:10:46

But isn't that what leaders are supposed to be doing? Looking down the road, identifying impediments, and clearing as much as they can?

Andrew Davis - Copado - DevOps and Testing for Low Code21:10:58

Westrum’s Law: “The higher the stakes, the rougher the play” ouch.

Gene Kim, ITREV, Program Chair22:10:05

The mind expanding manifestation of requisite imagination for me: @rdaitzman yesterday walking business partners thru what could go wrong, and inviting discussion of response/etc, and elevating risks.

👏 2
Gene Kim, ITREV, Program Chair22:10:20

(such a great presentation with @christina_yakomin!)

Dave Mangot - DevOps transformation professional22:10:29

Lesson, don’t let your company acquire other companies 😂

😂 3
Javier Magaña - Walmart22:10:55

TIL: The background for Boeing Culture

Ron Westrum22:10:10

don't get lost in the aquisition

Dave Mangot - DevOps transformation professional22:10:14

Sounds like when the families combined, it was more like the Gambino family

Gene Kim, ITREV, Program Chair22:10:31

(thanks to @ronwestrum, I’ve watched 9 hours of Boeing documentaries. admittedly some parts at 1.8x speed)

😄 2
Joshua Barton22:10:50

I now feel like I need to go find those..

Ron Westrum22:10:34

tony soprano indeed!

😁 1
Chris Gallivan (Tasktop-Planview)22:10:40

Don't hire someone named Stonecipher

Ganga Narayanan22:10:50

So when a company goes through a M&A or major leadership change, there's a major cultural shift

Scott Jaffa (Principal Engineer, ValidaTek)22:10:56

The story of Boeing’s culture change after the merger is terrifying.

Matt Saunders22:10:16

I know how this ends ... 737 MAX 😞

😢 2
Scott Jaffa (Principal Engineer, ValidaTek)22:10:09

They are lucky that starliner didn’t make it on the same level at the max.

Bryan Finster - Defense Unicorns (Speaker)22:10:25

Escaped before his fail caught up to him.

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear22:10:04

That seems to happen far too often unfortunately. 😞

Bryan Finster - Defense Unicorns (Speaker)22:10:31

Yep. Red Shirts always picking up the pieces

💯 1
Meghan Glass - PrdMgr Best Buy22:10:35

"family" culture can often mean toxic -- unlike what I think the "good family" culture can be which is a performing/generative team

👍 1
Javier Magaña - Walmart22:10:10

True. I wonder how many people have a negative context around the word family.

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere22:10:09

Curious about this too. In his book No Rules Rules, Netflix CEO Reed Hastings talks about creating a culture less like a family and more like a professional sports team. I'm very curious how folks from Netflix would respond to this, as well as what kind of culture Netflix would be classified as under the Westrum model.

👀 1
👍 3
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear22:10:10

I’ve definitely heard of some shady companies using the “family” angle to avoid fairly treating employees.

Jeffrey Fredrick, Author-Agile Conversations22:10:28

I think lots of different analogies can work, because the magic isn’t in the words, it is in the shared understanding, and does it generate the “will to believe”.

👍 4
Meghan Glass - PrdMgr Best Buy22:10:17

I think organizations should recognize their culture explicitly and then the relationship between employee and organization can start from a place of trust. If we see these topologies as analogies and not intentional cultivation then there isn't collective ownership over the culture

💯 1
Jeffrey Fredrick, Author-Agile Conversations22:10:15

Completely agree It isn’t enough to put the words in a powerpoint, it is a shared mindset you need to build.

❤️ 2
Christina Tan, Strategy at Blameless, Speaker DOES2122:10:50

unfortunate loss.. when culture not explicitly stated

Alex Duff22:10:51

"Family" speaks to connection, "team" less so (?).

Christina Tan, Strategy at Blameless, Speaker DOES2122:10:14

Yeah, agree that genuine connection is critical. I've seen it come at the expense of team performance/business needs, which is when it's no longer working/generative..

Matt Saunders22:10:03

how can they get this generative culture back, when they're now renowned for their planes falling out the sky? 😞

Sara Gramling22:10:14

How do you find out whether your culture is working (or not)?

👀 2
Ganga Narayanan22:10:38

Culture is something most don't quite understand since it's thought of hard to measure objectively

Ron Westrum22:10:40

You can find out by doing worker surveys

❤️ 1
👍 5
Scott Jaffa (Principal Engineer, ValidaTek)22:10:59

any details / resources you can point to?

Ryan Taylor - Senior Geospatial Developer, GISinc22:10:49

Curious about what kind of questions should be asked to find faults in a company culture.

👀 1
jeff.gallimore (CTIO - Excella, he/him)22:10:34

i’ve used @nicolefv’s work in “accelerate” with much success. she is a surveying aficionado.

👍 1
BJ Preece22:10:19

employers that actively avoid surveys or bury signals are likely pathological...

☝️ 1
Ron Westrum22:10:31

right, DevOps actually has worker surveys for this

Ron Westrum22:10:18

Ask people what kind of culture it is. They will tell you

👏 2
Gene Kim, ITREV, Program Chair22:10:38

I love that the State of DevOps Research validates the Westrum Organizational Typology — @ronwestrum

👏 1
Ron Westrum22:10:03

I love that, too

🙏 1
Ganga Narayanan22:10:47

"If it were not for Boeing, there would be no free world." Profound!

Dave Mangot - DevOps transformation professional22:10:13

Promises kept can be such a hard part of being a middle manager if the leaders above you keep shifting priorities

Matt Saunders22:10:09

is it wise to expose the leaders' inability to stick to a plan to those lower down? 🙂

😮 1
James T Snell22:10:21

Maybe Avro could have filled the void. Whatever, Boeing did great.

Ron Westrum22:10:41

A culture of kept promises is good

Gene Kim, ITREV, Program Chair22:10:44

“all the smart people at Boeing were assigned to work on Supersonic Transport project.” 😆 (leaving few for 747 project.)

Ron Westrum22:10:37

I have great respect for Joe Sutter

Virginia Laurenzano NSA/MARFORCYBER22:10:57

love that the leadership secret here is "make people feel better"

💯 5
👍 1
Gene Kim, ITREV, Program Chair22:10:12

After watching 3 documentaries on the 747, here’s the one I’d recommend that was produced by National Geographic, coincident with recent launch of 747-8.

✈️ 1
Gene Kim, ITREV, Program Chair22:10:31

The original 747 project is nuts: 75K engineering drawing. Can’t even comprehend how one designs something like that.

Dave Mangot - DevOps transformation professional22:10:32

I so distinctly remember seeing the “upstairs” in the 747 for the 1st time

Ron Westrum22:10:40

symbolic management is important!

Matt Saunders22:10:09

go big or go home!

BJ Preece22:10:16

this reminds me of Kent Beck 3X. After proving your idea/MVP, everything is a threat to your survival.

Scott Jaffa (Principal Engineer, ValidaTek)22:10:20

Great story! RIP Queen of the Skies!

Scott Nasello - eBay - Sr Director, Engineering Productivity22:10:22

@ronwestrum - back to tech maestros ... Does the tech maestro create the culture? or Does the culture reveal the tech maestro?

Ron Westrum22:10:51

The culture allows the tech maestro to rise

👍 4
🙏 1
Gene Kim, ITREV, Program Chair22:10:18

Reviewing my notes: each 747 engine costs $20MM — 4 of them. $80MM. So they attach at very last minute, so payment made at last possible minute — minimize time before assembly and sale to airline!

Matt Ring (he/him) - Sr. Product/Engineering Coach, John Deere22:10:37

That factory picture reminded me of this scene in Idiocracy... 😝

😆 3
Ron Westrum22:10:50

Gene steals all my thunder!

😆 6

Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring!   Enter your feedback for those talks here:

Virginia Laurenzano NSA/MARFORCYBER22:10:31

what did the Feds do to interfere? curious to know

Gene Kim, ITREV, Program Chair22:10:22

I’ll find the quote in the book for you.

Virginia Laurenzano NSA/MARFORCYBER22:10:59

@ronwestrum told me to read it and I will 😉

Gene Kim, ITREV, Program Chair22:10:52

Sorry, @ronwestrum!!! 😆 Just wanted to prove to you that I did my homework! 😆

😆 1
❤️ 1
Ron Westrum22:10:53

The Feds nixed Alan Mulally

Ron Westrum22:10:40

Read book American Icon about Mulally

👍 2
Randy Shoup22:10:45

"If you break a covenant, there is a price"

❤️ 7
Dave Mangot - DevOps transformation professional22:10:08

This talk is an emotional rollercoaster!!

💯 6
🥲 1
Matt Saunders22:10:38

the best talks often are!

Geri O’Toole22:10:57

I’m thinking of a few people I knew well who work there. Wow.

Gene Kim, ITREV, Program Chair22:10:49

“employees felt that management were preventing them from doing the right thing”

😞 2
Geri O’Toole22:10:14

.. when you’ve just joined a new org and see so many similarities… :woman-facepalming::skin-tone-2: where do I start to help?

Geri O’Toole22:10:49

(Asking myself but will take suggestions. Lol)

Ron Westrum22:10:34

Say it ain't so!

❤️ 1
Dave Mangot - DevOps transformation professional22:10:23

@allspaw is going to pop up and say adaptive capacity!

Tashfeen Mahmood22:10:24

I love the "circle of faith". Takes Product Management (being customer-centric) to a whole other level

❤️ 2
Dave Mangot - DevOps transformation professional22:10:53

“saved their lives” sounds so much like the Toyota philosophy of not building cars, but enabling people to transport their families to soccer games etc.

👍 1
💯 4
Brian Smith22:10:03

GEMBA - Circle of trust

Randy Shoup22:10:08

This framing of a 'circle of trust' is really wonderful. I'm going to rewatch this many times, I predict.

👍 2
Gene Kim, ITREV, Program Chair22:10:08

Thank you so much, Dr. @ronwestrum — we so much appreciate your work!!!

❤️ 12
💯 1
Brian Smith22:10:13

Wow, thank you!

❤️ 1
Javier Magaña - Walmart22:10:19

Amazing presentation. Thank you @ronwestrum

James T Snell22:10:21

That was great, thank you @ronwestrum. 🙂

Dave Mangot - DevOps transformation professional22:10:24

how do you give a virtual standing ovation??

👍 4
BJ Preece22:10:44

thanks for the insight @ronwestrum

Randy Shoup22:10:03

Thank you @ronwestrum!

👏 2
Dakota Wandro22:10:03

@ronwestrum can you elaborate on what you mean by "potential consulting"?

Ron Westrum22:10:26

Always a pleasure!

👏 2
Daniel Cahill - Engineer - Ontario Systems22:10:59

Did you hint last DOES that there might be a new book to be on the lookout for?

Nick Eggleston, IT Architect (DMs welcome)22:10:09

@ronwestrum will you be hanging out with us in Gather or slack for the during the rest of the conference?

Ron Westrum22:10:21

book will come in time!

🎉 3
Gene Kim, ITREV, Program Chair22:10:36

^^^ Hahaha. Says every author ever. 😆

😂 3
Gene Kim, ITREV, Program Chair22:10:48

Thank you, all!!! Catch you later tonight and tomorrow!

Ron Westrum22:10:49

I will be on and off, got to teach

📖 2
Denver Martin, Dir DevSecOps, he/him22:10:00

Heading to #gather no donuts...

🥐 1
Nick Eggleston, IT Architect (DMs welcome)22:10:53

@ronwestrum hopefully i'll get to join a Gathering with you sometime (when you're not teaching 😂)

Ron Westrum22:10:20

#gather for a few minutes

Ron Westrum22:10:47

Look forward to talking with you. Nick


past midnight here, happy hour/gather to those in a more favorable TZ 😴