Fork me on GitHub
#ask-the-speaker-plenary
<
2020-10-14
>
inactive00:10:08

This took me a year to understand; the problem of “shipping faster than you learn.” 🤯

😩 1
5
Jack Vinson - flow00:10:32

Can apply anywhere. Physical (cars, consumer goods) and technical products (software).

Ferrix Hovi - Head of DevOps - Siili00:10:19

@genek101 Oh, that rhymes "speaking a foreign language faster than you can produce the words." Makes perfect sense because of a few colleagues I had 😄

😆 2
Tim Haagenson, Software Engineer, American Airlines00:10:34

Shipping faster than you learn is often a subtle signal that the power dynamics are still wrong. The dev team is improving the only area they feel empowered to impact.

👍 3
👆 2
John Cutler00:10:45

Great call out @timothy.haagenson798

pcn00:10:51

I've always liked that - "you don't have internal customers"

Woody Evans00:10:58

"You don't have internal customers unless they are giving you money." Exactly.

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host00:10:07

> The dev team is improving the only area they feel empowered to impact. Doing the only thing to stave off burnout

pcn00:10:16

Internal customers are ways of setting up a fight over how to try to screw each other out of a penny

inactive00:10:39

One of my fave lines from Unicorn Project is from @bryan.finster — “‘The business’ is not our customer! They’re our colleagues!”

💯 2
👍 6
Ferrix Hovi - Head of DevOps - Siili00:10:56

Another anti-pattern in a SAFe transformation is not explaining each team the benefits they are supposed to be getting from all those new ceremonies. That leads to a very lax implementation that doesn't really empower the team when it usually does help when somebody spends a while with them.

👍 3
Steve Pereira - Value Stream Consulting00:10:27

Absolutely true. There's often no feedback mechanism that clearly signals when the rituals start costing more value than they generate.

Andy Nelson, Product Mgr - CH Robinson00:10:24

As read from the "Book of Cutler Chapter 4 Verses 1:20"

🙌 5
😁 1
❤️ 1
Ferrix Hovi - Head of DevOps - Siili00:10:53

I am close to productising the "three week intervention to redeem a former SAFe team"

inactive00:10:58

“what would you do differently if you got promoted two levels today?”

Craig Cook - IBM00:10:31

Does buying a new car count?

Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech00:10:47

Literaly opened this thread to see what serious people said while I thought of my Maserati:see_no_evil:

😂 2
Andy Nelson, Product Mgr - CH Robinson00:10:51

As a product guy that was a breath of fresh air

Tor Flatebo00:10:08

what was the 3rd point?

John Cutler00:10:09

The Right Brain Treatment is my new Band Name

🧠 2
2
1
🎸 1
Michael Dennison00:10:10

Excellent presentation

Nick Eggleston (Nix) - IT Architect - (actively looking)00:10:17

@john.cutler so much to think about and reflect on! will have to listen to this again.

🤙 1
inactive00:10:23

Thank you @john.cutler — and now, here’s @pdmoore, presenting on another source of dissatisfaction of how tech is often managed!

👏 3
Daniel Cahill - Engineer - Ontario Systems00:10:46

Did someone write down John's questions to think about over the next few days?

inactive00:10:54

cc @john.cutler

Andy Burton00:10:08

I was writing as fast as I could but have some gaps. Need to hit the library and rewatch that bit. 🙂

John Cutler00:10:16

Yup. One moment.

John Cutler00:10:56

@U015MQR554N What beliefs must you personally let go of. Tto set about not just sitting at the table, but leading your businesses, and using design and technology to support step changes and sustainable growth ? What would you do if suddenly you were give a two level promotion? How would you navigate that.  What steps do you need to take to be able to connect the day to day work of your teams out to the benefits and outcomes in the world outside your org walls? To help your organization learn as fast as you ship? What can you do to overcome the inherent biases / dichotomies / stereotypes that exist about “tech” in your company? To be more than builders and runners? To turn everything you do into the business?

thankyou 3
Jack Vinson - flow00:10:48

Company as the product - Think about how we think positively or negatively about the big companies out there? I'd buy any product from X. Or I'd never buy a product from Y.

1
Jack Vinson - flow00:10:48

Some companies have had some nice successes in using technology to support their product - its part of the complete story/product/offering.

Ferrix Hovi - Head of DevOps - Siili00:10:57

What? @genek101 do you want to make us angry?

Dave Mangot - Mangoteque00:10:35

Hmm, IT as a cost center, this sounds familiar

😞 1
👆 1
inactive00:10:40

“Zone to Win” — @mik and I often talk about what a fabulous book it is. @pdmoore talks about how, in a just world, this book would be as impactful as “Crossing the Chasm.” Did I misrepresent that, @pdmoore?

👍 1
Peter Moore00:10:05

You got it right Gene

inactive00:10:20

…a source of some dissatisfaction, I imagine. 🙂

Peter Moore00:10:52

I'll talk about these metrics at the end of my talk

inactive00:10:01

When put this way, it makes so clear why so many metrics and management models applied to “IT” are completely wrong — Productivity Zone is the cost center. Everything else requires different goals, objectives, metrics.

Ferrix Hovi - Head of DevOps - Siili00:10:21

@genek101 have you mapped that over Cynefin?

inactive00:10:31

I can totally see how the 4 Zones have different dynamics, as per Cynefin. cc @pdmoore @mik

Ferrix Hovi - Head of DevOps - Siili00:10:09

Yeah the "win/loss" ratio could be a "complexity compromise" version of revenue vs. plan when planning isn't so easy.

Ferrix Hovi - Head of DevOps - Siili00:10:12

Not sure if a direct overlay makes sense or not.

James (TeamForm) - helping teams at scale00:10:05

@john.cutler Thanks for that - very insightful and aligns very much so to our way of thinking. Would love to chat more about it.

2
inactive00:10:35

I am so fascinated by this Splunk case study: the structure and dynamics when there’s a team for Salesforce, Netsuite, etc…. Like Team of Teams, like functional Ops silos. The universality of this problem blows me away. Regardless of what level of the application stack you’re in!

🙌 1
John Cutler00:10:34

Great case studies @pdmoore

👍 1
inactive00:10:35

Edmunds story. Wait for it… 😂😂😂 (You’ll love this, @scott.prugh)

inactive00:10:06

The hostile vendor. “Go pound sand.” 😂😂😂

Nick Eggleston (Nix) - IT Architect - (actively looking)00:10:44

Link to the story or what to search for to read deeper on this?

Pete Nuwayser IBM he/him/his00:10:23

YES I /knew/ you were going to say "hostile vendor"

Daniel Cahill - Engineer - Ontario Systems00:10:35

With that openended question, I could see different answers from different areas of the organization. Is there a suggestion of who to start with for asking these types of questions?

Scott Prugh - Chief Technology Officer - CSG00:10:43

Grrrr... Hostile vendors... We need to create a support group...

😂 7
😁 1
jeff.gallimore (CTIO - Excella, he/him)00:10:14

A new Birds of a Feather session? :thinking_face:

inactive00:10:03

Hahaha. So good…. Beautiful story, @pdmoore, of liberating an org from a hostile vendor.

👏 2
Philip Day00:10:28

Love a specific case study

Manny Avila - Consulting Engineer, Cloud and Datacenter - JD Finish Line00:10:28

anytime you get a chance to tell Oracle to pack sand....take it

Peter Moore00:10:55

Revenge is always great

Ferrix Hovi - Head of DevOps - Siili00:10:26

Depends on which end you are holding 😄

inactive00:10:03

😂😂😂

inactive00:10:33

I love GraalVM, the next generation JVM, coming out of Oracle Labs. I’d love to see people pay for it, to pay back all the amazing R&D that is going into that project. (Any other fans of GraalVM out there?)

inactive00:10:48

(Lots of Clojure people doing amazing things in it)

inactive00:10:52

Finally looked this up — didn’t quite fully believe it! But look!

4
🤙 1
📈 1
Philip Day00:10:53

Have seen this graph before, it's insane

inactive00:10:15

Now one of the world’s most valuable companies — was in a race with Apple to be the first $2T market cap company!

inactive00:10:33

A phenomenal explanation of the before/after of Microsoft!!!

🔥 3
🤙 1
inactive00:10:58

A real aha moment for me, @pdmoore

James Cham00:10:25

Just to be clear, is @pdmoore saying that MSFT was using the zone methodology intentionally or in an emergent way?

Peter Moore00:10:01

It was using it intentionally to pivot its business model

👍 3
🔥 1
inactive00:10:00

“why most tech initiatives fail: they’re using Performance Zone metrics for initiatives in the other zone.”

👍 1
jeff.gallimore (CTIO - Excella, he/him)00:10:01

Mistake: using performance metrics to measure incubation zone investments!

👍 3
inactive00:10:06

Thank you @pdmoore!!!!

👏 3
👍 2
💯 3
Jack Vinson - flow00:10:19

Always wonder about overlapping other "four quadrant" models when hearing about these - particularly Cyenfin and when we use the "wrong" mindset in a given situation.

John Cutler00:10:34

Thanks @pdmoore

James Cham00:10:44

So many great moments here!

Brad Nelson00:10:54

I love that story about MSFT, I think it was Simon Sinek who I first heard it from, but it shows the power of leadership and the right mindset!

inactive00:10:30

And thinking about the problem correctly!!!

👍 2
inactive00:10:06

So great to see you, @james812!

Peter Moore00:10:14

I will be doing an Ask me Anything session tomorrow from 12:35-1:35 if you have further questions or want to talk about any of the frameworks and tools I presented today.

👍 1
Rich B - DevOps is my career change00:10:18

what I learned - nothing is easy and this is no different

Ferrix Hovi - Head of DevOps - Siili00:10:37

Nothing personal but I fell asleep at 3:23 am listening to the final presentation. Surprising myself with stamina and recapping in the morning.

1
Philip Day00:10:16

Flagging here at 01:28

Nick Eggleston (Nix) - IT Architect - (actively looking)00:10:45

Lots of really good material today, it's going to take a while to digest.

👍 2
🙌 1
Fred Ghahramani00:10:29

Peter Moore and John Cutler - thank you for great end to our day today. Some my best sessions today

👍 1
Nick Eggleston (Nix) - IT Architect - (actively looking)00:10:43

Who out there is really good at networking?

inactive00:10:35

PS: I’m blown away by the Happy Hours that are happening — the creativity and resources put into making some incredible experiences are really amazing…. Like seriously, imagine the budgets put into physical events being deployed into a virtual event. Kudos!!!

inactive00:10:45

THANK YOU SO MUCH!!! I hope you enjoyed Day 1!!! See you tonight!!!!

Matt Ring (he/him) - Product/Engineering Coach, John Deere00:10:33

@jeff.gallimoreI don't know how to be a croissant in a Zoom meeting! 😆

🥐 2
1
inactive00:10:02

THANK YOU, @pdmoore!!!!! Thank you, all!!!

Peter Moore00:10:27

I really appreciate the opportunity to share my ideas with this community who is so committed to peer to peer learning Sent from my iPhone

👍 2
Andy Nelson, Product Mgr - CH Robinson01:10:29

@pdmoore Cheers to you - great way to end the day. See you at the AMA tomorrow!

Jeffrey Fredrick, Author-Agile Conversations15:10:19

next feature request you can expect @U015KPGSEF8 is the ability to suggest tracks. 😄

🤞 1
Ben Williams - Arvest Bank - Sr Data Pipeline Dev15:10:22

I listen to anime OP/ED and Japanese Punk/Rock/Metal/Ska from a twitch vTuber.

Patrick Debois15:10:29

@ferrix How did you guess my virtual platform start command 🙂

😆 2
👍 1
Patrick Debois15:10:47

Also great opportunity for a sponsor or good cause bidding party

👍 1
Ferrix Hovi - Head of DevOps - Siili15:10:40

@U015KPGSEF8 Even if the newbies create whatever new deployment commands, one should anyway alias make deploy to them.

Patrick Debois15:10:39

well I suggest make deploy-staging and make deploy-production

Ferrix Hovi - Head of DevOps - Siili15:10:29

The CD takes care of that 😄

Patrick Debois15:10:01

CD is too slow for realtime ops

pcn15:10:42

No stress, y'all

Ferrix Hovi - Head of DevOps - Siili15:10:34

Oh, and there's this song about insufficient test coverage https://www.youtube.com/watch?v=bo_efYhYU2A

Jeffrey Fredrick, Author-Agile Conversations23:10:23

This next plenary clearly needs this as the hold music while we wait: https://www.youtube.com/watch?v=0AAbY9Pvp6A

3
Jess Meyer - IT Revolution (she/her)15:10:47

<!here> Welcome @afurtado and @lauren.knausenberger from the United States Air Force as our opening plenary after brief opening remarks!

Dwayne Holmes15:10:16

@afurtado & @lauren.knausenberger love the concept of the iron bank

Austin Wimberly15:10:23

Ok it's about that time folks...whose excited about today!

🎉 2
👍 1
Christopher S Donahue15:10:03

The count down is ON ...

Christopher S Donahue15:10:16

Its going to be a Great Day!

Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:02

Good morning @genek101 and @jeff.gallimore

👋 1
inactive15:10:54

Please welcome @afurtado and @lauren.knausenberger! I’m so delighted they’re able to share their amazing story!!!

🙏 1
👏 6
🤙 1
Austin Wimberly15:10:51

I want that t-shirt! @afurtado

👍 3
🤙 1
2
☝️ 2
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:07

give me a bunch of compliments and ill send you one

Neil Kalinowski15:10:37

I like your beard.

👕 1
Virginia L. NSA Technical Director15:10:43

ditto. #agileAF is one of my favorite tags

Austin Wimberly15:10:47

Yep the beard game is strong!

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:16

just kidding- im the government, i cant send you free stuff. but thanks for compliments

5
Austin Wimberly15:10:36

I am also a Veteran ...Desert Storm - Army so don't hold that against me..lol!

Chris Gallivan, FCA, Builder of JOY15:10:58

Technically it’s our free stuff :)

Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech15:10:15

Over here in former-Europe when the government promises something they ALWAYS do it and they give us free stuff all.the.time #BorisFTW

Neil Kalinowski15:10:24

I was a Government contractor for the AF for 15 years. I know how it goes. Thanks!

Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:44

#AgileAF has at least 3 overloaded meanings 😊

Neil Kalinowski15:10:04

Agile Adam Furtado?

Neil Kalinowski15:10:16

Those are the only two interpretations I can think of 😉

DevOpsManagerPDX15:10:22

“It’s not wise to upset a Wookie” Really though, coolest shirt every Flyboy @afurtado

Jon Smart [Better Value Sooner Safer Happier]15:10:05

Gosh. Lost for words regarding these business outcomes.

1
inactive15:10:12

This is such an amazing story from @afurtado, describing how the accidental Doctors Without Borders led to an unflinching re-assessment of how software is built and run.

👍 2
Mik Kersten (Project to Product, Tasktop)15:10:25

Powerful start @afurtado. The Kessel Run story is amazing.

👍 1
inactive15:10:34

“collective disregard for the status quo” 🙏

1
inactive15:10:03

“12 parsecs or less” 😆

Scott Prugh - Chief Technology Officer - CSG15:10:10

Yeah. This is a crazy story and crazy outcomes to manage... This is what courage looks like... @afurtado

👍 2
Lou Sacco15:10:16

@afurtado how can I join the rebel alliance?

😆 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:35

There is an AMA later today and Im happy to directly recruit all of you! 🙂

🎉 2
Jeffrey Fredrick, Author-Agile Conversations15:10:44

“You shouldn’t have to go back in time to go to work.”

Scott Prugh - Chief Technology Officer - CSG15:10:45

Oh. Yeah. The Stockholm Syndrome... We talk about "learned helplessness" @afurtado

1
inactive15:10:46

“You shouldn’t have to go back in time to go to work.” So good, @afurtado!!!!

👍 3
Roman Pickl - technical pm - Elektrobit15:10:48

Stockholm syndrom. Nice. I mentioned that recently as well in a discussion 💯

Chris Gallivan, FCA, Builder of JOY15:10:04

Nothing inspires more than the post office starting a dojo

😂 2
inactive15:10:33

“Turn the US Air Force into a software company that can help us win”

👍 2
inactive15:10:01

“a minor triumph is pushing Microsoft Office updates into the field”

😱 1
Neil Kalinowski15:10:56

You don't want to know how long the AF stayed on Windows XP.

Ferrix Hovi - Head of DevOps - Siili15:10:08

Oh, we are not cushioning expressions today.

Chris Gallivan, FCA, Builder of JOY15:10:20

This seriously eliminates ALL excuses

👍 1
Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:22

It’s rather amazing to see positive change in such a huge bureaucracy

👏 3
Jon Smart [Better Value Sooner Safer Happier]15:10:38

$1b spent, three failed previous attempts

inactive15:10:48

Commercial airliners do not refuel their airplanes in mid-air! cc @ross.clanton410 and all my friends at American Airlines! 🙂

😂 1
inactive15:10:19

“large scale changes may require erasing entire whiteboard”. 😂😂😂

🙏 1
👍 1
Stephen Magill [MuseDev]15:10:28

“Excel macro involved somewhere” 😄

1
😂 1
😅 1
Austin Wimberly15:10:40

But imagine if they did..how far could we go?

Luke Rettig (Sr Director Product Management - Target)15:10:44

being comfortable with bureaucracy is an interesting concept. I wonder if @schmark is going to talk about that later 🙂

❤️ 1
😂 1
Dave Mangot - Mangoteque15:10:44

Like watching @dominica,I always trust a speaker who has sticky notes next to them on the board

👍 1
inactive15:10:55

😱😱😱😱. (Excel Macros difficult to unit test. 🙂 I know from personal experience, how fraught with errors they can be. 🙂

😱 2
Ganga Narayanan15:10:05

That looks like one of those dependency boards I've seen!

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:08

Jigsaw’s success is most inspirational story for DOES. 🔥 Kessel run is amazing.

👍 3
inactive15:10:17

Fuel savings $425K per day!!!

💰 1
Brandon Brown - Nike - Sr. SW Engineer II15:10:31

i love that the gui looks like the whiteboard. take care of the user.

👌 1
👍 3
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:26

There's a great story about how we came to that UI. Tried to get way too creative first and landed back to simplicity and user comfort. Im so glad you noticed that.

Zell Gagnon15:10:44

I was really impressed with the experience reports from that project -> It was a really great demonstration of genji genbutsu

inactive15:10:48

“We hit roadblocks, because we exited within the world’s largest bureaucracy.” So good, @afurtado!!!

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)15:10:53

@schmark has some ideas as to how to apply summo mechanics to this problem.

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:28

I honestly never thought I’d be inspired by military software

🙌 2
🤯 1
👆 1
Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech15:10:46

"After years they had still not got their first ever deployment of whatever it was" - that

Virginia L. NSA Technical Director15:10:53

Check out PlatformOne, too, for inspiration

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:22

Engineers who started and now lead PlatformOne were the original Jigsaw devs! Enlisted!

inactive15:10:58

I finally met @lauren.knausenberger at the Spark Tank competition she helped run at the Air Warfare Symposium, modeled after the TV show Shark Tank.

👍 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:12

5 times a day inside the DoD; yet we’re struggling to get to that in a small private company.

😆 3
Lauren Knausenberger15:10:23

Ha! That was an awesome day. IT was so great to have you there @genek101

Jim DeMarre15:10:07

So finally moved beyond WWMCCS ...

Brandon Brown - Nike - Sr. SW Engineer II15:10:30

these institutions are only as good as we make them. thanks for the effort here. @afurtado

Jon Smart [Better Value Sooner Safer Happier]15:10:16

Impediment: cyber security accreditation. Months to accredit. => Continuous ATO process. Impediment to flow alleviated

👍 1
👏 1
inactive15:10:24

It is so amazing to hear about how @lauren.knausenberger and @afurtado met, who had common goals and were mutually supportive!

inactive15:10:47

“six weeks inside of USAF was considered a triumph” — @lauren.knausenberger

Jon Smart [Better Value Sooner Safer Happier]15:10:53

Automatically accredited as the process was accredited 🔥

Chris Gallivan, FCA, Builder of JOY15:10:57

And to make a small enhancement to vehicle software takes 14.5 months :(

Ferrix Hovi - Head of DevOps - Siili15:10:57

@afurtado You mentioned fuel savings. How many million are the red tape savings?

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:04

Lauren was an absolute gamechanger for us. As @jonathansmart1 mentioned, she obliterated our biggest constraint

💯 2
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:24

Theory of Constraints in action 😄

🙌 1
inactive15:10:42

“the result was that our most innovative would hide.” — @lauren.knausenberger. What a phenomenal observation!

👀 1
Shawn Hisaw15:10:03

so glad to hear we are not the only folks stuck in excel...fueling fighter jets with excel...WOW

💯 1
Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:06

Was there any attempt to stamp it out that had to be overcome?

James Simon, FCA, Solution Architect15:10:11

Game Changer. Everything I have ever heard at my office has just become an excuse

👍 3
😂 2
inactive15:10:50

This is such a great example of how support from the top levels eliminated very powerful, very entrenched barriers and obstacles.

💯 4
❤️ 3
🤙 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:23

you can buy that on Amazon- it talked about me getting flown to the White House to save the world

😆 1
🇺🇸 1
Dave Mangot - Mangoteque15:10:28

"Do I have to write my own unit tests?" "Yes, that's an order!"

😀 1
😂 3
🇺🇸 1
inactive15:10:35

This is so true — when @mik @jeff.gallimore and I visited Kessel Run, there was a delegation from a top intelligence agency there! 😂😂😂😂

1
Austin Wimberly15:10:42

I love that @afurtado has Kanban or similar board in his background.

danielschwartzer15:10:46

How did you get a charter to build the rebel group?

Jon Smart [Better Value Sooner Safer Happier]15:10:47

Initial band of 5: IMO, there is a key point there. It started descaled.

👍 3
Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:54

Mind-blowing talks like these are why I love DOES ❤️

❤️ 4
Ferrix Hovi - Head of DevOps - Siili15:10:21

The doors in the US all seem the same.

inactive15:10:39

Kessel Run is now 1300+ people. When we visited, they hired scores of people THAT WEEK!!! A feat of incredible organization — the energy was amazing.

🤯 1
Jon Smart [Better Value Sooner Safer Happier]15:10:46

What I'm hearing: Descale before you scale. Start small. Prove it

👍 4
🤙 1
👏 1
Eduardo Rodrigues Semensati (Procter and Gamble)15:10:57

seems like a pattern across a lot of groups presenting. @genek101 the problem solver :thumbsup:

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:25

@afurtado did you reorg using the team topologies model?

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:19

The book absolutely played a huge part in how we organized

Ganga Narayanan15:10:32

Inverse Conway maneuver to restructure the organization!

Chris Gallivan, FCA, Builder of JOY15:10:10

So @genek101 is the common thread!

😆 1
Scott Prugh - Chief Technology Officer - CSG15:10:27

Yes!! Inverse Conway Maneuver!! @afurtado @erica.morrison

👍 4
Virginia L. NSA Technical Director15:10:34

We say no because we love our customers 😉

Ferrix Hovi - Head of DevOps - Siili15:10:44

Inverse Conway always remember blurting out "Inverse Godwin"... well, it was descriptive of the maneuver that time. 😄

Andrew Davis - Copado - DevOps for Salesforce15:10:46

Those are some of the most intimidating looking Girl Scouts I've ever seen

😂 2
💪 1
inactive15:10:49

“Just doing what the customers are yelling at you to do doesn’t serve their goals”

Luke Rettig (Sr Director Product Management - Target)15:10:00

don’t give those customers what they want, give them what they need

Nick Eggleston (Nix) - IT Architect - (actively looking)15:10:02

Sometimes saying NO is the best way to love your customer

😂 3
❤️ 3
Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech15:10:07

"It's the responsibility of any platform or product to really understand their customers" - yup

❤️ 1
👍 1
Nick - developer at BNPP15:10:15

That got me puzzled for a second ))

inactive15:10:38

“We’re finding places where we have a Bus Factor <= 1” ❤️❤️❤️

❤️ 1
Justin Heimburger - Edward Jones Team Lead, Platform as a Service15:10:48

@afurtado I'm curious what percentage of capacity you reserve for unplanned work

☝️ 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:36

A ton right now! When we finally dove into this we realized that we were planning to 90-95% capacity and added 40% unplanned work on top of it.

😮 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:13

So we started limiting the capacity, set WIP limits, and allowed our teams to get work down and started tracking the nature of the unplanned work

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:28

Over time we are automating and documenting those issues away and increasing capacity of the teams

Justin Heimburger - Edward Jones Team Lead, Platform as a Service15:10:03

Good stuff. How are you tagging and tracking your unplanned work?

Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:14

A mixture of chat bots and GitLab labels

🔥 1
Jon Smart [Better Value Sooner Safer Happier]15:10:07

Psych Safety again :thumbsup:

👍 1
❤️ 3
inactive15:10:44

“I saw the tangible results of absence of Psych Safety. Some elements of military structure makes it difficult; on battlefield, it can save lives, but…”

🎯 2
💯 2
Ffion Jones (Partner, PeopleNotTech)15:10:59

Tangible negative effects of not having psychological safety! - so clear to see when it's not there, hard to hear though

inactive15:10:08

“none of our military wear uniforms, b/c we don’t want some aspects of those interactions to affect how we code.”

👍 1
Ron Alvey15:10:27

@afurtado How did you increase productivity of the developers but still meet segregation of duty requirements for auditors?

Steve Spear15:10:27

Not wearing uniforms: very Naval Reactors ADM Rickover.

1
👍 1
Manny Avila - Consulting Engineer, Cloud and Datacenter - JD Finish Line15:10:00

I was temp deployed to the submarine named after Rickover...ssn 709

👍 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host15:10:57

a wild Steve Spear & Rickover makes an appearance

👍 1
Luke Rettig (Sr Director Product Management - Target)15:10:58

black friday event for weeks months and years…. or COVID :man-facepalming:

inactive15:10:02

“We have an impossible product platform — a user scenario that has happened before… we want to be as prepared as possible; imagine being in retail, but not know when Black Friday will happen.”

👀 1
🎯 1
inactive15:10:02

“We have an impossible product platform — a user scenario that has happened before… we want to be as prepared as possible; imagine being in retail, but not know when Black Friday will happen.”

👀 1
🎯 1
Robin Morrison (Agile Solutions)15:10:06

Very interested to hear the challenges and how you addressed them with getting actual buy-in and adherence to non hierarchy in actuality..

Adam Furtado, Chief of Platform, Kessel Run (USAF)15:10:41

We got success under the radar. We did not have support right away- we kept getting told to stop using the name Kessel Run and to fall in line. We chose to not... then once you get in FastCompany, the support comes shortly after...

James Simon, FCA, Solution Architect15:10:50

How is governance done? As code (policy, security etc) is Platform highest importance?

Fred Ghahramani15:10:06

where do I get the t-shirt .... Kessel Run ?

inactive15:10:10

Love all these sci-fi references. Undeniably such a part of this culture we’re in! 🙂

Steve Spear15:10:11

Rickover at commissoining of first nuc, the USS Nautilus. Could have all the sartorial splendor befitting an admiral. But he shows up in the gray flannel suit…

❤️ 1
Jason Nelson - Centil - product agility coach15:10:36

I hope we discuss some of the bottlenecks to platform consistency...

Ferrix Hovi - Head of DevOps - Siili15:10:04

Structure may make psych safety hard. However, commands sink in better if the psych safety is built in when the heat is off.

👍 1
💯 1
Mik Kersten (Project to Product, Tasktop)15:10:18

The “push through all levels of classification” is an interesting technical/organizational problem

🎯 1
👍 1
bsktcase16:10:10

I'm still back on Continuous ATO. Compliance office, your argument is invalid.

Jason Nelson - Centil - product agility coach16:10:27

yes - lack of a consistent cloud connection costs the gov't billions in local rework

Lauren Knausenberger16:10:42

Here's my take on the buy in piece. We started with laser focus on a problem that had lots of senior level eyes. The team started delivering big results with few resources and showed the obvious business case. After they broke the security barrier everyone realized it was not just innovation over there, that it could also defeat the bureaucracy.

📣 1
👏 9
👀 1
💪 2
🎯 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host16:10:01

This sounds like one of the stories in “War & Peace & IT”. Start small, set an objective, prove results, then double down.

inactive16:10:08

I love what @lauren.knausenberger — enable people to find their gifts, and reach their highest potential!

🎯 2
💯 1
inactive16:10:17

Like Steven Segal. 😂😂😂😂

😆 2
👍 2
Dwayne Holmes16:10:31

security is the hardest team to convince, and I don't know why

Nick - developer at BNPP16:10:01

Smart people solving problems in uncomfortable way - that's a picture I will remember!

inactive16:10:01

^^ which is why I loved CISO of Fannie Mae @christopher_d_porter talk yesterday!!!

👍 1
Lauren Knausenberger16:10:05

That's why we brought in the hacker teams. We've demonstrated that the new way is better.

💯 2
👍 1
🚀 2
Jason Nelson - Centil - product agility coach16:10:15

"ruthlessly attack legacy infrastucture." yes @lauren.knausenberger !

👍 3
🔥 1
Fred Ghahramani16:10:36

Lauren, how do you deal with military structure opposing your path? too many generals in the way

Lauren Knausenberger16:10:56

We actually find that generals get on board with a well structured plan. And if you have the data and business case to support it most people automatically jump on board.

inactive16:10:47

Congrats on your promotion to Deputy CIO of USAF, @lauren.knausenberger — a testament that the organization is valuing all the work you’re doing, and that they want/need you to do more!!! In service of an important mission.

🎉 2
🙌 3
🚀 2
Randy Shoup (Chief Architect eBay; he/him)16:10:54

Really inspiring. I love the continued refocus on the mission, and the ruthless application of modern practices

👏 1
❤️ 1
👍 1
Dwayne Holmes16:10:55

@afurtado LOVE that shirt!

Chris Gallivan, FCA, Builder of JOY16:10:01

Structure is the key and the lock

Ferrix Hovi - Head of DevOps - Siili16:10:02

AgileAF ❤️

❤️ 3
💛 1
🙌 2
Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:16

A couple great questions on platform consistency/parity... it's a nightmare use case tbh. Infrastructure of different types (commercial cloud, on prem, edge), on different closed networks (with classification and access difficulties), with networking complexity abound. Maintaining parity so the development environments are as close as possible to what the various production environments looks like

☝️ 3
😱 1
Jim DeMarre16:10:16

defeat the bureaucracy

Nick - developer at BNPP16:10:34

join the USAF, see the world 🙂

Dwayne Holmes16:10:44

@afurtado how much k8's are you using?

👍 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:46

We are currently in the middle of a shift from PCF to k8s. Almost all of our platform services are on k8s now, working to shift all the mission applications over as we speak.

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)16:10:03

@afurtado That's such a great way to leverage your DOES platform. Hands down my favorite session so far. You all have a really exciting vision.

👍 1
Rikard Ottosson - Psychological Safety (People Not Tech Ltd)16:10:05

So @afurtado, I didn't get that bit about ticketing systems with regards to Developer commnication? What was done?

Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:49

We still have a ticketing system... but we tried to make platform feature requests a little more customer focused and include human interaction to really understand what our customers need instead of just reacting to the nameless, faceless ticket system.

Rikard Ottosson - Psychological Safety (People Not Tech Ltd)16:10:25

OK, makes perfect sense, that does address the main problem with ticketing systems, but also lets the ticketing system solve the problem it was probably brought in to solve, i e track tickets.

Alyssa Lundgren - Centil - DevOps Engineer16:10:14

‘Amazingly complex problems you cannot solve anywhere else.’ 💯 @lauren.knausenberger

👍 1
Victor Suarez16:10:25

@afurtado @lauren.knausenberger Fantastic talk! Congratulations on fighting red tape and moving the Air Force to something bigger, better, faster... !

👍 1
Jon Smart [Better Value Sooner Safer Happier]16:10:35

Great progress and journey @afurtado @lauren.knausenberger Thank you for sharing.

👏 2
👍 3
inactive16:10:59

It was humbling to see how Spark Tank was designed to “enable airmen to solve other airmen’s problems” — people interested in seeing this remarkable event can watch it here: https://www.youtube.com/watch?v=4gqVVtkp_A8&amp;t=1031s

👍 1
Matt Cobby (NAB)16:10:12

@afurtado - Do you ever use SaaS products as part of your platform? How long does it take to go through your governance process at the moment? I work in a highly regulated environment (although not as much as USAF) and it's a slow process that often stops us even trying.

🙌 1
👀 1
inactive16:10:33

^^ IMHO, a wonderful model of attempting to enable innovation at scale at the edges, supported by core.

Steve Spear16:10:53

@lauren.knausenberger Great point about expanding industrial base. Huge flip in terms of who does the cutting edge R&D and tech innovation. Used to be .gov .mil spent and led and .com followed (50's and 60s). Total inversion now with .com innovation 80+% of the whole.

🤯 4
👍 5
🤙 1
Ferrix Hovi - Head of DevOps - Siili16:10:05

@genek101 Works on a personal level as well. Ideas come from the periphery.

Virginia L. NSA Technical Director16:10:07

@lauren.knausenberger preach. USG/DoD/IC easy is hard

👍 2
Dan Sloan, Cox Automotive16:10:37

@lauren.knausenberger - That definitely resonates ... the easy stuff can be what trips us up the most!

Paula Thrasher - PagerDuty16:10:56

I love the vision of not just being "good enough for government" but just being, the most world class software organization, period. Its audacious and achievable, but revolutionary.

Virginia L. NSA Technical Director16:10:25

originally that phrase was a complement

Paula Thrasher - PagerDuty16:10:59

I think there are some places where the government is still world class. Unfortunately, that has become an exception, and I agree with @steve773 that that change happened in the late 90s/early dot com era.

inactive16:10:09

@afurtado request to @lauren.knausenberger: how do we measure major programs on outcomes, instead of things like vanity metrics. “Wish granted.”

jeff.gallimore (CTIO - Excella, he/him)16:10:12

There’s an Ask Me Anything session with @afurtado during the morning break at 10:35am Pacific as a follow-up to the talk from him and Lauren this morning. You can get the link to join the discussion here: https://doesvirtual.com/ama-links

Matt Masuda - Quicken Loans16:10:12

@afurtado love the focus on outcomes

Lance Taylor16:10:17

@afurtado @lauren.knausenberger As an AF Reservist in the 3D career field and a leader in tech for financial services as my day job this is so exciting!! Watching the Air Force lead the way for the DoD is great and I would love to know how to get plugged in.

🙌 1
inactive16:10:29

Thank you, @afurtado and @lauren.knausenberger!!!

👏 2
👍 1
Dwayne Holmes16:10:32

great job!

👍 1
Larry Abel - PnC DevOps at USAA16:10:33

Thanks @lauren.knausenberger and @afurtado! Such an inspiring and motivating story.

🙏 2
👍 1
Fred Ghahramani16:10:44

awesome segment Adam and Lauren

👍 2
Austin Wimberly16:10:07

Thank you @afurtado and @lauren.knausenberger - great presentation!

🤙 1
👍 3
Steve Spear16:10:14

@paula.thrasher for what it’s worth, the phrase “good enough for government work” was once a compliment, that a company could claim that their quality reliability etc was good enough that even the demanding customers on the government side would be willing customers.

🤯 4
😂 3
💯 1
inactive16:10:44

OMG. That’s… incredible…

Paula Thrasher - PagerDuty16:10:08

I believe it. Someday I still need to have this conversation with you: How do companies and organizations go from great to bad? E.g. How did NASA go from moon shot (innovative, risk rewarding) to challenger disaster (low trust, messengers shot, risk adverse)? I think its fascinating how places lose greatness (so we can learn to avoid that)

Steve Spear19:10:10

@paula.thrasher Great question. Let’s find time to discuss this. Doing some significant work with companies that have had the from great to …. experience. Some ideas to share. <mailto:Steve@HVELLC.com|Steve@HVELLC.com> Maybe next week or the week after? Steve

👍 1
inactive05:10:57

cc @afurtado @lauren.knausenberger Re: that incredible quote from @steve773. It’s heart-breaking that this phrase has turned into the opposite of its original meaning! 😥😥. 😂😂. If there’s two people who can flip the meaning again, it’s y’all! 🙂

Adam Furtado, Chief of Platform, Kessel Run (USAF)12:10:12

I love it, @steve773. I've only ever heard it sarcastically I guess!

inactive16:10:20

Please welcome @dwayne!!! I’ve wanted him to share his story for years! And now, finally, he can! 🎉🎉🎉

Scott Bullitt Thompson16:10:39

Thank you, @afurtado for the reminder of Gall's Law.

Venkata L - American Airlines16:10:07

Inspirational speech, thanks for sharing your success story @lauren.knausenberger @afurtado

👍 2
👏 1
Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:15

Im going to figure out how to make our swag available. Response heard. 😂

🔥 4
👍 17
😎 2
Andy Nelson, Product Mgr - CH Robinson16:10:35

@afurtado Since you are a fan fiction hero does that include the matching NASA Apollo Mission control glasses as well? #AgileAdamFurtado

Christopher S Donahue16:10:00

Thanks, Great Story and Well Done! @lauren.knausenberger @afurtado

❤️ 1
Sandra Ortega - Director Storage and Backup - MetLife16:10:02

@afurtado @lauren.knausenberger Thank you for sharing your experience. Do you have internships for the development work you are doing where College Students can participate and help as they learn the skills they need to join the Air Force after graduation?

Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:08

We have a pretty substantial internship opportunities- they mainly are focused on future AF Civilians

Sandra Ortega - Director Storage and Backup - MetLife16:10:51

That makes sense. Thanks for responding so quickly.

Dave Mangot - Mangoteque16:10:07

Lol, LinkedIn knows everything about Dwayne and his previous company

Dwayne Holmes16:10:17

delete delete delete

😂 4
Adam Furtado, Chief of Platform, Kessel Run (USAF)16:10:24

Thanks for the kind words everyone- there is an AMA later if anyone has any more questions. https://doesvirtual.com/ama-links. 10:35am Pacific

🇺🇸 3
thankyou 2
🎉 1
Ferrix Hovi - Head of DevOps - Siili16:10:43

Micromonoliths... hmm :thinking_face:

Lauren Knausenberger16:10:11

Great job @afurtado. It's always great to spend time with you and @genek101. So glad everyone got some value out of hearing our story. Definitely join for the AMA.

👏 3
❤️ 3
thankyou 1
Lauren Knausenberger16:10:48

I hope everyone enjoys the conference.

inactive16:10:56

I’m in awe of the platform that supports all the revenue-generating applications, of which $30B - $50B of annual flow through!!!

Dwayne Holmes16:10:12

yeah food, and OTAs

Dave Mangot - Mangoteque16:10:23

Ohhh, didn't Sun used to call that network tone?

Lou Sacco16:10:40

#automate_everything

inactive16:10:02

“Dial tone: no one cares about the tech that powers the phone infrastructure”: “It just better work. Otherwise, the business gets upset.” 😂😂😂

🤙 1
1
😂 1
Dave Mangot - Mangoteque16:10:07

Ruby on Rails, all of a sudden the containers makes a lot more sense!

inactive16:10:42

(Although I was amazed when I saw my first cell phone switch. ESS-6? It was just a SPARCstation 2 with an SBus card in it. Was actually quite disappointing… 🙂

🎉 1
Luke Rettig (Sr Director Product Management - Target)16:10:52

❤️ Ruby on Rails. so good with containers

Lou Sacco16:10:18

Node.js+React even better! 🙂

inactive16:10:28

@dwayne observation of “Lawyer vs. Secretary” blows me away: even if lawyer can type, file faster than secretary, they should do lawyer work, not clerical work!

Dwayne Holmes16:10:39

lol, Ruby on Rails is awesome for fake devs like me...I'm on the node.js + react bandwagon now

👍 3
Luke Rettig (Sr Director Product Management - Target)16:10:00

i am also a fake dev. probably why i like it

😆 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host16:10:57

Rails is under appreciated. Amazing and shocking to hear it mentioned at DOES.

👍 4
inactive16:10:22

The opportunity cost is very high — similarly, “every hour they don’t focus on things that don’t have amazing business value” is a huge waste. Thus the incredible value of platforms to elevate dev productivity!

Dave Mangot - Mangoteque16:10:26

Maximize the work not done?

Fred Ghahramani16:10:39

love = "every hour that Developers does not provide value..."

inactive16:10:02

PS: Fans of Rails will LOVE the presentation from @eileencodes: closing keynote on Day 3. The 7 year journey to upgrade GitHub from Rails 2 to Rails 5!!!!!!! It’s such an incredible talk!!!

👀 3
Chris Gallivan, FCA, Builder of JOY16:10:28

A promotion to Developer !

😂 4
inactive16:10:28

“…moving to this hotel company might have been worst career move ever; from VP to contractor; project could be cut… and from view of harbor to sitting at table, in room with no windows.” 😂😂😂😂

inactive16:10:54

“most people said I was a fool.” 😂❤️. <--- @dwayne

Brandon Brown - Nike - Sr. SW Engineer II16:10:55

@dwayne love that courageous move!

Mik Kersten (Project to Product, Tasktop)16:10:14

Just noticed this went live a moment ago, more stories from that amazing Kessel Run trip that @genek101 @jeff.gallimore and I had, from @afurtado’s colleague Bryon Kroger. Like the amazing preso we just heard, some mind expanding learnings from their experiences! https://projecttoproduct.org/podcast/bryon-kroger/

🔥 4
jeff.gallimore (CTIO - Excella, he/him)16:10:26

What an amazing trip. One of the highlights of my year.

Mik Kersten (Project to Product, Tasktop)16:10:16

@dwayne That story on only getting the worst slots, geez!

Dwayne Holmes16:10:31

yeah, 12-5 was horrible

Luke Rettig (Sr Director Product Management - Target)16:10:48

no time like the present to run a perf test 🙂

Dwayne Holmes16:10:37

back to the future and tell my future self

inactive16:10:33

“because we weren’t revenue generation: so only slot we could get environments at midnight to 5am; legacy teams had the best slots” 😭😭😭😭

inactive16:10:09

cc @UBA0KD19P 😂😂😂😂

Dwayne Holmes16:10:39

my wife almost left me 🙂

🙈 4
👀 1
😭 2
😲 1
Mik Kersten (Project to Product, Tasktop)16:10:22

Glad that she didn’t and that we are getting the learnings from this war story. I do miss awk…

Dwayne Holmes16:10:37

yes, I kid...she was amazing

inactive16:10:05

From @jsnover13: “bash: the disease that you don’t die of, but die with.” 😆 😆 😆 😆

1
inactive16:10:41

cc @U015KPGSEF8

Brandon Brown - Nike - Sr. SW Engineer II16:10:13

there's linux, then there's everything else...

👍 1
Ferrix Hovi - Head of DevOps - Siili16:10:25

@mik You can return to awk any day. If you have a mac, you probably have it already 😄

Mik Kersten (Project to Product, Tasktop)16:10:06

LOL, I really should, and yes I bring up the shell on my mac on occasion still

Dave Mangot - Mangoteque16:10:34

"Containers are immutable!", Well, it depends.

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host16:10:39

Dwyane dropping evolutionary wisdom right now

Matt Cobby (NAB)16:10:39

That was my "A-ha!" moment for containers when I realised that they could just be a single line bash script all the way through to a bloated app. Most enterprise tech focussed on the bloated end in their marketing.

👍 2
inactive16:10:05

“we learned [to what extent] a small platform team could support many other teams”

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host16:10:12

So true about containers on VMs, then onto orchestration

Ben Williams - Arvest Bank - Sr Data Pipeline Dev16:10:37

My Data Pipelines are all CLI driven, they just have fancy UIs that call them

Dwayne Holmes16:10:38

we wrote two orchestration engines for containers on VMs before helm and kustomize

Ferrix Hovi - Head of DevOps - Siili16:10:42

Not a given that you get all six.

Dwayne Holmes16:10:55

we were running go templates before helm

😎 1
inactive16:10:56

^^^ totally. What an audacious ask from @dwayne!

Mik Kersten (Project to Product, Tasktop)16:10:11

@dwayne So did I connect the dots right, that you and your team figured out this framework twas catalyzed by wanting to avoid needing to be up 12-5am?

😳 1
Ferrix Hovi - Head of DevOps - Siili16:10:17

"Development tools"? Are you mad? NOBODY GETS TOOLS!

Dave Mangot - Mangoteque16:10:40

Really setup for success there in 2017 Dwayne 😮

Dwayne Holmes16:10:45

basically, we had to be extremely efficient because we were going to be up at night

❤️ 2
Pete Nuwayser IBM he/him/his16:10:52

The only thing worse than a "DevOps Team" is "Another DevOps Team"

2
😂 3
Matt Cobby (NAB)16:10:41

But every app has a DevOps team! How else are they going to get a pay rise!?

Pete Nuwayser IBM he/him/his16:10:54

SHHHHH DON'T GIVE IT AWAY MATE

Dwayne Holmes16:10:01

lol, assembly line

inactive16:10:59

I love that the Ops organization not only didn’t understand what @dwayne, and had to compete with another platform initiative!!! “Once again, I’m all by myself!”

Ferrix Hovi - Head of DevOps - Siili16:10:25

Service Catalogs, what a novel idea.

Mik Kersten (Project to Product, Tasktop)16:10:53

Now that’s a good leading indicator metric

Mik Kersten (Project to Product, Tasktop)16:10:01

“Devs chose our pipelines 100% of the time”

❤️ 5
🙌 3
jeff.gallimore (CTIO - Excella, he/him)16:10:29

That’s such an important and profound statement.

Chris Gallivan, FCA, Builder of JOY16:10:11

No developer ever asked for a catalog

👍 4
inactive16:10:30

@dwayne and I were joking about how frustrating it could be to see all the compute instance types available in cloud providers. “I just want to run my code. I don’t want to learn about all 80 VM types, how much memory, disk.“😂

👍 2
🎯 1
Nick - developer at BNPP16:10:53

git commits VS the marketplace. Brilliant!

🤯 1
inactive16:10:45

A profound observation — sometimes we don’t actually want choice. I just gas . I don’t care what the blend is.

👍 4
Dwayne Holmes16:10:53

@chris.gallivan we had base images for everything tomcat, Nginx, apache, java (play/spring), etc

👍 2
Chris Gallivan, FCA, Builder of JOY16:10:26

We’ve been playing with terraform for similar thing

Dwayne Holmes16:10:03

so they just had to use the pipeline

Ferrix Hovi - Head of DevOps - Siili16:10:10

@mik That's a great indicator. I once deleted the virtual machines with "my baby", the test tooling I had been developing as my main job. The reason: devs loved the new option and this was the way to get testing adopted.

👍 1
Camilo Piedrahita - Bancolombia - IT Manager16:10:27

@dwayne do you have service now integrated to pipeline? any step in the pipeline or batch?

Eduardo Rodrigues Semensati (Procter and Gamble)16:10:31

@dwayne: is this framework available in the slides in high quality? My architect hat tells me I need to take a closer look into that ;)

1
Zell Gagnon16:10:35

Onsi Fakuri (Cloud Foundry) said it most artfully: Here is my source code Run it on the cloud for me I do not care how

🙏 1
☝️ 1
Pete Nuwayser IBM he/him/his16:10:39

Automate away your toil ➡️ Create capacity to solve more problems ➡️ get more problems to solve

🤙 1
Dwayne Holmes16:10:19

@capiedra we were very SLOW in implementing SNOW. They have zero vision on how to implement K8's with SNOW

Ferrix Hovi - Head of DevOps - Siili16:10:29

So, service catalogs always lose

Dwayne Holmes16:10:32

the last release was supposed to help

😂 2
Evangeline K16:10:23

moving to cloud is new for me, can anyone recommend a checklist for cloud security (vs hosted/on prem security) ie. did I miss the details of how @lauren.knausenberger streamlined the long lead time to approve for deployment? is containers still the preferred destination vs ie. Azure PaaS?

Matt Cobby (NAB)16:10:06

It's very dependant on your own internal governance rules but a rule of thumb is that Azure will get you somewhere faster in governance as MS understands enterprises better. AWS is ore of a build it yourself approach to governance. It's a long process to asses each service one by one. Start with SoxII compliance as a standard for each cloud service.

Evangeline K16:10:50

thx @U016879K6L9

Dwayne Holmes16:10:30

@rodriguessemensati.e you should come to my AMA tomorrow, norm (my architect who helped me the last half of my journey) will be on. We can provide slides and answer any questions

👍 1
Eduardo Rodrigues Semensati (Procter and Gamble)16:10:00

awesome, thanks man! Let me check the calendar and ensure I have that one tagged

inactive16:10:37

A moment of rare “dev rage” for me: when I went through the flowchart of “which database do I use?” https://cloud.google.com/products/databases. cc @rshoup 😂 (I ended up using MySQL, because all the choices frustrated me)

Rikard Ottosson - Psychological Safety (People Not Tech Ltd)16:10:38

"Developers are all like 'I't's the network'..." I feel attacked

😂 3
🙏 1
Andy Burton16:10:54

I'm fighting the container vs. service catalog fight at our company as well. Thanks for both acknowledging this fight and giving me some new ammo 🙂

Lou Sacco16:10:19

How about Rancher?

Lou Sacco16:10:38

Did you guys consider for K8s management?

Dwayne Holmes16:10:42

@occasl if you want multi-cloud, I think you need to standardize on a platform

Matt Cobby (NAB)16:10:47

99.9% of you don't need multi-cloud. Don't do it unless you have an extreme use case

srujit biradawada16:10:46

I'm very positive SNow Orchestration will lead to future end-to-end automations and self-healing applications 😊

Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:51

@occasl we are using Rancher.. it is great.

💯 1
Dwayne Holmes16:10:06

we were using Open Shift....but we were planning to switch to Anthos from Google

Andy Burton16:10:23

We've moved from OpenShift (container customer 1) to K8S w/ Rancher. Seems so much more straightforward and automatable.

Lou Sacco16:10:17

Us too...early adopters since 1.x

inactive16:10:19

Such an interesting point: “high touch, concierge level service” wasn’t what @dwayne wanted to do. He gave this to the competing DevOps/SRE teams, because he wanted to focus on platform development! cc @dan.sloan

Craig Larsen - he/him - Solution Design Group Mpls16:10:26

@dwayne compelling story (about doing a rollback taking minutes instead of hours). Love this!

❤️ 1
Myles [Sooner Safer Happier]16:10:51

"Ops product teams" - makes complete sense.

💯 4
Rikard Ottosson - Psychological Safety (People Not Tech Ltd)16:10:01

Yeah @dwayne this is good stuff. Very good stuff

Dan Sloan, Cox Automotive16:10:22

This is a terrific session @dwayne! Love how you describe "productizing"!

💯 1
Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:26

K8s is part of our rebellion movement,

Ferrix Hovi - Head of DevOps - Siili16:10:32

Investing in the right tools. I hate how rare this is.

Dwayne Holmes16:10:41

k8's and base images are game changers for ops

thankyou 1
Evangeline K16:10:52

there are such inspirational talks here, thanks “Once again, I’m all by myself!” - resonates

❤️ 1
Dave Mangot - Mangoteque16:10:56

I'm a sucker for a good Ops story, this is great

Chris Gallivan, FCA, Builder of JOY16:10:58

Anything thoughts on terraform vs containers ?

inactive16:10:18

“no one likes being told at Stop #90 that they need to go back to Step #1” 😂😂😂

Dwayne Holmes16:10:19

@chris.gallivan two different things; terraform is for infrastructure....containers is for apps

inactive16:10:39

“Kubernetes is hard”.

👆 5
💯 3
👀 3
Ferrix Hovi - Head of DevOps - Siili16:10:09

Reminds me: "Do you need kubernetes or a solution?"

Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:17

getting old WOW engineers to adopt K8s is hard…

Dwayne Holmes16:10:19

people should focus on consumption of K8's not installation

Dwayne Holmes16:10:32

@denver.martin that is why we abstracted

✔️ 1
❤️ 1
Ferrix Hovi - Head of DevOps - Siili16:10:49

@dwayne Maybe installation is hard if you need several people for it 😄

inactive16:10:02

@dwayne platform supported the work of thousands of developers, both internal and those in large system integrators. (Did I get that right, Dwayne?)

Lou Sacco16:10:06

@dwayne I think I'm being inspired to take this to the next level.

❤️ 2
Dwayne Holmes16:10:07

DevOps teams are the new awesome! Legacy teams pay the bills....abstraction was our way to give DevOps power to legacy teams

Derick Stenftenagel - Director - Cloud and Platform Svcs, Edward Jones16:10:14

@dwayne Why did you feel 5 cloud providers was important?

👀 5
Rikard Ottosson - Psychological Safety (People Not Tech Ltd)16:10:28

"You don't need to run Kubernetes today. " That's my takeaway

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:11

the only good reason to explore kubernetes IMHO is CI/CD

Dwayne Holmes16:10:34

start small and run containers in VMs

👍 1
Dwayne Holmes16:10:43

then go to k8s when you are good with containers

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:32

That I am not sure of. We did this and found no value in it other than learning to write Dockerfiles well. In fact we had to build quite a lot of tooling to orchestrate containers. it gets ugly

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:45

it took us a little over a year to migrate to kubernetes. before that we had spent almost 6-9 months experimenting with docker and docker-compose for CI/CD.

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:15

unfortunately docker-compose cannot be used in production. so we ended up with dev/prod disparatiy, leading to frequent drift in configuration

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:45

eventually we decided to go to kubernetes purely CI/CD and took it to prod for dev/prod pairty

Dwayne Holmes17:10:22

docker is not k8, however I believe people need to get good with managing containers before they then migrate to orchestration with K8

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers17:10:13

i meant to say that we didn't really find a lot to learn about containers outside of kubernetes, at least from an application perspective.

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers17:10:24

even from an ops perspective i feel there wasnt much to learn outside of kubernetes. just enough operational knowledge of docker was enough for us. that IMHO cxan be gained by just using docker for developmeent as well

Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers16:10:31

Funny thing is Kubernetes out of the box doesnt work for anyone. It's mistaken for PaaS. It is a lot of other components on top of Kubernetes that streamline workflow and abstract complexity is what makes a good PaaS.

Michael Winslow16:10:46

Fighting against everyone ... amen

Dwayne Holmes16:10:48

@genek101 you are correct, we supported multiple service providers and internal devs

Brandon Brown - Nike - Sr. SW Engineer II16:10:17

@dwayne love this story. Thanks for being here.

💯 2
Luke Rettig (Sr Director Product Management - Target)16:10:22

love this @dwayne - your career path is so awesome

Ferrix Hovi - Head of DevOps - Siili16:10:24

Do not pass up promotions... true... maybe I still should have 😄

Lucas Melo16:10:47

@dwayne Are your teams considering Cloud specific PaaS services today?

inactive16:10:51

“I was offered a VP role in 2017; I didn’t take it because I wanted to be hands on keyboard, and I didn’t like politics. But that decision was did a disservice to our team, because our destiny wasn’t in our control.”

Michael Winslow16:10:33

@dwayne We should probably talk about this. I'm not a VP yet, but I made the switch. And I also feel like I did a disservice sometimes!

🙌 1
Matt Cobby (NAB)16:10:24

This is the paradox of every tech career. We love to build but if we are successful we end up moving away from what we love.

Luke Rettig (Sr Director Product Management - Target)16:10:32

this made me thing about @scott.prugh going from a 10x developer to a 10x leader. I think Career Pathing is a big theme that we haven’t traditionally talked about at DOES

👍 2
Matt Cobby (NAB)16:10:50

Maybe it should be a topic for the next one @genek101

inactive18:10:41

Roger that. Cc @jeff.gallimore @scott.prugh @UDL8V8FFS

👍 1
Dwayne Holmes16:10:03

@brandon.brown2 thank you!

Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:05

@dwayne agreed, that is why we made K8s our rebellion path, where we did the skunk works approach and then shared with eng and devs as we heard of the needs… now it is in the open and is seen as Containers 1st with new dev, and starting to look at retro older apps now..

Rasika V16:10:08

great talk ! thank you @dwayne \

Dwayne Holmes16:10:40

@rasika.vaidya thank yoyu

inactive16:10:16

@dwayne will indeed be hosting some awesome sessions later in the conference — @jessicam and @jeff.gallimore will post details on when/where!

👍 1
💯 1
Maya Leibman16:10:17

@dwayne great talk! Loved the combination of technical and leadership messages. We are on a similar journey at American Airlines. Let’s connect!

👍 2
❤️ 2
Eduardo Rodrigues Semensati (Procter and Gamble)16:10:25

@dwayne: security in the container, I saw Artifactory in the diagram, were you using XRay as well as its IDE plugin as well or something else?

Dwayne Holmes16:10:27

@lucas.demelo no, I'm not a fan of AKS, GKE, EKS because they are different....as a dev you want consistent APIs...same with K8s

👍 1
Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:42

Voting for K8s deep dive as a session…. @dwayne for Pres, write in ballots…

😀 1
👍 1
1
1
Randy Shoup (Chief Architect eBay; he/him)16:10:56

@dwayne I really want to hear the talk you are describing now 🙂

Dwayne Holmes16:10:59

@rodriguessemensati.e Artifactory Xray, Snyk IDE

Myles [Sooner Safer Happier]16:10:03

"forensics with containers" would be great to hear how this was addressed

✔️ 1
Steve Gertz16:10:13

Loved it

👏 1
inactive16:10:14

Thank you, @dwayne!!!!!

👏 2
Pete Nuwayser IBM he/him/his16:10:15

Whoo hoo @dwayne that was terrific! 👏

Andy Weldon16:10:16

@genek101 @dwayne Another vot for the deep dive!

Dwayne Holmes16:10:17

@maya.leibman thank you

James Simon, FCA, Solution Architect16:10:20

You can certainly come do it with us :)

Jess Meyer - IT Revolution (she/her)16:10:20

@dwaynes follow up session is tomorrow during lunch!

Ben Williams - Arvest Bank - Sr Data Pipeline Dev16:10:25

I want to hear about env var use, because I haven't had good luck with them.

Barry DeBlieck16:10:28

Wow! That was out of the park!

inactive16:10:34

Please welcome Dr. @steve773!!!

👋 5
🐿️ 1
Joseph Laihee16:10:35

That was awesome

Steve Gertz16:10:37

@genek101 Deep dive for K8!

Dwayne Holmes16:10:50

@denver.martin awesome to hear! that is what we did

Rob Parkhill, Director SW Engineering, Hexagon AP16:10:53

Awesome talk! Thanks for sharing @dwayne

🙌 1
Yelena Gouralnik16:10:05

Deep dive on K8s and Docker is a must! Awesome job @dwayne

4
Steve Spear16:10:23

How this next guy @steve773 doesn’t stink.

😂 11
😱 1
Matt Cobby (NAB)16:10:59

High Velocity Learning - Love that term.

🤙 2
Shalini Nair16:10:03

@dwayne Great perspectives... thanks for sharing !👏

Steve Spear16:10:05

@nickeggleston Thanks!

Dwayne Holmes16:10:26

@bwilliams4 we used environment variables for conf. We were hosting env variables in hashi vault which was proxied through aqua securityy

👍 1
Nick - developer at BNPP16:10:29

Not storing them at git?

Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:27

about to check off one of my bucket list items — hear @steve773 give a speech at a conference…

🙌 2
3
👍 2
Mik Kersten (Project to Product, Tasktop)16:10:46

This concept of structure and dynamics is so important in my opinion. The foundation for first principles of everything we’re doing.

👍 5
🤙 1
Steve Spear16:10:13

@denver.martin Thanks Denver.

Courtney - CTO at Zulily16:10:01

Yes!! Relationships FTW!

🎉 6
Steve Spear16:10:15

@chawklady Thanks.

Ferrix Hovi - Head of DevOps - Siili16:10:16

Top-3 slides in the conference

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host16:10:51

Love the energy in your presentation @steve773 😄

5
💯 6
🤙 1
🎯 1
Steve Spear16:10:05

@adam Thanks!

Marc Boudreau (Enterprise Architect)16:10:32

@steve773 always enjoyable 🙂

Kate Chapman - Wikimedia Foundation - Systems Architecture16:10:44

was chatting with my boss and said "sorry gotta go to this talk" and then gave the title and left our call 🙂

🙌 2
inactive16:10:44

“so many problems. all the time!”

Michael Winslow16:10:46

So many stories and lessons come out of studying Disney history!

Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech16:10:47

@steve773 - @genek101 mentioned the "structure and dynamics" when I interviewed him for my People Before Tech book and asked him why we should ever bother with the concept of "organisation" so you're in my book 🙂

jeff.gallimore (CTIO - Excella, he/him)16:10:55

Can he keep this energy level up for the whole talk???

👍 1
Ferrix Hovi - Head of DevOps - Siili16:10:57

Disneyland is closer to DevOps than 19th century Japanese navy. But still looking for the arc of the story to appear.

Ferrix Hovi - Head of DevOps - Siili16:10:13

Okay, so we are going back to Japan now.

Dave Mangot - Mangoteque16:10:03

LOL, @jason.cox on the slide 🙂

inactive16:10:18

The moment when @jason.cox and @brian.l.scott appear in @steve773 presentation!

👍 1
🎉 1
Michael Winslow16:10:34

He mentioned the bracelet! We used that story in our Forum paper!!! https://myresources.itrevolution.com/id006657108

❤️ 3
🙌 1
Eduardo Rodrigues Semensati (Procter and Gamble)16:10:58

@steve773: very energetic speech and awesome bow tie :thumbsup:

Scott Prugh - Chief Technology Officer - CSG16:10:11

Yes!! We have to be solving problems collaboratively!! @steve773

Nick - developer at BNPP16:10:11

I dig the bow tie!

✔️ 1
Ross Clanton16:10:13

Amazing talk @dwayne. Very relevant for problems we are looking to solve here at AA in terms of how we are managing our infrastructure.

🙌 1
Matt Ring (he/him) - Product/Engineering Coach, John Deere16:10:17

Anybody else have a big stupid grin watching @steve773's talk? 😁 I've watched a few of these and I frickin' love the energy. Also, will @genek101 try and subtly shoo him off the virtual stage this year if he goes over time again? 😆

😁 3
Steve Spear16:10:21

@eduardoatubisoft bowtieclub

bowtie 9
Eduardo Rodrigues Semensati (Procter and Gamble)16:10:43

wrong Eduardo, but I am obviously IN haha

👍 1
Bryan Finster - Walmart (Speaker)16:10:48

Shattered Sword explains why assigned roles on a team are a pretty bad idea. 🙂

👍 1
Ferrix Hovi - Head of DevOps - Siili16:10:11

@steve773 Do you make it a principle to approach from afar every time?

Philip Day16:10:12

@steve773's talk about the Battle of Midway was one of my favourites at DOES20 London - incredible insight and energy

❤️ 2
Steve Spear16:10:36

@philipday Thanks!

Steve Spear16:10:49

@chris.gallivan covid’s coming.

😳 1
Ferrix Hovi - Head of DevOps - Siili16:10:17

When does Psych Safety pop up?

Michael Winslow16:10:31

INPUT = Ignorance / OUTPUT = Understanding!!! -- Brilliant!

🤙 1
Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:37

hmm, Goldratt would say it is to make money…. hmm

👍 3
Robyn Talbert, American Airlines16:10:54

@steve773 Great energy and excitement - Yesterday was the Navy's birthday. USN!

Steve Spear16:10:16

@robyn.talbert Anchors away!

Matt Masuda - Quicken Loans16:10:28

Whatchu talking bout, @genek101?

😄 1
inactive16:10:34

I thought this was such a profound observation: exactly how problems are solved are invisible and ephemeral, tacit processes, was a shock.. Then I read this tweet: https://twitter.com/cm_stead/status/1314079840354541571 “Something really important about writing source code: Writing the code is not the work. The code is an artifact OF the work.” 🤯

🤯 4
Michael Winslow16:10:53

"Whatcha Talkin' Bout, Willis" reference! What does @steve773 not know???

Pete Nuwayser IBM he/him/his16:10:19

It takes diff'rent swarms to solve the world...

Ffion Jones (Partner, PeopleNotTech)16:10:54

All the pictures of how all the pieces fit together...but how do you create the ideas?

Steve Gertz16:10:21

Have this type of problem all the time; license to innovate in the organization

inactive16:10:23

😂 @steve773 rattling off names of people involved in recording his session: @jessicam @erin @annp

Luke Rettig (Sr Director Product Management - Target)16:10:25

go back and rewatch @steve773’s talks back to back. how he ties structure and dynamics into every talk and pieces them together is truly mad scientist. Rewatching lit up so many light bulbs that i never caught the first time.

Ferrix Hovi - Head of DevOps - Siili16:10:35

Supercell CEO said that in his days at Digital Chocolate successful games were invariably created despite the process.

Ffion Jones (Partner, PeopleNotTech)16:10:44

Dominant architecture forming the organisational structure - and culture?

👍 2
1
inactive16:10:17

(Meta-comment: the way that @steve773 uses the medium of the “small screen” video on web format is dazzling. I re-recorded all my introductions after seeing what could be done in this medium. I thought the best you could get was a news anchor, reading the news; then I realized how much more it could be…. My intros are a pale shadow of Steve’s, but much better than before. 🙂

👍 2
Jack Vinson - flow16:10:36

Steve is almost as high-paced as Gene. Hard to stop and chat.

😀 3
Steve Spear16:10:50

@jackvinson I’m with ya Jack!

Ffion Jones (Partner, PeopleNotTech)16:10:41

The president of the entire system didn't care? Shocked...

❤️ 1
inactive16:10:24

Not uncommon! the “up 8 and down 3” problem. Or The Square in The Unicorn Project!

🔲 1
👍 2
inactive16:10:56

How do you translate and convey tacit knowledge across generations?!

Steve Gertz16:10:58

How good is your on-boarding, documentation, and how often do we update those docs

Ferrix Hovi - Head of DevOps - Siili16:10:00

Okay. I'll have to watch this taking notes at 0.5x speed

✔️ 7
🐢 1
Jack Vinson - flow16:10:36

This is a great argument for classic knowledge management. (And why we struggled with it - continue struggling with it.)

Steve Spear16:10:56

@pnuwayser I think Gene must have a replay button somewhere…

Pete Nuwayser IBM he/him/his16:10:53

oops - looks like I have an old account from a previous DOES here somewhere

inactive16:10:27

Yep! Will be in Watch Library by break time! http://videolibrary.doesvirtual.com/

Ben Williams - Arvest Bank - Sr Data Pipeline Dev16:10:13

The York Minster Cathedral took 252 years to complete. Construction began in 1220 and ended in 1472.

🤯 1
1
Steve Spear16:10:48

@bwilliams4 I think St John the Divine is still underway…

Jack Vinson - flow16:10:03

Isn't Gaudi's cathedral still going?

😀 1
Ferrix Hovi - Head of DevOps - Siili16:10:15

Trains, Disney, Japan, Breast Cancer... okay. Did I miss something?

Eduardo Rodrigues Semensati (Procter and Gamble)16:10:45

There was something with the navy and war books too?

Daniel Lemire16:10:50

reminds me of James Burke's Connections

💯 2
Marc Boudreau (Enterprise Architect)16:10:13

@U01AM405MAT was thinking the same

Ferrix Hovi - Head of DevOps - Siili16:10:24

Okay. I'll watch this finish line camera speed later

Ganga Narayanan16:10:41

Ok, I need to rewatch the recording, take more notes!

Myles [Sooner Safer Happier]16:10:57

"cancer treatment is emergent work"

Steve Spear16:10:02

@jackvinson Gaudi? Dude was a MADMAN! Fantastic place.

2
Jack Vinson - flow16:10:18

Is this a sign of complexity or human processes (or both)? The battleships in the WWII era were built in a few years.

Ben Williams - Arvest Bank - Sr Data Pipeline Dev16:10:14

atoms are harder than bits, or at least that is the theory

Pete Nuwayser IBM he/him/his16:10:30

@genek101 we really need to encourage @steve773 to use a wider shot so that we can get the full effect of his hand and arm movements. That, or we need three screens.

inactive16:10:31

Didn’t post this ten minutes ago: Anyone thinking of Conway’s Law right now? This was a huge, huge and recent aha moment for me. The Dominant Architecture encompasses all implications of Conway’s Law! cc @rshoup @scott.prugh

👍 5
😎 1
Nick - developer at BNPP16:10:31

this. And the Inverse Convey Manoeuvre

Scott Prugh - Chief Technology Officer - CSG16:10:51

Conways Law is a case of Dominant Architecture and Tools coding itself into the org...

Brandon Brown - Nike - Sr. SW Engineer II16:10:37

enter ML: deal with the huge complexity, almost incomprehensible complexity.

Nick - developer at BNPP16:10:40

I thought that the way we solve problems is actually well researched

Marc Boudreau (Enterprise Architect)16:10:44

I love the whack-a-mole anology

1
Jack Vinson - flow16:10:28

holy crud. arbitrary inputs -> arbitrary outputs -> arbirtrary measures

🤯 2
1
🤙 2
inactive16:10:51

oof, right?

inactive16:10:30

yelled at, and graded by arbitrary metrics… Oof.

🎯 3
2
Ffion Jones (Partner, PeopleNotTech)16:10:52

#psychological-safety anti pattern!

inactive16:10:16

“who does what work, on behalf of whom, and when”

Denver Martin - Sr. Mgr Cloud Ops Infrastructure16:10:37

as the work changes, the things that get measured and how measured should change too..

Scott Nasello - Nike - Director of Digital Order Support16:10:40

I'm live slacking DOES talks, including one of my heros: @steve773 talk on the NIKE slack instance. It's my hack to try to get some external information to busy folks that don't read enough or don't attend conferences 🐿️

🙌 1
❤️ 2
💪 2
👏 3
slack 1
Jack Vinson - flow17:10:35

remember live blogging?

Chris Gallivan, FCA, Builder of JOY17:10:37

I’m doing same on google chat - a bit harder !

Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech17:10:40

Blimey that's cool! I used to do that on Twitter for Finovate presentations and demos back in the day. Will you come see us on Track 3 at 2:05 and if so any slides I can send you to make your chronicle-doing easier?:)

👍 1
Scott Nasello - Nike - Director of Digital Order Support17:10:09

@U015CDT342K - I have a theory that posting high signal information will encourage spontaneous water cooler discussion ...

Brandon Brown - Nike - Sr. SW Engineer II17:10:35

thanks for bringing it inside @UDKEYN3DH

👍 1
Scott Nasello - Nike - Director of Digital Order Support17:10:35

💯 @brandon.brown2 - I tagged you in that channel

👍 1
Pete Nuwayser IBM he/him/his17:10:10

value stream management?

Billy Jo (he/him)17:10:11

@steve773 Excellent talk, but quick feedback on visuals - white on cyan is really hard to see/read with the resolution of the stream

inactive17:10:12

…and would underscore absurdity when Advil decisions escalate all the way up to healthcare system CEO… so many similarities with Team of Teams story yesterday with @jessica.reif and @david627

😩 3
Steve Spear17:10:25

@denver.martin right on. as the situation changes, so should the measures.

Scott Prugh - Chief Technology Officer - CSG17:10:08

Yes!! A reference to Team of Teams @david627 @jessica.reif @steve773

🙌 1
Ffion Jones (Partner, PeopleNotTech)17:10:54

Wrong people connected the wrong way...the definition of collaboration gone wrong?

Steve Spear17:10:25

@ffion collaboration gone wrong? Chaos!

💯 2
🎯 1
Ffion Jones (Partner, PeopleNotTech)17:10:30

Chaos indeed...set the teams up before you start the work and let culture do some of the heavy lifting for you instead!

Larry Abel - PnC DevOps at USAA17:10:59

Similar parallels to the message in The Fifth Discipline by Peter Senge

Steve Spear17:10:00

@genek101 nice splice!

📹 1
Michael Winslow17:10:15

I just screamed when he said "SPEED!"

😆 1
😄 1
❤️ 2
Steve Spear17:10:40

@larryabel Thanks Larry. I arrived at grad school when 5th Disicpline, Re Engineering, and Machine that Changed the World at the same time.

💯 1
💪 1
jeff.gallimore (CTIO - Excella, he/him)17:10:43

“The key is speed.” As Gene says, “It’s not big beats small, it’s fast beats slow.”

👏 4
inactive17:10:14

First heard from Chris O’Malley, CEO, Compuware; who was quoting Rupert Murdoch.

Michael Winslow17:10:06

I personally will never go below 55mph!

Jon Smart [Better Value Sooner Safer Happier]17:10:36

Fast and small beats big and slow 🙂

Scott Prugh - Chief Technology Officer - CSG17:10:12

Yes!! Give time back to people to make them successful!! @steve773

💯 1
🔁 1
👍 2
Jon Smart [Better Value Sooner Safer Happier]17:10:35

Optimise people for speed to value

💯 2
Ffion Jones (Partner, PeopleNotTech)17:10:27

Absolutely but I was wondering if there was a correlation

👍 1
Jon Smart [Better Value Sooner Safer Happier]17:10:57

Economics of trust and collaboration, I agree. And an aligned incentivisation to the actual economics, over role silo based stank ranking incentivisation

Mik Kersten (Project to Product, Tasktop)17:10:01

“Structure drives dynamics”, YES, and why I’m still putting all my focus on helping orgs establish and measure value stream structures to break through the org chart/silo structures that have been ossified at the level above the delivery teams. @me1208 Team Topologies has a catalog of structures that support the dynamics of what we’re after. That really is all it takes. When I’ve seen orgs improve the structure at the team of teams level to support the dynamics of DevOps a thousand flowers bloom…

👍 7
Jon Smart [Better Value Sooner Safer Happier]17:10:50

Surely it's $ install -f spotify_model ? 😉

2
Paula Thrasher - PagerDuty17:10:17

I was also making the Project to Product and Team topologies connections - great segway to @mik talk up next

Courtney - CTO at Zulily17:10:27

Always amazing @steve773 - thank you!

👍 1
inactive17:10:28

Thank you, @steve773!!! I learn something in every interaction with him!

🙌 6
4
👍 3
Marc Boudreau (Enterprise Architect)17:10:46

Great as always @steve773! Stay awesome

👍 3
Ferrix Hovi - Head of DevOps - Siili17:10:59

Okay. Not the right presentation to be interrupted by a call from mom. Luckily I need to watch this a few times again anyway.

👍 1
😄 1
Ffion Jones (Partner, PeopleNotTech)17:10:21

Thanks @steve773 this was super useful!

👍 1
Scott Prugh - Chief Technology Officer - CSG17:10:29

Nice job @steve773 Always Delightful!!

👏 1
Manny Avila - Consulting Engineer, Cloud and Datacenter - JD Finish Line17:10:30

YES for tools...I want to know how you went from problem to solution...what did the decision tree look like?

Michael Winslow17:10:33

That was so great

Larry Abel - PnC DevOps at USAA17:10:38

Great reminder @steve773 Thank you!

inactive17:10:42

Please welcome Dr. @mik!

👋 2
Austin Wimberly17:10:50

`Loved that energy...great presentation!

👍 1
Jess Meyer - IT Revolution (she/her)17:10:52

Keep the conversations going in #ask-the-speaker-more !

Ross Clanton17:10:07

I always love hearing @steve773 speak... Awesome talk yet again....

🐿️ 1
👍 1
jeff.gallimore (CTIO - Excella, he/him)17:10:16

Fast start to Day 2, right! Give us an emoji to indicate your experience so far today. And get ready for some more great talks…

Michael Winslow17:10:16

I just invited all of my Tech Debt folks! Both Engineers and PMs.

Andrew Tam17:10:16

Thanks @steve773!!!

Ferrix Hovi - Head of DevOps - Siili17:10:17

The plenaries are not over and this feels a bit more overwhelming than yesterday. Soldiering on.

Ganga Narayanan17:10:19

To paraphrase Francis Bacon, some talks are to be watched with popcorn, some in a relaxed setting with a glass of wine, some with extra hot coffee, fully caffeinated, alert and charged up. And re-watched again. And again. To be inspired. I think Steve's talk one was the last category. 🙂 Thank you for the energy, Steve! Also interested to check out the website and the book!

Pete Nuwayser IBM he/him/his17:10:50

and some... with cayenne pepper 😄

inactive17:10:24

@jessicam @jeff.gallimore Can you re-post the link to @steve773 AMA, when / where? Thank you!

Mik Kersten (Project to Product, Tasktop)17:10:53

Thanks @scott.prugh! I’ll answer questions here live and then during the conf AMA at 5:30pm PT here: https://tasktop.zoom.us/s/94693819688

👍 1
Steve Spear17:10:19

@ganga.n some excerpts from the book…

🙏 2
👍 2
Ganga Narayanan17:10:19

Thank you so much @steve773!

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host19:10:08

@steve773 do you have a public link to this?

inactive17:10:31

I’m so glad @mik is taking time to explain more slowly the symptomology of what often goes wrong in software, using Flow Framework constructs — I wish I had understood this better before writing The Unicorn Project!!!

Mik Kersten (Project to Product, Tasktop)17:10:44

I think you understood it well enough 🙂

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host17:10:44

Now the talk title makes much more sense

2
👌 1
😂 1
Mik Kersten (Project to Product, Tasktop)17:10:03

But thank you for encouraging me to finally take more time to explain the Flow Metrics @genek101!

Duena Blomstrom, Psychological Safety Dashboard CEO, Author PeopleBeforeTech17:10:04

"Convoluted red form with a linear green attachment" 🙂

Mik Kersten (Project to Product, Tasktop)17:10:28

I am now going to start describing tech debt exactly that way 🙂

❤️ 1
inactive17:10:30

“not an eyesight problem, but instead visual agnosia.” Not much an ophthalmologist can do!!! What a powerful thing to know!!!

Mik Kersten (Project to Product, Tasktop)17:10:45

For me there is so much depth in that analogy!

Paula Thrasher - PagerDuty17:10:45

@mik I love the reference to this book. I read this in college (there are some other awesome stories in the book as well - its really about approaching problems differently)

Ferrix Hovi - Head of DevOps - Siili17:10:09

@mik Ah, so you need to keep current with awk to keep the occipital lobe working.

Craig Cook - IBM17:10:13

@steve773 Your talk reminded me of this book "The No-Limits Enterprise: Organizational Self-Management In The New World Of Work" Have you seen it?

Steve Spear17:10:49

Thanks I haven’t I’ll look it up.

Craig Cook - IBM17:10:44

I attended a meetup where the author talked about it. Very interesting organization model. Would be scary to compete against it.

Dominica DeGrandis, Author - Making Work Visible17:10:31

For people unable to see tech debt for what it is - see why it needs to be prioritized - this talk is for you! Thank you @mik

👍 2
Mik Kersten (Project to Product, Tasktop)17:10:00

@paula.thrasher Yes, so good. And putting 50% on tech debt blindly is like subscribing 1" think lenses to someone who has visual agnosia.

👍 2
Paula Thrasher - PagerDuty17:10:41

I think the reason flow metrics work better is that they talk to the qualities business cares about - speed to market. I think we often ask business "can I upgrade the FooBar system", when that is silliness to them. If we asked, can I work on something that makes me deploy days faster, we'd get a different answer.

Dominica DeGrandis, Author - Making Work Visible17:10:21

Tech debt creates scope changes which in turn are a source of tech debt.

inactive17:10:26

What a powerful thing for execs to fully internalize! The full implications of all those scope changes caused by Sales!

🎯 1
Ferrix Hovi - Head of DevOps - Siili17:10:35

The graph does look like a hat.

😆 2
inactive17:10:38

(no CI/CD improvements would fix that problem!!)

Ferrix Hovi - Head of DevOps - Siili17:10:32

maybe you fix the tech debt with more features?

😭 2
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host17:10:50

Flow Metrics = Business Vital Signs 💯

👍 4
Anna Noak (she/her) - IT Revolution17:10:21

Head over to the Tasktop booth to pick up a free copy of Project to Product (while supplies last).

Dominica DeGrandis, Author - Making Work Visible17:10:23

Measuring flow opens the door to see the health of your end-to-end value streams

🤙 1
💯 4
inactive17:10:42

Flow Efficiency: 100% is the awesome feeling when you’re on a single person project, no handoffs! (Finally learned that when we recorded this session.)

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)17:10:45

@mik Have you correlated Flow Efficiency to team happiness measures?

Craig Gorveatt (Tasktop)17:10:43

hey Andrew good to "see" you again... let me know if you had any other questions and we can catch up about Flow Metrics

👍 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host17:10:58

I’m getting goosebumps from this like when I read Project to Product. This is language we need to leverage senior execs.

💯 2
Mik Kersten (Project to Product, Tasktop)17:10:06

we see correlations between better flow efficiency and higher employee NPS

👍 1
🎉 1
Joseph Laihee17:10:22

That waiting for approval time is not long enough in the graph

Mik Kersten (Project to Product, Tasktop)17:10:32

2nd ideal in UP; Focus, Flow and Joy

Ferrix Hovi - Head of DevOps - Siili17:10:34

@genek101 100% flow doesn't help if you have 100 years of work in line for you due on Friday 😄

😂 2
inactive17:10:45

Referencing @scott.prugh’s amazing achievement described yesterday of improving Flow Load (% of time spent on new features) from 15% to 55%.

Mik Kersten (Project to Product, Tasktop)17:10:05

Also, low Flow Efficiency causes an increase in deadlines and escalations, higher Flow Efficiency

Ferrix Hovi - Head of DevOps - Siili17:10:58

@mik The low Flow Efficiency feels directly analogous to Kingman formula.

Mik Kersten (Project to Product, Tasktop)17:10:36

and lack of purple = no tech debt work visible

inactive17:10:36

I love these explanations because we’re seeing the causation between all these factors, and the symptomology expressed, both before and after.

Ferrix Hovi - Head of DevOps - Siili17:10:02

Hmm... Is this my company in the graph? 😄

inactive17:10:04

“72 work items blocked on Core Backend Services” !!! “painful legacy constraint”

Mik Kersten (Project to Product, Tasktop)17:10:15

That’s the interesting thing, it’s all causations in a dynamic system. Amazing to watch the over time. I want to create an animation over months…

💯 2
👍 2
👏 1
Jon Smart [Better Value Sooner Safer Happier]17:10:50

I guess you can see the trends over time?

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)17:10:25

Interesting that the field header for that field is "Project" 😉 Wouldn't that have some product-oriented name?

Ferrix Hovi - Head of DevOps - Siili17:10:09

@mik What's the source of that data? Ticketing?

Mik Kersten (Project to Product, Tasktop)17:10:27

Jira still uses that terminology. This org modeled their value streams over dozens of Jira projects.

Jon Smart [Better Value Sooner Safer Happier]17:10:33

@mik sounds like they had a push not pull system of work

Mik Kersten (Project to Product, Tasktop)17:10:22

Yup! Still sadly rare for me to see pull based systems.

👍 1
Mik Kersten (Project to Product, Tasktop)17:10:55

@ferrix: yes, work item data across tools including Jira, ServiceNow, ADO, GitHub, etc

inactive17:10:58

Favorite part of this story: “business kept putting more devs on the front-end systems; the EXACTLY WRONG THING TO DO! Must flow that talent to the Core Backend System and platforms!”

💯 4
🤙 1
Jon Smart [Better Value Sooner Safer Happier]17:10:21

If WIP was an enabling constraint, they'd have blocked, causing pain, to help address it? @mik

Mik Kersten (Project to Product, Tasktop)17:10:19

Yes I think so. The next diagnostic is exactly about that point.

👍 1
Ferrix Hovi - Head of DevOps - Siili17:10:26

Okay. So we could have could potentially dig those out from what we have. Neat.

inactive17:10:29

“they were doing the exact wrong thing: they were putting low cost contractors on the Core Backend System!”

💯 2
Paula Thrasher - PagerDuty17:10:38

So since @steve773 just talked about the importance of making the process of creating value explicit and visual - this is way of measuring and making the hidden work visible. I love the pairing of the talks this morning.

❤️ 3
👍 1
Dominica DeGrandis, Author - Making Work Visible17:10:41

THIS - Make all work visible - including tech debt!

👍 5
🙌 1
💯 1
Mik Kersten (Project to Product, Tasktop)17:10:56

Yup, in the end all we’re doing here is trying to deliver on Dominica’s vision to make all work visible, and to show the complex dynamics of that work 🙂

Ferrix Hovi - Head of DevOps - Siili17:10:18

The alternatives to making tech debt visible: kill it or die.

Jon Smart [Better Value Sooner Safer Happier]17:10:29

If pull based system of work, with WIP limited, it would raise the pain level to address it!

inactive17:10:46

I found this inspiring: they’ve gotten so good at improvement at Tasktop, improvement projects now must pay off within 6 months. (did I get that right, @mik)

Mik Kersten (Project to Product, Tasktop)17:10:51

ie, need to see flow payoff within 6 months now, even for major API/architecture/platform work

inactive17:10:31

PS: the idea of having more descriptive Slack usernames came from @mik when we debriefed after London conference. 🙂

❤️ 1
Vaidik Kapoor - VP Eng (DevOps, Security) - Grofers17:10:32

Negleccted WIP is the most common problem

jeff.gallimore (CTIO - Excella, he/him)17:10:33

Neglected WIP! Thief neglected work, @dominica!

😎 1
Dave Mangot - Mangoteque17:10:07

"IT can't keep up with the business", I've heard that one! 🙂

Mik Kersten (Project to Product, Tasktop)17:10:23

@jonathansmart1 That’s what you were foreshadowing I think 🙂

👍 1
inactive17:10:25

The now obvious doom scenario when incoming work far exceeds completion of work.

👍 2
inactive17:10:44

“any promise you now make is unlikely to be met”

😩 2
Mik Kersten (Project to Product, Tasktop)17:10:09

The amazing thing to me is how often we still see this happening given how well this community has documented these problems!

💯 2
jeff.gallimore (CTIO - Excella, he/him)17:10:58

…and alas, not everyone who needs to hear these messages is in this community yet.

Paula Thrasher - PagerDuty17:10:32

I think @dominica puts the reason on it: We say yes because it feels good (we feel important, helpful, nice) - saying no is socially hard.

👍 2
1
Levi Geinert17:10:48

Agreed, we all have similar issues, that our peers have often solved and shared how they solved; yet we have much work to do to implement in our own orgs.

👋 2
🙏 1
Ferrix Hovi - Head of DevOps - Siili17:10:18

This is really fancy. However a simple burn-up chart will show that you are adding too much scope. Interesting.

Jon Smart [Better Value Sooner Safer Happier]17:10:31

"Saying No doesn't make you an ass" 🙂 @dominica

❤️ 10
🤙 2
Scott Nasello - Nike - Director of Digital Order Support17:10:05

Saying Yes to incremental work is saying Maybe or No to prior commitments

💯 3
Dominica DeGrandis, Author - Making Work Visible17:10:50

Why saying No matters http://videolibrary.doesvirtual.com/?video=467719259 from the Andy Patton show w/ @jonathansmart1

🎉 1
Mik Kersten (Project to Product, Tasktop)17:10:28

@ferrix Years ago I would track this all with burn-ups. I don’t find that a good visualization, and often too narrow a measurement of the value stream. Similar for CFDs. It’s good info, but the dynamics of improvements, changes, don’t pop.

Jon Smart [Better Value Sooner Safer Happier]17:10:33

"Improving daily work is as important as daily work" @genek101

🙌 5
Mik Kersten (Project to Product, Tasktop)17:10:44

I should have made that point more clear, ie, that fixing neglected WIP will drive second ideal and eNPS

🙏 1
inactive17:10:51

PS: I love seeing this: virtual equivalent to excited murmuring in crowd. 🙂

2
😂 2
1
Ferrix Hovi - Head of DevOps - Siili17:10:51

@mik I can see that. It will make "there is a problem somewhere" very apparent, but does not help in finding the bottleneck

👍 1
Carlos Faria17:10:55

Would be amazing to have a deep dive techniques on how to implement Flow Framework! I´m working with my team trying to solve this puzzle

Mik Kersten (Project to Product, Tasktop)17:10:30

Happy to! Hit me up at <mailto:mik@tasktop.com|mik@tasktop.com>

😀 3
Mik Kersten (Project to Product, Tasktop)17:10:56

and amazing how quickly that is observable!

Michael Winslow17:10:35

So glad @mik has not mentioned any Large Cable Operators! :rolling_on_the_floor_laughing:

😆 5
1
😂 1
inactive17:10:03

“6 agile teams; 1000 flow items in progress. 1000 large features? seems like something is wrong… let’s dig in deeper…” 😂

😀 3
Eduardo Rodrigues Semensati (Procter and Gamble)17:10:13

love the example and how you structure the presentations @mik! Can definitely related to some of the examples and the guidance provided here is valuable information. Have been wanting to read the book for a while, time to pick it up 😉

👍 4
Lauren Kaye (Tasktop)17:10:22

We’re giving away e-book copies at our booth if you want to grab one 🙂

👍 1
Eduardo Rodrigues Semensati (Procter and Gamble)17:10:08

I think I already did, you are right!! I will start reading virtually, but I like real books and I need some more to fill the huge bookcase I bought for my new flat 😉

Lauren Kaye (Tasktop)17:10:09

Drop us on message on our xpo-tasktop slack channel and we’ll get a physical copy sent out to you!

❤️ 1
Danielson Samuel17:10:31

Done is Done when customer receives the value

💯 3
❤️ 2
Ed Marshall - Enterprise Agility - Deloitte17:10:57

@mik what do did you mean by other work in the comment about balancing the backlog and other work in the WIP case study?

Mik Kersten (Project to Product, Tasktop)17:10:58

I was referring to all of the work on the backlog, according to the four flow items. So a lot of tech debt work was happening, the other work I think was defects, features and risks. Does that answer your question? Not sure I fully understood.

Ed Marshall - Enterprise Agility - Deloitte21:10:34

Thanks Mik. That answers it. Much appreciated. I always enjoy your sessions and podcasts.

inactive17:10:00

Aha! By changing criteria to Dev Complete but Blocked Downstream, dev now is obviously not the bottleneck — it’s downstream in Ops, infrastructure, etc…

Ferrix Hovi - Head of DevOps - Siili17:10:11

1013/6 is amazing. I've seen ~200 stories for a "team" of 12 people.

inactive17:10:14

From 1000+ work items down to 280.

Bryan Finster - Walmart (Speaker)17:10:37

"We're done. Just need to test it"

😢 3
😆 2
😂 2
Scott Nasello - Nike - Director of Digital Order Support17:10:10

something something tree in the forest falling

inactive17:10:38

“Absence of psych safety made it difficult to actually share data about what was going wrong!”

💯 3
Dominica DeGrandis, Author - Making Work Visible17:10:12

Again - Psychological Safety topic ! Check out this topic in today's Lean coffee w/ @chawklady

Jon Smart [Better Value Sooner Safer Happier]17:10:40

Workflow Obscurity. 99.999% of organisations.

👍 1
Bryan Finster - Walmart (Speaker)17:10:10

People are afraid of observable delivery metrics.

👍 6
Levi Geinert17:10:38

Without safety

👍 1
Jon Smart [Better Value Sooner Safer Happier]17:10:26

Fear Driven Development @bryan.finster

😂 1
Jon Smart [Better Value Sooner Safer Happier]17:10:26

Fear Driven Development @bryan.finster

😂 1
Matt Cobby (NAB)17:10:29

I see relationship driven development. "It's who you know.... "

👍 1
Bryan Finster - Walmart (Speaker)17:10:20

Fear lead to hate. Hate leads to end users suffering.

😢 2
😂 5
Levi Geinert17:10:43

I wanted to click the laughy face, but its more 😢 face.

Mik Kersten (Project to Product, Tasktop)17:10:28

@bryan.finster That’s why the point of psycholigica safety is about to come up!

👍 3
Bryan Finster - Walmart (Speaker)17:10:09

100% We work very hard to build trust with teams so they can use the measures as functional tools.

Bryan Finster - Walmart (Speaker)17:10:30

We also yell at managers who disrupt that.

Dominica DeGrandis, Author - Making Work Visible17:10:55

Again - Psychological Safety topic ! Check out this topic in today's Lean coffee w/ @chawklady

🙌 2
Bryan Finster - Walmart (Speaker)17:10:27

My biggest regret, I can't hang out with Courtney!

😭 2
Michael Winslow17:10:35

"If you put any word before DONE, it means NOT DONE." - Uncle Duane 🙂

👍 4
🙏 1
Paula Thrasher - PagerDuty17:10:47

Treatment plan is Value Stream Mapping (shameless plug: I'm talking about how to do this later today Track 2)

🎉 3
Dana Finster - Walmart InfoSec17:10:02

amazing talk @mik! I am going to organize a viewing party for my team when the recordings are publicly available!

Mik Kersten (Project to Product, Tasktop)17:10:35

Thanks @UDB2Y762J! You can also request a session here: https://projecttoproduct.org

Dana Finster - Walmart InfoSec17:10:02

Great - Thanks for the info!

Matt Masuda - Quicken Loans17:10:09

@mik Love that you call out the Unicorn Project ideals

👍 2
❤️ 2
Mik Kersten (Project to Product, Tasktop)17:10:51

@U01D79RH3ME yes, it basically boils down to data-drive whack-a-mole! But the amazing thing is how much Flow Time can be reduced via the first 2 or 3 biggest moles.

❤️ 3
Danielson Samuel17:10:46

Measure the system from outside in

Michael Winslow17:10:49

A word cloud of DOES would have WACK-A-MOLE large and centered! 🙂

😆 2
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host17:10:54

Stop worrying about cycle time. Focus on Flow metrics. 💯

Jon Smart [Better Value Sooner Safer Happier]17:10:23

"A system of local optimums is not an optimum system", Goldratt

Randy Shoup (Chief Architect eBay; he/him)17:10:56

Slides are so clear, and just dense enough. Setting a high bar for exec communication!

Ferrix Hovi - Head of DevOps - Siili17:10:02

Two observations so far: 1. Things like WIP limits, VSM, flow etc. really work as tools 2. The tools are not exactly new and shiny 😧

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host17:10:05

I guess I can skip all that work assigned to me then 😄

Ferrix Hovi - Head of DevOps - Siili17:10:23

YES! If you know what you are doing, you are better off choosing your work instead of assignments.

Carlos Faria17:10:11

@mik Very good talk! If you can share this deck is going to be amazing, as well!

inactive17:10:12

Thank you, @mik!!!!

👍 3
👏 4
🤙 1
Jon Smart [Better Value Sooner Safer Happier]17:10:38

Nice one @mik, thanks for sharing. Great to see the real world examples

Ganga Narayanan17:10:40

Wow! That was amazing, @mik!!

💯 4
inactive17:10:55

Please welcome @lucasc5 and @lewir7, two auditor friends from Nationwide Insurance!

❤️ 1
👋 1
Ganga Narayanan17:10:08

As soon as I saw the "hat", I couldn't unsee it!

Philip Day17:10:23

Incredible slides + pace from @mik

Levi Geinert17:10:25

Audit panel has got a lot of runway at the Banks!

Ken Kennedy17:10:30

@mik Great presentation. To all managing large development teams, you need these metrics or you're overly dependent on anecdotal opinions. We're using Tasktop and seeing tremendous value.

🙌 3
💡 1
😎 1
Philip Day17:10:36

Yes the Nationwide talk in London was brilliant

❤️ 2
thankyou 1
inactive17:10:48

Yep, that’s right, everyone. We’re going to have two auditors teach us some awesome stuff!

👍 2
dgmorales17:10:38

Amazing talk @mik! The kind of talk you need to place your seat belt on, because he talks FAST! Loads of content. Awesome!

👍 1
Ganga Narayanan17:10:30

Today's talks have been on overdrive!

inactive17:10:59

“Oh, no… not the auditors. do they get paid by the finding?” 😂😂😂

👍 1
😆 5
😅 1
Levi Geinert17:10:13

"What were you doing @genek101?"

😂 3
inactive17:10:21

“in my ten years in audit… my paycheck has never changed based on # of audit findings.” 😆

Ferrix Hovi - Head of DevOps - Siili17:10:56

Hmm... are these just two benevolent auditors then :thinking_face:

😂 3
Nick - developer at BNPP17:10:26

That's an old trick. Good Auditor and Bad Auditor 😄

😂 5
Levi Geinert17:10:07

Works for architecture too! 😆

🙂 1
inactive17:10:37

^^ Hypothesis: maybe they’re the only two benevolent auditors on the planet? hahaha

😂 5
1
Jack Vinson - flow17:10:51

Role is to come as a different set of eyes. Nice.

Ferrix Hovi - Head of DevOps - Siili17:10:58

@genek101 The last good auditors?

Danielson Samuel17:10:00

I enjoy MythBusters

Nick - developer at BNPP17:10:16

what's bad with the checklist?

inactive17:10:44

Static checklists for dynamic systems almost always lead to tears… and accidents and disasters. cc @steve773

💯 3
inactive17:10:00

cc @lucasc5 and @lewir7

Jack Vinson - flow17:10:51

Checklists can be great tools, but easily mis-used (I have a checklist, I can turn off my brain!) 🙂

Philip Day17:10:55

penalty flags sounds a bit... auditor 🙂

😆 1
😭 1
Ferrix Hovi - Head of DevOps - Siili17:10:10

Okay. I can see where this is going. Auditors follow checklists, Ops follow checklists. That's a bond!

2
Bryan Finster - Walmart (Speaker)17:10:25

Can't checklist be automated?

2
🙌 1
Frotz Faatuai (Cisco IT - he/him)17:10:59

I’ve done that (automated my audits). Made my quarterly audit deliverables very simple (and delegated).

👏 2
David Blezard - University of New Hampshire17:10:09

@bryan.finster A checkmark for your checklist! And yes, checklists can and probably should be automated.

Bryan Finster - Walmart (Speaker)17:10:49

I've the opinion that manual verification of a rule is compliance theater.

EmanuelMedina - Bancolombia17:10:30

And every manual approval is an untrust point

Zell Gagnon17:10:23

checklists are about outputs, but auditing should be focused on outcomes, as with most activities

Ferrix Hovi - Head of DevOps - Siili17:10:02

I used to work in a company wanting to make finance and procurement friends. How hostile can the top floor be?

Nick - developer at BNPP17:10:07

checklist is just a tool.

Nick Eggleston (Nix) - IT Architect - (actively looking)17:10:22

So glad Audit is being included in the DevOps work 😊

❤️ 3
Ed Marshall - Enterprise Agility - Deloitte17:10:42

David Marquet has some good advice on working with auditors in Turn The Ship Around

👍 2
Jack Vinson - flow17:10:42

If you are doing things "just because they said so" please connect with your auditors to understand why.

👍 1
Ferrix Hovi - Head of DevOps - Siili17:10:43

@nickeggleston wait... is that going to be on my backlog soon?

Nick Eggleston (Nix) - IT Architect - (actively looking)17:10:49

It’ll automatically be added via template

Steve Gertz17:10:53

Personally, I see the checklists as ok; ISO appendix A has 30ish controls, we should have an answer for each thing

Patrick Gill, Product Owner - American Airlines17:10:58

The same way that there is dominant architecture in other industries, Audit has some pretty tried and true ways to test things. Partnership can be built in the way risk is assessed and scope is determined! I've been an auditor in the past and have had IT leaders invite us into the dev teams. It's possible to have the best of both worlds

🎉 3
inactive17:10:59

That’s awesome! I was due-paying member of ISACA and IIA for nearly a decade. In particular, I always admired how great the IIA was as a professional organization!

1
Patrick Gill, Product Owner - American Airlines17:10:51

After all of these years, I still carry my CISA cert in my email signature 😎 I learned more about systematic approaches during those years than I even realized back then!

Ferrix Hovi - Head of DevOps - Siili17:10:22

Crossfunctional team: designer, developer, sre, auditor...

😂 1
Clarissa Lucas, IT Audit Director17:10:15

@nick.kritsky Checklists are a useful tool, but often those using them stick only to the checklist. Sometimes they can't see the risk staring them in the face because it isn't on the checklist.

inactive17:10:40

“We love telling your leaders what a great job you’re doing!” — @lewir7

❤️ 2
Bryan Finster - Walmart (Speaker)17:10:00

DevSecQualBusAudInfraOps. Could just talk about flow. 😉

👏 1
😀 1
inactive17:10:40

Thank you, @lucasc5 and @lewir7!! And make sure to check out their awesome breakout session!

❤️ 2
jeff.gallimore (CTIO - Excella, he/him)17:10:40

Fast start to Day 2, right! Give us an emoji to indicate your experience so far today. And get ready for some more great talks…

🔥 8
🎯 1
❤️ 3
🤙 1
💪 2
👏 1
Ganga Narayanan17:10:57

Auditor collaboration over audit negotiation. Thank you! 🙂

😂 2
🙌 1
👍 1
Nick - developer at BNPP17:10:30

Thanks @lucasc5 and @lewir7

😄 2
Nick - developer at BNPP17:10:32

I remember the time when our team has our first external application pen test done by the ouside vendor.

Nick - developer at BNPP17:10:23

And how developers went all the way from "Who are those stupid guys?" to "Wow, we didn't know about this hole! Thank you" . It was very interesting to observe

Nick - developer at BNPP17:10:34

Myself, I have been audited so many times, that I have probably developed the Stockholm syndrome with the auditors

😀 2
Carlos Faria17:10:49

I´m more the type who is traumatized! 😣

Martin (TeamForm)17:10:33

There's some really good conversations to be had with chief risk people, auditors, lawyers about your business objectives (we're in this together), and how they're implemented by tech. showing that the day to day stuff is automated so the humans can focus on the challenges at the edges demonstrates the buy in to those business objectives - and shows that you're not just robots :)

👍 2
Steve Spear18:10:42

@jillmead2018 Great preso. No stink eye for sure!

❤️ 3
Jill Mead19:10:18

@steve773 Thank you! I am glad you liked it.

Ross Clanton18:10:03

Awesome talk @jillmead2018. So funny and such a familiar story... 😹

❤️ 2
😂 1
Jill Mead19:10:47

Thanks @ross.clanton410 So proud to say that I worked with you in a former life.

Ross Clanton19:10:21

We're hiring 😉

Jill Mead19:10:16

@ross.clanton410 😃

Jesse Getzie19:10:58

@jillmead2018 I missed the live stream of your session, so watching now over our "lunch break"...looking forward to it!

❤️ 1
Jesse Getzie20:10:00

"We took a sabbatical in the dojo...to truly understand our engineering customers" 👏

👏 1
Jill Mead20:10:18

@UDK01BXPT Thank you! I am so glad you enjoyed it!

inactive23:10:55

Please welcome @erica.morrison!!!

Jess Meyer - IT Revolution (she/her)23:10:33

Soon, we welcome @erica.morrison to the plenary stage for Q&A!

❤️ 1
inactive23:10:04

For those of you who didn’t make the connection, @erica.morrison is from CSG — the same org as @ken.kennedy and @scott.prugh! 🎉🎉🎉

👍 3
❤️ 2
Dominica DeGrandis, Author - Making Work Visible23:10:26

"Failure doesn't have to be catastrophic - it's how you respond." @erica.morrison

Dave Mangot - Mangoteque23:10:50

Can even be turned into an opportunity

👍 1
inactive23:10:25

…it still amazes me that a certain date evokes memories of a crippling outage… “everyone knows what you’re talking about when you say ‘2/4’”

🎯 1
inactive23:10:29

So riveting. “continuous rolling outages eventually led to feelings of helplessness”. (or was “hopelessness?“) @erica.morrison

Kurt A23:10:03

Sounds like electrical service in California these days

Dave Mangot - Mangoteque23:10:41

Spanning tree? I had my money on DNS

👍 2
😂 1
Dominica DeGrandis, Author - Making Work Visible23:10:37

Incredibly - a situation where it's not DNS 😎

😂 1
Kurt A23:10:25

but it was in the network 😄

😉 1
Joel Boekankamp23:10:13

if I had a nickel every time we had spanning tree cause an outage over my career.......

💰 2
inactive23:10:38

..stories like this make me so insistent to purge IP addresses from my daily life… If I’m trying to figure out a CIDR block, something is very, very wrong…

😂 5
Joel Boekankamp23:10:32

I usually know its a bad day when i'm having to review wireshark captures....

😱 1
inactive23:10:05

…one that truly evokes awe at its scope and complexity…

inactive23:10:17

…and damage wrought..

inactive23:10:50

yeesh. don’t read before you go to sleep… you won’t be able to… 🙂

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:34

I like how we use 'complex' as a placeholder for 'incomprehensible' in all kinds of situations.

🤯 1
Rich B - DevOps is my career change23:10:16

blameless post mortem analysis

👍 4
Nick Eggleston (Nix) - IT Architect - (actively looking)23:10:09

I’ve enjoyed the lessons about incident response from other safety domains in previous years 😊

👍 1
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:58

I like the layered response. Good lean approach - countermeasures at all levels and not a 'solution'....

inactive23:10:47

“LNO”: wait… I didn’t pick this up before. That’s the term used in Team of Teams?

Frotz Faatuai (Cisco IT - he/him)23:10:00

Calling out RCA data points and LTF data points during remediation so that the scribe captures that data is hugely helpful, but I agree it is NOT the focus of the remediation bridges.

Rikard Ottosson - Psychological Safety (People Not Tech Ltd)09:10:14

Yeah this bit spoke to me. I'm quick to make the y-incision during an outage to look at root cause, rather than put a suit and sunglasses on the corpse and take it to the party to resume service.

Dave Mangot - Mangoteque23:10:47

where did the 14 number come from?

Erica Morrison23:10:39

We had a lot of discussions about who and how many. This was based on a combo of factors - recommendations from Black Rock 3, our own internal assessment of who was strong here, and internal discussion about change agents

❤️ 1
inactive23:10:07

they started with 14 incident commanders: they started small, to gain some experience, lessons learned…. two people on call at any given time. first on bridge makes second unnecessary — they’d go back to sleep

Ken Kennedy23:10:03

@erica.morrison you're being nice. lesson #1 for executives like me was to shut up and let the SMEs work the issue. 😃

🤯 3
😀 1
Erica Morrison23:10:28

Stand by

✔️ 1
John Allspaw23:10:08

@ken.kennedy - there might be something in my talk next on this topic

❤️ 4
Eduardo Rodrigues Semensati (Procter and Gamble)23:10:06

interesting, we got the opportunity to engage BlackRock for training in our company as well, more specifically driven by the IT Groups supporting our tech work in the manufacturing plants (where incidents can escalate very quickly, become physically dangerous as well as represent a direct revenue impact as the machines have to stop producing)

Erica Morrison23:10:09

I highly recommend them

👍 1
Ian Silverwood (IT Manager at Ubisoft)23:10:11

@erica.morrison How do you maintain the level of quality in incident response teams over time when things are slow for long periods of time (5-6 months without major incidents)? Do you do fire drills?

Erica Morrison23:10:41

We haven't done fire drills yet, but have discussed. We do leverage lower level outages (not major incidents) for newer ICs to train on. I haven't run into the problem of not getting enough incidents to IC yet....here's to hoping that becomes a problem 🙂

🙏 1
Ian Silverwood (IT Manager at Ubisoft)23:10:49

Good luck then, I hope it happens! I guess we have rich people's problems 😛.

inactive23:10:23

“Congrats! You’re so good at on-call, here’s more! You’re now on-call for your service, and all the other services, too! We were worried that this wouldn’t work.”

3
Kurt A23:10:28

The sheer scale of this incident management implementation is amazing

👍 2
2
😲 1
Grayden Wagner23:10:34

where does the term Bridges come from?

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host23:10:58

I think it’s conference call bridges?

2
Grayden Wagner23:10:58

"bridge" = "call"... roger

inactive23:10:08

Hearing about the amazing outcomes from better incident calls and supporting proceses still blows me away, @erica.morrison

👍 3
inactive23:10:28

Hearing this makes it so clear how poorly run so many incident calls I’ve been on are…. OMG. “Hi, sorry I’m late. Can someone catch me up?” 😱😱 😂😂

💯 6
👍 1
😆 1
Adam Bowman23:10:18

Literally making many dev jobs more humane

Erica Morrison23:10:36

Hi, this is __, can you tell me what's going on? has largely been banished. Not completely, but we're getting there

Dave Mangot - Mangoteque23:10:12

Read the backchannel in Slack @genek101! 😆

inactive23:10:13

^^^ totally. It evokes memories of having to drop everything and explain again the current state,..

Scott Prugh - Chief Technology Officer - CSG23:10:50

That’s when IC says: “ Gene: Stand by!”

inactive23:10:47

TFW when you laugh and cry at the same time, @erica.morrison

Dave Mangot - Mangoteque23:10:06

By the time you read the backchannel, you've lost so much of what has happened since you joined, could easily take 45 minutes

Scott Prugh - Chief Technology Officer - CSG23:10:37

A structured whiteboard helps with this and a great scribe

💯 2
Kurt A23:10:53

Somewhat like this conference - consider it as if it is an incident... :-)

John Allspaw23:10:16

It might be time to go on-call again for something, @genek101!

4
Pete Nuwayser IBM he/him/his23:10:57

If Jez Humble can be an SRE...

❤️ 1
inactive23:10:35

Just talked with Jez a week ago — he’s loving it! (He finally was able to do this, because you can now do it WFH!)

Pete Nuwayser IBM he/him/his23:10:18

"Hey Gene, I've got a really hot incident, wanna be a fly on the wall?"

John Allspaw23:10:20

are people on call sleeping somewhere else than their home?

😂 1
Frotz Faatuai (Cisco IT - he/him)23:10:45

Checkpoints times have been a boon for us. Also having a running PROBLEM RESTATEMENT (an iterative / dynamic problem statement) also helps my bridges by being constantly updated at each experiment / step.

Erica Morrison23:10:41

Yes, the conditions part of the CAN report is important to restate and evolve

inactive23:10:51

OMG, @allspaw (and anyone else): I’ll share stories about some of the things going wrong during this conference — BTW, the first one we did in London… I was shaking for an hour after we got through the first 30 minutes… 😂😂😂 😭😭😭

1
Nick Eggleston (Nix) - IT Architect - (actively looking)23:10:20

Love to hear about it. Where will u do the sharing?

inactive23:10:22

… I can share them perhaps in one of the Virtual Happy Hours tonight! 🙂

Kurt A23:10:17

But which one? There are so many to choose from...

Bryan Finster - Walmart (Speaker)23:10:33

Our's is the after party. @genek101, coming to the after party? We may have a Clanton there.

Grayden Wagner23:10:52

Fema's Online ICS training is surprisingly good as an intro.

Chris Hunt, SRE at Stack Overflow23:10:35

What size of org does this level of framework become necessary?

Frotz Faatuai (Cisco IT - he/him)23:10:32

I would suspect that the metric is defined as an unacceptable MTTR.

1
Erica Morrison23:10:34

I recommend it for any size org. Even a call with more than a few people starts to get chaotic and break down without structure.

💯 1
Kurt A23:10:54

Is there a public incident report available regarding the incident that is being talked about?

Kurt A23:10:16

It sounds remarkably similar to several Qwest/Century Link/Lumen outages that have happened within the last couple of years.

Scott Prugh - Chief Technology Officer - CSG23:10:26

Note this paper we are publishing:

🎉 1
✔️ 2
Eduardo Rodrigues Semensati (Procter and Gamble)23:10:39

is it building on the concept presented here? How different is it from the standard IMS framework that blackrock trained you folks at?

Kurt A23:10:48

I don't think I noticed that one in the IT Rev library - when do you plan to release it?

👍 1
Scott Prugh - Chief Technology Officer - CSG23:10:46

It is being published this fall... It covers more than just the IMS framework. Some patterns that we have discovered and/or formalized with others.,

👍 2
inactive23:10:35

@scott.prugh @erica.morrison Do you have archetypes for the “types of incidents”, which people specialize in? (It’s like you have a sommelier for outages!!!)

1
Erica Morrison23:10:56

Ha! It's typically around products that individual is most familiar with.

🙏 1
Jeffrey Fredrick, Author-Agile Conversations23:10:07

I appreciate hearing the willingness to listen to your own experience and go against the experts when you have conflicting data.

👍 2
Scott Prugh - Chief Technology Officer - CSG23:10:08

This paper does cover many patterns we have leveraged.

❤️ 1
Ian Silverwood (IT Manager at Ubisoft)23:10:15

I'm really impressed by the amount of training dedicated to Incident Commanders. It is a hard role to do efficiently, so having that many people in those rotations is awesome.

Frotz Faatuai (Cisco IT - he/him)23:10:32

Being in the monitoring system, my favorite past time is being at the end of the finger pointing “must be a monitoring problem”… 😉

Erica Morrison23:10:48

Ha! Many of our monitoring tools fall within my organization as well so I can relate!

Jon Sturdevant - Tech Advisor - BlueCross BlueShield of SC23:10:46

@erica.morrison - have you found that some ICs work more incidents than others? And if so, is that driving change to the domains they are part of?

Erica Morrison23:10:32

If you look at it for a period of say, a few weeks or even a month or two, absolutely it ebbs and flows. For our most senior leaders, I would say this tends to balance out over time. We do have a formal post-incident analysis process that includes briefing the company on the outage and determine improvements. So, if a particular area is struggling, it will get a focus

👍 1
Scott Prugh - Chief Technology Officer - CSG23:10:19

These are some of the patterns:

💯 1
🤯 1
inactive23:10:41

Freaking brilliant, @scott.prugh!

Michael Winslow23:10:10

I love that you abandoned "if you break it you IC it".

👍 2
Dominica DeGrandis, Author - Making Work Visible23:10:22

Yes! "There are some things you cannot not learn except by doing." @erica.morrison

👍 3
John Allspaw23:10:03

“Learning incident analysis skills by reading an article (or filling out a template) is akin to trying to learn to ice skate by reading a book about ice skating.”

inactive23:10:02

TFW when you realize you’ve used up almost all the error budget for the year. “Do we have enough to make it home?” — @erica.morrison

Erica Morrison23:10:53

It was such a sinking feeling to eat up must of the budget so early in the year....and such a great feeling to still be able to bring it home within budget when the year was done

inactive23:10:31

“an experience that stick with me my entire career” Wow.

👍 1
Dave Mangot - Mangoteque23:10:34

I have a love/hate relationship with these

Pavan Kristipati23:10:07

wonderful story

👍 1
Scott Prugh - Chief Technology Officer - CSG23:10:12

Awesome job @erica.morrison!!

👍 3
Brian Gallop23:10:14

Does an error budget instill a fear of failure?

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:13

I just had a client describe it as "punishment for bad developers". - sigh -

Erica Morrison23:10:39

It has to be coupled with continuous improvement and psychological safety. If you aren't going to do something as a result of things going wrong, then it absolutely just punishes. But, if you have a culture where it's safe to talk about failure, look for ways to improve, and have capacity to implement improvements, it can be a way to show all the progress being made and end up being a celebration.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:21

Absolutely. And I think it's a brilliant concept that's much more easily understandable than the intangible tech debt.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:12

But we're a consultancy and many of our clients ask us to get involved because their own system or SDO culture is fundamentally broken - so we see behaviour like this a lot. And it is very very hard to change.

Brian Gallop23:10:24

So is it used as more of a canary metric, just to show if you're trending in a direction you don't want to go?

Brian Gallop00:10:06

Complementing it w/ psychological safety and continuous improvement makes lots of sense... and probably self control. 🙂

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)00:10:36

No. We use it as a tool to tackle tech debt and balance backlogs. What we normally do is insist on running our own team as a separate project to build a "go and see model" - of how things can be different. That is usually enough to convince clients to consider the possibility that there is a different way... and that gives us an opportunity to work on trust, safety and improvement...

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)00:10:02

(oh and we embed a couple of client developers, so there is an internal voice to champion the 'better' way)

Brian Gallop00:10:27

Got it, thanks for the extended explanation!

Jason Trent23:10:18

@erica.morrison excellent presentation!

👍 1
Michael Winslow23:10:20

I know some people who took the BlackRock training. It seems you got your money's worth more than them!

Erica Morrison23:10:08

Absolutely. We discussed trying to handle ourselves. No way would we have made the progress we did on our own. Those guys are awesome

Carter McHugh (He/Him)23:10:28

Fantastic work @erica.morrison!

👍 3
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)23:10:30

Great talk. Honest and open. We've been spending a lot of time on incident analysis and resolution lately even though we don't have large distributed systems - we feel like it's just a pivotal part of DevOps. Good to get some validation.

👍 1
Mik Kersten (Project to Product, Tasktop)23:10:31

This is such an important story and so well delivered. I’m going to recommend it to everyone in our org. Awesome presentation @erica.morrison!

👍 1
Fred Ghahramani23:10:59

Fantastic presentation. way to present us at CSG

👍 1
Adam Hawkins, SRE at Skillshare, smallbatches.fm Host23:10:01

Sverige :flag-se: 😄 I lived in Gothenburg for a few years. My sister went to school in Lund. Great times 😄

Michael Winslow23:10:11

Great job @erica.morrison

1
👍 3
Adam Bowman23:10:13

👏👏 Bravo Erica!

💯 2
👍 2
inactive23:10:38

That was an amazing presentation, @erica.morrison — thank you!!!!

🤙 1
👍 3
1
brian kram Principal Systems Engineer Roche Diagnostics23:10:44

Erica: your 'impact minutes' metric sounds interesting. It only works though if it is cumulative in small bits - not all in one incident.

Erica Morrison23:10:58

It can work both ways. The goal would be to NOT have it all be in one incident. But the system is intended to measure pain caused to customers...whether it's through a thousand papercuts or one big cut

brian kram Principal Systems Engineer Roche Diagnostics23:10:24

Thank you. Sounds like two closely coupled needs - not one. So I was wondering if you considered a secondary metric to account.

Erica Morrison23:10:08

We don't - we just use the cumulative total

Steve Gertz23:10:15

IC is the way; it's a truly national system that's deployed everywhere and focuses on being able to work consistently

Steve Gertz23:10:19

Any thoughts on value of using the CAN metric? I've been doing it when I'm doing a call, but frequently can't get others to. Haven't pushed the training, but considered

Steve Gertz23:10:43

(Not metric, but normalized communication pattern)

inactive23:10:43

Oh, running behind! So glad you’re here, @allspaw!

Ashlee Laux23:10:20

@erica.morrison amazing presentation!

🎉 1
💯 1
👍 1
Nick Eggleston (Nix) - IT Architect - (actively looking)23:10:23

Believing X and X being true are often different things

Mik Kersten (Project to Product, Tasktop)23:10:35

Oh that was interesting, the low signal strength

🔥 1
Nick Eggleston (Nix) - IT Architect - (actively looking)23:10:43

We found the root cause of the incident and they have been fired... 🔥

1
💥 1
Frotz Faatuai (Cisco IT - he/him)23:10:41

One of our interesting metrics are No Root Cause Problems in Problem Management.

Luke Rettig (Sr Director Product Management - Target)23:10:43

until these 2 presentations, i’ve always thought about what happens after the incident is recovered (post mortems, etc)… have never thought about how to learn pre and during the incident… 🤯

Erica Morrison23:10:23

We had talked about improving outage calls prior to implementing IMS, but it just didn't feel like 1) It would make that much difference and 2) That it could be done. Boy, was I wrong. I'm glad ACL pushed us to make changes here. And once you get the hang of it, it's really not very hard

👏 1
Jesse Getzie23:10:04

"The state of maturity in the industry on learning form incidents is low" @allspaw any recommendation on "getting started" for large orgs starting?

John Allspaw23:10:02

Incident analysis is a set of skills that very few organizations have and recognize as a specific expertise. Supporting and encouraging the building of these skills by leaders is necessary.

👍 1
Jesse Getzie00:10:37

Thank you @allspaw I will read and share, and thank you for your support at our past Learning Events.

thankyou 1
Scott Prugh - Chief Technology Officer - CSG23:10:31

Pattern #1 is: Make Incidents Visible and Part of Everyday Work...

👍 1
inactive23:10:15

…I’m dazzled by some of the common themes in this talk and @steve773, so much reinforced by the concrete details and emotional impact of @erica.morrison’s amazing presentation…

Kurt A23:10:10

The AMA with @steve773 ended up hitting on these themes quite a bit too

Matt Masuda - Quicken Loans23:10:58

The pointy haired boss: "Make this simpler for my boss, and even simpler for his boss"

Mik Kersten (Project to Product, Tasktop)23:10:27

“Mean time to oversimplify”, ha!

upvotepartyparrot 4
😂 1
Frotz Faatuai (Cisco IT - he/him)23:10:38

We cascade Major Incidents against CMDB so that the responsible Service Owners and Service Executives have to tell the story of what the “blunt ends” and “sharp ends” are doing to remediate. Would be interested to hear how people solve this problem without big ITIL process (Incident Management, Major Incidents, Problem Management).

John Allspaw23:10:27

A different perspective would be to bring attention to what is being learned. Effective remediation depends critically on how rich/broad the understanding of the incident is. 🙂

inactive23:10:01

@steve773 talks about the “vast and rugged topography of the problem” — major incidents/outages are clearly an extremely example of this.

Jim Cuff, VP Engineering / CTO Consultant (he/him)23:10:21

"believe their presence has a positive influence!"

⬆️ 1
inactive23:10:30

…still wondering how @allspaw captured that ceiling camera shot…. 🙂

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)23:10:06

@allspaw Is the observation about leadership involvement in bridges/calls true for orgs that score high on Transformational Leadership/Westrum Culture?

Andrew Hughes - Manager, DevOps Service Delivery QA (TRIMEDX)23:10:50

In other words, are there ways for leaders to actually be helpful in these calls given the right cultural maturity etc

John Allspaw23:10:51

Ask the people doing the hands-on work what leaders can do to be helpful in those situations — before an incident takes place.

👍 1
Dave Fugleberg23:10:10

I'm not sure which position is more frustrating during an incident- the blunt end or the sharp end...by the time you're at that place in your career where you're at the blunt end, it's frustrating because you no longer know enough to be useful

Kurt A23:10:14

I liked the "quick check of who is to the left and right" before saying that HIPPO presence is not positive

Adam Hawkins, SRE at Skillshare, smallbatches.fm Host23:10:25

I just hit this last week when I was paged. 3 levels of hierarchy were “participating” by pinging me for updates and possible fixes … before creating a call with another 10 engineers.

Taylor Barnett00:10:38

Oof, that’s rough

Nick Eggleston (Nix) - IT Architect - (actively looking)23:10:48

Aligns with whether the company embraces Learning Culture?

Eduardo Rodrigues Semensati (Procter and Gamble)23:10:57

@genek101, @allspaw: doesnt the comment on leaders not being useful in bridges go a bit against of what show in the phoenix project book where the main characters are always present in the incident and long nights? Or by leaders you really mean C-Suite and Executives?

2
inactive00:10:39

…spraying drink all over laptop. 😂😂😂 (and for the record, I don’t think anyone is asserting that leaders aren’t needed on outage calls. But, yes, Sarah was deliberately portrayed as particularly unhelpful in the outage calls depicted in Unicorn Project, evidencing some of these phenomena.)

John Allspaw00:10:32

I’m saying that leaders’ belief that they’re being helpful rarely matches the belief of others who they’re trying to “help”

Eduardo Rodrigues Semensati (Procter and Gamble)00:10:09

thanks for answering! indeed, most of the cases they just put more pressure and take the practitioners focus away of solving the problem at hand

John Allspaw00:10:19

Indeed, and often folks don’t feel as free to be as verbose with their colleagues when leaders are present.

2
inactive23:10:07

…question I’ve wanted to ask @allspaw all year: Can you give an example of how you, as VP Ops at Flickr or CTO at Etsy, that you fell into one of these technology leader traps?

💯 2
Kurt A23:10:24

doesn't apply to them - sounds like Dunning-Kruger

👍 2
Mik Kersten (Project to Product, Tasktop)23:10:41

I learned this the hard way years ago and at that point made sure that I never participate directly in incident response channels, much as I thought I knew better and thought it was positive for me to be engaged. Wish I’d seen this talk then!

👍 3
👀 1
Luke Rettig (Sr Director Product Management - Target)00:10:45

i try to listen in incognito, but refrain from direct participation. Is this similar to what you do @mik?

pcn00:10:59

I always appreciate it when someone who knows the history asks "can you describe what you expect to happen if you do <whatever we're talking about>"

pcn00:10:17

Whether they're management, IC, or executive

Nick - developer at BNPP23:10:45

wow. Technical leaders look really bad in this talk

😂 3
John Allspaw00:10:27

Yes. Yes they do. @genek101 has always encouraged me to provide real talk.

❤️ 5
inactive00:10:34

…well, wait until @allspaw describes what goes wrong with hands-on practitioners! 😆 😆 😆

😂 2
inactive00:10:45

…well, wait until @allspaw describes what goes wrong with hands-on practitioners! 😆 😆 😆

3
😮 2
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)00:10:06

There's a very old meme from the British army - the only order a British officer ever has to issue is "carry on Sergeant Major". It separates the practitioners from the leadership....

😁 2