Fork me on GitHub
#ask-the-speaker-plenary
<
2020-06-23
>
Fernando Cornago08:06:47

hello! adidas is ready here!

👟 5
🙌 14
😁 1
🤝 2
😀 1
2
👏 2
👍 3
Margueritte Kim (CEO, IT Revolution, she/her)08:06:56

So good to see your face on the screen, my friend @fernando.cornago!

Fernando Cornago08:06:24

same @mvk842 missing you in person this year!

Fernando Cornago08:06:39

@genek101 is nice, but you are on the next level!!!

2
Jeffrey Fredrick, Author, Agile Conversations08:06:10

IT --> Technology is a good rename. How did that come about?

👍 2
Jeffrey Fredrick, Author, Agile Conversations08:06:10

IT --> Technology is a good rename. How did that come about?

👍 2
Jeffrey Fredrick, Author, Agile Conversations08:06:33

… or should I have said @fernando.cornago442? There’s two of you!

Daniel Eichten08:06:53

That was actually a proposal of our CIO last year. Sometimes you need outside in persepective. Or just rename to create a reboot.

5
Fernando Cornago08:06:55

really good decision

Kurt A08:06:58

Sounds like you've taken Team Topologies to heart 🙂

Fernando Cornago08:06:03

from managing IT to enable biz with tech

Fernando Cornago08:06:11

we started before the book!

✔️ 1
Fernando Cornago08:06:25

😛 but we called differently

Fernando Cornago08:06:33

consult, co-create and swat-delivery

👍 1
Jeffrey Fredrick, Author, Agile Conversations08:06:42

managing IT to enable biz with tech < reminds me of the CAMS blog post from @patrick.debois256 way back in 2012 http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/

👍 1
Jevon White08:06:18

Platform is a product - so true!!

👍 11
Jeffrey Fredrick, Author, Agile Conversations08:06:19

“You don’t want to be the bottleneck” < words to live by!

👍 9
Jose Mingorance08:06:09

Is everyone @adidas required to use the enterprise platform?

Jose Mingorance08:06:09

Is everyone @adidas required to use the enterprise platform?

Daniel Eichten08:06:48

We have a very lightweight tech request process if you want/need to deviate. But actually we make it very appealing to use the Platform.

Daniel Eichten08:06:25

So short answer no, but longer answer is: better come with a good reason if you want to deviate.

👍 2
Jose Mingorance08:06:18

Thanks that makes sense. At Fannie Mae we started with one single CI/CD platform but we are stating to offer 2 solutions as we move to the cloud.

Daniel Eichten08:06:23

Yep, that is usually also happening over here that we have 2-3 choices globally you can pick from. We don't believe in one-size-fits-all.

👍 4
Nick Eggleston (Nix) - IT Architect08:06:14

How often do you see deviation requests and approvals?

Fernando Cornago08:06:28

more in the beginning

Fernando Cornago08:06:54

but after 2 years we made so easy to use some of the offering and invested so much into onboarding and training that doesn't make a lot of sense

Fernando Cornago08:06:16

we even had issues by implementing serverless at scale as we made it very easy to jump on K8s

John Boyes08:06:19

@fernando.cornago442 and @daniel.eichten in terms of platform evolution, do you have any structures to help the platform teams build the platforms which their customers actually need, rather than the platform which the platform team think they need (or want their customers to need)?

Fernando Cornago08:06:14

structures? every team knows their customers and we also measure reaches

Fernando Cornago08:06:31

we rarely implement an evolution without a team hand-by-hand

👍 1
John Boyes08:06:44

Maybe structures is not the best word. Techniques, then 🙂

Fernando Cornago08:06:56

at the end, we are the tip of the Communities of Practice

Luke08:06:04

@fernando.cornago442 - Was the loss in revenue directly linked to the outages?

Fernando Cornago08:06:26

our digital analytics team is providing hte numbers

Fernando Cornago08:06:28

for the P1s only

👍 1
Fernando Cornago08:06:37

for P2 and below we don't measure

Fernando Cornago08:06:46

they provide as part of the RCA

Markus Lauttia08:06:09

I bumped into Adidas DevOps Maturity framework a few days ago: https://github.com/adidas/adidas-devops-maturity-framework. Are you actively using it and what kind of experiences you have had with it?

👍 8
Markus Lauttia08:06:09

I bumped into Adidas DevOps Maturity framework a few days ago: https://github.com/adidas/adidas-devops-maturity-framework. Are you actively using it and what kind of experiences you have had with it?

👍 8
Fernando Cornago08:06:58

Yes, the Delivery teams use it and we update it frequently

Fernando Cornago08:06:09

it's really good and was born from 3 or 4 areas at the same time

Fernando Cornago08:06:28

we have an agile maturity framework, that at the end links to the DevOps one for the tech teams

Christian Rudolph08:06:32

can you share more how you linked both. we have currently 2 separate teams (1 for Agile, 1 for DevOps) which are developing there own.

Markus Lauttia09:06:27

Thanks! I found the framework very valuable myself. It's always easier to start own work when someone else has done the groundwork 😄

Liz Rice08:06:21

Hi @Fernando! In the Dojo slide it showed you’re migrating from Quay to Harbor - I’d be really interested in the reasons for that?

Liz Rice08:06:21

Hi @Fernando! In the Dojo slide it showed you’re migrating from Quay to Harbor - I’d be really interested in the reasons for that?

Daniel Eichten08:06:49

Simple reason: global reach. http://quay.io wasn't available in China mainland. 😉

👍 2
Fernando Cornago08:06:28

yep, harbour we install as a sidecar of all our stack (K8s, Kafka, Kong, Jenkins aaS...) in private and public

👍 1
Fernando Cornago08:06:39

fits better and slightly cheaper

Daniel Cahill - Engineer - Ontario Systems08:06:16

Is there a similar reason you chose Harbor over ECR?

Liz Rice08:06:28

Did you consider running your own instances of Quay? (I’m interested because they are submitting to CNCF)

Daniel Eichten08:06:39

@U015ZKPR2H2 actually not really we looked into harbor, and http://port.us the point in time. http://Quay.io was only available on-prem as enterprise subscription and that was provided by our k8s partner giantswarm.

Daniel Eichten08:06:23

But I can connect you to the team who did it that time and what their current point of view on the registries are.

Liz Rice08:06:48

That would be really helpful, yes please

Daniel Eichten08:06:08

should I forward to your cncf address?

Liz Rice08:06:16

<mailto:liz@lizrice.com|liz@lizrice.com> is good for this, thanks!

Jose Mingorance08:06:21

We are also using DOJOs to help the teams learn through immersive learning working on real product backlogs with the help from Product and Techn coaches who have "hands of the keyboard" vs "hands of the whiteboard" skills.

👍 3
Gene Kim - Program Chair - ITREV08:06:46

(I’m loving the fact that @fernando.cornago442 is answering all these questions in near real time — thank you Fernando!!!)

👍 2
💯 12
1
Gene Kim - Program Chair - ITREV08:06:46

(I’m loving the fact that @fernando.cornago442 is answering all these questions in near real time — thank you Fernando!!!)

👍 2
💯 12
1
Luke Rettig (Target)08:06:13

i’m stressed out for Fernando and Daniel

😆 2
Luke08:06:23

And keeping up!!

Jose Mingorance08:06:07

Hey @genek101, I was bummed I missed your session @ Fannie Mae due to vacation.

Olivier Jacques08:06:17

Big up vs. actual f2f conference!

Fernando Cornago08:06:52

I'm still a romantic

Fernando Cornago08:06:01

and prefer f2f with a round table afterwards

💯 2
Jeffrey Fredrick, Author, Agile Conversations08:06:56

have you shared your RCA process anywhere @fernando.cornago442 @daniel.eichten?

Jeffrey Fredrick, Author, Agile Conversations08:06:56

have you shared your RCA process anywhere @fernando.cornago442 @daniel.eichten?

Fernando Cornago08:06:51

Not, yet... I will check with the Digital SRE team... it's great

👍 2
Jeffrey Fredrick, Author, Agile Conversations08:06:53

That would be great! I find it really interesting to compare the process from organization to organization.

Fernando Cornago08:06:26

Hi Rene, Jeff, SRE we are piloting in the eCOM space

Fernando Cornago08:06:04

basically we try it to be an enablement team that helps DevOps teams

Fernando Cornago08:06:31

if you ask me, our vision is that it stays central for the 4 or 5 most critical value streams of the company

Fernando Cornago08:06:50

and at the same time set up the standards for release/incident and problem managmenet of hte company

❤️ 1
Sandeep Joshi08:06:18

So SRE teams are more of enablers and help other teams with standards/advisory.

💯 1
Fernando Cornago08:06:48

that's the vision, for the time being they still hold a lot of ops resonsiibility

Fernando Cornago08:06:00

as they were born from L1,L2 Ops teams

Fernando Cornago08:06:26

now they keep track of the tams taking over ops for themselves and the ones not doing it

Sandeep Joshi08:06:09

I thought so. I had a similar setup at few clients and we introduced “swarming” and embedding SRE as part of our team’s platform dev guidelines. It worked well at some places where SRE became everyone’s business. Some places, it is still maturing. Thanks for the details.

Szilard Szell (Eficode)08:06:26

@fernando.cornago442 Great to see QA Strategy is a key to succeed. What is the main point to focus on in a renewed QA Strategy?

Szilard Szell (Eficode)08:06:26

@fernando.cornago442 Great to see QA Strategy is a key to succeed. What is the main point to focus on in a renewed QA Strategy?

Fernando Cornago08:06:11

wow.... too big topic... it's more about overall strategy

Fernando Cornago08:06:16

focus on dependencies and processes

Fernando Cornago08:06:22

more than testing within the teams

Fernando Cornago08:06:33

contract testing will play a role technically

Fernando Cornago08:06:48

in the big entities that travels from one product to another

Fernando Cornago08:06:07

I can give you the contact of our Test leaders

2
👀 2
Szilard Szell (Eficode)08:06:16

E2E Quality assurance, with shift left and shift right? And I loved to see that you focus on Exploratory testing next to automation

Szilard Szell (Eficode)08:06:41

Contact would be great! Thanks

Nick Eggleston (Nix) - IT Architect08:06:55

Would love to hear more about this as well...

Andrew Salt08:06:17

@fernando.cornago442 Any more info on Dojos?

Andrew Salt08:06:17

@fernando.cornago442 Any more info on Dojos?

Fernando Cornago08:06:40

for us... putting tech experts with products experts together

Fernando Cornago08:06:55

with a contract of success and time to get there

Fernando Cornago08:06:05

the rest... the teams work together as a single team for the time of engagement

Fernando Cornago08:06:21

outcome is value for the receiving team and updates on practices for the platform team

Fernando Cornago08:06:54

I can put you in contact with our Lean Delivery leader that is the most experienced one with that

Fernando Cornago08:06:02

even if we use the practice in the Big Data Platform too

Andrew Salt08:06:02

Thank you 👍

Nick Eggleston (Nix) - IT Architect08:06:29

Love to hear more about this topic too!

Yannick Muller09:06:14

Hi @fernando.cornago looking forward to know more about how to put in place DOJO practice and ways to organize it.

Hannah Beech08:06:43

@fernando.cornago442 What's the general make up of your teams? Are they multi-discipline? What's the (average) ratio of qa to dev?

👍 1
Hannah Beech08:06:43

@fernando.cornago442 What's the general make up of your teams? Are they multi-discipline? What's the (average) ratio of qa to dev?

👍 1
Fernando Cornago08:06:41

strong techies that want to be techies their whole life

Fernando Cornago08:06:50

we have a handful of QA experts

Fernando Cornago08:06:07

that work for our team as well as help the rest of the QA leaders of the company with QA platforms and practices

Fernando Cornago08:06:32

I cannot breath! ;););)

😆 1
😂 4
3
Fernando Cornago08:06:32

I cannot breath! ;););)

😆 1
😂 4
3
Jeffrey Fredrick, Author, Agile Conversations08:06:10

though I guess that means you might end up with questions all week!

😂 1
Stefan (Stieber)08:06:00

@fernando.cornago  how do you cope with this level of transparency within your organisation?

Stefan (Stieber)08:06:00

@fernando.cornago  how do you cope with this level of transparency within your organisation?

Fernando Cornago08:06:55

it's easier to have transparency that not to have it

Stefan (Stieber)08:06:53

in my experience, some hiereachies avoiding this kind of transparancy

Vlad Ukis08:06:29

it depends on the culture and the current level of bureaucracy in the organization, how far you can go with transparency increase 🙂

Stefan (Stieber)08:06:01

in my case -1 😉

Fernando Cornago08:06:43

well... some areas have some elefants under the carpet

Fernando Cornago08:06:52

but the TCO transparency will make this visible

Fernando Cornago08:06:06

market budgets, etc...

👍 3
Vlad Ukis09:06:12

@fernando.cornago442 do you have transparency on cloud cost consumption by team and service?

Jeffrey Fredrick, Author, Agile Conversations08:06:03

I’m reminded of that “great artists steal” quote

👍 6
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:11

I will ask my “how do you guys measure Psychological Safety?” question later then @fernando.cornago442 and @daniel.eichten when you guys caught your breath 😆

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:11

I will ask my “how do you guys measure Psychological Safety?” question later then @fernando.cornago442 and @daniel.eichten when you guys caught your breath 😆

Daniel Eichten08:06:00

@U015CDT342K not really directly. We measure out employer NPS which can be an indicator.

Pete Nuwayser IBM he/him/his08:06:27

@U015CDT342K have you seen other systems of measurement for p.s.?

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:41

True it can if you’re forensic about engagement but they can love you and not necessarily be as magical as they can be so happiness if good but but it’s not all of PS by far - we also measure flexibility,courage, speaking up, negative behaviours etc - anyhow happy to chat don’t wanna monopolise you. @UDKF6GJ59 we make one and seen some in-house ones in banks too

Ann Marie Fred - IBM09:06:58

Have you seen https://engineering.atspotify.com/2014/09/16/squad-health-check-model/ ? We’ve had luck with this framework, and you could easily extend it to add a question about Psychological Safety.

Pete Nuwayser IBM he/him/his08:06:31

What was in that sandwich?! noodles, pickles, flat meat? (apologies for the off-topic question - it can wait 😂)

Pete Nuwayser IBM he/him/his08:06:31

What was in that sandwich?! noodles, pickles, flat meat? (apologies for the off-topic question - it can wait 😂)

Luke08:06:48

Be good to know how you guys manage users/access via GitOps

Luke08:06:48

Be good to know how you guys manage users/access via GitOps

Giulio Vian08:06:23

it makes sense if you think of a Git Repo with all your definition in, say Terraform or CloudFormation, and the PR merge will apply the change

Luke08:06:48

@U015LV0MPUN Yeah, we currently use GitOps for Kubernetes via Flux.

Luke08:06:19

Interested to know if it’s just Terraform or something a little more special 😉

Giulio Vian08:06:16

the trick, IMHO, is to have a repo just for the security/account stuff well, I love Terraform and have some ideas of how to implement this

Daniel Eichten08:06:33

Bitbucket is integrated into Azure AD for SSO. And then a linked account request is basically one file. Yaml/Json depending on your preference. If you wanna change you only change the file. PR goes in and the guys sitting on top of aws review. Rest is then automated filling terraform templates or CDK scripts.

Luke08:06:50

@daniel.eichten Thank you!

Giulio Vian08:06:51

for my information, no CloudFormation, only TF or CDK?

Daniel Eichten08:06:54

Well was a journey. Started with TF only, went to CloudFormation, back to mix of both. Now all Terraform and some CDK.

👍 3
Dmitry Luchnik08:06:59

To add on top - in the data mesh implementation we do in adidas, permissions to get access to the data will also goes through git: PR from a subscriber as access request tool, approved by the data owner. Self service from that perspective. Rest - is automation.

👍 1
Giulio Vian08:06:40

thx @U015V9PNT9R and @daniel.eichten I would love to do something similar but I see lots of org issues in our company Looking forward for insight on the org/social issue in the next days

Fernando Cornago08:06:58

thanks @U015V9PNT9R for jumping!!!

Jiří Klouda08:06:42

Data Puddles” !!! Oh my! Stealing that one!

👍 3
😁 2
1
Jen08:06:28

@fernando.cornago it'd be great to understand what specific technologies/tools are you using to measure the business value of sw engineering/sw delivery?

Jen08:06:28

@fernando.cornago it'd be great to understand what specific technologies/tools are you using to measure the business value of sw engineering/sw delivery?

Fernando Cornago09:06:45

well... in our Global Metrics Portal is an open field where each team can put a value

Fernando Cornago09:06:48

depends a lot of the domain

Fernando Cornago09:06:01

ecom has nothing to do with Supply Chain

Fernando Cornago09:06:07

or Creation technologies

Fernando Cornago09:06:48

in our case.... we automate the metrics of adoption technically (number of jobs in jenkins, number of apis, etc.) measured by the areas using them

Fernando Cornago09:06:54

and we enter NPS manually

Fernando Cornago09:06:06

and actionise the comments

Fernando Cornago09:06:58

and sorry, Global Metrics Portal is our tool collecting metrics from JIRA, GIT, Jenkins, Sonar, ServiceNow....

👍 2
Fernando Cornago09:06:09

to measure SDLC

👍 2
René Lippert08:06:31

@daniel.eichten Can you share more about the watchdog strategy you mentioned on the Azure and GC useage? Is that something you have as code and might be even able to share this code?

René Lippert08:06:31

@daniel.eichten Can you share more about the watchdog strategy you mentioned on the Azure and GC useage? Is that something you have as code and might be even able to share this code?

Daniel Eichten08:06:41

Right now we make use of Dome9 with custom rulesets and extract percentages on how compliant accounts are to the rulesets. That is being read out via API and put into the metrics portal.

Daniel Cahill - Engineer - Ontario Systems08:06:15

What led to the decision that every 6 sprints is the right interval for the cleanup work to improve velocity?

👍 5
Daniel Cahill - Engineer - Ontario Systems08:06:15

What led to the decision that every 6 sprints is the right interval for the cleanup work to improve velocity?

👍 5
Daniel Eichten08:06:36

In the past we usually parked 15% capacity of the teams for technical debt and cleanup acitivities. So I'd say its something that developed over time. Although it was hard to convince some biz colleagues in the beginning most of them see the benefit in the meantime.

👍 2
Fernando Cornago09:06:06

in this case... the team decided themself

👍 2
James Head08:06:32

hello! SwissRe iptiQ is ready here!

👋 2
Luke08:06:43

Thank you Adidas team!!

thankyou 1
Gene Kim - Program Chair - ITREV08:06:45

THANK YOU @daniel.eichten @fernando.cornago442!!!

👏 3
🙌 23
thankyou 5
👍 5
3
Jose Mingorance08:06:52

Does your platform provide the agile teams with low leve software engineering metrics and trends i.e. Commit frequency, build frequency, failure rates, test execution, etc?

Jose Mingorance08:06:52

Does your platform provide the agile teams with low leve software engineering metrics and trends i.e. Commit frequency, build frequency, failure rates, test execution, etc?

Daniel Eichten08:06:11

Yes, correct! We have all the low level metrics and are increasing now piece-by-piece as well as aggregates.

James Head08:06:56

great job adidas

thankyou 1
👍 2
1
Areti Panou08:06:59

@fernando.cornago442 You've mentioned something about improving developer productivity. What metrics are you using for that?

6
Areti Panou08:06:59

@fernando.cornago442 You've mentioned something about improving developer productivity. What metrics are you using for that?

6
Edu Escartí08:06:14

Yeah! Very interested in this

Victoria Mayo08:06:06

Hi everyone! IptiQ here. And SUPER presentation by Adidas!!

thankyou 2
Jon Smart [Better Value Sooner Safer Happier]08:06:09

Great job @fernando.cornago442 @daniel.eichten

3
🎉 1
thankyou 1
Yannick Muller08:06:10

@fernando.cornago can you explain what's the goal of Licence to automate program ? Who is managing the program ? (is this the marketing team ?)

Yannick Muller08:06:10

@fernando.cornago can you explain what's the goal of Licence to automate program ? Who is managing the program ? (is this the marketing team ?)

Daniel Eichten08:06:29

Program was initiated by the automation team in Platform Engineering. So rather technical. But they calculate together with their biz partners the RoI.

Rui Melo08:06:16

Great session Adidas. Thanks for sharing and inspiring!

thankyou 3
Jose Mingorance08:06:23

Nicely done paisano and Daniel!!!!

thankyou 1
Matthew Joyner08:06:39

Great job! 👏

thankyou 1
Chris08:06:40

Thanks Adidas for the inspiring talk

thankyou 3
Brandon Ferrari08:06:48

@fernando.cornago442 Thank you for the presentation! I'm curious, how do you gather the metrics for your performance? I'm trying to do something similar for my company as well

Daniel Eichten08:06:24

Would be great!

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:31

Better be! Getting airport-lounge withdrawals at last

Guillaume Rémy08:06:25

Great Keynote, thank you

upvotepartyparrot 1
thankyou 1
Jess Meyer - IT Revolution (she/her)08:06:00

Thank you @fernando.cornago @daniel.eichten!!!!

thankyou 1
Jeffrey Fredrick, Author, Agile Conversations08:06:12

In the future all talk titles will be emoji based

😁 6
❤️ 1
👍 1
Jess Meyer - IT Revolution (she/her)08:06:31

@james839 @victoria_mayo are here to answer your questions!!

👋 7
Luke Rettig (Target)08:06:55

@fernando.cornago @daniel.eichten - do you have formal Product Managers for your platform products or do your Engineering Managers lead that function?

Luke Rettig (Target)08:06:55

@fernando.cornago @daniel.eichten - do you have formal Product Managers for your platform products or do your Engineering Managers lead that function?

Daniel Eichten08:06:29

No real formal product managers. Engineering Leads do this in role-union. As they are internally usually speaking engineer to engineer that works pretty well.

👍 6
Luke Rettig (Target)08:06:36

thanks…. its a continued debate at Target. I find that if the leaders are customer focused and treat the engineering teams they are enabling as a customer with choices, then it works.

Ciaran Byrne08:06:56

Thank you @fernando.cornago and @daniel.eichten

thankyou 1
Edu Escartí08:06:17

Hi @fernando.cornago! I also work for a German company (http://XING.com) and we are also doing our DevOps transformation right now and a lot of what you said totally resonated. I have a very specific Cloud question to @daniel.eichten: We know that in Germany there is a very present PR-related cloud oposition. How was the cloud movement? Was it driven bottom-to-top? What success stories can I pitch as a simple team-lead and Cloud enthusiast to help my VP pitch cloud adoption to our C-level?

Edu Escartí08:06:17

Hi @fernando.cornago! I also work for a German company (http://XING.com) and we are also doing our DevOps transformation right now and a lot of what you said totally resonated. I have a very specific Cloud question to @daniel.eichten: We know that in Germany there is a very present PR-related cloud oposition. How was the cloud movement? Was it driven bottom-to-top? What success stories can I pitch as a simple team-lead and Cloud enthusiast to help my VP pitch cloud adoption to our C-level?

Daniel Eichten08:06:43

@U015LG3MT7G Cool! Send some greetings to Jens Pape please. On your question: yes I can echo this but it usually also differs to what companies you talk to. Big global enterprises like us or BMW are usually quite open to adopting public cloud. Super small startups as well. What I typically see is a oppostition in the small to medium sizes – sometimes family run – companies.

Daniel Eichten08:06:56

And don't get me wrong there are also good reasons to not go to the cloud. Although I'd say data privacy shouldn't be on the list. You just have to do it right. This even resonates in the fact that the german BaFin now allows banks and insurances to make use of public cloud.

Edu Escartí08:06:12

Sure will send Jens your greetings 🙂 Very nice guy. Yeah, you’re probably very right… XING being 100% DACH focused now, maybe falls into some “cultural” - pitfalls. Thanks… BaFin seems like a good argument to pitch 😄

Vlad Ukis08:06:12

I am also in Germany. In the regulated Healthcare domain working for Siemens Healthineers. At the Digital Health business unit, lots of our products run on the Microsoft Azure Cloud.

👍 1
Edu Escartí08:06:28

Wow @U016H286F6U

Christian Rudolph09:06:08

@U015LG3MT7G I can only mirror your challenges we face the Same problem at TUI especially in Germany. We proved it in the UK (where we had senior buy-in) and then we got some of the germans to evangelise it in the teams.

Edu Escartí09:06:53

Really interesting @ULMEV2R47. Seems to be “a thing” in Germany. Do you see results already? I think teams could be easily motivated to go cloud, but we faced sometimes slow-downs due to trying to use OnPrem practices with cloud

👋 1
Christian Rudolph09:06:50

Yes we see results and now we have a movement for things to move over. However if we operate it like On-prem we fail a lot of time, based on costs comparison. If we then optimize together with our Enabler teams we see better results. So training and review of tools/practices are very important. However I can only encourage to take a small cost bump through lift’n’shift instead of trying to do it directly correct.

Jeffrey Fredrick, Author, Agile Conversations08:06:21

RiskTech < new term for me

🎉 3
😀 1
Cindy Vineberg08:06:56

Ca you provide a few more details about the Org change for inverse Conway @fernando.cornago442

Maik Himstedt08:06:13

@james839 Can you give some info about how you setup support process at IptiQ? I understand that you had a greenfield, right?

Maik Himstedt08:06:13

@james839 Can you give some info about how you setup support process at IptiQ? I understand that you had a greenfield, right?

James Head08:06:19

it depends on the support

James Head08:06:58

we were able to leverage the group for workplace, productivity tools and corporate services

James Head08:06:33

which gave us time to buils out our own capabality

Maik Himstedt08:06:28

How do you deal with support requirements (e. g. 24/7 you build it, you run it) and autonomous product teams?

9
James Head09:06:30

we built out our teams around DDD for delivery

James Head09:06:59

and then communities of practice around data, test, product, platform engineering

Maik Himstedt09:06:07

How do you position the teams in regard to your support organisation (if you have any)?

James Head09:06:52

Don't really have a seperate support org

James Head09:06:02

rotated through engineering teams

Maik Himstedt09:06:39

Does this imply a minimum team size?

Gene Kim - Program Chair - ITREV08:06:29

@victoria_mayo “I don’t do retail insurance” 😂

😁 3
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:48

@james839 ”trying to change mindsets and getting nowhere fast”?!? Get outta here! Unheard of 😆

😆 7
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:48

@james839 ”trying to change mindsets and getting nowhere fast”?!? Get outta here! Unheard of 😆

😆 7
James Head08:06:57

familiar right!

Edu Escartí08:06:55

Also @daniel.eichten do you transparently communicate costs of their infra to the teams using it? Is that some KPI they take a look regularly?

Edu Escartí08:06:55

Also @daniel.eichten do you transparently communicate costs of their infra to the teams using it? Is that some KPI they take a look regularly?

Daniel Eichten08:06:02

This year there is a mix, but from next year the platform team actually is aimed to have 0€ budget. So that means every product team is being recharged for the services they use.

Daniel Eichten08:06:55

And yes: we had some surprise moments in the past when a half-years-cloud budget is burned in one month cause someone spun up the biggest managed Oracle DB you can find in the catalog. 😉

3
Edu Escartí09:06:14

We got also funny things like “Hmmm why is this Sandbox account no-one is using burning 1k a month?” Literally burning money

Edu Escartí09:06:16

For me being really Data-Center heavy company and having the “it’s cheaper” argument always waved for PRO-On-PREM I find this cost-focus really interesting to demistify

Jeffrey Fredrick, Author, Agile Conversations08:06:26

“going native” sounds much like “joining the rebellion” :)

😀 3
😁 3
👍 2
Jeffrey Fredrick, Author, Agile Conversations08:06:26

“going native” sounds much like “joining the rebellion” :)

😀 3
😁 3
👍 2
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:05

I like how you guys are 10 mins into it and still on how you created the team

👍 2
Shamse Alam08:06:04

What is your support model?

Bahram08:06:16

Unfortunately most slides are not possible to read. Too small and blurry.

Bahram08:06:16

Unfortunately most slides are not possible to read. Too small and blurry.

James Head09:06:49

We can share after

John Booth08:06:24

Build vs Buy is something I'm looking to achieve, would be interested how you convinced the executive team. Mine is very rooted in Buy

John Booth08:06:24

Build vs Buy is something I'm looking to achieve, would be interested how you convinced the executive team. Mine is very rooted in Buy

James Head08:06:06

its certainly a journey thats for sure, it took quite some time.

Victoria Mayo08:06:33

Build was important from the outset but that work was done by our iptiQ Life and Health colleagues who came about 6 years before us. Swiss Re wanted to create that access and create it greenfield, but the buy option can always remain on the table as part of inorganic growth.

James Head08:06:08

we split out the tech into 3 key areas and focused on build in the front end

James Head08:06:28

we bought things like HR, finance, risk apps

James Head08:06:50

then middle tier, was mixed between build and opensource

John Booth09:06:25

this makes sense thanks. I'm looking to identify which are the products that really differentiate us from our competitors and look to build there. Buy for standard services such as the ones you meantion (HR, Fianance, etc)

James Head09:06:11

maybe have a look at wardley mapping it gives some nice ideas about technical evolution and can help focus the mind on where to build v's buy

💯 1
James Head09:06:38

generally speaking the bias is build for differentiation and buy for commodity

Gene Kim - Program Chair - ITREV08:06:39

This is so great, @victoria_mayo and @james839! @victoria_mayo — I meant to ask: what does a law degree let you do? Were you ever a practicing lawyer, and if not, how close were you?

Gene Kim - Program Chair - ITREV08:06:39

This is so great, @victoria_mayo and @james839! @victoria_mayo — I meant to ask: what does a law degree let you do? Were you ever a practicing lawyer, and if not, how close were you?

Victoria Mayo08:06:44

I was indoctrinated into Swiss re from the outset and never had my heart set on being a lawyer, so not a qualified solicitor in the UK. Maybe one day for a new challenge 😉

John Booth08:06:43

Thanks, will take a look. I haven't tried this

James Head09:06:05

Yes, Love it

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech08:06:02

Ah, this is the clip that eternally cuts in the middle of my binaural sounds on YouTube during a Pomodoro :the_horns:

Gene Kim - Program Chair - ITREV08:06:12

“let’s not destroy a 100+ year old organization before we even launch” 😆

😆 11
2
Gene Kim - Program Chair - ITREV08:06:20

“I didn’t want to crush all this creativity” 😆

😂 4
👏 1
Tom Sheeran08:06:48

"to be innovative, you are always going to be one step ahead of regulators" Absolutely

👍 2
💯 3
Thomas Williams09:06:32

Agile methodology — came with too much agile evangelism. 😄 Always.

thumbsup_all 2
👍 1
Gene Kim - Program Chair - ITREV09:06:50

“we’re agile — we don’t need milestones. but we WANTED milestones.” (Sorry, I know I’m supposed to ask questions, but it’s so GOOD!)

👍 3
Jon Smart [Better Value Sooner Safer Happier]09:06:16

"Agile religion to the letter, not the spirit" @victoria_mayo

💯 2
Gene Kim - Program Chair - ITREV09:06:18

@jonathansmart1 calls them the “Scrumdamentalists”!!!

4
😆 1
😂 8
💥 1
🤯 1
Daniel Cahill - Engineer - Ontario Systems09:06:29

This is giving me an Aha revelation that I don't hear much conversations from my compliance team outside of the "how". What are some ways or types of questions I can use to understand the "why"? Do you have suggestions on how to do this proactively instead of only when there is something urgent?

👍 2
Daniel Cahill - Engineer - Ontario Systems09:06:29

This is giving me an Aha revelation that I don't hear much conversations from my compliance team outside of the "how". What are some ways or types of questions I can use to understand the "why"? Do you have suggestions on how to do this proactively instead of only when there is something urgent?

👍 2
James Head09:06:40

I would focus on trying to establish what the outcomes they are looking for.

James Head09:06:56

rather than the how, focus on the why and what.

👍 1
Victoria Mayo09:06:11

I also think a lot is down to engagement with compliance and legal. If people only speak to them when it’s urgent, they will learn to only give urgent and often conservative advice. Bring them into the business meetings, let them get the context. People can be nervous to do so, but they’re not there to be the police! They’re there to support. We only get paid if the business does well after all 😉

👍 1
Akis Sklavounakis09:06:46

Loved the 'applying the letter vs the spirit'. Too many going through the motions without the values and principles.

👍 5
Thomas Williams09:06:15

Real progress is messy and difficult. Oof.

👍 1
1
😀 1
Thomas Williams09:06:15

Real progress is messy and difficult. Oof.

👍 1
1
😀 1
Vlad Ukis09:06:52

you have to enjoy the process to endure it 🙂

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:31

@genek101 @jonathansmart1 is a brave man, many know where he lives

1
Gene Kim - Program Chair - ITREV09:06:07

…I loved that confession about “yeah, I was pushing agile forms too much” from @james839 — soo good!

Gene Kim - Program Chair - ITREV09:06:07

…I loved that confession about “yeah, I was pushing agile forms too much” from @james839 — soo good!

James Head09:06:44

yup, guilty as charged!

Luke Rettig (Target)09:06:52

i have remind my teams all of the time about the difference between “being Agile” vs “doing Agile”… usually includes a “i dont give a f* about how long your sprints are of if you had the grooming meeting or not”… all in good spirits

Kurt A09:06:59

Book pointer: Agile Conversations FTW (h/t @jtf)

💯 1
👋 1
James Head09:06:51

I must say I'm very much reformed now!

😄 2
Jeffrey Fredrick, Author, Agile Conversations09:06:10

I appreciate @victoria_mayo pointing out the challenge that a diverse teams brings. Diversity should be a strength, however if the people don’t know how to discuss their differences then the team can end up with low psychological safety.

👍 4
Jeffrey Fredrick, Author, Agile Conversations09:06:10

I appreciate @victoria_mayo pointing out the challenge that a diverse teams brings. Diversity should be a strength, however if the people don’t know how to discuss their differences then the team can end up with low psychological safety.

👍 4
Victoria Mayo09:06:09

Completely agree. Diversity should be strived for, but be prepared to spend time to manage it from the outset

Gene Kim - Program Chair - ITREV09:06:10

“numerous warnings; DON”T TOUCH ANYTHING IN PRODUCTION!” 😂

😂 24
😅 2
😆 2
Gene Kim - Program Chair - ITREV09:06:10

“numerous warnings; DON”T TOUCH ANYTHING IN PRODUCTION!” 😂

😂 24
😅 2
😆 2
Davy Kenis09:06:49

we should introduce testing in production in that case :rolling_on_the_floor_laughing:

Robert09:06:27

We need a Phoenix Emoji 😉

Patrick Debois09:06:22

wait I just pushed out a change to the livesite 🙂

👏 1
Steve Spear09:06:18

@steve773 is here for questions.

🙌 1
👋 1
Gene Kim - Program Chair - ITREV09:06:21

PS: I have a new appreciation for how terrifying these launch events — it was one of my favorite scenes in Unicorn Project. But the first 30m of this online conference was… a bit rocky. THANK YOU for bearing with us!! Thanks to @sam and @patrick.debois256 and the ITREV team for getting us through this!

👍 15
4
☺️ 1
Jose Mingorance09:06:34

Don't breath on those apps or servers. How many times have legacy technology that has been disregarded for years become an anchor to your ability to deliver fast and safe?

🎉 2
Jose Mingorance09:06:34

Don't breath on those apps or servers. How many times have legacy technology that has been disregarded for years become an anchor to your ability to deliver fast and safe?

🎉 2
Jeremy McGee09:06:31

Conversely: sometimes those reliable, well-established systems and ways of working are the anchor you need to deliver fast and safe changes on top

Jeremy McGee09:06:43

(we did a lot of COVID-related change here at Nationwide on the back of COBOL-based mainframe systems and existing digital journeys -- well known, reliable and safe -- with great well-embedded teams)

👍 1
James Head09:06:40

😂 crazy day launch day

Philip Day09:06:16

@victoria_mayo said the key to gaining traction was the diverse executive team finally gelling together... could you elaborate on that please

Philip Day09:06:16

@victoria_mayo said the key to gaining traction was the diverse executive team finally gelling together... could you elaborate on that please

Victoria Mayo09:06:35

It’s hard when it’s a diverse team, especially in terms of ‘thought diversity’ and even more so when you’re starting greenfield. Everyone has a different view and even vision. Once we cemented the short and long term goals and we learnt to communicate effectively, the machine really clicked into place.

Philip Day09:06:09

Thanks... communicate effectively among each other, or to the wider iptiQ team?

Philip Day09:06:50

Getting a shared vision...was that a matter of offsites etc? Leadership selling it in or something more collaborative?

Victoria Mayo09:06:27

Both. We have this concept of going from a village to a city. Communicating insurance business goal context to a wide, mostly tech centric company , has been a learning process.

Philip Day09:06:31

Thanks, very interesting

Victoria Mayo09:06:57

We had A LOT of offsites. I think they’re a great tool, but always interested in other ways as sometimes you can get offsite fatigue!

Ferrix Hovi - Head of DevOps - Siili09:06:52

Official unicorn? Who grants that one?

Ferrix Hovi - Head of DevOps - Siili09:06:52

Official unicorn? Who grants that one?

Jiří Klouda09:06:18

Unicorn Certification Authority

Victoria Mayo09:06:03

Think his name is Charlie from Candy Mountain...

Victoria Mayo09:06:16

Also, investors 😁

Ferrix Hovi - Head of DevOps - Siili09:06:05

I bet there is a Chinese vendor for great-looking badges and trophies.

Ferrix Hovi - Head of DevOps - Siili09:06:55

And their potentially western competition

Gene Kim - Program Chair - ITREV09:06:50

IKEA!!! 🎉 So great!!! Congrats @victoria_mayo and @james839!!!

👍 2
🎉 10
Gene Kim - Program Chair - ITREV09:06:50

IKEA!!! 🎉 So great!!! Congrats @victoria_mayo and @james839!!!

👍 2
🎉 10
James Head09:06:25

Some other big ones in the pipeline too!

Jeffrey Fredrick, Author, Agile Conversations09:06:08

@victoria_mayo: can you send my a copy of your slides via snail mail? 😉

😀 1
1
Gene Kim - Program Chair - ITREV09:06:14

“Can you please send me my policy statement by mail?” (ah, the non-digital natives! 😆 )

😂 1
Gene Kim - Program Chair - ITREV09:06:14

“Can you please send me my policy statement by mail?” (ah, the non-digital natives! 😆 )

😂 1
Jeffrey Fredrick, Author, Agile Conversations09:06:34

heh, both reacting to the same thing 🙂

Chris09:06:28

Not surprised... We’re doing mechanical watches after all...

😆 1
James Head09:06:33

no actors where used in the making of this advertisement !!:rolling_on_the_floor_laughing:

😆 7
😀 1
Gene Kim - Program Chair - ITREV09:06:22

“iptiQ is hiring all over the world!”

👍 4
Gene Kim - Program Chair - ITREV09:06:22

“iptiQ is hiring all over the world!”

👍 4
Luke09:06:44

Great during these times!

James Head09:06:23

100% iptiQ in 4 continents

💯 1
Nick Eggleston (Nix) - IT Architect09:06:21

100% remote (given the pandemic)?

Victoria Mayo09:06:28

Depends on your location and the rules there, some people have chosen to come into the Zurich office occasionally, but otherwise yes, everyone has been doing 100% remote!

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:17

Really great job @victoria_mayo and @james839 both in building an InsureTech challenger but also in building with the WoTNotWoW Agile bit at DNA level -and in keeping this about the EQ of it-

🙌 3
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:17

Really great job @victoria_mayo and @james839 both in building an InsureTech challenger but also in building with the WoTNotWoW Agile bit at DNA level -and in keeping this about the EQ of it-

🙌 3
Thomas Williams09:06:44

Thank you Victoria and James!

Evgeny Potapov09:06:44

@victoria_mayo @james839 how far is iptiQ in terms of direct management from swiss RE management? is it more like a startup within a large company or a department within a large company? how supportive was the swiss RE management in terms of innovations? I saw the cases when the top management was initially all in for innovation but only later realized that it might "break" the existing culture and was reluctant to implement innovation thanks for a great talk!

Evgeny Potapov09:06:44

@victoria_mayo @james839 how far is iptiQ in terms of direct management from swiss RE management? is it more like a startup within a large company or a department within a large company? how supportive was the swiss RE management in terms of innovations? I saw the cases when the top management was initially all in for innovation but only later realized that it might "break" the existing culture and was reluctant to implement innovation thanks for a great talk!

James Head09:06:50

I would say its like a friendly VC

👍 1
James Head09:06:12

so its now a standalone business unit in the SR Group

James Head09:06:27

it has its own CEO, leadership team, platform etc.

James Head09:06:51

and is funded by the group with some oversight from a risk/ cost perspective

Victoria Mayo09:06:01

It’s very close! So our iptiQ CEO’s boss reports directly to the Group Swiss Re. And our Group CEO is very passionate about iptiq. I can say our group CEO is very keen to disrupt the reinsurance culture, including Swiss re’s own. We can be a bit elephant like - reliable but slow. IptiQ allows for genuine agility so so long as our underwriting remains sharp, top management is very supportive

👍 1
Evgeny Potapov09:06:53

got it, thanks!

Liz Rice09:06:46

That "friendly VC" model can be super-powerful if the team is really given freedom to operate e.g. using different tools from the parent company

👍 2
Evgeny Potapov09:06:28

@U015ZKPR2H2 right, feel like the best way to implement changes (especially in a regulated niche)

Evgeny Potapov09:06:16

@james839 @victoria_mayo thanks again, might be a great case for other companies

James Head09:06:47

it was really powerful

James Head09:06:58

I think we had the best of both worlds

James Head09:06:00

the security and budget from a big corp but with autonomy on tech, ways of working, hiring etc.

Philip Day10:06:48

Sounds analogous to product teams, at larger org scale

Matthew Joyner09:06:51

Thanks for sharing 👏

Jon Smart [Better Value Sooner Safer Happier]09:06:57

Great job @victoria_mayo @james839

👏 2
Mark Mulvany09:06:09

Thanks for a very interesting session

Philip Day09:06:10

Thanks both!

Victoria Mayo09:06:14

Thanks for having us everyone! 😊😊

💯 5
Michael Kalbermatter09:06:16

Great story! Greetings from Bern 🙂

Jess Meyer - IT Revolution (she/her)09:06:19

Thank you @victoria_mayo @james839 !

Hannah Beech09:06:31

@victoria_mayo It was really interesting to hear from a legal and compliance perspective. Thank you for the talk.

2
Hannah Beech09:06:31

@victoria_mayo It was really interesting to hear from a legal and compliance perspective. Thank you for the talk.

2
Victoria Mayo09:06:05

I’m so glad! 😊

Hannah Beech09:06:03

As someone who's had to deal with pushback from developers on whether or not to fix bugs, I love it when I'm working on a compliance product - it's comforting when you've got compliance on your side to talk to devs as to why it needs fixing!

Victoria Mayo09:06:04

Bug fixing has become a daily part of my job that I never expected coming from Reinsurance!!

mohitraina09:06:32

nice talk 👏

Jess Meyer - IT Revolution (she/her)09:06:44

@steve773 speaking next and here for your questions!

James Head09:06:44

Thanks everyone!

4
👏 2
Gene Kim - Program Chair - ITREV09:06:51

Amazing presentation, @victoria_mayo and @james839!!! PS: here’s a talk from a legal/compliance executive at Nike! Let me know if I can introduce you to Anne Bradley! https://www.youtube.com/watch?v=13C95oShKgQ&amp;t=4s > DOES18 Las Vegas — Leveraging the power of a matrixed organization to solve problems and build solutions bigger than the individual. > > Build a Bigger Team - Nike > > Anne Bradley, Chief Privacy Officer and Global Counsel for Nike Direct > Courtney Kissler, Vice President, Nike Digital Platform Engineering

👏 1
Gene Kim - Program Chair - ITREV09:06:51

Amazing presentation, @victoria_mayo and @james839!!! PS: here’s a talk from a legal/compliance executive at Nike! Let me know if I can introduce you to Anne Bradley! https://www.youtube.com/watch?v=13C95oShKgQ&amp;t=4s > DOES18 Las Vegas — Leveraging the power of a matrixed organization to solve problems and build solutions bigger than the individual. > > Build a Bigger Team - Nike > > Anne Bradley, Chief Privacy Officer and Global Counsel for Nike Direct > Courtney Kissler, Vice President, Nike Digital Platform Engineering

👏 1
James Head09:06:30

that would be great!

Gene Kim - Program Chair - ITREV09:06:12

And here’s some fantastic auditors from Big Four from panel session last year — they were from audit/assurance practice, not the consulting side. They’re so great, too! https://www.youtube.com/watch?v=iiQY9qiDQCE&amp;list=PLvk9Yh_MWYuwXC0iU5EAB1ryI62YpPHR9&amp;index=15&amp;t=174s > Matt Bonser, Director, Digital Risk Solutions, PricewaterhouseCoopers LLP > Yosef Levine, Managing Director, Global Technology Controls, Confidentiality & Privacy, Deloitte > Jeff Roberts, Senior Manager, Advisory Services, Ernst&Young > Michael Wolf, Managing Director Modern Delivery Lead, KPMG

Gene Kim - Program Chair - ITREV09:06:10

Will do — I’ll send you an email introduction in the next day or two!

Steve Spear09:06:58

@genek101 thanks for the nice introduction.

🙏 1
Richard Vodden09:06:21

It is actually impossible not to smile when @steve773 starts speaking 🙂

🙂 12
😆 3
👍 4
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)09:06:46

@genek101 you talked a lot about community, where does the community persist outside of the summits? Is there a permanent home for exchanging ideas?

👆 6
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)09:06:46

@genek101 you talked a lot about community, where does the community persist outside of the summits? Is there a permanent home for exchanging ideas?

👆 6
Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx09:06:50

This Slack stays open all year round but it tends to be pretty quiet outside of conferences. Maybe we could use it more?

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:50

I was gonna say - found resources on here from a couple of years ago - it’s not going anywhere

2
Gene Kim - Program Chair - ITREV09:06:58

For sure — the Slack instance stays open. I’m open to any ideas you have on how to enable people to engage with each other, help each other, etc., outside the conferences. cc @jeff.gallimore, who can help operationalize it. 🙂

Nick Eggleston (Nix) - IT Architect09:06:32

Agreed... pretty dead between conferences... love to find a why to keep the flow going...

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx09:06:28

Barry at SEACON conference keeps conversations going by having occasional sessions outside of the actual conference itself. Maybe some DOES curated virtual sessions (Zoom? Slack? Something on this platform?) more frequently than once / twice a year?

💯 1
Mark Goble09:06:55

Yep DataScience festival does something similar with meet-ups to share real world learnings in between major events

Jose Mingorance09:06:16

Is the video frozen for you?

Jose Mingorance09:06:16

Is the video frozen for you?

Richard Vodden09:06:40

it borked for me, but refreshing brought it back 🙂

Steve Spear09:06:54

I think the speaker is just a stiff…:-)

😂 4
9
😁 2
Steve Spear09:06:54

I think the speaker is just a stiff…:-)

😂 4
9
😁 2
Jose Mingorance09:06:44

Rich content clogged my Wifi 🙂

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:32

Stealing this one for the next virtual keynote

✔️ 1
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:23

not your entire presentation! The joke :face_with_hand_over_mouth:

👍 2
Steve Spear09:06:05

@simonw Thanks Simon!

Gene Kim - Program Chair - ITREV09:06:06

PS: I told @steve773 after his recording how mind expanding it was — I learned so much about how to use the medium of the “small screen.” Like Bill Nye the Science Guy! 🙂 You can see some adjustments I made to my own recordings after watching him!

Steve Spear09:06:59

Ship on the left looks like old school “Man of War.” More and more guns bristling from the side of the ship.

Gene Kim - Program Chair - ITREV09:06:43

I still can’t believe the tech changes given that there’s only 3 years of difference in those ships!!!

👍 3
Steve Spear09:06:00

the introduction of the “dreadnought” into naval aresanals

🤯 1
Steve Spear09:06:54

repurpose what we do; repurpose how we do it.

Preston Gibbs - DevOps Dojo Sensei - Walmart09:06:17

If the U.S Government can change things on how they design their ships in just 3 years to better serve their mission. Then we all should be able to change our organization's to better serve the organization's mission with DevOps. It just takes people with the idea and the drive to keep push for better outcome for everyone.

Preston Gibbs - DevOps Dojo Sensei - Walmart09:06:17

If the U.S Government can change things on how they design their ships in just 3 years to better serve their mission. Then we all should be able to change our organization's to better serve the organization's mission with DevOps. It just takes people with the idea and the drive to keep push for better outcome for everyone.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)09:06:54

They just need a lead from the Brits, 5 years earlier..

Preston Gibbs - DevOps Dojo Sensei - Walmart09:06:49

Thats the whole point of the conference right. Learn from other's success and failures and then put it in to practice for what works for you.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)09:06:55

Actually reading about it, it was a parallel development... John Ericsson a Swede introduced it in the Monitor...

👍 1
Jeffrey Fredrick, Author, Agile Conversations09:06:22

I’ve got a feeling this keynote is likely to come up in the #bof-leadership channel later…

💯 1
Pete Nuwayser IBM he/him/his09:06:38

"distributed experimentation" 🤯

🎉 1
💪 2
🤯 5
👍 1
2
Pete Nuwayser IBM he/him/his09:06:38

"distributed experimentation" 🤯

🎉 1
💪 2
🤯 5
👍 1
2
Simon Williams09:06:35

That graphic was like a horizontal OKR map to me

Anand09:06:45

Excellent keynote! We are struggling with centralisation vs de-centralisation debate. Would be good to understand how you break down problems against strategy

Steve Spear09:06:12

@anand.patil Most of my knowledge is from movies 🙂 But seriously, I’ve got a riff on how “The Martian” demonstrates “recursive” pre mortem” to show how to go from huge problem to individual pieces (and then piece them back again).

2
🙌 2
Jose Mingorance09:06:16

You can share lessons learned also through Blameless Postmortems. We do host a monthly session on the last Friday of the month called Blameless Fridays where teams come share learnings from recent failures in Production. Stole this idea from the Google SRE practices.

👍 7
Jose Mingorance09:06:16

You can share lessons learned also through Blameless Postmortems. We do host a monthly session on the last Friday of the month called Blameless Fridays where teams come share learnings from recent failures in Production. Stole this idea from the Google SRE practices.

👍 7
Richard Vodden09:06:57

the power of a blameless postmortem should not be underestimated iMHO!

1
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:13

As long as you don’t push it too far and start toying with Failure awards and rewards like l saw a team do 😵

John Allspaw15:07:50

I’d love to hear about your resistance to that idea, @U015CDT342K!

John Allspaw15:07:13

I have a very positive take on the notion of ‘failure awards’ (if I understand your meaning of it)…so much that as CTO we brought a great deal of productive attention to it: https://www.infoq.com/articles/crafting-resilient-culture/

Jeffrey Fredrick, Author, Agile Conversations09:06:02

“the information will be parsed and go to the right places” < sounds like Westrum’s generative culture

Pete Nuwayser IBM he/him/his09:06:10

opportunity, authority, ability RESPONSIBILITY to experiment

👍 4
Anand09:06:25

Technology choice decisions - Should it be a EA led process or should it be a decision made in the trenches??

Anand09:06:25

Technology choice decisions - Should it be a EA led process or should it be a decision made in the trenches??

Anand09:06:33

Our CIO gave us a Cliched answer - It depends :)

🙂 2
Anand09:06:44

Looking forward

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx09:06:33

(Also happy to chat more in detail - my 1/2 hour is a bit of a blistering pace run through lots of modern EA)

2
Jose Mingorance09:06:50

Make failures an enterprise learning opportunity not a shaming one. Creates a safe culture where problems are not shoved under the rug.

👍 9
Jose Mingorance09:06:20

EA should set Enterprise guard rails but leave a decent level of local autonomy.

👍 3
Jose Mingorance09:06:20

EA should set Enterprise guard rails but leave a decent level of local autonomy.

👍 3
Mark Goble09:06:28

Yep ‘freedom in a framework’ was a great eye opener for me. thinking about how you can do ‘improv’ within the guardrails

Steve Spear09:06:45

@jose_mingorance @richard431 So, The Martian was a deconstructive pre mortem. “Astronaut Mark” won’t get home. Why not? No food no shelter no communication no transportation. Why no food? didn’t pack enough for more than a short period and don’t have fresh food. Why not? no soil, no nutrients no water, etc. Why not? … OH! We can fertilize soil. We can burn fuel and make water…

Gene Kim - Program Chair - ITREV09:06:54

“I have no idea! I am utterly ignorant of how to solve this problem” 😂

thumbsup_all 1
Jeffrey Fredrick, Author, Agile Conversations09:06:16

“problems” vs “exercises” < very nice

Steve Spear09:06:39

@jtf Thanks. Those guys were on to something.

Jose Mingorance09:06:59

I see Red People!!!!

Gene Kim - Program Chair - ITREV09:06:34

@akis.sklavounakis David Silverman, co-author of Team of Teams, will be talking tomorrow. @steve773 and I have had great fun talking with him.

❤️ 5
👍 2
Steve Spear09:06:04

@akis.sklavounakis Thanks Akis. Team of Team is terrific.

👍 1
Tom Coudyzer09:06:08

I want an "aha" emoji... was in need of it several times already

👍 1
🎉 1
1
Stijn Claes09:06:10

love the energy of your talk @steve773

❤️ 7
💯 7
Stijn Claes09:06:10

love the energy of your talk @steve773

❤️ 7
💯 7
Arne Rossmann09:06:48

@steve773 great and vivid presentation!!!

Steve Spear09:06:51

@stijn.claes Thanks. Several pots of coffee coming into the recording. Keeps the energy high and keeps the duration short 🙂

😁 1
4
Jeffrey Fredrick, Author, Agile Conversations09:06:00

heh. I just read a blog post on this mistake, deciding in advance, on Friday, this time using Sauruman and LotR: https://acoup.blog/2020/06/19/collections-the-battle-of-helms-deep-part-viii-the-mind-of-saruman/

Jeffrey Fredrick, Author, Agile Conversations09:06:00

heh. I just read a blog post on this mistake, deciding in advance, on Friday, this time using Sauruman and LotR: https://acoup.blog/2020/06/19/collections-the-battle-of-helms-deep-part-viii-the-mind-of-saruman/

Tom Sheeran09:06:04

@victoria_mayo reinforces innovation will always be ahead of "regulations"

👌 1
Giulio Vian09:06:27

@steve773 the projection in the Pacific didn't started at the end of XIX with Philippines?

gerald09:06:30

@stijn.claes - agree! nice talk by @steve773. The talk reminds of a couple of CCRP papers - “Network Centric Warfare”, “Power to the Edge”, and “The Agile Organization”.

Ciaran Byrne09:06:56

Great observations on the issue of 'Decided' @steve773

Steve Spear09:06:00

@ciaran.byrne Thanks. We’re going through “decided” with Covid, right? People trying to “plan” for September as if there is one possibility as opposed to “prepare” for the possible Septembers we might encounter.

😲 4
😑 1
Davy Kenis09:06:16

life will throw you situations you don’t have the answer on.... love the quote @steve773!!

thumbsup_all 4
Nick Eggleston (Nix) - IT Architect09:06:23

Hubris is a real problem for real-world learning.

gerald09:06:28

AFAIK the firing of someone with a novel idea happened also in US i think around 2002 - MC02 with Lt Gen Van Riper being fired and the war games being restarted

Jose Mingorance09:06:31

All plans are outdated as soon as the first bullet is fired.

💯 4
Richard Vodden09:06:54

“smart beats stupid just about every day” 😄

😂 5
Andre Lee-Moye, DZone09:06:09

That reminds me of a quote: "Everyone has a plan until they get punched in the face" @jose_mingorance

😆 2
👍 4
1
Daniel Cahill - Engineer - Ontario Systems09:06:54

@steve773 There is a lot of good takeaways about how to help lift issues up to leadership. Do you have suggestions on how to build the knowledgebase horizontally and learn from other teams?

Olivier Jacques09:06:55

@steve773 - COVID could be a new reference, alternative to famous war battles? Learning from distributed experimentation (enabled by technology) vs. a battle plan?

Olivier Jacques09:06:55

@steve773 - COVID could be a new reference, alternative to famous war battles? Learning from distributed experimentation (enabled by technology) vs. a battle plan?

Nick Eggleston (Nix) - IT Architect09:06:23

Should be a fascinating area of study...

Nick Eggleston (Nix) - IT Architect09:06:41

So many experts and vested interests vying for control... and top-down structures having a field day...

Steve Spear09:06:02

@andrelm Thanks Andre. I believe that was Mike Tyson…

Steve Spear09:06:02

@andrelm Thanks Andre. I believe that was Mike Tyson…

Andre Lee-Moye, DZone09:06:08

It is! I genuinely had no idea that was from Mike! The things you learn at conferences...

Richard Vodden09:06:26

@steve773 bravo 🎉

Nick Eggleston (Nix) - IT Architect09:06:28

@steve773 - where will you be active on Slack after the talk?

Steve Spear09:06:36

@dacahill7 Thansk Daniel. Have a separate riff on building “knowledge sharing” mechanisms.

👍 2
Steve Spear09:06:36

@dacahill7 Thansk Daniel. Have a separate riff on building “knowledge sharing” mechanisms.

👍 2
Daniel Cahill - Engineer - Ontario Systems09:06:07

@steve773 Is this already recorded or upcoming?

Kasia Karpinska09:06:42

@steve773 Great talk!!

Thomas Williams09:06:44

Wow, great engaging talk @steve773! Thank you!

Matthew Joyner09:06:45

Great talk 👏

Steve Spear09:06:52

Thanks all!

👏 1
Gene Kim - Program Chair - ITREV09:06:57

Thanks @steve773!!!! So great!! (I understand you’ll be available for further Q&A in a Zoom or something? Can you share when/where to go?) 🙏

👋 3
❤️ 2
Gene Kim - Program Chair - ITREV09:06:57

Thanks @steve773!!!! So great!! (I understand you’ll be available for further Q&A in a Zoom or something? Can you share when/where to go?) 🙏

👋 3
❤️ 2
Alistair Doran09:06:12

Excellent and captivating session

Steve Spear09:06:28

@jessicam Hey Jess. To where shall I go for Zoom talk?

Marc Boudreau (Enterprise Architect)09:06:01

At first wasn’t certain, completely had me at the end... well done

👍 1
Steve Spear09:06:09

I’ll move over to “ask the speaker” channel now.

Siva Sethu09:06:15

Great Session

Jeffrey Fredrick, Author, Agile Conversations09:06:15

A video on a better way to learn from your wargames (UK WW2 vs the pre-war Japanese experience described by @steve773): https://www.youtube.com/watch?v=fVet82IUAqQ

Vlad Ukis09:06:16

Intriguing :)

Steve Spear09:06:43

@mboudreau327 Thanks Marc.

👍 1
Jess Meyer - IT Revolution (she/her)09:06:59

@steve773 Hi Steve! We'll hold your Q&A during networking - I'll work with you on getting that set up

Jose Mingorance09:06:03

How do you solve SOX and SOD concerns for the App Teams to be able to change the code and deploy it to production?

Jose Mingorance09:06:03

How do you solve SOX and SOD concerns for the App Teams to be able to change the code and deploy it to production?

Tom Sheeran09:06:52

Autromate Automate Automate

Jose Mingorance09:06:32

Automated gates?

Olivier Jacques09:06:52

You can have peer reviews and let the pipeline release to production. SOX tenets do not mandate that reviewers belong to different orgs or even teams.

👍 2
Jose Mingorance09:06:06

Being in financial industry the same person that has access to the code cannot deploy

Jiří Klouda09:06:34

SOX Auditors are quite happy to certify automations.

Tom Sheeran09:06:47

in CI/CD, NO ONE deploys. Deployment is automated based on pushes to master

👍 1
Hannah Beech09:06:02

Having quality gates such as Definition of Ready and Definition of Done also helps with SOX compliancy in my experience

Jose Mingorance09:06:56

We do have CI/CD and deployments are automated but we are required to have a DevOps engineer be the one that can push that button. It cannot be a software developer.

Jose Mingorance09:06:06

We are thinking of implementing a pull request like process where the change will be reviewed and approved before the code gets deployed.

Tom Sheeran09:06:20

An approval step immediately before deployment might make sense. But the criteria should be "did all the automated tests pass? was there a peer review? It should not require more than about 15 minutes to "approve" a deployment.

👍 1
Jonny Leigh09:06:46

Need a talk / breakout on deployment automation in Financial Services

👍 2
Simon Williams10:06:36

@U015CS308FR disclaimer: this presentation is from our customer conference and Goldmans are investors in our company, but thought it might be interesting: https://www.youtube.com/watch?v=Q4QbL-BToLg

Jonny Leigh10:06:27

thank you, watching it now

Jose Mingorance10:06:43

@U015T8ST8UD we are very close to that. We have all testing, code quality and security scans automated running on a continuous basis. What we are missing is the quick approval in our pipeline and the biggest step convince Internal Audit, Risk and Controls that it will meet the spirit of SOX and SOD.

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech09:06:01

This confession is giving me high anxiety and l am not even sure what that certificate meant!

😂 2
Olivier Jacques09:06:05

@dominica.degrandis - this was a great story. TLS certificates are one of the biggest source of unavailability. The story is very inspiring.

Tom Ayerst09:06:23

I miss DevOps Cafe!

1
Kenny Johnston - Gitlab09:06:11

I feel like we are missing a star wars reference

Kenny Johnston - Gitlab09:06:11

I feel like we are missing a star wars reference

Kenny Johnston - Gitlab09:06:53

Agile is definitely the Empire.

Steve Spear09:06:13

@rohrersm Thanks Simon. I’ve got a piece about the interwar period, 1917 Trenches to Blitzkreig. Here’s a preview. Looking forward to comparing with what you mentioned.

🙂 1
Steve Spear09:06:13

@rohrersm Thanks Simon. I’ve got a piece about the interwar period, 1917 Trenches to Blitzkreig. Here’s a preview. Looking forward to comparing with what you mentioned.

🙂 1
Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx09:06:31

Really interesting, thanks @steve773. Tim Harford’s audio goes beyond Clayton Christensen and into Architectural Innovation https://timharford.com/2019/12/cautionary-tales-ep-6-how-britain-invented-then-ignored-blitzkrieg/ JFC Fuller’s tale is just fascinating on its own though.

Steve Thair (DevOpsGroup)09:06:53

Have you read Stephen Bungay's book, the Art of Action? He talks a lot about Von Moltke etc

Steve Spear10:06:07

@rohrersm Gene mentioned something about “structure and dynamics.” I’ve got a piece around “the right way to organize. Attached, below.

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx10:06:18

Thanks so much! This looks Conway’s Law adjacent too. I will take time to take a read.

Andre Lee-Moye, DZone09:06:12

"DevOps didn't come from Agile, but is a reaction to Agile" TIL

❤️ 1
😂 1
Olivier Jacques09:06:57

Great 7 minutes spent. Thanks.

Kenny Johnston - Gitlab09:06:20

Great talk @catswetel!

Matt Wheeler10:06:39

I'm really glad to have caught this. Thank you @catswetel

Cat Swetel12:06:32

aw thank you! it was fun ❤️

Gene Kim - Program Chair - ITREV09:06:52

Thank you @cdavis for Confession! Thank you @damon and John Willis for hosting the Lightning talks!

🙌 3
Gene Kim - Program Chair - ITREV09:06:52

Thank you @cdavis for Confession! Thank you @damon and John Willis for hosting the Lightning talks!

🙌 3
Nick Eggleston (Nix) - IT Architect10:06:50

@genek101: Is John Willis not joining us on Slack?

Cat Swetel12:06:25

yes, thank you Damon and John. I had so much fun preparing my lightning talk! :lightning:

Jeffrey Fredrick, Author, Agile Conversations09:06:22

“Will I live with purpose?” < :thumbsup:

🙌 7
👍 1
Luke09:06:47

What a guy

Pete Nuwayser IBM he/him/his09:06:47

When my weight shoots up after holiday meals: "that's just my error budget"

🙂 4
😂 5
Mark Mulvany10:06:22

Unbelievable talk very inspirational

Kasia Karpinska10:06:12

that was indeed powerful

Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech10:06:14

”l can help people - l am a people!” - genius stuff from @damon

🎉 8
Jose Mingorance10:06:23

DOJO seems to be an overloaded term in the industry. I would be interested in learning the different implementations. For us it is an Immersive learning program acquiring new skills in Product, Tech, Agile and Lean while working on real products and real backlogs. Agile squads/teams learn together. No siloed role training.

👍 4
Jose Mingorance10:06:23

DOJO seems to be an overloaded term in the industry. I would be interested in learning the different implementations. For us it is an Immersive learning program acquiring new skills in Product, Tech, Agile and Lean while working on real products and real backlogs. Agile squads/teams learn together. No siloed role training.

👍 4
Vlad Ukis10:06:14

the confusion is like with very many other terms in the industry 🙂

Jose Mingorance10:06:35

Agile/DevOps alphabet soup

Preston Gibbs - DevOps Dojo Sensei - Walmart10:06:11

So for the dojo team i'm apart of with @UDA3SLC4F we do our embeds by join the team in there space. We do VSM with the team and start by breaking down the work they have in flight currently and then work our way backwards to the backlog. During this time we are insuring they have a template pipeline for each of there tech stacks within that team. Insure they are able to gather all the metrics they need to measure improvement while also discussing the road blocks the team faces and addressing those issues. We become a part of that team for a 6 weeks and introduce the best practices we have learned and adapt them to be able to work within these teams.

👍 2
Jose Mingorance10:06:20

Do you also provide Product Management coaching?

Jose Mingorance10:06:05

BTW, I am part of the DOJO consortium with Walmart, US Bank, Verizon, Target, etc

Bryan Finster - Walmart (Speaker)10:06:57

We provide help with whatever is blocking their flow.

Preston Gibbs - DevOps Dojo Sensei - Walmart10:06:10

Yes, We provide everything and anything the team needs.

Bryan Finster - Walmart (Speaker)10:06:54

We're very focused on delivery outcome goals, so it's less open ended than some other dojos.

Ferrix Hovi - Head of DevOps - Siili14:06:30

We have Dojos anything from a half-an-hour discussion to introduce some core concepts or tools to a handful of people joining a team or team of teams for weeks or months to do whatever together to accelerate them. So, for me, even locally, it is a very diverse thing but always very specifically "investing time working together to improve something".

Eadwin13:06:48

@daniel.eichten A bit late hopefully you are still around - We are struggling with the lock in conversations at the moment and obsessing with the myth of portability. Do you have any advice for convincing the exec about loving the lock in? or rather picking the lock ins you love?

Eadwin13:06:48

@daniel.eichten A bit late hopefully you are still around - We are struggling with the lock in conversations at the moment and obsessing with the myth of portability. Do you have any advice for convincing the exec about loving the lock in? or rather picking the lock ins you love?

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)13:06:39

I've got one. Every technical decision is a sunk cost. Every economic decision should be based on marginal cost and marginal return. So decide where you're going to put X today based on the ROI.

👏 2
👍 2
Daniel Eichten14:06:30

@U015W1V5D8S sorry for the late reply. Had to join a couple of other meetings here unfortunately. I can strongly recommend to read and use material provided by Gregor Hohpe on http://martinfowler.com: https://martinfowler.com/articles/oss-lockin.html. Please specifically check the matrix. And typically we pick based on the accepted lock-in topic. So if we see that a high level service is giving us a real benefit we go all in. E.g. we did this earlier this year launching a product completely based on aws serverless technologies. If there are services on the other hand that we don't like or have limitations we can't live with, we pick something vanillla and try to decouple from underlying IaaS as much as possible. E.g. we didn't liked EKS nor AKS but do vanilla k8s managed by a partner. Or we didn't wanted to go with Kinesis and didn't really liked MSK so we enrolled our own Kafka on aws. But if we'd have to move it will be not a huge deal.

👍 3
Daniel Eichten14:06:11

@U016GR2UA3S yes we do this as well. But usually for us we also include Time-to-Market as a big item on top.

👍 2
Jess Meyer - IT Revolution (she/her)13:06:58

<!here> @steve773 will be doing a follow-up Q&A to his morning keynote at 330pm BST (in 90 min). Join here: https://us02web.zoom.us/j/8908483265

👍 4
🍾 2
☝️ 1
alexbf (event staff)14:06:59

This could be you in here! https://us02web.zoom.us/j/8908483265 (live now)

2
👍 1
🎉 1
alexbf (event staff)14:06:59

This could be you in here! https://us02web.zoom.us/j/8908483265 (live now)

2
👍 1
🎉 1
Kurt A14:06:36

@jessicam will you be posting those extra bits of info here or somewhere else?

Jess Meyer - IT Revolution (she/her)14:06:26

@kboth_does Steve is posting in the general channel!

Kurt A14:06:48

@steve773 ^^

Scott Prugh - Chief Technology Officer - CSG14:06:37

@erica.morrison Up next!! Good luck!

🎉 4
👍 2
Gene Kim - Program Chair - ITREV15:06:46

“…it started off as routine server maintenance…“. 😭😭😭

Gene Kim - Program Chair - ITREV15:06:27

“…you could feel it walking through the hallways, the absolutely crushed morale.” “see, DevOps doesn’t work!”

😥 2
Gene Kim - Program Chair - ITREV15:06:27

“…you could feel it walking through the hallways, the absolutely crushed morale.” “see, DevOps doesn’t work!”

😥 2
Luke Rettig (Target)15:06:05

its oddly similar to the stages of grief

☝️ 1
Ferrix Hovi - Head of DevOps - Siili15:06:11

The mental states of the Responsibility Process: Denial Lay Blame Justify Shame Obligation --- Responsibility ...

Ferrix Hovi - Head of DevOps - Siili15:06:03

Okay, that's upside down and there is a side exit of Quit there on the side... These are easy to detect in others, harder in self and boy does it make sense to be aware.

Olivier Jacques15:06:22

@erica.morrison - how was the outage connected to "DevOps does not work"? Was it a timing question?

Olivier Jacques15:06:22

@erica.morrison - how was the outage connected to "DevOps does not work"? Was it a timing question?

Erica Morrison15:06:18

The sheer fact that we had an outage like this. There was a belief that we should have prevented this and that we should have been able to fix much faster if DevOps did work.

👍 4
🙏 2
Tom Ayerst15:06:49

People create narratives. I am guessing that anyone in the org that didn’t like DevOps used it as a stick to beat people with.

👍 2
Gene Kim - Program Chair - ITREV15:06:00

@erica.morrison It just occurred to me that the horrors of this outage was so great that it now defines a day on the calendar. Wow.

👍 1
Gene Kim - Program Chair - ITREV15:06:00

@erica.morrison It just occurred to me that the horrors of this outage was so great that it now defines a day on the calendar. Wow.

👍 1
Erica Morrison15:06:18

Yes, we actually noted the anniversary this year. It's a day we'll never forget.

Gene Kim - Program Chair - ITREV15:06:06

“…the best training they’ve ever had…” (on incident command)

Scott Prugh - Chief Technology Officer - CSG15:06:41

That BlackRock3 training is truly excellent... We had SVPs and EVPs take it...

Brandon Ferrari15:06:46

@erica.morrison How do you pick who are the incident commanders?

Brandon Ferrari15:06:46

@erica.morrison How do you pick who are the incident commanders?

Erica Morrison15:06:53

It can be anyone who shows the ability to lead. We tend to leave our SMEs out of this group because they are so often involved in the actual troubleshooting. The IC role is a dedicated role and if done right, the only thing that person is doing is running the call. For us, that has largely meant managers are the group free to run a call vs lower level technical team members who are assisting with troubleshooting. It absolutely does not have to be that way and in fact, the training insists that your normal title should not come into play for ICs. At the end of the day, you need someone with the confidence to run a call, make tough decisions (by asking the right questions), assert authority, and be technical enough to at least understand context.

👍 4
💯 3
3
Alex van Assem15:06:08

Why first training and then then the pilot>

Alex van Assem15:06:08

Why first training and then then the pilot>

Erica Morrison15:06:09

IMS/IC is a pretty foundational shift. Even after having researched it, I didn't fully understand how to apply it until I took the training and practiced it. Then, it all clicked. And we needed support from others on the call to ensure we were following protocols, behaviors, and roles

Scott Nelson Windels16:06:24

The shift to broad training has been fantastic in our org. We kicked off our shift a couple of years ago with Blackrock3, and have since expanded to require all engineers in onboarding to take an Incident Responder course. Staring out with training puts everyone on the same page and gives them some confidence.

👍 1
Scott Prugh - Chief Technology Officer - CSG15:06:27

@alexa IMS/IC has certain behavior protocols and roles that are important for many stakeholders to understand. We needed many leaders across groups to understand those.

Scott Prugh - Chief Technology Officer - CSG15:06:27

@alexa IMS/IC has certain behavior protocols and roles that are important for many stakeholders to understand. We needed many leaders across groups to understand those.

Luke Rettig (Target)15:06:15

I’ve heard a lot of through line themes today related to Leader Training, Stakeholder understanding/education, Business appreciation.

Karl Marfitt15:06:06

Given me a test idea, random vLAN killing and low layer LLDP/spanning-tree protocol or other weird packet network storms/floods seem like particularly evil tools to add to a network chaos testing kit 🙂

😱 1
😈 2
Karl Marfitt15:06:06

Given me a test idea, random vLAN killing and low layer LLDP/spanning-tree protocol or other weird packet network storms/floods seem like particularly evil tools to add to a network chaos testing kit 🙂

😱 1
😈 2
Ferrix Hovi - Head of DevOps - Siili15:06:25

random firewall accept rule disabling for a host

Ferrix Hovi - Head of DevOps - Siili15:06:17

oh this is 😈: password expiring for the most senior member of staff

Karl Marfitt15:06:55

Indeed, when hypervisors and other cloud platforms offer APIs to configure (and misconfigure!) things like virtual firewalls, switches, LBs and other network infrastructure elements as code, things like dropping a whole VLAN or certain low layer protocol/traffic become more likely/realistic. Fun for dev/test but scary from an ops perspective!

Ferrix Hovi - Head of DevOps - Siili15:06:45

oh... and false positive health check!

👍 1
Ferrix Hovi - Head of DevOps - Siili15:06:49

I think you can chaos engineer anything during office hours that at some point woke you up in the middle of the night and made you haul donkey to the DC

Ferrix Hovi - Head of DevOps - Siili15:06:24

okay, and most health checks are false positive... HTTP 200 is not the same as "has had connection with the DB in the last fifteen minutes"

Karl Marfitt15:06:57

Had a personal false positive at home today, my IDS caught a malformed packet that apparently looked like an emerging threat and decided to stop trusting and auto-block the whole http://slack.com domain. Doh! 😕

Ferrix Hovi - Head of DevOps - Siili15:06:12

Based on the phishing I get for my Slack credentials, I would happily announce that this Slack thing must be something evil.

Mikko Niemi15:06:40

@erica.morrison You mentioned that many people felt angry and wounded because of the incident. Who were the people who got past that and understood that your organization must learn from the incident? So champions of this learning experience?

Mikko Niemi15:06:40

@erica.morrison You mentioned that many people felt angry and wounded because of the incident. Who were the people who got past that and understood that your organization must learn from the incident? So champions of this learning experience?

Erica Morrison15:06:45

A select group of leaders from our organization saw the need and championed this. It was not initially welcomed by all, but I believe almost everyone now sees the value in retrospect

thankyou 1
Mikko Niemi15:06:14

Did this group of leaders had strong buy-in to DevOps?

Alex van Assem15:06:57

How did the COVID lockdown impacted your current Incident Response?

👍 2
Alex van Assem15:06:57

How did the COVID lockdown impacted your current Incident Response?

👍 2
Erica Morrison15:06:02

I think if you asked most people, they would say it looks pretty much the same. Our mean time to assemble is probably faster now that most people are home most of the time anyway

Scott Prugh - Chief Technology Officer - CSG15:06:54

One thing we did find is that you can run urgent events with incident command. We delayed a deployment due to covid across 65 people. That call was expertly coordinated by @UPVTFM656 using Incident Command.

Scott Nelson Windels16:06:33

In our experience there hasn't been an impact in our Incident Response at Slack, if anything when folks are on call for our Major IC rotation, they are likely to be closer to the keyboard. No hard decisions about where to try and grab a quick lunch when it is your kitchen.

Jonny Leigh15:06:42

How many outages do you get? How do you decided that an incident is worth of opening a bridge and going into "full" Incident Response mode?

Jonny Leigh15:06:42

How many outages do you get? How do you decided that an incident is worth of opening a bridge and going into "full" Incident Response mode?

Erica Morrison15:06:56

I don't have the count handy, but we go into full incident response mode on issues we deem "Major Incidents." Typically, that is some sort of major functionality in a product or products is not working.

thankyou 1
Hugo Serrano15:06:55

great point - trying to get updates all the time during an outage is highly disruptive to someone that is trying to fix the issue

👍 3
Daniel Cahill - Engineer - Ontario Systems15:06:52

I am normally one of the incident responders for one of our main products, no matter what team I'm on. I've tried to implement some of these strategies and worked with the product teams to have an incident response team with defined roles, really similar to some of these steps. We even acted through some past incidents to understand roles better. However, after a few months, incidents returned back to the normal chaos and explaining to managers instead of problem solving when I wasn't there to try to help the team use the flow again. Did you see regressions back to the old way? If so, how did you get people to buy back in to your incident framework?

Daniel Cahill - Engineer - Ontario Systems15:06:52

I am normally one of the incident responders for one of our main products, no matter what team I'm on. I've tried to implement some of these strategies and worked with the product teams to have an incident response team with defined roles, really similar to some of these steps. We even acted through some past incidents to understand roles better. However, after a few months, incidents returned back to the normal chaos and explaining to managers instead of problem solving when I wasn't there to try to help the team use the flow again. Did you see regressions back to the old way? If so, how did you get people to buy back in to your incident framework?

Erica Morrison15:06:57

We have not had a lot of regression. This is one of the reasons I highly recommend engaging with a group like Blackrock 3 and hitting a broad swath of people. We then had a formal program around this with the pilot and the checkpoints, with official reporting to executive leadership on a regular cadence about progress. There is enough visibility and buy-in at this point that it continues on its own momentum.

👍 1
Ferrix Hovi - Head of DevOps - Siili15:06:31

@dacahill7 Sounds familiar. When being pulled back, a change in approach that helped a ton was: "Help them only in ways that makes them own the product"

Ferrix Hovi - Head of DevOps - Siili15:06:31

I think it aligns with my anti-disservice promise when coaching a team: 1. I will not do anything alone 2. I will not finalize anything 3. You'll own whatever I did

Nick Eggleston (Nix) - IT Architect15:06:48

@U016HQVUEE4 interesting ... do you have more to share on this?

Ferrix Hovi - Head of DevOps - Siili16:06:50

@nickeggleston Umm... I'll take that as an incentive to write a blog post on it. The whole thing is something I have cooked up when thinking of how "they can own their own path". It appears that the amount the teacher is speaking is inversely proportional to the amount the learners are completing their mental model. So, given that the three points can be turned around into: 1. If I am not working with them, how are they learning? 2. If I am taking the joy of completing something, what are they enjoying? 3. I am less important than the results.

Daniel Cahill - Engineer - Ontario Systems16:06:07

If you remember, I would love if you would comment on this thread with that blog post

2
Ferrix Hovi - Head of DevOps - Siili16:06:37

No promises, but duly noted 😄

Jason Yavorska15:06:15

Did you find that the company then wanted to treat everything as an incident since it was so well managed?

Jason Yavorska15:06:15

Did you find that the company then wanted to treat everything as an incident since it was so well managed?

Erica Morrison15:06:13

We have had some requests to use IC on lower priority incidents, which I mostly take as a compliment to the process but not something that we can fully staff for every lower level ticket. We do try to use a mini version of this for any bridge we have, with some people simply wearing multiple hats. We haven't hit a point where the odds and ends requests are unsustainable

Pat Eyler15:06:46

Our incident manager have been asked to manage a number of non-incident stuff because incident management "works so well"

Jason Yavorska16:06:15

I worked for a company (won't name names) where they built a great incident management department, and then everything became an incident because it was worked so great to just manage everything that way forever.

Gene Kim - Program Chair - ITREV15:06:40

“we measure the level of pain that we cause our customers.” “after 2/4, we ate up the entire budget of allotted impact to customer.”

Gene Kim - Program Chair - ITREV15:06:17

“you will look at this incident as a blessing, even though it won’t feel like it at the time.” (holy cow.)

👍 2
💯 1
Davy Kenis15:06:57

great talk! Love it!

👍 1
👏 1
Gene Kim - Program Chair - ITREV15:06:07

Thanks @erica.morrison — such an incredible story, and wonderful teachings!!!

✔️ 5
2
Gene Kim - Program Chair - ITREV15:06:07

Thanks @erica.morrison — such an incredible story, and wonderful teachings!!!

✔️ 5
2
Karl Marfitt15:06:11

Brilliant talk, love the idea of (often hidden/invisible or less visible!) complexity and failure combining to amplify a negative feedback loop but ends up being a huge positive learning experience, we should celebrate failure and anomalies, it's how we learn 🙂

👍 2
Brandon Ferrari15:06:13

Awesome talk, thank you!

👍 2
Pete Nuwayser IBM he/him/his15:06:33

@erica.morrison wonderful talk, thanks for the transparency! 🙏

👍 2
Gene Kim - Program Chair - ITREV15:06:35

Fans of this talk will love the talk tomorrow that @scott.prugh is giving on the amazing work at CSG

👍 1
Gene Kim - Program Chair - ITREV15:06:35

Fans of this talk will love the talk tomorrow that @scott.prugh is giving on the amazing work at CSG

👍 1
Kurt A17:06:41

Thanks for the intermittent preview of coming attractions (to follow your cinematic mind set)

Matthew Joyner15:06:36

Thanks for sharing 👏

👍 1
Pat Eyler15:06:37

Great talk, Thanks!

👍 1
evakasiak15:06:40

love this one Avoiding failure requires failure - some things you can't learn except by doing.

👍 1
Alistair Doran15:06:08

Excellent insight.

Pete Nuwayser IBM he/him/his15:06:36

@genek101 I see what you're doing... you're filling the afternoon with incident management talks to make sure we stay awake.

Pete Nuwayser IBM he/him/his15:06:36

@genek101 I see what you're doing... you're filling the afternoon with incident management talks to make sure we stay awake.

Gene Kim - Program Chair - ITREV15:06:01

…and so you won’t sleep tonight, too! 🙂

😂 12
Ciaran Byrne15:06:40

Wow - what an incredible talk, and what great (hard won) advice. Thanks, @erica.morrison 👏👏

Hugo Serrano15:06:41

great talk @erica.morrison! 🙂 thanks for sharing!

👍 3
ClaudiaCardona15:06:51

Great talk 👏 @erica.morrison

👍 1
Gene Kim - Program Chair - ITREV15:06:22

@allspaw: on typical reporting on incidents: “we think we’re doing astronomy, but we’re actually doing astrology” 😆

2
Vishakh Viswanathan15:06:24

“Why do we fall? So that we can learn to pick ourselves back up.” – Batman. Great Talk @erica.morrison! Thank you:)

👍 3
Scott Prugh - Chief Technology Officer - CSG15:06:08

Hah. "We think we are doing astronomy but we are doing astrology instead." @allspaw

👍 5
Kurt A15:06:21

This gap that @allspaw is talking about reminds me of James Mickens points in his USENIX security talk: https://www.usenix.org/conference/usenixsecurity18/presentation/mickens

Ivan Krnic15:06:22

this one is the best so far 🙂

Gene Kim - Program Chair - ITREV15:06:36

David Crossman (co-author of Team of Teams) has a pretty interesting take on blunt end vs. sharp end, and the sometimes vast difference between the two.

Gene Kim - Program Chair - ITREV15:06:36

David Crossman (co-author of Team of Teams) has a pretty interesting take on blunt end vs. sharp end, and the sometimes vast difference between the two.

Kurt A15:06:59

Was just thinking of that point too

Kurt A15:06:47

And what JSOTF did to bridge the gap

Sam Guckenheimer15:06:09

Leaders on the blunt end need to do oncall and participate on the bridge too.

Sam Guckenheimer15:06:09

Leaders on the blunt end need to do oncall and participate on the bridge too.

John Allspaw15:06:33

Wait for it, Sam

👍 4
Kurt A15:06:23

HIPPO tend to introduce reality distortion fields and infringe psychological safety

Sam Guckenheimer15:06:53

The view looks best from the mountaintop.

Gene Kim - Program Chair - ITREV15:06:25

Hahahaha. @allspaw It’s so interesting to have seen the episode before it airs on TV! (Biting tongue!! 😆 )

Brian Martin15:06:59

I was once called out to an incident bridge, ironically while in a hotel room in London, and we wound up with 2 VP's, 1 Senior Director and 6 Directors on the bridge. It did not go well.

😂 3
Gene Kim - Program Chair - ITREV15:06:26

Giving each other PowerPoint presentations? Hahaha

Brian Martin15:06:34

An hour into the bridge no one could say what the issue was or even if there actually was an incident

Kurt A15:06:20

@brian.martin sounds like a good lead up to a joke...

🙂 1
Scott Prugh - Chief Technology Officer - CSG15:06:04

Incident Command has some good protocols to deal with HIPPOs...

Gene Kim - Program Chair - ITREV15:06:30

(I had to look that up — I meant to look it up after @allspaw’s recording. 🙂

Kurt A15:06:07

ding! 🔔 🔔 🔔 about the fears of higher leaders thinking that incidents reflect on their leadership

😭 3
👍 3
🎉 1
Davy Kenis15:06:44

technology leaders don’t believe it applies to them :rolling_on_the_floor_laughing:

😂 4
Kurt A15:06:50

Waiting for the requisite Dunning-Kruger citation 😄

😀 1
John Allspaw15:06:24

@kboth_does I’ll let the audience make that connection for themselves!

Kurt A15:06:45

didn't want to play buzzword bingo?

Gene Kim - Program Chair - ITREV15:06:08

PS: I’m loving the interaction with speakers while their talk airs. I’ve never experienced anything quite like this before!

👏 8
Gene Kim - Program Chair - ITREV15:06:08

PS: I’m loving the interaction with speakers while their talk airs. I’ve never experienced anything quite like this before!

👏 8
Markus Lauttia15:06:13

it's a bit double edged though. Slacking (no pun intended) and watching at the same time is a bit difficult 😄

👍 2
Kasia Karpinska15:06:03

True, but imo it really helps with retaining the main ideas, like a study group

👍 4
Kenny Johnston - Gitlab15:06:07

I love this model for virtual conferences, I was a bit skeptical about the canned recording of the talks but being able to interact live the attendees and the speaker during the talks is awesome. Great work team!

❤️ 1
Duena Blomstrom, PO Psychological Safety Works, Author PeopleBeforeTech16:06:33

l try to answer Tweets when on panels on stage but mostly to flaunt the multitasking- this was seriously good and it goes well with the current trend of podcast-listening while doing other work surveys found is devs behaviour @genek101

Gene Kim - Program Chair - ITREV16:06:32

Me, too! Totally love tweeting on panels as a display of a feat of multitasking prowess. 🙂 But I’m observing that watching talks, and Slack pushes me to capacity. Unlike every conference experience, I didn’t have Twitter front and center.

Siva Sethu15:06:14

The problem is also compounded by the fact that in many organizations, the Technology leaders on the blunt end are the ones that communicate with business leaders and hence try to abstract the problem statement for business consumption by toning down the incident complexity

Siva Sethu15:06:14

The problem is also compounded by the fact that in many organizations, the Technology leaders on the blunt end are the ones that communicate with business leaders and hence try to abstract the problem statement for business consumption by toning down the incident complexity

Kurt A15:06:11

I'd say that such comms patterns are evidence of dysfunction in themselves.

Kurt A15:06:47

Nice summary: "you don't need the chart...just ask the [insert emphatic here] questions"

👍 1
Sam Guckenheimer15:06:15

@siva.ss this needs to go both ways...they should provide customer context as well as just broadcasting out

John Willis15:06:56

Love this slide...

👆 1
Gene Kim - Program Chair - ITREV15:06:38

…now that I think about it, I’m a bit surprised that @allspaw hasn’t dropped in five PDFs into this channel by now… 😆

😂 4
Gene Kim - Program Chair - ITREV15:06:38

…now that I think about it, I’m a bit surprised that @allspaw hasn’t dropped in five PDFs into this channel by now… 😆

😂 4
Sophie Weston15:06:19

I wish he would!

😂 2
Jennifer Velasquez15:06:45

@allspaw Wouldn’t this be the efficiency or flow metric of the incident resolution?

John Boyes15:06:46

Interesting point about Mean Time to Restore having questionable value in this context - curious how that tallies with the findings in Accelerate, where MTTR is one of the four key metrics? :thinking_face:

John Boyes15:06:46

Interesting point about Mean Time to Restore having questionable value in this context - curious how that tallies with the findings in Accelerate, where MTTR is one of the four key metrics? :thinking_face:

Ferrix Hovi - Head of DevOps - Siili15:06:49

I think a low MTTR, as in < 1 h, covers those things. However, the metric is not enough to teach how to drive it down.

Ferrix Hovi - Head of DevOps - Siili15:06:29

The way I teach others about MTTR is "you need to get people out of there to average below an hour"

John Boyes16:06:12

Thanks @U016HQVUEE4. Can you expand on “you need to get people out of there to average below an hour” - I think I understand but not sure. Thanks 🙂

Ferrix Hovi - Head of DevOps - Siili16:06:04

So, when you get nightly incidents that involve waking up, driving and debugging, you need 10 half-hour incidents to fix your MTTR to < 1h. So, removing people from your equation and automating robustness is the way to go.

👍 2
John Allspaw22:06:53

Adding automation can increase challenges for people in incidents as much as it can help.

John Boyes02:06:50

That’s interesting, @allspaw, v grateful if you can expand on this?

John Allspaw12:06:53

Automation, especially successful automation, hides how it does what it does. Incidents often require people trying to make sense of what the automation was doing or is doing, which is difficult when the irony is that it would ‘abstract away’ that

👍 1
John Allspaw12:06:46

There are studies on how influential this paper has been, and continues to be, after 30+ years: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf

👀 1
John Boyes12:06:02

Thanks, that makes sense :thumbsup:

Ferrix Hovi - Head of DevOps - Siili12:06:04

Very true. Black box automation bad.

Ferrix Hovi - Head of DevOps - Siili12:06:09

What I have caught myself saying many times: "When you are automating something as simple as deployment and you are suddenly designing an algorithm, you are probably hurting the future you by making it too complex"

John Boyes12:06:55

Automation is code, and code gets read (way) more often than it gets written. If the automation code is written with care for readability (which is still not the norm, sadly, I find), then the problem should be mitigated somewhat I guess . . https://www.goodreads.com/quotes/835238-indeed-the-ratio-of-time-spent-reading-versus-writing-is

John Boyes13:06:07

This conversation has made me think of something new to me: the concept of designing for ease of incidents.

Ferrix Hovi - Head of DevOps - Siili13:06:37

Nicely put. That has been a standard practice in automation for me, but the wording is new 😄

John Boyes13:06:04

Yep, it sounds obvious doesn’t it, but I have to say I’ve rarely seen or been in teams who have explicitly designed for ease of incidents. It’s making me think that it would be great to have ease of incidents as one of the standard considerations for every new feature to take into account . .

Ferrix Hovi - Head of DevOps - Siili13:06:19

So that's a cocktail of other abilities...

Ferrix Hovi - Head of DevOps - Siili13:06:50

testability for sure, traceability, transparency, repeatability...

John Boyes13:06:35

yep . . what really got me thinking was John A’s insightful comment that it’s the successful automation that can be particularly opaque. Means that communicating the intent and purpose of each piece of automation is super important, I’m thinking.

John Allspaw13:06:15

all automation has the opportunity to be “black box” @U016HQVUEE4 🙂

Ferrix Hovi - Head of DevOps - Siili13:06:42

The quote brings to mind Joel Spolsky with leaky abstractions. So somebody has to fix the plumbing eventually.

John Allspaw13:06:32

and when the plumbing breaks, it puts the human operator in an unenviable position

Ferrix Hovi - Head of DevOps - Siili13:06:03

I hope it is the fresh water side of the plumbing (50% of the time it isn't)

John Allspaw13:06:13

to your very first question, @U016GS8A1UG: my understanding is that in Accelerate, the metric isn’t literally the calculated mean of a set of timings collected

Ferrix Hovi - Head of DevOps - Siili13:06:00

After 13h outage, getting back to the <1h mean is probably not the factual priority

John Allspaw13:06:00

in any case, even if it was referring to that (which would fall down as being useful to represent a non-normal distribution of unique data, nevermind the fact that the “beginning” and “ending” of an incident is negotiable and much more flexible than many admit) it still doesn’t have any explanatory or predictive value.

Ferrix Hovi - Head of DevOps - Siili13:06:13

It does have the feature of not reducing on its own when repeating problems are not addressed.

Ferrix Hovi - Head of DevOps - Siili13:06:36

So it is predictive of the current best case to be expected.

John Allspaw13:06:05

it’s not, actually. the number (averaged over what time period?) doesn’t indicate anything about the incidents

John Allspaw13:06:01

if we look at a plot of this and ask to forecast past what’s in the plot, analysis there doesn’t make sense and empirically isn’t valid 🙂

John Allspaw13:06:39

even if we slice it by severity, or slice it by customer impact, or other axes

John Allspaw13:06:06

this is really tough for leaders to grok, which is why I put it in my talk

John Boyes13:06:58

> Traditionally, reliability is measured as time between failures. However, in modem software products and services, which are rapidly changing complex systems, failure is inevitable, so the key question becomes: How quickly can service be restored? We asked respondents how long it generally takes to restore service for the primary application or service they work on when a service incident (e.g., unplanned outage, service impairment) occurs, offering the same options as for lead time. The above is from the Accelerate book, where they introduce MTTR as one of their 4 key metrics.

Ferrix Hovi - Head of DevOps - Siili13:06:01

Yeah, I see that side of it. However, it is indicative of "this organization is struggling to fix anything in under a couple of days" or "At any given time there are probably up to x dumpsters on fire and that seems to be acceptable"

John Allspaw13:06:01

@U016GS8A1UG trying to find the word average in there. If you look closely at the questions asked in the survey, they ask about <x or >y time, not asking for the distribution

John Boyes13:06:50

Yep, agreed, here’s what they asked for in the survey: > less than one hour > less than one day > between one day and one week > between one week and one month > between one month and six months > more than six months

John Allspaw13:06:15

@U016HQVUEE4 this is the point I made in my talk…what makes it indicative about the “organization”’s abilities, and not the complexity of the system they’re charged with being responsible for?

John Allspaw13:06:21

@U016GS8A1UG lemme ask: did you take part in the survey?

John Boyes13:06:31

I did not . .

John Boyes13:06:29

My original reason for asking, btw, was that (from my non-expert position) I’ve been a fan of the 4 key Accelerate metrics and have been evangelising them to the organisations I work with - was asking as I was curious if you were highlighting some situations where MTTR wouldn’t be valuable.

John Boyes13:06:52

Really appreciate the useful info you’re giving us

John Allspaw13:06:56

the key issue is that the length of time an incident takes can be influence by a multitude of things, especially what’s defined as an “incident”, and who gets to say when it began or ended

John Allspaw13:06:40

you’ll notice that the book doesn’t speak about literally a calculated average across data points spread in time

Ferrix Hovi - Head of DevOps - Siili13:06:09

@allspaw Having been the only guy to be woken up in the middle of the night to fix network problems where the network wasn't very complex, it was more of a staffing according to the needs problem than a complexity thing, seeing a high MTTR does not point you to complexity or, for example, enough people.

John Allspaw13:06:12

which is what “MTTR” means and many organizations actually looks at like tea leaves

Ferrix Hovi - Head of DevOps - Siili13:06:11

Can be argued that the organization is biting more than it can chew since it has complexities that it cannot digest in the middle of the night...

John Allspaw13:06:28

@U016HQVUEE4 and that reflection would be excellent data, were I doing our assessment project in ACL for your company. I wouldn’t ask just you, and I’d ask more details about that.

John Boyes13:06:37

True. Yes was going to say that (of course) MTTR contains the word mean, which suggests a calculated average (to a layman anyway)?

John Boyes13:06:31

ps I’m sold on what you’re saying about incidents learning, namely the need to talk to people rather than relying on arbitrary data

Ferrix Hovi - Head of DevOps - Siili13:06:35

@allspaw Yup and at the end of the day it is a matter of which types of problems you can handle

Ferrix Hovi - Head of DevOps - Siili13:06:28

"Maybe we invest on these kinds of risks next"

John Allspaw13:06:54

who is “they” “we” “you’ organization…the key idea is to understand in a grounded and concrete way what is difficult for people, what rationales they have for doing the things they do, and then begin to aggregate based on that data, not to assert an abstracted aggregate view that can’t be supported by underlying data

John Allspaw13:06:50

qualitative data and research is indeed difficult to do, but engineers (aka the people responsible) don’t tend to see their experiences reflected in these high-level statistics and therefore can be skeptical (rightly so) when decisions that effect their work are based on them

Ferrix Hovi - Head of DevOps - Siili13:06:55

So would you simply ditch MTTR?

John Allspaw13:06:51

I never collected it at Etsy where I was CTO. The value is negligible compared to other forms of learning from incidents.

John Boyes13:06:37

The situation I often find myself in as a coach is that an organisation are going down a path of truly awful “transformation” metrics, and I have the chance to influence them to do something better. My go to recently has been the 4 key Accelerate metrics, as they are much better than what the organisation would otherwise choose.

John Allspaw13:06:28

I’d argue that what is in Accelerate is a fine basis to stand on. Just note that it’s not literally “Mean Time To X”

👍 1
John Allspaw13:06:41

@U016HQVUEE4 CFR?

John Allspaw13:06:46

definitely same issue with that.

Ferrix Hovi - Head of DevOps - Siili13:06:42

When used in a team for the team I've found that to be helpful for seeing progress when starting from some instability.

John Allspaw13:06:46

‘change’ is ambiguous, negotiable, and when a metric collapses a high diversity “population” into a single value, it wipes away the important parts to understand

John Allspaw14:06:57

change and even fail can be really flexible, and tend to get stretched if what they describe is used for decisions

John Boyes14:06:36

Here’s the definition for CFR from the Accelerate book > We asked respondents what percentage of changes for the primary application or service they work on either result in degraded service or subsequently require remediation (e.g., lead to service impairment or outage, require a hotfix, a rollback, a fix-forward, or a patch).

John Allspaw14:06:44

it just doesn’t capture the genuine ambiguity and difficulty people face, and doesn’t shine light on situations where productively anticipating an adverse situation might arise and headed off

John Allspaw14:06:02

@U016GS8A1UG if I asked you what percentage of messages in this Slack in the past two days had questions in them, what would you say?

John Boyes14:06:49

Not many, I think (though I’ve asked a few 😉 )

John Boyes14:06:54

Why do you ask?

John Boyes14:06:15

There’s another one, oops 😆

John Allspaw14:06:39

if I were to use the Slack API to collect all messages during that time period and tabulated questions versus non-questions and then compared that to your answer of ‘not many’ - might I find a difference?

John Boyes14:06:37

is thinking :thinking_face:

John Boyes14:06:28

You would get a precise answer compared to my very vague answer?

John Boyes14:06:46

You made the point above that the imprecise ranges asked for in the survey are fine - so conclusion is that (for MTTR and CFR) precision will be arbitrary and is not required, is that right?

John Allspaw14:06:51

the data they’re collecting is about the respondent’s perspective, which is valid data. It’s not, however, representative of what I think sometimes readers walk away with thinking it represents (even if the book spends ink on interpretation about the data)

John Allspaw14:06:50

from my understanding, the conclusions Accelerate make from the data about respondent’s perspectives on how long it takes an incident to be handled is not at all the same as what companies we see use it for, which is trending and tabulating and decision-making

John Boyes14:06:45

In Accelerate the metrics are proposed as measures for software delivery effectiveness.

John Boyes14:06:00

They’re not proposed as measures for incident learning, for instance (if I recall correctly)

👍 1
John Allspaw14:06:22

Right, which is very different in many ways than a VP of Eng looking at a chart of “MTTR” and concluding that the teams need to “buckle down” or “hire more people” or “move quicker to the cloud”

Ferrix Hovi - Head of DevOps - Siili14:06:53

They are proposed as measures for devops maturity or something like that but that doesn't prevent setting bonus targets based on them.

John Allspaw14:06:27

these are significantly more valuable than wasting time and money tabulating ‘length’ of incidents

Ferrix Hovi - Head of DevOps - Siili14:06:49

So, a high MTTR and CFR will tell you that "there is some struggle somewhere" or "it seems like the struggle is not constant". 😄

John Boyes14:06:04

This conversation is very useful for me, thank you both

John Allspaw14:06:36

A high MTTR and CFR can also tell you what the collector of that data wants to tell you

Ferrix Hovi - Head of DevOps - Siili14:06:03

Gotcha. Maybe it is an individual struggle 😄

Ferrix Hovi - Head of DevOps - Siili14:06:49

BTW, in my current environment those get collected automatically as a side product. Standardises the way they are gamed, at least.

John Allspaw14:06:12

a CSS change that moves a logo into barely noticeable different position…and a CSS change that hides a “Checkout” button. a 100-line code change….and a 1 line code change. <- the 100 line is only an HTML comment, the 1 line is a feature-flag turning on a new feature

John Boyes14:06:32

@allspaw so would you recommend that, if applying the 4 Accelerate metrics in an organisation, (for software delivery effectiveness, not learning from incidents, I mean) to use a similar approach to that taken in the book - i.e. individual surveys with broad ranges?

John Allspaw14:06:03

a blog post being published (hosted on entirely different systems) trigged an outage with http://Etsy.com once. is a blog post a change? was it included in a CFR as a ‘change’ to begin with?

John Allspaw14:06:09

@U016GS8A1UG I’d ask the authors 🙂 research can be done with all sorts of methods, and the authors might even use different methods in the future

John Allspaw14:06:47

I gotta run, but thanks for chatting!

👍 1
John Boyes14:06:23

Thanks for the chat!

John Boyes14:06:14

Would be great to get your thoughts on this conversation too, @genek101 - I love that Slack is asynchronous, so I don’t feel too guilty about asking you, please take your time as I suspect you’re more than a little busy right now! 🙂

Gene Kim - Program Chair - ITREV15:06:35

“…often hands on practitioners don’t capture what made the situation difficult.” Ah, yes. The daily work arounds and near misses! So good!

Vishakh Viswanathan15:06:45

Enabling decentralized decision making and a continuous learning culture is something key here. Since the folks closer to the problem are better informed. However, fear of failure and lack of trust can be a major hurdle in getting to that team dynamic. I wonder how this cultural change can be achieved?

Vishakh Viswanathan15:06:45

Enabling decentralized decision making and a continuous learning culture is something key here. Since the folks closer to the problem are better informed. However, fear of failure and lack of trust can be a major hurdle in getting to that team dynamic. I wonder how this cultural change can be achieved?

Kurt A15:06:05

See Team of Teams and Agile Conversations

Vishakh Viswanathan15:06:40

Thank you. Will check it out! 🙂

Sam Guckenheimer15:06:50

The most important measurement is time-to-learn-and-improve which is all about measuring the learning for reuse

👍 1
Craig Cook - IBM15:06:15

What makes this incident difficult... Great question.

🙂 1
Jonathan Evason15:06:16

what were the books and blog post link?

Jonathan Evason15:06:16

what were the books and blog post link?

Sophie Weston15:06:26

'Tyranny of Metrics' was one of the books - didn't catch the title of the other.

Todd Brooks15:06:34

Safety Can't Be Measured by Andrew Townsend and The Tyranny of Metrics by Jerry Muller

👍 1
Jonathan Evason15:06:51

Nice one, thank you both.

Gene Kim - Program Chair - ITREV15:06:17

“often hands-on practitioners don’t read the PIRs from other teams”

😢 1
Gene Kim - Program Chair - ITREV15:06:17

“often hands-on practitioners don’t read the PIRs from other teams”

😢 1
Kenny Johnston - Gitlab15:06:10

@allspaw - are you going to later cover best practices for writing those reviews in such a way that they are consumable by others?

Davy Kenis15:06:13

and that’s exactly the place where you can learn A LOT!!

John Allspaw15:06:20

@U015U3PFCPM not in a 30min talk, but note that it’s not just about writing, it’s the analysis that happens prior and during the analysis

👍 2
Kenny Johnston - Gitlab15:06:56

:thumbsup: - where would you point me to learn more?

Ferrix Hovi - Head of DevOps - Siili15:06:02

That does somehow rhyme with the high and low context environments from earlier today.

Sam Guckenheimer15:06:11

you need to distinguish proximate cause from contributing factors

Gene Kim - Program Chair - ITREV15:06:58

…ah, I forgot. @allspaw hasn’t been putting the PDFs here, because the links are in his slides! Would you mind putting the links into this channel, John? 🙏

Gene Kim - Program Chair - ITREV15:06:46

NTSB = “U.S. National Transportation Safety Board”. Famous for studying airline crashes. https://www.ntsb.gov/Pages/default.aspx

Gene Kim - Program Chair - ITREV15:06:46

NTSB = “U.S. National Transportation Safety Board”. Famous for studying airline crashes. https://www.ntsb.gov/Pages/default.aspx

Tom Ayerst15:06:00

Totally read that as “U.S. National Trampoline Safety Board”. I may need a lie down…

1
Jennifer Velasquez15:06:15

Maybe in this virtual delivery, it would be helpful to have one place to go for all reference materials for any presentation.

👍 2
Brian Martin15:06:07

'Signals that people are learning', love that!

Davy Kenis15:06:20

love the way you’re giving concrete tips and examples @allspaw !!

❤️ 2
😀 2
Ivan Krnic16:06:30

These tips are so useful for any retrospective.

❤️ 2
Daniel Cahill - Engineer - Ontario Systems16:06:41

"Half of my job is to get people to genuinely look forward to and participate in the next incident anaylsis" Oof. I need to get better at this.

👍 2
John Allspaw16:06:42

Taking action item generation out of group review meetings is probably the most controversial suggestions, we’ve found.

🤯 1
John Allspaw16:06:42

Taking action item generation out of group review meetings is probably the most controversial suggestions, we’ve found.

🤯 1
Pete Nuwayser IBM he/him/his16:06:43

I can imagine. "But what are we going to do?" "You're going to sit on the feedback and take it in for at least 24 hours before writing a single action item."

gerald16:06:00

is it just me or the video got cut?

1
Brian Martin16:06:25

I'm still fine

😱 1
Nick Eggleston (Nix) - IT Architect16:06:39

@allspaw regarding incident analysts being outsiders... how do you think the result of the inquiry into the Challenger would have been different without Richard Feynman?

Nick Eggleston (Nix) - IT Architect16:06:39

@allspaw regarding incident analysts being outsiders... how do you think the result of the inquiry into the Challenger would have been different without Richard Feynman?

Sam Guckenheimer16:06:09

I doubt anyone else would have used a glass of icewater on stage teh way he did

1
John Allspaw16:06:47

@nickeggleston ask your CTO to lead the next group incident review meeting….it’d look a little like that.

John Allspaw16:06:47

@nickeggleston ask your CTO to lead the next group incident review meeting….it’d look a little like that.

gerald16:06:20

i can imagine CTO asking very high level questions. not much on the details. :(

Ferrix Hovi - Head of DevOps - Siili16:06:38

OMG, YMMV but the first three CTOs coming to mind leading any conversation...

Jonathan Evason16:06:54

The comparison with solving coding problems when we've come away from the computer and taken a break makes sense. We don't always solve the problem there and then.

👍 3
gerald16:06:59

@brian.martin - thanks. i think others (but not everyone) is having an issue. hmm weird

gerald16:06:40

nice talk though. the change of pov from just solving incidents to learning from it is nice

John Allspaw16:06:51

Ok leaders here in the channel: who’s going to take me up on these challenges?

👍 3
3
Gene Kim - Program Chair - ITREV16:06:03

“and for technology leaders, that’s my dare for you”

Gene Kim - Program Chair - ITREV16:06:11

Well, @erica.morrison, that’s some pretty awesome kudos from @allspaw!!! I agree!!! 🎉

🙂 1
Scott Prugh - Chief Technology Officer - CSG16:06:14

Really nice job @allspaw and @erica.morrison!!

upvotepartyparrot 2
1
✔️ 1
Ron Chittaro16:06:15

Great talk @allspaw

gerald16:06:30

great talk, @allspaw!

Aidan Moriarty16:06:47

Very inspiring stuff :)

John Allspaw16:06:50

I’m super honored to follow up @erica.morrison! Thanks for listening folks!

🙌 1
John Allspaw16:06:50

I’m super honored to follow up @erica.morrison! Thanks for listening folks!

🙌 1
Erica Morrison16:06:57

Our entire neighborhood lost power like one minute into your talk. Grrrrrr, I missed it. We have power again now - will have to check out the recording. Thanks again for all you have taught us!

Erica Morrison23:06:12

I was able to watch the recording this afternoon....fantastic presentation!

John Allspaw23:06:38

Thanks! High-five!

Sophie Weston16:06:54

My favourite talk of the day - hands down! Thank you @allspaw! Already want to go back and listen again!

Thomas Williams16:06:08

Great Talk, John. Thank you!

Brandon Ferrari16:06:12

Thank you @allspaw! Wonderful talk

Gene Kim - Program Chair - ITREV16:06:13

@jeff.gallimore will be sharing when/where @allspaw will be hosting a Q&A Zoom (or whatever) session later today! Thanks for doing that, John!!!!

Jason Milley16:06:18

Fabulous presentation @allspaw

Pat Hughes16:06:20

thanks @allspaw

Hugo Serrano16:06:36

Thank you @allspaw! great talk 🙂

Akis Sklavounakis16:06:45

@allspaw Unless the 'other' teams have equity or skin-in-the-game generally, I don't see how would this happen...Thanks for the insights.

Fokko Veegens16:06:05

@jwillis Where did you get that great t-shirt? (excuse me if off-topic)

Ciaran Byrne16:06:05

Thank you @allspaw 👏👏

😀 1
Karl Marfitt16:06:19

Superb talk, thanks @allspaw thinking of wild goose chases down rabbit holes reminded me of one particular 'near incident' lesson I learned a long time ago... OMG was that a Natural England logo and some maps my sense of deja vu is so strong today :thinking_face:

👍 2
Pete Nuwayser IBM he/him/his16:06:43

@helen.beal phrases I wasn't expecting to hear at DOES: "These are the sheep." 🙂

😆 4
🐑 3
🐏 2
Pete Nuwayser IBM he/him/his16:06:43

@helen.beal phrases I wasn't expecting to hear at DOES: "These are the sheep." 🙂

😆 4
🐑 3
🐏 2
Pete Nuwayser IBM he/him/his16:06:32

But seriously, Kingley Vale looks beautiful - would love to visit there.

Helen Beal10:06:50

Haha I'm glad you enjoyed it - I thought I'd do something completely different! It is a very special place and you should come visit! 🙂

Matthew Joyner16:06:47

Awesome talk !! 👏

Bryan Finster - Walmart (Speaker)16:06:22

@allspaw I frequently study reports for motorcycle accidents because I try to learn from others to stay alive. With an organization like mine that has so many stacks and so many different environments, what suggestions do you have to make incidents relevant to others in related stacks more obvious? I'd probably not attend a Mainframe incident review if I didn't think the information would help me improve.

☝️ 1
Bryan Finster - Walmart (Speaker)16:06:22

@allspaw I frequently study reports for motorcycle accidents because I try to learn from others to stay alive. With an organization like mine that has so many stacks and so many different environments, what suggestions do you have to make incidents relevant to others in related stacks more obvious? I'd probably not attend a Mainframe incident review if I didn't think the information would help me improve.

☝️ 1
John Allspaw16:06:27

Doing incident analysis well means understanding how others understood the event. When you can represent what was surprising or difficult for people closest to the incident (especially after they’ve expressed it in interviews) - in a compelling narrative, people start to build expectations that they can get something out of a well-captured analysis even if they’re not expert in the technical details.

👍 2
Daniel Cahill - Engineer - Ontario Systems16:06:27

@allspaw It was mentioned tracking specifically who is "voluntarily" reading the incident reports. If an important part of my job as a practioner is to get people excited about incident response, do you have suggestions to find ways to get people motivated and energized?

👍 5
Daniel Cahill - Engineer - Ontario Systems16:06:27

@allspaw It was mentioned tracking specifically who is "voluntarily" reading the incident reports. If an important part of my job as a practioner is to get people excited about incident response, do you have suggestions to find ways to get people motivated and energized?

👍 5
John Allspaw16:06:17

We find that practitioners are already enthusiastic…with engineers, once we get them talking about incidents…we can barely get them to stop!

John Allspaw16:06:07

The key is to identify what about the incident was/is mysterious or interesting to them, and represent that faithfully in the writeup. Once people see a demonstration that writeups can be a place where they can learn things they can’t elsewhere (and that it reads different than what they’ve been used to in past reports) … the interest will be difficult to stop.

👍 2
Ferrix Hovi - Head of DevOps - Siili16:06:37

I think there is some pride in finally finding the solution and you can tap into that to get them started and then you'll have to keep their timeboxes 😄

John Allspaw16:06:25

Yes, @U016HQVUEE4 - you can’t stop people from thinking of ideas, but you can instruct them to write them down during the group review meeting, so they can get away from the fear of losing the idea.

John Allspaw16:06:35

but yes: “engineers are gonna engineer”

Ferrix Hovi - Head of DevOps - Siili16:06:35

I guess it's a matter of engineering aloud

John Allspaw16:06:01

FWIW, we’ve worked with a client whose writeups later were getting 10-20 unique views per day up to 6 months after the incident, and people were still commenting, highlighting, and linking to it from other places

Ferrix Hovi - Head of DevOps - Siili16:06:25

Oh wow. Since engineers are not usually authors by profession, that level of interest is really impressive.

John Allspaw16:06:22

you might be interested to know that a significant part of the audience were folks from customer support, product management, and design

Ferrix Hovi - Head of DevOps - Siili16:06:24

I have noticed that side jobs as a stand-up comedian and a copywriter really help with writing exceptionally readable company-internal blog posts.

Ferrix Hovi - Head of DevOps - Siili16:06:30

So I'd go on to assume that there is a certain requirement for clarity and tone of voice to those documents that has made them very valuable outside the tech bubble.

John Allspaw16:06:53

Yep. Where there’s specific technical jargon, they’ll link to more tech details elsewhere. Where diagrams and pictures can be used to make things clearer, they’re used. Writing in plain language helps immensely.

John Allspaw16:06:36

A great deal is the idea that you’re writing them to be read, not just writing them to be filed.

Ferrix Hovi - Head of DevOps - Siili16:06:28

I have noticed that "dumb it down" is not the way to go but ruther "general smart it up"

👍 1
Gene Kim - Program Chair - ITREV16:06:35

JP missed his calling as a lawyer. 🙂

Slackbot16:06:21

Reminder: Don’t miss an opportunity for live Q&A with @allspaw at 6:25pm London time as a follow-up to his closing talk today. Join the discussion at https://us02web.zoom.us/j/83625560792?pwd=c1Y3b1dTTklqQ3NOTTkxa2N6SGxGdz09#success

alexbf (event staff)16:06:58

<!here> Jon Smart is in his happy hour room! Come join us here for an AMA: https://us02web.zoom.us/j/8908483265

John Willis16:06:20

I'll join the fun.. Here's a zoom link where I'll be hanging out to talk about the Handbook or Beyond the Phoenix Project.  https://us02web.zoom.us/j/87567853321?pwd=RVY2MjFQaUwxSmpQSWY0azRmTFNZZz09

Kurt A17:06:34

Following up in the @allspaw Q&A re Conway's law - the authors of Team Topologies use it to argue for intentionally structuring your teams in order to facilitate communication patterns that are helpful.

Kurt A17:06:47

@tom.ayerst ^^

Steve Spear19:06:32

@kboth_does attaching some materials on the “right way to organize” that might be of interest, apropos of Team Topologies reference and @allspaw+A. Happy to discuss.

📚 3
😀 1
👍 1
Steve Spear19:06:32

@kboth_does attaching some materials on the “right way to organize” that might be of interest, apropos of Team Topologies reference and @allspaw+A. Happy to discuss.

📚 3
😀 1
👍 1
Jeffrey Fredrick, Author, Agile Conversations06:06:03

More in my @steve773 read stack! Who needs sleep…

💯 2
Kurt A06:06:20

Thanks @steve773!

Steve Spear09:06:06

Thanks. Let me know what you think. Steve

Kurt A11:06:18

Right now these are joining my backlog of post-conference material to absorb. One downside of the virtual conference is that I don't have 10 hours in an airplane to work off the backlog. On the other hand, I did discover that jet lag happens even virtually - waking up at ~0300 BST this morning...

🙂 2