Fork me on GitHub
#plenary-discussion
<
2022-05-11
>
Gene Kim, ITREV, Program Chair08:05:21

Hello! It was so good to catch up with so many friends yesterday, and meet new ones — excited to start again this morning!

👏 5
❤️ 10
3
Gene Kim, ITREV, Program Chair09:05:10

Playlists in the Video Library are starting to answer the question of “where do we start?” https://videos.itrevolution.com/playlists/index.html

❤️ 2
Alex Broderick-Forster, IT Revolution, Event Staff09:05:15

Alex “playlists” Broderick-Forster right here

2
🎧 1
yerbamate 1
Ann Perry - IT Revolution09:05:52

Starting us off this morning is @moira.cheng, IT Operations - DevSecOps Transformation Programme Manager, Vodafone, here to present, Enabling IT Operations - Transforming to DevOps

👏 2
Gene Kim, ITREV, Program Chair09:05:22

Hello, @moira.cheng!! Thank you for sharing your story!

👋 2
👏 1
Moira Cheng09:05:04

Hi everyone 😄 So excited to have you listening today. Would love your feedback ❤️

👋 4
👏 2
❤️ 2
🎉 1
Quinn Daley09:05:43

Vodafone is adding 7,000 engineers? Just thinking about how hard it is for us to add even 1! 😄

🤯 11
👏 2
Moira Cheng09:05:40

Yes a big challenge.... includes upskilling and reskilling too 🙂

👍 1
Quinn Daley09:05:14

I will be taking lots of notes during this

❤️ 1
Sandeep Dabas09:05:15

Thanks Moira, I was writing the same 😉

Randy Shoup09:05:15

What is meant by 'insourcing'? Taking currently outsourced work in house with full-time employees?

Vlad Ukis09:05:39

I would be interested in how this large-scale hiring program is executed.

Matt Cobby (Director of Engineering, Deloitte)10:05:19

I think it is virtually impossible to hire 2, 5, 7000 skilled engineers but it is possible to hire 7000 passionate people and train them. Very smart.

Moira Cheng10:05:52

@rshoup yes on insourcing - bringing dev work in that we used to get done by 3rd parties - back in house with FTEs.

1
👍 1
🙌 1
Moira Cheng10:05:33

@vladyslav.ukis unfortunately I don't have that level of information. But up-skilling and reskilling is a big part of it.

Gene Kim, ITREV, Program Chair09:05:40

I learned from @moira.cheng how much Vodafone was changed by the acquisition of the Cable and Wireless enterprise business.

Gene Kim, ITREV, Program Chair09:05:32

“what? you want to build it and run it? it seemed like an odd thing back then…” 😆

❤️ 2
👍 1
jeff.gallimore (CTIO - Excella, he/him)09:05:49

that one platform team… sounds like the start of “the rebellion” within vodafone :rolling_on_the_floor_laughing:

💯 3
2
Gus Paul - Morgan Stanley09:05:58

Note that my partner from my talk yesterday is our head of change management. Collaboration is key!

👏 2
❤️ 3
Moira Cheng09:05:09

Yes Gus - you're on the top of my list to speak to and network with later 🙂

Gus Paul - Morgan Stanley09:05:28

Look forward to it 🙂

Clarissa Lucas, IT Audit Leader09:05:11

@gus.paul - Agreed! Check out the plenary talk on Agile Auditing this afternoon, where Audit and Technology leaders co-present on collaboration 🙂 Loving the high-collaboration talks!

Ann Marie Fred - Red Hat09:05:28

This is great:

❤️ 3
👍 1
jeff.gallimore (CTIO - Excella, he/him)09:05:08

“find the way or pave the way” 🙌

❤️ 6
Gene Kim, ITREV, Program Chair09:05:14

@moira.cheng: “I resolved to either find the way… or pave the way” Love it!

👏 5
jeff.gallimore (CTIO - Excella, he/him)09:05:18

@moira.cheng this is such an excellent analysis of the current state

❤️ 1
Gene Kim, ITREV, Program Chair09:05:19

“engagement would happen just before production” ah, yes. that. 🙂

😥 1
😭 1
Quinn Daley09:05:15

interestingly we have this same problem but it’s reversed - it’s the dev teams we need to encourage to be more collaborative, not the ops team

Quinn Daley09:05:36

(well, not just the dev but things like product and delivery on a product team)

Gus Paul - Morgan Stanley09:05:19

Moira are you sure you don't really work at Morgan Stanley? 😂

😂 1
😆 2
1
Sandeep Dabas09:05:36

When I saw your presentation, I thought of the same level because I was comparing both companies :)

Al Ramos (Morgan Stanley)10:05:05

Great presentation @moira.cheng! Happy to share Morgan Stanley’s Wealth Management Tech Operations and SRE journey! Over the past 6-7 years, we significantly reduced incidents by 70%, operating costs and toil.

🙌 1
❤️ 2
Gus Paul - Morgan Stanley10:05:26

Oh yes...Moria you definitely want to talk to Al as well as me!

Moira Cheng10:05:24

@aramosnj & @gus.paul for sure 100% looking forward to catching up.

😃 1
Gene Kim, ITREV, Program Chair09:05:51

it is funny to what extent there are universalities in these dynamics, isn’t it?

💯 1
Gene Kim, ITREV, Program Chair09:05:57

a call for volunteers! no budget, but fueled by people’s curiosity. so interesting!!

👏 1
1
❤️ 2
Luke Rettig - Target09:05:34

calling all people who want to be part of the Rebellion

🎯 4
Rudi van der Kaaden - TOPdesk09:05:11

intrinsic motivation is key in changing the mindset of 'the mass'

👍 1
Gene Kim, ITREV, Program Chair09:05:58

this is something I loved about @moira.cheng’s program — modeling “new ways of working” in all aspects of running this volunteer program. what a disarming way to teach and learn and participate!

❤️ 1
Ann Marie Fred - Red Hat09:05:26

I love your use of design thinking.

❤️ 1
Randy Shoup09:05:27

Learning the real challenges and resolving them 👍:skin-tone-5:

Gene Kim, ITREV, Program Chair09:05:43

“they had never used agile on a project before” (or so they thought! 🙂

Vaidik Kapoor (Speaker) - Technology Consultant09:05:34

❤️ that optimism @moira.cheng! #failfastandlearn

👍 1
Gus Paul - Morgan Stanley09:05:36

curious how much senior management sponsorship you had. or at what point...i felt without that we'd have way less people paying attention.

👍 1
Moira Cheng09:05:13

I was really fortunate Gus. My senior management team empowered me to do it how I want, in any way I want. I haven't disappointed them so far 😂. Still have their support 💯

❤️ 2
Musthafa Soukathali09:05:44

Here it comes...Hearing SAFe for the first time since yday!

jeff.gallimore (CTIO - Excella, he/him)09:05:01

“what’s the worst that could happen?” a phrase used by many great change-agents and innovators :rolling_on_the_floor_laughing:

💯 2
😂 2
❤️ 1
Ann Marie Fred - Red Hat09:05:06

Call for Code uses Agile planning with volunteers, too! Sometimes it’s the best way to make sure the team gets aligned and actually finishes things.

Radoslaw Wiankowski09:05:14

That’s what true DevOps is to me. Ways of working, getting people together, asking them to participate and co-create. Not just technical gimmicks. ❤️

❤️ 1
Moira Cheng09:05:54

Thank you @rwiankowski that's a real complement for me ❤️

Gene Kim, ITREV, Program Chair09:05:27

marshaling all the energy of volunteers who want to solve their parochial problems. neat.

jeff.gallimore (CTIO - Excella, he/him)09:05:14

a great way to connect to the “what’s in it for me” (WIIFM)

Robert Ruzitschka - DevOps Guild Lead09:05:48

A lot of that resonates strongly!

Scott Prugh (DOES Prog Committee)09:05:03

My good friend Mauricio always said "Its not possible to make it any worse right?"

❤️ 3
😆 1
Ann Marie Fred - Red Hat09:05:45

Survey - always a good start.

❤️ 1
Jack Lagare09:05:14

I’m off to a good start then. 😆 When I took over leadership of an internal services team, surveys became my tool to understand better what the current situation is (aside from actual conversations of course), what are individual aspirations, etc.

1
Gene Kim, ITREV, Program Chair09:05:48

50% responded to survey — how many, @moira.cheng? So good!

👏 2
Gus Paul - Morgan Stanley09:05:36

Our experience was engineers were not bothered about agile, but we had massive engagnement on devops concepts. Meet the people where they are...

👍 1
Robert Ruzitschka - DevOps Guild Lead09:05:24

Coming from a bigger enterprise organization, I feel that the first thing that you want to achieve is changing the conversations about Engineering.

Moira Cheng09:05:21

Keen to explore in more detail what kind of Engineering conversations you mean @robert.ruzitschka. Genuinely intrigued. Lets connect

Robert Ruzitschka - DevOps Guild Lead09:05:44

Happy to! Will talk tomorrow about our experiences.

👍 1
Robert Ruzitschka - DevOps Guild Lead09:05:31

I see a huge overlap in approach and experiences. Seems good 😀.

👍 1
Ann Marie Fred - Red Hat09:05:49

49% learning outside of working hours, wow.

Gene Kim, ITREV, Program Chair09:05:05

What incredible evidence that people want to learn — and signals of their aspirations.

👏 1
💯 1
Anshul09:05:06

@moira.cheng - can you take an example of a challenge you mentioned?

Moira Cheng10:05:06

@anshul.kumarbansal which types of challenges was i talking about? Local Market Challenges or DevOps Adoption (survey results) challenges? Local Markets were more about lack of involvement in agile projects, knowledge, resourcing, process perceptions (well we had about 63 pain points overall). Survey results for IT Operations indicated they didn't have the opportunity, knowledge or training.

Robert Ruzitschka - DevOps Guild Lead09:05:29

That numbers are pretty astonishing!

1
Mike Wilkinson09:05:06

In the drive for “productivity” - the space for learning has been massively reduced! Huge problem in the services industry at the moment

☝️ 1
👀 1
jeff.gallimore (CTIO - Excella, he/him)09:05:08

i believe @ben.grinnell will be sharing about this tomorrow

👍 1
Radoslaw Wiankowski09:05:53

It’s probably a problem everywhere. I see that so often both internally and at customers. People are so swamped with work that they don’t have the headspace to learn, not even mentioning the time…

Scott Prugh (DOES Prog Committee)09:05:59

@ikrnic Spoke about this yesterday!!

🙌 1
🙇 2
Radoslaw Wiankowski09:05:52

thanks, will catch up!

👋 1
Ivan Krnic (Director of Engineering at CROZ)10:05:22

Yes, really a problem - one way we tackled this was reducing the amount of non-strategic work in the funnel to open up capacity for experimentation... basically, lean portfolio management principles applied.

Moira Cheng10:05:32

We talked about this in our Birds of a Feather session on Culture and Leadership yesterday. Some suggestions were booking fixed capacity in Agile projects for learning. 70/20/10 rule mentioned By @ferrix I know as a rule, this is hard to enforce in individual virtual agile projects and overall professional development (outside of projects) needs to be catered for too.

Ferrix Hovi - Head of DevOps - Siili10:05:33

The 70:20:10 (customer needs / improvement of daily work / personal growth) is hard to push through. If the top management agrees that this should be the way, there might be some middle-management friction which requires assertiveness from teams to actually do it.

👍 1
Radoslaw Wiankowski10:05:38

I experience a different challenge. The team members have such a strong sense of responsibility that they do it to themselves. With the 10/15/20 % of time promised to engineers for learning and experimentation, people don’t use it because they care so much to get the work done and provide the quality. I see more and more the importance of the middle management in this - they need to fully embrace it, to the point that they actively encourage engineers to learn and develop.

👍 1
Steffen Kreusch09:05:23

It´s true that we had many volunteers as part of Moira´s programme with very little experience on how to run agile projects but the decision to get it done in this way was the best decision ever as this gave me awesome insights into this matter which I wouldn´t have learned by all the trainings we have available in our Vodafone University. I genuinely love this this programme ❤️

❤️ 2
3
👍 1
Gene Kim, ITREV, Program Chair09:05:40

It’s a startling and astonishing initiative — kudos!!! 🎉

Steffen Kreusch09:05:01

thanks Kim, that´s what it is indeed:slightly_smiling_face:

Gene Kim, ITREV, Program Chair09:05:49

Such startling numbers — I wonder if it’s because @moira.cheng targeted individual learning and aspirations, as opposed to… sorry, grasping for words, but something like “you’re going to learn DevOps because It’s Good For You.”

1
Gus Paul - Morgan Stanley09:05:14

there are many enterprise engineers out there who are eager to change they way they work once they are given "permission". The latent enthusiam is amazing

🎉 3
👏 1
Matt Cobby (Director of Engineering, Deloitte)10:05:16

We created a team based around engineer training, certification and developer productivity and unleashed a wave of enthusiasm. 7000 people trained later, 1500 certified... all by giving permission. No-one was forced, nothing was mandated. All by free choice.

👍 2
Nick Eggleston, IT Architect (DMs welcome)10:05:06

Invite vs inflict (which I learned from @jonathansmart1 )

👍 1
Gene Kim, ITREV, Program Chair09:05:23

learning and defining/identifying conditions that were favorable to devops.

💯 1
Gene Kim, ITREV, Program Chair09:05:21

😆 “no more CABs”. Now I’m sure that she wasn’t at MS, @gus.paul. 😆

Scott Prugh (DOES Prog Committee)09:05:56

One of my favorite days at work was when we cancelled CAB!!

Gus Paul - Morgan Stanley09:05:36

Team Topologies shows up again. My favourite book!

🙌 3
❤️ 13
😂 2
ttop 1
Pulak - Accenture DevSecOps Practice Lead10:05:07

@matthew @me1208 feel proud

😄 1
🙌 1
Peter Fassbinder09:05:28

Great talk - how many FTEs did / do you have in the tranformation core team? For how long do you plan to run this transformation program?

Moira Cheng09:05:00

Right now - dedicated to the programme full time 1 FTE (me) and 2 x 0.5 FTE. The rest of the volunteers (circa 40) are part time - roughly spending 4-8 hours a week. And NO CONTRACTORS 😂

🎯 1
Gus Paul - Morgan Stanley09:05:19

Lots of famiilar learnigns here from our early days. We also used gamefication for teams adopting practices by giving them badges right on their source code repos. that turbo charged our adoption

3
Marco Delgado [Ocado Technology, Engineering Practice Lead]09:05:38

Maybe jumping ahead but could you share some of the unified DevOps KPIs you used?

Moira Cheng09:05:28

Yes we have a view of them (MVP KPI list and Non MVP KPI list), but they're not all implemented yet. We're working on trying to make them real - getting source data from various pipeline and reporting systems - and doing a PoC to pull them all together this Increment.

Marco Delgado [Ocado Technology, Engineering Practice Lead]09:05:01

cool thanks, did you use the DORA metrics for this or other custom ones?

Moira Cheng13:05:29

Hi @marco.delgado ok so we're attempting to start our Unified DevOps KPIs with this list - Selected for MVP: MVP: Predictability: JIRA/ADO MVP: Incidents/MTTx: ITSM (=>Central Tool =>Andrea Martin's dashboard) MVP: %Success Deployment PRD: ITSM (how to connect incidents post change) MPV: Mean Cycle Time: JIRA/ADO MVP: Deployment Time: Pipeline (Azure/Jenkins/Gitxx) MVP: SEMM Maturiy: SEMM App MVP: ITOM Maturity: Excel MVP: Throughput: JIRA/ADO MVP: Code Quality: SonarQube Enterprise

Gene Kim, ITREV, Program Chair09:05:48

middleware application! used SRE techniques!

Gene Kim, ITREV, Program Chair09:05:19

“Guidance for IT Operations Managers”

💯 1
Radoslaw Wiankowski09:05:32

the pressurised middle management come up again… That’s an amazing idea I’m going to steal from Moira :)

Scott Prugh (DOES Prog Committee)09:05:28

Turns out that DevOps is technology agnostic!!

1
💯 1
Luke Rettig - Target09:05:49

this is such a great talk and a great framework for anyone out there trying to jump start their company’s transformation. this should be a 🔖 for any Tech Leader who just finished The Phoenix Project and is scratching his/her head about how where to start

🎯 3
❤️ 1
Ann Marie Fred - Red Hat09:05:09

Agreed, it’s a transformation playbook.

Nick Eggleston, IT Architect (DMs welcome)10:05:55

Which playbook? (For those catching up)

Luke Rettig - Target10:05:45

@nickeggleston - check out the opening talk from Moira from Vodafone.

Moira Cheng10:05:51

Oh my word. Thank you @lucas.rettig and @ann.marie.99 That is such a high complement. Thank you. Never did I expect a year ago, that we'd come this far so soon. There's so much more to do though.❤️

🙌 1
Ann Marie Fred - Red Hat10:05:29

A lot of us muddle through similar efforts, and your summary and explanation of the process is excellent, @moira.cheng. 👍 Looking forward to hearing how it’s going again in a year or two. 🙂

❤️ 1
Nick Eggleston, IT Architect (DMs welcome)13:05:04

It's exciting to see how much you can accomplish so quickly

❤️ 1
Clarissa Lucas, IT Audit Leader09:05:53

@moira.cheng - This playbook information is very intriguing and is sparking a lot of ideas on how I can apply a similar concept to my audit team's journey. Can you share more information about the maturity assessment and unified DevOps KPIs?

👍 4
Gus Paul - Morgan Stanley09:05:16

Not Vodafone, but we have very strictly stuck to the DORA metrics when showing progress. We resisted a lot of push to use other KPIs at the start but we've shown the value of sticking ot those 4 simple metrics now

👍 3
Gus Paul - Morgan Stanley09:05:56

Having a whole squad dedicated to meaurement has been key though. the PO of our metrics squad is a genius

Clarissa Lucas, IT Audit Leader09:05:47

@gus.paul a squad dedicated to measurement?! 🤯 Genius is an understatement!

Nick Eggleston, IT Architect (DMs welcome)10:05:03

Does this help ensure quality in metrics, i.e. the same metrics consistently get measured in the same way, having the same meaning, and thus can be usefully compared?

Gus Paul - Morgan Stanley10:05:21

Yes exactly. It's the foundation of all we do. It's also resulted in a data lake where all our jiras/build stats/changes/incidents etc are all in one place

🎯 3
Moira Cheng11:05:56

That's exactly what we're trying to get to @gus.paul - Wish i had a whole squad and a dedicated PO on this though!

Moira Cheng11:05:29

@lucasc5 - yes i'm trying to see how we can share the maturity assessments - one is digital! On KPIs we have a list of Metrics (the MVP ones and non-MVP ones) i can share. The aim in this PI is to try and collate the MVP metrics from our source systems (whether it's pipeline tools or ITSM tools) or to see how much effort it would take to pull them all into a visible datalake and dashboard.

Werner Pantke11:05:58

Sharing that list would be great 👍

Moira Cheng13:05:05

@lucasc5 @werner.vaarwerk This is what we're attempting first as MVP MVP: Predictability: JIRA/ADO MVP: Incidents/MTTx: ITSM (=>Central Tool MVP: %Success Deployment PRD: ITSM (how to connect incidents post change) MPV: Mean Cycle Time: JIRA/ADO MVP: Deployment Time: Pipeline (Azure/Jenkins/Gitxx) MVP: SEMM Maturiy: SEMM App MVP: ITOM Maturity: Excel MVP: Throughput: JIRA/ADO MVP: Code Quality: SonarQube Enterprise

👍 1
Gene Kim, ITREV, Program Chair09:05:17

@moira.cheng this is such a different narrative than the stories of “frozen middle” — what do you attribute that to? So neat!

❤️ 2
Scott Prugh (DOES Prog Committee)09:05:51

It would be interesting to understand what capabilities help combat the frozen middle problem

Moira Cheng10:05:50

Hmm... simply put i think getting the 'frozen- middle' bought into what we're trying to do. Acknowledging that their teams need to learn and evolve, and even better, getting them to nominate individuals who can co-create with us, so they're part of that evolution journey.

am09:05:28

Why are almost all the speakers describing the world before DevOps as "Waterfall and monolithic"? I mean, Agile and SOA predate DevOps.

Musthafa Soukathali09:05:39

guidance for IT operations is a real need in large corporations. In many companies ops is 60% of the IT work. They do need a playbook like what @moira.cheng shows here

👍 3
Gene Kim, ITREV, Program Chair09:05:41

inform upcoming technology upgrade choices!

Pulak - Accenture DevSecOps Practice Lead09:05:41

@moira.cheng you seem to have that "relentless optimism" we must all strive for all the time

🎉 4
1
Moira Cheng10:05:19

Well everyone has their odd moments, but I have to say I am naturally inclined to be optimistic. And when working with volunteers - I want them to be happy working on the programme. Optimism and motivation spreads naturally through others 🙂

David Hawes-Johnson (DevOps Enablement - BT)09:05:48

This is a really nice journey - personally I find the playbook approach much better than the "follow the methodology" that is often used. Giving teams the guidance and information they need to determine their own path within the guardrail and towards a shared outcome.

👍 3
thankyou 1
Randy Shoup09:05:57

For the skeptics, how did you motivate them to get involved? Was it a coalition of the willing from the beginning?

Moira Cheng10:05:28

Hi @rshoup I have to say, because we are running with volunteers - i've hardly had any skeptics. The Vodafone Tech 2025 strategy, and our OneTechTeam Digital & IT vision also helps. It was a coalition of the willing - yes - from the beginning. I'm a firm believer that skeptics will learn from the success of others given time 🙂

👍 1
Randy Shoup10:05:38

👍:skin-tone-5:

Marcelo Pazzinatto09:05:05

great talk at @moira.cheng. Really resonating with me as I'm currently working on the (often arguably neglected) 'Ops' side of DevOps transformations. DevOps principles of 'you build it you run' it is the starting point, but embedding Ops in code and getting product teams to understand 'how' to do that requires stuff like SRE techniques (not necessarily roles), and the other elements of your playbook. Great stuff.

👍 4
❤️ 1
👏 1
Neil Waller09:05:03

@moira.cheng this is a great and refreshing talk, some really useful points in here. I am in a similar position so some really useful points for to reuse. Thanks so much

👍 1
👏 1
❤️ 1
Musthafa Soukathali09:05:48

@rshoup - skeptics - tough job to motivate them through sessions and presentations. But they do notice when they see their peers do great (motivation through peer pressure works!)

👍 2
1
Gus Paul - Morgan Stanley09:05:48

in a big compamy socialising the work is a full time job ...it takes a while but it pays dividends in the end. We have 1500+ at our quarterly internal devops days now

Moira Cheng10:05:32

@gus.paul wow! Think i can learn some lessons from you on this. Would be great to find out more about your journey (other than Automating Change Management)

Gene Kim, ITREV, Program Chair09:05:48

“let’s go to market! let’s launch!”

1
Ann Marie Fred - Red Hat09:05:49

I think that we need to get our SREs to sit with our developers 1-1 to define SLIs in particular. Otherwise it doesn’t work.

❤️ 1
Vlad Ukis12:05:01

Why would 1-1 be required?

Ann Marie Fred - Red Hat12:05:54

The SREs are usually much better at parsing through cryptic logs to generate the metrics that the business wants. And the developers know enough about their application behavior and logs to help the SREs find the right information, or to add logging if needed. You need the domain knowledge of the application and the log aggregation / searching / alerting. Sometimes you get lucky and they’re the same person, but not often.

Vlad Ukis13:05:01

Yes, sure but why would "1 SRE talking to 1 Team" not work instead of "1 SRE talking to 1 Dev"?

Ann Marie Fred - Red Hat18:05:55

I think you only need one of each? It’s usually very application specific. What were you thinking of, Vlad?

Vlad Ukis19:05:53

Perhaps I misunderstood you. My impression was that e.g. if you have a 200 devs org, you would have SREs talk to the 200 devs.

Ann Marie Fred - Red Hat13:05:04

Ah, gotcha. Not like that. But say my 200 devs are divided across 30 smaller teams who work together, and they have 10 SREs who work on the service. I would want to have some general training for an hour or two (like a presentation) for everyone. But I would also want each of those 10 SREs to spend a day or two sitting with about 3 developers - training 1 developer per smaller team - over the course of a few weeks. That’s pair programming to hand-hold the developers through learning how to instrument their own application logs and how they (or SRE) will define the metrics and monitor them. And then those developers can teach the rest of their team how to work with SRE. I think we need that.

Vlad Ukis13:05:34

Yes, understood! That make sense 🙂

😁 1
David Hawes-Johnson (DevOps Enablement - BT)09:05:53

Would be interesting to understand if SRE (ops) is taking away focus away from DevOps. (EDIT: i.e. does SRE and the OPS association make it easier for some enterprises to look at SRE as a more traditional model).

Matt Cobby (Director of Engineering, Deloitte)10:05:29

I don't think so. I know those that say SRE is the new DevOps and I think that is based on a misunderstanding of DevOps (automation vs value delivery via software). DevOps, for me, was very much about shift left, removing silos, removing waste, reduced time to value. SRE is a specialisation in that story taking the lessons learnt from DevOps and Software Delivery to every day operations. SRE does appear to have a much easier path as it fits into the mindset of ITIL and Service Management. The aspects of SRE around control, quality of service fid an easy home in a traditional enterprise. The question of error budgets less so.

👍 1
Moira Cheng10:05:53

Hi @slack1599 I don't think SRE is taking away focus per say from DevOps, but i do believe that SRE practices are easily applied and can be a first step to improving traditional IT solutions. The fact that SRE practices also promote a DevOps culture and feedback loops with Dev is the start to a blossoming collaboration and opens more doors for full DevOps application/adoption

👍 1
Marcelo Pazzinatto10:05:11

@slack1599 @matthew.cobby @moira.cheng SRE helps with the 'how' to do Ops in DevOps, very much a collection of principles and techniques like SLO and Toil management to focus on customer-based reliability and continuous automation in the Ops space. DevOps itself doesn't get that prescriptive. I think the DevOps vs SRE debate started because (at Google at least) SREs are dedicated roles aligned to but outside their product teams (for their critical systems at least, for others it's you build it you run it). Thing is, the SRE techniques can be used with or without SRE 'roles'. You don't even need to call it SRE to avoid confusion if that's the case. We're using that approach as we embed Ops into product teams i.e. we're merely asking/teaching our DevOps product teams to use key SRE techniques as we 'expose' them to production more directly and ask them to run it. The same principles of collaboration, automation, etc are there. SRE implements DevOps.

❤️ 1
👍 1
1
Moira Cheng13:05:04

Will look out for this @vladyslav.ukis Thanks for sharing

Marcelo Pazzinatto13:05:47

me, too. Thanks @vladyslav.ukis 🙂

Marcelo Pazzinatto13:05:51

perhaps we need more talks covering this theme here in future DOES events.

Moira Cheng13:05:31

@marcelo.pazzinatto this is exactly what we're trying to achieve here. We're looking at ensuring SRE techniques can be used by all, and are not perceived to just something a specific Team / Role (e.g. Software Engineers) can do. I think the Google definition has actually caused some misinterpretations that only specific roles or teams can apply SRE practices and techniques.

Marcelo Pazzinatto13:05:13

I agree on both counts. As a collection of techniques helps balance velocity vs reliability and control toil levels, perfectly suitable for product teams in full DevOps mode. I think for large enterprises still transforming, it also helps with the journey to 'embed' Ops into their product teams. It's a different, arguably easier story for greenfield, cloud-native and new teams....

👍 1
Marcelo Pazzinatto13:05:52

I don't remember a DevOps conference (including Google's own Next) where this DevOps vs SRE theme didn't come up 🙂

👍 2
Gene Kim, ITREV, Program Chair09:05:04

“our philosophy: the more the merrier!”

Musthafa Soukathali09:05:44

I think that the first clarification that should happen is - clarify what you are after - Site RE or Service RE. I see lot of mixup on these

💡 1
Gene Kim, ITREV, Program Chair09:05:46

“how can we help each other reduce MTTR?” a surefire way to start a great conversation!

👏 3
Toli09:05:46

@genek this often is followed by ‘how do we measure it? Let’s measure it!’ Have you seen this done well? Should it be measured quantitatively at all? Is it more about confidence?

👍 1
Randy Shoup09:05:46

We've found great value in identifying and motivating DevOps Ambassadors. Huge unlock. (We call them "Velocity Champions", though ;-))

👍 3
👀 1
Gus Paul - Morgan Stanley09:05:17

Similar. we have an internal change agent network of wnthsiasts to spread the word

Matt Cobby (Director of Engineering, Deloitte)09:05:56

I like Velocity Champions.... it speaks to the delivery of value at PACE. Too often a DevOps Champion was usually the person who ran Jenkins. :rolling_on_the_floor_laughing:. I might use this way of describing it.

Randy Shoup09:05:09

Our initiative is the "Velocity Initiative", so ...

Alvin Chuang09:05:02

I was at Vodafone way back when, it's so fantastic to hear the progress you've made in modernizing - thanks for the great talk @moira.cheng!

❤️ 1
Quinn Daley09:05:12

“if you’re a delivery manager please try to involve your operations team earlier” - YES

❤️ 1
Scott Prugh (DOES Prog Committee)09:05:49

@moira.cheng I love getting Ops Features into a shared backlog!!

👍 2
Luke Rettig - Target09:05:11

yes! - because they are required features! Product peeps need some upskilling too

Moira Cheng09:05:55

Absolutely agree - we need to change the tide here. That's why getting Ops involve early in the Agile project team set up / structure is key. Another thing is Product owner vs. Service Owner concept. Product owners need to have accountability for service and ongoing in-life operational and CX improvements too. This unfortunately does not often get fair prioritization in the overall backlog. Education of Product Owners, Product Managers and Project Managers needed. Ops is nowhere in SAFe

👍 1
Toli09:05:02

Great talk @moira.cheng! We have a version of DevOps ambassadors embedded in teams and it worked well for us. The evolution of this is a Community of Practice around DevOps and its socio-technical practices

👍 2
Musthafa Soukathali09:05:03

Simple yet clean journey @moira.cheng. Thank you

Gus Paul - Morgan Stanley09:05:30

One challenge we have is operations team supporting 5 or 6 or more different stream aligned teams....so they dont have the bandwidth to be fully engaged with every dev squad

👍 1
Moira Cheng09:05:29

Yes @gus.paul we've had to reassess how we structure our Ops teams and are providing guidance to Ops Team managers on how to scale. This is to enable participation in Agile project ceremonies, have new skills in planning - a bit more upfront. Happy to really get into the details 121.

Nick Eggleston, IT Architect (DMs welcome)10:05:20

It would be great if you had that discussion in Gather or somewhere interested folks can listen and learn 🥐

Scott Prugh (DOES Prog Committee)09:05:37

@moira.cheng We really need that magic DevOps Spell!! Where can we buy that?

😆 2
1
1
Quinn Daley09:05:39

thank you @moira.cheng!

Manikandan Balasubramanian09:05:48

Great talk @moira.cheng. Thank you

Ann Marie Fred - Red Hat09:05:09

Thank you @moira.cheng - my favorite talk of the Summit so far.

❤️ 2
Gene Kim, ITREV, Program Chair09:05:25

Hello, @ben.connolly! I enjoyed your session on OKRs last year!

Sandeep Dabas09:05:26

Looking forward to see DevOps, Me being in IT Operations in Vodafone 🙂 Amazing presentation 👏 @moira.cheng

❤️ 1
👏 2
Moira Cheng09:05:20

@sandeep.dabas get in touch, we'd be glad to welcome you into our programme or as a DevOps Ambassador:slightly_smiling_face:

❤️ 1
Sandeep Dabas09:05:06

Sure will do 🙂 Thanks.

Ian Richards09:05:50

Great talk, thank you @moira.cheng

👍 1
Marcus Cziomer09:05:51

That was very inspiring @moira.cheng! Hope you don't mind my stealing some ideas ;)

Moira Cheng09:05:59

Of course, it would be my pleasure if anyone took anything from my session and re-used it elsewhere. Happy to connect too if you'd like more details 🙂

Marcus Cziomer10:05:10

I will surely connect! I am already in Confluence trying to articulate how to get some of your learnings into our own DevOps journey 😊

❤️ 1
Moira Cheng11:05:11

Coincidently Confluence is where we're putting our Playbook:smile: 🎯

❤️ 1
Scott Prugh (DOES Prog Committee)09:05:57

Great talk @moira.cheng!!! Keep up the great work!

👏 1
❤️ 1
Radoslaw Wiankowski09:05:10

@moira.cheng such an amazing story! what a holistic approach. And that you’ve started looking at it only last year! a huge WOW! I’m sure your playbook will be used as inspiration and guidance by many. Thank you!

❤️ 1
thankyou 1
Moira Cheng09:05:30

Thank you so much. I'm really glad you enjoyed it ❤️

Gene Kim, ITREV, Program Chair09:05:13

“has helped us blast thru the inertia…”

Boris Martinovic09:05:15

Thank you @moira.cheng for sharing your journey!

👍 1
Moira Cheng09:05:18

For those of you i haven't replied to yet... I will shortly. Sorry haven't been able to keep up with all the positive comments and questions 😁

❤️ 7
👍 1
BMK-SECTION6-TransformationArchitect09:05:31

Well done, Kudos to you and your team

thankyou 1
Gene Kim, ITREV, Program Chair09:05:47

1. by our people, for our people; empowered, engaged, by goals they set 2. reaching out to business to get their goals 3. stronger voice to Ops community

👍 2
👏 4
❤️ 1
1
Alex Broderick-Forster, IT Revolution, Event Staff09:05:07

Enabling IT Operations - Transforming to DevOps (Vodafone) Moira Cheng, IT Operations - DevSecOps Transformation Programme Manager, Vodafone 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708550161/

🙏 3
🔖 4
Gene Kim, ITREV, Program Chair09:05:07

Thank you, @moira.cheng!!! Awesome!

👍 3
🎉 2
❤️ 2
Matt Cobby (Director of Engineering, Deloitte)09:05:28

Zero Budget transformation, no-one forced to be on it. Nice.

BMK-SECTION6-TransformationArchitect09:05:02

:thinking_face: - @matthew.cobby I am thinking where you were and where I am 🙂

Matt Cobby (Director of Engineering, Deloitte)09:05:15

😂. I know I now need budget but with the right team and the right motivation, you can shift a company.

💯 1
Ann Perry - IT Revolution09:05:29

And now, please welcome @simone.steel, Chief Data and Analytics Officer & CIO for Enterprise Data Platforms, Nationwide Building Society, presenting From DevOps to DataOps

👍 1
👏 2
Peter Fassbinder09:05:51

@moira.cheng 👏

thankyou 1
Gene Kim, ITREV, Program Chair09:05:42

Hello, @simone.steel! I’m so delighted that you’ll be teaching us today, pointing us to something that is elusive, profound, and important, relevant to everyone!

Gene Kim, ITREV, Program Chair09:05:30

among other goals: “make best possible use of data for our membership; keep data secure, safe, compliant”

Manikandan Balasubramanian09:05:48

@simone.steel, Looking forward to your session on DataOps

Gene Kim, ITREV, Program Chair09:05:29

17MM members; 15K colleagues in branches/offices

Leena Pradhan09:05:32

Great story @moira.cheng ! Loved it.

👍 1
Musthafa Soukathali09:05:43

'member' and not just a 'customer'

👍 2
💯 1
Luke Rettig - Target09:05:15

at my company, Target - large US retailer, we call our customers our “guests.” Because, how do you treat your guests when they come to your house?

❤️ 3
Musthafa Soukathali09:05:41

That makes the company as - company with purpose

❤️ 2
Rich James09:05:47

This is a nuance of a mutual. We are a membership organisation, and owned by our members - not share holders.

👍 2
BMK-SECTION6-TransformationArchitect09:05:32

Aha - Value Streams @simone.steel - dropped my pen, paying my full attention now

👍 1
Gene Kim, ITREV, Program Chair09:05:01

Key responsibilities: • customer master data • making data available • data governance • analytics

✔️ 2
Simone Steel09:05:46

+ Business insights (BI, MIS)

Gene Kim, ITREV, Program Chair09:05:48

(for decades, I’ve wanted a word that means “property of having integrity”: @simone.steel used “integral.” In past, I’ve tried using “integrous”)

👍 2
❤️ 1
Gene Kim, ITREV, Program Chair09:05:34

“my job in a nutshell: to ensure that use and management of data in the Society is ethical, valuable, safe and compliant”

Gene Kim, ITREV, Program Chair09:05:30

fascinating; mortgage of properties; we have access on home carbon emissions; how can we help members finance homes to make their homes greener

🙌 3
Gene Kim, ITREV, Program Chair09:05:05

I love all these beautiful titles; Chief Proposition and Marketing Officer!!!! Brilliant.

👍 1
Musthafa Soukathali09:05:50

Members' properties are safe and efficient -> our business is safe and efficient. What a purposeful business model!

❤️ 1
Lilian Mitroglou - DevOps Coach10:05:26

DevSecFinDataQualityOps 😄

😆 2
1
BMK-SECTION6-TransformationArchitect09:05:23

I am curious - how many data folks here attending DOES (Data Engineer, Scientist, Data modelers, DBAs, Analytics, BI)? cc - @genek

1
Gene Kim, ITREV, Program Chair09:05:49

earliest data records: 200 BCE: celestial motions. (!!)

jeff.gallimore (CTIO - Excella, he/him)09:05:14

the origin story of data analysis!

❤️ 1
Gene Kim, ITREV, Program Chair09:05:06

“do we know what purpose of data is?”

Quinn Daley09:05:30

people still using Microsoft Excel 2200 years later

😂 2
😀 2
😄 1
Szymon Osiecki09:05:55

Data Analyst - one of the oldest profession in the world ;)

🙂 1
✔️ 1
Gene Kim, ITREV, Program Chair09:05:03

dawn of enterprise data science; job of the data scientist is arguably the job of geneticist, physicist, economist; but now it’s an enterprise capability

👍 1
Gene Kim, ITREV, Program Chair09:05:37

“what is flowing through the software is the data of the business”

👌 2
💯 1
Gene Kim, ITREV, Program Chair09:05:43

“data professionals are citizen developers, not End Users” <<<<<

Gene Kim, ITREV, Program Chair09:05:13

From @cbergh: “between 30-50% of company employees use or manipulate data in their daily work” (arguably making this problem space bigger than devops!)

☝️ 2
🤯 1
BMK-SECTION6-TransformationArchitect09:05:25

agree - It is already too late, but we need to bring our Data friends to this DevOps party and bring them to this mainstream. Without Data we cannot move forward.

Luke Rettig - Target10:05:18

its quite the conundrum. I use this quote in my products on why our teams need to ruthlessly investigate every “export to Excel” that happens in our system. Data is an integral part of how to drive your business and needs to be highly aligned with your strategy (unified Business/Tech)

👍 1
BMK-SECTION6-TransformationArchitect10:05:41

The power of data - some enterprises are so so behind; Most of the compliance, privacy is about "DATA"

BMK-SECTION6-TransformationArchitect10:05:39

Agree, recently I was listening to one of the org presentation (nothing to do with DevOps) they were saying Data Driven Decisions and Data Insights in their strategy for Years 2022; For me this sounds like they are 5 years behind thinking like this.

💯 1
jeff.gallimore (CTIO - Excella, he/him)09:05:00

:rolling_on_the_floor_laughing: 😳

😆 1
Vaidik Kapoor (Speaker) - Technology Consultant09:05:58

where have we seen this before? :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing:

1
Szymon Osiecki09:05:19

There is nothing better than spending a day or two on automating a thing that takes 5 minutes 😛

1
😃 2
Gus Paul - Morgan Stanley09:05:50

LAzy developers are the best developers. "Why am i always doing this 5 minute thing...let me automate it"

❤️ 2
Gene Kim, ITREV, Program Chair10:05:37

Makes me think how often I use Excel to assemble/explore data, or do one computation.

👍 1
Gene Kim, ITREV, Program Chair09:05:34

“getting the data pipelines working; not databases, data lakes, data warehouses. data pipelines…”

❤️ 1
jeff.gallimore (CTIO - Excella, he/him)10:05:30

like an “insights value stream”…

Musthafa Soukathali10:05:25

Only person with a common goal is just the CEO usually. Looking forward to hear how you brought that down to lower levels

🤯 1
😀 2
Simone Steel10:05:16

Unfortunately this is all too common. Part of the education here is for the C-suite!

🤯 1
Gene Kim, ITREV, Program Chair10:05:06

• what is the hypothesis that biz is trying to prove/disprove? • how do I connect data domains that may be very far apart in org (!!) • produce actionable insight

👍 1
Gene Kim, ITREV, Program Chair10:05:41

“don’t use DataOps as an expression meaning “devops for data”"

BMK-SECTION6-TransformationArchitect10:05:40

I do have question why most of u sdo not see "DevOps" including Data - instead we want DataOps? @simone.steel

👀 1
👍 3
👏 1
Simone Steel10:05:28

My instinct is that we focus on improving what is under our span of control (am I a tech professional or a data professional?) I.e. changing practices I can control is easier than creating common (cross-departmental) practices.

💯 1
1
Gene Kim, ITREV, Program Chair10:05:33

I think what is so fascinating about this talk is that we don’t have a (widely used) name for the concept of what @simone.steel is talking about. Seems to include reframing what the actual inputs/outputs of applications/orgs should be. Thoughts, @scott.prugh @rshoup (my fave architecture)

🙏 1
Randy Shoup10:05:01

Agreed. As always, start with clear problem statement(s), identify impediments and bottlenecks, start knocking them down.

Luke Rettig - Target10:05:59

In my opinion, its a take on Conway’s law, but instead of your architecture being a representative of your org structure, its how you influence, drive change, name things, etc, being a representative of where you sit in your org and what your title is. Once again - we have to align on our common problems, and tear down those silos.

💯 4
Gene Kim, ITREV, Program Chair10:05:29

“data: some deterministic; some predictive (??)” <--- did I get that framing right? Such a neat way to unify those two types of computation going on.

Simone Steel10:05:14

I think this classification is helpful because we can explain and replicate software behaviour (deterministic), but we cannot do it in the same way with predictive analytics (non-deterministic, learning all the time).

Gene Kim, ITREV, Program Chair10:05:37

“data is often treated as IT projects; instead, it is a power user domain”

Musthafa Soukathali10:05:15

Analytics projects are different animals than IT project. 100%.

1
BMK-SECTION6-TransformationArchitect10:05:43

Most of the organizations - We lost track of how many ETL jobs are running, active and not in use; duplicate data sets in number of data lakes; Often the argument is about "Who owns the data?" - common assets like Customer, Products are good example for this.

👍 2
Simone Steel10:05:54

Even more worrying if we think about environmental and sustainability views: #dataisnotfree #dataisnotgreen

💯 1
jeff.gallimore (CTIO - Excella, he/him)10:05:43

lack of alignment in incentives — sounds like the “core chronic conflict” of the data world. just like with “dev” and “ops”.

Gene Kim, ITREV, Program Chair10:05:32

North Star: “integrity of data and timeliness of decisions, enabled by”

jeff.gallimore (CTIO - Excella, he/him)10:05:10

Gartner: only 8% of our data science apps reach production. :thinking_face:

jeff.gallimore (CTIO - Excella, he/him)10:05:26

…and “we want to get it up to 75%”

1
Gene Kim, ITREV, Program Chair10:05:20

“get away from heavy ETL, brute force wrangling” (ouch)

BMK-SECTION6-TransformationArchitect10:05:53

Data space is changing so fast - but practices remain old; We need to bring this new thinking and share with our data friends

am10:05:25

We are watching a pre-recorded video, despite the LIVE caption.

🙂 1
Quinn Daley10:05:14

the organisers have been very open about the conference working this way - it allows us to talk to the speakers during their talk, which I appreciate

❤️ 3
Andrew Salt - ScholarPack10:05:48

Absolutely brilliant way of doing it 🙌

🎉 1
Randy Shoup10:05:24

Coming full circle with Statistical Process Control from the 1980s. Great to see that used on data processes and not just industrial processes :-).

👍 1
Musthafa Soukathali10:05:05

DataOps is combination - data pipeline (agile) + data management (DevOps) + data observability (Data Science) @genek - probably this answers the question why dataops than devops

👍 1
Gene Kim, ITREV, Program Chair10:05:33

“Please think about your role in creating better connections thru a lens of Data Operations”

Gene Kim, ITREV, Program Chair10:05:06

An incredible call to action: “create co-mentorships between software and data professionals” (!!)

❤️ 6
🎯 1
Alex Broderick-Forster, IT Revolution, Event Staff10:05:27

From DevOps to DataOps (Nationwide Building Society) Simone Steel, Chief Data and Analytics Officer & CIO for Enterprise Data Platforms, Nationwide Building Society 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708550174/

👏 2
Gene Kim, ITREV, Program Chair10:05:42

Thank you, @simone.steel!!! 👏

🙏 1
BMK-SECTION6-TransformationArchitect10:05:45

Excellent session @simone.steel - appreciated

🙏 1
Ann Perry - IT Revolution10:05:47

🌟 And now, we welcome the team from Schlumberger, @smohan3, Global Practices and Competency Manager and @lester2, Global ALM Practice Champion here to present Season-Based Governance 🌟

👏 1
Nigel Lester10:05:54

Hi All – @smohan3 & I are delighted and honoured to be here and presenting at the DOES22.

Musthafa Soukathali10:05:54

A real story telling session. Loved it @simone.steel

❤️ 4
🙏 1
1
Gene Kim, ITREV, Program Chair10:05:04

Hello, @smohan3 and @lester2!!!

👋 1
😀 1
🙂 1
Billy Hudson - ScholarPack - DevOps Engineer10:05:09

Really got our Devops team thinking about our data team here @simone.steel, thanks!

👏 1
Sujaa Deepak10:05:00

thank you @genek , Happy to be here!

Simone Steel10:05:25

If I miss any of your questions, please reach out to me in the private channel. Thank you!

thankyou 1
Randy Shoup10:05:57

Obrigado, @simone.steel. Great talk.

👍 1
Gene Kim, ITREV, Program Chair10:05:41

2000 software engineers; $3.2B; oil exploration and production; help access energy for all

Gene Kim, ITREV, Program Chair10:05:10

“our software products; until 2017, quarterly releases; offerings include reservoir simulators; market leader for 30 years (!!!)” compute heavy; means customers need to buy large amount of servers” cloud offering allows customers to use without having to do lots of capacity planning/acqusition

👏 1
Gene Kim, ITREV, Program Chair10:05:45

Talking with @smohan3 and @lester2, it was wild learning about what it’s like selling to customers who operate in safety-critical cultures, resulting in a safety-critical mindset at Schlumberger.

Gene Kim, ITREV, Program Chair10:05:31

“detailed planning, far out in future”

Gene Kim, ITREV, Program Chair10:05:39

“taken as commitment by stakeholders”

Gene Kim, ITREV, Program Chair10:05:32

“only most critical artifacts now need signoff”

Gene Kim, ITREV, Program Chair10:05:06

“culture of creating (long) lists of features to develop” (this is so great! and familiar!) 😆

👏 1
Virginia Laurenzano NSA/MARFORCYBER10:05:35

never to be seen again (from the doc store)

😆 1
Musthafa Soukathali10:05:35

Hypothesis driven development - haven't seen org finance embracing this concept. Some details about how you educated and convinced them will be helpful

Nigel Lester10:05:03

Add HDD this into process to encourage its adoption and provided training to help with its use.

Nigel Lester10:05:38

Just reviewing this again. Got this wrong. We have not changed the way finance works. A very interesting topic though, on how to roll out new techniques.

Muhammad Tabrez - TCS10:05:37

Hypothesis Driven Development :thumbsup:

👏 2
Philip Day10:05:02

The annual release cycle was "working very well" according to the blurb Then we did cloud and had to change to faster release cycles I'm not clear here... was there a business trigger for the paradigm shift?

👀 1
Gene Kim, ITREV, Program Chair10:05:41

• visions • goals • bets

Gene Kim, ITREV, Program Chair10:05:02

“we don’t govern how the teams operation; goals are owned by the team”

❤️ 3
Giulio Vian, Unum10:05:10

"season is not a release cadence"

❤️ 1
Nigel Lester10:05:00

Yup, this is a key point.

Gene Kim, ITREV, Program Chair10:05:31

“plans are presented at beginning/end of each season” “anyone in community can attend”

jeff.gallimore (CTIO - Excella, he/him)10:05:35

@smohan3 how well did people take to the concept of “bets” and “hypothesis driven development”?

Sujaa Deepak10:05:47

its been a "continuous" work in progress - educating, coaching etc. :)

😉 1
BMK-SECTION6-TransformationArchitect10:05:59

@smohan3 - What roles are there in Goal Owners? I mean who are the people and what is their title in the org chart?

Luke Rettig - Target10:05:55

“releases are not tied to the end of a season, they are continous”

Nigel Lester10:05:22

Correct, the aim is teams release when ready and not wait till the end of the season

💯 1
Gene Kim, ITREV, Program Chair10:05:38

“by the way, we didn’t get any of this right the first time around” 😆

❤️ 4
BMK-SECTION6-TransformationArchitect10:05:28

In some enterprise even after few iterations the result is same as first LOL

1
Luke Rettig - Target10:05:35

very frequent theme in every talk and a great reminder for everyone. Just start, get it wrong, pivot, but most importantly START

❤️ 1
💯 2
Nick Eggleston, IT Architect (DMs welcome)10:05:49

Start and persist, iterate, which means somewhere there’s an understanding that failure-learning will be part of the process so the failures don’t empower the skeptics to torpedo the initiative.

Musthafa Soukathali10:05:52

This sounds very explorative model (meaning not a prescriptive one). While this appears effective for learning and doing, how does it scale. When you are not reasonably prescriptive at least around boundaries, things will go out of control!?

Sujaa Deepak10:05:38

actually we found higher the alignment higher the autonomy. It does require a clear vision to be set and understood by the entire product team.

Sujaa Deepak10:05:19

Yes there was a significant shift and business need to releasing more frequently.

❤️ 1
👍 1
Nigel Lester10:05:24

We came across the season concept from MSFT and is covered in multiple talks but here is one: Agile at Microsoft By Aaron Bjork https://youtu.be/-LvCJpnNljU?t=1417

Sujaa Deepak10:05:19

the goal owners are roles in BUsiness, dev and ops

thankyou 1
Gene Kim, ITREV, Program Chair10:05:30

“DORA workshops generated hundreds of improvement plans”

🙌 1
Gene Kim, ITREV, Program Chair10:05:48

Congrats on the award, @lester2 and @smohan3!

🎉 3
Musthafa Soukathali10:05:10

@smohan3 - how did you onboard the finance team on the HDD? WHat adjustments they needed to make to their financial planning process?

👀 3
Al Ramos (Morgan Stanley)10:05:20

Good to see multiple industries leveraging DORA metrics. With wide adoption it would be interesting to see if we can benchmark to see how we are doing?

Matt Cobby (Director of Engineering, Deloitte)10:05:33

You can do an industry benchmark. I have one open in front of me and you can select which industry to benchmark to. It is based on the State of DevOps survey data

Nigel Lester10:05:48

The Google DORA Assessments do give comparisons with others in the industry.

Gene Kim, ITREV, Program Chair10:05:09

“because decades of habits, much unlearning was required.”

Nigel Lester10:05:27

It is really important to read the definitions of the DORA metrics – the high level details are in the State of DevOps reports https://www.devops-research.com/research.html#reports Need to take particular care with Lead time for changes – this is from code commit to deployment in production. In ACCELERATE by Forsgren, Humble & Gene this is described as the delivery lead time (and does not include the design & validation part of lead time).

👏 1
1
Giulio Vian, Unum10:05:37

and they underline that was a practical need seems that a consistent measure the left chunk of a delivery process has too many variations and challenges

Gus Paul - Morgan Stanley10:05:16

And if you work somehwere that change lead time can be measured in days, you cna break it down into further stages 🙂

Fang Yuan10:05:21

LinearB measures DORA from accessing your Git repos so you can do this exact breakdown into various phases @gus.paul. From there we can identify the bottlenecks in your dev process and help your devs merge and deploy faster with a workflow optimization chatbot called WorkerB. Come to our booth if you want to learn more! #xpo-linearb-automate-dev-team-improvement

Giulio Vian, Unum14:05:03

IMHO we should keep delivery lead time as defined in Accelerate and use specific terms for any metrics that extends the measurement span

Fang Yuan14:05:31

Delivery lead time as in first commit to prod? @gvian

Fang Yuan14:05:06

Yup. That's what we measure in Cycle Time. Coding-Pickup-Review-Deploy. 👍

Malte Fiala10:05:57

@smohan3 @lester2 Thanks a lot for this excellent talk. As some of the aspects look like they are coming from Basecamp, I guess the betting process is based on basecamp as well? Did you use it as is or did you adapt it? Additionally: Do you use aspects of ITIL 4? If so, which ones are the most important for you to use?

👀 1
Sujaa Deepak10:05:44

no not based on basecamp but similar. Our devops practices are aligned with ITIL4 but ITIL4 is more adopted on the services side than the software development lifecycle...

👍 1
Gene Kim, ITREV, Program Chair10:05:02

“ending seasons at year end was (problematic)” (holidays, year end rituals, etc)

Gene Kim, ITREV, Program Chair10:05:45

“this has liberated us from onerous processes” (love it!!!)

Gene Kim, ITREV, Program Chair10:05:39

Thank you so much, @smohan3 and @lester2!!!! 👏

Ann Perry - IT Revolution10:05:02

And now....here to present Product Org Design: Flow Follows Form, it's @mik, Founder and CEO of Tasktop

Musthafa Soukathali10:05:45

Thanks @smohan3 and @lester2 for the great talk and active responses in the thread

👍 1
Gene Kim, ITREV, Program Chair10:05:55

“architecture (must be? should be?) isomorphic to the org”

❤️ 1
Peter Fassbinder10:05:23

@smohan3 @lester2 Great presentation, relates to my experiences quite well. I'd be interested to get your feedback on our Continuous Quality Assurance / Continuous Conformance approach that I presented yesterday (https://videos.itrevolution.com/watch/707351867/), and which is also featured in the latest issue of the DevOps Enterprise Journal (https://myresources.itrevolution.com/id006657137/DevOps-Enterprise-Journal-Spring-2022). Does this relate to your approach? If you have some time to have a look at it just drop me a line either in Slack or on LinkedIn.

Nigel Lester10:05:29

Your talk caught my eye, and I will be watching it with interest.

Peter Fassbinder11:05:36

I look forward to your feedback

Sujaa Deepak11:05:39

same, will come back to you!

Ivan Krnic (Director of Engineering at CROZ)10:05:54

We have mess above agile teams 🙌

❤️ 1
😂 1
jeff.gallimore (CTIO - Excella, he/him)10:05:55

“spotify has moved on from the spotify model” :rolling_on_the_floor_laughing:

😜 6
👀 1
😅 1
BMK-SECTION6-TransformationArchitect10:05:02

so they moved to next track 😉

Quinn Daley10:05:28

the “things break down at team-of-teams level” is so relatable

👍 1
BMK-SECTION6-TransformationArchitect10:05:36

Always love @mik presentation; Very compelling and great takeaways

1
👍 1
Gene Kim, ITREV, Program Chair10:05:42

(there’s a fascinating story of how McKinsey picking up on Spotify model in one of their whitepaper has caused epic impacts on the orgs who hired them)

👀 8
Radoslaw Wiankowski10:05:47

could you maybe share a link to the paper? or the story? :)

❤️ 1
Alex Broderick-Forster, IT Revolution, Event Staff10:05:03

Season-Based Governance (Schlumberger) Sujaa R. M. Deepak, Global Practices and Competency Manager, Schlumberger Nigel Lester, Global ALM Practice Champion, Schlumberger 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708550186/

👍 1
Gene Kim, ITREV, Program Chair10:05:24

decouple to accelerate flow. now what?

Gene Kim, ITREV, Program Chair10:05:52

(what could go wrong? 😆

Virginia Laurenzano NSA/MARFORCYBER10:05:59

May need to rewatch at a slower speed – @mik is talking so fast 😉

🐎 1
3
😂 3
1
👍 1
Gene Kim, ITREV, Program Chair10:05:24

might be my fault: I squeezed him into a 15m slot. 🙂

😂 1
Olav Maassen (Tasktop)10:05:29

The link will be posted after the session. Or feel free to reach out to us any time.

🙏 1
Gene Kim, ITREV, Program Chair10:05:01

but OTOH, I’m not sure he speaks any slower in conversation. 🙂

😁 1
😂 1
1
Virginia Laurenzano NSA/MARFORCYBER10:05:48

In his defense, I absolutely need more coffee AND always enjoy learning from him

👏 2
Alex Broderick-Forster, IT Revolution, Event Staff10:05:12

@vmshook video library can run in both slower and faster speeds 🙂

👏 1
🤌 1
Alex Broderick-Forster, IT Revolution, Event Staff10:05:43

Then you can choose extra deep voice Mik or high pitched Mik.

😂 3
Margueritte Kim (CEO, IT Revolution, she/her)10:05:42

Now you’re just showing off @alex

Mik Kersten (Project to Product, Tasktop)17:05:59

One day I will learn to slow down… I keep telling myself. All these DevOps Enterprise topics are just too exciting to me for the time being 😄

😁 2
Gene Kim, ITREV, Program Chair10:05:10

“the messy matrix”

🎯 3
Musthafa Soukathali10:05:14

architecture -> zaha hadid -> flow -> value streams.... Fantastic!

Scott Prugh (DOES Prog Committee)10:05:10

The Messy Matrix!! Great term! @mik

3
Gene Kim, ITREV, Program Chair10:05:43

configure org, then let it run

Gene Kim, ITREV, Program Chair10:05:08

Pattern Language; from Chris Alexander, the famous building architect

jeff.gallimore (CTIO - Excella, he/him)10:05:36

and another reference to Team Topologies

Gene Kim, ITREV, Program Chair10:05:22

by the way, there’s something called the Alexander effect: his observation that great architects made great use of his book “A Pattern Language.” Bad architects did worse when using his book. cc @rshoup

❤️ 4
🔖 1
Randy Shoup11:05:05

I feel like there is a Shu Ha Ri point in here.

Gene Kim, ITREV, Program Chair10:05:25

“action: reduce manual testers; outcomes: worse quality” (why? software wasn’t designed for automated testing!)

Randy Shoup10:05:04

But also, adding manual testers ==> worse quality, per Elisabeth Hendrickson 😉

✔️ 2
Gene Kim, ITREV, Program Chair10:05:14

hahahaha. thinking about things like this lead to madness. 🙂

Gene Kim, ITREV, Program Chair10:05:45

@mik talking about how seductive the concept of “single threaded owner” is — here’s expected benefits of merging product and engineering were:

Gene Kim, ITREV, Program Chair10:05:29

Actual outcomes… “uh oh…”

Musthafa Soukathali10:05:08

challenging the 'business and tech merged together and lived happily EA

👍 1
Gene Kim, ITREV, Program Chair10:05:40

“most profound moment; all hands meeting; most senior engineer asked, Mik, you’re such a fan of Team Topologies; so why do we keep overloading cognitive load and complexity domains? over and over again?” “I realized something was very wrong. signals were not reaching me.”

☝️ 3
1
BMK-SECTION6-TransformationArchitect10:05:44

TT Book & Cognitive load - a good challenging question by senior engineer to @mik - Love that

Gene Kim, ITREV, Program Chair10:05:47

“code jams were gone; engr community events gone; how did this happen on my watch? I’m an engineer!” “the configuration of the org suppressed engineering/platform signals, like cognitive overload; product signals were coming through fine, though.”

Gene Kim, ITREV, Program Chair10:05:16

“we should have fewer meetings, but no; important signals were no longer reaching me”

Gene Kim, ITREV, Program Chair10:05:48

“why did Spotify move on from Spotify model? no engineering leader owned enough of the org.”

3
Vlad Ukis13:05:15

I would like to delve into this to understand the point at which this happens

BMK-SECTION6-TransformationArchitect10:05:56

Curiosity - another question: How many of your organizations are adopting or exploring Team Topologies and seriously thinking about Cognitive load?

👀 3
2
Gene Kim, ITREV, Program Chair10:05:22

“results of changing org again? we’ve already hired 3 amazing engineers in last month”

Gene Kim, ITREV, Program Chair10:05:22

“faster moving consumer products need strong product signals; others may need strong engineering signals.”

Robert Ruzitschka - DevOps Guild Lead10:05:44

@lbmkrishna: We have not fully implemented the approach but the main ideas about Enabling teams and the core platform ideas have deeply shaped how we work.

❤️ 1
Ivan Krnic (Director of Engineering at CROZ)10:05:56

design via pattern languages, gauge via Flow Framework

Musthafa Soukathali10:05:59

@mik just gave a great business case for VPs/Heads of engineering.

👍 2
Alex Broderick-Forster, IT Revolution, Event Staff11:05:21

Product Org Design: Flow Follows Form Dr. Mik Kersten, Founder and CEO, Tasktop 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708550206/

Quinn Daley11:05:26

Thank you @mik - very interesting and relatable!

👏 1
BMK-SECTION6-TransformationArchitect11:05:28

@mik talk is real DevOps, so fast 🙂 LOL

Gene Kim, ITREV, Program Chair11:05:30

Fantastic podcast further exploring these topics, from @mik

Ian Mahoney11:05:43

@mik - Great presentation! We're just embarking on this path at New Look Retailers 🙂

❤️ 1
Ann Perry - IT Revolution11:05:51

Closing out the morning keynotes is @stephen, Vice President, Product Innovation at Sonatype, here to present, What does log4j teach us about the software supply chain?

👏 1
Christina Biangslev11:05:01

Thank God for recordings.... I need to watch that 10-15 times more. Amazing

💯 6
BMK-SECTION6-TransformationArchitect11:05:17

I often do that - 3-10 times in small micro learning space; All this DOES are highly packed with great content, quality, experience & ideas. You cannot afford to miss that

❤️ 2
Stephen Magill [Sonatype]11:05:07

Hello! So happy to be here!

👍 1
👋 3
🎉 1
Giulio Vian, Unum11:05:24

very interesting is there a way to realise team overload except for asking team members?

👀 1
BMK-SECTION6-TransformationArchitect11:05:52

Looking forward to it @stephen

Gene Kim, ITREV, Program Chair11:05:56

(although it does make one wonder about the life choices we’ve all made. 🙂

💣 1
🎯 2
Radoslaw Wiankowski11:05:11

Really nice presentation from @mik but I’d love to see the “full version” 😉 Felt like scratching the surface of a topic that deserved an hour at least 🙂

⬆️ 1
Gene Kim, ITREV, Program Chair11:05:47

“at end of 2021, we had this amazing natural experiment” 😆

Ann Marie Fred - Red Hat11:05:28

I love it - and I noticed how many times the Log4J/Log4Shell topic showed up in talks this year.

☝️ 3
BMK-SECTION6-TransformationArchitect11:05:21

so as in the previous years Equifax; It is better that we make sure we are not on the headline for wrong reasons 🙂

Stephen Magill [Sonatype]11:05:41

haha. better than the “natural experiment of 2020” 😕

💯 1
Gene Kim, ITREV, Program Chair11:05:45

@paul.d.fox is speaking next in one of the breakouts. It’s an amazing description of the log4j events at Morgan Stanley.

👍 1
👀 1
Gene Kim, ITREV, Program Chair11:05:16

“The Large Hadron Collider took a decade to build and cost around $4.75 billion”

Stephen Magill [Sonatype]11:05:38

we see this idea of exemplary cohorts in update behavior, release frequency, security response time. there tends to be clustering like this for various outcomes.

Virginia Laurenzano NSA/MARFORCYBER11:05:52

@stephen - did you (or others from Sonatype) get to participate in the EO NIST responses?

Stephen Magill [Sonatype]11:05:55

Yes, we submitted a couple of responses to that (on SCA and static analysis).

👍 1
jeff.gallimore (CTIO - Excella, he/him)11:05:19

log4j exemplars: updated in 1-2 days

💯 1
Stephen Magill [Sonatype]11:05:18

yes, quite good — especially considering it dropped on a Friday

💯 1
Stephen Magill [Sonatype]11:05:38

but then again the weekend is the time for open source contributions, right? 🙂

jeff.gallimore (CTIO - Excella, he/him)11:05:06

a culture of keeping dependencies up to date ➡️ better security posture. 🙌

Matt Cobby (Director of Engineering, Deloitte)11:05:20

Do it until it stops hurting!

💯 3
☝️ 1
Matt Cobby (Director of Engineering, Deloitte)11:05:57

Like going to the gym. Patch every week. It's going to hurt but eventually you'll be ripped and secure !

💪 2
Stephen Magill [Sonatype]11:05:14

in other words, we got to see how the sausage in made

Gene Kim, ITREV, Program Chair11:05:34

great times — 5 patches in less than a month.

Muhammad Tabrez - TCS11:05:50

Thanks @mik for the High Voltage Lighting talk

Stephen Magill [Sonatype]11:05:52

it is really interesting that there are still folks who get “stuck”, even among that early updating cohort. worth more investigation.

Quinn Daley11:05:02

in our case log4j was almost exclusively a dependency of our dependencies / tools etc, rather than something we could readily update ourselves. I’m guessing this will come up in your talk!

☝️ 2
Stephen Magill [Sonatype]11:05:11

good lead-in! “transitive dependencies” are coming up next 🙂

🙌 1
1
Stephen Magill [Sonatype]11:05:23

So given the trend I just described, I’m curious how long you had to wait for those transitive dependencies to be updated. Do you have a sense for that?

Marco Delgado [Ocado Technology, Engineering Practice Lead]11:05:34

@stephen is there a link to this graph with live data by any chance?

piotr.papros11:05:46

this is really behind the scenes data! Love it

Gene Kim, ITREV, Program Chair11:05:25

integrating dependency updates into their daily work

Gene Kim, ITREV, Program Chair11:05:18

“wall of red…marches inexorably to the right.”

Ann Marie Fred - Red Hat11:05:32

Good: updating dependencies manually every couple/few days. Better: automatically getting PRs for updates and simply approving them. Probably one of the big differences between exemplars and laggards.

1
💯 1
Stephen Magill [Sonatype]11:05:12

definitely! the more automation the better.

✔️ 1
Mitesh (DOES Event Staff / Engineer at Gaiwan)11:05:36

This is still manageable in the Java ecosystem (even better in Clojure). But transitive dependencies are hell in the node ecosystem. Sometimes a simple library ends up with over 1500 transitive deps 🚒 🧑‍🚒 🙈

Ann Marie Fred - Red Hat11:05:07

At least ‘npm audit fix’ exists and helps a lot.

💯 1
Ann Marie Fred - Red Hat11:05:41

Can you imagine if it didn’t? 💀

☠️ 1
Mitesh (DOES Event Staff / Engineer at Gaiwan)11:05:01

It definitely is helpful, but having so many dependencies change we also need to ensure things don't break right? I feel there isn't enough community focus on keeping libraries "backword compatible"

1
jeff.gallimore (CTIO - Excella, he/him)11:05:00

the community is getting faster at addressing issues. 🙌

Fokko V.11:05:15

Love this graph!

Scott Prugh (DOES Prog Committee)11:05:03

"Transitive Dependencies Matter" They also really stink... @stephen 🙂

Mitesh (DOES Event Staff / Engineer at Gaiwan)11:05:03

No wonder developers are scared and become laggards

😆 1
Gene Kim, ITREV, Program Chair11:05:33

PS: the fantastic presentation from @ben.dodd and Fliss Bennee talk about covid prevalance in sewage. Fave line: “even world class researchers can’t resist saying the word ‘poo’” :laugh:

Robert Ruzitschka - DevOps Guild Lead11:05:35

The time it takes you to update a critical vulnerability is likely another good compound metric showing engineering maturity. Watch out, DORA. 😀

Vlad Ukis13:05:47

It would be interesting to research what this would be predictive of!

Robert Ruzitschka - DevOps Guild Lead13:05:52

Thinking about it, I guess it is already covered in the existing metrics. 😀

Robert Ruzitschka - DevOps Guild Lead13:05:41

@vladyslav.ukis It would definitely show that you know about your inventory, can deploy quickly, have an effective process and situational awareness. If you can fix by just deploying a new version of the dependency, it most likely also shows that you have not been neglecting the topic and that you are fairly up to date, so the change to the system is small.

Robert Ruzitschka - DevOps Guild Lead13:05:53

I have seen systems that were so old that moving to a secure version of the library means first finding a developer who still knows the code, then upgrading your Java version then upgrading your build system, then upgrading your framework and only then you can upgrade the dependency. Now we are talking about months to fix a zero day vulnerability. Ahh, accumulating technical debt may have pretty ugly results!

Erik Greathouse11:05:44

Transitive Dependencies are a very large problem for very complex systems. The lag of update is expanding at each link in the supply chain.

✔️ 1
Stephen Magill [Sonatype]11:05:31

yes, the deeper the dependency the longer the update takes to “trickle down”

✔️ 1
Katharine Chajka (Tasktop)11:05:33

How do you prioritize what percentage gets addressed vs not, is it noted somehow

Stephen Magill [Sonatype]11:05:52

one good practice is to focus first on applications that are in production and open to the internet. work based on security risk.

Erik Greathouse11:05:09

It is critical to understand the impact of each dependency. Not just as a isolated patch but how it fits in the entire system or better system of systems.

Katharine Chajka (Tasktop)11:05:03

Thank you, sounds like it is maybe part of the prioritization process

💯 1
jeff.gallimore (CTIO - Excella, he/him)11:05:47

so we’re stalled on remediation at 2/3? 😩

😞 1
Philip Day11:05:52

Sorry dumb question (not a software professional) but why are the vulnerable versions still available for download?

Stephen Magill [Sonatype]11:05:54

pulling vulnerable dependencies would break all those builds that are using those versions. there’s been discussion of this, but it’s generally considered too disruptive.

Philip Day11:05:27

^follow up Q on that thread 🙂

Gene Kim, ITREV, Program Chair11:05:56

“75% of dependencies were never upgraded”

Quinn Daley11:05:32

Deps that never get updated is such a frustration! My recent example is https://uppload.js.org/ which was all the rage as an amazing toolkit for frontend file upload / modification etc. And then within a few months its only developer left that org and it just stopped getting updates. I guess if deps are popular enough (more popular than Uppload) someone else will usually take on the maintenance?

Stephen Magill [Sonatype]11:05:47

this is an important point! which dependencies you choose matters a lot. Choosing deps with an active community and large contributor base helps a lot.

Quinn Daley11:05:04

yeah - my lesson was learnt with that one!

Stephen Magill [Sonatype]11:05:38

also dependencies that upgrade their dependencies quickly, so you inherit transitive fixes quickly. (MTTU, which we discuss in the Software Supply Chain Report measures this)

1
Robert Ruzitschka - DevOps Guild Lead11:05:37

@philipday Builds will break, everything stops - Hell will break loose!

Philip Day11:05:43

Thanks! So orgs are in a bind: • Do a migration you might not have resources for • Persist a vulnerability How do they avoid getting into this trap in the first place? Assuming that vulnerabilities are when not if.

Robert Ruzitschka - DevOps Guild Lead11:05:34

It needs the understanding that keeping Technical Debt in check is a regular part of engineering work - not a side note. There is no way around it.

Matt Tennison11:05:51

Fwiw I've found tooling around automating dependency updates is pretty good, at least in the JS/TS eco-system (e.g. Renovate - https://www.whitesourcesoftware.com/free-developer-tools/renovate/). It spins up PRs with version bumps, and can even automatically merge those if you want, taking it out to prod (if you're practicing CD) automatically. If you keep on top of it it shouldn't be too much effort, excluding major version bumps. It's when it's neglected for a few years that it becomes really painful in my experience

Philip Day11:05:16

@robert.ruzitschka I guess not just understand Tech Debt, but your org has to fund it too So you whatever you build, you take on a financial liability that if your org doesn't commit to in perpetuity, you end up in this trap and then the data in your own care, your own product, and the supply chain downstream of it is all jeopardised What happens if your ownership changes to something less diligent? Iirc Cory Doctorow has written on this (he's written so much I can't find it), when private equity takeovers happen they are incentivised to slash costs such as IT spending, creating shareholder value (for themselves) in the short term but externalising costs such as security risk (and other problems) for many stakeholders further ahead

Philip Day11:05:56

Thanks @matt.tennison!

Robert Ruzitschka - DevOps Guild Lead11:05:38

@philipday you touch on many important points that are far beyond the realm of engineering.

Robert Ruzitschka - DevOps Guild Lead11:05:10

If an organization takes ownership and has a software product that is important for it's business success, it must be aware that continuously changing this product is a necessary fact. Even if you don't add any features, the world around your system will change - and thus make your system obsolete over time (it will become legacy - oh no!). So we must make sure to keep our systems in a changeable state. Not doing this will incur high costs and efforts later in time as change is unavoidable. Think about a system that uses an old log4j version. There is no way how you can keep it - it would be a business risk.

Robert Ruzitschka - DevOps Guild Lead11:05:45

@philipday If the sole purpose of the owner is to extract as much money as possible and externalize risk, I don't think we are talking about a sound and sustainable product strategy 😀. So there is a more fundamental issue to solve. I am not denying that this approach is common but the possibility to apply good practices requires good will and understanding on all sides. I have done a lot of talks to explain TechDebt in our company and it helps to explain context and business impact.

Robert Ruzitschka - DevOps Guild Lead11:05:32

Sorry for being so verbose 🥲.

Gene Kim, ITREV, Program Chair11:05:30

Amazing line in @paul.d.fox presentation : “why are you telling me about a log4j vulnerability? I’m a .NET app.” (why? how? find out in his talk!!! 🙂

jeff.gallimore (CTIO - Excella, he/him)11:05:08

1. know what you have and where it is 2. know where you get your stuff and have controls on it 3. have the capability to make changes with speed and stability

1
👍 1
Alex Broderick-Forster, IT Revolution, Event Staff11:05:12

What does log4j teach us about the software supply chain? Stephen Magill, Vice President, Product Innovation, Sonatype 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708550222/

📝 1
Quinn Daley11:05:14

Thank you @stephen this was great!

👏 1
🎉 1
Fokko V.11:05:18

A process for approving components, wouldn't that be a constraint to the DevOps teams?

Stephen Magill [Sonatype]11:05:14

Depends on how it’s implemented. You can make approval processes very quick with pre-vetted components and automated analysis of new components. But it absolutely can slow things down if it’s manual and under-resourced.

👍 1
Scott Prugh (DOES Prog Committee)11:05:10

Agreed with @stephen here. Teams choosing their own tools doesn't mean "all the tools". You should have a preferred/vetted list and create a decentralized process to proposes changes like ADRs in wikis or github. Beyond safety, you also get shared knowledge/common understanding and consisteincy.

👍 2
Fokko V.11:05:18

There is nothing in the Sonatype product line to support this, right? It would be something more upfront in the process?

Gene Kim, ITREV, Program Chair11:05:23

On that cheery note, morning plenary sessions come to an end!! 😆

🎉 1
Mitesh (DOES Event Staff / Engineer at Gaiwan)11:05:16

Thank you @stephen this was a great talk! 🙌

Ann Marie Fred - Red Hat11:05:52

So much lovely data…

💯 1
Gene Kim, ITREV, Program Chair11:05:47

PS: if you appreciate breaks being preserved in the schedule, please thank @annp and @jeff.gallimore — if it weren’t for them, morning programming would have overflowed the entire morning break. 🙂

❤️ 12
🐱 1
Nick Eggleston, IT Architect (DMs welcome)11:05:36

…and has the advantage of previewing all the talks before the event, right? 😉

💯 1
Gene Kim, ITREV, Program Chair11:05:56

It’s always tough to pick plenary content among such good choices — the default is to just jam another talk in, to the significant dismay of the team. 🙂

😁 1
Nick Eggleston, IT Architect (DMs welcome)13:05:24

Noticed quite a few folks saying they need to watch the videos 3+ times to try and focus and extract value from them. I have a hypothesis that the virtual model allows for deliver of the content and speaker/community interaction in a more "exploded" way, where the video is broken in to small "bites" as each slide or point is made and discussion swarms or circles around the before moving on to the next "bite"

Nick Eggleston, IT Architect (DMs welcome)13:05:30

Another related idea is doing the conference as a capstone for content that is released earlier and more continuously, allowing for a subscription model as presentations happen more-or-less continously over time and then a big get together to blast through the "best of" and a few new items in person or a virtual multi-day to celebrate and summarize. This also lets Gene do his very much appreciated value-add to the talks (kind of like he does with the Idealcast or the video-topic-lists). The model feels more agile/devops aligned with small batch released into "production", and is similar in a way to what AWS and Microsoft do with their big conferences (in that most of the features and products have already quietly been released or are in beta but there's a focus and spotlight on them for the conference).

Nick Eggleston, IT Architect (DMs welcome)13:05:00

What do you think @genek? Any value here or is this just crazy talk?

Alex Broderick-Forster, IT Revolution, Event Staff15:05:56

@nickeggleston love the idea of “bites”. As for the continuous release, the idea has struck me before as well. I think it’d be amazing to have weekly/bi-weekly/monthly “watch parties” where we showcase 1-2 new talks, but do it all year. Basically becomes a DevOps Enterprise Summit Membership Lecture Series. For AWS, I’ve noticed they do it over a longer period. Does it feel like attention fades, or does it help with engagement?

Stephen Magill [Sonatype]11:05:05

As Jeff just said: for all those interested in talking about supply chain further, join me for the Birds of a Feather session on it later today.

🪶 1
💯 5
Al Ramos (Morgan Stanley)12:05:57

Great points.. I know you don't talk about SLOs.. 🙂 How are you tracking SLO's and have you implemented error budgets?

👀 1
Vlad Ukis13:05:17

Interested as well

Slackbot16:05:08

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png

Gene Kim, ITREV, Program Chair16:05:15

Hello everyone! I’m super excited to watch @lucasc5 and @tod.bickley present!!

❤️ 1
Ann Perry - IT Revolution16:05:00

Let's now welcome @tod.bickley, AVP Information Risk Management for Identity and Access Management, and @lucasc5, Technology Audit Director from Nationwide Insurance. They will share, Adventures in Agile Auditing: Don’t Just Survive Your Audit… Thrive in it!

Gene Kim, ITREV, Program Chair16:05:34

AUDIT!!! “fear, dread, frustration”. 😆 Sorry, @lucasc5!!!

😂 4
Gene Kim, ITREV, Program Chair16:05:20

“What are minds when you hear that the auditors are coming?”

🕺 1
Erik Greathouse16:05:25

I love auditors - I will say that I am one of the few. But I like the second set of eyes to ensure things are working as intended. P.S. I am in the minority 😞

Erik Greathouse16:05:53

As an Assessor (RMR & CMMC) - I get a mixed bag some love the insights others treat me like Medusa

Gene Kim, ITREV, Program Chair16:05:38

To really go full circle, @tod.bickley, you may have to do a rotation in audit first! (Something that I’ve heard is quite common in organizations — I think it was Continental Airlines who first told me that high performing / high potential leaders were sent to do a rotation in internal audit!)

Tod Bickley16:05:25

@genek We actually do that! In fact Clarissa's leader is a former IT Executive

Gene Kim, ITREV, Program Chair16:05:38

That’s so interesting!! Who is that? cc @lucasc5

Clarissa Lucas, IT Audit Leader16:05:22

Pat Shanahan is my leader. He is an incredible leader and empowers his team to experiment with new ways of working 🙂

Gene Kim, ITREV, Program Chair16:05:55

The family of products under IAM.

Ethan Culp16:05:39

I did a rotation in Audit last year after coming from the IT org! Absolutely believe IT folks show go to Internal Audit and vice versa. Definitely something that should be in more orgs.

👏 1
🔥 1
Clarissa Lucas, IT Audit Leader16:05:55

We love having IT associates rotate through audit. We learn a lot from each other!

Vlad Ukis08:05:04

@culpe2 how long did you spend in Audit? Did you go back to IT afterwards?

Gene Kim, ITREV, Program Chair16:05:31

Kudos to you, @culpe2! Looking back, I wish I had done a year in internal audit somewhere in my career — so few other places you can actually see end-to-end processes.

Tod Bickley16:05:31

If anyone has more interest in the Product Model designs, I did a session last year that goes into detail on how we organized this. Check it out!

Ferrix Hovi - Head of DevOps - Siili16:05:42

Developers rotating to any place with repetitions... that's a lot of scripts and loops that get written 😄

Gene Kim, ITREV, Program Chair16:05:47

To auditors: “we can’t have you inflict your waterfall processes on us.” (I actually never thought that was something you could say…)

❤️ 3
Gene Kim, ITREV, Program Chair16:05:10

“became part of our IAM team; attended our standups, sprint reviews…“. 🤯

jeff.gallimore (CTIO - Excella, he/him)16:05:21

💡 “when the auditors show up, they’re really just generating demand for your team,” might explain a lot of the typical reactions to the auditors showing up.

Clarissa Lucas, IT Audit Leader16:05:37

Completely understandable!

Gene Kim, ITREV, Program Chair16:05:44

From @lucasc5 describing how the entire audit engagement model is (I thought inherently) a waterfall approach; scope, plan, fieldwork, report.

Hasan Yasar16:05:16

Typically auditor asks for verification/evidence. Ideally DevOps pipeline can(should) generate all the artifacts.

👏 4
Ferrix Hovi - Head of DevOps - Siili16:05:38

Actionable Insights... that's a real improvement to Insights

🎯 2
Gene Kim, ITREV, Program Chair16:05:11

“self organizing teams” — “we included someone very well versed in auditing standards.”

Nigel Lester16:05:15

The 2021 State of DevOps report had some feedback on documentation which was: We found that teams with quality documentation are: • 3.8 times more likely to implement security practices • 2.4 times more likely to meet or exceed their reliability targets • 3.5 times more likely to implement Site Reliability Engineering (SRE) practices • 2.5 times more likely to fully leverage the cloud

❤️ 5
Ethan Culp16:05:33

@hyasar - absolutely! most of these automated pieces in the CI/CD generate checks and automate evidence that an auditor would be looking for. Automated testing is absolutely your friend in an audit.

👍 2
Gene Kim, ITREV, Program Chair16:05:49

(When I was IIA member, the audit standard was AS2, then AS5. I wonder what it is now…)

👏 1
Gene Kim, ITREV, Program Chair16:05:01

“I didn’t have to go project manager to get updates — instead, I could just go to planner board.”

🎉 3
👏 1
💯 1
Gene Kim, ITREV, Program Chair16:05:23

“push method: project manager assigns control tests to staff. instead, we pulled work: staff self-assigned their work.”

Gene Kim, ITREV, Program Chair16:05:45

(better capacity balancing, and better match between assignments and person’s skills and goals!)

Topo Pal16:05:03

Daily stand-ups with audit sounds really scary..but not at Nationwide!

👏 1
🔥 2
🙌 2
👍 1
Ferrix Hovi - Head of DevOps - Siili16:05:25

@topo.pal If it scares you, do it more often 😄

👍 2
Ferrix Hovi - Head of DevOps - Siili16:05:37

So, maybe the problem is that we're not in a daily collaboration with audit.

Gene Kim, ITREV, Program Chair16:05:38

attended developer daily team standups! (often requests for information or followup for documentation; goes to point of contact, get routed, clarify requests… took days or weeks!)

Gene Kim, ITREV, Program Chair16:05:19

(leans back, takes long drag from cigarette, and says, “perfect.” 😆

😂 4
Tod Bickley16:05:43

The audit "work" was managed the same as other demand. Engineers pulled cards and worked audit requests, the same as the other requests

❤️ 2
Gene Kim, ITREV, Program Chair16:05:52

using standups, we got info during standup, or later that day.

Virginia Laurenzano NSA/MARFORCYBER16:05:07

you solved the 'bring me a rock' problem with daily standups. awesome!

❤️ 1
jeff.gallimore (CTIO - Excella, he/him)16:05:28

the journey toward psychological safety in that context is fascinating

❤️ 3
Gene Kim, ITREV, Program Chair16:05:45

The interactions that @tod.bickley is describing is like exact opposite of some stories I’ve heard: “the auditors of coming; make sure you don’t bring up X, Y, or Z.”

☝️ 1
🤐 1
Tod Bickley16:05:07

Psychological safety is critical. You need to be able to have tough, but "safe" conversations to move forward

❤️ 3
Ferrix Hovi - Head of DevOps - Siili16:05:09

This will naturally lead to a CA pipeline any day now.

Gene Kim, ITREV, Program Chair16:05:09

or “set ABC on fire, so they don’t notice XYZ”. 😆

Gene Kim, ITREV, Program Chair16:05:23

Okay, here it is: “what about independence?!?”

Erik Greathouse16:05:34

I have seen stand-ups and sprints save or kill an effort. What are good sources for practices or approaches to evolve great interaction?

Tod Bickley16:05:25

At the foundational level, you need to have buy in and trust. The process is the process.

Erik Greathouse17:05:22

@tod.bickley That makes sense, I guess my challenge is I am a matrixed employee so I see many varying levels of maturity.

Gene Kim, ITREV, Program Chair16:05:40

> To be independent, the auditor must be intellectually honest; to be recognized as independent, he must be free from any obligation to or interest in the client, its management, or its owners. https://pcaobus.org/oversight/standards/auditing-standards/details/AS1005#:~:text=To%20be%20independent%2C%20the%20auditor,its%20management%2C%20or%20its%20owners.

🔖 2
📝 1
👍 1
Gene Kim, ITREV, Program Chair16:05:07

@lucasc5: “the key is maintaining our decision rights”

🎯 2
Gene Kim, ITREV, Program Chair16:05:22

“it’s not @tod.bickley’s decision; those are our decisions” (Nice)

Ferrix Hovi - Head of DevOps - Siili16:05:45

@lucasc5 How about the Five Ideals of Auditing next?

❤️ 1
1
Gene Kim, ITREV, Program Chair16:05:02

I think PCAOB is just a US thing: Public Corporation Accounting Oversight Board — the folks who audit the auditors, created after Enron, Worldcom, etc.

Gene Kim, ITREV, Program Chair16:05:37

“Okay, why would one want to partner this way with auditors?”

Topo Pal16:05:06

💥 1
🙌 2
🔥 1
Jason Cox - Disney16:05:22

Loved the three ways infographic. Is there a high rez version?

❤️ 2
Leah Brown - IT Revolution16:05:01

Sure. Here you go!

❤️ 1
thankyou 1
Gene Kim, ITREV, Program Chair16:05:27

These results are nuts AMAZING. 🔥

🥜 1
👏 1
😂 2
Ethan Culp16:05:38

Love to see anything that makes audits shorter. Huge

👏 1
Ferrix Hovi - Head of DevOps - Siili16:05:09

Hmm... do these map somewhat with the DORA metrics?... hmm

Clarissa Lucas, IT Audit Leader16:05:28

@ferrix - That's next on my list: determining consistent metrics to measure success and progress as we expand our use of these newer ways of working.

Gene Kim, ITREV, Program Chair16:05:59

Wow. 2x more coverage, 1/2 time required to open issues; 3x issues with progress

Scott Prugh (DOES Prog Committee)16:05:21

Wow! Great results @lucasc5 @tod.bickley

❤️ 2
Ferrix Hovi - Head of DevOps - Siili16:05:39

Happy about spending less time being audited?... "Surprise".

❤️ 1
1
Gene Kim, ITREV, Program Chair16:05:42

Quotes from @tod.bickley team!!!

Jason Cox - Disney16:05:13

This type of feedback is gold. Hard to fold into an OKR. 🙂

❤️ 1
Gene Kim, ITREV, Program Chair16:05:15

“[Auditors] were very positive, worked hard to understand and communicate gaps; all members were enjoyable to work with.” 🔥

Erik Greathouse16:05:18

Love the Feedback on my fridge, I recommend a prase jar for the hard days.

Topo Pal16:05:27

Auditors make good friends

❤️ 2
Gene Kim, ITREV, Program Chair16:05:42

(We missed @tod.bickley and his team so much, we created some more findings, just so we can hang out again. 😆

😂 5
❤️ 1
Ferrix Hovi - Head of DevOps - Siili16:05:26

"Hey, we found a discrepancy, don't worry, we also brought cake."

🍰 1
Alexander Fritz16:05:31

I ask myself if there are enough auditors available to join all dailies of the teams? or are they join just selected DevOps Teams?

Ivan Krnic (Director of Engineering at CROZ)16:05:16

Auditors are like new DBAs 😅

😂 3
Clarissa Lucas, IT Audit Leader16:05:27

@alexander.fritz - The auditors join the daily stand-ups when we're doing specific work in that space.

👍 1
🙌 1
Gene Kim, ITREV, Program Chair16:05:03

(In all seriousness, this is one of the most startling presentations I’ve seen — I told them it was like another 2009 Allspaw/Hammond ‘10 deploys/per day by dev & ops cooperation’ presentation.)

❤️ 1
jeff.gallimore (CTIO - Excella, he/him)16:05:39

this is definitely a milestone.

❤️ 1
Scott Prugh (DOES Prog Committee)16:05:32

"10 audits/day??"

😆 4
👏 5
👍 1
😄 1
Virginia Laurenzano NSA/MARFORCYBER16:05:55

continuous authorization/auditing - a dream we have in DoD

❤️ 1
👍 1
Fokko V.16:05:13

Very inspirational story @lucasc5 and @tod.bickley, thanks a lot!

❤️ 1
👏 1
Ferrix Hovi - Head of DevOps - Siili16:05:17

@scott.prugh Maybe there should be something for the auditors in the pipeline then...

👍 1
piotr.papros16:05:37

so auditors also want to help us 😄 sure they do... but in many cases is feels soo much different ... let's pass more of those messages, great story to hear!

🙌 1
Gene Kim, ITREV, Program Chair16:05:42

THANK YOU, @lucasc5 @tod.bickley!!! Wild, right?

🎉 6
❤️ 2
👏 1
Topo Pal16:05:47

"Person who is doing the audit should not be able to close the issue" - just kidding

😆 3
Ann Perry - IT Revolution16:05:48

And now, a special presentation from the co-authors of Team Topologies, @me1208 and @matthew

🍿 4
Tod Bickley16:05:59

@vmshook we have been working on continuous auditing.. maybe something for Vegas!

🎉 1
Manuel Pais, speaker, co-author Team Topologies16:05:28

Hi everyone! I hope you enjoy the presentation 🍿

🙌 4
Gene Kim, ITREV, Program Chair16:05:37

Hello, @matthew and @me1208!!!

🙌 4
Leah Brown - IT Revolution16:05:56

Hooray! You can also learn more in @me1208 and @matthew new workbook: https://itrevolution.com/remote-team-interactions-workbook/

❤️ 1
Philip Day16:05:30

Guys you've got to group your slack channels 🙂 (aimed at many of my colleagues)

😄 1
Gene Kim, ITREV, Program Chair16:05:33

Just @-here to ask for help!

jeff.gallimore (CTIO - Excella, he/him)16:05:43

@matthew are you sure it’s not in the kubernetes channel???

Scott Prugh (DOES Prog Committee)16:05:53

Its always K8s or Microservices

💯 1
Erik Greathouse16:05:47

LOL, does @matthew have camera at my office 😉

😆 1
1
🎯 2
Gene Kim, ITREV, Program Chair16:05:11

\@here: I need help with database provisioning! (Notify all 3147 people? YES!)

😂 4
☝️ 1
🔥 4
Manuel Pais, speaker, co-author Team Topologies16:05:26

Just do it! Worst case scenario none of the 3147 people can help, right?

😆 2
bala gopal16:05:57

Looks like a job for "Brent" :-p

💯 2
piotr.papros16:05:35

every organizaiton has a Brent

🤫 1
Matthew Skelton (co-author of Team Topologies)16:05:03

Sadly, some organizations have like 10 or 50 Brents 😕

😂 1
💯 1
Gene Kim, ITREV, Program Chair16:05:54

“check Slack one more time… who is this rando, and why are they bugging me?” 😆

😁 1
Gene Kim, ITREV, Program Chair16:05:00

“I’ll just send them the docs” 😆

jeff.gallimore (CTIO - Excella, he/him)16:05:21

RTFM. the “F” is silent, @topo.pal

😄 1
😆 1
Ferrix Hovi - Head of DevOps - Siili16:05:55

@ channel Can you help me with creating a database? No?

😅 1
Virginia Laurenzano NSA/MARFORCYBER16:05:49

buying the book is good enough, right?

😆 5
😜 2
Matthew Skelton (co-author of Team Topologies)16:05:43

We actually insist that you record your own role play video 🎭 😉

Virginia Laurenzano NSA/MARFORCYBER16:05:33

lolz. I have on my list to RE-read your book. Recording videos is generally beyond possible where I work, but I definitely make my Marines role play

😄 1
👍 1
Ferrix Hovi - Head of DevOps - Siili16:05:55

I wonder if this guy really bought all those books...

3
jeff.gallimore (CTIO - Excella, he/him)16:05:29

it’s just the demo :man-shrugging:

Matthew Skelton (co-author of Team Topologies)16:05:31

Buying and reading are two separate things 📚

Ferrix Hovi - Head of DevOps - Siili16:05:01

Yes and buying means you have them locked away on your Kindle.

Jim Moverley16:05:24

this is a great presentation 😄 😄

🙌 2
1
Erik Greathouse16:05:05

I like GitLab documenting the change in the handbook and notifying the greater team when there is a major change. Not just change in Slack naming.

❤️ 1
💡 1
Jim Moverley16:05:31

gitlab work flow is amazing

💯 1
Gene Kim, ITREV, Program Chair16:05:06

“need 10x throughput and scaling”. “I need it in 3 days”

piotr.papros16:05:18

hilarious 🙂 and useful, and i am on my way to buy the book 🙂

🙌 1
🙏 1
1
Gene Kim, ITREV, Program Chair16:05:51

“I don’t need docs; I need 10x more TPS”

bala gopal17:05:56

Now you sound like "Sarah".. Jus saying

👏 1
Matthew Skelton (co-author of Team Topologies)17:05:11

Have you worked on a platform team before, Virginia? You seem familiar with the approach 😄

Manuel Pais, speaker, co-author Team Topologies17:05:31

and again... and again... and again...

Virginia Laurenzano NSA/MARFORCYBER17:05:50

Yup. May have had to help some admins learn new behaviors ;)

😄 1
Gene Kim, ITREV, Program Chair17:05:39

“I’ll just send the docs again.”

Ferrix Hovi - Head of DevOps - Siili17:05:04

Now look at these guys... this collaboration is not working at all... and I bought their book about teams... 😄

😆 6
🎉 2
1
Erik Greathouse17:05:39

If at first, you don't succeed say it louder... or again (old sailor saying)

Gene Kim, ITREV, Program Chair17:05:10

“please rate me 5 stars”

Margueritte Kim (CEO, IT Revolution, she/her)17:05:27

@genek Says this every time he drops me off somewhere.

1
😆 1
Ferrix Hovi - Head of DevOps - Siili17:05:30

"Would you recommend this database team to your enemies?"

😆 3
☑️ 2
Gene Kim, ITREV, Program Chair17:05:14

“I can’t believe we dropped all the production tables two years ago — and now we’re both working here! Wild, right?”

😀 1
1
Gene Kim, ITREV, Program Chair17:05:27

“Ah” “Ah” “Umm”

😆 1
Gene Kim, ITREV, Program Chair17:05:35

“We go live next Monday.”

Vaidik Kapoor (Speaker) - Technology Consultant17:05:46

I have to say that this format is super cool. This is like one of those tech talks where you do a live coding session, just that you are showing how code ends up being bad :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing:

😀 3
☝️ 2
1
🙌 2
Alex Broderick-Forster, IT Revolution, Event Staff17:05:41

love the pauses with explainer voice, too.

👍 1
piotr.papros17:05:26

and talk always helps ... no matter how disciplined we are with shared understnading documents

💯 1
1
Ferrix Hovi - Head of DevOps - Siili17:05:42

Is there a curl version of this API?

😆 3
Lloyd P17:05:05

Slack workflows and modals could be super helpful for some of those situations 👀

☝️ 1
👍 2
Lloyd P17:05:40

We even have a workflow to keep on top of watering the office plants 😅

1
🌱 1
❤️ 1
Manuel Pais, speaker, co-author Team Topologies17:05:51

In the workbook there are some interesting Slack automation examples from a UK company called Autotrader, e.g. creating/updating a channel based on metadata in a service repo.

Erik Greathouse17:05:21

I would caution Slack is a great tool but security should be included to protect before implementing a workflow or app.

🔒 1
Gene Kim, ITREV, Program Chair17:05:31

@me1208 I’m dying to know — does @matthew get his 10x scaling needs met?

1
😂 2
Manuel Pais, speaker, co-author Team Topologies17:05:17

Nah, this was just for the camera. Bob ended up doing all the scaling and db support work for the walking tours team, weekends and all. Bob has left the company.

2
Gene Kim, ITREV, Program Chair17:05:29

Or… (They study the problem, bring in consultants, and after a week, they upgrade the VM from f1-micro to n2-standard-2, everyone happy)

💥 1
Malte Fiala19:05:17

Are you sure they did not just produce a powerpoint explaining the basics of amazon machine sizes after one week?

1
Ferrix Hovi - Head of DevOps - Siili17:05:08

I am on the fence with booking meeting through two Miro spaces. Would raise the threshold from the Outlook abusive behaviours of stealing time without asking... but the UX is actually more pleasant.

Gene Kim, ITREV, Program Chair17:05:31

Thank you, @me1208 and @matthew!!!

👏 7
🙌 3
1
Manuel Pais, speaker, co-author Team Topologies17:05:59

Thank you for the feedback and helping make this presentation more engaging!

💯 1
Erik Greathouse17:05:30

Thank you so much... I have been reading Team Topologies and this was a great way to complement.

🙏 4
1
Gene Kim, ITREV, Program Chair17:05:23

Thank everyone for awesome Day 2! Day 3 is fantastic, too!!!

👏 7
Slackbot17:05:30

Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png

Alex Broderick-Forster, IT Revolution, Event Staff17:05:46

Adventures in Agile Auditing: Don’t Just Survive Your Audit… Thrive in it! (Nationwide Insurance) Tod Bickley, AVP Information Risk Management for Identity and Access Management, Nationwide Insurance Clarissa Lucas, Technology Audit Director, Nationwide Insurance 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708649004/ How to Help Teams Interact in a Remote-First World - A Team API Role Play Example Manuel Pais, IT Organizational Consultant, Co-author of Team Topologies Matthew Skelton, Founder at Conflux, Co-author of Team Topologies 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708649039/

Malte Fiala19:05:03

Regarding Auditing: Anyone around here having knowledge regarding Cloud ITAR compliance? If so, please let me know, I'd love to chat with you!