This is just an appreciation post for all organizers and speakers in US and Asia time zones... these 3 days must have been crazy for you...thanks 🍺 ...or should I say ☕ 😅

Reminder: Remember all those talks you attended the first two days of the Summit? Please submit your feedback for those! It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png

Reminder: We want to hear your stories from the Summit. What did you learn? Whom did you meet? What ideas are you taking back with you? What actions are you planning to take? Post in #summit-stories! https://devopsenterprise.slack.com/files/UATE4LJ94/F01CD9VE6JE/stories.png

Looking forward to our fantastic Day 3 plenary morning talks! First up will be @christina_yakomin and @rdaitzman from Vanguard! 🎉 🎉

Head here to grab the journal and some free ebooks! https://members.itrevolution.com/free-ebooks

✨ Opening up Day 3 is @christina_yakomin, SRE and @rdaitzman, Senior Manager - Intermediary Technology Platform, of Vanguard, here to present: Iterative Enterprise SRE Transformation ✨

Vanguard Financial: created the index fund in 1975: in 2021, they had $8 trillion Assets Under Management (!!)

PS: I loved the NPR Planet Money episode on index funds and Vanguard Financial founder John Bogle here: https://www.npr.org/sections/money/2019/01/23/688018907/episode-688-brilliant-vs-boring

Vanguard: no physical retail presence — all contact is thru phone or web.

@rdaitzman now is certified on wine serving and tasting! Wow! 🎉

@christina_yakomin @rdaitzman From that NPR Planet Money interview, it sounded like technology was key from the very beginning to drive down management overhead costs, ideally down the zero — is that assumption correct?

(I think I heard somewhere that some of those initial mainframe applications are still the system of record for customer accounts, which use VSAM tables?)

i can imagine what things looked like (or didn’t, as the case may be) with that lack of visibility. it’s like flying blind.

The initial route to cloud was to “lift and shift” the PaaS to cloud, which would bring all the applications that run on it with it!

@christina_yakomin what were the problems that the “lift and shift” approach caused?

Failing to account for the work required to keep Kube clusters healthy - I’ve seen that mistake made multiple times!

(route to the cloud; some embraced serverless Lambdas, some ran in Kubernetes)

TIL https://asq.org/quality-resources/fmea

@christina_yakomin you mention AWS ECS (Fargate). Interesting you decided to go that route rather than K8S. We also run on Fargate and it works really well with low overhead. Interested to hear why you chose Fargate?

As part of Chaos Engineering effort, @christina_yakomin crashed build systems , to see the mayhem that would ensue?!

So many cloud migrations I see are just that. Lift and shift driven by a data centre contract ending - the benefits of the cloud transformation are much greater than that

Fire Drills - I've been discussing the idea that we do office building evacuations on a regular basis and test alarms every week, so why don't we test our operational outage procedures every week?

(these fire drills are a great way to train engineers who are just starting to be trained to go on-call. 😆 )

OMG. The ultimate heisenbug problem — how do you recreate the build system failures only caused by high build loads! (Not in the code — in the build systems that build/test the code, that cause the build system to fail!!!)

load testing (kinda) the build system… cool

Break testing CICD pipeline! Great!

@christina_yakomin It would be great to see designs for the changing architecture from monolithic to microservice and moving to cloud/containers

Cloning dashboards - interesting story.

“teams no longer had to submit tickets to get metrics.”

How did you balance between really required logs and not so critical logs?

(When Google Cloud deprecated v2 of their monitoring/graphing service, I went from too many dashboards to none. 😆 😭 )

(I may have missed a couple of emails warning about deprecation…)

“Logging costs increased exponentially.” “We could no longer see important error messages.” 😆

I thought CloudWatch was a log aggregator? Do you have a different term for it?

up = good, down = bad: good for executives, bad for engineers. :rolling_on_the_floor_laughing:

Product owners to buy-in into SRE concept..

Does your build pipeline slow down? Do you have so much logging you struggle to find what you are looking for? Do the amount of alerts you get overwhelm you? - well - the good news is, you are doing something right. the bad - you may need to iterate your SRE. Here’s a talk from @christina_yakomin that can help you

@christina_yakomin Monitoring is traditionally a role that falls on support/Ops, how did you get Developers excited to put together, build and use their own dashboards/metric data? was it putting them oncall? 😆

How big is the whole SRE team?

@christina_yakomin Yes! I love OpenTelemetry! Having great distributed tracing really improves understanding and 2 of the High Performance Capabilities!

How do the SRE leads interact or collaborate with DevOps teams?

I love this story from @rdaitzman — newer Vanguard Advisory business unit, which help external financial advisors who help with retirement planning, etc. As investment industry all race to lower costs, this is an important initiative for Vanguard — “it’s our first SaaS product.” I LOVE IT!

“first time we’ve ever built an application like this.” So cool, @rdaitzman

What differentiates the SREs from the traditional BAU teams?

Talking to business partners about potential failures, wow.

“Failure modes and effects analysis: we engage both engineers and business partners; deliberately understand what business response should be: comms and notification, processes that would be activated.” <--- so amazing.

Totally, @ann.marie.99 — so great!!!

Great learning from this session! Loving it.

@christina_yakomin could you kindly explain what the SRE champions do vs. the SRE Leads? Do SRE leads have accountability for SRE techniques and practices for their application and the SRE champions have a more holistic view across different project teams/product lines?

“uh, yeah, we’d have to send an apology letter to everyone, apologizing for incorrect account balances and values — and it may be a week before correct account balances can be shown. can we delay the deployment until we’ve tested this?” (I was telling @rdaitzman that Phoenix Project would have been written totally differently had I heard this talk before writing it!!) I just love this presentation from Robbie and @christina_yakomin!

Also @christina_yakomin what's your view on having DevOps Coaches and SRE Coaches? Would you amalgamate?

From buzzword to mindset - real transformation.

“every hour taken from engineer to do things like SRE training is an hour they’re not writing a feature — how do we balance?” “boring week-long training?” (I doubt it’s actually boring!!)

This is always a challenge - getting people to spend time on training when they could be doing more on delivering.

budgeting pops up again as something that needs to change to support continued transformation. 💰

@christina_yakomin Is there an easy answer for the question: “given these success stories, to what extent have you gotten all the leadership support you’ve been seeking?”

Learning how to set up your first metrics and alerts is HARD. 😅

“in pockets where implemented, blameless PIR leads to amazing breakthroughs”

Are Vanguard engineers co-located or a distributed team?

Nicely done @christina_yakomin and @rdaitzman!!

THANK YOU SO MUCH, @christina_yakomin and @rdaitzman!!

⭐ Now introducing @andrew.letherby, Service Owner of the Customer Insight Platform, HMRC and @caitlin.smith, Delivery Lead, Equal Experts, presenting: Dynamic Decision Making at HMRC - Moving to Data as a Product During a Pandemic. Here to answer any questions with Caitlin today is @mercedes.gozalbomorag, CIP Service Designer. ⭐

Enjoyed your presentation very much! Helps me a lot!

Amazingly insightful and completely relevant to what we're going through in Vodafone as well. Thank you @christina_yakomin d @rdaitzman

Iterative Enterprise SRE Transformation (Vanguard) Christina Yakomin, SRE, Vanguard Robbie Daitzman, Senior Manager - Intermediary Technology Platform 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/708991000/

Amazing presentation, learning, insights. Thanks so much @christina_yakomin , @rdaitzman

@christina_yakomin Nice to hear covering most of the e2e journey for transformation. Good luck with next steps 👏

Hello, @andrew.letherby and @caitlin.smith014!!! Continuing the amazing story told last year from HMRC!

…this is the story hinted at last year… so great that we’ll learn more about how UK averted economic ruin thru the heroic work done at HMRC!

“HMRC: you may know them from income tax, VAT.. but much more… import/export, customs declaration, manage govt banking service that support govt orgs (payments and repayments across govt)” (!!)

@andrew.letherby mentioned to me that of the 20 government services deemed “critical,” 17 are run by HRMC. Became very relevant as pandemic started shutting society down, as these services were depended upon by some of the most vulnerable in population. Thank you for your amazing work, @andrew.letherby and team!

as the flow of work coming in goes up, and capacity stays the same, i wonder what would happen… :thinking_face:

(What didn’t make it into @andrew.letherby introduction: he spent over 20 years at HMRC, including investigations of banana smugglers. 🙂

“pandemic led to mandate that said, if you can work from home, you should; entire sectors of economies unable to trade.”

“HMRC: get financial help as fast as possible to those in need.” (Hundreds of billions GBP put into economy — but it was an obvious target for organized crime (!!!))

“we thought about what HMRC analysts needed; we’ll have an unprecedented # of cases to analyze, to detect whether it was a fraudulent claim” (all citizens were entitled to monies; however, criminals creating fictitious organizations claiming monies were not.)

“to prepare for uncertainty, we designed for flexibility”. 💯

@caitlin.smith014 was tasked with more than doubling the size of this team supporting these efforts on VERY short notice…. (less than 4 weeks for the initial efforts, I think?)

the astonishing opportunity: “could we automate the detection/prevention to filter fraudulent/erroneous transactions, or even prevent it”

“we focused on well understood problems: specific attributes already used by business analysts; addresses, bank accounts, where we had good sources of information, could be verified; good sources for intelligence within and outside of HRMC, how specific accounts/addresses were/are being used.” “by sticking to these familiar concepts: we check a thing, we decide good/bad, and decide what next step is; these familiar concepts enabled business to know how to use these capabilities.”

“could we assess claims while people were filling out form?” <----- WOW

“good claims would be sent through — potentially problematic ones would be reviewed. external data could be brought to bear, to increase fidelity.”

The first iteration of the insights service for the Statutory Sick Pay scheme took four weeks to build. The existing team integrated our existing data pipeline. We only grew the team when new capabilities were added.

“to scale efforts, we needed to onboard new team members; we reached out to people who already worked on digital platform at HMRC; so many wanted to come back, b/c of their great experiences at HRMC.” ^^ amazing — what a testament to HMRC being a great place to work?? (am I getting that right, @caitlin.smith014?)

“we discouraged overtime; only had to do it once or twice;”

“we brought in a burnout specialist” ❤️ ❤️ ❤️ ❤️ ❤️ ❤️ ❤️

“we brought out burnout specialist; some were at home alone, where colleagues were a lifeline; others had families, struggling with homeschooling. we knew we had to look after team.” ❤️

Proof of their ability to scale with pandemic

“3,960 claims per minutes — far higher than could have been analyzed manually”

“100% of claims were being assessed upfront — another first” <-- ensuring that public monies going to those in need (vs fraudulent claims!)

“150% more users; 2.3MM searches internally” <-- showing how widely these capabilities were accessed, as well as supporting internal workload needed to process claims.

https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1652349707995149?thread_ts=1652349509.705569&amp;cid=C015DQFEGMT “The culture within HMRC Digital is perhaps extremely different from what people might expect. It’s a great place to work full of brilliant and dedicated people trying to do the right thing for the public. “The whole organisation came together to deliver for citizens during the pandemic”

“data was not only available, but it was safe and secure.” 🎉

@caitlin.smith014 - how did you scale up that user-onboarding, did you change the platform to make it simpler to adopt, or improve the training materials/engagement?

“data is our product”

“looking back: without those teams, we wouldn’t have been able to build [these critical capabilities] at HMRC.” What an incredible contribution, @caitlin.smith014!

“Our role: deliver a critical service, but at the same time, because of its attractiveness to criminals: we could prevent/detect it happening, have information to pursue people who took money they weren’t entitled to.”

“the whole of HMRC had a single focus for this period of time; delivering of critical services; we had a unique line of communications from delivery teams to policy makers in central government, setting levels of risk appetite very clearly, so we could balance delivery of services vs management of risk. “this clear line of communications made decision making clear and direct; we are attempting to preserve this post-pandemic; policy goals directly and alongside delivery teams.” Amazing. Unity of mission/purpose, allowed massive decentralized execution.

Help they’re looking for: how does one fund innovation across a large service?

Thank you so much, @andrew.letherby and @caitlin.smith014 — was amazing to hear about this side of the story of the heroic work at HRMC!

Dynamic Decision Making at HMRC - Moving to Data as a Product During a Pandemic Andrew Letherby, Service Owner of the Customer Insight Platform, HMRC Caitlin Smith, Delivery Lead, Equal Experts 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/708991017/

✈️ Let's welcome back @pieter.jordaan, here to give an update on TUI Group's Transformation Journey ✈️

(I’m so fascinated that UK HMRC has obviously created a reputation of being an amazing place to work. ❤️ 🎉 I’m less sure this is the case for similar agencies in other countries…)

Thank you for giving us an update on the heroic work you talked about last year, @pieter.jordaan!!! First data point: he obviously hasn’t been fired. 🙂

(Which was on my mind when I reached out to him — was wondering whether my email to him would bounce. 🙂

How utterly visionary: use period when there was 95%+ drop in transactions to do massive modernization/rewrite of most core applications.

Second data point: focus was the more core processes at TUI Group: airplane inventory, routing, pricing, sales, seating, fulfillment, ticketing. “We completely rewrote these applications.” (!!) I still can’t get over this.

“We rebuilt these applications in record time.”

the opposite of other business who saw increase in work

“We are now on the second season running on these new platforms.” (I.e., they’re working, and supporting all relevant business operations.) 🤯

“we digitalized so much of the travel experience: flight cancellation, baggage updates, what bus to get on; these are what make/break travel experience.” (This is something that thousands of people used to do; hold signs to show people what bus to get on!)

“we weren’t flying; we weren’t sending people on airplanes; it’s easy to say ‘let’s rebuild it.’ “but now it was time to say ‘let’s fly!!’” <--- more scary to say! 😆

Is it only me... but this talk is making me want to book a flight (of course via TUI) to try the experience and go on holiday somewhere:rolling_on_the_floor_laughing:

“rebuild the aviation platform while flying” 👈 a different take on “the rebuild the airplane in flight” :rolling_on_the_floor_laughing:

A company having bad times at the markets but nonetheless spending money in renovating their IT! One needs to have this mindset (instead on 'we must reduce costs now') and the sufficient funds to do that! Hats off!

This is a great story of how "survival instincts" change normal operating behavior to deliver outcomes that can never be done with "BAU behavior". We also seem to see this post war-room or code-yellow working scenarios where teams get amazing things done and afterwards go back to "BAU" and fail to achieve these results. Question: How do we deliver these outcomes in "BAU times"??

^^^ (wait for the testimonial!!! it’s wild.)

“Business Migration follow IT Consolidation” - interesting to see Organizational structure evolution as companies continue to align business and tech strategy into one

“we didn’t have the luxury to start small; we had to go big.” (focus was moving largest market to cloud, to save the most $$$ fastest.)

“If you took a holiday in last year, you were flying on our new platform!” (!!!)

It’s funny: I really did ask him, “were you fired? of are you a hero?” His answer is amazing and profoud.

Free Solo is a terrifying movie and I love that quote!

Alex Honnold: “Free Solo movie” “If you fall, you die.”

“Mimsdai Purja: climbed 22K foot peaks (?) in 6 months; previous record; 14 years!!!) ^^^ “no one wanted to be associated with him, no one wanted to give him $$, because they didn’t want to be associated with failure.” “to put in context, most climbs would take 3 months to get ready.”

“then climbed another mountain 48 hours later (!!)”

Some of @pieter.jordaan’s comments early in this talk about UX reminded me of this recent travel IT/system UX horror show that appeared on my linkedin feed - BA need to get on this conference! (It happened to Dan North personally, who happens to be a major figure in the software world, and is therefore able to articulate it particularly well) https://dannorth.net/2022/04/20/a-bad-system/

“they didn’t set a new milestone; they broke barriers, and set a whole new mindset for future efforts”

This commentary on risk is so key. Many folks in business view the status quo as the least risky path and change as the riskiest path. This stance emboldens sunk cost fallacy.

“2.5 years ago, this was an idea on powerpoint slide; now it’s reality.”

I’m reminded of Shigeo Shingo who changed the way of thinking about die cast setup, reducing them from over a day to single minute tasks (Single Minute Exchange of Die - SMED). It was a game changer in manufacturing

(I know a tech executive at a travel booking company who looks at this story with genuine envy.)

“Everything is risky, but doing nothing is the riskiest approach”

“embolden successes” 🔥

“how can we make it less wrong?” … such a critical mindset

“we try to make the one-way decisions early; avoid them when we can. why? because we know our design is wrong.”

“how can we make it less wrong” is a much better mindset than “adopting best practices”

“what is the magic? was it cloud? devops? the real magic is in small, motivated and focused teams”

Yes!! Doing nothing is often the riskiest approach!! Also, doing something that isn't working(or working well) is also a risky approach. Even late in the game, be willing to change your approach if there are better options to success!

Love that 'less wrong' point - we often use similar "what's least worse choice we can make - let's try that"

this is mostly a business decision to invest in improving when tide is low to be ready when the tide is high and sail

“there is so much passion in products now; collaboration in remote-first; all had to be figured out post-pandemic; all very difficult dynamics.” “was made clear when Ukarainian engineers, who we tried to help as much as possible; take care of your personal situation; we planned for them to take time off; but despite all this, even though they are in cities being bombed, they keep showing up to work; there’s no pressure to work, but it’s driven by passion and comraderie”

the great [story] is that business realized that every company is a sw company (cit.) and invested in IT

Your Design is Wrong. How to make it Less Wrong? This statement is such a mindset changing @pieter.jordaan

“when people find love for their product, and a place to hook onto exciting vision, that’s the treasure. Money isn’t the scarce commodity; small motivated teams are.”

@moira.cheng here’s the testimonial… a TUI Group customer, who also happens to be an engineer at Google!

“I would show this to the team, who worked so hard during pandemic… the fruit of our labors has put us into the position post-COVID that is completely different pre-COVID.”

another element is the joy of engineers allowed to redo, improve and change

“when he started, everyone thought he was crazy; he did in 14 months what everyone else did in 14 years; he had no money, had to re-mortgage his home to fund his climbs.”

There is already an assumption that the design is wrong which means, it's quite possible that some issue may crop up in future in the design, but that should not stop us from proceeding

“We’re hiring!”

(I am so grateful for @pieter.jordaan for putting that presentation together — so much was based on my question, “did you get fired, or are you a hero?” His response was so startling, thoughtful, and even profound.) THANK YOU, @pieter.jordaan!!

Really inspiring, thanks @pieter.jordaan!

⚡ Now introducing DevOps Enterprise Summit Programming Committee member, @ben.grinnell, Managing Director and Global Lead of the Technology and Digital at North Highland, presenting Sourcing in the Age of Digital Business ⚡

TUI Group: Transformation Journey Pieter Jordaan, Group Chief Technology Officer - Technology Transformation, TUI Group 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/708991029/

Up next: @ben.grinnell — he’s been to all 15 DevOps Enterprise Summits, has been on the Programming Committee since 2015, but has never spoken here!

Really inspiring - thanks for scheduling me after that one Gene

An uncomfortable moment: @ben.grinnell pointed out a bias we’ve had against outsourcers — deliberate, as we typically favored enterprises who brought in engineering talent. But as a result, we’ve been overlooking something very important — the result is this presentation, and the start of an important and new effort.

“enterprising using outsourcers, and outsourcers are under-represented here.”

“In my government service, there was nothing that supported trust, collaboration; they were set against each other; impossible to implement great practices discussed here in that context.”

“every enterprise I speak to are trying to grow their capabilities. True?” <--- put emoji here.

Every enterprise I speak to is trying to grow their internal IT capability. If you are working for a large enterprise, let us know if that's the case in your firm? 👍 👎

'like it or not, they're in the IT business'

“being fastest is now more important than being first”. 🔥

Profound question: “do I have to insource to accelerate my DevOps goal” There’s a knee-jerk reaction, given how common “70% outsourced -> 30% outsource.” Answer is more complicated!

There was a great session yesterday from TCS, about bringing DevOps to their customer portfolio — let me find link.

💯 to the lack of alignment across functions

@ben.grinnell does an amazing job framing the challenge for outsourcer and their clients.

The right hand side of this slide should read: 1.Help growing their internal capability and reducing the dependency 2.Increased automation to reduce delivery/maintenance effort and time 3.Collaboration with competitors in multi-supplier engagements to help everyone bring their ‘A’ game

Trying to help outsourced IT be more effective is feeling like the next iteration in the DOES journey: First not unicorns, but horses. Now not horses as we think they should be, but horses as they are.

Most enterprises with aspirations to grow their internal IT capability are falling short of their goals, why? – Slack Poll - please vote with the emoji reaction below for the 1-2 most impacting your enterprise or reply with others: 1. It's an aspiration without a plan or concerted effort:eyes: 2. A lack of leadership alignment about what that means or the priorities:open_hands: 3. A lack of constructive collaboration and shared ownership with HR:sweat: 4. There is a lot more to growing a sustainable internal capability that the current scope:open_mouth: 5. Struggling to put a compelling proposition to the market:mega:

When @ben.grinnell said, “I hear clients say to their outsourcers, ‘I don’t want your people learning on the job.’ That is the opposite of what they should be saying.” One of the most profound things I’ve ever heard!!

“Your mess for less” era is dead.

The 2018 State of DevOps report warns against functional outsourcing: Analysis shows that low-performing teams are 3.9 times more likely to use functional outsourcing (overall) than elite performance teams…

Outsourcer goals: “grow margin, secure long term revenue” Client wants “I want someone who has done this 10 times before”

@ben.grinnell - what challenges do you see with budget/headcount from executives when insourcing?

Procurement wants multiple firms competing for work — difficult to create One Team.

Argh!! Procurement managing IT outsourcing is a HUGE anti-pattern. We aren't buying servers and trying to get the lowest price!

Agreed to some extent - "less for mess", should be stopped @ben.grinnell

MSP has to develop workforce; “I don’t want your people learning on the job” <-- “that phrase has to be put to rest; we need everyone growing their careers.” 🤯

how does “don’t ban learning for suppliers” gel with IR35 here in the UK? I assume there are tax implications of including suppliers in employee activities, annoyingly

Balance between funding products/projects vs. community is key — power follows money. (This came up in the Dr. Westrum work in “Sidewinder” book — this balance is actually big tension in functional organizations! @ben.grinnell has finger on a very important problem.)

The idea that you can treat people like cogs that can just be ‘dropped in’ fully functional, shows a complete misunderstanding of what we all do. Even if you know a technology forwards, backwards and inside out, you will still need to learn about the existing systems and ways of doing things in that organisation before you can be effective.

(and is at the root of what is called the Second Toyota Paradox — that Toyota didn’t have cross-functional teams, instead they had very powerful functional silos. Much like how Google SRE is organized.)

Can we deprecate the phrase "Centre of Excellence" (CoE) and use "Centre of Enablement" instead? It sets the right tone to switch from controlling to helping those who are responsible for a line of business to achieve their goals and business value?

@bernard.voos usually enterprise headcount constraints getting in the way of the capability build, and too much focus on the individual cost rather than value

Haha. “Many will point of SODR functional outsourcing finding.” (yes.)

This session is not in the video library, is it? I can't find it.

“Do you buy T&M time & materials or outcomes?”

(@hagedorn video will be posted as soon it finishes airing.)

“if you want people to commit their careers are, you need to be very clear about your goals, what you’re trying to build” <--- relevant to any leader

“don’t ask partners to do something that’s not already in their repretoire, engagement model, etc.” 🔥

Superb analysis and summary @ben.grinnell

“suppliers -> partners”. (interesting, right? 🔥 )

“Where has service providers enabled growth of capabilities, and how have they been rewarded?” (we see this more on app dev, less on app maintenance.) We’ll have a track dedicated to this, to focus attention on this. Join the Birds of Feather on this topic!!! THANK YOU, @ben.grinnell!!! Amazing work!!!

Sourcing in the Age of Digital Business Ben Grinnell, Managing Director and Global Lead of the Technology and Digital Service Line, North Highland 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/708991041/

Thank you @ben.grinnell, a great talk with lots of insights.

Thanks for this insight @ben.grinnell

Get your copy of to soon to be released book that @wbensing coauthored here! It's free while supplies last: https://myresources.itrevolution.com/purchase.aspx?appId=006657138&amp;shelfId=ce07b281-2e9b-455c-9e22-0d095c7a4570&amp;code=invuarc

Great talk @ben.grinnell!! I'd love to help with this problem!!

great talk! thanks!

Hello, @bill.bensing (or is it @bill.bensing) — agh, not sure about how Slack does name conflicts. 🙂

✨ Let's now welcome, @wbensing, Software Factory - Managing Architect at Red Hat and co-author of the upcoming Investments Unlimited, here to present, We're Sorry, Love DevOps ✨

Hello all!!!

"CI/CD was yesterday" - yes! @bill.bensing

“Imagine you’re the CEO, and you’re at home, making pizza with family.. and you get a call from board member.”

MRIA - Matter Requiring Immediate Attention - aka Its on Fire

MRIA: “matter requiring IMMEDIATE attention” how do you get an MRIA? by not effectively responding to a bunch of MRAs: “matter requiring attention”. (Audit findings. 🙂 😆

@tapabrata.pal #rockstar!

“This is one of my best experiences” — I hear this so much when one gets to work on projects with kindred spirits on important problems. So true!!! Which is why I love this community so much!

Jason’s and Topo’s playlists here! https://videos.itrevolution.com/playlists/index.html

“organizations” === “efforts that require efforts larger than can be done by one person alone”. (which is why we need to say it so often! 🙂

“this is more than someone saying ‘we think you should do this’ — it’s what successful organizations have done.” (and not gotten fired. 🙂

Codification of Governance

“audit toil” — so much discussed yesterday by @lucasc5 and @tod.bickley. (one of the most startling presentations I’ve ever seen.)

(toil that was okay, because “they’re only auditors”. 🙂

it shifts the approach from using controls based on heavy documentation with manual reviews to automated controls that uses evidence as the “digital exhaust” of daily work

(more YAML! 😆

very cool idea to use rego/OPA for codifying compliance @wbensing! that's how you do governance at scale!

@kapoor.vaidik, we have this example in the opensource upstream

Good DevOps Practices = Better Governance

(@bill.bensing side note: Rego/OPA is written in Haskell, yes?)

Oh yeah! Re-purposing CAB!

CAB ➡️ Modern Governance Platform Team 🤯

@genek, OPA does follow the functional paradigm! I have to admit, I don't know if it's written in Haskell...I'm rapidly looking that up!!

why not? traditional CABs don’t work anyway. 🤷 (don’t believe that? read Accelerate.)

@jeff.gallimore #truth

Congrats to entire team behind “Investments Unlimited” — it’s fabulous!

Join some of Bill's coauthors today in a live podcast interview around modern governance!! Today at 1:50 pm BST: https://members.itrevolution.com/live/links

This book is amazing! We will talk more about insights from Investments Unlimited with co-authors during AMA later in the day. Please remember to register, link is here in the session description. See you all! :) https://doeseurope2022.sched.com/event/11x66/0800-devops-live-podcast-session-with-co-authors-of-investments-unlimited?iframe=no

Continuous compliance in an automated manner. Great talk!

thank you @bill.bensing - lots to think about… the road to changing to automated governance sounds painful but worth it

We’re Sorry, Love DevOps Bill Bensing, Software Factory - Managing Architect, Red Hat and Co-author of Investments Unlimited (https://members.itrevolution.com/free-ebooks) 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/708991065/

@quinn.daley, welcome!!! It's well worth the pain

Thank you @bill.bensing!!!

But yes, painful it can be!

excited for the new book @wbensing !

Thank You All!

:fire:*Want to hear more about _Investments Unlimited*?_ Our final live _Ask-Me-Anything_ of the conference to do just this is ahead at 1:50 BST! You won't want to miss this LIVE AMA! Ivan Krnić, from CROZ, will talk to Topo Pal, Jason Cox, Bill Bensing, and John Rzeszotarksi, co-authors of _Investments Unlimited._ <https://us06web.zoom.us/webinar/register/WN_3son46R4TAiLLcFYaKG9cA> ALSO: Be sure to download your *free copy of the book* on the conference platform under the tab "FREE EBOOKS." <https://members.itrevolution.com/free-ebooks>

(PS: @scott.prugh @rshoup You just observed how these videos are more coupled than you’d expect — it’s surprisingly difficult to move talks around, because of dependencies that are impossible to change; all videos recorded independently, asynchronously, sequence not known in advance.)

Reminder: The action has moved to the breakouts! Join the following channels to interact with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4

Reminder: The action has moved to the breakouts! Join the following channels to interact with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4

Reminder: The final plenary sessions are starting again in 5 minutes. Start making your way back to your browser. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png

ANNOUNCEMENT: we are still waiting to get the approvals needed to air my interview with ING Bank CTO Ron van Kemenade — So we will be airing the amazing presentations from Disney (@jason.cox @brian.l.scott, and then Dr. @steve773! If we can get the necessary approvals, we’ll air Ron’s talk at the end of the day — otherwise, we will put that amazing interview in the Video Library, and we’ll send out an email.

And NOW, we are excited to introduce the team from The Walt Disney Company – Alexi Varanko, VP, Transformation Engineering; @jason.cox, Director, Global SRE and @brian.l.scott, Sr Staff Technical Evangelist, Global SRE. They will be presenting: Disney Global SRE — Creating Digital Magic

Hi everyone! 🙌

Hi, @jason.cox and @brian.l.scott!!!! Jason has been with us from the very beginning!!! 🎉

👋:skin-tone-4: Hello!

getting thanked by senior leaders all the time? just a typical day for @jason.cox 🤷

Baby Yoda! 😆

Was that Galaxy’s Edge ride?

“technology is critical as Disney enters Direct to Consumer world — we’re in 2nd or 3rd inning” (in baseball, there are 9 innings 🙂

and now we all want to work at Disney

I had an outrageous time reading this massive book of bringing Marvel Cinematic Universe to the big screen — https://www.amazon.com/gp/product/1419732447/

Love that video because it is full of the people we get to work with. Diane is our CIO - my boss ^ 2. She is awesome. Hope to have her join us one of these days. 🙂 Nick is our CTO at Animation, great partner. Same with Aaron (ESPN), Mike and Joe (Streaming).

Grogu! Doesn’t get more magical than him.

"Optimize for Innovation" is really a lot about story creation and story telling.

Yes! (Where did you find an actor that small who could play him? (her?))

Great book! “Ride of a LIfetime!”

Must of read that book a few times now

Side note on SRE - so much great content from many of this community on SRE at DOES - check it out: Site Reliability Engineering (SRE) Playlist - https://itrevolution.com/jason-cox-sre-playlist/

“we went from 10s of servers, to 100s.. to 1000s… to 10,000s”

There are over 100 servers just for one attraction (Toy Story Mania) ... good example of digital explosion of the business.

Tickets are must - lol

“the internet group” classic - those peeps over there doing interweb things

“there’s so much you need to know” — and that was back when life was much, much simpler… all you needed to know was Linux, Apache, MySQL, Perl, maybe nginx.

Now its all the container and serverless tech

“everything required a ticket — even going to bathroom, you’d need a ticket” 😆

Sleepy server is no good

“grumpy threw errors, sleepy had lots of latency, bashful disappeared from the network for days at a time.” :rolling_on_the_floor_laughing:

At Sun in 1986, I thought it’d be cute to name my Sun 4 workstation “panic” — dumb idea. Every log message gave me a heart attack.

Jason always has the best slides

I think @jason.cox idea of “Embedded” changed Ops for so many organizations — I remember how shocking it was when he first presented on this in 2016, and how rapidly it was adopted across industry!

I really love the embedded model, really helps us understand our business and how to bring Value to our guests

“Platforms! And that’s we made a massive bet on… Mesos!” (quoting a friend)

https://itrevolution.com/why-the-full-stack-engineer-is-problematic/

Yes! ^^ Full Stack Teams not Engineers - Jason Cox, Christian Posta, Cornelia Davis, Dominica DeGrandis, Jim Stoneham, Thomas A. Limoncelli. (2019). Full Stack Teams, Not Engineers. *P*ortland, Oregon: IT Revolution. https://itrevolution.com/book/full-stack-teams-not-engineers/

"Treating Operations like Software" - Powerful

“as SREs, we are technology sherpas”

I think one of the reasons people thank @jason.cox and team so much: having platform engineers available to help, is so helpful. Because, I found learning docker pretty difficult, and kubernetes was super super difficult. Better to just ask your buddy to help!

Using the first-person way to describe your job really helped us shape our identity and how we optimize our team/brand/efforts for success within Disney.

Great way to describe the job in first person. Imbibes more ownership.

"Define who you are through that first-person"

@jason.cox engagement model:

Super interesting: “Our role from Global SRE — help those SREs connect with other SREs”

(share the pockets of awesome; cross-pollination of cool)

Going to quote myself - "Great pollinators of cool"

I agree with full stack team. Full stack engineer is not a thing.

Flexible Support Assignments: months to years

Knowing the business side as an engineer helps so much.

highest level: Dedicated Embedded Team: embedded within the business unit, and now you’re able to bring to SRE Global (mothership) needs from edges of organization (where the money is at!)

“we ship happiness” 🙌

yes, each service knows the ins and out. Pretending for a global SRE to manage platform+ srvice both does not work

"Better, Faster, Safer & Happier"

Better, Faster, Safer, Happier sounds very much like BVSSH ❤️

Remember when I first talked to @jonathansmart1 about this, he would say... Sooner Safer Happier - Jonathan Smart - https://itrevolution.com/sooner-safer-happier/ 100%

Better, Faster, Safer, Happier. (Loving how these words are being used so widely — go @jonathansmart1!)

Hi, @brian.l.scott!!! Congrats on your new role!!!

Thanks @genek!

“We bring security folks to help everyone learn how to make their things more secure!” 🎉

For some reason I didn’t expect Continuous Delivery to Smugglers’ Run.

How utterly awesome — so cool, @ann.marie.99!!!

Imagine... sitting in ride vehicle, see an issue, pull out your laptop to pull down the code, fix it, push it to the repo and see the CI/CD deliver and deploy the new code on the next cycle. Fast feedback = better products = more happiness. ; )

It's pretty incredible!

I love these stories of elevating productivity of some of these epic / legendary groups, @brian.l.scott!!!

“Excel spreadsheets” “opening tickets” 🎉

I love being able to tell stories of our SRE Teams!

“I’m embedded in Industrial Light and Magic.” (mic drop) @brian.l.scott

what a presentation.. where do i sign up 😄

✨And we're EXCITED to present the next session, An Interview with Ron van Kemenade, CTO of ING Bank ✨

yayyy

THANK YOU @jason.cox @brian.l.scott!!!!

Thank you all!

Thanks everyone!

great presentation. SRE stories are most inspiring

Will the presentation be available to download and re-watch please?

“only reason it wasn’t worse was that you couldn’t rate an app with less than one star” 😆 I loved that joke. Went into Unicorn Project.

Ron van Kemenade 2016 London presentation: https://videos.itrevolution.com/watch/524438094/

a mainframe is difficult to power down? confirm four times, please!

Fans do not turn off for several hours, even after you power it off — because there’s very few scenarios where you wouldn’t turn it back on!

Nobody designs a mainframe for a graceful exit...

the problem wasn’t the mainframe — “the problem was the 250 applications were running on it; payment and many of our product engines, like mortgages, consumer loans, business loans, a lot of business processes related to those products, our customer and agreements, uh, administration was running on. Part of the financial accounting was running on it. So consider it literally the core of the bank.”

“Mainframes are one of the most long living and fully supported and very much alive platforms in the world still I’m supporting a lot of enterprises, governments.”

“It’s harder to find great programmers in ALGOL, PL/1, COBOL anymore. The ones we have are leaving the bank” — retiring.

People do retire 😄 😄

This apparently was a race against time — the knowledge pool was shrinking the whole time over the 7 years of this project!!

How easy is it to find staff for these mainframes now? And projection for the future...

“success of this required trusting your engineers; and endurance to persevere over such a long conviction. and absolute conviction that there’s value in doing this”

“No one wants to sit on pile of legacy that no one understands anymore; it’s not easy, but a sheer necessity” (!!)

"pile of legacy"

“no one wants to sit on a pile of legacy that no one understands anymore” :rolling_on_the_floor_laughing:

“when they heard what we wanted to do, located, they literally said, ‘Ron, you’re crazy.’”

“Along the way, we encountered many… complications. People retiring faster than expected, funding / priority issues.” (I tried not to laugh, but the story is so good.)

Imagine a journey when the people required to succeed kept disappearing (retiring, which they deserved!!!)

the courage it takes to “let’s go; if we have problems, we’ll fix it on the other side — we must have enough bravery to keep going.”

another common element of these journeys: courage.

yes! very much evokes the image of a person climbing a mountain, with party shrinking as climb goes on. @jeff.gallimore

“I truly understand your concerns; but I take responsibility, and now we’ve got to go.”

“if you don’t take the last step, you can always turn back” (oooh…). “we must leave this world behind.”

Burn the ships

"Now I can actually retire"

“he was standing next to the guy who was there in 1980 when they wheeled in the mainframe; and he was there, powering it off. he was not sad; he was proud; he was the last man to turn it off; his account to power it off”

“now I can actually retire”

https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1652375175872059?thread_ts=1652375151.331719&amp;cid=C015DQFEGMT

how many have worked on COBOL here 🙌. Raise your hand. I did 😄

went from 2 day international payment settlement to 15 minutes; it wasn’t the code — it was the way all those applications were chained together!!!!

“Wow, what happened. You processed my dollar payment in 15 minutes!” (So great!!!)

“No one wants to have PL/I on their CV” (anymore)

“we scheduled it to 5 years; it took 7 years” — but they will benefit from this project every year forever.

@jason.cox will the slides be made available for your presentation? Some useful points I would like to share with my team

Disney Global SRE — Creating Digital Magic Alexi Varanko, VP, Transformation Engineering, The Walt Disney Company Jason Cox, Director, Global SRE, The Walt Disney Company Brian Scott, Sr Staff Technical Evangelist, Global SRE, The Walt Disney Company 🎉 Now Available for Sharing: https://videos.itrevolution.com/watch/709121287/ Slides: https://github.com/devopsenterprise/2022-virtual-europe/blob/main/Jason%20Cox%20-%20Brian%20Scott%20-%20Alexi%20Varanko%20-%20DOES%202022.pdf

@jason.cox @brian.l.scott You all rocked.

These replatforming / rearchitecture efforts require so much courage and conviction because you only get the benefits at the end. It's all or nothing. And while you are in the transition period between architectures things are worse -- you have 2 systems to keep in sync, in addition to all of the horrors of the old system.

“must have conviction that there will be benefits to customers and firm; the risk was that every year, critical knowledge is leaving the firm. every CIO must be worried about this — a real risk”

“every CIO who takes responsibility must think about [the risk of] knowledge legacy, because it will sneak up and haunt you.”

@rshoup @rshoup This bit is so interesting… wait for it…

There are great parallels here with the TUI story from @pieter.jordaan around risk. Many times doing nothing is WAY riskier than forging the new path!

There is almost nothing riskier then your talent evaporating away...

Agile spagetti eating!

what if your plate of spaghetti looks like this?

I.e., if architecture is sufficiently homogeneous and no partitions, it actually doesn’t matter where you start. It’s all the same, so it doesn’t matter where you start,” COME ON, that’s amazing, right?!?

If the system is heterogenous but modular, where you start matters little too.

Can you confirm with an emoji that you appreciate the amazingness of that story? I want to share this with Ron! THANK YOU!!! ❤️

:unicorn_face: And now, bringing us to a close, our very own @genek and @steve773, who will teach us, What is Architecture, and Why It Matters

Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png

ARCHITECTURE!! Very pertinent given that last talk!!! Hi Dr. @steve773!!

Yes, @genek, your book will come out THIS YEAR.

this year Gene this year!!!!! 😄

Bring on the questions and comments folks 🥊

(Incidentally, when I took that class in 2014, I discovered in Boston that I forgot to pack pants. 😱 )

2023 everyone. Book comes 2023. Just in time for the 10th Anniversary of IT Revolution !!!

“First we design our buildings, and then they design us” — Winston Churchill

We behave on them, and then they behave on us…how we architect our engineered systems determines how they architect us.

Function over form!

Fascinating. With what @steve773 said: the problem that Ron van Kemenade was trying to solve was the calcified pathways of work, caused by the 250+ mainframe applications that they could no longer change. Change the applications, and suddenly you can go from 2 day settlement times to 15m. Not the applications, it’s the way they’re arranged.

@jim@james.moverley Structure determines dynamics, form determines function.

Very relevant to the @mik talk yesterday!

Chapter 4 about Alcoa achieving 200X delta on Safety. Chapter 5 about huge deltas on R&D productivity and quality.

Everyone starts at a very low low level of competency and capability.

@steve773 quality journey from worst to first started with the toyota pet toyopet, right?

Some thoughts on Toyota and why it found it so important and differentiating to institutionalizing it’s management system…https://conta.cc/38lVhFI

So interesting to consider the opposite of this: “people trapped in a system that constrains their full problem solving potential” (the common battle told in almost all DevOps Enterprise stories!)

@jeff.gallimore yes. 1958. Toyopet. Terrible car and terrible production system and terrible labor managment relations. Worst in the world. Within 20 years, best in the world.

Comparing the great orgs from the horrible orgs — you can isolate the dependent variable to one thing… the management system, which yields the structure and architecture of which people are working within.

“Fragile vs. Agile” 🙂

PS: it’s been so super fun to work on a project with Trent Hone, whose work @steve773 has been citing for 5+ years. 🎉

Integrated systems: Inflexible in Dev and Fragile in Ops Modular systems: Flexible in Dev and Resilient in Ops

“we were reliant upon 250 applications that we could no longer change” (Right?!?)

“modular and nested” that sounded like an important point

This theme has shown up so many times today: @ben.grinnell said: “every major program had 7+ sourcing partner”; Ron vK mentioned something similar at ING 2010, where nothing could get done.

This is a tremendous framing. So powerful. Strong parallels to Dave Farley's new book Modern Software Development 🙂

this is so good, i will rewatch it few more times 🙂 dr Spear , i have just become a fun of your work !

@jeff.gallimore Modular + nested --> mean we can express our creativity locally and in parallel and get to flexible Dev and resilient Ops.

@piotr Thank you!

@genek little that’s complementary about a GM plant…certainly not the GM management…ugghh.

I heard from a friend: “we are all coupled together, because of the technology platform we run on — in order to get anything urgent done, you see these signals like pebbles on a pond, as nodes try to signal to each other. Only CEO intervention can get the system oriented onto one urgent things, usually a customer issue.” Coupling.

OBIDOS!

@genek “some systems constrain or even extinguish the creativity of people in the system…” That’s what despots do!

@genek nested modularity…necessary for having a well functioning liberal democracy?

As an architect : it does become clear how the design affects flow - this can also be said true of the low-level software architecture of components and how developers can interact. What fascinates me is the "reverse Conway manoeuvre" and the impact of the architecture on your enterprise organisation!

This should be fundamental teaching for everyone in the industry. Across all roles. No exceptions.

API were a Microsoft mantra for all their products

@rwiankowski Thanks! We’re ready to take our show on the road. PRODUCT PLUG! We’re bringing my exec ed course at MIT Sloan on line by this Summer, in which we build on the idea of simplified, standardized, stabilized, and synchronized.

😭

this afternoon has a spaghetti theme

To @james.moverley’s point…all depends on the direction of design. Start with your silos and see what you design, or start with the design objective of nested modular, etc. and see what silos are needed. This is a huge idea in the idea of EMERGENT DESIGN (architecture). Initially, the profile of problems that have to be solved in the baseplate engineered system determine who has to talk about what with whom Eventually, when the engineering system’s structure stabilizes, so does the organizations. The problem being, both calcify…rarely does a change in the problem profile result in a change in the organization’s architecture.

PS: here’s the original picture:

blood curdling, no?