Great question! Off the top of my head, I’m thinking about 30% have “regressed,” usually due to a change at the top of the tech org. 😭 😭 😭

One of the unicors has a funny 1984 like story @ffion and I were told a year ago fit for maybe virtual happy hour. Must be in the 30%

We laugh, because we cannot cry, right?

Is it regressing or remaining? I saw agile flushed by new management and there are no two pieces intact in that puzzle after that. However flushing the CD from version control is a bold move rarely seen.

We have come a long way at CSG!! https://techbeacon.com/devops/10-companies-killing-it-devops-2020

Are you saying some horses got to it at better times and overran them? @ferrix

that is very cool indeed @scott.prugh

I am saying that culture is demolished but some of the assets remaining will erode.

Also, it takes more to be a unicorn than it used to :unicorn_face:

Would be a good little blog post to publish that format and explain why you use it.

https://itrevolution.com/love-letter-to-conferences/

:thumbsup:

I only hear Gene, no sound

Same

@patrick.debois256 @jeff.gallimore

Silly Firefox ... new tab opened an "auto play" Sorry for the noise

that's what i call CD

haha... tested in production. it works.

hey - first tested in staging 🤡

@rikard has whatsapped to tell us before you did!:) how cool. Reality TV of DevOps

I’m still reeling for this story…

Ahhh nice, thanks for that!

None. It must be your network. Hahahaha. (Real answer: many. 🙂

Great editing then 😄 I produce some videos so it’s all interesting to see how other get on that regard. :thumbsup::skin-tone-3:

Thanks @me1342! @ross.clanton410 and I are excited to be here.

(I’m talking to you, @me1342 🙂

Thanks for having us, @genek101!

and #ask-the-speaker-more is a great addition

We’re hoping to have all videos publicly viewable by end of conference. It’s a NYTimes-style paygate — watch up to X videos for free. We’re planning on publishing a portion to YouTube.

@nickeggleston post the topic in #birds-of-a-feather to gauge interest. then if you want to host something, bring your own Zoom account — just post a link. that said, if the topic is close enough to one of the existing BoF themes, you might just bring it into one of those sessions to get instant traction.

thanks Daniel, and welcome back !

http://videolibrary.doesvirtual.com

Or did you mean the GitHub link for the Slides?

click plus next to channels

type bof and hit enter

They are all in there...#bof

Thank you, Marc!

You should see them when you search on bof

Thank you!

It’s also on GitHub.

Do you have the address?

@chris.gallivan Do you have the link?

Dropbox: http://itrevolution.com/DOES20-db GitHub: http://itrevolution.com/DOES20-git

PS: I’m such a huge fan of the video library — all the London videos are in there, too. And I’ve used it so much in the last couple of weeks, to help prepare for this conference!

Hi Ann. Thanks for the watch link. For what it is worth, that link is hard to find in the ITRev website, on the events.itrev page etc.

Steveeeeee.....

Hi Ann. But…I am on line and logged in today. Yay!

Yay!!!

Looking forward to your two sessions!!

Me too. I hope they don’t suck. 🙂

Miss you too Luke. Glad to see you here at the conference.

We all miss Ross!

@nickeggleston The excitement is both top down and bottoms up. We have terrific leaders who are paving the way and we have a lot of grass-roots practitioners who are filled with infectious enthusiasm.

You’d think safer would’ve made the cut :)

Me too! It's awesome to hear the journey we've been on here at American

@mattmasuda!!! Long time no speak 🙂

Hey Lucas!

Hope things are going well for you!

Going good! Thanks, same to you!

@denee.ferguson We started really talking about this in earnest about 4 years ago

@jtf Udemy has been super helpful for us

Thanks @maya.leibman. I was wondering if there was a particular course or set of courses on Udemy that you feel had the biggest impact?

We set learning journeys, curating various courses for different roles/skills. We could deep dive on this more in our networking session this afternoon if there is more interest...

The ones I personally led were a combination that depended on the team. If there were DevOps engineers they were engaged, but the team breakout was product team specific. If they had QAs we had them involved since their work factored into the current value stream.

We try to have both business and IT in the room - lots of a-ha's in the process...

+ product / business stakeholders...

Whole product teams!

stay tuned. Maya is about to answer that question in the talk 🙂

The short answer is get Finance to help you lead the transformation...

You really need forward thinking people in Finance. Have your folks call our folks

such a reality… I used to be told “make your finance partner your friend” quickly followed by “keep your friends close and your enemies closer” back in the day

Seriously!

It sounded like 1/3 Grow, and 2/3 Run.

ROFL... That's where we met. That day completely changed the course of my career... 🙏 @genek

Right back at you, buddy!!!

And the course of DevOps for the Whole Community!

and a 5 tab excel spreadsheet with all future costs and resource identified for the next 5 years

Just getting back up to this question. Hopefully our evolution of measurements (inputs -> outputs -> outcomes) answered this.

how to measure "Improve Ops"?

The big KPIs are around frequency of airline-grounding problems and the time to restore and then vulnerability management. If those are in check, we're in a manageable spot

See also, OODA loops, PDCA, etc.

such a fan of Sam Walton quotes and thought… coming from a Target guy…

He was DevOps in the 60s

Leaders are there to remove impediments from the product teams!

I thought you’d say that!

Something like this ...

We created a taxonomy of products (oriented around experiences & capabilities) then mapped funding to the products & product groups.

Thanks. That sounds significantly more granular than TBM. It was difficult to read the slides in the live presentation, I will definitely take a closer look.

thats the Ross i know!

😹

Bauhaus!

Slamming Sky Five!

And what roles were represented?

We didn't leave anyone behind. We asked everyone to participate (all 4,000!). Initially we didn't have any central team leading it. Now with Ross we have about 30 people (including coaches) who are pushing.

What was the role (or roles) of those making the invitiation to 4000 people, and what was the resonse from that large community initially?

One of the most difficult parts of a transformation is not leaving anyone behind. easy to just say jump on or your OUT!. lol

Come to our networking breakout later this afternoon. 🙂

I will!

Hangars ROCK!!!!!

@chris.gallivan Looking forward to seeing you there!

@chris.gallivan, did you have a specific question in mind?

No just looking for you!

Continuous improvement!

Agree @maya.leibman! Kudos to you for having the courage to start where you are!

I'd love to say we have that one figured out, but we're still working it. Very open to ideas on how to do it though.

One question we used at TIM was in a survey asking something like “Do you believe there is mutual trust and respect within your team?” … across teams? … between departments? In our first survey we had a very low baseline. When that came up we felt we were making progress. < @ross.clanton410

We have HR representatives on each working group, but we haven't made changes in the HR group yet. Mainly, we don't feel those artificial constructs are holding us back...yet.

These tracks are very cross-functional w/IT, business, HR, etc.. all involved.

Thank you. Great talk and amazing progress!

We have a bunch of evangelists. We plan to discuss Innersource later at our networking session this afternoon. You should come check it out...

Will do, thanks

I know - isn't it so weird to see people next to each other?!?

It is. BTW... I consulted there the last half of 2019 and first half of 2020. Good times.

might have been Ross that implemented “Make Decisions, Not Decks”

i still have the sign hanging in my conference room

In many ways we've gotten more productive with COVID. People enjoy having the heads down time. But we also need the ability for teams to come together and we've started bringing folks back in order to get the best of both worlds

Maybe just read the comments and reply without having the video playing? 😄

It's kinda weird to hear it for the first time live. Y'all are keeping me busy with Q&A though so it's more like background noise 🙂

Turn the sound way down!

We're definitely still working on that. Change is hard. The words we're using is "Empower teams and try not to put our personal leadership stamp on everything"

The PSA values are relatively new in our transformation. We had a previous set, but these simplified them further. now working on the incentives/rewards around it..

I love them. At Barclays we found that, in the end, line management often have more discretion than they realise and whilst the "late majority" population make noice about the incentive system it perhaps was not as much of a blocker as sometimes presented.

Did you remember to put the tray tables up and return the seat back to its full upright position when you were done?

A Seat at the Table War and Peace and IT

thanks

https://itrevolution.com/war-and-peace-and-it/

The eBook is currently on sale for $2.99!

Objectives and Key Results

Objectives and Key Results

@chris.movick “Measure what Matters” is the best place to start with ORKs. https://www.hawkins.io/book-review/measure-what-matters/

I have a non-popular opinion of OKRs when they are used as a mindless replacement or when they eskew the people work -which they too often do-

Pretty sure I agree with your opinion. OKRs are not individual performance metrics.

It's old but stands I think @bryan.finster - BTW hello! https://www.linkedin.com/pulse/psychological-safety-hard-lets-try-okrs-instead-duena-blomstrom

I see what you did there 😉

@jackvinson We still struggle with this. The words we use are "empower teams and try not to put your own personal leadership stamp on everything." We just need leaders to realize their job is to remove impediments. And need to control trying to change what the team is doing.

I liked that value statement. It implies more of the in-service-to mentality, rather than having to prove "my" value (as a leader).

I like the line from Team of Teams : eyes on, hands off. Teams need to provide transparency, and that transparency allows leadership to let the teams make decisions. Then it is the job of the leaders to endorse those decisions to reinforce that the team had the authority to make the decision.

Definitely making a push towards IaaS but in the meantime they function as platform products

@jason.johs @asgeir.olafsson @erica.morrison Lets do this!! PR to prod in 1 hour!!

"Ditch the release calendar" :)

Sounds good 🙂

Custom off the shelf

commercial off the shelf software - bought from vendor

Commercial Off the Shelf

Commercial Off The Shelf software

Or that.

thanks all. TIL 😄

It was a key part of our success. Breaking the deployments into smaller chunks and driving testing automation were two necessary hurdles we had to overcome to get to CD.

Commercial off the shelf

https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1602606576334300

commercial off the shelf software - bought from vendor

Thanks

couldn’t agree more!

I hope it gets taught... the alternative makes me cry.

I can take this straight to higher ups in my company. Framing is really important. My gut tells me this will really help product owners collaborate with engineering.

I already pinged the whole management team on company Slack about this... publicly 😄

@maya.leibman @ross.clanton410 Curious if your leadership focus leans more Transformational Leadership or Servant Leadership focused (or if you emphasize different things at different leadership levels)? I'm having this debate within my org right now. A potential trade-off we're having is the pendulum swing from prescriptive to servant leadership is creating a gap in people focused on vision and strategy.

I think it's a bit of both. Transformational in how we focus on vision and outcomes. Probably more Servant in terms of how we want leaders engaging with and supporting thier teams.

The job of each of the layers above the team is to remove the impediments that the teams are facing.

“The opposite for courage is not cowardice, it is conformity. Even a dead fish can go with the flow” …I’m pretty sure you need to expense that Wookie mask. yw.

🙌

Yes. I actually own many of these shared services as I have most of the infrastructure products in my org. We treat our enablement shared services (e.g. Hangars / Dojos) as products too.

Would love to connect and understand the process you went through to formulate those product teams.

Would be happy too. You can join our networking session and see if we can cover some if it there. Happy to connect further later as well

Great, see you at the networking session; currently working on an effort to align our Infrastructure/Shared IT Services groups into product teams while the rest of the IT organization is under a similar transformation

@jason.hobbs or @steven.leist this would be a great one for you to respond to.

@derick.stenftenagel honestly, it’s something we’d been trying to do for some time, with relatively low success. Each time, we had big ideas and big plans, it always came from ‘top down’ and we treated it like a large effort or a project. In that mode, we always seemed to struggle to invest enough time and tools etc in moving the needle. It always involved tool purchases and big up front roadmaps, and they were mostly about trying to automate what we already did. It wasn’t until we took a more informal “run at it” last year using an Improvement Kata, NOT as a big sponsored project, but as a few hours each week that people ‘with skin in the game’ - the actual people doing the work - voluntarily participated in. At that point, we starting making strides (really small ones at first, every couple of weeks) to just improve one thing at a time. The Improvement Kata was the key. The Improvement Kata allowed us to just inspect each thing we learned every day / week, and start asking things like “what if we just stopped doing this?” or “how can we just make this one thing better?” - The people involved were the people that owned these systems and processes, and quite honestly, at first, most of the big changes just came from changing process and removing gates. No tools, No tech. Our first “next target condition” was “Release 2x per week” - when we got there, we realized it was a burden and people were maxed out. The next “next target condition” after that became “release in the middle of the day” - each time we reached a new target condition, we relieved the burden on the PEOPLE and made sure to not make work harder, but smarter and smaller. Eventually there were so many people “all in” with their newly freed up time that it just snowballed. Happy to answer more questions about this in the #bof-american-airlines channel, and happy to connect directly with anyone who wants to know more. The tech and Kubernetes aspects are super cool as well, but they were definitely means to an end, versus a planned target.

I hesitate to think of hacks when we're talking about people, but I think you have to start with incentives, specifically how they align (and don't) with the mission

The right way has to be the easy way, and ideally the rewarding way

@sam793 We started with the willing and built successes that they could share with others - then we give them different platforms to share those stories

@steveelsewhere - agree, always interesting to reflect on how these incentives and measures influence the decisions get made, when these are not transparent its incredibly difficult to drive change and collaborate

Could not agree more... 🙂

@ana.torres love it, it’s so easy to try and go BIG fast especially in orgs as large as AA! But doing as you say, getting small wins, making these willing leaders heros one by one and letting that replicate is the right way to effect org wide change

I was going to say exactly that - changing whole orgs isn't possible and an exercise in futility. Maya's story about the business advocates meeting was true. But we now have quite a few business leaders at the table and actually making change happen in their orgs.

try out #ask-the-speaker-more

Yes... but... then I can't pay attention to the current talk... maybe it jsut a personal limitation.

Totally fair. 🙂

Nobody has asked a question there, so probably not just you. 🙂

Changing channels breaks flow... it's awesome when it all goes into one video session and chat... just need to automate separating it into different buckets after it winds down... so hopefully later-watchers can come and ask follow-ups there. Hard to implement in slack, though, I am guessing. If only there were some Dev-types around... :rolling_on_the_floor_laughing:

😆

whoah

That is awesome!!

epic!

AA has a lot of people attending this year :)

Wow! VERY nice @misty.shafer!

Okay I need that.

@misty.shafer WHERE ARE YOU?!?!?

Our Star Wars media room!

That is so cool. I have media room envy...

Agreed! Well done. Very helpful.

Hey Ravi, come see us at our networking session this afternoon and we can connect further.

line one in all of our job descriptions

Incredible talk. So inspiring.

Great Presentation , it was a good realization of how far we have come !

@srinivas.sarathy We continue to improve our approach but yes, we still allocate funds to products and continue to work on OKRs so that they're truly outcome focused. We are continuously improving and working on leader mindsets!

Yep, funding is all allocated to products now. We're in year one in the new funding model. Still pivoting some project behaviors, but we are getting better every day at OKRs and prioritization of funding

We have de-centralized the majority of our releases. We have also localized change.

We still track CRQs but execution is de-centralized

decentralized even for infrastructure ?

We measure what is called an Impact Minute = MTTR * Failure Group Size

That gives us a measure of not just how long recovery takes but how many transcations, subscribers, etc are affected.

percentages of changes that result in degraded service, require a hot fix, or rollback

We currently measure rollback directly, since that can be instrumented. We re also tracking defect rates. I've no idea how to instrument the difference between a new change and a roll-forward hotfix on on a high performing team.

Change failure rate is an interesting one. With a high cadence of changes, no such thing as one root cause (a conflux of factors coming together), it can be hard to determine causality.

no doubt, as velocity increases the fuzziness increases.

We seem to get stuck on if my change completes by my measure, but causes impact to your product if that should be counted on the change failure rate. I was curious what others thought, thank you for your input.

If metrics are used to judge from the outside instead of helping things improve, then to goal will be lost. The goal is "how do we better serve the customer?" "I deployed and I'm fine, but it broke you. /shrug" doesn't serve the customer.

If people aren't focused on actual outcomes, then they need better incentives.

Honestly one of my favorite books. I cannot recommend it enough.

Dropbox: http://itrevolution.com/DOES20-db GitHub: http://itrevolution.com/DOES20-git

Thanks @jtf!

That was fast, does this include the slides from the keynote and this current talk?

Thanks a ton

I haven’t checked. I think so. Videos of breakout sessions are available now, and the plenary sessions later: Video library: http://videolibrary.doesvirtual.com

Must be same with the slides, I don't seee the plenary slides yet

The dropbox link is populated but the Git one appeared empty to me.

This issue manifested itself as a DOS attack... Basically the protocol stack on AIX had differnet TCP/IP window sizes which caused packet backup and servers to tip over

Sounds like something that may make its way into a novel...

<huge smile> Totally. And of course you took note Dominica. Thank you.

Big red flag!

“and the Solaris servers were already powered down.” 😂😂😂

lol

stunning STUNNING lack of communication/coordination.

But how many people do we run into who are convinced they WILL make it worse? (Reflecting bak to the AA discussion of preferring Action over Analysis)

Tansactions per second

And just for context, our customers' digital transformations fueled this explosive growth. Great problem to have... right?

That's phenomenal .

Old ideas are new again. I think your analysis fits. :thumbsup::skin-tone-3:

I think one could even argue that depending on how non-identical your (pre-Infrastructure as code) environments are and how heavily non-automated deployments depend on who does them, many release processes fit the (Cynefin) definition of complex - doing what you think is the same thing as last time may result in different outcomes, resulting in major unpredictability. And of course in this case, any variation is usually in the unwanted direction.

I want to agree and extend @mathias.eifert. Even in the case where we’ve automated everything—or perhaps even especially in the case where we’ve automated everything—we can end up with emergent complexity. The interaction of lots of simple pieces, when there is interdependence, can yield unexpected complex responses.

Good point, @jtf! That's perhaps the underlying reason why monitoring the processes that we have turned "simple" (by constraining them through automation) for brittleness is a key counterbalance in order to maintain resilience.

change request

Sounds like the ops team was pretty customer-centric, though! #silverlinings

It is surprising how pervasive the thinking is that developers left to their own devices write terrible code. Does this happen with other job roles?

Aren't developers just optimistic? So experiencing the hostile production environment enlightens them on how to be better at their craft.

Off-topic: That eye makeup tho! 😍

Ops members are part of the same team that builds the software

We told them the job requirements were different. We told them we'd help them build the required skills. Some dove in. Some opted out of the company.

Definitively drives different behavior. <smile>

You have no idea. Question #1 was are we going to pay them more because they're going to be on call.

when i was a dev manager back in the day… i once had an ops manager tell me to “eat my own dog food” and then I was put on call. Let’s just say that i started pushing better code!

I love it! Creating a shared view of delivering value. Set the tone for the transformation - breaking the culture of “my”.

How long have you been doing this?

we have been on the journey for 1 year + now

did your org provide training for folks to ramp up on tech skills?

Not a lot of training outside of automation. We relied heavily on leadership and front line managers to lead the transformation

It, literally, might be the only way to truly improve it.

YES! They will script it instantly.

Yeah, I mean of course, it has to be done right, but too often devs are siloed away from problems the code exhibits in prod.

I used to be the dev sitting next to the ops guy doing repetitions. I saved him the next day.

The teams I support have been required to carry the on call phone. It makes it much more personal for them when they deploy code. 😀

on-call carry-on

I wonder how that impacts on NPS, Engagement, etc. Devs are so lucky to so often be in flow/zone and that's key to why they do what they do - the clean-up or fixing part is not flowy

clean up, defects, etc is a sign of low quality. Quality needs to be built in at the start, things like pair programming, swarming, test driven development, etc. Being on-call gives you a focus on high quality before you send your work to prod.

Seeing flames helps in building quality

DRI - Directly Responsible Individual - if your code is what breaks things in production - your phone rings

Well... if that's blameless 😄

I am starting to develop a pet peeve against "blanket blameless" 🙂

DRI was explained to me by PM at Microsoft on how they drive code quailty and accountability while empowering devs to release on demand.

@cncook001 being on call and dealing with unintended consequences helps bring awareness to the wider effects of what one does....

Right. It leads to discussions how do I not get woken up for this code release. Things like multi region deployments, blue/green, canary's etc.

I agree with these comments. It really allows for accountability (which isn't a bad word) for their work instead of a false belief that operations is "someone else's problem". As stated above, quality needs to be in the forefront of everyone's mind from the beginning. I jokingly tell the folks I work with that it is their BBQ on Saturday that will be impacted if they don't think of quality/reliability. It usually only takes one time! 😂

Indeed. It definitely drives behavior in the right direction. And we know that there is no culture change by fiat, rather culture is created by daily behaviors. Therefore... change the behavior = change the culture.

@me1342 Blanket blameless sounds like too much. However, the "you make it, you fix it" at extremes is "you fail to fix it completely the first 48 hour shift, you are the one to wake for the next 48 hour shift, when your loaned time expires" is a dumb way to work. So, "actively claiming team responsibility" would be a better option where finding the culprit is trumped by finding the heroes.

Hmmm... maybe send an automated email to a person whose code was patched off-hours according to git blame... 😄 "Hey, did you know, someone made a change very late.. what could that have been?"

You need enough depth in the teams to have a healthy rota, and to be able to excuse the on-call person from project work so that they can directly fix broken prod things during the day without also needing to contribute to team output.

Good conversation... there is definitely tension between individual and team responsibility for issues related to code quality. Each "failure" is a learning and growth opportunity, and having a learning culture is important to the success of the business and happiness of the employees...but... if there's clear line of direct responsibility for the code and an expectation that you will be engaged if it breaks, then the risk of being called at odd hours should hopefully drive attention to both obvious and higher-order effects of a given change.

This was often justification for keeping Dev and Ops separate. We are regularly audited and you just need to show the right controls are in place. Design/Build/Run teams are key.

I agree - was just wondering what specific controls you arrived at. I'm collecting stories!

This brings back memories! LOL

me too

See our paper on this: https://itrevolution.com/book/breaking-the-change-management-barrier/

We basically decentralized our chnage processes...

awesome, thanks Scott, will definitely check it out later

Frankly, it took 18-24 months but getting the right tools in place was more of a long pole than upskilling. attrition (planned and unplanned) was close to 20% over those two years.

ott Prugh - SVP Software Engineering - CSG

everyone liked this ...

i just started making a list of the 5x and 10x developers i know… 🤯

should a technical leader still be writing code? would you be concerned if you saw your cio merging to master on a Saturday, at 3am?

slightly kidding 🙂

Seems like the 10x devs might want to stay 10x devs. And be extremely successful that way. Reward leaders and 10x devs so you can have both.

completely agree Jeremy… these are the best people i know, so I’m going to think through who could be the 1 leader and start having the career conversations

Our entire senior leadership team were all lead developers at one time, but we all became sold on the idea that we could have more impact by leading than coding.

To a person, I think we all still do a little coding now and then. Love it too much to quit entirely.

This!

Amazing @scott.prugh!

I think they were tracking the percentage of features that were decoupled from the deployment, I’m assuming feature flagging or some of branch by abstraction approach.

We measure the % of features not batched into a release(ROD%). Basically it acts as forcing function to decouple features from a releases.

we had quarterly releases and no code drops between release. now 60-70% of our features are delivered when complete (released on demand). the batch size of our quarterly releases is that much smaller.

We had quarterly planning and quarterly release. We now plan quarterly and on demand and release every day.

Thanks

Thanks for sharing that Gene. I wonder if imposter syndrome played a part in Scott's relictance to put the spotlight on his story.

So is this the path forward for a Brent or a Maxine? Does one cease being an IC or is that a different path?

Great that we're all speaking about it! Our session about Psychological Safety and Impression Management is on Track 3 day 2 at 2:05pm

Have you done any work on the interaction between psychological safety and imposter syndrome?

Your leadership needs to model the behavior you want to see. Start there.

Basically Change Behavior First to Change Culture. See Shooks Model of Change

i'll look up that book

Take a look at Lean Enterprise. It is referenced there

This book illustrates a good way to shift the culture of a team to become a learning organization: https://www.amazon.ca/Around-Story-Turning-Followers-Leaders/dp/1591846404

How does Read and Research voraciously balance against the pressure to deliver features faster?