This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # ask-the-speaker-more (15)
- # ask-the-speaker-plenary (1192)
- # ask-the-speaker-track-1 (220)
- # ask-the-speaker-track-2 (196)
- # ask-the-speaker-track-3 (323)
- # ask-the-speaker-track-4 (212)
- # birds-of-a-feather (10)
- # bof-arch-engineering-ops (1)
- # bof-covid-19-lessons (1)
- # bof-leadership-culture-learning (1)
- # bof-project-to-product (14)
- # bof-sec-audit-compliance-grc (2)
- # demos (7)
- # faq (4)
- # games (69)
- # games-self-tracker (2)
- # gather (5)
- # happy-hour (39)
- # hiring (10)
- # lean-coffee (13)
- # project-to-product (12)
- # psychological-safety (1)
- # summit-help (79)
- # summit-info (156)
- # summit-stories (3)
- # xpo-anchore-devsecops (5)
- # xpo-cloudbees (4)
- # xpo-copado (1)
- # xpo-epsagon (1)
- # xpo-gitlab-the-one-devops-platform (13)
- # xpo-harness (1)
- # xpo-hcl-software-devops (9)
- # xpo-ibm (4)
- # xpo-itrevolution (16)
- # xpo-launchdarkly (26)
- # xpo-mirantis-devops (10)
- # xpo-pagerduty (11)
- # xpo-redgatesoftware-compliant-database-devops (8)
- # xpo-snyk (3)
- # xpo-sonatype (4)
- # xpo-split (25)
- # xpo-synopsys-software-integrity (4)
- # xpo-tasktop (10)
- # xpo-tricentis (4)
I do love these conferences and get a lot back from them. I do sometimes get frustrated though. A bit like getting into artisan coffee and having to go back to instant! So much to do in my area 🙂
Sure you come back from conferences willing to change the world in one (let's say two) weeks, and got a bit of frustration needing a few additionnal monthes... 😉
Any transformation is a long journey, days or months or years depends on the culture and commitment
Frustrating, and yet it helps me keep the faith. It isn’t just my imagination, these things exist in the world!
we've changed organisation structures many times over the last three years so it often feels like starting again with new leaders!
If culture and mindset don't change, probably you can exhaust all available leaders and stay in the status quo indeed
Don't underestimate the therapy effect of this conference. It helps with the enthusiasm gap to know that we are not alone and many companies have the same problems and are solving them!
thats a great point, with all the setbacks this does help to reinvigorate and provide the energy to keep pushing!
i heard a quote once… “the really hard problems are not with the hardware, software, or firmware… they’re with the peopleware.”
Tech issues so much easier than human ones... Hopefully some powerfull AI once doesn't come to this conclusion...
Classic book: https://www.goodreads.com/book/show/67825.Peopleware
I think I got “all problems are people problems” from https://www.goodreads.com/book/show/566213.The_Secrets_of_Consulting
Ah, the birth of the Next Gen Ops and Infrastructure track, from @jason.cox
The SME talks like the one from Chris Strear yesterday, so invaluable it's crazy
The literal firehose of information talks from John Willis are great memories, I remember him saying a few years ago that K8s will come in big... 😄
@genek101 (@genek), what are the other conferences that are favorites of yourself and other members of the programming committee, and why?
Really appreciate the networking time, @genek101! It’s a big difference from other virtual conferences I’ve attended.
@jonathansmart1 played a huge part in helping us frame the networking at DOES and @dominica for many years brought leadership and collaboration to our Lean Coffee sessions!
The networking time is such a huge part of what makes DOES a fantastic experience. Thank you @jonathansmart1 and @dominica. Let us know how we can help make it better.
Glad to hear you’ll be keeping lessons learned from the virtual conference in the physical world
...and hopefully continuing some virtual conferences or integrating virtual and physical, once the world gets back to normal.
Many thanks to @jonathansmart1 to helped crack the code on how to get networking time right!
Do you think future conferences will be a blend of virtual and in person or only in person?
I hope they find a way to blend the physical and virtual together (and have some virtual only conferences as well). This will be a very interesting topic for experimentation, feedback and improvement.
discussions and those participating in them have been phenomenal : gather has come into its own 🙂
love the analytics on watch usage… can help hone in on insights as to problems the community is facing and looking to get more details on!
Will the Q&A from Slack be integrated into each video in the Video Library? It would be useful.
@nickeggleston that would be terrific. it’s in the backlog!
Anyone not in Slack is missing 1/2 the conference
...and those not using Gather Town are missing the other <large percentage>
I’m looking forward to being there today! Wasn’t able to join yesterday… and felt I was missing out. 😞
A good question, @davidorsi72 — we’re not quite sure yet. If we have time, I’d love to brainstorm about this with y’all. There’s so many interesting ideas to bring into IRL conferences next year.
I wrote about this in the morning - everyone seems to be thirsty for in-person more than last year https://www.linkedin.com/pulse/live-from-does-learning-devops-heroes-agile-covid-duena-blomstrom/?published=t&trackingId=lgwVDpA1RYWyTQ4L7QItiA%3D%3D
Happy to get involved in those discussions. I attend some regular meetups which would have been in person in Edinburgh. As it’s virtual, we get people from across the globe, so increases our diversity. They are trying to get their head round this too. I do miss meeting people in person. I attended DOES 18 in London and it was a game changer for me.
Also happy to get involved with this, since I am one of those who have found the virtual experience even more valuable than in-person. There's a risk that will be lost, without being very deliberate as in-person ramps up again.
A warm welcome this morning to @colas.a and @olimpia.nitti from Procter & Gamble 👏
I've recommended @mik’s talk last year on Flow Framework to many, many people through our org subscription 🙂
If you enjoy listening to Mik, make sure you catch his talk today on OKRs and Flow Metrics (12.05 BST), as well as checking out his podcast with incredible guests including Gene Kim, Brian Solis (Salesforce), Maya Liebman/Ross Clanton (American Airlines), Pieter Jordaan (TUI Group), Adrian Cockroft (Amazon), Dr. Nicole Forsgren, Don Renertsen and more, the list in endless...https://projecttoproduct.org/podcast/
I think the org sub is a great way to share the videos internally, as well as raising awareness of DOES and providing funding for growth.
Well my audible account is packed with tens/hundreds of books including all of itrevolution, Dr Goldratt... But didn't know this podcast, thanks!
"They really didn't go beyond the title of the book" - that's why mine are so darn long - some people never do
I’m so delighted that you’re presenting, @colas.a and @olimpia.nitti!! This is such a great story, bringing technology back to business units and brand!
For now, Track 2 is captionless if you prefer!
Captions are useful - use preference to turn on / off will be good one
Definitely automated captions, yet useful (spotted religious -> releases 🙂 )
@colas.a - how did you come up with the 70/30 split? Have you adjusted it over time?
@colas.a - how useful did you found the maturity survey to drive progress?
Love that teams are incentivized and declare themselves agile, rather than having it forced from above.
@bernard.voos -- It was not designed up front. It was where the cost ended. I expect central costs to go down over years and adoption of tools to go up so it will likely evolve towards 60/40 or 50/50.
@colas.a, would love to hear more about the training you put on for business leadership
What sort of topics were covered in the Business Leader training? I’m curious if you saw greater buy-in from them after this training and what they learned from the training that helped them ‘get’ agile/devops better.
@olivier.jacques -- The survey helped people get a better understanding of where they were. It opened their eyes to areas they were not considering and rebaselined where they were. Just for that it was helpful. On top, we created some elements of a competition. Nobody likes having a 1000 points when their colleagues have 4000
Can you share more about this in detail... such as the questions and answer sets and how you scored them? And how frequently and in what form you did follow-up...
It's the leaderboard nature of this that is the "Aha!" moment. It is always going to drive teams
Can you share bit more detail on your , welcome to your devops challenge, what does this look like. What outcome were u driving from this challenge. Thx
I love this glimpse behind the business of Pampers from @olimpia.nitti!
The business leadership training tried to demystify both Agile and DevOps and explain it in simple terms. We have used a few different approaches, including comparing IT to one of our manufacturing plants (I wonder where we got that inspiration 🙂).
Gary Gruver has a DevOps certification that uses this same analogy
Did you bring in any of @jonathansmart1's rebranding of Agile+Devops as "Better Value, Sooner, Safer, Happier"?
Love making the change while keeping the budget the same.
I often feel that money isn't the problem, it's how you waste time/money in a large org that is the problem.
I agree @matthew.cobby, and actually too much money can then be a barrier to improvement, because it allows us to continue the waste.
Interesting (old) book on transforming teams by replacing habits by other habits (and great portion of the book is about P&G) - "Power of Habit" https://www.amazon.fr/Power-Habit-What-Life-Business/dp/081298160X
Great content @colas.a and @olimpia.nitti, thanks for sharing the story. How are you planning to roll out Agile practices across all teams globally? Or has this already been achieved?
@olivier.jacques -- Culture is what we repeatedly do. To change culture we had to change what we do.
Very much our hypothesis in Agile Conversations.
If folks can tag book recommendations with the bookmark emoji 🔖 like I did above then we can pull those out of the logs afterwards and bundle them.
https://docs.google.com/document/d/17nHB3aDOsUI2f3W5EShyoFQiK7NMwIJ8QFkUCANVPbk/edit# And https://www.goodreads.com/list/show/162704.DevOps_Enterprise_Summit_2021_Reading_List Tries to collect most of them as well 🙂
Neils Pflaeging, Organise for Complexity (2014, 2020) 🔖 https://www.nielspflaeging.com/books/ https://www.amazon.co.uk/Organize-Complexity-High-Performance-Organization-Publishing/dp/0991537602 About half of it is in these slides https://www.slideshare.net/npflaeging/special-edition-paper-organize-for-complexity-part-iii
did you find demonstrable improvements in LT buy-in (for example providing presumably business PO's) after the LT training
@mpsibilskyte -- This is global already. We have over 120 teams in multiple Business Units and Central teams across almost every region
This is a huge win, congratulation on that! Are you pursuing a centralised Agile tooling approach or a more “let the teams choose what they work with”? And what’s your rationale behind it?
The answer is both. We have a number of tools managed centrally. In some areas, we even have several tools managed centrally as they meet slightly different use cases or needs. In some areas, we do not provide any central tool for now. Still, we foster collaboration within a Community of Practice so that we learn from each other. Eventually, some of them may become centrally managed if there is a scale advantage (but that is not mandatory).
I am curious if the 'scale workstream' (rebuilding legacy to microservices) and the ' innovation' workstream (build new features on target), impacted employee happiness/motivation in those streams?
@colas.a @olimpia.nitti this and how did you measure (if you did)?
@colas.a @olimpia.nitti thanks for sharing! Are you bringing together 'our business' and IT into multidisciplinary teams?
@colas.a @olimpia.nitti - "only" for web sites, or do you have other services where this transformation was rolled out?
@jonathansmart1 We are doing that more and more. It is not perfect but we are doing much better. We are now even getting 'business' to appoint the POs, instead of all of them coming from IT.
Was the training for business leadership curated by your teams? If so, I'd be curious to know what types of content you found most successful.
@david.sampimon we didn’t split the teams between old and new…the entire group wanted to get rid of the old and jump on the new
this was a big learning for me… i experimented with both and found the teams that owned both, time and time again, had much better outcomes!
@olivier.jacques -- This example is about our Pampers website but we have examples across the board: Supply Chain, HR, Sales...
@olimpia.nitti when you give the advice of "know what you want, know your KPIs and outcomes you are after" (which I agree with), were these made transparent to everyone across all layers and how the progress you made towards them was looking at all times?
Was the progress towards KPIs reported via OKRs?
Yes to both questions - all KPIs are shared within the full team - the team members have to know what’s being valued
many thanks Olimpia.
@olimpia.nitti How much of the solutions devising was done on ground level by the teams to help test what will deliver towards the targets? I am interested in the empowerment part especially.
“Change is optional — survival is not mandatory” 😂 I love that Deming quote.
fantastic story, congratulations @olimpia.nitti and @colas.a - what happens next? Are you done with transforming?
@adrian.jones The journey continues. We have not stopped in 183 years, no reason to do now
@olimpia.nitti You mentioned that your teams were energized by the challenge they faced. But I think it's hard to strike a good balance here and not overwhelm teams. Can you give any insights on how you made sure that the challenge is not "too much" for your teams?
@colas.a How did you bring the leadership trainings around? Did you curate these or did the respective teams curate their own leaders?
THANK YOU, @colas.a and @olimpia.nitti! Next up this morning, @jakob.knutsson657 and @daniel.claesson from H&M Group!
Statement from @olimpia.nitti - " Dont want only Product but experience " -- exactly everyone should work towards it.. thanks for sharing your experience
Absolutely brilliant transformation journey at P&G. Thank you for sharing @colas.a and @olimpia.nitti. It is interesting to hear that you kept the budget flat. I would like to hear more about where did you start and how was it planned ?
@christian.kullmann -- We have curated a number of central materials but then we have left the teams curate / adapt to their needs. At the same time, some teams created new materials that we then turned into global materials. Reapplication is rewarded!
And then you had them hold workshops and trainings in their respective departments with your support if necessary?
@andreas.baernthaler I didn’t impose the timing but asked the team to come back with a plan…so it was self paced
In my experience it is far too common to impose timelines from the outside. I think it's brilliant that you empowered the team to decide on their own. Thanks for sharing!
And now please welcome @jakob.knutsson657 and @daniel.claesson!
Definitely could use a small break in time between presentations to finish up and transition the chat... hard to focus on both at once.
There is a 5m break today, that wasn’t consumed by talks going late like yesterday. 🙂
and also scrolling through all the slack questions, answers & important pointers
Yes, while I like the -plenary channel, so I don't have to switch between channels, it would also be nice to have the Q&A in a channel per presentation (and/or maybe per author or group). I realize this is difficult.
Also, threading of the Q&A and queueing to make sure questions aren't missed is another place for improvement.
@colas.a @olimpia.nitti Thank you. Great to see the success journey as a co-P&G'er in this forum :)
I'm watching the videos on my phone, so I can go to the bathroom or kitchen as needed, but then I have to play catch-up on the Slack.
It's also hard for the speakers as the Q&A goes on for a while after their video ends, so they don't get a chance to be part of the audience for the video that follows.
That breaks the flow... I see that as more of a channel to ask questions later when something occurs. Threading is definitely an unsolved problem here.
“I am Fredrik” < that caught my attention more than is reasonable. 😄
P&G’s DevOps Journey with @colas.a & @olimpia.nitti is already live for sharing here! https://videolibrary.doesvirtual.com/?video=551641782
@jakob.knutsson657 - I would have taken two to 4 weeks until fairly recently.,.....its been known as months for us.....but we are getting better and its now days
2-4 weeks for onboarding new developer (hmm, how many of us very well connect with this statement), even after 4 weeks - still missing some permissions
we have an unofficial goal that a new developer should make his first
commit PR on the second working day.
Talk to me later @lbmkrishna. Also Dharmesh Gordhan back at BNZ.
This is a great topic for more discussion. Let us know where you continue... maybe we can get a new channel set up? @alex or a suggestion of a better way to greate topic groups that persist...
Great idea @nickeggleston - what should we call the channel?
I once wrote an internal paper on how much it cost to replace an engineer. A very conservative estimate was AU$100K, probably more but it was harder to quantify. It's more efficient to keep your hearts & minds than to get a new one, no matter how fast it is to prod.
@matthew.cobby - Few individuals like us we are still fighting for that "freedom" - we are yet to get liberate mate
I just love the background, the clothing rack in it's natural jungle environment 😂
@lbmkrishna This was one of my “favorite” parts in the Unicorn Project — engineers quitting before they were ever onboarded, contractors on-site for months, still unable to get access to things they need.
Still day-to-day business in some companies 😮
Since almost two years The Unicorn Project - The developer onboarding is still the same 🙂
@genek what real world examples were the inspiration for this? (redacting names as necessary)
"The number of deployments is a true competitive advantage". Nice!
A channel for engineers called “absolutelynoidea”!!! So freaking awesome! 😂
💯 creates psychological safety !! helps normalize "i dont know, but let's find out"
now that our Slack is for our entire enterprise, not just Tech, I wonder what kind of questions would get posted now :thinking_face:
#absolutelynoidea now live!
#absolutelynoidea now live!
Mhm curious how these bubbles were actually broken down into the teams @jakob.knutsson657? I've seen slides like that a lot and found that the actions after are much more exciting 🙂
“159 posts, 729 replies, 220+ mentions in #absolutelynoidea” — what a marvelous way to model a learning culture!!
#absolutelynoidea now live!
I find it takes a huge amount of time for people to get comfortable “working in public”, to ask questions and interact in a shared channel.
Exactly, I think it's worth striving for this kind of safety in a team.
I find it common for people to keep splintering off into private channels because “we don’t want to spam people”.
Agreed, I find myself thinking that way and not wanting to bother the large community with questions that may be obvious or already answered elsewhere.
A good setup for messaging applications helps a lot with that. For example in Slack: having specific channels for asking about help on a certain topic, having channels with very specific topics for discussion, and always encouraging people to use threads to keep the spam in the "main" channel low.
Love a shared space for knowledge sharing! Anyone have a bookclub at their orgs?
Awesome idea. Do you have one?
Yup. We started with Accelerate and now are currently reading Agile Conversations.
@becky.woof at the moment we've got as far as assemble the group (not many at the moment but think thats a good thing). Then to get everyone to nominate a book, a reason why we should look at it and then we vote on it. My preference probably would be for Accelerate, project to product or Unicorn project. So many to choose from now 🙂
We started with a similar process as @eazyd247 mentioned. We found we started with about 10% engagement from our department. I'd like to find ways of including more individuals from across our company in the future though.
have you had much issue keeping the momentum, people keeping up with the reading and the attendance?
We did notice that in the first one. I feel like that is somewhat to be expected. We did a retro exercise after that one to attempt to make things better for the current (second one).
We found that even if we had a small group, we could share our high-level findings more widely (on slack, or wider departmental syncs)- just bite-sized points. A few more people got involved that way!
#absolutelynoidea - How do you keep useful content floating to the top without drowning in new questions/noise?
Is this naturally promoted or is there a specific team/person behind that?
#absolutelynoidea now live!
We have appointed topic owners in the community, examples could be around API & Events or Cloud. They are monitoring the chat-channels and also update articles, patterns & practices. But then the community also step in and answers and make suggested changes based on their inputs.
@jakob.knutsson657 - how did you know that the culture had shifted in the right direction, especially among senior leaders?
definitely interested in this one too. Great question
I think when we could prove actual value was a first. But it's a long and gradual journey. Then more studies, science and success stories around devops / cloud also emerged. We also had some leaders believing in the "new" ideas.
How do assure people have slack to work on community topics?
We make it part of the Definition of done, to hsare the work with the community.
On what level? Each story? Which community? Topic specific?
On the idea of slack this is a great post about it https://fs.blog/2021/05/slack/ Many roles become less effective the more efficient their time is managed
It is not that I am not aware of this bu tin many cases the situation is pretty complex. I wrote a piece about it actually: https://rruzitschka.medium.com/the-challenge-of-internal-communities-in-an-enterprise-it-organization-fbe198b1f0e4
We have appointed topic owners in the community, examples could be around API & Events or Cloud. They are monitoring the chat-channels and also update articles, patterns & practices. But then the community also step in and answers and make suggested changes based on their inputs.
I love that quote from @jakob.knutsson657 about @daniel.claesson observing how productive it was when the merchandiser sat next to the developer, instead of being put between two departments. So good.
Like the naming of "Business Tech" to make it visible to everyone
Always surprised/shocked when discussing outside of the company, people telling me you're quite knowledgeable on the business, and you work in tech? Is it supposed to be mutually exclusive?
IMHO - more & more Enterprises still in this fix - anything and everything slow. Some end up to whitelist URLs/ IP address for developer tools to access required updates
@jakob.knutsson657 and @daniel.claesson you mentioned that Product teams were formed around unique business problems. Has that created any challenges on the same users needing to interact with multiple product teams?
@jakob.knutsson657 How do you balance aligned autonomy? Allowing teams to be fully autonomous in their ability to make decisions to get value VS following standards and keeping on the foundation you touched on?
DevOps was about bridging silos, collaborating at a larger scale. Continuing that trend is getting collaboration across the silos of business and technology. So H&M creating a “Business Tech” group seemed like the end state of where the ever expanding collaboration takes us. That make sense @vladyslav.ukis?
I love all these statements: “don’t be a service department,” “technology must be embedded into our DNA”.
+100 to "biz understands tech better and tech to understand biz better" @daniel.claesson
BTW, how you found the balance what to learn on both sides to raise that understanding to the next level?
Great talk. Thank you for sharing @daniel.claesson and @jakob.knutsson657
Gene referring this about Audit: https://www.youtube.com/watch?v=FyfLnuceQcU
I really enjoyed the auditor talks from that year. It helped me understand their perspective and partner better with Audit internally. We even initiated a value stream analysis to get audit more integrated and automated into the process.
“Oh, no. Not the auditors. What are THEY doing here?” 😂 “I hear they get paid by the finding” 😂. @lucasc5
the “paid by the finding” comment had me laughing out loud.
reminds me of the stories of testers being paid by the number of bugs they found… and developers by the number of bugs they closed.
Wait. There were some QA shops that were bonused on # of defects found, right? After all, if they don’t find anything, what are they doing all day? 🙂
Mythbusters for roles, might just be a good series of videos :thinking_face:
I’m not sure how many of you noticed, but I have a copy of Montgomery’s Auditing on the top shelf on my bookshelf. I definitely don’t recommend buying this, unless you are studying to become an auditor. 🙂 https://www.amazon.com/Montgomerys-Auditing-Vincent-M-OReilly/dp/0471140635/ref=sr_1_6?dchild=1&keywords=montgomery+auditing&qid=1621418506&sr=8-6
I won't put that on the reading list from the conference 😂
Then again, it might just be interesting to read anyway :thinking_face:
I noticed strategic placement of The Phoenix Project and The Unicorn just outside each shoulder 🙂
Well an auditor would say that, then you don't know they're getting paid by the finding 😂 I should say, I've never had an issue with an auditor, they've always been very helpful.
@david.read There are InfoSec or Compliance Audits as well
Good question, they look into so many things. Depends on the industry as well
We also run open source audits at Synopsys so it can really mean anything
Basically any part of dev or business that requires or comes with an external ruleset can be audited
Internal Auditors look into risks that could prevent an organization from achieving its objectives. Those can be financial risks, compliance risks, operational risks...
Tune into our break-out session tomorrow (DevOps and Audit: A Great Partnership Part 2). We'll briefly review Internal Audit's role in an organization
@lucasc5 - one thing that I hate - when we ask developer to fill and answer the risk control Excel spread sheet 🙂
@lbmkrishna - I'm cringing right now 😆 That's how we as auditors get a bad rep.
I noticed that on your shelf @genek101 - glad to see it's part of your regular reading 😉
It's in the office waiting for me when I return to in-person meetings :)
Turn the Ship Around has a great approach to this called, “Embrace the Inspectors”: > Embrace the inspectors turned out to be an incredibly powerful vehicle for learning. Whenever an inspection team was onboard, I would hear crew members saying things like, “I’ve been having a problem with this. What have you seen other ships do to solve it?” Most inspection teams found this attitude remarkable. As a result, Santa Fe was getting superior grades on inspections. https://davidmarquet.com/turn-the-ship-around-book/
i love this book for so many reasons, but had forgotten that topic was in there. one more reason to love it.
Yeah it was a real gem of a concept and I found it helped in a huge way with our most recent audit, which we PASSED by the way! After 2 failed audits, we were really proud of it.
I have to say my experience with auditors has generally also been good. They’ve also been generally open to new approaches as long as you can explain how the approach meets the outcomes/standards/regulations they’re checking against.
“Now let’s talk about your relationship with YOUR auditors.” @lewir7 🙂
I feel like this conversation will require a reclining couch. 😂
What's your view on Audits being the first step towards DevOps? A good audit can set goals and highlight gaps
Internal Audit can definitely shed light on areas that need improvement. Management can definitely leverage that as an opportunity to implement DevOps practices to address those gaps. Great idea!
@lucasc5 i respect you busting these myths but I have seen some of these happen unfortunately, And some of these issues, internal IT auditors on my team acknowledge. I really wish it was not like it though. I have been working with auditors for external audits for 3 years. And some of these myths were challenges. Thankfully we managed to work our way through by really trying to understand why is it that we are doing this audit thing. But then it made sense. What really helped me get it all was http://dearauditor.org/.
I found some of it to be a training issue with how they were inducted in the industry
When leading change, it's a good idea to ensure that internal or external audit are your best friend from day one! Auditors will either 'encourage' teams into better ways of working, or keep teams back in a theatre of control. Very few people argue with an audit point! Internal audit have been a huge lever for good in my experience
I do find a massive communication gap in this success though.
IT leaders and managers dont want to understand purpose of audits
Auditors for some reason also fail to explain this well. This is probably also not encouraged fully by the larger audit firms that larger companies end up working with
@lucasc5 @lewir7 My experience with auditors was absolutely positive so far. I am very interested in how GitOps is seen by auditors. I would love to get a chance to discuss about this topic with you.
Check out our break-out talk tomorrow (DevOps and Internal Audit: A Great Partnership Part 2). We may address your question in greater detail there. If not, I'd love to get more insight from you on how we can address it.
@genek is a Team Auditor superfan!
Oh, I saw that. Just thought you’d enjoy this view as well.
Audit is like anything else - shift it left! Get the audit team engaged at the start rather than the end.
Tune in to our break-out session tomorrow (DevOps and Internal Audit: A Great Partnership Part 2). We talk about some of the ways we're shifting left :)
Haha! And there’s a copy of the Institute of Internal Auditors Professional Practices Framework book among the red books. 🙂 (I should have hung my ISACA CISA certification up, too. 🙂
We anchor back to the IIA Standards/red book! :)
PS: I learned last year from @lucasc5 that there’s no actual physical “red book” anymore. 🙂 Thank you again so much for the talk — you, too, @lewir7
It was our pleasure! Thank you for giving us a platform.
Brilliant as usual @lucasc5 - much more relaxed than "Audit" finding interview 🙂
Thanks @lucasc5 and @lewir7! That was fantastic, going to show this to my auditor BFF’s! They will love it.
@andrew.sturrock Yes, totally agree. It falls into the same set as "we're not going to tell you what we're going to test otherwise we won't find any bugs"
@tim.wyatt - hearing that breaks my heart! We hate surprising our clients. If my client's are surprised at the end of an audit, I didn't do my job very well.
@genek @jeff.gallimore - we need more of this Audit / GRC "eye opening sessions"
Huh! I’m noticing that everyone are especially loving these “boundary spanning” talks?
I think it comes down to a similar effect like when you're having cross functional teams, more viewpoints mean overall better coverage
i think they resonate and help us story tell that DevOps isnt just a tech problem. I noticed a lot of questions/interest during the P&G talk about getting the Biz Leadership on board. I think these boundary spanning really help our brains firm things up
:thumbsup: Y’all are going to love some of the talks coming up, then! 🙂
Thank you @lucasc5 and @lewir7. I will try to keep that in mind the next time i talk to our auditors
@lucasc5 @lewir7 - The issues I see is that in the name of "Audit" - we over engineer things in an Enterprise (not by auditors though)
We see this a lot too. I visually cringe when someone tells me they're doing something because audit told them to. Either we didn't explain well enough why the gap was important, or it wasn't truly a risk. In the past few years, we added a scorecard metric for ourselves to encourage us to identify cost savings and efficiency opportunities. That helps with some of the overengineering too.
Coming up in a few minutes: @leanne.bridges and @mark.rendell from Nationwide Building Society!
I think a key point to think about with audits is remembering why your doing them? If they provide some problem areas and you don't act on them what the point of doing it the first place. Its like everything you need to view it as the beginning of the process not the end lets stop doing checkbox exercises
Start with why as would S. Sinek say (but not only), you'll get more buy in when people understand why they're asked to do things, and knowing it from start rather than as a gate at the end, would certainly to better results and less reworks and better image of the audit. Looks quite similar to security which can either be seen as enabler and bringing value or just as a blocker breaking plannings
100% I and many others can't invest in doing something if we don't understand why or we're doing it. Filling in a spreadsheet because Legal say so isn't good enough
Security in DevOps is to prevent problems down the line no developer likes being called in to help fix P1 issues or similar but by following good security practices they could prevent future pain thats worth a little effort now to avoid future pain
One other common issue between Engineers and Auditor - The language / terms that we use
Most auditors make an attempt to use neutral language but legalize and developer shorthand are problems for both groups
TLAs don't help have you ever counted how many acronyms you use in your org/team daily/weekly
Too short tbh. I would have liked 10 mins better.
The relationship of engineering teams with audits and regulatory requirements definitely is a thorny one. It is not so easy sometimes to convince teams that these things are necessary - otherwise you will be out of business. On the other hand it needs to be understood that implementations of regulatory requirements can be done in many cases in way that they don't impede flow. This can be achieved with close cooperation between risk/auditing and engineering teams. But my experience is that in many cases engineers just don't want to be bothered.
and the teams are bothered too much by cumbersome tooling that must be used to fulfil the regulations
My view is that its also important to understand why the regulation exist look at the UK and USA talking about cyber crime and IoT security its being discussed and talked about because its a problem. I'd always suggest people use regulation as the organization stick to implement good changes. Don't worry about the exact specific but make sure your doing what right for your team/product
Fully agree to this @robertso: Why is always super important. My main issue was that in my experience engineers embrace freedom and hate regulation like the plague.
@vladyslav.ukis Being able to satisfy regulatory requirements is a reason as good as it gets. It is non negotiable. But of course it needs to be explained.
Regulation very rarely wants to limit innovation or freedom but encourage best practices. Regulations can be interpreted in a million different ways and I've worked with orgs who take the very strict approach vs the more open model. Its about understand what problem the regulation is trying to solve look at the key recommendations and implement those how they make the most sense for you.
@annp Hi. I answerd in our other channel about meeting with Mik. QUESTION: What’s the link to watch live?
Hi Gene. Am I in the right place. I think so. Intelligent control guy right before Richardson?
And speaking of audit and compliance, here’s another talk on achieving compliance objectives from @leanne.bridges and @mark.rendell (who apparently now works with compliance people all the time. 🙂
H&M Group – Integrating Business and Tech with @daniel.claesson & @jakob.knutsson657 is live in the Video Library! https://videolibrary.doesvirtual.com/?video=551641791 From Your Auditor Friends: What We Wish Every Technology Leader Knew with @lucasc5 & @lewir7 is live in the Video Library! https://videolibrary.doesvirtual.com/?video=551641804
@leanne.bridges - helping colleagues build good things ❤️
the do, but naive creatives temperamentally seem to dislike them
For me it's the critical thing @richard.james - if we don't work together, it's almost anti-value
You've already had a couple of absolutely GOLD quotes in here mate - I'm going back through and stealing shamelessly to use "back at the ranch" on posters with your face on 🙂
I still like Mark Schwartz language around 'enabling' rather than 'corrosive' controls - this is the agenda for me 🙂
indeed. his perspective that a CI/CD pipeline is simply an “automated bureaucracy” (full of rules and controls) blew my mind.
Totally that - we are using this language/phraseology directly with @mark.rendell and @leanne.bridges here - 'enabling and embedded controls'
'automated bureaucracy' is worse than 'manual bureaucracy' if a control doesn't add value, we shouldn't operate it at all..
Safe autonomy is something I've seen more and more :thinking_face:
"accountable freedom" - love it. Good morning from New York 😄
Good Morning @katharine.chajka... we're aiming to find the balance for 'safety + agility'. We've learned that fewer, easier to understand parameters enable folk to work with autonomy
It's actually the name of Leanne, Mark and my home Community back at Nationwide - Resilience & Agility
I loved this from @mark.rendell - when he shared the idea of "agile adolescents", I hung my head in shame and self-recognition ❤️ 😄
“DevOps people are like teenagers; love to challenge status quo; but they have live services, and guardians have thankless task. They’re morally and legally responsible, but need to enable autonomy.” So good, @mark.rendell!
I feel seen :rolling_on_the_floor_laughing: Very much like a teen sometimes in what I want to do...
Organize like an engineer; lead like a parent
I mean the most common whiteboard guard rail is the note to check if you're using permanent markers 😂
Made that mistake at a public sector client. Visited again a few months later, scribbles were still on the board 😊
In that case @jonathansmart1 was probably a good thing. If you once come at our place, I'll make sure you have only permanent markers 😉
The "traceability for those accountable" bit is so crucial! Big thing to consider when creating boundaries
I think it's critical. We apply a bit of a 'forensic' approach to understand how we'd review the outcome before we build it.
We're in the process of learning this the hard way, lots of policies got put in place without any effort into traceability, they are definitely creating pain down the line now
I feel your pain @philipp.boeschen650 sometimes policies are created like sticking plasters... if you've too much ambiguity from the policy layer, the control tier becomes un-workable. Reach out if you wanted to explore further on that one..
In all seriousness, I love how auditors (and lawyers) talk about “reliance” — http://i.e.inare reliant upon a given control (from “not at all” to “entirely reliant — if that control fails, you are actually sunk.“)
This was one of the most illuminating learnings from audit for me. cc @lucasc5 @leanne.bridges @lewir7 @mark.rendell
It's especially important when things go really wrong... it's also usually where you end up with 'over control' (often in reaction to a previous 'really bad' situation).
That's a tough one. I don't want teams doing unnecessary, over control activities. However, being able to find and rely upon a backup control when the primary one fails has saved us in an external audit. Layers of control, defense in depth, can be nice to have.
I'm glad that you've seen the value of this @andy.hinton - it's solid learning when you go through those processes and realised the value of what might otherwise be seen as burdensome
Today I learned that I'm a guardian of a teenager. Suddenly, my interactions with our Engineers make so much more sense. 😆
Shouldn't risk be assessed right up front, when one knows the least about what one is building? :rolling_on_the_floor_laughing:
Someone (@jonathansmart1) has a chapter in a well-loved book dedicated to that particularly anti-pattern 😄
I know you posted this in jest, but risk should be assessed up front and should be updated as you learn more.
we continuously adjust the risk profile created early on, the the product evolves, the risk evolves and we can apply controls proportionate to that point in time..
"Fully Informed up Front. Build the right thing the right way" - love it
it makes the world of difference... you know what you're getting into before you get going.
@mark.rendell @leanne.bridges I'm a colleague of @jonathansmart1 your Intelligent Control presentation is brilliant. May I get a copy? cheers, John
(wanting a copy -- I can't claim to be a colleague of @jonathansmart1)
They’ll also be in the Video Library — @alex will be posting it shortly.
@jlpryan @nickeggleston https://videolibrary.doesvirtual.com/?video=550704454
"build the right thing and build the thing right" - control stories and understanding why you needed them in the first place - "why" is so often lost!
You can really see the whole traceability thing pulled through the entire process now 🤯 So good
if there's value in doing this work it's to enable traceability; creating the data that tells the story of how and why the product was built makes a huge difference when you're trying to provide any kind of assurance..
(I’m not quite sure I understand @leanne.bridges’s motivation of doing all this — doesn’t she get paid less if the product teams get fewer findings? 🙂
😂 Maybe some informed executive could find out she makes more value for the business and the product team less miserable? Hypothesis... 🙂
Hi @genek101 and @christian.lefevre - I always say that I need to 'do myself out of a job'. When an organisation has achieved 'control by design' and operates continuous assurance, you don't need as many people like me telling you whether you need to worry - because you understand the risk inherent in your own work... So weirdly, my personal objective is to enable lasting change that supports continuous improvement for an organisation.... but I'm a problem solver so guessing that's where this becomes rewarding for me..
Oh Thoughtspot, interesting haven't seen that around a lot! How's that working for you @leanne.bridges?
it works really well. We wanted to ensure we were 'system agnostic' and flexible in terms of the data we could pool. Thoughtspot enables us that 'layer' that we can adjust based on the depth of data available to inform that 'real-time' view
Think Tableau with a google search like engine for your data underneath, it's some pretty interesting tech https://www.thoughtspot.com/
We use ThoughtSpot in a number of ways to support our BVSSH journey - @zsolt.berend and @marc.price can give the 'inside track' on how we leverage to support measurement/visualisation of flow, golden thread and embedded control 🙂
Oh interesting! What's BVSSH tho :thinking_face:
Hehe another thing to note for future investigation
Bring Value Sooner Safer Happier - Super book by @jonathansmart1
Right, yeah 🙂 Heard of it it's still on my shelf to be read... so many books so little time
T/S provides a rich front-end, it is great for self-serve data, insights consumption
Have you seen benefit in the self service aspect, do people actually actively explore the dataset? We never got to a big enough PoC to find data for that
yes, they do - we treated our dashboard as a product from the beginning. Start-up way, started small with some innovators, early adaptors as customers, establishing feedback loop. Now we have 700+ active customers, colleagues at all level, all based on pull.
it has been a journey to unlocking measurability and through conversations unlearning fear of data, and experts providing reports to measure for learning behavioural
That is super cool! Did you face any problems getting your data into the right shape underneath? Thanks for answering these! 🙂
yes, we did, we pull data from jira and snow and e1 - and cont. improving the data models for more flexible insights creations. However it has also been the goal to keep it as agnostic to ways of working as possible so e.g. there is no mandate on using a set of states (it would be an antipattern as context is unique so flow of work will differ) so we made lead time calcs. e.g. agnostic
Something of note here; we’re in the process of migrating to dashboards for our Data and Analytics Value Streams. The Intent is that this will supersede our need to ‘report’ on risk and move toward use of real data to inform the risk positio…
more than happy to give a view of Thoughtspot if anyone would be interested 🙂
What you can see btw is the absolute chance (need) to increase capability in the organisation - look at the techniques, tools and principles that underpin the integrated, enabling and 'always on' controls ❤️
Yes @tim.bassett and Metrics and Insights are a key part of our overall Ways of Working enabling function //cc @berendzs
partially - where a control is automated or semi-automated, we can enable dynamic data.
Hi @vladyslav.ukis sometimes a control is manual (i.e. that decision needs to be made at that tier of committee to ensure that it's within governance' so you end recording that the event happened and logging associated documents to support the execution of the control. That help?
“this is helpful when you have legal proceedings.” (Everyone stops laughing.)
those are the days when you're really grateful for how you made records of the build....
@leanne.bridges I love how this is driving more collaboration and engagement
HEy @berendzs - glad you're here... @philipp.boeschen650 had a Q on ThoughtSpot... you'll be able to enrich that thread with your mega-knowledge..
Also @tim.bassett was interested too... maybe we need a session on data-centric controls assurance informing risk??
yes - our problem is that we want to automate "stuff" but people forget about dashboards.....we have agility metrics under control (sort of - v1 at least), however we are now gathering metrics for other "stuff" - but manually. Always interesting to see how others do it...
one of "my things" is reporting/data should fall out of the process and not be additional work....
Yeah we were at a similar point when we played with it, but the price point was just too high for us to get enough ROI
The whole, metrics should just fall out, that seems to be hard to create though, is there something special you're doing towards data schemas or when collecting data to get it more "in line"?
Hi... this bit is the bit that is often overlooked... I say (almost everyday) that we have to build the data that enables us to measure. The key here is JIRA because we can get a status for manual, semi-automated or automated controls as a data point that we can add to dashboards. This is how we're able to then streamline the assurance (if we don't enable this data creation, we end up doing the work twice)
"I say (almost everyday) that we have to build the data that enables us to measure." I think that's the key thing that needs to be driven from the top, I've unfortunately been in some positions were we never got time or resources to build proper data foundations and then got pushed to deliver, deliver, deliver which never worked out as people thought...
@philipp.boeschen650 -a large part of my career was spent in the Public Sector focussed on maximising value for the money we invested. One of my personal aims is to achieve as much of this as possible with existing enterprise licensed tooling minimising investment. Much of our 'build' is driven through internal enthusiasm rather than investment and we're currently experimenting with Power BI as it's within our Teams licensing and is widely available. This should cost less than the alternative.
@tim.bassett ref excel.... we're experimenting with Power BI/Power Platform in Teams for increased availability..
Yeah I do think sometimes, especially we as engineers, tend to want to re-develop rather than using existing stuff - Trying to get ahead of that problem seems like a very good use of time! 💯
Hey @berendzs - a lot of love for the Thoughtspot work from yourself and @marc.price - deffo a topic for the next DOES 😄
more than happy to @richard.james v
Huge credit to @leanne.bridges with their show and tells. The open invitations to those regularly attract a very diverse and senior group of people - and not just to observe - loads of ideas flowing :)
the best thing about those @mark.rendell is how much we get out of it too 🙂
"pulling the work into one place [JIRA stories] improved visibility [of control work]". Work is where the work is, rather than in 20 spreadsheets.
"invite over inflict" from Intelligent Control - that is the future stance of compliance and control
we align risk people to the work.. once the risk assessment is complete and thereafter continuous, we align a risk partner to be the persistent and primary point of contact... They'll bring in any additional specialist risk support to ensure that the depth and breadth of risk is understood and the right control stories are surfaced
yes on the empowerment - take the time to support, can't flip a switch!
one of our absolute favourite adopted phrases from @jonathansmart1 ❤️
used regularly by our Chief Operating Officer, as well as across our WoW community and colleagues 🙂
this is the best example of BVSSH i’ve seen @leanne.bridges! definitely making my gears turn on how to double down with my team
In addition to the great work done at Nationwide, see chapter 6 in Sooner Safer Happier, a chapter just on this topic.
That's great to hear - thankyou. I'd be happy to share more detail on anything specific..
"Empowerment can feel threatening". Yes, I've seen this, where people are given more empowerment and yet don't take it.
@christian.lefevre Yes. A need to intentionally nurture psychological safety. As per Amy Edmondson, (1) Set the stage (2) Invite participation (3) Respond positively
Thanks @jonathansmart1 I’m considering a dojo as well to help people train and gain confidence too, but shouldn’t substitue to demonstrating trust by handing the keys of the real world too I guess.
Yes, sounds good. And there needs to be safety to experiment and fail. If not, even with new knowledge, there will be no empowerment taken
I've quite often seen 'learned helplessness' with everyone shrugging their shoulders, pointing at someone or something else
I like the idea of L David Marquet in turn the ship around, of having people declare intention and validating, I would guess such endorsement will reinforce confidence in taking the empowerment, which should transition to a stronger ownership tranforming the declaration of intention to informing. (kinda removing the small wheels to the bicycle)
@leanne.bridges - this reminds me of the idea of “Enabling Bureaucracy” from @schmark’s book, “The (delicate) Art of Bureaucracy” 😉 https://itrevolution.com/delicate-art-of-bureaucracy/
Totally this - we use Mark's language around "enabling bureaucracy" rather than "corrosive bureaucracy" in our context ❤️
This part - "Intelligent Control" can have 'unintended consequences' of causing alienation for colleagues and existing work
“In some people’s heads, we were calling their controls, or worse, them, unintelligent.” 😆 @mark.rendell
Personally, I'm very wary of anything that is described as 'intelligent' - it's usually to do with hubris of the inventor
The talk about replacing the word intelligent with control reminds me of this chrome plugin I've used in the past https://chrome.google.com/webstore/detail/ai-just-some-if-statement/ihdinhfmngbefhhajfjankbpphnflfbd?hl=en
See also: SmartTVs (sigh), Smart Meters (sigh), Smart Home (😬 ), Smart Cities (🙈 )
Perhaps this is crazy, but I wish we had an "andon cord" to stop the video presentation while the discussion is ongoing on Slack... and then release the video to proceed when the discussion on the current point dies down.
@leanne.bridges @mark.rendell How do you overcome the challenge to have a risk expert available when it is needed. How does the Risk CoE facilitate availability of Risk experts on the moment the Agile teams need them
I answered up there but will repeat here: e align risk people to the work.. once the risk assessment is complete and thereafter continuous, we align a risk partner to be the persistent and primary point of contact... They'll bring in any additional specialist risk support to ensure that the depth and breadth of risk is understood and the right control stories are surfaced
Sounds a lot like the adjust support teams @jonathansmart1 talked about in his videos
@nickeggleston Safety Teams. Multidisciplinary GRC type SMEs aligned to value stream, long lived. e.g. InfoSec, Data Privacy, Compliance, Fraud, Anti Money Laundering, etc.
Agree @nickeggleston and as @jonathansmart1 highlights - having the right group of people, with the right capabilities makes the world of difference to how we build things
We have been doing things around OPA for validating e.g. container images are safe to release to production
the primary stuff right now is about validating that dependencies you put in approved/not marked as risk
That's a great topic @gus.paul what we're doing with Intelligent Control is partly about tying together smart technology solutions like that with the formal official controls.
Inner Source for your controls... have you had many contributions to those controls and are they for engineers or risk?
That's a hot topic and work in progress @matthew.cobby Just last week the Data and Analytics community got Inner Source access so that they can make contributions!
Interesting.... I'd like to follow up and see how you are going. We have an Inner Source programme and one of our products is our compliance as code. It's been a sleeper so far but we have ambitions
There is a fundamental "industry" problem where everyone needs their "bit" in the DevOps portmanteau, how does that get solved?
@jonathansmart1 references the 'narrow' (Tech) or 'broad' (Org) definitions of DevOps - we talk increasingly about how we are bringing Change and Run together as the broader agenda 🙂
I need to read the book! I think the problem is that departments that are not mentioned are not included, so there needs to be a way of describing that inclusivity
my reference to the 'must do and the should do' is because that helps you work out who 'must' be in the room and who doesn't really need to be there... The other challenge though is having a trust environment where folk 'not in the room' are ok with that and trust their colleagues to bring them in if they need to be there.
empathy and shared goals.... has anyone found a way to ctrl+c ctrl+v this everywhere? :rolling_on_the_floor_laughing:
we create something based on our best understanding in the moment. That understanding has to be imperfect. Waht we’re trying to do is get feedback early and often about what we didn’t understand so we can correct and improve.
definitely - by having a 'critical friend' in the room to answer the thorny questions - can help you go faster..
Theme coming out in many of the talks - by focusing on people and empowering people we save TIME
Thank you, Leanne and Mark! We are honored to welcome @jmrichardson1, also known as Admiral John Richardson, who will be presenting next!
I think we need a framing around the Risk, not around the Control. The Risk is the outcome; the Control is just an output along the way.
Yes. The formal Controls are context sensitive depending on the risk profile and risk appetite
Not one size fits all set of Controls (lowest common denominator) as is usually the case.
Hey @rshoup - do you mean the 'wrapper' for the work or through the work?? We use the outcome of the URA to provide a baseline risk profile and define 'why' controls are required and to what extent they're mandatory. It helps us prioritise the work in that way but gives us a persistent line of sight to 'what happens' if we don't invest in that specific control activity, how does that affect the risk profile... We didn't dive into that on this session but I'd be happy to explore further with you.
I totally agree @rshoup but there is also often a significant volume of established controls processes that you need to meet where they currently are.
@leanne.bridges Sorry, I meant that we need to frame the problem in terms of the Risk, not in terms of the Control. Reducing the Risk is the outcome / goal / business value. A particular Control is one candidate way to achieve it, but there may be others, as you pointed out. The Controls are not themselves valuable! I've had so many conversations with auditors that take this path. "Tell me your process and prove you are following it", they say. And I try to reframe the conversation around the Risk itself and how we can reduce that.
Your model sounds great. I'll be sharing it with our internal auditors 🙂.
Glad to have helped…. The experience you mention though makes me think of human-robots… we have to challenge processes and controls to ensure that they add value (otherwise audit is valueless)
This was really insightful @leanne.bridges @mark.rendell thanks so much!
To @ffion comment> It has to be about the people. Their collective intelligence is the source of the systems we create and it’s their needs that systems exist to satisfy.
It's that systems thinking that allows us to release our teams from a blame dynamic and fix the system collectively
Thanks @leanne.bridges and @mark.rendell, also going to share this with my risk management peers!
Up next: Admiral John Richardson, former Chief of Naval Operations, retired. @jmrichardson1!
Really enjoyed the Idealcast episodes with @jmrichardson1!
It was an incredible experience recording 4 hours of interviews with @jmrichardson1 with @steve773! https://itrevolution.com/the-idealcast-episode-15/
Did you get into a discussion about Gen4 reactors that are being talked about being manufactured in a plant?
We did not get into some of these important technologies...subject of another talk!
i feel like until DevOps revolution came along I was under water too 😉
That is a personal goal right there: spend a couple of years underwater (but then from a leisure perspective :)
11 years underwater - things you can look forward to if you bought crypto in the last few months 🙂
haha which channel is for crypto discussions? :rolling_on_the_floor_laughing:
“push ownership out to the furthest capable edge”
i love the inclusion of “capable”. that’s important.
these same principles are reflected in the talk from david silverman at last year’s summit… https://videolibrary.doesvirtual.com/?video=432219018
team comes back stronger. Thinking of how this is recognized. :thinking_face:
As for recognition, there are a host of both individual and unit/team awards that we used to recognize coming back stronger
Holy cow, @jmrichardson1 — it occurred to me after we recorded your talk, that the Officer of the Deck had to formulate the response, not the ship commander. In general, how junior can these OODs be?
Expertise is an important element of delegation. Often overlooked.
These are young officers - maybe 27-28 years old...their teams on watch average 20 years old
If you want to be competitive, you must deliberate about developing leaders
“if you want to be competitive, if you want to seize every fleeting opportunity that comes your way, you must think about how you develop your leaders” — @jmrichardson1
Puts the jobs ads which say must have 15+ years of DevOps experience in perspective and to shame
Startling — 28 yo officer can be responsible for $1B destroyer, and we require 15 years of experience to write deployment automation scripts.
Highlights organizations where the hiring manager didn't write the ad to me or worse organizations incapable of learning from and training new talent
Framework - what are the attributes that our leaders have (different for every org) - dependent on mission of the org - goal that your system has to be tuned for
Tuning the system for building leaders, that’s excellent.
"leadership factory" - I'm not sure we've something like that in Germany :)
Framing around being a "leadership factory" is so unique though, who else thinks of themselves that way?
the only other time i’ve heard this term was another US Navy reference — the book Turn the Ship Around from David Marquet.
Yeah and maybe many more should but it's categorically problematic in most places
This sounds a lot like the ‘Technical Maestro’ profile you raised in your interview with Ron Westrum, @genek.
2nd - character and integrity - acting consistent with the values of the organiation
What happens when confidence exceeds necessary competence?
Character and integrity - what are some of the recommended ways to work on identifying gaps and developing these?
3rd - connections - sense of confidence that if those leaders ran into a situation that they would not hesitate to reach back to get more support
@GeneKim is the Admiral here in the chat with us? If not where can we ask him questions?
Really similar to the Elements of Trust - Motivation, Competence, Reliability. But I love the first-order focus on Connections. We mostly miss that human element.
Monitoring for integrity and character - it takes positive energy and assessment by senior leaders, and close monitoring of the “conversations” amongst the team. The key is taking every opportunity to provide positive guidance in this regard, otherwise you’ll be dominated by the bad news..
Such a powerful concept! I was wondering how you conduct these assessments and monitoring. Any tips?
The only way I found is to engage with the third element - connections! This is very hard to assess from a distance - leaders at every level need to be connected personally to their leaders and teams to get the best sense. This seems to me to be the most important thing...once the rights people are in place, the rest is providing good guidance and sharing ownership!
Reminds me of the three groups in 'Turn the Ship Around' - Competence, Clarity and Control
@genek101 I loved the IdealCast episodes you did with Admiral Richardson, and also the Team of Teams people. It's interesting how the military has adapted to this empowered teams approach faster than the Enterprise!!!
@jmrichardson1: Dr. @ronwestrum recently introduced me to Rabinow’s Law: “if you have a dope at the top, you will have, or soon will have, dopes all the way down.” This struck me as hilarious and quite profound — does that resonate with your own experiences?
Almost sounds like a corollary to the Peter Principle!
A players hire A players; B players hire C players. Then I modify it to say "hire and retain"/
😂 Yes, @rshoup, I totally think this is a corollary of Rabinow’s Law.
Chapter 5 has the Naval Reactor (Admiral Rickover) case to which Gene referred.
I found that every ship took on the characteristics of it’s commanding officer. It was uncanny...
"attributes tuned to your organization", appropriately connected - push ownership out to the most capable and furthest edge of the org
Yes, that was exactly what the “dope at the top” principle reminded me of — @steve773 and I have been discussed how it illustrates why leadership is so important.
It makes sense to me why in a military context, avoiding asking for support can cause catastrophic problems. Makes me wonder why I hesitate to ask for help myself when I need it.
How cool did this invitation sound?
I had a really terrific time speaking with you today about leader development. It was a real honor to be part of this discussion, and I'm grateful to Gene for including me. I intend to continue to develop and sharpen the ideas I touched on to make them more useful to leaders in business. If you'd like to participate in that, it's super easy and I'd be privileged to include you. Just send an email to <mailto:BrinyDeep@SendYourSlides.com|<mailto:BrinyDeep@SendYourSlides.com>> with the subject "leaders" — you'll get an automated response that will loop you in and give you access to a few products that I put together in the Navy.
Thank you so much, @jmrichardson1!! And now, we welcome @mik talking about OKRs and DevOps!
Very very cook talk, thanks
leadership is so, so, SO important. thanks for your insights!
I’ve learned so much from @jmrichardson1 in every interaction I’ve had with him — I encourage you to take advantage of this fleeting opportunity! 🙂 Thank you, @jmrichardson1!!! (and thank you to @steve773 who introduced me to him!!)
Steve and Gene - thank you so much for bringing me into this amazing forum!
Thanks @jmrichardson1 as an ex Royal Navy person your stories just now and with Gene on the podcast really resounded. My own research over the past year on Agile brought me to 'oh, enterprises need leadership 1.01', so it's great that there feels like a zeitgeist around leadership in the industry now.
Totally agree! It was always a learning experience when I had the honor of working together with the RN!
And speaking of leadership! Here’s @mik talking about how leaders screw up the OKR planning process!! 🙂
@jmrichardson1 The failure patterns here will be about the opposite of radical delegation…
(In what should be the most startling contrast of great vs. disastrous leadership: here’s @mik! Wait, that probably didn’t come out quite right. 🙂 “How CEO’s screw up OKRs in their org in surprising and tragic ways and how to fix it.”
I sat in on this recording and @mik did this in one take! He’s such a pro. Though I do miss selecting his walk-on music. 😉
Objectives - the what - value, KRS - benchmarks for how. 3-5 OKRs, cascade
It's impressive this is the first time that incident was mentioned, expected it to be much more prominent 😂
https://en.wikipedia.org/wiki/2021_Suez_Canal_obstruction It's a fun one to catch up on
Leadership Development in the U.S. Navy from Admiral John Richardson is live in the Video Library for sharing: https://videolibrary.doesvirtual.com/?video=550704528
Turns out @genek is the source of that quote even though I tried for quite a while to attribute it to Goldratt!
Trivia fact: @mik told me he couldn’t find the quote “any improvement not made at the bottleneck is an illusion” in The Goal. I laughed at him, told him to look again. And then I looked, and holy cow, I couldn’t find it, either. (I think it came from Beyond The Goal, but there’s no easy way to search 7 hours of audio lecture in a hurry. 🙂
Looks like it is from Beyond: https://books.google.com/books?id=Gax-DwAAQBAJ&pg=PT52&lpg=PT52&dq=%22any+improvement+not+made+at+the+bottleneck+is+an+illusion%22&source=bl&ots=WbkoXy2Gif&sig=ACfU3U3lyvvduU0Fp2aiEJO-VEJFjXshHA&hl=en&sa=X&ved=2ahUKEwiT95fqzdXwAhWBVc0KHRuhAAoQ6AEwAnoECAIQAw
KPIs are health metrics. help you set guardrails
KPIs are equivalent to health indicators (e.g. heartrate) KRs are movement (e.g. we've travelled 5 miles from point A in 1 hour)
And orgs will tend to have many more KPIs than OKRs, the KPIs are longer lived and less related to a planning window, etc.
velocity and acceleration are both KPIs. you could have key results for each (increase our velocity by 10%, increase acceleration by 5%)
Lead Time 85th percentile is a KPI Reducing the 85th percentile LT by 10% is a KR
So maybe something like “our outcome is to win the next race”. To achieve this we will need to improve the performance, including “increase our top end velocity by 10%” “increase acceleration by 5%” “decrease pit stop time by 3%” < helpful @robertso?
There may be loads of KPIs There should not be loads of KRs (limit WIP)
measuring KPIs help you track performance vs OKRs and ensure you hit your OKRs
KPIs show your current status, health. Key Results shine a light on improvement (leading and lagging)
Some KRs are likely NOT existing KPIs (especially leading KRs, e.g. clicks on online advertising as a leading indicator of future sales)
and how would we even know whether our OKRs are slowing us down or not if we're not measuring!
is this evidence of scenius - having trouble attributing a quote or idea to a single person?
Interesting.... could well be. And if we accept it we stop being so famished for exact attribution and more open to collaboration
This was the biggest realization to me, that OKRs were being used to snap teams back into waterfall planning.
I think even when it hasn't happened, teams fear this. I want to ask "who hurt you?"
This was quite the aha moment — team roadmaps shouldn’t be part of the OKRs: they’re different axes in the planning process. Otherwise, OKR results in “where’s my feature?” 😆 @mail832 also talks about this tomorrow.
Yes absolutely, having both Product Roadmaps and OKRs.
I especially love when the roadmap shows outcomes and not "things" - like "make it easier for users to find the most relevant pages first time" rather than "improve relevancy search algorithm" or whatever
"Where is my feature" - Never have I heard, where is my debt item 😢
To the business debt items don't have value. Changing that perception is important. But i agree. Ive never heard it either
Yeah we're having that discussion inside of our orgs right now, at some point they become so big and everyone sees the symptoms of never dealing with the debt, suddenly it becomes a thing to talk about 👀
I like the term 'revenue-protecting work' (debt) and 'revenue-generating work' (features) - I think it was @dominica that I first heard use these terms.
That is actually super good framing! Def gonna steal that one!
add security risk work to protecting revenue. Seeing how much risk work you're doing (or not doing) is important too
Agreed - something teams at my company are just starting to do, wasn't previously something we were doing explicitly. You'll be pleased to hear that it's the adoption of Tasktop Viz that is driving this change! 😀
that's great @sophie.weston129. Making work visible and seeing improvements is a great motivator for change
everyone wants more features more quickly, pushed down on top of the value stream, capacity being overloaded.
@philipp.boeschen650 That is a key point, and why only the feature icons are green in this slide and why they’re the ones that contribute most to WIP.
@mik - have you worked with any companies in the past year that had to completely restructure their flow as their business shifted from retail to direct to consumer?
would you have OKR only on Value Stream level or would you do this is as well on a lower level?
@mik describing how OKR process can elevate/escalate system level bottlenecks. Very cool. (as opposed to “where’s my feature?“)
By improving transparency on capacity and the importance of aligned priorities, we no longer get questions of can you please do this, but now frequently get questions like can you not just add some capacity.. as if developers are a commodity you can scale up and down at any time. Anyone else struggling with that?
and they will deliver directly more outcome. Yes we have this discussion as well
The question reminds me of the discussion in The Mythical Man Month by Brooks
organizational OKRs reminds me of the "from your auditor friends" talk this morning - making it visible to encourage that collaboration and delegation
@bernard.voos Yes! And some did that extremely effectively. I saw OKR hoirzons shorten to mothly or very 2 weeks in some cases at start of pandemic and drive very fast and positive change.
“business results are LAGGING measures.” <--- obvious, but I’m not sure if I fully understood and internalized until @mik pointed this out.
I really like how the leading vs lagging vs team metrics have been called out in the presentation
@mik - thanks! I would love to hear more, because this is an area I (and my teams) struggle with.
@bernard.voos Hit me up in the #project-to-product channel and I can elaborate…
improvement of daily work in support of the OKR - rather than being pushed as the lowest priority...
Such a nice example of how a budget constraint was removed by using OKR as a vehicle to describe importance of what this team did.
I’ve seen this happen repeatedly, the hosting cost bubble reduced via tech debt reduction. Even though it’s an overly-simplistic case for tech debt investment, definitely a compelling one.
“Goal: zero days wait state on business validation of features” 🎉
@mik in that policy vlaue stream example, I am confused by the flow KR. whenever I have done OKRs, we have tried to keep KRs very directly related to the objective. How does that KR gets us to say that the objective has been achieved? also pointing to that we can increase the flow but deliver features that customers dont love 😕
0 days wait state on business input - fits the really audacious goal criteria of OKRs!
I always viewed NPS as a leading metric for income/profit. So interesting to see it framed as lagging to flow metrics
@kapoor.vaidik Those are two really key questions, that’s why the two KRs complement each other. One about how to delivery more features (flow) the other about whether those features are making a difference (NPS).
okay. its like deployment frequency and change fail rate
we have struggled with high level OKRs in our org though. often an OKR like that has not translated very well in shared udnerstanding amongst teams. we are not sure why. someitmes we attribute that to lack of maturity in relatively younger teams and middle management
i was wondering if you have also observed this. not specifically the cause but the effect of OKRs not being well udnerstood in teams?
we usually end up making KRs very specific to the objective. for example: Make our service loved by our customers: • Increase NPS by X • Increase conversion rate for new users by Y • Increase repeat order frequency by Z And then every product engineering team is allowed make their own relevant OKRs aligned to those
and as a general theme of practice improvement (not derived from a formal flow study i must admin), we take up excellence OKRs
Hi @kapoor.vaidik - if you head over to #project-to-product and @mik, he or the team can get back to you. Alternatively, head over to the #xpo-tasktop channel where someone can try to help you! Mik is also hosting an AMA tonight with special guest, Pieter Jordaan, Group CTO at TUI! 🙂
OKRs & DevOps: From Micromanagement Misery to Finding Flow from @mik is now live in the Video Library, and available for sharing: https://videolibrary.doesvirtual.com/?video=550704601
And together they drive the business objective. Hit me on on #project-to-product if you’re interested in more.
Nice job @mik I liked tying the OKRs to the flow metrics!
Thank you @mik for your very valuable contributions to our community
Thanks @scott.prugh! I documented some more details on these practices and happy to share.
@mik Could share a bit more how to measure flow - what metrics are valid? (Lean time)?
@marcin.praczko1 For more on measuring flow, check out the talk from DOES Vegas: https://videolibrary.doesvirtual.com/?video=467489079 or the resources at: https://projecttoproduct.org and happy to chat more on the #project-to-product channel!
@mik How would you relate the 4 Key Engineering KPIS (from State of Devops/Accelerate) with OKRs?
Good question - this blog post from Mik covers this topic (amongst other things) - https://www.linkedin.com/pulse/measuring-what-matters-software-delivery-tasktop-viz-story-kersten
@robert.ruzitschka In my view they are critical telemetry at the team level, and should be set as OKRs for a value stream if they are the bottleneck (eg, improve deployment frequency x” where x is frequent enough for deployment frequency to stop being a bottleneck.
@mik thanks for the thought provoking talk. Now I have to try and process all the discussion it produced
Thank goodness for the DOES video replay function! 🙂 There is also plenty of related content on: • Tasktop YouTube channel - https://www.youtube.com/user/Tasktop • Tasktop blogs - https://tasktopblog.wpengine.com/
I’ve heard managers say “KPIs are now called OKRs”…
That is about as painful as "DevOps is just a pipeline right?"
Yes, heard that one as well! 🙂
Understanding new concepts is hard, and managers typically have limited time, so they develop these mapping heuristics... hard to get attention...
Yeah empathy when explaining and not diving too deep goes a long way
Not everyone is as deep into the topic as you are, is a good mantra to keep in mind when talking about stuff
I think it is very important not to mix KPIs and OKRs, or the OKR deployment will not result in any improvement to the dynamics of your planning.
So true - OKR's are a mindset shift. Big interdependence with Psychological Safety - can't have successful OKR's without the psychological safety.... In my experience, organisations that have not embraced the mindset of OKRs (e.g. re-named KPI's) also have issues with psychological safety that may need to be solved first/in parallel
That statement can be expanded to loads of things I think, the cultural aspects are always a big factor, usually the techniques and approaches just help shape that and force you to think about what kind of culture you want to foster
Ultimately I don't think you can avoid thinking about your own special situation and evaluate it
come see us at the project to product or tasktop channel to see it in action!!
Think of the Flow Metrics as a leading indicator for measuring process improvement.
Morning Plenary Links P&G’s DevOps Journey with @colas.a & @olimpia.nitti — https://videolibrary.doesvirtual.com/?video=551641782 H&M Group – Integrating Business and Tech with @daniel.claesson & @jakob.knutsson657 — https://videolibrary.doesvirtual.com/?video=551641791 From Your Auditor Friends: What We Wish Every Technology Leader Knew with @lucasc5 & @lewir7 — https://videolibrary.doesvirtual.com/?video=551641804 Intelligent Control - Enabling Safety At Speed with @leanne.bridges & @mark.rendell — https://videolibrary.doesvirtual.com/?video=550704454 Leadership Development in the U.S. Navy with Admiral John Richardson — https://videolibrary.doesvirtual.com/?video=550704528 OKRs & DevOps: From Micromanagement Misery to Finding Flow with @mik — https://videolibrary.doesvirtual.com/?video=550704601
Yes, I find this very much true, and why the way that leadership approaches the initial rollout of OKRs is so important.
@jeff.gallimore when/where is the AMA for @jonathansmart1?
@nickeggleston all the links are on this page https://doesvirtual.com/ama-hh-links
"When setting OKRs, focus on the destination, not on the means to get there". migrating from activities to value based OKRs... "delivering an initiative is not enough. We must fulfill it successfully" - in addition to measure what matters and radical focus, I thought this site was pretty concise as well - https://felipecastro.com/en/okr/success-criteria-types-key-results/
Yes that is a good resources, thanks @katharine.chajka!
The action has moved to breakouts! Join the following channels to chat with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4
Coming back from the break in a few minutes, we are so excited to welcome @nora from Jeli! 🎉
Hi all! Excited to be here 😄 looking forward to answering your questions
<!here> plenary started up again! @nora is here to answer questions.
Up next: @nora on what we can and really need to be learning from incidents!
This is going to be so interesting, incident analysis and learning is such an underrated field!
> An investment which was made on your behalf, but without your consent.
“we’re merely making efforts to stop future incidents — but we’re not improving the incident review process…” and thus not yielding additional knowledge we should be getting from them, that could create so much more value.
"Incident response and system management" is the phrase I use after watching previous fantastic DOES talks on incident management (Erica Morrison, John Allspaw, Richard Cook) Looking for to this one a lot
We definitely use it as a serious idea though, incident reviews are an opportunity to improve work for our teams, not just reduce errors
@nora — did I capture the gist of that statement? https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1621441168307900
"It's the system telling you where your understanding is wrong" 🙂
yes @genek! folks tend to expect to improve incidents, without giving more time to spend on incident reviews
Gary Klein: “Performance improvement = more errors + more insights” — we’re good at the “error” part, much less good at the “insight” part
it's easy to measure error reduction, it's hard to measure insight generation...
“we were doing most of the learning, as opposed to the people on the actual product teams” 🙂
so less time ends up being spent on it, when it's incredibly important
“incident analysis is not about the incident — it’s about understanding delta between ‘system as imagined’ vs ’system in reality.”
“the ‘socio’ in ‘sociotechnical systems’!” <--- so good!
That's exactly the best incidents to learn from 😮
Yes, individual incident reviews are not that valuable - it's the general trends across many incidents which are much more helpful for revealing the underlying issues.
I agree with that (depending on how that’s done) - I was disagreeing that individual reviews aren’t that valuable
I have a very dorky way to describe this… individual errors should be analyzed by machines. The first derivative of individual errors (error budget burn rate --> incidents) should be analyzed by SREs. The second derivative of individual errors (incident and reliability trends) should be analyzed by leadership.
@davidstanke532 nice - definitely some truth in that - derivatives are more relevant up the hierarchy
"incident and reliability trends" - does anyone know good blog posts / videos / etc. on this topic?
Hi Vlad! I’ve got some content on this which unfortunately is not in a state to be shared outside of NDA. But I’m starting to think there’s appetite for it (I’ll take your question as a sign!), and hopefully I can find time to write about it.
Cool! Please, email <mailto:firstname.lastname@example.orgemail@example.com> once you have sth to share. Looking forward, @davidstanke532!
Ah, so true, it's not a blameless post mortem just because you call it that 😜
you can't get the trends across incidents without spending time on individual incident reviews though
I've tried to set up monthly reviews much like Jon spoke about in previous conferences but I've struggled to get traction.
I sat in one just yesterday that folks were asking each other how different parts of the system worked, they were all refining their mental models, it was one of the most collaborative, learning experiences I had ever seen
I've led one of that kind once for one of our systems, still learning from it after months... So valuable! Other areas do post mortems but in them there are no developers because they are too busy and need to be "protected" 😞
It has an interesting parallel with safety practice, if there’s an accident due to human error you wouldn’t disregard it, you would try to engineer systems to reduce the opportunity for human error
Listening to @nora talk thru this interview is like listening to Sherlock… or Monk. 🙂
We gained so much insight from just talking to Kirin and we improved: • on call transfers • new employee on-call rotations • this particular kafka broker
exploring the context is such rich territory for learning
I would bet only the 3rd one would've been improved if we hadn't done a thorough review
To be honest I like the opportunities incidents bring. Assuming of course you have ways to navigate through the crisis in the first place. It’s always great opportunity to learn something from them and use it to foster a culture of constant learning and improvement. Incidents can be your leverage to drive attention of stakeholders to important issues that were neglected and contributed to the incident in the first place. But use it wisely as it can be double edged sword if your organisation prefers control over retrospective and improvement.
This shows though, how much effort goes into that incident review, compared to just sitting together in a meeting for 1h .... It is a lot of up front information gathering that is so interesting!
definitely @philipp.boeschen650 - but there are certainly smaller things you can do to raise the floor
Oh now I'm curious, I've got a few ideas but would love to get some inspiration!
I love how @nora is describing how we improve the “socio” part of the system — which in all domains of knowledge work, is a larger part of the value creation process. The parts that aren’t the code. I think this more concretely describes so much of what Allspaw talks about, which can be pretty abstract. cc @jtf
And this is also one of the hardest parts (I think) to convince leadership and other team members of.
That “socio” part jumped out to me. Remember that That “socio” jumped out to me. Reminded me that Westrum is a sociologist.
@nora i’ve noticed some of the language you use (or the absence of it). no “why” questions. lots of “tell me about…”
Excellent talk about the implications of words: https://www.usenix.org/conference/srecon19asia/presentation/eckhardt
@nora who is the best person to do the interviews? A Director, Product Owner or a dedidacated Incident Review team? Or someone different?
^^ absolutely @jeff.gallimore it puts people less on the defensive
Wow. Promo packets as contributing factor to incidents!
“we noticed that there was an uptick in incidents during certain parts of the year…“. (uh oh?). “around when promotion packets were due.” (OH NO!) 😆
We tend to use the 5 why’s, do you think there’s a better way of tracing root causes?
We’ve moved away from 5 why’s because of many reasons in this post by John Allspaw: https://www.kitchensoap.com/2014/11/14/the-infinite-hows-or-the-dangers-of-the-five-whys/
Also, we’ve replaced “root causes” with “contributing factors”. Also from John Allspaw https://twitter.com/allspaw/status/168001737628721154?lang=en https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/
This is a better way https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf
Never ascribe to malice what can adequately be explained by incompetence; never ascribe to incompetence what can adequately be explained by perverse incentives.
Walmart paid bonuses based on company performance. Then they put a tiered structure in place based on individual performance reviews. The first was motivating. The second had the opposite result. Even if you got an high performing bonus you knew that a co-worker was being hurt because they had a poor manager.
Oh wow... incidents caused by people rushing to finish work because that's what they were being measured on! 🤯
I find this story to be so hilarious, tragic, delightful, complicated, thought provoking… c @nora
@rshoup “promo packets” definitely resides in the domain of the “socio” part of the “sociotechnical system!”
I talked to one organization who paid bonuses based on number of features delivered and gave bonuses to PM's based on them rescuing troubled projects.
That puts me to have one more ques in my backlog which I will talk to you later 🙂
I love the flow of this presentation, it's so well done 🤯
Different dynamic in UK companys, but it would be interesting if the same correlation existed with end of year objectives/bonus & pay rise time....
on the product we're trying to get out, we're pushed so hard to get it out that the incidents are coming in to fast to review
One of the most fascinating incident reviews was associated with the fire on the Forrestal carrier. There were actually two reports, one by a special review committee, and another stimulated by the CNO. The most interesting part is to see what changed as a result. The CNO kept getting progress reports about the changes that were being made, this was Thomas Moorer.
Thoughts on metrics/dashboards that measure individuals.? always feel the risk of that gettin weaponised/toxic quickly is high
not tracking individuals -- but highlighting what made things difficult for them is key here
Hello, Dr. @ronwestrum! So delighted you’re here — I’m curious to what extent this reminds you of the deliberate ways that accident reviews are conducted in other domains?
The number of incident calls I've been on where the question "what changes happened" hasn't been asked at the top of the call
Are there many examples of using examples of how other industries do their incident reviews for better Digital incident reviews? I'm thinking in my companies space, wells event reviews look at all aspects
That’s great, @ronwestrum — I’ll send you a link to former CNO Admiral Richardson (@jmrichardson1) who gave a lecture earlier today, on radical delegation and leadership development. (Who oversaw investigations of a series of accidents and near-misses in South Pacific, not discussed, but much has been written about)
Especially reviewing "trivial" incidents is such a good idea, always leads to good stuff
because people are more open, and it creates space for when we do need to review the non-trivial ones
@nora Loving this talk This is key to work our team is doing now. You mention Jeli isn't available yet, what is the timeframe for when it will be available? We literally were just talking about needing the visualization component of our analysis. 😍
When the focus/goal of post-incident work is to produce preventative actions, the resulting description tends to be thin and narrowly technical. As a result, people build an expectation that they won’t learn much from reading them…so they don’t. (full disclosure, we at Adaptive Capacity Labs are huge fans and partners with @nora and what Jeli is doing!) ❤️
So good seeing you, @allspaw! Looking forward to catching up soon1
Trivia fact, @ronwestrum — I think I was introduced to your work by Jez Humble, who was introduced to it by @allspaw, who has been referenced several times in the last 30m.
@shoemakermk we are in closed beta right now 😄 and taking on a certain amount of customers! happy to connect with people from this community that feel like they could benefit from it today. you can DM me in here, <mailto:firstname.lastname@example.orgemail@example.com>, or https://www.jeli.io/contact-us/
Socio factor. I guess the industry doesn't look at the business factor / cost many a time. @nora
(PS: Dr. @ronwestrum is giving a talk tomorrow, and part of it is a discussion of the change in culture at Boeing after the McDonnell Douglas acquisition, and how it affected the 787 and 737 MAX programs)
counter point on external reviwers...need to be careful how many external people at once, or it turns into a bit of a star chamber
When do you feel like you've hit diminishing returns on investigating an incident and are ready to move on?
Incident reviews are a muscle to be trained :thinking_face: That sounds like fun workshop material 🙂
“I’ve seen people get promoted by conducting this type of incident reviews.”
When the focus/goal is to capture and represent the richest understanding of the event for the broadest audience, readers look forward to discussing what they found new or interesting with others, a network effect. This results in write-ups being seen as valuable resources in their normal work.
I do recommend folks timebox it (especially when learning) it can help it not feel like it's dragging on, and can help not burn out the investigators
@nora / @allspaw how to make the #IncidentTeam more inclusive. Many a times they are considered as Ops folks not part of your value chain.
@nora is the value that this person got due to this? Understanding how their component fit into the larger system as a whole?
they changed the way they were going to consume a particular service
@nora in your opinion, what are the events when you see developers looking for and reading incident reviews / postmortems?
at first it's more like "oh gosh, terraform again -- I def need to attend this one to get my opinion across"
@nora -- Managing Incident and writing RCA become more tough when C Executive are involved and some time not sure when we dont know why the problem occur in a short span of time period :)
Does anyone use the information found in the incident reports to the troubleshooting section of your runbook?
🎊 Incident Analysis – Your Organization’s Secret Weapon by @nora is already available to share here: https://videolibrary.doesvirtual.com/?video=551641823 (video & slides)
Thank you, @nora! And now, closing us out today, is none other than @corey...
@nora one of the best talks so far, thank you!
@nora That was an amazing talk! So much learning packed into that time. Thank you! 👏:skin-tone-5:
Hi everyone, thank you for attending my talk: You Suck at Cloud and It's
Not All Your Fault.
Head over to http://LastWeekinAWS.com/DOES and sign up to receive:
• The Unconventional Guide to AWS Cost Management
• Understanding Data Transfer in AWS Diagram
• Recommended interviews and podcast episodes
And now, my nonsense begins.
one of the best ROIs of Incident analysis is how much it levels everyone up, especially the people analyzing the incident -- the more folks you have doing this, the more expertise builds, the better your system functions
Welcome @corey!! Super excited that you're presenting!
Shameless plug: I'll actually be talking about The Anatomy of Three Incidents from Google, Stitch Fix, WeWork next week where we applied a similar pattern of very detailed post-incident investigation and followup: https://99percentdevops.com/blameless/?utm_content=166199815
Going from incidents and on-call to someone who definitely does not want to wake up at 3am 😂
The amount of money that we collective spend on cloud always boggle my mind.
And we can't walk over to their desk and yell when they are unstable. 😞
Clearly cloud is not blameless. :face_with_raised_eyebrow:
After asking for cloud access and being denied, eventually I paid my own way. In a strange way it felt good because I got what I wanted - the learning
@saket.kulkarni We conducted a blameless postmortem and blamelessly determined it's all your fault.
Kidnapping your account manager’s pets used to work, until AWS got better security
I want to have beer with @corey :rolling_on_the_floor_laughing:
Today after the conf in the Gather Bar, then thereafter (if that closes) in FinsterChat... coordination in #happy-hour
AWS PMs name their dogs things like "Simple Elastic Barkenfloof 2.0 for Containers."
Which begs the question what happened to Barkenfloof 1.0 (or 1.0.7, for that matter)
That multi-az thing is a big pain in the neck with kubernetes in AWS BTW
"...and not in the fun way I do" :rolling_on_the_floor_laughing:
But maybe the nerds you need to pay to do it properly are more expensive than the savings at AWS...
BUILD is coming up next week... lots of material for your special blameful narration...
@corey is there an opinion on the new macie stuff actually? Have not looked at it yet...
Better pricing but still not great unless you bound it and calculate the costs carefully first.
That sounds like my expectation, unfortunately 😄
@corey 's talk is the one getting the most smileys so far. But a serious topic: bills!
I think the soft jazz piano backup is my second favorite part of this talk.
It’s a good thing none of the brick counters came to the summit @james.simon1165 - this would set us back decades
It’s probably a short sighted view, but this just makes me glad we use Fargate 😄
That picture has been referenced so much by me when explaining AWS traffic stuff 😂
What do you take for the headache staring at the AWS Data Transfer Costs diagram must inevitably give you, @corey?
(This point led to a blog post: https://www.lastweekinaws.com/blog/s3s-durability-guarantees-arent-what-you-think/ )
11 9s, or the one question that is in every AWS certification exam ever
Wait, we don’t have to know the Data Transfer Costs diagram for the certification exam, do we? 😰
@ashulman I'm not sure; I'm conversant with American backwoods hicks, but am unfamiliar with the backwoods hickery inherent to other geos.
“It doesn’t exist because it’s not on YouTube.” that’s some sound logic there. :thinking_face:
OK but neither are a lot of the individuals working cloud in our orgs ,but that doesnt prevent them from picking up these axioms and amplifying them as facts
Fair. I've just seen the other side of these things on the AWS bill. It... shapes opinions.
The best part of consulting is seeing how these tropes play out in lots of different shops.
Ok...Corey...I think my DevOps Engineers will contradict here massively...
I was a human version of this emoji 😱 during that tshirt story
Just these days we copy the whole platform from AWS over to Azure...which is not too hard because we did all Infrastructure as Code
@dubravko Is this a pile of VMs, a managed database or three, and maybe a load balancer?
@corey, hard to fight you, you are rhetorically more brilliant than me... 😉
(Short version is: you're probably correct for your environment. I'm speaking in the general case, but you've thought about your specific constraints and context way, way, way more than I have.)
Which is probably correct, as we never discussed my certain context (at least I can't remember)
Exactly! My point isn't "you're doing it wrong," it's that if you're chasing multi-cloud because you read it in an in-flight magazine or something PLEASE STOP IMMEDIATELY.
Oh no...when we started our journey in 2017 we asked which cloud to use. No one could tell us so we stayed cloud agnostic on purpose. And as of today we need to serve Azure and AWS...
@corey than's for the canonical pronunciation of GitHub. I'll be using that and crediting you. It's a perfectly cromulent way of saying it.
"go all in on one cloud and then acquire a company that is on a different cloud? good, leave them alone! " so, so good. easier said than done for the command & control leaders 😜
Kept trying to get the sales rep to find out who to talk to about an ELA... crickets...
The term is "EA (enterprise agreement)" for the zero dollar enterprise agreement, or the EDP (Enterprise Discount Program addendum) or PPA (private pricing addendum).
It's always good to know exactly what to look for. Will the standard reps know about these, or do you have to know someone inside?
Painful topic. Since Conti has this saving contract with AWS no developer can forecast its costs anymore...
I fact checked @corey’s claim that he's #1 and #2 in search results for AWS contract negotiations, it checks out 👏
I do. And Controlling provides me how much I need to participate on all fees and stuff. AWS cost report and Conti Controlling never fit. And Conti Controlling is not able to go down on project level. Not nice...
Knowledge is power — here’s an amazing 2003 video of Warren Buffet and Charlie Munger talking about how made $120MM selling Costco call options to traders who were blindly using the (Nobel Prize winning idea) Black-Scholes option theory to calculate option values. https://www.youtube.com/watch?v=Fd4lfVNJljk&t=16s
Who wants to scale down anyway, scaling up is way more fun!
your hourly $$ cloud rate should correlate to your traffic/usage otherwise you are likely still in data center mindset- GREAT ADVICE!! 🙏
OMG got the cloud providers in one -- turn it off because they got distracted by something shiny :rolling_on_the_floor_laughing:
Interesting: we AWS-certified architects like crazy, and got mostly serverless (cloudier) implementations as a result. So, is the best recipe to reduce your AWS bill to get archi/developers to AWS certify?
We ended up having Lambda everywhere and it's somewhat of a mess in some areas though :thinking_face: Knowledge itself helps picking the right technology in the end tho!
@corey does Amazon have the testicular fortitude to have you Keynote re:Invent?
don't let developers near your cloud - I had one leave a GPU instance they were experimenting with running for a couple of weeks 😄
quinnypig for the winny pig great talk. I like shiny things too, so I work on Azure, but I know I can apply these principles there too. Thanks!!
🎉 You Suck at Cloud and It’s [Not] All Your Fault by @corey is available for sharing here: https://videolibrary.doesvirtual.com/?video=552401856 (video & slides)
Thanks @corey, big fan.
I hope you had a fantastic Day 2 here at DevOps Enterprise! And as great as today was, we have an equally amazing day tomorrow! @mail832 from the UK Government Digital Service is sharing her experiences of how the UK government used her Government as a Platform services to help protect the health of UK citizens, often the most vulnerable, over the last year. @steve773 and I will share what we’ve learned about the common themes of high performers across every industry, building upon the models of structure and dynamics. Dr. @ronwestrum will give a lecture on information flow in organizations, and part of it is a discussion of the change in culture at Boeing after the McDonnell Douglas acquisition, and how it affected the 787 and 737 MAX programs. And @jason.cox gives us an update on the amazing Technology Management Rotation program that he’s served as the executive sponsor for, and we hear from leaders who went through the program! And we hear from Dr. J. Goosby Smith (@drjgoosbysmith) on the elements that create a sense of inclusion in teams, a necessary element for people to feel engaged and bring their best and most creative work. Those are just a few of the many highlights — there are so many good talks tomorrow!
Thank you @corey. I had a great laugh and some thoughts to take home 😄
And hey, thank you to our sponsors, who have done an amazing and DAZZLING job who have truly challenged themselves to engage with this our community in so many creative ways, during a global pandemic that has forced us all to be stuck in endless Zoom/Teams/WebEx calls for months on end. I can’t tell you how delighted and impressed I am with all the delightful experiences they’ve brought into their Virtual Happy Hours and more. Thank you to our sponsors for doing this — we couldn’t put on this Summit without them!
Help us show them some love, visit their booths, stop by their happy hours and enter their prize drawings!
If I can help with anything, please ping me. If the answer fits in a short conversation I'm not hoarding it any. 🙂
If I can help with anything, please ping me. If the answer fits in a short conversation I'm not hoarding it any. 🙂
Come to the Bar in Gather Town and help us feel worse about the fees we pay AWS
@corey -hope you can stop by in Gather Town Bar later
@ferrix, @bryan.finster486, @james.moverley, @dale.eb.pluthero, @sumit.agarwal and I are all still in the bar
The bar has been abandoned at this point. Dale won last man standing.
Great chatting at the bar yesterday evening. @corey We were chatting about multi-cloud and what that means for critical infrastructure like telecom and would have loved to hear your thoughts.
I try to avoid speaking to specific industries if I don't have deep experience within them.
@corey Can you post the link for your resources here, so people can download them?
There are definitely some differences of opinions on the multi-cloud v single cloud popping up in the scenius lately!
How do I stay in the scenius after DOES ends tomorrow?
@mik - you will be surprised, how much time in back & forth in these decisions go in an Enterprise. Multi-Cloud sounds like rketing than realizing the capability
(For the people who didn’t get a screenshot in time, @corey 🙂
Thanks for the great, informative and entertaining presentation, Corey!
OK, off to hosting a Happy Hour AMA with @pieter.jordaan! Link is: https://tasktop.zoom.us/j/94661630316
You’re just going to blame me anyway. It shouldn’t cost him too much.
Plenary Links from Today ——————— P&G’s DevOps Journey with @colas.a & @olimpia.nitti — https://videolibrary.doesvirtual.com/?video=551641782 H&M Group – Integrating Business and Tech with @daniel.claesson & @jakob.knutsson657 — https://videolibrary.doesvirtual.com/?video=551641791 From Your Auditor Friends: What We Wish Every Technology Leader Knew with @lucasc5 & @lewir7 — https://videolibrary.doesvirtual.com/?video=551641804 Intelligent Control - Enabling Safety At Speed with @leanne.bridges & @mark.rendell — https://videolibrary.doesvirtual.com/?video=550704454 Leadership Development in the U.S. Navy with @jmrichardson1 — https://videolibrary.doesvirtual.com/?video=550704528 OKRs & DevOps: From Micromanagement Misery to Finding Flow with @mik — https://videolibrary.doesvirtual.com/?video=550704601 Incident Analysis – Your Organization’s Secret Weapon with @nora — https://videolibrary.doesvirtual.com/?video=551641823 You Suck at Cloud and It’s [Not] All Your Fault with @corey — https://videolibrary.doesvirtual.com/?video=552401856
Going back to incident analysis... we often hear of the shuttle disasters, but here's a potential catastrophe that was successfully (narrowly) averted, for the Cassini/Huygens Saturn probe, c. 2000 https://www.thespacereview.com/article/306/1 https://www.sebokwiki.org/wiki/How_Lack_of_Information_Sharing_Jeopardized_the_NASA/ESA_Cassini/Huygens_Mission_to_Saturn
Excellent case indeed. What organizations in tech capture near-miss stories like this one?
for sure - I was just commenting on how rare (if at all) companies look closely at near-miss cases (never mind publish about them) like NASA has, like in this case 🙂
As a (lapsed) private pilot, it’s interesting to reflect on how you read all of the Air Accident Investigation Branch reports including near misses. You learn so much. As you say @allspaw , rare to see orgs do this. https://www.gov.uk/aaib-reports
@jonathansmart1 we hear a lot about the big failures (Dreamliner, Challenger) in talks and books, but less about the successes. My memory of the tv documentary on Cassini, this engineer persists with his concern and eventually is allowed to prove his point. I wonder what other cases are there of good psychological safety enabling a whistleblower to be heard, concerns acted on, and disaster averted.
for anyone going to happy hours—making it easy with all the links in one place: https://doesvirtual.com/ama-hh-links