This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-10-14
Channels
- # ask-the-speaker-track-1 (411)
- # ask-the-speaker-track-2 (347)
- # ask-the-speaker-track-3 (540)
- # ask-the-speaker-track-4 (399)
- # bof-american-airlines (2)
- # bof-arch-engineering-ops (10)
- # bof-covid-19-lessons (1)
- # bof-cust-biz-tech-divide (10)
- # bof-leadership-culture-learning (4)
- # bof-next-gen-ops (8)
- # bof-overcoming-old-wow (3)
- # bof-project-to-product (2)
- # bof-sec-audit-compliance-grc (37)
- # bof-transformation-journeys (1)
- # bof-working-with-data (2)
- # demos (78)
- # discussion-connect-february (1226)
- # games (43)
- # happy-hour (195)
- # help (76)
- # hiring (20)
- # lean-coffee (47)
- # networking (17)
- # project-to-product (1)
- # psychological-safety (10)
- # summit-info (249)
- # summit-stories (23)
- # xpo-delphix (25)
- # xpo-digitalai-accelerates-software-delivery (3)
- # xpo-harness (1)
- # xpo-hcl-software-devops (5)
- # xpo-infosys-enterprise-agile-devops (6)
- # xpo-instana (4)
- # xpo-itmethods-manageddevopssaas (1)
- # xpo-itrevolution (18)
- # xpo-launchdarkly (6)
- # xpo-logdna (2)
- # xpo-moogsoft (4)
- # xpo-muse (2)
- # xpo-nowsecure-mobile-devsecops (6)
- # xpo-opsani (7)
- # xpo-pagerduty (19)
- # xpo-pc-devops-qualifications (3)
- # xpo-planview-tasktop (43)
- # xpo-plutora-vsm (3)
- # xpo-redgatesoftware-compliant-database-devops (6)
- # xpo-servicenow (14)
- # xpo-snyk (3)
- # xpo-sonatype (7)
- # xpo-split (2)
- # xpo-sysdig (15)
- # xpo-teamform-teamops-at-scale (4)
- # xpo-transposit (11)
- # xpo-tricentis-continuous-testing (1)
This took me a year to understand; the problem of “shipping faster than you learn.” 🤯
Can apply anywhere. Physical (cars, consumer goods) and technical products (software).
@genek101 Oh, that rhymes "speaking a foreign language faster than you can produce the words." Makes perfect sense because of a few colleagues I had 😄
Shipping faster than you learn is often a subtle signal that the power dynamics are still wrong. The dev team is improving the only area they feel empowered to impact.
> The dev team is improving the only area they feel empowered to impact. Doing the only thing to stave off burnout
Internal customers are ways of setting up a fight over how to try to screw each other out of a penny
One of my fave lines from Unicorn Project is from @bryan.finster — “‘The business’ is not our customer! They’re our colleagues!”
not sure if you know, but you can actually learn "Rockstar" https://github.com/RockstarLang/rockstar
Another anti-pattern in a SAFe transformation is not explaining each team the benefits they are supposed to be getting from all those new ceremonies. That leads to a very lax implementation that doesn't really empower the team when it usually does help when somebody spends a while with them.
Absolutely true. There's often no feedback mechanism that clearly signals when the rituals start costing more value than they generate.
I am close to productising the "three week intervention to redeem a former SAFe team"
Literaly opened this thread to see what serious people said while I thought of my Maserati:see_no_evil:
@john.cutler so much to think about and reflect on! will have to listen to this again.
Thank you @john.cutler — and now, here’s @pdmoore, presenting on another source of dissatisfaction of how tech is often managed!
Did someone write down John's questions to think about over the next few days?
I was writing as fast as I could but have some gaps. Need to hit the library and rewatch that bit. 🙂
@dacahill7 What beliefs must you personally let go of. Tto set about not just sitting at the table, but leading your businesses, and using design and technology to support step changes and sustainable growth ? What would you do if suddenly you were give a two level promotion? How would you navigate that. What steps do you need to take to be able to connect the day to day work of your teams out to the benefits and outcomes in the world outside your org walls? To help your organization learn as fast as you ship? What can you do to overcome the inherent biases / dichotomies / stereotypes that exist about “tech” in your company? To be more than builders and runners? To turn everything you do into the business?
3Company as the product - Think about how we think positively or negatively about the big companies out there? I'd buy any product from X. Or I'd never buy a product from Y.
Some companies have had some nice successes in using technology to support their product - its part of the complete story/product/offering.
What? @genek101 do you want to make us angry?
Hmm, IT as a cost center, this sounds familiar
“Zone to Win” — @mik and I often talk about what a fabulous book it is. @pdmoore talks about how, in a just world, this book would be as impactful as “Crossing the Chasm.” Did I misrepresent that, @pdmoore?
When put this way, it makes so clear why so many metrics and management models applied to “IT” are completely wrong — Productivity Zone is the cost center. Everything else requires different goals, objectives, metrics.
@genek101 have you mapped that over Cynefin?
I can totally see how the 4 Zones have different dynamics, as per Cynefin. cc @pdmoore @mik
Yeah the "win/loss" ratio could be a "complexity compromise" version of revenue vs. plan when planning isn't so easy.
Not sure if a direct overlay makes sense or not.
@john.cutler Thanks for that - very insightful and aligns very much so to our way of thinking. Would love to chat more about it.
I am so fascinated by this Splunk case study: the structure and dynamics when there’s a team for Salesforce, Netsuite, etc…. Like Team of Teams, like functional Ops silos. The universality of this problem blows me away. Regardless of what level of the application stack you’re in!
Link to the story or what to search for to read deeper on this?
With that openended question, I could see different answers from different areas of the organization. Is there a suggestion of who to start with for asking these types of questions?
Grrrr... Hostile vendors... We need to create a support group...
Hahaha. So good…. Beautiful story, @pdmoore, of liberating an org from a hostile vendor.
anytime you get a chance to tell Oracle to pack sand....take it
Depends on which end you are holding 😄
I love GraalVM, the next generation JVM, coming out of Oracle Labs. I’d love to see people pay for it, to pay back all the amazing R&D that is going into that project. (Any other fans of GraalVM out there?)
Now one of the world’s most valuable companies — was in a race with Apple to be the first $2T market cap company!
Just to be clear, is @pdmoore saying that MSFT was using the zone methodology intentionally or in an emergent way?
“why most tech initiatives fail: they’re using Performance Zone metrics for initiatives in the other zone.”
Mistake: using performance metrics to measure incubation zone investments!
Always wonder about overlapping other "four quadrant" models when hearing about these - particularly Cyenfin and when we use the "wrong" mindset in a given situation.
I love that story about MSFT, I think it was Simon Sinek who I first heard it from, but it shows the power of leadership and the right mindset!
I will be doing an Ask me Anything session tomorrow from 12:35-1:35 if you have further questions or want to talk about any of the frameworks and tools I presented today.
what I learned - nothing is easy and this is no different
Nothing personal but I fell asleep at 3:23 am listening to the final presentation. Surprising myself with stamina and recapping in the morning.
Lots of really good material today, it's going to take a while to digest.
Peter Moore and John Cutler - thank you for great end to our day today. Some my best sessions today
PS: I’m blown away by the Happy Hours that are happening — the creativity and resources put into making some incredible experiences are really amazing…. Like seriously, imagine the budgets put into physical events being deployed into a virtual event. Kudos!!!
@jeff.gallimoreI don't know how to be a croissant in a Zoom meeting! 😆
I really appreciate the opportunity to share my ideas with this community who is so committed to peer to peer learning Sent from my iPhone
next feature request you can expect @patrick.debois256 is the ability to suggest tracks. 😄
https://www.youtube.com/watch?v=K1b8AhIsSYQ comes to mind
I listen to anime OP/ED and Japanese Punk/Rock/Metal/Ska from a twitch vTuber.
Oh, and a make deploy song https://www.youtube.com/watch?v=-FMhUNSIxks
And the song for the on-calls: https://www.youtube.com/watch?v=LXAIAl3ZcWg
Oh, and a song about delivery frequency https://www.youtube.com/watch?v=YVVTZgwYwVo
For post mortems https://www.youtube.com/watch?v=FscIgtDJFXg
I’m upgrading the K8 cluster: https://www.youtube.com/watch?v=bwLkI5Id6cA
https://www.youtube.com/watch?v=SwYN7mTi6HM&list=RD9FMLghVlrc0&index=23
@patrick.debois256 Even if the newbies create whatever new deployment commands, one should anyway alias make deploy to them.
Oh, and not really DevOps... still concerns all of us https://www.youtube.com/watch?v=uABDLKyVm8o
Now there’s a mic drop moment: https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1602689101021200?thread_ts=1602688381.014700&cid=C015DQFEGMT
Oh, and there's this song about insufficient test coverage https://www.youtube.com/watch?v=bo_efYhYU2A
There is only one CI/CD song: https://www.youtube.com/watch?v=H5UK40sSo8I
This next plenary clearly needs this as the hold music while we wait: https://www.youtube.com/watch?v=0AAbY9Pvp6A
<!here> Welcome @afurtado and @lauren.knausenberger from the United States Air Force as our opening plenary after brief opening remarks!
Please welcome @afurtado and @lauren.knausenberger! I’m so delighted they’re able to share their amazing story!!!
give me a bunch of compliments and ill send you one
You are supercalifragelisticexpialidocious!
just kidding- im the government, i cant send you free stuff. but thanks for compliments
I am also a Veteran ...Desert Storm - Army so don't hold that against me..lol!
Over here in former-Europe when the government promises something they ALWAYS do it and they give us free stuff all.the.time #BorisFTW
I was a Government contractor for the AF for 15 years. I know how it goes. Thanks!
“It’s not wise to upset a Wookie” Really though, coolest shirt every Flyboy @afurtado
This is such an amazing story from @afurtado, describing how the accidental Doctors Without Borders led to an unflinching re-assessment of how software is built and run.
Powerful start @afurtado. The Kessel Run story is amazing.
Yeah. This is a crazy story and crazy outcomes to manage... This is what courage looks like... @afurtado
There is an AMA later today and Im happy to directly recruit all of you! 🙂
“You shouldn’t have to go back in time to go to work.”
Oh. Yeah. The Stockholm Syndrome... We talk about "learned helplessness" @afurtado
Stockholm syndrom. Nice. I mentioned that recently as well in a discussion 💯
Nothing inspires more than the post office starting a dojo
Oh, we are not cushioning expressions today.
It’s rather amazing to see positive change in such a huge bureaucracy
Now I'm sorry @ffion and I didn't swear
Commercial airliners do not refuel their airplanes in mid-air! cc @ross.clanton410 and all my friends at American Airlines! 🙂
being comfortable with bureaucracy is an interesting concept. I wonder if @schmark is going to talk about that later 🙂
Like watching @dominica,I always trust a speaker who has sticky notes next to them on the board
😱😱😱😱. (Excel Macros difficult to unit test. 🙂 I know from personal experience, how fraught with errors they can be. 🙂
Jigsaw’s success is most inspirational story for DOES. 🔥 Kessel run is amazing.
i love that the gui looks like the whiteboard. take care of the user.
There's a great story about how we came to that UI. Tried to get way too creative first and landed back to simplicity and user comfort. Im so glad you noticed that.
I was really impressed with the experience reports from that project -> It was a really great demonstration of genji genbutsu
“We hit roadblocks, because we exited within the world’s largest bureaucracy.” So good, @afurtado!!!
@schmark has some ideas as to how to apply summo mechanics to this problem.
I honestly never thought I’d be inspired by military software
"After years they had still not got their first ever deployment of whatever it was" - that
Engineers who started and now lead PlatformOne were the original Jigsaw devs! Enlisted!
I finally met @lauren.knausenberger at the Spark Tank competition she helped run at the Air Warfare Symposium, modeled after the TV show Shark Tank.
5 times a day inside the DoD; yet we’re struggling to get to that in a small private company.
Ha! That was an awesome day. IT was so great to have you there @genek101
these institutions are only as good as we make them. thanks for the effort here. @afurtado
Impediment: cyber security accreditation. Months to accredit. => Continuous ATO process. Impediment to flow alleviated
It is so amazing to hear about how @lauren.knausenberger and @afurtado met, who had common goals and were mutually supportive!
And to make a small enhancement to vehicle software takes 14.5 months :(
@afurtado You mentioned fuel savings. How many million are the red tape savings?
Lauren was an absolute gamechanger for us. As @jonathansmart1 mentioned, she obliterated our biggest constraint
“the result was that our most innovative would hide.” — @lauren.knausenberger. What a phenomenal observation!
so glad to hear we are not the only folks stuck in excel...fueling fighter jets with excel...WOW
Game Changer. Everything I have ever heard at my office has just become an excuse
This is such a great example of how support from the top levels eliminated very powerful, very entrenched barriers and obstacles.
you can buy that on Amazon- it talked about me getting flown to the White House to save the world
"Do I have to write my own unit tests?" "Yes, that's an order!"
This is so true — when @mik @jeff.gallimore and I visited Kessel Run, there was a delegation from a top intelligence agency there! 😂😂😂😂
Initial band of 5: IMO, there is a key point there. It started descaled.
The doors in the US all seem the same.
Kessel Run is now 1300+ people. When we visited, they hired scores of people THAT WEEK!!! A feat of incredible organization — the energy was amazing.
What I'm hearing: Descale before you scale. Start small. Prove it
seems like a pattern across a lot of groups presenting. @genek101 the problem solver :thumbsup:
@afurtado did you reorg using the team topologies model?
The book absolutely played a huge part in how we organized
Yes!! Inverse Conway Maneuver!! @afurtado @erica.morrison
You can see more here: https://www.youtube.com/watch?v=f4et0EGvKXA&feature=youtu.be
Inverse Conway always remember blurting out "Inverse Godwin"... well, it was descriptive of the maneuver that time. 😄
Those are some of the most intimidating looking Girl Scouts I've ever seen
don’t give those customers what they want, give them what they need
Sometimes saying NO is the best way to love your customer
"It's the responsibility of any platform or product to really understand their customers" - yup
@afurtado I'm curious what percentage of capacity you reserve for unplanned work
A ton right now! When we finally dove into this we realized that we were planning to 90-95% capacity and added 40% unplanned work on top of it.
So we started limiting the capacity, set WIP limits, and allowed our teams to get work down and started tracking the nature of the unplanned work
Over time we are automating and documenting those issues away and increasing capacity of the teams
Good stuff. How are you tagging and tracking your unplanned work?
A mixture of chat bots and GitLab labels
AKA "Reduce Human Debt" - good job @afurtado
“I saw the tangible results of absence of Psych Safety. Some elements of military structure makes it difficult; on battlefield, it can save lives, but…”
Tangible negative effects of not having psychological safety! - so clear to see when it's not there, hard to hear though
“none of our military wear uniforms, b/c we don’t want some aspects of those interactions to affect how we code.”
@afurtado How did you increase productivity of the developers but still meet segregation of duty requirements for auditors?
I was temp deployed to the submarine named after Rickover...ssn 709
a wild Steve Spear & Rickover makes an appearance
black friday event for weeks months and years…. or COVID :man-facepalming:
“We have an impossible product platform — a user scenario that has happened before… we want to be as prepared as possible; imagine being in retail, but not know when Black Friday will happen.”
“We have an impossible product platform — a user scenario that has happened before… we want to be as prepared as possible; imagine being in retail, but not know when Black Friday will happen.”
Very interested to hear the challenges and how you addressed them with getting actual buy-in and adherence to non hierarchy in actuality..
We got success under the radar. We did not have support right away- we kept getting told to stop using the name Kessel Run and to fall in line. We chose to not... then once you get in FastCompany, the support comes shortly after...
How is governance done? As code (policy, security etc) is Platform highest importance?
Love all these sci-fi references. Undeniably such a part of this culture we’re in! 🙂
Rickover at commissoining of first nuc, the USS Nautilus. Could have all the sartorial splendor befitting an admiral. But he shows up in the gray flannel suit…
I hope we discuss some of the bottlenecks to platform consistency...
Structure may make psych safety hard. However, commands sink in better if the psych safety is built in when the heat is off.
The “push through all levels of classification” is an interesting technical/organizational problem
yes - lack of a consistent cloud connection costs the gov't billions in local rework
Here's my take on the buy in piece. We started with laser focus on a problem that had lots of senior level eyes. The team started delivering big results with few resources and showed the obvious business case. After they broke the security barrier everyone realized it was not just innovation over there, that it could also defeat the bureaucracy.
This sounds like one of the stories in “War & Peace & IT”. Start small, set an objective, prove results, then double down.
I love what @lauren.knausenberger — enable people to find their gifts, and reach their highest potential!
Smart people solving problems in uncomfortable way - that's a picture I will remember!
^^ which is why I loved CISO of Fannie Mae @christopher_d_porter talk yesterday!!!
That's why we brought in the hacker teams. We've demonstrated that the new way is better.
"ruthlessly attack legacy infrastucture." yes @lauren.knausenberger !
Lauren, how do you deal with military structure opposing your path? too many generals in the way
We actually find that generals get on board with a well structured plan. And if you have the data and business case to support it most people automatically jump on board.
Congrats on your promotion to Deputy CIO of USAF, @lauren.knausenberger — a testament that the organization is valuing all the work you’re doing, and that they want/need you to do more!!! In service of an important mission.
Really inspiring. I love the continued refocus on the mission, and the ruthless application of modern practices
A couple great questions on platform consistency/parity... it's a nightmare use case tbh. Infrastructure of different types (commercial cloud, on prem, edge), on different closed networks (with classification and access difficulties), with networking complexity abound. Maintaining parity so the development environments are as close as possible to what the various production environments looks like
We are currently in the middle of a shift from PCF to k8s. Almost all of our platform services are on k8s now, working to shift all the mission applications over as we speak.
@afurtado That's such a great way to leverage your DOES platform. Hands down my favorite session so far. You all have a really exciting vision.
So @afurtado, I didn't get that bit about ticketing systems with regards to Developer commnication? What was done?
We still have a ticketing system... but we tried to make platform feature requests a little more customer focused and include human interaction to really understand what our customers need instead of just reacting to the nameless, faceless ticket system.
OK, makes perfect sense, that does address the main problem with ticketing systems, but also lets the ticketing system solve the problem it was probably brought in to solve, i e track tickets.
‘Amazingly complex problems you cannot solve anywhere else.’ 💯 @lauren.knausenberger
@afurtado @lauren.knausenberger Fantastic talk! Congratulations on fighting red tape and moving the Air Force to something bigger, better, faster... !
Great progress and journey @afurtado @lauren.knausenberger Thank you for sharing.
It was humbling to see how Spark Tank was designed to “enable airmen to solve other airmen’s problems” — people interested in seeing this remarkable event can watch it here: https://www.youtube.com/watch?v=4gqVVtkp_A8&t=1031s
@afurtado - Do you ever use SaaS products as part of your platform? How long does it take to go through your governance process at the moment? I work in a highly regulated environment (although not as much as USAF) and it's a slow process that often stops us even trying.
^^ IMHO, a wonderful model of attempting to enable innovation at scale at the edges, supported by core.
@lauren.knausenberger Great point about expanding industrial base. Huge flip in terms of who does the cutting edge R&D and tech innovation. Used to be .gov .mil spent and led and .com followed (50's and 60s). Total inversion now with .com innovation 80+% of the whole.
@genek101 Works on a personal level as well. Ideas come from the periphery.
@lauren.knausenberger - That definitely resonates ... the easy stuff can be what trips us up the most!
I love the vision of not just being "good enough for government" but just being, the most world class software organization, period. Its audacious and achievable, but revolutionary.
I think there are some places where the government is still world class. Unfortunately, that has become an exception, and I agree with @steve773 that that change happened in the late 90s/early dot com era.
@afurtado request to @lauren.knausenberger: how do we measure major programs on outcomes, instead of things like vanity metrics. “Wish granted.”
There’s an Ask Me Anything session with @afurtado during the morning break at 10:35am Pacific as a follow-up to the talk from him and Lauren this morning. You can get the link to join the discussion here: https://doesvirtual.com/ama-links
@afurtado @lauren.knausenberger As an AF Reservist in the 3D career field and a leader in tech for financial services as my day job this is so exciting!! Watching the Air Force lead the way for the DoD is great and I would love to know how to get plugged in.
Thanks @lauren.knausenberger and @afurtado! Such an inspiring and motivating story.
@paula.thrasher for what it’s worth, the phrase “good enough for government work” was once a compliment, that a company could claim that their quality reliability etc was good enough that even the demanding customers on the government side would be willing customers.
I believe it. Someday I still need to have this conversation with you: How do companies and organizations go from great to bad? E.g. How did NASA go from moon shot (innovative, risk rewarding) to challenger disaster (low trust, messengers shot, risk adverse)? I think its fascinating how places lose greatness (so we can learn to avoid that)
@paula.thrasher Great question. Let’s find time to discuss this. Doing some significant work with companies that have had the from great to …. experience. Some ideas to share. <mailto:Steve@HVELLC.com|Steve@HVELLC.com> Maybe next week or the week after? Steve
cc @afurtado @lauren.knausenberger Re: that incredible quote from @steve773. It’s heart-breaking that this phrase has turned into the opposite of its original meaning! 😥😥. 😂😂. If there’s two people who can flip the meaning again, it’s y’all! 🙂
I love it, @steve773. I've only ever heard it sarcastically I guess!
Here's an article on it: https://fcw.com/blogs/lectern/2018/01/good-enough-for-government-work-kelman.aspx
Please welcome @dwayne!!! I’ve wanted him to share his story for years! And now, finally, he can! 🎉🎉🎉
Inspirational speech, thanks for sharing your success story @lauren.knausenberger @afurtado
Im going to figure out how to make our swag available. Response heard. 😂
@afurtado Since you are a fan fiction hero does that include the matching NASA Apollo Mission control glasses as well? #AgileAdamFurtado
@afurtado @lauren.knausenberger Thank you for sharing your experience. Do you have internships for the development work you are doing where College Students can participate and help as they learn the skills they need to join the Air Force after graduation?
We have a pretty substantial internship opportunities- they mainly are focused on future AF Civilians
That makes sense. Thanks for responding so quickly.
Lol, LinkedIn knows everything about Dwayne and his previous company
ha...I did the same thing
Thanks for the kind words everyone- there is an AMA later if anyone has any more questions. https://doesvirtual.com/ama-links. 10:35am Pacific
2Micromonoliths... hmm :thinking_face:
Great job @afurtado. It's always great to spend time with you and @genek101. So glad everyone got some value out of hearing our story. Definitely join for the AMA.
1I’m in awe of the platform that supports all the revenue-generating applications, of which $30B - $50B of annual flow through!!!
Ohhh, didn't Sun used to call that network tone?
“Dial tone: no one cares about the tech that powers the phone infrastructure”: “It just better work. Otherwise, the business gets upset.” 😂😂😂
Ruby on Rails, all of a sudden the containers makes a lot more sense!
(Although I was amazed when I saw my first cell phone switch. ESS-6? It was just a SPARCstation 2 with an SBus card in it. Was actually quite disappointing… 🙂
❤️ Ruby on Rails. so good with containers
@dwayne observation of “Lawyer vs. Secretary” blows me away: even if lawyer can type, file faster than secretary, they should do lawyer work, not clerical work!
lol, Ruby on Rails is awesome for fake devs like me...I'm on the node.js + react bandwagon now
i am also a fake dev. probably why i like it
Rails is under appreciated. Amazing and shocking to hear it mentioned at DOES.
The opportunity cost is very high — similarly, “every hour they don’t focus on things that don’t have amazing business value” is a huge waste. Thus the incredible value of platforms to elevate dev productivity!
PS: Fans of Rails will LOVE the presentation from @eileencodes: closing keynote on Day 3. The 7 year journey to upgrade GitHub from Rails 2 to Rails 5!!!!!!! It’s such an incredible talk!!!
“…moving to this hotel company might have been worst career move ever; from VP to contractor; project could be cut… and from view of harbor to sitting at table, in room with no windows.” 😂😂😂😂
Just noticed this went live a moment ago, more stories from that amazing Kessel Run trip that @genek101 @jeff.gallimore and I had, from @afurtado’s colleague Bryon Kroger. Like the amazing preso we just heard, some mind expanding learnings from their experiences! https://projecttoproduct.org/podcast/bryon-kroger/
@dwayne That story on only getting the worst slots, geez!
no time like the present to run a perf test 🙂
“because we weren’t revenue generation: so only slot we could get environments at midnight to 5am; legacy teams had the best slots” 😭😭😭😭
Glad that she didn’t and that we are getting the learnings from this war story. I do miss awk…
From @jsnover13: “bash: the disease that you don’t die of, but die with.” 😆 😆 😆 😆
@mik You can return to awk any day. If you have a mac, you probably have it already 😄
LOL, I really should, and yes I bring up the shell on my mac on occasion still
"Containers are immutable!", Well, it depends.
Dwyane dropping evolutionary wisdom right now
That was my "A-ha!" moment for containers when I realised that they could just be a single line bash script all the way through to a bloated app. Most enterprise tech focussed on the bloated end in their marketing.
So true about containers on VMs, then onto orchestration
My Data Pipelines are all CLI driven, they just have fancy UIs that call them
we wrote two orchestration engines for containers on VMs before helm and kustomize
Not a given that you get all six.
@dwayne So did I connect the dots right, that you and your team figured out this framework twas catalyzed by wanting to avoid needing to be up 12-5am?
"Development tools"? Are you mad? NOBODY GETS TOOLS!
Really setup for success there in 2017 Dwayne 😮
basically, we had to be extremely efficient because we were going to be up at night
But every app has a DevOps team! How else are they going to get a pay rise!?
I love that the Ops organization not only didn’t understand what @dwayne, and had to compete with another platform initiative!!! “Once again, I’m all by myself!”
Service Catalogs, what a novel idea.
@dwayne and I were joking about how frustrating it could be to see all the compute instance types available in cloud providers. “I just want to run my code. I don’t want to learn about all 80 VM types, how much memory, disk.“😂
A profound observation — sometimes we don’t actually want choice. I just gas . I don’t care what the blend is.
@chris.gallivan we had base images for everything tomcat, Nginx, apache, java (play/spring), etc
@mik That's a great indicator. I once deleted the virtual machines with "my baby", the test tooling I had been developing as my main job. The reason: devs loved the new option and this was the way to get testing adopted.
@dwayne do you have service now integrated to pipeline? any step in the pipeline or batch?
@dwayne: is this framework available in the slides in high quality? My architect hat tells me I need to take a closer look into that ;)
Onsi Fakuri (Cloud Foundry) said it most artfully: Here is my source code Run it on the cloud for me I do not care how
Automate away your toil ➡️ Create capacity to solve more problems ➡️ get more problems to solve
@capiedra we were very SLOW in implementing SNOW. They have zero vision on how to implement K8's with SNOW
moving to cloud is new for me, can anyone recommend a checklist for cloud security (vs hosted/on prem security) ie. did I miss the details of how @lauren.knausenberger streamlined the long lead time to approve for deployment? is containers still the preferred destination vs ie. Azure PaaS?
It's very dependant on your own internal governance rules but a rule of thumb is that Azure will get you somewhere faster in governance as MS understands enterprises better. AWS is ore of a build it yourself approach to governance. It's a long process to asses each service one by one. Start with SoxII compliance as a standard for each cloud service.
@rodriguessemensati.e you should come to my AMA tomorrow, norm (my architect who helped me the last half of my journey) will be on. We can provide slides and answer any questions
awesome, thanks man! Let me check the calendar and ensure I have that one tagged
A moment of rare “dev rage” for me: when I went through the flowchart of “which database do I use?” https://cloud.google.com/products/databases. cc @rshoup 😂 (I ended up using MySQL, because all the choices frustrated me)
"Developers are all like 'I't's the network'..." I feel attacked
I'm fighting the container vs. service catalog fight at our company as well. Thanks for both acknowledging this fight and giving me some new ammo 🙂
99.9% of you don't need multi-cloud. Don't do it unless you have an extreme use case
I'm very positive SNow Orchestration will lead to future end-to-end automations and self-healing applications 😊
@occasl we are using Rancher.. it is great.
we were using Open Shift....but we were planning to switch to Anthos from Google
We've moved from OpenShift (container customer 1) to K8S w/ Rancher. Seems so much more straightforward and automatable.
Such an interesting point: “high touch, concierge level service” wasn’t what @dwayne wanted to do. He gave this to the competing DevOps/SRE teams, because he wanted to focus on platform development! cc @dan.sloan
@dwayne compelling story (about doing a rollback taking minutes instead of hours). Love this!
Yeah @dwayne this is good stuff. Very good stuff
This is a terrific session @dwayne! Love how you describe "productizing"!
Investing in the right tools. I hate how rare this is.
there are such inspirational talks here, thanks “Once again, I’m all by myself!” - resonates
I'm a sucker for a good Ops story, this is great
@chris.gallivan two different things; terraform is for infrastructure....containers is for apps
Reminds me: "Do you need kubernetes or a solution?"
getting old WOW engineers to adopt K8s is hard…
@dwayne Maybe installation is hard if you need several people for it 😄
@dwayne platform supported the work of thousands of developers, both internal and those in large system integrators. (Did I get that right, Dwayne?)
DevOps teams are the new awesome! Legacy teams pay the bills....abstraction was our way to give DevOps power to legacy teams
@dwayne Why did you feel 5 cloud providers was important?
"You don't need to run Kubernetes today. " That's my takeaway
the only good reason to explore kubernetes IMHO is CI/CD
That I am not sure of. We did this and found no value in it other than learning to write Dockerfiles well. In fact we had to build quite a lot of tooling to orchestrate containers. it gets ugly
it took us a little over a year to migrate to kubernetes. before that we had spent almost 6-9 months experimenting with docker and docker-compose for CI/CD.
unfortunately docker-compose cannot be used in production. so we ended up with dev/prod disparatiy, leading to frequent drift in configuration
eventually we decided to go to kubernetes purely CI/CD and took it to prod for dev/prod pairty
docker is not k8, however I believe people need to get good with managing containers before they then migrate to orchestration with K8
i meant to say that we didn't really find a lot to learn about containers outside of kubernetes, at least from an application perspective.
even from an ops perspective i feel there wasnt much to learn outside of kubernetes. just enough operational knowledge of docker was enough for us. that IMHO cxan be gained by just using docker for developmeent as well
Funny thing is Kubernetes out of the box doesnt work for anyone. It's mistaken for PaaS. It is a lot of other components on top of Kubernetes that streamline workflow and abstract complexity is what makes a good PaaS.
@genek101 you are correct, we supported multiple service providers and internal devs
love this @dwayne - your career path is so awesome
Do not pass up promotions... true... maybe I still should have 😄
@dwayne Are your teams considering Cloud specific PaaS services today?
“I was offered a VP role in 2017; I didn’t take it because I wanted to be hands on keyboard, and I didn’t like politics. But that decision was did a disservice to our team, because our destiny wasn’t in our control.”
@dwayne We should probably talk about this. I'm not a VP yet, but I made the switch. And I also feel like I did a disservice sometimes!
This is the paradox of every tech career. We love to build but if we are successful we end up moving away from what we love.
this made me thing about @scott.prugh going from a 10x developer to a 10x leader. I think Career Pathing is a big theme that we haven’t traditionally talked about at DOES
@dwayne agreed, that is why we made K8s our rebellion path, where we did the skunk works approach and then shared with eng and devs as we heard of the needs… now it is in the open and is seen as Containers 1st with new dev, and starting to look at retro older apps now..
@dwayne will indeed be hosting some awesome sessions later in the conference — @jessicam and @jeff.gallimore will post details on when/where!
@dwayne great talk! Loved the combination of technical and leadership messages. We are on a similar journey at American Airlines. Let’s connect!
@steve773 - excited for your talk!
@dwayne: security in the container, I saw Artifactory in the diagram, were you using XRay as well as its IDE plugin as well or something else?
@lucas.demelo no, I'm not a fan of AKS, GKE, EKS because they are different....as a dev you want consistent APIs...same with K8s
Voting for K8s deep dive as a session…. @dwayne for Pres, write in ballots…
"forensics with containers" would be great to hear how this was addressed
I want to hear about env var use, because I haven't had good luck with them.
Awesome talk! Thanks for sharing @dwayne
@bwilliams4 we used environment variables for conf. We were hosting env variables in hashi vault which was proxied through aqua securityy
about to check off one of my bucket list items — hear @steve773 give a speech at a conference…
This concept of structure and dynamics is so important in my opinion. The foundation for first principles of everything we’re doing.
Love the energy in your presentation @steve773 😄
was chatting with my boss and said "sorry gotta go to this talk" and then gave the title and left our call 🙂
@steve773 - @genek101 mentioned the "structure and dynamics" when I interviewed him for my People Before Tech book and asked him why we should ever bother with the concept of "organisation" so you're in my book 🙂
Disneyland is closer to DevOps than 19th century Japanese navy. But still looking for the arc of the story to appear.
Okay, so we are going back to Japan now.
The moment when @jason.cox and @brian.l.scott appear in @steve773 presentation!
He mentioned the bracelet! We used that story in our Forum paper!!! https://myresources.itrevolution.com/id006657108
@steve773: very energetic speech and awesome bow tie :thumbsup:
Yes!! We have to be solving problems collaboratively!! @steve773
Amazing talk @dwayne. Very relevant for problems we are looking to solve here at AA in terms of how we are managing our infrastructure.
Anybody else have a big stupid grin watching @steve773's talk? 😁 I've watched a few of these and I frickin' love the energy. Also, will @genek101 try and subtly shoo him off the virtual stage this year if he goes over time again? 😆
wrong Eduardo, but I am obviously IN haha
Shattered Sword explains why assigned roles on a team are a pretty bad idea. 🙂
@steve773 Do you make it a principle to approach from afar every time?
@steve773's talk about the Battle of Midway was one of my favourites at DOES20 London - incredible insight and energy
hmm, Goldratt would say it is to make money…. hmm
@steve773 Great energy and excitement - Yesterday was the Navy's birthday. USN!
I thought this was such a profound observation: exactly how problems are solved are invisible and ephemeral, tacit processes, was a shock.. Then I read this tweet: https://twitter.com/cm_stead/status/1314079840354541571 “Something really important about writing source code: Writing the code is not the work. The code is an artifact OF the work.” 🤯
All the pictures of how all the pieces fit together...but how do you create the ideas?
😂 @steve773 rattling off names of people involved in recording his session: @jessicam @erin @annp
go back and rewatch @steve773’s talks back to back. how he ties structure and dynamics into every talk and pieces them together is truly mad scientist. Rewatching lit up so many light bulbs that i never caught the first time.
Supercell CEO said that in his days at Digital Chocolate successful games were invariably created despite the process.
Dominant architecture forming the organisational structure - and culture?
(Meta-comment: the way that @steve773 uses the medium of the “small screen” video on web format is dazzling. I re-recorded all my introductions after seeing what could be done in this medium. I thought the best you could get was a news anchor, reading the news; then I realized how much more it could be…. My intros are a pale shadow of Steve’s, but much better than before. 🙂
The president of the entire system didn't care? Shocked...
Not uncommon! the “up 8 and down 3” problem. Or The Square in The Unicorn Project!
How good is your on-boarding, documentation, and how often do we update those docs
Okay. I'll have to watch this taking notes at 0.5x speed
This is a great argument for classic knowledge management. (And why we struggled with it - continue struggling with it.)
oops - looks like I have an old account from a previous DOES here somewhere
The York Minster Cathedral took 252 years to complete. Construction began in 1220 and ended in 1472.
Depends on your definition of "DONE" though :)
Trains, Disney, Japan, Breast Cancer... okay. Did I miss something?
There was something with the navy and war books too?
Okay. I'll watch this finish line camera speed later
Is this a sign of complexity or human processes (or both)? The battleships in the WWII era were built in a few years.
atoms are harder than bits, or at least that is the theory
@genek101 we really need to encourage @steve773 to use a wider shot so that we can get the full effect of his hand and arm movements. That, or we need three screens.
Didn’t post this ten minutes ago: Anyone thinking of Conway’s Law right now? This was a huge, huge and recent aha moment for me. The Dominant Architecture encompasses all implications of Conway’s Law! cc @rshoup @scott.prugh
Conways Law is a case of Dominant Architecture and Tools coding itself into the org...
enter ML: deal with the huge complexity, almost incomprehensible complexity.
I thought that the way we solve problems is actually well researched
holy crud. arbitrary inputs -> arbitrary outputs -> arbirtrary measures
as the work changes, the things that get measured and how measured should change too..
I'm live slacking DOES talks, including one of my heros: @steve773 talk on the NIKE slack instance. It's my hack to try to get some external information to busy folks that don't read enough or don't attend conferences 🐿️
1Blimey that's cool! I used to do that on Twitter for Finovate presentations and demos back in the day. Will you come see us on Track 3 at 2:05 and if so any slides I can send you to make your chronicle-doing easier?:)
@me1342 - I have a theory that posting high signal information will encourage spontaneous water cooler discussion ...
One can only hope
I'm a hopeless romantic and optimist
💯 @brandon.brown2 - I tagged you in that channel
@steve773 Excellent talk, but quick feedback on visuals - white on cyan is really hard to see/read with the resolution of the stream
…and would underscore absurdity when Advil decisions escalate all the way up to healthcare system CEO… so many similarities with Team of Teams story yesterday with @jessica.reif and @david627
Yes!! A reference to Team of Teams @david627 @jessica.reif @steve773
Wrong people connected the wrong way...the definition of collaboration gone wrong?
Chaos indeed...set the teams up before you start the work and let culture do some of the heavy lifting for you instead!
Similar parallels to the message in The Fifth Discipline by Peter Senge
@larryabel Thanks Larry. I arrived at grad school when 5th Disicpline, Re Engineering, and Machine that Changed the World at the same time.
“The key is speed.” As Gene says, “It’s not big beats small, it’s fast beats slow.”
Yes!! Give time back to people to make them successful!! @steve773
Absolutely but I was wondering if there was a correlation
Economics of trust and collaboration, I agree. And an aligned incentivisation to the actual economics, over role silo based stank ranking incentivisation
“Structure drives dynamics”, YES, and why I’m still putting all my focus on helping orgs establish and measure value stream structures to break through the org chart/silo structures that have been ossified at the level above the delivery teams. @me1208 Team Topologies has a catalog of structures that support the dynamics of what we’re after. That really is all it takes. When I’ve seen orgs improve the structure at the team of teams level to support the dynamics of DevOps a thousand flowers bloom…
I was also making the Project to Product and Team topologies connections - great segway to @mik talk up next
Okay. Not the right presentation to be interrupted by a call from mom. Luckily I need to watch this a few times again anyway.
YES for tools...I want to know how you went from problem to solution...what did the decision tree look like?
You'll love this @ffion and @rikard
Fast start to Day 2, right! Give us an emoji to indicate your experience so far today. And get ready for some more great talks…
The plenaries are not over and this feels a bit more overwhelming than yesterday. Soldiering on.
To paraphrase Francis Bacon, some talks are to be watched with popcorn, some in a relaxed setting with a glass of wine, some with extra hot coffee, fully caffeinated, alert and charged up. And re-watched again. And again. To be inspired. I think Steve's talk one was the last category. 🙂 Thank you for the energy, Steve! Also interested to check out the website and the book!
@jessicam @jeff.gallimore Can you re-post the link to @steve773 AMA, when / where? Thank you!
Thanks @scott.prugh! I’ll answer questions here live and then during the conf AMA at 5:30pm PT here: https://tasktop.zoom.us/s/94693819688
We’re on with Mik! Join us 🙂 https://go.tasktop.com/DOES-Vegas-20-Mik-AMA.html
@steve773 do you have a public link to this?
Join @steve773 for AMA today at 2:45pm PST - https://doesvirtual.com/ama-links
I’m so glad @mik is taking time to explain more slowly the symptomology of what often goes wrong in software, using Flow Framework constructs — I wish I had understood this better before writing The Unicorn Project!!!
Now the talk title makes much more sense
But thank you for encouraging me to finally take more time to explain the Flow Metrics @genek101!
"Convoluted red form with a linear green attachment" 🙂
I am now going to start describing tech debt exactly that way 🙂
“not an eyesight problem, but instead visual agnosia.” Not much an ophthalmologist can do!!! What a powerful thing to know!!!
@mik I love the reference to this book. I read this in college (there are some other awesome stories in the book as well - its really about approaching problems differently)
@mik Ah, so you need to keep current with awk to keep the occipital lobe working.
@steve773 Your talk reminded me of this book "The No-Limits Enterprise: Organizational Self-Management In The New World Of Work" Have you seen it?
I attended a meetup where the author talked about it. Very interesting organization model. Would be scary to compete against it.
For people unable to see tech debt for what it is - see why it needs to be prioritized - this talk is for you! Thank you @mik
@paula.thrasher Yes, so good. And putting 50% on tech debt blindly is like subscribing 1" think lenses to someone who has visual agnosia.
I think the reason flow metrics work better is that they talk to the qualities business cares about - speed to market. I think we often ask business "can I upgrade the FooBar system", when that is silliness to them. If we asked, can I work on something that makes me deploy days faster, we'd get a different answer.
Tech debt creates scope changes which in turn are a source of tech debt.
What a powerful thing for execs to fully internalize! The full implications of all those scope changes caused by Sales!
The graph does look like a hat.
maybe you fix the tech debt with more features?
Flow Metrics = Business Vital Signs 💯
Head over to the Tasktop booth to pick up a free copy of Project to Product (while supplies last).
Measuring flow opens the door to see the health of your end-to-end value streams
Flow Efficiency: 100% is the awesome feeling when you’re on a single person project, no handoffs! (Finally learned that when we recorded this session.)
@mik Have you correlated Flow Efficiency to team happiness measures?
hey Andrew good to "see" you again... let me know if you had any other questions and we can catch up about Flow Metrics
I’m getting goosebumps from this like when I read Project to Product. This is language we need to leverage senior execs.
we see correlations between better flow efficiency and higher employee NPS
@genek101 100% flow doesn't help if you have 100 years of work in line for you due on Friday 😄
Referencing @scott.prugh’s amazing achievement described yesterday of improving Flow Load (% of time spent on new features) from 15% to 55%.
Also, low Flow Efficiency causes an increase in deadlines and escalations, higher Flow Efficiency
@mik The low Flow Efficiency feels directly analogous to Kingman formula.
I love these explanations because we’re seeing the causation between all these factors, and the symptomology expressed, both before and after.
Hmm... Is this my company in the graph? 😄
That’s the interesting thing, it’s all causations in a dynamic system. Amazing to watch the over time. I want to create an animation over months…
Interesting that the field header for that field is "Project" 😉 Wouldn't that have some product-oriented name?
Project = Core Backend Services
@mik What's the source of that data? Ticketing?
Jira still uses that terminology. This org modeled their value streams over dozens of Jira projects.
Yup! Still sadly rare for me to see pull based systems.
@ferrix: yes, work item data across tools including Jira, ServiceNow, ADO, GitHub, etc
Favorite part of this story: “business kept putting more devs on the front-end systems; the EXACTLY WRONG THING TO DO! Must flow that talent to the Core Backend System and platforms!”
If WIP was an enabling constraint, they'd have blocked, causing pain, to help address it? @mik
Yes I think so. The next diagnostic is exactly about that point.
Okay. So we could have could potentially dig those out from what we have. Neat.
“they were doing the exact wrong thing: they were putting low cost contractors on the Core Backend System!”
So since @steve773 just talked about the importance of making the process of creating value explicit and visual - this is way of measuring and making the hidden work visible. I love the pairing of the talks this morning.
THIS - Make all work visible - including tech debt!
Yup, in the end all we’re doing here is trying to deliver on Dominica’s vision to make all work visible, and to show the complex dynamics of that work 🙂
The alternatives to making tech debt visible: kill it or die.
If pull based system of work, with WIP limited, it would raise the pain level to address it!
I found this inspiring: they’ve gotten so good at improvement at Tasktop, improvement projects now must pay off within 6 months. (did I get that right, @mik)
ie, need to see flow payoff within 6 months now, even for major API/architecture/platform work
PS: the idea of having more descriptive Slack usernames came from @mik when we debriefed after London conference. 🙂
"IT can't keep up with the business", I've heard that one! 🙂
@jonathansmart1 That’s what you were foreshadowing I think 🙂
The amazing thing to me is how often we still see this happening given how well this community has documented these problems!
…and alas, not everyone who needs to hear these messages is in this community yet.
I think @dominica puts the reason on it: We say yes because it feels good (we feel important, helpful, nice) - saying no is socially hard.
Agreed, we all have similar issues, that our peers have often solved and shared how they solved; yet we have much work to do to implement in our own orgs.
This is really fancy. However a simple burn-up chart will show that you are adding too much scope. Interesting.
Saying Yes to incremental work is saying Maybe or No to prior commitments
Why saying No matters http://videolibrary.doesvirtual.com/?video=467719259 from the Andy Patton show w/ @jonathansmart1
@ferrix Years ago I would track this all with burn-ups. I don’t find that a good visualization, and often too narrow a measurement of the value stream. Similar for CFDs. It’s good info, but the dynamics of improvements, changes, don’t pop.
"Improving daily work is as important as daily work" @genek101
I should have made that point more clear, ie, that fixing neglected WIP will drive second ideal and eNPS
@mik I can see that. It will make "there is a problem somewhere" very apparent, but does not help in finding the bottleneck
Would be amazing to have a deep dive techniques on how to implement Flow Framework! I´m working with my team trying to solve this puzzle
Happy to! Hit me up at <mailto:mik@tasktop.com|mik@tasktop.com>
So glad @mik has not mentioned any Large Cable Operators! :rolling_on_the_floor_laughing:
“6 agile teams; 1000 flow items in progress. 1000 large features? seems like something is wrong… let’s dig in deeper…” 😂
love the example and how you structure the presentations @mik! Can definitely related to some of the examples and the guidance provided here is valuable information. Have been wanting to read the book for a while, time to pick it up 😉
We’re giving away e-book copies at our booth if you want to grab one 🙂
I think I already did, you are right!! I will start reading virtually, but I like real books and I need some more to fill the huge bookcase I bought for my new flat 😉
Drop us on message on our xpo-tasktop slack channel and we’ll get a physical copy sent out to you!
@mik what do did you mean by other work in the comment about balancing the backlog and other work in the WIP case study?
I was referring to all of the work on the backlog, according to the four flow items. So a lot of tech debt work was happening, the other work I think was defects, features and risks. Does that answer your question? Not sure I fully understood.
Thanks Mik. That answers it. Much appreciated. I always enjoy your sessions and podcasts.
Aha! By changing criteria to Dev Complete but Blocked Downstream, dev now is obviously not the bottleneck — it’s downstream in Ops, infrastructure, etc…
1013/6 is amazing. I've seen ~200 stories for a "team" of 12 people.
Hey @mik - I don't see your slides on github --> https://github.com/devopsenterprise/2020-Las-Vegas-Virtual cc @alex @annp
something something tree in the forest falling
“Absence of psych safety made it difficult to actually share data about what was going wrong!”
Again - Psychological Safety topic ! Check out this topic in today's Lean coffee w/ @chawklady
I see relationship driven development. "It's who you know.... "
Fear lead to hate. Hate leads to end users suffering.
@bryan.finster That’s why the point of psycholigica safety is about to come up!
100% We work very hard to build trust with teams so they can use the measures as functional tools.
Again - Psychological Safety topic ! Check out this topic in today's Lean coffee w/ @chawklady
Treatment plan is Value Stream Mapping (shameless plug: I'm talking about how to do this later today Track 2)
amazing talk @mik! I am going to organize a viewing party for my team when the recordings are publicly available!
Thanks @dana.finster! You can also request a session here: https://projecttoproduct.org
@levi.geinert500 yes, it basically boils down to data-drive whack-a-mole! But the amazing thing is how much Flow Time can be reduced via the first 2 or 3 biggest moles.
Stop worrying about cycle time. Focus on Flow metrics. 💯
"A system of local optimums is not an optimum system", Goldratt
Slides are so clear, and just dense enough. Setting a high bar for exec communication!
Two observations so far: 1. Things like WIP limits, VSM, flow etc. really work as tools 2. The tools are not exactly new and shiny 😧
I guess I can skip all that work assigned to me then 😄
YES! If you know what you are doing, you are better off choosing your work instead of assignments.
@mik Very good talk! If you can share this deck is going to be amazing, as well!
Decks will appear here https://www.dropbox.com/sh/77udvz76kbkerrd/AAA67_LZ9iWNBxu_0JmBlnOLa/Keynotes?dl=0&subfolder_nav_tracking=1 - uploading today's as we speak, I'm told
Nice one @mik, thanks for sharing. Great to see the real world examples
Please welcome @lucasc5 and @lewir7, two auditor friends from Nationwide Insurance!
@mik Great presentation. To all managing large development teams, you need these metrics or you're overly dependent on anecdotal opinions. We're using Tasktop and seeing tremendous value.
Yep, that’s right, everyone. We’re going to have two auditors teach us some awesome stuff!
Amazing talk @mik! The kind of talk you need to place your seat belt on, because he talks FAST! Loads of content. Awesome!
“in my ten years in audit… my paycheck has never changed based on # of audit findings.” 😆
Hmm... are these just two benevolent auditors then :thinking_face:
^^ Hypothesis: maybe they’re the only two benevolent auditors on the planet? hahaha
@genek101 The last good auditors?
Static checklists for dynamic systems almost always lead to tears… and accidents and disasters. cc @steve773
Checklists can be great tools, but easily mis-used (I have a checklist, I can turn off my brain!) 🙂
Okay. I can see where this is going. Auditors follow checklists, Ops follow checklists. That's a bond!
I’ve done that (automated my audits). Made my quarterly audit deliverables very simple (and delegated).
@bryan.finster A checkmark for your checklist! And yes, checklists can and probably should be automated.
I've the opinion that manual verification of a rule is compliance theater.
checklists are about outputs, but auditing should be focused on outcomes, as with most activities
I used to work in a company wanting to make finance and procurement friends. How hostile can the top floor be?
David Marquet has some good advice on working with auditors in Turn The Ship Around
If you are doing things "just because they said so" please connect with your auditors to understand why.
@nickeggleston wait... is that going to be on my backlog soon?
Personally, I see the checklists as ok; ISO appendix A has 30ish controls, we should have an answer for each thing
The same way that there is dominant architecture in other industries, Audit has some pretty tried and true ways to test things. Partnership can be built in the way risk is assessed and scope is determined! I've been an auditor in the past and have had IT leaders invite us into the dev teams. It's possible to have the best of both worlds
That’s awesome! I was due-paying member of ISACA and IIA for nearly a decade. In particular, I always admired how great the IIA was as a professional organization!
After all of these years, I still carry my CISA cert in my email signature 😎 I learned more about systematic approaches during those years than I even realized back then!
Crossfunctional team: designer, developer, sre, auditor...
DevSecAuditOps
@nick.kritsky Checklists are a useful tool, but often those using them stick only to the checklist. Sometimes they can't see the risk staring them in the face because it isn't on the checklist.
DevSecQualBusAudInfraOps. Could just talk about flow. 😉
@lucasc5 and @lewir7 breakout session! https://sched.co/eVGV
Thank you, @lucasc5 and @lewir7!! And make sure to check out their awesome breakout session!
Fast start to Day 2, right! Give us an emoji to indicate your experience so far today. And get ready for some more great talks…
I remember the time when our team has our first external application pen test done by the ouside vendor.
And how developers went all the way from "Who are those stupid guys?" to "Wow, we didn't know about this hole! Thank you" . It was very interesting to observe
Myself, I have been audited so many times, that I have probably developed the Stockholm syndrome with the auditors
There's some really good conversations to be had with chief risk people, auditors, lawyers about your business objectives (we're in this together), and how they're implemented by tech. showing that the day to day stuff is automated so the humans can focus on the challenges at the edges demonstrates the buy in to those business objectives - and shows that you're not just robots :)
@jillmead2018 I missed the live stream of your session, so watching now over our "lunch break"...looking forward to it!
"We took a sabbatical in the dojo...to truly understand our engineering customers" 👏
Soon, we welcome @erica.morrison to the plenary stage for Q&A!
For those of you who didn’t make the connection, @erica.morrison is from CSG — the same org as @ken.kennedy and @scott.prugh! 🎉🎉🎉
"Failure doesn't have to be catastrophic - it's how you respond." @erica.morrison
…it still amazes me that a certain date evokes memories of a crippling outage… “everyone knows what you’re talking about when you say ‘2/4’”
So riveting. “continuous rolling outages eventually led to feelings of helplessness”. (or was “hopelessness?“) @erica.morrison
Incredibly - a situation where it's not DNS 😎
if I had a nickel every time we had spanning tree cause an outage over my career.......
..stories like this make me so insistent to purge IP addresses from my daily life… If I’m trying to figure out a CIDR block, something is very, very wrong…
I usually know its a bad day when i'm having to review wireshark captures....
I like how we use 'complex' as a placeholder for 'incomprehensible' in all kinds of situations.
I’ve enjoyed the lessons about incident response from other safety domains in previous years 😊
I like the layered response. Good lean approach - countermeasures at all levels and not a 'solution'....
Calling out RCA data points and LTF data points during remediation so that the scribe captures that data is hugely helpful, but I agree it is NOT the focus of the remediation bridges.
Yeah this bit spoke to me. I'm quick to make the y-incision during an outage to look at root cause, rather than put a suit and sunglasses on the corpse and take it to the party to resume service.
We had a lot of discussions about who and how many. This was based on a combo of factors - recommendations from Black Rock 3, our own internal assessment of who was strong here, and internal discussion about change agents
they started with 14 incident commanders: they started small, to gain some experience, lessons learned…. two people on call at any given time. first on bridge makes second unnecessary — they’d go back to sleep
@erica.morrison you're being nice. lesson #1 for executives like me was to shut up and let the SMEs work the issue. 😃
interesting, we got the opportunity to engage BlackRock for training in our company as well, more specifically driven by the IT Groups supporting our tech work in the manufacturing plants (where incidents can escalate very quickly, become physically dangerous as well as represent a direct revenue impact as the machines have to stop producing)
@erica.morrison How do you maintain the level of quality in incident response teams over time when things are slow for long periods of time (5-6 months without major incidents)? Do you do fire drills?
We haven't done fire drills yet, but have discussed. We do leverage lower level outages (not major incidents) for newer ICs to train on. I haven't run into the problem of not getting enough incidents to IC yet....here's to hoping that becomes a problem 🙂
Good luck then, I hope it happens! I guess we have rich people's problems 😛.
“Congrats! You’re so good at on-call, here’s more! You’re now on-call for your service, and all the other services, too! We were worried that this wouldn’t work.”
I think it’s conference call bridges?
Hearing about the amazing outcomes from better incident calls and supporting proceses still blows me away, @erica.morrison…
Hearing this makes it so clear how poorly run so many incident calls I’ve been on are…. OMG. “Hi, sorry I’m late. Can someone catch me up?” 😱😱 😂😂
Hi, this is __, can you tell me what's going on? has largely been banished. Not completely, but we're getting there
^^^ totally. It evokes memories of having to drop everything and explain again the current state,..
By the time you read the backchannel, you've lost so much of what has happened since you joined, could easily take 45 minutes
A structured whiteboard helps with this and a great scribe
Just talked with Jez a week ago — he’s loving it! (He finally was able to do this, because you can now do it WFH!)
Checkpoints times have been a boon for us. Also having a running PROBLEM RESTATEMENT (an iterative / dynamic problem statement) also helps my bridges by being constantly updated at each experiment / step.
OMG, @allspaw (and anyone else): I’ll share stories about some of the things going wrong during this conference — BTW, the first one we did in London… I was shaking for an hour after we got through the first 30 minutes… 😂😂😂 😭😭😭
Our's is the after party. @genek101, coming to the after party? We may have a Clanton there.
What size of org does this level of framework become necessary?
I would suspect that the metric is defined as an unacceptable MTTR.
I recommend it for any size org. Even a call with more than a few people starts to get chaotic and break down without structure.
Is there a public incident report available regarding the incident that is being talked about?
It sounds remarkably similar to several Qwest/Century Link/Lumen outages that have happened within the last couple of years.
is it building on the concept presented here? How different is it from the standard IMS framework that blackrock trained you folks at?
I don't think I noticed that one in the IT Rev library - when do you plan to release it?
It is being published this fall... It covers more than just the IMS framework. Some patterns that we have discovered and/or formalized with others.,
@scott.prugh @erica.morrison Do you have archetypes for the “types of incidents”, which people specialize in? (It’s like you have a sommelier for outages!!!)
I appreciate hearing the willingness to listen to your own experience and go against the experts when you have conflicting data.
This paper does cover many patterns we have leveraged.
I'm really impressed by the amount of training dedicated to Incident Commanders. It is a hard role to do efficiently, so having that many people in those rotations is awesome.
Being in the monitoring system, my favorite past time is being at the end of the finger pointing “must be a monitoring problem”… 😉
Ha! Many of our monitoring tools fall within my organization as well so I can relate!
@erica.morrison - have you found that some ICs work more incidents than others? And if so, is that driving change to the domains they are part of?
If you look at it for a period of say, a few weeks or even a month or two, absolutely it ebbs and flows. For our most senior leaders, I would say this tends to balance out over time. We do have a formal post-incident analysis process that includes briefing the company on the outage and determine improvements. So, if a particular area is struggling, it will get a focus
Yes! "There are some things you cannot not learn except by doing." @erica.morrison
“Learning incident analysis skills by reading an article (or filling out a template) is akin to trying to learn to ice skate by reading a book about ice skating.”
TFW when you realize you’ve used up almost all the error budget for the year. “Do we have enough to make it home?” — @erica.morrison
It was such a sinking feeling to eat up must of the budget so early in the year....and such a great feeling to still be able to bring it home within budget when the year was done
I just had a client describe it as "punishment for bad developers". - sigh -
It has to be coupled with continuous improvement and psychological safety. If you aren't going to do something as a result of things going wrong, then it absolutely just punishes. But, if you have a culture where it's safe to talk about failure, look for ways to improve, and have capacity to implement improvements, it can be a way to show all the progress being made and end up being a celebration.
Absolutely. And I think it's a brilliant concept that's much more easily understandable than the intangible tech debt.
But we're a consultancy and many of our clients ask us to get involved because their own system or SDO culture is fundamentally broken - so we see behaviour like this a lot. And it is very very hard to change.
So is it used as more of a canary metric, just to show if you're trending in a direction you don't want to go?
Complementing it w/ psychological safety and continuous improvement makes lots of sense... and probably self control. 🙂
No. We use it as a tool to tackle tech debt and balance backlogs. What we normally do is insist on running our own team as a separate project to build a "go and see model" - of how things can be different. That is usually enough to convince clients to consider the possibility that there is a different way... and that gives us an opportunity to work on trust, safety and improvement...
(oh and we embed a couple of client developers, so there is an internal voice to champion the 'better' way)
I know some people who took the BlackRock training. It seems you got your money's worth more than them!
Absolutely. We discussed trying to handle ourselves. No way would we have made the progress we did on our own. Those guys are awesome
Great talk. Honest and open. We've been spending a lot of time on incident analysis and resolution lately even though we don't have large distributed systems - we feel like it's just a pivotal part of DevOps. Good to get some validation.
This is such an important story and so well delivered. I’m going to recommend it to everyone in our org. Awesome presentation @erica.morrison!
Sverige :flag-se: 😄 I lived in Gothenburg for a few years. My sister went to school in Lund. Great times 😄
Erica: your 'impact minutes' metric sounds interesting. It only works though if it is cumulative in small bits - not all in one incident.
It can work both ways. The goal would be to NOT have it all be in one incident. But the system is intended to measure pain caused to customers...whether it's through a thousand papercuts or one big cut
Thank you. Sounds like two closely coupled needs - not one. So I was wondering if you considered a secondary metric to account.
IC is the way; it's a truly national system that's deployed everywhere and focuses on being able to work consistently
Any thoughts on value of using the CAN metric? I've been doing it when I'm doing a call, but frequently can't get others to. Haven't pushed the training, but considered
Oh that was interesting, the low signal strength
We found the root cause of the incident and they have been fired... 🔥
One of our interesting metrics are No Root Cause Problems in Problem Management.
until these 2 presentations, i’ve always thought about what happens after the incident is recovered (post mortems, etc)… have never thought about how to learn pre and during the incident… 🤯
We had talked about improving outage calls prior to implementing IMS, but it just didn't feel like 1) It would make that much difference and 2) That it could be done. Boy, was I wrong. I'm glad ACL pushed us to make changes here. And once you get the hang of it, it's really not very hard
"The state of maturity in the industry on learning form incidents is low" @allspaw any recommendation on "getting started" for large orgs starting?
Incident analysis is a set of skills that very few organizations have and recognize as a specific expertise. Supporting and encouraging the building of these skills by leaders is necessary.
What it looks like when progress is happening: https://www.adaptivecapacitylabs.com/blog/2019/11/20/markers-of-progress-incident-analysis/
Thank you @allspaw I will read and share, and thank you for your support at our past Learning Events.
1Pattern #1 is: Make Incidents Visible and Part of Everyday Work...
…I’m dazzled by some of the common themes in this talk and @steve773, so much reinforced by the concrete details and emotional impact of @erica.morrison’s amazing presentation…
The pointy haired boss: "Make this simpler for my boss, and even simpler for his boss"
We cascade Major Incidents against CMDB so that the responsible Service Owners and Service Executives have to tell the story of what the “blunt ends” and “sharp ends” are doing to remediate. Would be interested to hear how people solve this problem without big ITIL process (Incident Management, Major Incidents, Problem Management).
A different perspective would be to bring attention to what is being learned. Effective remediation depends critically on how rich/broad the understanding of the incident is. 🙂
@steve773 talks about the “vast and rugged topography of the problem” — major incidents/outages are clearly an extremely example of this.
"believe their presence has a positive influence!"
@allspaw Is the observation about leadership involvement in bridges/calls true for orgs that score high on Transformational Leadership/Westrum Culture?
In other words, are there ways for leaders to actually be helpful in these calls given the right cultural maturity etc
Ask the people doing the hands-on work what leaders can do to be helpful in those situations — before an incident takes place.
I'm not sure which position is more frustrating during an incident- the blunt end or the sharp end...by the time you're at that place in your career where you're at the blunt end, it's frustrating because you no longer know enough to be useful
I liked the "quick check of who is to the left and right" before saying that HIPPO presence is not positive
I just hit this last week when I was paged. 3 levels of hierarchy were “participating” by pinging me for updates and possible fixes … before creating a call with another 10 engineers.
@genek101, @allspaw: doesnt the comment on leaders not being useful in bridges go a bit against of what show in the phoenix project book where the main characters are always present in the incident and long nights? Or by leaders you really mean C-Suite and Executives?
…spraying drink all over laptop. 😂😂😂 (and for the record, I don’t think anyone is asserting that leaders aren’t needed on outage calls. But, yes, Sarah was deliberately portrayed as particularly unhelpful in the outage calls depicted in Unicorn Project, evidencing some of these phenomena.)
I’m saying that leaders’ belief that they’re being helpful rarely matches the belief of others who they’re trying to “help”
thanks for answering! indeed, most of the cases they just put more pressure and take the practitioners focus away of solving the problem at hand
Indeed, and often folks don’t feel as free to be as verbose with their colleagues when leaders are present.
…question I’ve wanted to ask @allspaw all year: Can you give an example of how you, as VP Ops at Flickr or CTO at Etsy, that you fell into one of these technology leader traps?
I learned this the hard way years ago and at that point made sure that I never participate directly in incident response channels, much as I thought I knew better and thought it was positive for me to be engaged. Wish I’d seen this talk then!
i try to listen in incognito, but refrain from direct participation. Is this similar to what you do @mik?
I always appreciate it when someone who knows the history asks "can you describe what you expect to happen if you do <whatever we're talking about>"
…well, wait until @allspaw describes what goes wrong with hands-on practitioners! 😆 😆 😆
…well, wait until @allspaw describes what goes wrong with hands-on practitioners! 😆 😆 😆
There's a very old meme from the British army - the only order a British officer ever has to issue is "carry on Sergeant Major". It separates the practitioners from the leadership....