This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-10-15
Channels
- # ask-the-speaker-track-1 (437)
- # ask-the-speaker-track-2 (251)
- # ask-the-speaker-track-3 (122)
- # ask-the-speaker-track-4 (136)
- # bof-american-airlines (3)
- # bof-arch-engineering-ops (3)
- # bof-covid-19-lessons (1)
- # bof-cust-biz-tech-divide (26)
- # bof-leadership-culture-learning (6)
- # bof-next-gen-ops (1)
- # bof-overcoming-old-wow (3)
- # bof-project-to-product (3)
- # bof-sec-audit-compliance-grc (11)
- # bof-transformation-journeys (4)
- # bof-working-with-data (1)
- # demos (57)
- # discussion-connect-february (1491)
- # games (41)
- # happy-hour (162)
- # help (96)
- # hiring (12)
- # itrev-app (10)
- # lean-coffee (65)
- # networking (16)
- # project-to-product (3)
- # summit-info (199)
- # summit-stories (60)
- # xpo-atlassian (1)
- # xpo-delphix (48)
- # xpo-gitlab-the-one-devops-platform (2)
- # xpo-infosys-enterprise-agile-devops (2)
- # xpo-instana (3)
- # xpo-itrevolution (1)
- # xpo-launchdarkly (10)
- # xpo-moogsoft (3)
- # xpo-muse (9)
- # xpo-nowsecure-mobile-devsecops (3)
- # xpo-opsani (5)
- # xpo-optimizely (1)
- # xpo-pagerduty (18)
- # xpo-pc-devops-qualifications (5)
- # xpo-planview-tasktop (4)
- # xpo-plutora-vsm (1)
- # xpo-redgatesoftware-compliant-database-devops (9)
- # xpo-servicenow (1)
- # xpo-snyk (2)
- # xpo-sonatype (8)
- # xpo-split (9)
- # xpo-sysdig (25)
- # xpo-teamform-teamops-at-scale (6)
- # xpo-transposit (4)
We’ve figured out that Management belongs on the Management Bridge while the techies belong on the Technical Bridge. Incident Command summarizes from Technical to Management and cascades requirements to the Technical bridge if it is inline with remediation-first.
If only management would stay on the maangement bridge....
I'm impressed by @allspaw’s ability to keep a straight face throughout these indictments on tech leaders management of incidents
“You don’t need this chart to ask deeper questions” — Spot on!
Same. Have noticed a little lag in last two talks.
Love this point! Just ask the questions, you don’t need the chart.
The amount of misleading on-the-fly interpretation of shallow data from charts of this sort is amazing, and I frequently have to monitor myself not to fall into the trap.
So, we are claiming that TLs don't know much about the incidents. And then, we ask them not to log on to the INC bridges.
I’m not claiming, I’m reporting what we’ve found. Didn’t tell people to not log onto bridges.
I'm sorry if I misunderstood. I've just had that impression from your talk, that you see TLs participating in incident response as an antipattern.
No, I’m saying that if it was an antipattern in the specific context of a given company, leaders rarely would detect that or people tell them.
Many leaders genuinely believe their presence is supportive and helpful - we see a gap between how others in the channel experience their presence, and how they think they’re being perceived.
This new way of conferencing is very tempting. Sometimes it's hard to concentrate on listening with so much interesting going on in the slack channel. Never had this issue when I was sitting in the large audience room and had no choice but to listen what the host is saying.
after verifying we don’t have any pitchforks
https://www.adaptivecapacitylabs.com/blog/2019/11/20/markers-of-progress-incident-analysis/
https://wallpapertag.com/wallpaper/full/a/2/0/569247-cool-terminator-2-wallpaper-1920x1200.jpg
Excellent how this directly follows some of @erica.morrison’s learnings!
“don’t capture what makes an incident difficult; only what was done to remediate” — Great call out!
The "How do i get people to stop asking me questions" approach to incident write ups...
I guess now it is time to massacre the practitioners who were laughing while you covered leadership? haha, awesome
@allspaw Do u have links to examples of poor and excellent write-ups?
I wish I did - as it turns out, our clients don’t feel comfortable publishing their writeups.
not sure if that was deliberate, but I love that this was opening chapter of Phoenix Project! 🙂
Yep! It's always BigIP in our world! :rolling_on_the_floor_laughing:
After the hosting team. Which is after the monitoring team.
Didn't @steve773 cite the issue with the Penn Station subway sign in his London talk - good example of fixing without learning
Thanks. But what's wrong with the sign? I'm guessing one of the things on it no longer exists?
Yeah, a lit sign they keep changing the bulb for, but it's no longer meaningful.
Here's the DOES version http://videolibrary.doesvirtual.com/?video=466913277
It's tough being 12 hours out of synch with the conference....
I talk a lot about this with people. My CTO gave a talk on it yesterday too. The “Automate all the things” trend is still too strong.
Yes!! Prioritize work on incidents when things are going well!!!
The interesting challenge is across the employee / contractor boundary…
Interesting perspective wrt A Learning Organization
“The expertise is coming from inside the house.”
BTW, what I find so exciting about this talk is that it reinforces the notion that we need a Structure where they enable the Dynamics where weak failure signals can be acted upon, aggressively, quickly, effectively (even generatively) — and for that matter, strong signals, like in massive outages. cc @allspaw
https://www.youtube.com/watch?v=2PjVuTzA2lk&feature=youtu.be
dont ask me, not the producer. Just sharing because the funny representation fits a bit the bashing John is presenting and some of the side discussions in the channel here 😛
I don't agree with the representation of a homogenous community of DevOps practitioners.
sorry, I guess my use of the word representation was misunderstood. I agree, we are not only dudes and the video could have been better on this. What I meant by "funny representation fits a bit..." is that they fit the description of Tech Leaders and Practitioners in John's talk as well as the finger point culture we were talking in the channel 🙂
Incidents are being prevented every day. We don’t recognize the events that don’t happen, but they’re prevented by people leaning on their experience with past incidents.
this the notion of the “virtual andon cord” that @steve773 talks about — make it easy so that any problem can be quickly captured, and mobilize leaders to quickly help. His stories of how this has helped in nursing and pharmaceutical labs are incredible. cc @jeff.gallimore
Do we have a virtual andon cord to generate learnings for this conference? 😁
Thanks @genek101 @nick.kritsky and @nickeggleston Please visit our site http://SeeToSolve.com The promo videos explains the idea of a virtual andon cord. That page is a little legacy, but enough of a appetite whetter that if there’s interest, we can pick up for a bigger discussion about those tools and others now in our portfolio.
I think in general we need to promote structure, rigor, and discipline. When will the industry move on from artisanal development and cowboy process?
The levels structure, rigor, and discipline is what is keeping you from having more and more significant incidents!
True that; it's hard because there's a spectrum: how much specialization should be done on the team vs how much should be consistent across teams
It's ironic - they embrace CI/CD and all forms of automated process.... but you tell them they have to do things the same way twice and they get all 'creative' on you...
Bryan is not normal… see comment on everyone else being wrong 😆
I'd rather sleep at night. Every developer should work with management who understand development and the things that negatively impact sleep.
sleep well my friend :sleeping_accommodation:
After this conference. Working CT and staying up PT is a killer.
“These meetings need to be prepared for (require skill)” - great point
My hypothesis is that Making Incidents Visible and part of Daily Work builds resilience and allows people to prepare to handle the big ones...
Ah, the whitepaper mentioned during Erica Morrison's talk. Definitely will be on my reading list. Thank you!
The "soak time" idea is an established brainstorming idea backed up by neuroscience
"one of the most mature things you can do is step away from the keyboard"
Yes, so leaders would do well to stop demanding and constraining how long an incident analysis should take
Instead, they should ask responders to estimate how long analysis will take.... (tongue firmly in cheek)
Collect data about outage - immediately. Generate action items - later
💯 Adding specific action items to a presentation :thumbsup::skin-tone-3:
At Google App Engine, we brainstormed about action items in the meeting, assigned "themes" of them to engineer / leads, and then brought the same group back 1 week later to discuss the (curated) action item suggestions. Hugely valuable to give the team that space to think and investigate.
"Solutions come to you when you're taking a walk, with friends" (when away from the screen), And I would argue - when under lower stress - when fight or flight is not in play.
in several places I've started walking meetings.
it's a kind of low level physical/cognitive activity that breaks the cycle of whatever stress loop people are in
I walk every day 4-5 miles. Some of that walk (about 1 -2 miles) i do with no input (no headphones). These are usually around 7:30 am, give your mind time to think through things without noise in your head is crucial to so many things. In todays world we are constantly bombarded with noise - have to take the headphones off for a while and let your mind work.
Great talk @allspaw! One thing we tried to overcome the postmortem "check the box" was to measure NPS around the post-incident review meeting. "How likely would you be to suggest your peers attend the incident reviews we hold in our company." 😁 It actually led to some really great feedback on how to make these reviews more effective. And, yes, it was scary as hell to our IC's when it was first suggested.
Great talk @allspaw! One thing we tried to overcome the postmortem "check the box" was to measure NPS around the post-incident review meeting. "How likely would you be to suggest your peers attend the incident reviews we hold in our company." 😁 It actually led to some really great feedback on how to make these reviews more effective. And, yes, it was scary as hell to our IC's when it was first suggested.
But once they started getting constructive feedback, they became some of our biggest proponents for it.
Thanks for listening, folks! And thank you, @erica.morrison for “duetting” with me again!
@allspaw How have you found documenting Red Herrings usable? I've never thought of documenting them, but I'm wondering what that would buy us. Thanks for the talk!
Understanding what makes following red herrings is incredibly valuable, because it can reveal mechanisms that can look like things they are not. People follow red herrings and rabbit holes because they appear to be productive. Only hindsight helps us see that they weren’t productive.
Also: stories of red herrings is fuel for genuinely compelling reading and stories that are told. This is perhaps the most valuable part, because it brings the reader in and requires the author to describe details what made it look like it was the right direction.
Join @allspaw at an AMA tomorrow 2:50pm PDT! https://doesvirtual.com/ama-links
OK, grabbing beer for my happy hour AMA starting soon, looking forward to chatting to people there 🙂 https://go.tasktop.com/DOES-Vegas-20-Mik-AMA.html
@allspaw says "Start tracking how often post-incident write-ups are voluntarily read by people outside of the team(s) closest to the incident." so simple, yet 🤯
One place I worked we had a "lessons learnt" database that was a mandatory part of the project process. At one point we queried how much it was used and discovered that the only people who had used it in the past 3 years were the people who posted their findings. No one had ever read it. Still, the PMO insisted on keeping it as part of the process and decided to make it part of people's KPI's to read it....
That latter point is why making them available and not ‘metricized’ is an important part. You want the analysis to be genuinely compelling reading, so people can build an expectation that they will discover new things they didn’t know before.
@nick.jenkins other "brilliant" idea is to store the post-mortems on the wiki space where nobody has access except for the IM team 🙂
We try and frame them as a story and publish them in slack. Everyone likes to read a thriller...
Come to my AMA today and I may have an answer! 😉
4I'm looking for the presentation from @dwayne, in dropbox, and cannot seem to find it.
Hello @annp. The slides for this presentation are still missing. Or I still cannot find it. 😞
Hi Javier - I believe that we will have them uploaded soon. I'll give you a ping.
Good morning, DevOps Enterprise Summit! Looking forward to an amazing Day 3!!! 🎉
@genek101: did you film all these transitions on the same day and change wardrobe in between?
YAS! "we deconstruct work so that teams CAN work independently"
(Something like that — three hangers with three pre-selected shirts, worn across many days. 🙂
“decisions can be made locally, without the vast escalations, such as to the CEO of the hospital system that Dr. @steve773 described yesterday.”
Soon we welcome @werner.loots and @ian.eslick for our opening plenary Q&A!
Structures are what we do; dynamics are what we get. Dynamics are emergent and can’t be perfectly predicted. You must probe-sense-respond.
I was amazed by @jtf observation of how the Westrum organizational typology model is indeed a description of how information / signals flow within an org — in other words, the dynamics.
Stories like these and the Steve Spear Idealcast episode really get me excited about growing learning organizations.
Team of Teams is a great description of information flowing to where it is needed, as per Westrum’s Generative culture.
So cool that both Westrum and Edmundson did most of their work in healthcare - shows the dangers of Human Debt in a much more evident way than we can see in tech
Our ability to handle complex problems is in part enabled by the 2nd Taylorist revolution where we standardize work through automation.
The darkside of Taylorism were a culture that prohibited learning at the edges. Now learning/automation is happening at the edges.
Freeing time and cognitive load for other more valuable problems
I'm totally going to use Momento as an example on a regular basis now.
We use Inception regularly to describe our worst architectures
Standardized work is fine for repetitive work (known unknowns, lean production), not so good for emergent work where experimentation is needed (unknown unknowns, agility)
Good Morning/Afternoon/Evening! Everyone to at #DOES20! Was great to be part of last two days! Looking forward to Day 3.
It was amazing hearing from @levi.geinert and @werner.loots talking about that small business loan program — they could apply for loans and potentially have it approved and have funds in their bank account within minutes or hours!!!
Such an impressive change of client experience!
I wonder what happened to Scheme :thinking_face: (which I learned from Dan Friedman CS311 at IU Bloomington)
IU! When were you there? I graduated December '98
No, I was a music major. 🙂 My wife may have, though. She did CIS via the business school.
Scheme never made it into the professional programmer toolkit; the ecosystem was very geared towards education and hobby programming. Clojure did make that transition (Rich’s original intention) as did Common Lisp for a short time in the 80's/90's. I talked to Sussman about replacing Scheme with Python at MIT some years ago and he was supportive - his comments amounted to “an elegant tool for a more civilized age” or rather assembling and managing systems of components is more important than expressing core concepts perfectly - and Python does provide lambdas so other than losing homoiconicity, you can still do alot of what made the intro Scheme course so great (which I both took and later TA’ed with Sussman).
That's what makes transformation stories of such organizations more interesting!
How did you get the appetite for improvement, for change, without a burning platform @werner.loots @levi.geinert?
When we’d expect the dominant architecture to fight against the change.
Definitely not easy, with the studios we found business lines that believed there was in their business lines. But this is often the hardest part. @ian.eslick might have more to add.
I’ve benefited from groundwork and leadership laid over the two years prior to my joining the bank (I joined about a year ago). Our new CIO Dilip Venkatachari worked closely with our Digital office to make the case for transformation. They provided some high level guidance like public cloud first, microservices, data APIs — and many of the folks in this conference made the overall value case for this transformation — the challenge now is turning that high level alignment into changes in process, tools, and culture across the bank. It’s hard, but definitely easier when you have strong top-down support.
One area we are looking to improve is how we communicate to our business leaders and make sure they understand what we’re trying to do and that we highlight the value as we create it. It is key to earning trust across the bank the next few levels down from our managing committee
it will never be this slow again < great insight, great framing of the need for learning & change
Some photos of @werner.loots and @levi.geinert: check out those crazy red Nike Air Maxes that they wore! A signal proudly shown by everyone involved in the effort.
What percentage of customers were digitally active prior to covid?
Great example about how preparing for learning & change increases resilience, your ability to respond to the unexpected.
@werner.loots @levi.geinert Can you post a picture of that amazing flyer describing the values behind the red shoes? I loved reading it in the box you sent me!!! ❤️❤️❤️
And congrats on your new role, @werner.loots — a testament to the value the transformation group created!!! 🎉🎉🎉
PS: @werner.loots’ comment about transformation being a combination of “careful internal planning” and wild “external events” was amazing…. Fortune favors the prepared…
Just over 70% of customers were digitally active before COVID, so that number increased somewhat, but actual activity a lot more.
Great example of swarming a problem. Reminds me of the Toyota supplier fire example from High Velocity Edge.
Mainframe engineering leader: “what we did this week would normally take 3 months”
Also reminds me of the opening of The Phoenix Project and The Goal where they respond to a crisis and ask “why can’t we be this fast all the time?”
That is so great, @werner.loots — what an amazing statement of values, modeled so brilliantly by leaders! Thank you!
“we had no idea that we could move so fast — that we could go from concept to production in one week” “what is we dramatically invested in technology?” “what if we doubled our technology next year” (!!!) @ian.eslick
Great to actually fund success rather than putting money into shoring up failing programs.
I love the language @ian.eslick is using — I was joking with him and @werner.loots that he’s cracked the code of framing tech debt very precisely using the language of economics. You know… “banker talk.”
1Shouldn’t be compromising strategy around architecture, as we wouldn’t compromise security
“how do we quantify our book of tech debt that we’re carrying around from year to year.” <--- nice! (banker talk 🙂
Make the right thing more attractive < something I’ve heard more than ones this week
2The rate is less important than the fact its compounding. @genek101 has used banker talk to find a way to our hearts!
This is great - I was in a meeting this morning where the same question of accelerating came up.
minimizing project costs is like a manufacturer dumping toxic waste -- I love this.
I love using the scorecard as a mechanism for making tradeoffs explicit. Make the work / tradeoff visible!
“I’ll spend $2MM right now, but we’re deliberately holding off on $20MM of investment that we’ll have to spend later.” So good! And so precisely phrased!
Explicit conversations for the win! And that project scorecard provides a nice structure for transparency and curiosity.
It’s definitely gaining traction; the real proof will be whether it is driving the right behavior 6 months from now when it’s just another process step…
Thank you, @ian.eslick and @werner.loots!!! I’m again dazzled by the precision of framing the positive value of great architecture and negative externalized value of tech debt!!!
I'd love to hear/understand how to price the value of the intangibles when constrained by traditional business case frameworks that want to tradeoff opex/capex...
Thanks @scott.prugh at present we aren’t trying to price intangibles, we translated the issue into a senior leadership alignment around strategy. We use the scorecard to document alignment with the strategy and the use of ‘short term throw away spend’ and ‘need to invest this in the long term’ is a gambit to start surfacing our backlog of debt to leadership. Will let you know next year how it played out.
we love comparing notes < great sign of a learning mindset
In the old days, eBay used NPV to prioritize projects. It was well-meant, but horrible in execution. Gameable estimates + the "toxic waste" of technical debt and poor programming practices.
@rshoup What did eBay replace NPV with? Any insights on how to engage finance in a more enlightened practice?
It wasn't replaced with anything principled, I'm afraid. I have my own ideas around option theory and portfolio risk management, but nothing concrete to offer yet :-).
@rshoup I agree. The domain of finance is NPV and capex/opex tradeoffs. It fails to capture the other capabilities.
Except ... Finance people are trained in option theory and managing risk, and have to think about whether to hedge against high impact, low probability events. I've found that CFOs and finance people are great allies in advocating for technical debt reduction. We just need to help them frame the problem with an investment / risk management lens.
or....hold my beer!!!
my dog it loving you ripping up the railroad tracks!
My favorite opening.... @jonathan_moore @michael_winslow
Love the tooth pick @michael_winslow and the bandana @jonathan_moore
well done, need to buy that hat as well haha
Costume expectations have been set for future talks...
Winner of the best hat in a devops talk goes to....
now I need both an engineering hat and a wookie mask.
@michael_winslow and @jonathan_moore bringing the "cinema" to the virtual stage. Great story telling.
This is eventually going to be a talk about SAFe release trains right?
Ha, sorry. I did actually buy it from the link I shared, but didn't read your profile name in detail before I responded!
New York - over-engineering leading to... re-engineering and re-work
I always love seeing people use Concourse CI - but I would definitely go down in options too
The rockstar engineer did not succeed in launching broadgauge
OMG! My two passions collide in one talk!! As a model train enthusiast, I am LOVING the railroad comparison!!! So fantastic!
Nice parallel between recording the context and Allspaw's talk yesterday
I was loving when @steve773 was talking about the "Dominant Architecture Emergence" yesterday!
Gosh, I’m interested in this - What is the timeline and lifespan of an ADR. I had an org that tried them and it did not go well at all 😞
There's a longer writeup available here: https://www.infoq.com/articles/architecture-guild-800-friends/
1sounds like our ARB (architecture review board), but a bit more hands-on. Like it!
sounds like our ARB (architecture review board), but a bit more hands-on. Like it!
HA. Lovely, gonna save your quote to share in our next ARB call haha
Best part of group based ADRs are when someone (new typically) comes along 18 months later and asks why the "heck" did we do this and not that. It's all recorded.
I think ADRs are more akin to an RFC process that you see in many OSS communities like https://github.com/rust-lang/rfcs, https://github.com/apple/swift-evolution and others. It also reminders me of Tanya Reilly's blog post on how SquareSpace does a similar process: https://engineering.squarespace.com/blog/2019/the-power-of-yes-if
ADRs optional to describe good practice, or mandatory with process controls around them?
It varies. The recommendations are written like IETF RFCs, where there are a mixture of MUSTs, SHOULDs, and MAYs in each ADR.
The strongest common level is usually SHOULD, based on our current company culture.
For me the primary benefit is not ending up years down the line wondering why on earth we chose tool x
This is an interesting question. We generally don't revisit to update consequences, per se, although that's a good idea. One of the reasons to record context is that this can be a good way to detect when your context has changed and a new decision might be warranted.
Maybe an automated reminder for a 6 month update to actual vs expected outcomes?
@jonathan_moore @nickeggleston the revisit is what turns trial and error into experiment. And experiment is way higher yield in terms of ignorance becoming understanding.
What I've found to work, is to use the quarterly OKR as a trigger to revisit the continuously emergent architecture envisioning / ADR, if not sooner.
1That reminds me of business case submissions and reviews that often include estimates of costs or savings... but does anyone go back and validate that later? :thinking_face:
The time it takes to finalize an ADR varies; sometimes a couple months of asynchronous working/conversation. So far, we haven't been doing it long enough for an ADR to expire or be obsoleted, but I expect we'll do something similar to what the IETF does with Internet Standards (marking an ADR as "obsolete").
TIL, just now: I always heard it as "fist of five" as well, but did some Googling and found it as "fist to five" (representing 0 to 5): https://www.lucidmeetings.com/glossary/fist-five
Did you have intransigent stakeholders, and if so, how does this system deal with them? (ex: Some person with an important title says, “We are using TravisCI and I’m not budging on that”)
would be cool to move all that into gh as well to gather all info went into the decision into one place
for the voting thing: how do you take in consideration cases where the folks dont know better? If they only know how to use a hammer, even for screws, they will likely vote for the hammer no?
tagging @jonathan_moore and @michael_winslow 🙂
There is a danger of large orgs get more votes out of sheer numbers. Regardless of the solution. That is when other orgs have to be able to make good arguments TBH. There was already a very large Jenkins contingent in our case, but the argument for Concourse prevailed in the end.
But you can get a lot of votes that "This will never work for me" which can sway the Hammer and Screws solution.
thanks for clearing that out Michael, appreciated!
Do you ever explain that they SHOULDN'T do that in any CD platform?
Sorry, I was answering other questions. What's the "that" you're referring to here?
Practices that just don't align to a CD flow. You should see the questions we get. :)
And no worries, the question firehose wore me out yesterday.
I’m always pleased to see people using Concourse CI and especially when people select to standardize on it ^_^
My snarky reply is that most executives don't spend time in GHE in practice. 😉 But we'll talk about leadership buy-in in a bit.
this was a particular challenge I’ve faced - a “director of engineering” cared a lot about using a particular tool
Architecture tooling choice by consensus is a rare things to see. So many architects are used to dictating to teams what they will use and are consequentially ignored.
the famous Ivory Tower Architect, who define the awesome world without touching the ground and listening to the actual users
Reminds me of using the Resolution protocol: “What will it take to get you in?” https://liveingreatness.com/core-protocols/resolution/
Argument I always hear from CTO is that apps all want n+1 solutions in the past
YES - ADRs to document team decisions to change default branch names to something more inclusive
I'm curious to know, for the ADRs, who are the people that vote? Is it anyone? Is it a select group of people?
@mboudreau327 and all developers have a say?
We didn't start polling until we had assembled a lot more context people could read, including a clear set of requirements, analyses of how each of the proposed solutions did or did not meet the requirements, etc. It wasn't until we had actually collected all that info that we let people open PRs to propose/advocate for a specific solution. We relied on the issue mechanism we showed as a way for people to ask questions and gain more context.
the puns should continue until the naming of things improves
oh no… I can see this going off the rails…
TIL: Resolution protocol! A quick scan gives me the feeling they are similar in spirit.
awesome thanks @jonathan_moore. Are you as the Chief Architect the tie breaker? I image you want someone to have a final say.
In this case, yes, we got down to two choices that seemed to have equal support/acceptance, and I made the final call.
Awesome, thanks @jonathan_moore
But there have also been cases where consensus gets reached on one option, and that's always my preference. I have not yet run into a situation where the broader technical community wanted to choose something that I thought was actively problematic enough that I needed to intervene.
I think providing the context behind the decision is key, that's what we miss often ourselves and I believe that creates the friction. If people have a clear understanding of the context it's often they'll come to the same conclusion.
What I eventually came to understand was that people often agreed about assessments: that option A is better than option B for characteristic X, and option B is better than A for characteristic Y. But people actually disagreed about whether X or Y was more important.
“choo choo choose… concourse” - gotta give him credit a little bit tho for real
It’s weird that this talk is about trains, Concourse gives you a great runway for airport jokes
And you wonder why some towns actually formed where they did... interesting
one of the biggest obstacles
I try to point out far more interesting work they COULD be doing.
I would love to be able to stay to do interesting work, but I keep being asked to work elsewhere...
Generally, what we found was that we had a bajillion separate installations of CI/CD tools, and it was rare to find someone whose full time job was running each of them. So fortunately, we didn't really threaten anyone's job directly, because we were just freeing up their time to work on the other stuff they were already working on. In the cases where there was a small team, we encouraged them to become the Concourse experts for their department and move up the stack to focus on implementing better CI/CD practices instead.
We did something similar. They either joined us in CD Platform or joined a product team.
yeah that's a good point, that's what we saw as well, many teams didn't really enjoy running their own Jenkins
Yep. And when we took those over and started systematically migrating and shutting them down, it really made cross-org collaboration on good CD practices a lot easier.
Great example of a vested interest fighting change. Local optimization for Erie!
This session though, is really driving home the importance of training for an organization... #dadjokes
The train analogy is particularly good because its easy to visualize. When I talk about work problems, I don't think I communicate in this type of clear and comprehensable way. Good to think about over the next few days.
*R*einforcing *O*perational *W*ork *D*iscipline *I*n *E*nterprise *S*ettings
@jonathan_moore @michael_winslow what was your process and tooling you used for your ADRs? Once an ADR was "complete", what is your process to revisit or update it?
I love this focus on communication. Lots of change efforts suffer because the champions of change suffer from the naive realism fallacy and assume that their view is not just correct but obviously correct.
Generally, we kept the ADRs in our source control system, and develop them as pull requests/issues on Markdown files kept in the repository. Once ADRs are finalized, we have an index of them in a Well Known Wiki Location where links get published. We haven't been doing this long enough to revisit/update anything yet, so...suggestions welcome!
maybe have the Wiki as a static website generated out of the MD files, triggering a redeployment through the PR in GitHub once you update the info in the MD file?
@jonathan_moore we have a very similar process ourselves but it's isolated to the one CI/CD platform, bringing in the RFC concept. We've recently been inspired by the https://github.com/apple/swift-evolution. One nice aspect of the metadata at the top is that you can build a nice index of the previous and current proposals: https://apple.github.io/swift-evolution/
We don't generate ADRs very quickly, so we haven't developed any special tooling for this. Usually updating a Wiki page with a link (and maybe an exported PDF) has scaled pretty well so far. But I like the idea of generating a site from the repository sources!
One thing I continue to struggle with for these platform adoption initiatives is how much to rebuild documentation for an internal context vs asking developers to read existing documentation. I often find the latter fails regularly and almost always necessitates building the internal documentation/videos/etc. It takes a huge amount of time though 😞
and in my experience, to make the 'documentation' reentrant, living, emergent, with learning and evolution
same here. A lot of people dont like reading the perfect documentation I spent hours writing and they ask for a training session instead (which is not scalable in a big company)
“don’t want to be left behind” < early majority reasoning (via Crossing the Chasm)
1I've heard ADKAR in a training curriculum context, this is the first time I've heard it about change management at large. Now that you say it, it feels obvious to use this in change communication - of course, a lot of change communication is about education! 🤯
One of the things I should mention: we partnered heavily with HR on this, particularly with someone who specializes in organizational change management: she's really the one who introduced us to ADKAR and helped us think through it. So I've actually only encountered it in a change communication setting. 🙂
Wow that is interesting @jonathan_moore - HR is often an underused partner in transformation but that is a great way to engage them.
Yes, we had two HR folks that were first-class members of the program team that designed and ran the transformation. Super valuable, would include again. 🙂
We ended up starting with the existing documentation as a "101 level" course, and just added documentation as a "102 level" course that talked about any differences/info that were specific to our context.
ohhhh that's a great idea @jonathan_moore, so basically accumulate existing public documentation into a "path" and then provide your own context on top of it... nice! Thank you!
Right. So, for example, Concourse supports multiple backends for handling secrets. We went with an existing external tutorial for the 101-level course, but the 102-level course talks about the specific backend we use at Comcast.
“Your team didn’t get any short term value from moving to ConcourseCI — it’s a lot of short-term work for no apparent short-term gain…” And yet there’s so much obvious value in standardizing “undifferentiated heavy lifting”
Jenkins had its issues, but did work well for many of our teams. There was an existing internal Concourse community run by at least one of our distinguished engineers. They had a very mature implementation and knew how to demonstrate the value very well. The Jenkins teams were more spread out.
That’s interesting, we are also trying to standardize on CI tooling, does it follow that the tools used by the teams that are doing CICD really well would be a good factor in the tool selection? Or am I reading too much into that?
@michael_winslow did you all build some kind of framework/library around Concourse? Or did you have each team define their own pipeline and scripts?
As an FYI we have a library used by everyone in Jenkins today.
@arthur.maltson We've taken a community approach. Teams are responsible for writing their own pipeline and scripts, but one of the things that we like about Concourse is that it has a very modular architecture. We have an inner source repository called "Recipes" that hosts a number of reusable scripts ("tasks" in Concourse parlance) that teams can mix and match from.
@jonathan_moore interesting, sounds like where we initially started as well but we found teams got very "creative" in getting to production and would skip critical steps.
who wrote all that test automation? Was it the teams writing their own or part of platform capability?
Very curious here. My feeling is most should be on teams
Tests are most effective when they are the documentation for how the system will work. If someone automates testing for me, then each of us is writing code based on our understanding of the desired outcomes. That doubles the chances of misunderstanding and the possibility of two different misunderstandings. It also delays quality feedback.
Bryan does that include compliance testing though? Should that be done by a governance team?
I'm only referring to behavior, not compliance or security. The teams cannot define that. Compliance and Security do. The platform should alert to issues as early as possible though.
So, how can the platform make it easy for Compliance to inject audit gates?
This is a question that is still a matter of learning for us
This last talk just gave me a method to start the conversation I think
@jwillis gave a talk to us and talked about some concepts there I really liked.
@jonathan_moore, @michael_winslow: awesome talk! Could really connect with what I am going through right now and took plenty of notes. Would love to connect and swap stories. Since you folks announced it publicly here, I will hold you on the offer 😉
Another great example of measuring the externalized costs of every team making local decisions — did I get that “banker talk” right, @levi.geinert @rshoup?
I personally come at this from environmental economics more than banking -- pollution, habitat destruction, climate change, etc., are the ultimate negative externalities.
The global benefits of standardization (architecture) are not always apparent locally
this is sooooo true!
Does anyone have good strategies for this? How do you win them over?
agreed. Focus the story on the ability to spend their time on innovation and not the undifferentiated heavy lifting.
Would you like to hand craft a Jenkins for 3 weeks or would you like to push this button and be writing come app code?
Follow the paved road and the architects and security will be more open to signing of your production release
We definitely ran into teams who said "We're already doing CI/CD on <something-other-than-Concourse>. Do we need to change?" I found the most effective reframing was that this was not actually about that team, but about something the larger organization was going to achieve together. "It's not about you, it's about us". That helped a lot.
It’s also about future you, or future others moving to different teams
@jonathan_moore @michael_winslow I love the conversational presentation style - thank you
@jonathan_moore and @michael_winslow - Love the talk and the theme of the talk...would love to connect and talk further as well!
Sure! Best place for me would be to find me on Twitter: https://twitter.com/jon_moore
This is a couple of years ago at this point, so my recall might be a little hazy! But one main area was that we liked Concourse's multi-tenancy support/architecture. But the reality is that we quickly determined, as we looked through all the proposed solutions against our requirements, that we could probably have been successful with any number of specific CI/CD platforms.
@jonathan_moore - We're working on standardising the undifferentiated heavy lifting and would also like to share stories. Great presentation, thank you.
That was a great talk 👏, thank you!
Great talk! @jonathan_moore and @michael_winslow !! How did u come up with the railroad metaphor?
Hmm, I'm not sure I remember how the idea came to us! But I got to do a lot of reading/research about the history of trains as a result. 😄
There's a training SaaS company in Indy that has built their culture/mission around this phrase "Do Better Work" ... "When we do better work, we live better lives." They've put out some good content in this domain. https://dobetterwork.com/
@jonathan_moore @michael_winslow thanks for the talk, are you only using Github to manage the tracking over ADRs? I feel a gap when trying to follow-up from analysis to implementation, any thoughts on that?
I've used ADRs in code and any follow up goes into the same repository. It's not a static point in time document but lives and evolves with PRs
we used some "central" repo with issue to track similar request... but dedicated repo is much better approach
We have one ARCH org for the whole Architecture Guild, and then each working group gets a repo.
WGs often produce multiple ADRs over their lifetime, so yes, they are living repos.
@drjgoosbysmith on the need to define variables clearly and precisely and correctly — oh, the memory of the pain of getting this wrong is all coming back to me now, @nicolefv!! 😆 😆 😆
to get the outcomes we want, we need to know what our goals are and be clear about it ! @drjgoosbysmith is one of the best
I love that analogy, and diverse gardens are better for the environment and wildlife. Love it.
What are the various attributes being measured when evaluating diversity... important question related to definitions
Love this — I often conflate these in my head, and @drjgoosbysmith examples are so stunningly effective at separating the two!
Haha, you definitely can't do that LOL 🙂
I’m really appreciating this extension of the gardener metaphor!
(In my introductory remarks, I admitted that when I first read her book 3 years ago, I had conflated them in my head. But not on my second reading!!! 🙏
In the book I think I used an airplane metaphor. I came up with the garden later. And it keeps developing!
(Also very much appreciating that the gardening metaphor was also mentioned by @david627 and @jessica.reif on Day 1.)
and you too! I'm so grateful for how much you've taught me through the years. you helped teach me so many of these things "before it was cool". a queen 👑
I am so proud to have @drjgoosbysmith. presenting her important work at DevOps Enterprise!
two presentations in a row with very well selected metaphors for simplification!!!!!
"Some of us haven't had a hard time staying at home because we live very quiet lives" - so very much this
Great presentations this morning. Loved the Railroad story tremendously and the current one.
From clearing toxic waste to building train tracks and now beautiful gardens. 🙂
I love how in @drjgoosbysmith book describes these orthogonal dimensions — splits all those fuzzy notions of inclusion and diversity into very precisely defined constructs.
It’s just so much to think about. That’s why I say that the technical mind is so needed in the areas of diversity and inclusion!
Love Charleston!!
(isn’t it kind of interesting/funny that the last talk about was about benefits of standardization, you know? 🙂
I am in awe of you. I need a Dr. J action figure. And one for my three boys.
Both-and thinking -- avoid false dichotomies. Love this so much, @drjgoosbysmith!
I'm loving this conversation. This quarantine has made me realize that even if I am uncomfortable around people, I still prefer that than completely isolated. Need some of that external energy
awesome explanation on minority/majority terms!
"Just b/c I'm in this <insert color> shrink wrap, you don't know me - you actually have to talk to me."
‘People are a wonderful mosaic. We are all complex hybrids.’
“coming soon to the marketplace!” “Generation Alpha” — all born in this millenia
I've got 5 of them 😆 Doing my part!
going from 2.5M born to 25M after quarantine 😆 my wife is a birth doula and she has been buuuusy...
"we should have an equal chance to succeed, have the things we need, get mentoring..." i love this
(I admit that I was embarrassed that when I read her book, I finally learned that Ubuntu was more than the name of a Linux distro. 🙂
Yes! That it is based on the African philosophy. Thus, it’s open/non proprietary nature. It is for ALL! Don’t have to be a rich programmer to develop on that operating environment!
Honestly - the reason I seized this career is because I felt so included at this conference 2 years ago
I came to the conference as head of QA. I left with a passion to make peoples lives better and an appreciation for the entire value stream. I joined a “band” of dojo practitioners. I paid my own way to visit target when they wouldn’t pay. I stood up a dojo and am now head of enablement (even though my title stay says QA). I found my peeps :)
Oh, you’re THAT Chris! FCA == “Fiat Chrysler Automotive”! I loved your presentation in 2018! Congrats on all the amazing achievements!!!
Gene - I seriously owe a debt to you and the community :) keep doing great things
Exactly. We are indelibly linked? To quote Rev. Dr. Martin Luther King, Jr. If you hurt, it impacts me too. So I support you to be your best. And vice Versa.
I have two iGeneration people in remote schooling today. 10 feet away from me. I am witnessing this RIGHT NOW.
"Nobody cares about what you know until they know you care!" - Former ATL Falcon Buddy Curry
I wish I could fully articulate how much that still affects me almost every day, if not every week
I learn something new from Dr. J every time I speak and listen to her. she's so good.
@drjgoosbysmith Any ideas on an example of a community that is diverse but not inclusive? seems like an unstable/unsustainable environment?
I think we see that in the workplace all the time. When we get back to office, scan at lunch and see if you don’t see grouped like “flowers”.
Absolutely. Very unpleasant. Because you get all of the ideas and perspectives from the diversity. But...if people don’t feel included, and they aren’t connected with colleagues, AND they don’t feel cared for and they have no mentoring...that’s a SAD garden. You’re so right. High turnover. Low engagement.
@drjgoosbysmith - You should win a Nobel Peace Prize.
I gotta say, I am so happy and impressed to be a tech conference where people are seeing this talk and are so engaged
So unbelievably rare. DevOps people don't know how good they have it :)
I have so much to learn and I am glad that others are showing interest
“Humans being” :heavy_heart_exclamation_mark_ornament:
This lineup today is championship caliber 🏆🥳 on how to blueprint a change in your org:star-struck:
"Witnessing inequity decreases trust and shows that the workplace is unfair."
“Visibility and reward is what makes people feel included.” YES.
If there's one thing this journey has taught me is that everyone on the team has something to contribute. "10x" developer isn't the goal. People who aren't as strong in coding have other perspectives that make the outcomes better.
Starting from a place of genuine care about all not just those who are similar is key
It is one of my pet peeves that we call people resources...can we start small and just stop doing this as a community?
I have been trying to do that myself. It's not easy, because it is everywhere in our tech community
That was so valuable. Already shared it with our diversity team for sharing out with the whole company, as we make at least one diversity topic required watching/learning each quarter. I got a chance to learn a lot on this topic recently, and some very specific ways that it relates to software teams and distributed teams from Denae For Robinson and @nicolefv: https://projecttoproduct.org/podcast/dr-nicole-forsgren-dr-denae-ford-robinson/
So good @drjgoosbysmith - thank you!
Also I liked your explanation on minority and majority. I once had a teacher(black) in high school also say she doesn't like that. Her explanation was; if you take all the different "minorities" they will make the majority. We shouldn't refer to ourselves as that. One of those things that just stuck with me.
Thank you assembling such a unique, intelligent, logical, thought-ful, and diverse group. That combination is MUCH more needed across EVERY human stratum. I’m honored.
so excited to see @jennifer.hansen there 😄 😄 😄
@chawklady was the very person person at DOES to stop and talk to me. She was very patient.
Thank you for the kind words @bryan.finster and for everything you have shared with me. You have DEFINITELY paid it forward. 🙂
I LOVE your comment. Yes! It does NOT take a million-dollar initiative to make an organization more inclusive! Every interaction matters.
I think this community does a pretty good job of living this @drjgoosbysmith
Thank you @nicolefv for suggesting Dr. J’s book to @genek101!
👏Amazing ! The best one on this topic that I have heard... Thank you, Dr J. Would love to have you address our org
Thank you @drjgoosbysmith - what a wonderful talk - I'd never heard of "intrapersonal inclusion" and am going to have to really sit with that
Exactly! We hear our own voices (telling us about being included or not; telling us how we feel about including others) loudest over all other voices.
I'm just glad there's a name for it! "Create your own psychological safety; notice when you are / aren't"
I'm not even going to try and take notes during this. Just going to maximize the screen and listen.
my fave phrase: “natural experiments”. doh. or is it “natural experiments” @nicolefv
As a hard of hearing person and an introvert, I've been able to get more out of this conference than I could've imagined, thanks to the format and the different modes of communication (including a Livetranscribe app I use on my phone that helps make sure I don't miss anything)!
mind sharing the app?? android or iphone? I feel like I misst things since I am also kind of hard of hearing
It's Android LiveTranscribe. I put my phone in front of the speaker. Helps me relax a bit. Conference calls used to drain me out. Now, not so much. Here's a screenshot from Nicole's talk going on just now 🙂
Dr. J and Dr. Forsgren - unbelievably proud to have b2b presentations by these amazing women!
This quote! "We're doing very well, we're barely hanging in there" 😂🤪
I have a love/hate relationship with WFH - ok with not seeing my co-workers (until I miss them)
WFH != WFH + Pandemic 💯. I think a lot of managers forget this, everyone's life is upside down, regardless of how composed they seem to be.
So agree - this is not flexible working, this is a weird pandemic version. We can get the real potential when this passes
Exactly, it's so hard to judge what WFH is actually like (never actually did it full time). Having young kids at home is hugely disruptive.
I did 6 years remote in 2004-2010. The tools have not improved a lot (okay, skype, Gdocs, Miro). But now the "normal people" are ruining my calendar by constant calls.
I can relate to opening I never want to go back I miss my co workers
I am missing an office beer tasting for this virtual conference.
There are 15 at the office because 5, including me, cancelled
This talk was what I used to say to people "but you HAVE to make it to DOES, @nicolefv is presenting the ONLY data there is on this"
wow... my two keep my hands completely full
how do you do that, wow
boundless energy clearly 🙂
Microsoft Office team: people are working earlier, working later, even through lunch. Amazing stats.
Yes - work expands to fill the space. No more lunch dip.
Repull of data from March to Sept is where you will see that.
Sept to Oct will show the events of Virtual/Home school.
The press in the UK is on and on about how there should be a drive to go back to the office as people are working less. Lies
I am having to tell my teams to take time off, not to over work…
Work expands to fit the space and we fill our calendars with no good reason other than the empty gap
This is so useful @nicolefv!! So glad you’re sharing it out
@dana.finster @subhashree2005 @nepobunceno our leadership needs to see this.
yeah totally can attest to working earlier and longer and all the time 😞
open source project creation equals what? more commits? more new projects? both?
My teams say they want to work more because they have already seen all the shows on Netflix… lol
crazy numbers, thanks for putting this together @nicolefv, very insightful !
more open source -- both activity (commits, etc) and creation (new repository creation)
@nicolefv but... how are they feeling? crispy? well done?
also lol when people would say "but surveys are dumb we need system data!" I love you all for saying "but wait what about the people!!!"
workers fill the space ➡️ workers work longer hours, therefore creating more space ➡️ companies fill the new space 🔁 lather rinse repeat
I'm definitely in that 1/3, feeling completely unproductive most of the time with this pandemic.
probably too much to ask to bucket into parent vs. non-parent
School age kids now in virtual/home school AND people who have parents who are high risk...
I think with parenting you would have a great amount of info to take in. Age of children being 1 of those huge factors.
I think this is important to remember. this is truly a wonderful thing for some people. let's not forget them
We're an entire neurodiverse family - I'm on the spectrum, kiddo's dad Aspergers (programmer extraordinaire) and kiddo (9) is HFA - they both need major distraction - child can only read/do homework or even Minecraft if Netflix's Horrible Histories is running in the background
@nicolefv btw, love your camera setup, I'm guessing you were using a DSLR? Love to bokeh
We just don't know what others are going thru and how the pandemic impacts them.
Worried about being judged, now we can disclose more - this has developed psychological safety to some extent, however I'm not sure all orgs are continuing to support this effect.
I think combining this with Dr. J's diversity and gardening talk - we need totaylor the answer to the person. Feels like we've run the "everybody must work in an office" and now, "Everybody must work at home" extremes ... the answer to what people need is.... it depends on the person - maybe the future is both-and
So insightful to talk about these elements of personal productivity, esp as relates to neurodiversity
Boxing breaks on Friday lunch time (as long as the pandemic dont close the gyms)!
I used to have walking 1:1 with my team members and we did laps in the office… to help get steps in… now we do walking 1:1 in our own neighborhoods while talking on Zoom.. nice to see the background change and see the outside world.
I love this! Always looking for creative ways to build teams and have better virtual meetings
...they’ve always preferred WFH, and they hope they never have to go back
We have more meetings. They are getting shorter. More context switching. No breaks.
cooking your own lunch is a pain sometimes indeed haha
completely privileged to have built a home office to specifically address some of these
I've built one too but it doesn't help with the kids 😞
My kids have to go outside the house and knock on an exterior door. It's pretty effective.
I think Miro etc does create more safety for some of those with more introvert characteristics
there are companies that didn't even have laptops 😮
So hard parenting and working. But we are getting into a nice rhythm... 7 weeks later?
If I hadn't switched to "unlimited" cable internet back in Jan, my 75GB a day would suck...
Re: too many meetings in the era of Covid WFH, sometimes we may be trying to overcompensate with the need to always be ON, which can be draining
Attention is a resource. Virtual meetings are exhausting in part because it keeps taxing our attention.
for those with kids, managing their online education is also a new task shifting item…
Too many meetings has always been a problem...twas ever thus
I always introduce my house friends when getting "into the picture". I'm in my home environment, they are part so why not introduce them instead of try to "hide" them. A lot of my colleagues know my cat, she's really a star getting into the picture 😉
Me too! "These are my dogs. Hang on, they think it's play time". Nothing like playing fetch while talking CD.
I introduce my kids on calls with my team. The kids want to know what I'm doing all the time and they ask "how are your team doing today?". I spent most of my time with my team, they are my social circle but my family is who I am.
My publisher asked if I can't write less about the challenge of life/work in light of homeschooling in the book and I fought them for it. Clearly unchilded to think it's a minor point
so true on how this impacts glue work 💯, I always found glue work to be much easier to do in person.
the famous: yo need to be in the office so leadership see you there and know you are working and remember you when the promotion discussion comes (like, really?)
@nicolefv the way that you work with data - it is amazing. The attention to details, delivering the outputs. It’s like I am watching DORA presentation again ))
Team Gratitude in every retro can not be understated IMO
this is a really interesting idea. I try to do this myself and tried to get my daughter to, but interesting to do it with the team.
I think this has been hard even for some of us that prefer to WFH because we could fulfill any social needs or get outside of the house in our own time (pre-pandemic).
I’ve been doing walking 1:1 in remote 1:1s. Funny it took so long for us to try it given that we did walking 1:1s while in the office.
Team Gratitude… Time to start using those internal “thank you” mechanisms and give thanks…
been advising clients to force people to take time off, minimum vacation policy
THIS --> watch out for ppl not taking vacation.
Dashboard of team mental health? • Cumulative PTO balances • % time spent in meetings • Other measures?
I had all of mine killed. No Alaska cruise for me. Now I'm trying to make it up before I lose it. I still have 26 days to burn before 2/1
Ooof - be careful not to turn people into numbers either (PTO days taken?). Lack of PTO is an indicator to go deeper and check in ❤️, not an numerical score to manage your "resources" 💡
Agree 100% was wondering if that could be one (of several) key indicators that WFH mental hygiene is off, especially for people that don't know how to raise their hand and say they're struggling.
We have two cultures. The West coast has "Unlimited PTO", which is a management scam to prevent PTO. I have a set amount of PTO.
Highly recommend trying to find the time to do the Yale Science of Wellbeing course as it goes into amazing depths about positive psychology and ways ahead to prevent burnout
Watch for team members who are forced to take time off because they begin losing it… (points to self)
It's about starting to re shape the way we work and coach our teams for the hybrid work environment, which is here to stay
DevOps: A Humane Way to Make Tech Sustainable @nicolefv
We’ve done Watch Parties (for a recent retirement) and did the Mystery Science Theatre 3000 through the entire movie…
@nicolefv is to me one of those people who has single-handedly reduced HumanDebt in SO many organisations with her work it's insane, With all we know we should all be working on it now though
I saw a team do flower arranging. it was fun, creative, and left something beautiful in everyone's homes
they should do it after watching @drjgoosbysmith’s talk! Put it all together 😄
"we can never do college admissions without the SATs" "Oh look, we can!" :face_with_rolling_eyes:
We did a game night - Scattergories and Pictionary. It was hilarious and also we learned more about each other’s hidden talents!
Yes, it was a low point when the category was boys names and the letter R. Our eldest son’s name is Reid. Guess what name was not on @genek101’s list!
this is so true, though one would argue in some respects it may be a good thing. If someone was a micromanager, they may discover their team can still be productive without them micromanaging.
so much more to be done in mental health resilience
https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1602782495281300
Micro managers suffering from lack of the ability to direct.
"Productivity is personal. WFH is personal."
Another often invisible class of isolated workers are folks who come from a first-in-the family-to-go-to-college/white collar career background. They often don’t have an existing network outside of work to provide guidance -and it is especially hard for this group to ask for help.
Remote beer tasking, remote wine tasting, remote DOES video watchparties... But some of us really need the time not in a call all the time.
A lot of this is like looking in the mirror and seeing problems I knew I've had and realize that trying to ignore it isn't always working. Very helpful.
OMG @nicolefv if ONLY someone had said this in the mid 80s when I entered tech. My company was great and on the front end of human issues (DEC...think PDPs, VAXen, and Alpha!)...but work-life balance-sustainability wasn’t at the front yet. The current and next generations thank you!
What is so startling to me is the negative impact of pandemic to managers and leaders — I suspect this will resonate very strongly with the experiences of this community! Thank you @nicolefv!!
@ffion and I are on and on about the people work including the need for Team Relaunches and understanding what works for each individual and each team - THIS is the time to redefine work, it would be such a waste to not take the opportunity
For the love, don't read work email on your phone people. Email is async, it can wait
That's another one. Comprehending asynk, flexible, etc. time to work these things out
@chris.gallivan 😅 I guess I'm not alone haha
that's definitely one way to get productivity up!
Hmm... really thinking to try this lol
I wonder how long I'd stay employed 😛
many thanks to my colleagues who have done work to supplement my own. huge thanks to the SAINTes team in particular ❤️
While in a virtual meeting should video be required? Can it be even enforced? Does it matter? I like to turn my video on but many never do.
Very interested i this topic. With no camera - hard to connect.
It can help or it can hurt. Can you decide on it at the team level?
Curious how camera's on can hurt? @me1342
I'd rather have camera on. But a lot of people claim bandwidth problems. One client actually told people to stop using video due to bandwidth issues.
We have remote folks with connectivity issues so they sometimes turn it off. But we are encouraged to turn it on when possible. It really helps.
I think there could be issues of making people use their video for personal reasons. :man-shrugging: Perhaps some are embarrassed, uncomfortable.
Curious how much the bandwidth issue is really the main concern.
We strongly encourage it but don't require it. Like Nicole just said - a lot of people are working through lunch, or odd hours, so requiring it all the time would be pretty restrictive for people. We shoot for like 80/20.
@mike_hoon Yes - I agree - my observations are that ppl are embarrassed w/ background - they don't have a fancy home office.
Also - some have told me that they don't look good enough - haven't put their makeup on yet:flushed:
We've also gone pretty casual. Not that there was ever a WFH dress code, but I have my video on with a t-shirt and hat most days - which I think helps others overcome worries that their appearance won't be professional enough, or a pet or child might wander in, or whatever.
@dominica - there are those that have a genuine near-phobia - neurodiversity, sense of self connected to outer appearance, a few different reasons
We joke at PeopleNotTech that the worst we look for an internal meeting the higher our Psychological Safety is. If only it were that simple 🙂
We do offer company approved backgrounds and encourage people to use them but it seems most people just don’t do it. Could be the a dress code thing like @sg.larson mentioned. I miss seeing faces so wish more people did it might be interesting to figure a strategy to get videos on.
@me1342 Would love to hear more. Sounds like I need to read your book 🙂
Book is not out till next year but a lot of the hints, suggestions and interventions that we do with some teams are detailed in our YouTube channel where we publish every Tuesday https://www.youtube.com/playlist?list=PLx8H_55ay1TLBRtaqrvyvCRmLrFIFM_tqand on LinkedIn in the Chasing Psychological Safety series: https://www.linkedin.com/newsletters/chasing-psychological-safety-6500675038520365056 and The Future Is Agile series:https://www.linkedin.com/newsletters/the-future-is-agile-6506407395235815424/
We also publish company backgrounds... although another background has become more popular than the official ones... the unfinished basement one of our senior VPs has been working from during quarantine.
unfortunately school starts super early here so I'm still up by 6am to get the kids ready 😞
Early pandemic, my reaction to leaders who admitted that working double-shifts was draining was “well, yeah” ;-(
They are reorganizing our entire building for secondary workspaces. We are home for good.
@damon so true, the pendulum will swing both ways.
re: video on/off in calls --- that tends to be a very cultural thing, and also depends on connectivity
How to create inclusion, safety, a sense of belonging, authentic connections, etc. in this new normal...
Agreed, I ❤️ VS Code Live Share 😄
Yes love VS Code - wish we could get live share to work through the vpn
Although - our GitHub is about to go on the internet - I hadn’t thought about that
Looking for more effective ways to collaborate without just having a Zoom call open.
Exhaustion and afternoon power naps have become a thing for me. I don't mind a couple of extra hours in the evening later.
👏 really enjoyed the talk and thanks for joining the happy hour last night too, @nicolefv!
really amazing talk, thank you!
Thanks @nicolefv! I will be shoving this at execs ASAP
@nicolefv awesome and very timely talk! How do u prefer contact after the conference ends?
👏@nicolefv Thank you ! The future work model hopefully will be influenced by employee choices!
Everyone's coping strategies are different. For me, it's completely unplugging after 6pm and not reengaging until the next morning. Not engaging at all on weekends. Others have completely different mechanisms 🙂.
“I got an email from someone I never heard of, about a conference I had never heard of”. hahahaha. @damon @patrick.debois256 😂😂😂😂😂
is it me or does it seem like this feed has gone 10X today? Probably correlated to the quality of the talks (and maybe the karaoke happy hour last night. so great to see everyone collaborating in realtime
@nicolefv When leaders themselves are overwhelmed and are struggling with their own productivity, they miss what's happening with their team. The details of this talk may help bring some of that to light.
damn, missed those times, started with NES and SNES haha
you are in a twisty maze of passages all alike!
(I was amazed when I found the cheat code for MacOS. Such a life changing experience! So amazing! Finally beat the game!)
I think that was for Double Dragon and Zelda… maybe Mike Tyson Punch Out…
What was the joke? “If Ops were a game, OpenStack would be the boss at the end?” 😆 (Amazing blog post by… umm, I’ll think of it!)
Which variety of "complexity" is causing this? What is really blocking the ability to have faster/fewer?
https://twitter.com/brecky0101/status/790497625610997760 Found it. It was by Justin Clayton.
“If there was a video game called Sysadmin, the final boss fight would be https://twitter.com/hashtag/OpenStack?src=hashtag_click.” —Justin Clayton
until you buy an amd ryzen thread ripper and run your program.. and everything breaks 😄
(About @damon’ background. I felt so bad that we didn’t tell him the conference was virtual — he recorded this in Vegas. 😆
I also didn't get the memo @damon, except I showed up in London looking for some guy named @genek101 😉
Humans still kick computers butts at large scale pattern recognition
"Automation supports the human operator rather than replace" With that as a premise, then teams will be more comfortable embracing automation
heh. I heard "Armies of Automation"
And at the same time, one of my favorite quotes from my friend @scott.prugh is that the machines are laughing at us because of our feeble efforts to do their work better than them.
Hah. I think I made that comment when doing a VSM of our VM process years ago.
I'm just going to send that to the next manager who insists they need manual acceptance testing.
Yes. Let the machines do what they are great at. But, don’t be foolish and expect them to do well what the human brain is great at.
That should have been in “Cards against Containers”
We are all BlackBoxes! Tacit vs. Explicit knowledge...and the tacit does not get recorded into dynamic organizational learning repositories
Yes, the experts and the tickets to them are 100% my life right now.
haha... i remember someone telling me that they were looking for a script that didn't run for a week until they found out it was a person being on vacation 😄 (and their standin on sick leave)
Let me throw one more expert problem in the mix - you design systems differently for experts vs novices. and if you have a mixed organization, you risk either designing a system where the experts are impeded - OR - a system were novices can easily make mistakes.
Those of us focused on Dev quality of life try to figure it out. We try to just tell the leaders what to say so that it becomes important. I'm always looking for examples.
Kinda look at most management as people i can get to repeat good things I tell them. 😄
standardised work = embodied knowledge…. a connection I hadn’t quite made before!
the knowledge of what is “the right level” is part of what is being embodied.
"That's what the money is for" ~Don Draper (Mad Men) <-- SMH :face_with_rolling_eyes:
I heard that, more or less, last month. In fact it was phrased, "you are well paid so I expect you to deal with a lot."
The trend is that human work ought to be focused on exceptions. If it’s reliably repeatable, then it’s machine repeatable. By applying people to the unique things, we increase value and satisfaction
We have a famous jeweler in Philadelphia who has a diamond in his beard. The CFO with the mask looks exactly like him! lol!
Have you been wanting to share these amazing talks with your colleagues? A big announcement! We’ve heard how excited you are about sharing the videos you’ve seen here at the DevOps Enterprise Summit — we’ve put together a short-term solution that will enable you to share any video in the Watch library with others who didn’t attend: http://videolibrary.doesvirtual.com/ Over the next couple of weeks, we will be enabling individual and organizational passes that some of you have purchased, and eventually a NYTimes-style gate that will allow a certain number of free views. We’ll give you more information when those options are available. In the meantime, read this post for instructions on how you can share videos with others right now. Thank you for all the help making this happen!!! https://devopsenterprise.slack.com/files/TASMB716H/F01CGBCQYUW
2When will the licenses be activated? We want to plan some watch parties
Is the content still going to roll over to YouTube at some point? If not, a tier for those without corporate financial support would be helpful.
@genek101 where is pricing information for individual and enterprise licenses? Or is that part of what's being put together now?
@bryan.finster seems to know something here... 
Should be on the IT Rev site to register for the conference. There was a place to sign up for individual and enterprise licenses.
A director in another area pointed it out and purchased for us.
@bryan.finster Ahh, I see it now, buried still on the registration page (at the bottom).
Should really, really think about taking a subset of these for universities, high schools and allow teachers and professors to use them for free. If we really want long term success the time to educate the next generation is now.
@damon where was the link to "The Ironies of Automation"?
talk about bad timing… the PDF link (that is linked to from multiple places on the web) is now dead
OK, I found a copy: https://arena-attachments.s3.amazonaws.com/5866627/714ed797298c0b2ef18481c31fdc4416.pdf?1578598175
which observability tools have helped you all achieve the goals for reliability?
reliability is a by-product of focusing on service availability. various ways to measure that. you can create "app heath checks" inside your app, ie call myapp/heath. it runs code you created to verify the service is healthy. or, external tools to measure your availability like synthetics.
and also how have you trained developers to instrument code for enabling observability?
Mine is fine but let me ask @jeff.gallimore and @jessicam to doublecheck. Thanks, Melissa!
@mhoskins I am looking into this - I am getting varying reports. Most people can hear ok. I am sorry you can't hear!
We think it has to do with actual recording and not the broadcast.
Posting this note I got from @jonathansmart1, because I suspect it will be of interesting to many other: > Morning Gene, hope it’s all going well. > I’d be interested to discuss that quadrant you had up. I’m not quite getting it. High standardisation is an impediment for experimentation. > > e.g. Six Sigma kills innovation. Top right doesn’t seem optimal to me, unless I’m not understanding it. > Low standardisation and high feedback is the sweet spot for emergent work, product development. i.e. top left. Which happens to be the same place as Complex in Cynefin. 🙂
I do agree that standards can be the opposite of innovation. But. I think standards are good as long as you change them often. I think about @steve773 s see-swarm-solve-share idea. Standard work is a way of sharing solutions. They have to be written in pencil so they can be revised and updated as you find new information. When they get calcified into SOPs that haven’t changed in three years...or they can only be changed with a bureaucratic process that’s where it stops being part of a continued improvement model
Also maybe ... six sigma kills innovation in the same way @jonathansmart1 Certified one way of working agile model kills innovation - when the process is more important than the problem you solve nd the value you create ... also if you six sigma a dominant architecture you can get really good at delivering that dominant architecture and do some good business that way. But you might miss a new emergent architecture as a result. Short term vs long term innovation?
@paula.thrasher @jonathansmart1 @genek101 @jeff.gallimore @nicolefv Thanks for bringing this up. The phrase “standard” has more than one meaning, some conducive to a learning dynamic, others not. The non conducive has Tayloristic overtones--a to down command and control imposition of (behavioral) requirements to which all us schmoes have to subscribe no matter how difficult they are. Conducive to an improvement dynamice are standards in terms of (well articulated and accurate) requirements of what is needed to meet the needs of the customer. Lastly, back to behavioral standards, standards as used by Toyota and others, standard means hypothesis--one’s best (justified) guess about action and outcome, which gets tested each use and potentially refuted by a single use. Once refuted, that’s trigger for improvement. One thought about this: I do believe you actually need (temporary) standards to improve just like you need a hypothesis to conduct a meaningful experiment. No hypothesis (no standard) means no experiment (ie no meaningful feedback), means no learning.
@steve773 Thank you for this. Thats a very good way to describe it. I like the point @nicolefv makes as well, how essential it is to have some controls/standards in an experiment in order to get meaningful, valid results. I love this scene in https://youtu.be/LGjREUEpkec when the everyman hero Emmet gives a speech that to sneak into the bad guys lair, instead of being creative and building a new thing, they should use instructions to build the standard model as a team, and then they would look like all the other ships and not draw attention. All the rebels balk at the idea (instructions! why would I want to do that!) but he sells the value of standards allowing everyday people come together and build something bigger than themselves. Maybe not the most "enterprise" analogy (or maybe I watch too many kids movies! ha), but this is how I think of why and when standards can be A Good Thing 😀
Found this that I thought was superb at capturing what I was trying to convey — an “old but good” writing on “XP vs Cowboy Programming.” http://wiki.c2.com/?CowboyCoding I thought the point about disregarding future and past, and factors that are ignored were superb. ———— A nearly complete absence of advance planning. Cowboy coders code in the moment, with little affection for considerations of the future or even the past. However, there is a branch of the cowboy discipline that thrives on the blue-sky possibilities of software, discussing what is going to be done until the sun rises. A disregard for the social aspects of cooperative programming, manifested as a disdain for all the trappings of communication between developers: comments, documentation, systems of names, explanations, reviews. Here again, some cowboys hang out together, often forming GarageShopEnterprises, sometimes, though rarely, with outstanding results. Apple might be an example. A focus on "hacking", or perhaps better, "active change". When a Cowboy Coder doesn't like something, he fixes it. It is this dynamic revisionary style which has produced perhaps the best merging of the CowboyCoding and BigDesign approaches. RAD, Incremental Development, the Spiral Model, and even Extreme Programming are all attempts to retain the aggressiveness and passion of the Cowboy while gaining the benefits of risk-aversion that underlies BigDesign.
cc @mik @jtf @steve773 @tal @paula.thrasher I think upper-right is where work is always treated as an experiment, where work can be repeatable, performed by others, potentially automated, and adjusted. I loved the @tal talk about high culture environments, because to me, it talks about when too much work is tacit, and requires a ton of time/effort/exposure to learn how “things actually work around here.” @damon talk pointed at this, as well — when Alice’s work can be standardized, it makes it done by Alice easier, as well as people besides Alice. (Or Brent.) I think @steve773 would claim that you cannot do experimentation with being able to repeat it (i.e., standardization). It doesn’t mean that it has to be rigid, just repeatable. Being able to perform mass experiments at scale at the edges (e.g., http://booking.com, Facebook, Etsy) is enabled by standardizing A/B experiment setup and run. Thoughts?
cc @mik @jtf @steve773 @tal @paula.thrasher I think upper-right is where work is always treated as an experiment, where work can be repeatable, performed by others, potentially automated, and adjusted. I loved the @tal talk about high culture environments, because to me, it talks about when too much work is tacit, and requires a ton of time/effort/exposure to learn how “things actually work around here.” @damon talk pointed at this, as well — when Alice’s work can be standardized, it makes it done by Alice easier, as well as people besides Alice. (Or Brent.) I think @steve773 would claim that you cannot do experimentation with being able to repeat it (i.e., standardization). It doesn’t mean that it has to be rigid, just repeatable. Being able to perform mass experiments at scale at the edges (e.g., http://booking.com, Facebook, Etsy) is enabled by standardizing A/B experiment setup and run. Thoughts?
Top right, high degree of standardisation, high degree of feedback, agree Toyota Production System. Complicated in Cynefin. Known unknowns. It's the "high degree of experimentation" that doesn't fit for me there
For emergent, unknowable work, we don't want standardisation
Really interesting point to discuss! I’m reminded of Donald Reinertsen making a similar point about variability when comparing lean manufacturing to software, and the idea (IIRC) that in software variation can be a source of value.
Also up there would be Mercury, Gemini, Apollo programs; Safety culture at Alcoa; Apollo space program was highly creative work in unknowable (or at least highly unknown) domain.
Top left is the sweet spot for experimenation, which is the sweet spot for unknown unknowns. i.e. product development
Standardization inhibits experimentation < I think we are thinking of different mental models of standardisation here.
Top right, minimise variability Top left, maintain optionality, we want variability
‘one best way’ vs ‘the best way we currently know’
In Gene Krantz book, it was clear that all mission controllers lived and died by their 3 ring binders. Their best understanding of the system, risk analysis, planned responses, stress tested in simulations. I explored this in many episodes of Idealcast, and became even more convinced after talking with @jtf!
I don’t remember where I read this (Toyota Kata maybe?) that while everything in the factory is documented, that document changes 400+ times a year. < you’re making a distinction between continuous improvement and experimentation, right @jonathansmart1?
Apollo 11 simulations kept killing 2 astronauts, revealing that final landing decision was being made to early. Got integrated into procedures. Apollo 13 was saved by documenting what normal looked like, enabling seeing what the genuine anomolies were.
Lean = known unknowns = standardised work = kaizen = continous improvement
Unique work = unknown unknowns = wanting variability = experimentation = agility != standardised work
There’s a Kaikaku example in Toyota Kata of asking a supplier to cut costs by 30% (I think that was the number)
Complex -> Complicated -> shallow dip into Chaos -> Complex -> Complicated 🙂
If I follow what you’re saying, the outcome of kaikaku is going to be standardized work, but the kaikaku itself isn’t standardized work.
standardized work is merely the prerequisite for being able to say, “if I do A, then B results.” It is the foundation of the scientific method. and ability to repeat prior work, regardless of what Cynefin quadrant you’re in.
Yes. Perhaps it's definition of experimentation, I'd expect to see experimentation on top right and top left.
If I’m thinking about the organization, an organization without standardized work might have trouble doing kaikaku because their daily work isn’t under statistical control.
Hard to judge an experiment without an established baseline to experiment from.
Standardised work inhibits experimentation. There is one way. Which is refined, generally small experimentation with variability limited
Actually, @steve773 is dubious that there’s anything in the top-left — the Y-axis is “degree of integration of feedback into standardized work.” I thought that was the Momento quadrant, so clung to it, because it was funny. But it’s really a “n/a”, nothing there makes sense, because there’s nothing to integrate feedback into. 🙂
Product development with much bigger experimentation needs to not be constrained by 'standardised work'
| Hard to judge an experiment without an established baseline to experiment from. It's unknowable, it's testing a hypothesis
| But it’s really a “n/a”, nothing there makes sense, because there’s nothing to integrate feedback into. Might be good to discuss verbally 🙂 It's Complex in Cynefin, isn't it? Unique, unknowable, unstandardisable work. Which needs fast feedback loops, to test a hypothesis. i.e. agile.
hmm… you might still have a baseline. Consider that example of seeking a radical breakthrough of cost or performance. You have a baseline you’re starting from.
IMHO, @jonathansmart1 — here’s the ends of the spectrum. Lower left: A dev randomly hacking away on things, seeing if something works. (in context of a high risk, high gain project.) Upper right: A dev being methodical, disciplined, taking good notes, learning what works and what doesn’t work, converging on the right answer. (also in context of a high risk, high gain project.)
Might be good to discuss verbally < lean coffee coming up. 😄
Upper right: can repeat the breakthrough, can explain it to others, etc. Lower right: gets lucky, can’t explain why it worked (or why it didn’t work)…
I think I am with @genek101 here. If people cannot just fork your code and build/run/test/repeat - it's not going to be a success
> A dev being methodical, disciplined, taking good notes, learning what works and what doesn’t work, converging on the right answer. (also in context of a high risk, high gain project.) For unique product development work, is it standardised work? We ask teams to not take a one size fits all approach
Just checked The Principles of Product Development Flow because what Jon is saying reminds me of Reinertsen. Chapter 4 is Exploiting Variability.
https://tulip.co/blog/lean-manufacturing/what-is-standardized-work-and-how-to-apply-it/
Having watched @patrick.debois256 work on standing up this online conference, I observe his paranoia of relying on external services, his methodical eradication of vulnerabilities, his intent listening for weak signals of failure. I’ve seen the massive differences of LL vs. LR. I fear LL. I want UR.
I guessing the creation of the online conference was not a documented one best way Standardised Work 🙂 Maybe it's becoming more standardised the second time around 🙂
Maybe this is the Dev+Ops, as in you want variability (experimentation) for the product development and you don't want variability for the Ops
For sure. But that convergence and iteration and improvement hinges on methodicalness and disciplined, obviously informed by a ton of experience and intuition. And we’ve only done it twice, but I bet @patrick.debois256 would bristle if you told him his work was in the LL quadrant. Hahaha.
Not suggesting LL. Top Left. High feedback, not following a cookbook in the creation
Thought: Can the process of experimentation, the steps, be standardized?
Is that what we mean as a disciplined experiment?
I mean, holy cow, we discovered on Sunday, that uploading video Dropbox links were causing intermittent video gray boxes — I laughed at the notion — but methodical analysis and experimentation isolated the cause, he found an known issue… It turns out Dropbox links are never copied — and that Dropbox files were having problems streaming, intermittently. And thus the decision to copy all video files (100GB+) into Easylive streaming platform. (And not as I had proposed: putting them into an S3 bucket or GCS bucket.)
@jtf I definitely agree. The defining factor is probably the mindset: the degree of methodicalness.
| Thought: Can the process of experimentation, the steps, be standardized? I'd suggest it's context dependent. What works for one firm, might not for another, based on culture, history, etc. Within a context, possibly. With the corresponding meta level ability to experiment on the standardisation of experimentation!
PS: I catch myself often drifting into LL, and I know it’s never a good thing, but I’ll do it anyway. (“I’ll only have to do it once!“). e.g., randomly changing config settings. rolling dice to find the one that works. 🙂
Still, Jon is definitely poking at a point in the model that I think isn’t totally settled in my mind. I’ve started by framing this as “how could this be true”, however I’m not fully satisfied yet.
Totally. Might be that standardized work is an unsuitable name, because it conjures up too many, umm…. things…
I interpret it in the lean production sense, minimise variability, which is fine for knowable, repeated, work. It's a killer for unknown work, unique product development, where we want variability, to be able to experiment, to be able to test a hypothesis
@jtf @jonathansmart1 Wanna do quick Zoom meetup? (I realized I need to poke head into the USAF bof.)
Track 3 right now: “know the baseline, use hypothesis, focus on outcomes” 😄
https://us02web.zoom.us/j/82798883823?pwd=czh1U3BOM0lINVIzSzk4ZGc5ZVYrUT09
Going to have to move to the #lean-coffee zoom room: https://us02web.zoom.us/j/6033264036?pwd=Q081a2tJbGVFWFE0YWZqS2doWlhwQT09
Whenever that conversation continues, I'd love to be a fly on the wall again.
i'm jumping in late, but not all standardized work inhibits experimentation. it depends on what is standardized and what you want to experiment on
I would also argue that complete lack of standardization would render most experiments meaningless. that is, if you cannot reliably duplicate or confirm data, methods, findings, then what you did was not an experiment, it was chaos and any results are as good as guesses
from another angle: experimentations provide value in part because you can identify what piece of what change provided value. that means that something ELSE (in fact, many something elses) needed to remain constant and unchanged, so you know it did not introduce a change
OMG, hi, @nicolefv!!! Yes, so true! (Sorry, didn’t see you — we jumped on a Zoom call. Like getting a huddle together in a hallway. I love conferences!! 🙂
In my view every experiment is unique and has a personality of its own.. I view mass experiments at scale as an experiment on its own.. You learn by doing. 🙂
This is fascinating. I conceptualize the TL as “0 to 1”, creating the new thing, and TR as “1 to many”, produce many identical copies of the thing (like an iPhone, for example), and then iterate the vNext.
If u have time come talk thru on Finster’s zoom tonight... link is in #happy-hour
Minutes away! All systems green! Maxine socks of +5 saving throws (and +7 for network problems) obviously working!!! 🙂
Please welcome our next speakers, @jennifer.hansen, @rakesh.goyal and @biswanath.basu!
PS: and also thank you to @jennifer.hansen for being on that amazing panel earlier today with @drjgoosbysmith @nazia.ali @diamond.f.lee7 @anamos21252 @chawklady!
“Capital One has 49K employees, of which 10K+ are software engineers” (!!!) @jennifer.hansen
It's always fascinating to see the ratio of engineers in a company and how that changes over time / industries.
Amazing how Capital One customer service support person just enabled person to buy gas on their credit card….
@biswanath.basu: I heard this correctly? This application was a vendor COTS package running on a mainframe, of which hundreds of millions of dollars of annual revenue relied upon?
yeah pretty much haha
we have a bunch of those all over 🙂
maybe soon we can virtually burn the mainframes too
Hahaha! BTW, what happened to the mainframe at the end? Did it get hauled off? Or still more workloads on it? 🙂
I can't unsee the CONFIDENTIAL on the bottom of the slides 🙂
it makes sense to do the whole end to end, best way to see if it's working
Not sure - where it finally ended. @rakesh.goyal can tell us where it finally landed
Including regulators as customers is very useful framing.
I am sure we have some people from IBM here who could help to track it down 🙂
I’m still in awe of scale and the creativeness of this re-engineering effort — with so much revenue that it supports, I can only imagine of the “what could go wrong” exercises…
Chaos Engineering?
If I had an hour to chop a tree, I’d hit it with my chainsaw, and drink a few beers :)
Try hitting refresh, or one of the other tracks — they’re all independent streams. cc @erin
Thanks, but then i could miss some details. I just have to stop reading your comments 🙂 for five minutes
LOL I was going to say, I'm pretty sure we don't have in office in Yellowknife. I think they just meant Canada lol
our Canadian offices are in Toronto and Waterloo
and there's a call centre in Toronto and Montreal
haha @jennifer.hansen and group are American, I forgive them 😛
haha make sure to do it in the summer 😛
I still wish we got more snow here in Toronto 😞
though last year was pretty nice 😄
two pizza principle is kind of mean to big guys who can eat one pizza by themselves... 😞
Retain legacy employee’s keep the tribal knowledge,,
"Travel to agents site and work directly with the customer" 💯
I talk to a lot of people who still ask, “Why do I need to go that fast?” For some things, maybe not. But not for customer-facing systems. 😞
I always try to quote the Continuous Delivery book, how the larger the change-set the greater the chance that something will fail
“But that’s why we do releases on the weekends. So we don’t impact customers.”
being able to deploy on demand…as opposed to being held hostage by your vendor. (BTW, was the vendor still supporting that package you were using, and still in business? @jennifer.hansen @rakesh.goyal @biswanath.basu)
^^^ Haha. That’s… incredible… But this shows that there are companies who don’t have the engineering talent to get off of their platform. Another recurring theme: without engineering excellence, the future of the company is now dependent on an external vendor.
a very competitive market too
For sure. @scott.prugh gave a presentation last year on how they’re migrating away from “hostile vendors,” where the business model of the vendor is diametrically opposed to their own. 🙂
I have lost the same engineer twice… he came back and still was not able to keep happy due too many barriers…
@tapabrata.pal you get the credit for helping imagine our software delivery cleanroom!
2 years ago you have e2e test in clean room. did you delete it? @tapabrata.pal
(Interesting I learned: both @venkatarakesh.devinen and @jennifer.hansen worked with @tapabrata.pal on creating these dev platform capabilities, which included the software clean room.) @venkatarakesh.devinen, how did it feel to move from the platform team to one “in the business,” closer to where the money is made? 🙂
It is really fruitful to see the years of tooling benefit the engineers. Good part was that there was no need to sell the Capabilities!!
@jennifer.hansen’s slide on clean room is an example. There many checks and e2e testing is there
1we leverage CDC based micro-services testing to break releases if any API would be introducing a breaking change.
(Auditors don’t actually smile. They just show their teeth.). 😆 (Sorry @lucasc5 @lewir7)
I was talking about these same topics to a local bank today. I'm definitely going to send them this talk. You're hitting all the roadblocks they threw at me.
Hi All, Question: How can we rapidly release code when the business refuse to accept the changes unless they manually test themselves?
Hi All, Question: How can we rapidly release code when the business refuse to accept the changes unless they manually test themselves?
I personally always find naming something an "experiment" is the best way to sneak something in
Build trust - share/evangelize capabilities, bring them into the automation fold
Go small and prove the case. It will become apparent to the business the opportunity cost of delaying.
The business cares about value - economic and strategic....if you have data points proven through pilots / experiments (whatever you call them), then the bets are easier to make for the business.
We’ve had some success separating the concept of “deployment” and “release”. You can deploy changes into production behind feature flags (dark). The business can then test all they want and flip the toggle on when they’re comfortable (“release”).
Ask the business if they are getting the results and functionality at the speed that they desire ... When they respond NO - then help them see the value and as said before - start small and prove the value
try to eliminate the us vs them and find some low hanging fruit that can provide big value
but it can be an long uphill battle depends on the organization. but if UAF can do it everyone else should be able to do it
Perhaps there's a communication gap between many members of the organization. Some buy in and others are stuck in their old "if it ain't broke" ways. They seem to trust their quality over ours. We have has small wins in certain parts of the org, but others are struggling. In ways, we are unintentionally (maybe we want to intentionally) "strangulate" our business users
it's only been 3 months, but the struggle is real and some don't get it. I guess I need to be more patient
@biswanath.basu noted. thank you. @bob.rion, we have sent surveys and the results were positive.
@ibathazi, thank you for trying to keep me level. How long was the UAF uphill battle?
try to find some ally who are willing to make the extra mile and show the value... some of the automated testing can do wonders to get people on board... one week test steps in couple minutes can do wonders to convince people...
we are fighting our uphill battle but some small quick wins get us some allies
I was going to echo what @jeff.gallimore said above. Separate deployment from release then empower them on when they want to release that value. This is definitely one of the challenges as you move from the projects to product world. I also tell my team it's way easier to debate 'working software' than getting into an impasse or debate upfront about how they'll accept MVP, etc. There's a BIG difference in the conversation when you can say this is working software deployed in Production that you can realize the value on any time versus will you accept X, Y, Z upfront? Another way I've convinced stakeholders to think smaller, but make a bigger impact is to focus on removing pebbles from your customer's shoes? Those are usually smaller changes that give stakeholders AND customers confidence you're in tune with their needs and can make an impact no matter how small.