This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-11
Channels
- # ask-the-speaker-track-1 (130)
- # ask-the-speaker-track-2 (99)
- # ask-the-speaker-track-3 (94)
- # ask-the-speaker-track-4 (172)
- # bof-arch-engineering-ops (2)
- # bof-leadership-culture-learning (2)
- # bof-project-to-product (23)
- # bof-sec-audit-compliance-grc (2)
- # discussion-main (730)
- # faq (6)
- # games (7)
- # games-self-tracker (1)
- # gather (28)
- # happy-hour (17)
- # help (34)
- # hiring (11)
- # networking (8)
- # project-to-product (1)
- # summit-info (28)
- # summit-stories (1)
- # xpo-bmc-ami-devops (1)
- # xpo-cloudbees (1)
- # xpo-cockroachlabs (2)
- # xpo-github-for-enterprises (1)
- # xpo-gitlab-the-one-devops-platform (3)
- # xpo-itrevolution (4)
- # xpo-launchdarkly (3)
- # xpo-lightstep-observability-incidentresponse (1)
- # xpo-linearb-automate-dev-team-improvement (9)
- # xpo-planview-tasktop (2)
- # xpo-snyk (1)
- # xpo-split (18)
- # xpo-stackoverflowforteams (2)
- # xpo-synopsys-sig (4)
- # xpo-tricentis-continuous-testing (2)
Hello! It was so good to catch up with so many friends yesterday, and meet new ones β excited to start again this morning!
Playlists in the Video Library are starting to answer the question of βwhere do we start?β https://videos.itrevolution.com/playlists/index.html
β¨ Starting us off this morning is @moira.cheng, IT Operations - DevSecOps Transformation Programme Manager, Vodafone, here to present, Enabling IT Operations - Transforming to DevOps β¨
Hello, @moira.cheng!! Thank you for sharing your story!
Hi everyone π So excited to have you listening today. Would love your feedback β€οΈ
Vodafone is adding 7,000 engineers? Just thinking about how hard it is for us to add even 1! π
What is meant by 'insourcing'? Taking currently outsourced work in house with full-time employees?
I think it is virtually impossible to hire 2, 5, 7000 skilled engineers but it is possible to hire 7000 passionate people and train them. Very smart.
@rshoup yes on insourcing - bringing dev work in that we used to get done by 3rd parties - back in house with FTEs.
@vladyslav.ukis unfortunately I don't have that level of information. But up-skilling and reskilling is a big part of it.
I learned from @moira.cheng how much Vodafone was changed by the acquisition of the Cable and Wireless enterprise business.
βwhat? you want to build it and run it? it seemed like an odd thing back thenβ¦β π
that one platform teamβ¦ sounds like the start of βthe rebellionβ within vodafone :rolling_on_the_floor_laughing:
Note that my partner from my talk yesterday is our head of change management. Collaboration is key!
@gus.paul - Agreed! Check out the plenary talk on Agile Auditing this afternoon, where Audit and Technology leaders co-present on collaboration π Loving the high-collaboration talks!
@moira.cheng: βI resolved to either find the wayβ¦ or pave the wayβ Love it!
@moira.cheng this is such an excellent analysis of the current state
βengagement would happen just before productionβ ah, yes. that. π
interestingly we have this same problem but itβs reversed - itβs the dev teams we need to encourage to be more collaborative, not the ops team
Moira are you sure you don't really work at Morgan Stanley? π
When I saw your presentation, I thought of the same level because I was comparing both companies :)
Great presentation @moira.cheng! Happy to share Morgan Stanleyβs Wealth Management Tech Operations and SRE journey! Over the past 6-7 years, we significantly reduced incidents by 70%, operating costs and toil.
it is funny to what extent there are universalities in these dynamics, isnβt it?
a call for volunteers! no budget, but fueled by peopleβs curiosity. so interesting!!
calling all people who want to be part of the Rebellion
intrinsic motivation is key in changing the mindset of 'the mass'
this is something I loved about @moira.chengβs program β modeling βnew ways of workingβ in all aspects of running this volunteer program. what a disarming way to teach and learn and participate!
βthey had never used agile on a project beforeβ (or so they thought! π
β€οΈ that optimism @moira.cheng! #failfastandlearn
curious how much senior management sponsorship you had. or at what point...i felt without that we'd have way less people paying attention.
I was really fortunate Gus. My senior management team empowered me to do it how I want, in any way I want. I haven't disappointed them so far π. Still have their support π―
βwhatβs the worst that could happen?β a phrase used by many great change-agents and innovators :rolling_on_the_floor_laughing:
Call for Code uses Agile planning with volunteers, too! Sometimes itβs the best way to make sure the team gets aligned and actually finishes things.
Thatβs what true DevOps is to me. Ways of working, getting people together, asking them to participate and co-create. Not just technical gimmicks. β€οΈ
marshaling all the energy of volunteers who want to solve their parochial problems. neat.
a great way to connect to the βwhatβs in it for meβ (WIIFM)
My good friend Mauricio always said "Its not possible to make it any worse right?"
Iβm off to a good start then. π When I took over leadership of an internal services team, surveys became my tool to understand better what the current situation is (aside from actual conversations of course), what are individual aspirations, etc.
50% responded to survey β how many, @moira.cheng? So good!
Our experience was engineers were not bothered about agile, but we had massive engagnement on devops concepts. Meet the people where they are...
Coming from a bigger enterprise organization, I feel that the first thing that you want to achieve is changing the conversations about Engineering.
Keen to explore in more detail what kind of Engineering conversations you mean @robert.ruzitschka. Genuinely intrigued. Lets connect
Happy to! Will talk tomorrow about our experiences.
I see a huge overlap in approach and experiences. Seems good π.
What incredible evidence that people want to learn β and signals of their aspirations.
@anshul.kumarbansal which types of challenges was i talking about? Local Market Challenges or DevOps Adoption (survey results) challenges? Local Markets were more about lack of involvement in agile projects, knowledge, resourcing, process perceptions (well we had about 63 pain points overall). Survey results for IT Operations indicated they didn't have the opportunity, knowledge or training.
In the drive for βproductivityβ - the space for learning has been massively reduced! Huge problem in the services industry at the moment
i believe @ben.grinnell will be sharing about this tomorrow
Itβs probably a problem everywhere. I see that so often both internally and at customers. People are so swamped with work that they donβt have the headspace to learn, not even mentioning the timeβ¦
Yes, really a problem - one way we tackled this was reducing the amount of non-strategic work in the funnel to open up capacity for experimentation... basically, lean portfolio management principles applied.
We talked about this in our Birds of a Feather session on Culture and Leadership yesterday. Some suggestions were booking fixed capacity in Agile projects for learning. 70/20/10 rule mentioned By @ferrix I know as a rule, this is hard to enforce in individual virtual agile projects and overall professional development (outside of projects) needs to be catered for too.
The 70:20:10 (customer needs / improvement of daily work / personal growth) is hard to push through. If the top management agrees that this should be the way, there might be some middle-management friction which requires assertiveness from teams to actually do it.
I experience a different challenge. The team members have such a strong sense of responsibility that they do it to themselves. With the 10/15/20 % of time promised to engineers for learning and experimentation, people donβt use it because they care so much to get the work done and provide the quality. I see more and more the importance of the middle management in this - they need to fully embrace it, to the point that they actively encourage engineers to learn and develop.
ItΒ΄s true that we had many volunteers as part of MoiraΒ΄s programme with very little experience on how to run agile projects but the decision to get it done in this way was the best decision ever as this gave me awesome insights into this matter which I wouldnΒ΄t have learned by all the trainings we have available in our Vodafone University. I genuinely love this this programme β€οΈ
Itβs a startling and astonishing initiative β kudos!!! π
Such startling numbers β I wonder if itβs because @moira.cheng targeted individual learning and aspirations, as opposed toβ¦ sorry, grasping for words, but something like βyouβre going to learn DevOps because Itβs Good For You.β
there are many enterprise engineers out there who are eager to change they way they work once they are given "permission". The latent enthusiam is amazing
We created a team based around engineer training, certification and developer productivity and unleashed a wave of enthusiasm. 7000 people trained later, 1500 certified... all by giving permission. No-one was forced, nothing was mandated. All by free choice.
Invite vs inflict (which I learned from @jonathansmart1 )
learning and defining/identifying conditions that were favorable to devops.
π βno more CABsβ. Now Iβm sure that she wasnβt at MS, @gus.paul. π
One of my favorite days at work was when we cancelled CAB!!
Team Topologies shows up again. My favourite book!
Great talk - how many FTEs did / do you have in the tranformation core team? For how long do you plan to run this transformation program?
Right now - dedicated to the programme full time 1 FTE (me) and 2 x 0.5 FTE. The rest of the volunteers (circa 40) are part time - roughly spending 4-8 hours a week. And NO CONTRACTORS π
Lots of famiilar learnigns here from our early days. We also used gamefication for teams adopting practices by giving them badges right on their source code repos. that turbo charged our adoption
Maybe jumping ahead but could you share some of the unified DevOps KPIs you used?
Yes we have a view of them (MVP KPI list and Non MVP KPI list), but they're not all implemented yet. We're working on trying to make them real - getting source data from various pipeline and reporting systems - and doing a PoC to pull them all together this Increment.
cool thanks, did you use the DORA metrics for this or other custom ones?
Hi @marco.delgado ok so we're attempting to start our Unified DevOps KPIs with this list - Selected for MVP: MVP: Predictability: JIRA/ADO MVP: Incidents/MTTx: ITSM (=>Central Tool =>Andrea Martin's dashboard) MVP: %Success Deployment PRD: ITSM (how to connect incidents post change) MPV: Mean Cycle Time: JIRA/ADO MVP: Deployment Time: Pipeline (Azure/Jenkins/Gitxx) MVP: SEMM Maturiy: SEMM App MVP: ITOM Maturity: Excel MVP: Throughput: JIRA/ADO MVP: Code Quality: SonarQube Enterprise
the pressurised middle management come up againβ¦ Thatβs an amazing idea Iβm going to steal from Moira :)
Turns out that DevOps is technology agnostic!!
this is such a great talk and a great framework for anyone out there trying to jump start their companyβs transformation. this should be a π for any Tech Leader who just finished The Phoenix Project and is scratching his/her head about how where to start
@nickeggleston - check out the opening talk from Moira from Vodafone.
Oh my word. Thank you @lucas.rettig and @ann.marie.99 That is such a high complement. Thank you. Never did I expect a year ago, that we'd come this far so soon. There's so much more to do though.β€οΈ
A lot of us muddle through similar efforts, and your summary and explanation of the process is excellent, @moira.cheng. π Looking forward to hearing how itβs going again in a year or two. π
It's exciting to see how much you can accomplish so quickly
@moira.cheng - This playbook information is very intriguing and is sparking a lot of ideas on how I can apply a similar concept to my audit team's journey. Can you share more information about the maturity assessment and unified DevOps KPIs?
Not Vodafone, but we have very strictly stuck to the DORA metrics when showing progress. We resisted a lot of push to use other KPIs at the start but we've shown the value of sticking ot those 4 simple metrics now
Having a whole squad dedicated to meaurement has been key though. the PO of our metrics squad is a genius
@gus.paul a squad dedicated to measurement?! π€― Genius is an understatement!
Does this help ensure quality in metrics, i.e. the same metrics consistently get measured in the same way, having the same meaning, and thus can be usefully compared?
Yes exactly. It's the foundation of all we do. It's also resulted in a data lake where all our jiras/build stats/changes/incidents etc are all in one place
That's exactly what we're trying to get to @gus.paul - Wish i had a whole squad and a dedicated PO on this though!
@lucasc5 - yes i'm trying to see how we can share the maturity assessments - one is digital! On KPIs we have a list of Metrics (the MVP ones and non-MVP ones) i can share. The aim in this PI is to try and collate the MVP metrics from our source systems (whether it's pipeline tools or ITSM tools) or to see how much effort it would take to pull them all into a visible datalake and dashboard.
@lucasc5 @werner.vaarwerk This is what we're attempting first as MVP MVP: Predictability: JIRA/ADO MVP: Incidents/MTTx: ITSM (=>Central Tool MVP: %Success Deployment PRD: ITSM (how to connect incidents post change) MPV: Mean Cycle Time: JIRA/ADO MVP: Deployment Time: Pipeline (Azure/Jenkins/Gitxx) MVP: SEMM Maturiy: SEMM App MVP: ITOM Maturity: Excel MVP: Throughput: JIRA/ADO MVP: Code Quality: SonarQube Enterprise
@moira.cheng this is such a different narrative than the stories of βfrozen middleβ β what do you attribute that to? So neat!
It would be interesting to understand what capabilities help combat the frozen middle problem
Hmm... simply put i think getting the 'frozen- middle' bought into what we're trying to do. Acknowledging that their teams need to learn and evolve, and even better, getting them to nominate individuals who can co-create with us, so they're part of that evolution journey.
Why are almost all the speakers describing the world before DevOps as "Waterfall and monolithic"? I mean, Agile and SOA predate DevOps.
guidance for IT operations is a real need in large corporations. In many companies ops is 60% of the IT work. They do need a playbook like what @moira.cheng shows here
@moira.cheng you seem to have that "relentless optimism" we must all strive for all the time
Well everyone has their odd moments, but I have to say I am naturally inclined to be optimistic. And when working with volunteers - I want them to be happy working on the programme. Optimism and motivation spreads naturally through others π
This is a really nice journey - personally I find the playbook approach much better than the "follow the methodology" that is often used. Giving teams the guidance and information they need to determine their own path within the guardrail and towards a shared outcome.
For the skeptics, how did you motivate them to get involved? Was it a coalition of the willing from the beginning?
Hi @rshoup I have to say, because we are running with volunteers - i've hardly had any skeptics. The Vodafone Tech 2025 strategy, and our OneTechTeam Digital & IT vision also helps. It was a coalition of the willing - yes - from the beginning. I'm a firm believer that skeptics will learn from the success of others given time π
great talk at @moira.cheng. Really resonating with me as I'm currently working on the (often arguably neglected) 'Ops' side of DevOps transformations. DevOps principles of 'you build it you run' it is the starting point, but embedding Ops in code and getting product teams to understand 'how' to do that requires stuff like SRE techniques (not necessarily roles), and the other elements of your playbook. Great stuff.
@moira.cheng this is a great and refreshing talk, some really useful points in here. I am in a similar position so some really useful points for to reuse. Thanks so much
@rshoup - skeptics - tough job to motivate them through sessions and presentations. But they do notice when they see their peers do great (motivation through peer pressure works!)
in a big compamy socialising the work is a full time job ...it takes a while but it pays dividends in the end. We have 1500+ at our quarterly internal devops days now
@gus.paul wow! Think i can learn some lessons from you on this. Would be great to find out more about your journey (other than Automating Change Management)
I think that we need to get our SREs to sit with our developers 1-1 to define SLIs in particular. Otherwise it doesnβt work.
The SREs are usually much better at parsing through cryptic logs to generate the metrics that the business wants. And the developers know enough about their application behavior and logs to help the SREs find the right information, or to add logging if needed. You need the domain knowledge of the application and the log aggregation / searching / alerting. Sometimes you get lucky and theyβre the same person, but not often.
Yes, sure but why would "1 SRE talking to 1 Team" not work instead of "1 SRE talking to 1 Dev"?
I think you only need one of each? Itβs usually very application specific. What were you thinking of, Vlad?
Perhaps I misunderstood you. My impression was that e.g. if you have a 200 devs org, you would have SREs talk to the 200 devs.
Ah, gotcha. Not like that. But say my 200 devs are divided across 30 smaller teams who work together, and they have 10 SREs who work on the service. I would want to have some general training for an hour or two (like a presentation) for everyone. But I would also want each of those 10 SREs to spend a day or two sitting with about 3 developers - training 1 developer per smaller team - over the course of a few weeks. Thatβs pair programming to hand-hold the developers through learning how to instrument their own application logs and how they (or SRE) will define the metrics and monitor them. And then those developers can teach the rest of their team how to work with SRE. I think we need that.
Would be interesting to understand if SRE (ops) is taking away focus away from DevOps. (EDIT: i.e. does SRE and the OPS association make it easier for some enterprises to look at SRE as a more traditional model).
I don't think so. I know those that say SRE is the new DevOps and I think that is based on a misunderstanding of DevOps (automation vs value delivery via software). DevOps, for me, was very much about shift left, removing silos, removing waste, reduced time to value. SRE is a specialisation in that story taking the lessons learnt from DevOps and Software Delivery to every day operations. SRE does appear to have a much easier path as it fits into the mindset of ITIL and Service Management. The aspects of SRE around control, quality of service fid an easy home in a traditional enterprise. The question of error budgets less so.
Hi @slack1599 I don't think SRE is taking away focus per say from DevOps, but i do believe that SRE practices are easily applied and can be a first step to improving traditional IT solutions. The fact that SRE practices also promote a DevOps culture and feedback loops with Dev is the start to a blossoming collaboration and opens more doors for full DevOps application/adoption
@slack1599 @matthew.cobby @moira.cheng SRE helps with the 'how' to do Ops in DevOps, very much a collection of principles and techniques like SLO and Toil management to focus on customer-based reliability and continuous automation in the Ops space. DevOps itself doesn't get that prescriptive. I think the DevOps vs SRE debate started because (at Google at least) SREs are dedicated roles aligned to but outside their product teams (for their critical systems at least, for others it's you build it you run it). Thing is, the SRE techniques can be used with or without SRE 'roles'. You don't even need to call it SRE to avoid confusion if that's the case. We're using that approach as we embed Ops into product teams i.e. we're merely asking/teaching our DevOps product teams to use key SRE techniques as we 'expose' them to production more directly and ask them to run it. The same principles of collaboration, automation, etc are there. SRE implements DevOps.
I am discussing these topics in depth in the upcoming https://www.amazon.com/Establishing-Foundations-Step-Step-Organizations/dp/0137424604
perhaps we need more talks covering this theme here in future DOES events.
@marcelo.pazzinatto this is exactly what we're trying to achieve here. We're looking at ensuring SRE techniques can be used by all, and are not perceived to just something a specific Team / Role (e.g. Software Engineers) can do. I think the Google definition has actually caused some misinterpretations that only specific roles or teams can apply SRE practices and techniques.
I agree on both counts. As a collection of techniques helps balance velocity vs reliability and control toil levels, perfectly suitable for product teams in full DevOps mode. I think for large enterprises still transforming, it also helps with the journey to 'embed' Ops into their product teams. It's a different, arguably easier story for greenfield, cloud-native and new teams....
I don't remember a DevOps conference (including Google's own Next) where this DevOps vs SRE theme didn't come up π
I think that the first clarification that should happen is - clarify what you are after - Site RE or Service RE. I see lot of mixup on these
βhow can we help each other reduce MTTR?β a surefire way to start a great conversation!
@genek this often is followed by βhow do we measure it? Letβs measure it!β Have you seen this done well? Should it be measured quantitatively at all? Is it more about confidence?
We've found great value in identifying and motivating DevOps Ambassadors. Huge unlock. (We call them "Velocity Champions", though ;-))
Similar. we have an internal change agent network of wnthsiasts to spread the word
I like Velocity Champions.... it speaks to the delivery of value at PACE. Too often a DevOps Champion was usually the person who ran Jenkins. :rolling_on_the_floor_laughing:. I might use this way of describing it.
I was at Vodafone way back when, it's so fantastic to hear the progress you've made in modernizing - thanks for the great talk @moira.cheng!
βif youβre a delivery manager please try to involve your operations team earlierβ - YES
@moira.cheng I love getting Ops Features into a shared backlog!!
yes! - because they are required features! Product peeps need some upskilling too
Absolutely agree - we need to change the tide here. That's why getting Ops involve early in the Agile project team set up / structure is key. Another thing is Product owner vs. Service Owner concept. Product owners need to have accountability for service and ongoing in-life operational and CX improvements too. This unfortunately does not often get fair prioritization in the overall backlog. Education of Product Owners, Product Managers and Project Managers needed. Ops is nowhere in SAFe
Great talk @moira.cheng! We have a version of DevOps ambassadors embedded in teams and it worked well for us. The evolution of this is a Community of Practice around DevOps and its socio-technical practices
One challenge we have is operations team supporting 5 or 6 or more different stream aligned teams....so they dont have the bandwidth to be fully engaged with every dev squad
Yes @gus.paul we've had to reassess how we structure our Ops teams and are providing guidance to Ops Team managers on how to scale. This is to enable participation in Agile project ceremonies, have new skills in planning - a bit more upfront. Happy to really get into the details 121.
It would be great if you had that discussion in Gather or somewhere interested folks can listen and learn π₯
@moira.cheng We really need that magic DevOps Spell!! Where can we buy that?
Thank you @moira.cheng - my favorite talk of the Summit so far.
Hello, @ben.connolly! I enjoyed your session on OKRs last year!
Looking forward to see DevOps, Me being in IT Operations in Vodafone π Amazing presentation π @moira.cheng
@sandeep.dabas get in touch, we'd be glad to welcome you into our programme or as a DevOps Ambassador:slightly_smiling_face:
That was very inspiring @moira.cheng! Hope you don't mind my stealing some ideas ;)
Of course, it would be my pleasure if anyone took anything from my session and re-used it elsewhere. Happy to connect too if you'd like more details π
I will surely connect! I am already in Confluence trying to articulate how to get some of your learnings into our own DevOps journey π
Great talk @moira.cheng!!! Keep up the great work!
@moira.cheng such an amazing story! what a holistic approach. And that youβve started looking at it only last year! a huge WOW! Iβm sure your playbook will be used as inspiration and guidance by many. Thank you!
For those of you i haven't replied to yet... I will shortly. Sorry haven't been able to keep up with all the positive comments and questions π
1. by our people, for our people; empowered, engaged, by goals they set 2. reaching out to business to get their goals 3. stronger voice to Ops community
Enabling IT Operations - Transforming to DevOps (Vodafone) Moira Cheng, IT Operations - DevSecOps Transformation Programme Manager, Vodafone π Now available for sharing: https://videos.itrevolution.com/watch/708550161/
:thinking_face: - @matthew.cobby I am thinking where you were and where I am π
π. I know I now need budget but with the right team and the right motivation, you can shift a company.
β And now, please welcome @simone.steel, Chief Data and Analytics Officer & CIO for Enterprise Data Platforms, Nationwide Building Society, presenting From DevOps to DataOps β
Hello, @simone.steel! Iβm so delighted that youβll be teaching us today, pointing us to something that is elusive, profound, and important, relevant to everyone!
among other goals: βmake best possible use of data for our membership; keep data secure, safe, compliantβ
at my company, Target - large US retailer, we call our customers our βguests.β Because, how do you treat your guests when they come to your house?
This is a nuance of a mutual. We are a membership organisation, and owned by our members - not share holders.
Aha - Value Streams @simone.steel - dropped my pen, paying my full attention now
Key responsibilities: β’ customer master data β’ making data available β’ data governance β’ analytics
(for decades, Iβve wanted a word that means βproperty of having integrityβ: @simone.steel used βintegral.β In past, Iβve tried using βintegrousβ)
βmy job in a nutshell: to ensure that use and management of data in the Society is ethical, valuable, safe and compliantβ
fascinating; mortgage of properties; we have access on home carbon emissions; how can we help members finance homes to make their homes greener
I love all these beautiful titles; Chief Proposition and Marketing Officer!!!! Brilliant.
Members' properties are safe and efficient -> our business is safe and efficient. What a purposeful business model!
I am curious - how many data folks here attending DOES (Data Engineer, Scientist, Data modelers, DBAs, Analytics, BI)? cc - @genek
dawn of enterprise data science; job of the data scientist is arguably the job of geneticist, physicist, economist; but now itβs an enterprise capability
βwhat is flowing through the software is the data of the businessβ
βdata professionals are citizen developers, not End Usersβ <<<<<
From @cbergh: βbetween 30-50% of company employees use or manipulate data in their daily workβ (arguably making this problem space bigger than devops!)
agree - It is already too late, but we need to bring our Data friends to this DevOps party and bring them to this mainstream. Without Data we cannot move forward.
its quite the conundrum. I use this quote in my products on why our teams need to ruthlessly investigate every βexport to Excelβ that happens in our system. Data is an integral part of how to drive your business and needs to be highly aligned with your strategy (unified Business/Tech)
The power of data - some enterprises are so so behind; Most of the compliance, privacy is about "DATA"
Agree, recently I was listening to one of the org presentation (nothing to do with DevOps) they were saying Data Driven Decisions and Data Insights in their strategy for Years 2022; For me this sounds like they are 5 years behind thinking like this.
where have we seen this before? :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing:
There is nothing better than spending a day or two on automating a thing that takes 5 minutes π
LAzy developers are the best developers. "Why am i always doing this 5 minute thing...let me automate it"
Makes me think how often I use Excel to assemble/explore data, or do one computation.
βgetting the data pipelines working; not databases, data lakes, data warehouses. data pipelinesβ¦β
Only person with a common goal is just the CEO usually. Looking forward to hear how you brought that down to lower levels
Unfortunately this is all too common. Part of the education here is for the C-suite!
β’ what is the hypothesis that biz is trying to prove/disprove? β’ how do I connect data domains that may be very far apart in org (!!) β’ produce actionable insight
βdonβt use DataOps as an expression meaning βdevops for dataβ"
I do have question why most of u sdo not see "DevOps" including Data - instead we want DataOps? @simone.steel
My instinct is that we focus on improving what is under our span of control (am I a tech professional or a data professional?) I.e. changing practices I can control is easier than creating common (cross-departmental) practices.
I think what is so fascinating about this talk is that we donβt have a (widely used) name for the concept of what @simone.steel is talking about. Seems to include reframing what the actual inputs/outputs of applications/orgs should be. Thoughts, @scott.prugh @rshoup (my fave architecture)
Agreed. As always, start with clear problem statement(s), identify impediments and bottlenecks, start knocking them down.
In my opinion, its a take on Conwayβs law, but instead of your architecture being a representative of your org structure, its how you influence, drive change, name things, etc, being a representative of where you sit in your org and what your title is. Once again - we have to align on our common problems, and tear down those silos.
βdata: some deterministic; some predictive (??)β <--- did I get that framing right? Such a neat way to unify those two types of computation going on.
I think this classification is helpful because we can explain and replicate software behaviour (deterministic), but we cannot do it in the same way with predictive analytics (non-deterministic, learning all the time).
βdata is often treated as IT projects; instead, it is a power user domainβ
Most of the organizations - We lost track of how many ETL jobs are running, active and not in use; duplicate data sets in number of data lakes; Often the argument is about "Who owns the data?" - common assets like Customer, Products are good example for this.
Even more worrying if we think about environmental and sustainability views: #dataisnotfree #dataisnotgreen
lack of alignment in incentives β sounds like the βcore chronic conflictβ of the data world. just like with βdevβ and βopsβ.
North Star: βintegrity of data and timeliness of decisions, enabled byβ
Gartner: only 8% of our data science apps reach production. :thinking_face:
Data space is changing so fast - but practices remain old; We need to bring this new thinking and share with our data friends
the organisers have been very open about the conference working this way - it allows us to talk to the speakers during their talk, which I appreciate
Coming full circle with Statistical Process Control from the 1980s. Great to see that used on data processes and not just industrial processes :-).
DataOps is combination - data pipeline (agile) + data management (DevOps) + data observability (Data Science) @genek - probably this answers the question why dataops than devops
βPlease think about your role in creating better connections thru a lens of Data Operationsβ
An incredible call to action: βcreate co-mentorships between software and data professionalsβ (!!)
From DevOps to DataOps (Nationwide Building Society) Simone Steel, Chief Data and Analytics Officer & CIO for Enterprise Data Platforms, Nationwide Building Society π Now available for sharing: https://videos.itrevolution.com/watch/708550174/
π And now, we welcome the team from Schlumberger, @smohan3, Global Practices and Competency Manager and @lester2, Global ALM Practice Champion here to present Season-Based Governance π
Hi All β @smohan3 & I are delighted and honoured to be here and presenting at the DOES22.
Really got our Devops team thinking about our data team here @simone.steel, thanks!
If I miss any of your questions, please reach out to me in the private channel. Thank you!
2000 software engineers; $3.2B; oil exploration and production; help access energy for all
βour software products; until 2017, quarterly releases; offerings include reservoir simulators; market leader for 30 years (!!!)β compute heavy; means customers need to buy large amount of serversβ cloud offering allows customers to use without having to do lots of capacity planning/acqusition
Talking with @smohan3 and @lester2, it was wild learning about what itβs like selling to customers who operate in safety-critical cultures, resulting in a safety-critical mindset at Schlumberger.
βculture of creating (long) lists of features to developβ (this is so great! and familiar!) π
Hypothesis driven development - haven't seen org finance embracing this concept. Some details about how you educated and convinced them will be helpful
Add HDD this into process to encourage its adoption and provided training to help with its use.
Just reviewing this again. Got this wrong. We have not changed the way finance works. A very interesting topic though, on how to roll out new techniques.
The annual release cycle was "working very well" according to the blurb Then we did cloud and had to change to faster release cycles I'm not clear here... was there a business trigger for the paradigm shift?
βwe donβt govern how the teams operation; goals are owned by the teamβ
βplans are presented at beginning/end of each seasonβ βanyone in community can attendβ
@smohan3 how well did people take to the concept of βbetsβ and βhypothesis driven developmentβ?
@smohan3 - What roles are there in Goal Owners? I mean who are the people and what is their title in the org chart?
βreleases are not tied to the end of a season, they are continousβ
Correct, the aim is teams release when ready and not wait till the end of the season
βby the way, we didnβt get any of this right the first time aroundβ π
In some enterprise even after few iterations the result is same as first LOL
very frequent theme in every talk and a great reminder for everyone. Just start, get it wrong, pivot, but most importantly START
Start and persist, iterate, which means somewhere thereβs an understanding that failure-learning will be part of the process so the failures donβt empower the skeptics to torpedo the initiative.
This sounds very explorative model (meaning not a prescriptive one). While this appears effective for learning and doing, how does it scale. When you are not reasonably prescriptive at least around boundaries, things will go out of control!?
actually we found higher the alignment higher the autonomy. It does require a clear vision to be set and understood by the entire product team.
Yes there was a significant shift and business need to releasing more frequently.
We came across the season concept from MSFT and is covered in multiple talks but here is one: Agile at Microsoft By Aaron Bjork https://youtu.be/-LvCJpnNljU?t=1417
βDORA workshops generated hundreds of improvement plansβ
@smohan3 - how did you onboard the finance team on the HDD? WHat adjustments they needed to make to their financial planning process?
Good to see multiple industries leveraging DORA metrics. With wide adoption it would be interesting to see if we can benchmark to see how we are doing?
You can do an industry benchmark. I have one open in front of me and you can select which industry to benchmark to. It is based on the State of DevOps survey data
βbecause decades of habits, much unlearning was required.β
It is really important to read the definitions of the DORA metrics β the high level details are in the State of DevOps reports https://www.devops-research.com/research.html#reports Need to take particular care with Lead time for changes β this is from code commit to deployment in production. In ACCELERATE by Forsgren, Humble & Gene this is described as the delivery lead time (and does not include the design & validation part of lead time).
and they underline that was a practical need seems that a consistent measure the left chunk of a delivery process has too many variations and challenges
And if you work somehwere that change lead time can be measured in days, you cna break it down into further stages π
LinearB measures DORA from accessing your Git repos so you can do this exact breakdown into various phases @gus.paul. From there we can identify the bottlenecks in your dev process and help your devs merge and deploy faster with a workflow optimization chatbot called WorkerB. Come to our booth if you want to learn more! #xpo-linearb-automate-dev-team-improvement
IMHO we should keep delivery lead time as defined in Accelerate and use specific terms for any metrics that extends the measurement span
@smohan3 @lester2 Thanks a lot for this excellent talk. As some of the aspects look like they are coming from Basecamp, I guess the betting process is based on basecamp as well? Did you use it as is or did you adapt it? Additionally: Do you use aspects of ITIL 4? If so, which ones are the most important for you to use?
no not based on basecamp but similar. Our devops practices are aligned with ITIL4 but ITIL4 is more adopted on the services side than the software development lifecycle...
βending seasons at year end was (problematic)β (holidays, year end rituals, etc)
βthis has liberated us from onerous processesβ (love it!!!)
β‘ And now....here to present Product Org Design: Flow Follows Form, it's @mik, Founder and CEO of Tasktop β‘
Thanks @smohan3 and @lester2 for the great talk and active responses in the thread
βarchitecture (must be? should be?) isomorphic to the orgβ
@smohan3 @lester2 Great presentation, relates to my experiences quite well. I'd be interested to get your feedback on our Continuous Quality Assurance / Continuous Conformance approach that I presented yesterday (https://videos.itrevolution.com/watch/707351867/), and which is also featured in the latest issue of the DevOps Enterprise Journal (https://myresources.itrevolution.com/id006657137/DevOps-Enterprise-Journal-Spring-2022). Does this relate to your approach? If you have some time to have a look at it just drop me a line either in Slack or on LinkedIn.
We have mess above agile teams π
βspotify has moved on from the spotify modelβ :rolling_on_the_floor_laughing:
Always love @mik presentation; Very compelling and great takeaways
(thereβs a fascinating story of how McKinsey picking up on Spotify model in one of their whitepaper has caused epic impacts on the orgs who hired them)
Season-Based Governance (Schlumberger) Sujaa R. M. Deepak, Global Practices and Competency Manager, Schlumberger Nigel Lester, Global ALM Practice Champion, Schlumberger π Now available for sharing: https://videos.itrevolution.com/watch/708550186/
May need to rewatch at a slower speed β @mik is talking so fast π
The link will be posted after the session. Or feel free to reach out to us any time.
but OTOH, Iβm not sure he speaks any slower in conversation. π
In his defense, I absolutely need more coffee AND always enjoy learning from him
@vmshook video library can run in both slower and faster speeds π
One day I will learn to slow downβ¦ I keep telling myself. All these DevOps Enterprise topics are just too exciting to me for the time being π
Pattern Language; from Chris Alexander, the famous building architect
by the way, thereβs something called the Alexander effect: his observation that great architects made great use of his book βA Pattern Language.β Bad architects did worse when using his book. cc @rshoup
https://ericnormand.me/podcast/the-christopher-alexander-effect
βaction: reduce manual testers; outcomes: worse qualityβ (why? software wasnβt designed for automated testing!)
But also, adding manual testers ==> worse quality, per Elisabeth Hendrickson π
hahahaha. thinking about things like this lead to madness. π
@mik talking about how seductive the concept of βsingle threaded ownerβ is β hereβs expected benefits of merging product and engineering were:
challenging the 'business and tech merged together and lived happily EA
βmost profound moment; all hands meeting; most senior engineer asked, Mik, youβre such a fan of Team Topologies; so why do we keep overloading cognitive load and complexity domains? over and over again?β βI realized something was very wrong. signals were not reaching me.β
TT Book & Cognitive load - a good challenging question by senior engineer to @mik - Love that
βcode jams were gone; engr community events gone; how did this happen on my watch? Iβm an engineer!β βthe configuration of the org suppressed engineering/platform signals, like cognitive overload; product signals were coming through fine, though.β
βwe should have fewer meetings, but no; important signals were no longer reaching meβ
βwhy did Spotify move on from Spotify model? no engineering leader owned enough of the org.β
Curiosity - another question: How many of your organizations are adopting or exploring Team Topologies and seriously thinking about Cognitive load?
βresults of changing org again? weβve already hired 3 amazing engineers in last monthβ
βfaster moving consumer products need strong product signals; others may need strong engineering signals.β
@lbmkrishna: We have not fully implemented the approach but the main ideas about Enabling teams and the core platform ideas have deeply shaped how we work.
design via pattern languages, gauge via Flow Framework
Product Org Design: Flow Follows Form Dr. Mik Kersten, Founder and CEO, Tasktop π Now available for sharing: https://videos.itrevolution.com/watch/708550206/
@mik - Great presentation! We're just embarking on this path at New Look Retailers π
β Closing out the morning keynotes is @stephen, Vice President, Product Innovation at Sonatype, here to present, What does log4j teach us about the software supply chain? β
Thank God for recordings.... I need to watch that 10-15 times more. Amazing
I often do that - 3-10 times in small micro learning space; All this DOES are highly packed with great content, quality, experience & ideas. You cannot afford to miss that
very interesting is there a way to realise team overload except for asking team members?
(although it does make one wonder about the life choices weβve all made. π
Really nice presentation from @mik but Iβd love to see the βfull versionβ π Felt like scratching the surface of a topic that deserved an hour at least π
βat end of 2021, we had this amazing natural experimentβ π
I love it - and I noticed how many times the Log4J/Log4Shell topic showed up in talks this year.
so as in the previous years Equifax; It is better that we make sure we are not on the headline for wrong reasons π
@paul.d.fox is speaking next in one of the breakouts. Itβs an amazing description of the log4j events at Morgan Stanley.
βThe Large Hadron Collider took a decade to build and cost around $4.75 billionβ
we see this idea of exemplary cohorts in update behavior, release frequency, security response time. there tends to be clustering like this for various outcomes.
@stephen - did you (or others from Sonatype) get to participate in the EO NIST responses?
Yes, we submitted a couple of responses to that (on SCA and static analysis).
yes, quite good β especially considering it dropped on a Friday
but then again the weekend is the time for open source contributions, right? π
a culture of keeping dependencies up to date β‘οΈ better security posture. π
Like going to the gym. Patch every week. It's going to hurt but eventually you'll be ripped and secure !
it is really interesting that there are still folks who get βstuckβ, even among that early updating cohort. worth more investigation.
in our case log4j was almost exclusively a dependency of our dependencies / tools etc, rather than something we could readily update ourselves. Iβm guessing this will come up in your talk!
good lead-in! βtransitive dependenciesβ are coming up next π
So given the trend I just described, Iβm curious how long you had to wait for those transitive dependencies to be updated. Do you have a sense for that?
@stephen is there a link to this graph with live data by any chance?
Good: updating dependencies manually every couple/few days. Better: automatically getting PRs for updates and simply approving them. Probably one of the big differences between exemplars and laggards.
This is still manageable in the Java ecosystem (even better in Clojure). But transitive dependencies are hell in the node ecosystem. Sometimes a simple library ends up with over 1500 transitive deps π π§βπ π
It definitely is helpful, but having so many dependencies change we also need to ensure things don't break right? I feel there isn't enough community focus on keeping libraries "backword compatible"
"Transitive Dependencies Matter" They also really stink... @stephen π
No wonder developers are scared and become laggards
PS: the fantastic presentation from @ben.dodd and Fliss Bennee talk about covid prevalance in sewage. Fave line: βeven world class researchers canβt resist saying the word βpooββ :laugh:
The time it takes you to update a critical vulnerability is likely another good compound metric showing engineering maturity. Watch out, DORA. π
Thinking about it, I guess it is already covered in the existing metrics. π
@vladyslav.ukis It would definitely show that you know about your inventory, can deploy quickly, have an effective process and situational awareness. If you can fix by just deploying a new version of the dependency, it most likely also shows that you have not been neglecting the topic and that you are fairly up to date, so the change to the system is small.
I have seen systems that were so old that moving to a secure version of the library means first finding a developer who still knows the code, then upgrading your Java version then upgrading your build system, then upgrading your framework and only then you can upgrade the dependency. Now we are talking about months to fix a zero day vulnerability. Ahh, accumulating technical debt may have pretty ugly results!
Transitive Dependencies are a very large problem for very complex systems. The lag of update is expanding at each link in the supply chain.
yes, the deeper the dependency the longer the update takes to βtrickle downβ
How do you prioritize what percentage gets addressed vs not, is it noted somehow
one good practice is to focus first on applications that are in production and open to the internet. work based on security risk.
It is critical to understand the impact of each dependency. Not just as a isolated patch but how it fits in the entire system or better system of systems.
Thank you, sounds like it is maybe part of the prioritization process
Sorry dumb question (not a software professional) but why are the vulnerable versions still available for download?
pulling vulnerable dependencies would break all those builds that are using those versions. thereβs been discussion of this, but itβs generally considered too disruptive.
Deps that never get updated is such a frustration! My recent example is https://uppload.js.org/ which was all the rage as an amazing toolkit for frontend file upload / modification etc. And then within a few months its only developer left that org and it just stopped getting updates. I guess if deps are popular enough (more popular than Uppload) someone else will usually take on the maintenance?
this is an important point! which dependencies you choose matters a lot. Choosing deps with an active community and large contributor base helps a lot.
also dependencies that upgrade their dependencies quickly, so you inherit transitive fixes quickly. (MTTU, which we discuss in the Software Supply Chain Report measures this)
@philipday Builds will break, everything stops - Hell will break loose!
Thanks! So orgs are in a bind: β’ Do a migration you might not have resources for β’ Persist a vulnerability How do they avoid getting into this trap in the first place? Assuming that vulnerabilities are when not if.
It needs the understanding that keeping Technical Debt in check is a regular part of engineering work - not a side note. There is no way around it.
Fwiw I've found tooling around automating dependency updates is pretty good, at least in the JS/TS eco-system (e.g. Renovate - https://www.whitesourcesoftware.com/free-developer-tools/renovate/). It spins up PRs with version bumps, and can even automatically merge those if you want, taking it out to prod (if you're practicing CD) automatically. If you keep on top of it it shouldn't be too much effort, excluding major version bumps. It's when it's neglected for a few years that it becomes really painful in my experience
@robert.ruzitschka I guess not just understand Tech Debt, but your org has to fund it too So you whatever you build, you take on a financial liability that if your org doesn't commit to in perpetuity, you end up in this trap and then the data in your own care, your own product, and the supply chain downstream of it is all jeopardised What happens if your ownership changes to something less diligent? Iirc Cory Doctorow has written on this (he's written so much I can't find it), when private equity takeovers happen they are incentivised to slash costs such as IT spending, creating shareholder value (for themselves) in the short term but externalising costs such as security risk (and other problems) for many stakeholders further ahead
@philipday you touch on many important points that are far beyond the realm of engineering.
If an organization takes ownership and has a software product that is important for it's business success, it must be aware that continuously changing this product is a necessary fact. Even if you don't add any features, the world around your system will change - and thus make your system obsolete over time (it will become legacy - oh no!). So we must make sure to keep our systems in a changeable state. Not doing this will incur high costs and efforts later in time as change is unavoidable. Think about a system that uses an old log4j version. There is no way how you can keep it - it would be a business risk.
@philipday If the sole purpose of the owner is to extract as much money as possible and externalize risk, I don't think we are talking about a sound and sustainable product strategy π. So there is a more fundamental issue to solve. I am not denying that this approach is common but the possibility to apply good practices requires good will and understanding on all sides. I have done a lot of talks to explain TechDebt in our company and it helps to explain context and business impact.
Amazing line in @paul.d.fox presentation : βwhy are you telling me about a log4j vulnerability? Iβm a .NET app.β (why? how? find out in his talk!!! π
1. know what you have and where it is 2. know where you get your stuff and have controls on it 3. have the capability to make changes with speed and stability
What does log4j teach us about the software supply chain? Stephen Magill, Vice President, Product Innovation, Sonatype π Now available for sharing: https://videos.itrevolution.com/watch/708550222/
A process for approving components, wouldn't that be a constraint to the DevOps teams?
Depends on how itβs implemented. You can make approval processes very quick with pre-vetted components and automated analysis of new components. But it absolutely can slow things down if itβs manual and under-resourced.
Agreed with @stephen here. Teams choosing their own tools doesn't mean "all the tools". You should have a preferred/vetted list and create a decentralized process to proposes changes like ADRs in wikis or github. Beyond safety, you also get shared knowledge/common understanding and consisteincy.
There is nothing in the Sonatype product line to support this, right? It would be something more upfront in the process?
On that cheery note, morning plenary sessions come to an end!! π
Protip for Github users: https://github.blog/2017-11-16-introducing-security-alerts-on-github/ π
Thank you @stephen this was a great talk! π
PS: if you appreciate breaks being preserved in the schedule, please thank @annp and @jeff.gallimore β if it werenβt for them, morning programming would have overflowed the entire morning break. π
β¦and has the advantage of previewing all the talks before the event, right? π
Itβs always tough to pick plenary content among such good choices β the default is to just jam another talk in, to the significant dismay of the team. π
Noticed quite a few folks saying they need to watch the videos 3+ times to try and focus and extract value from them. I have a hypothesis that the virtual model allows for deliver of the content and speaker/community interaction in a more "exploded" way, where the video is broken in to small "bites" as each slide or point is made and discussion swarms or circles around the before moving on to the next "bite"
Another related idea is doing the conference as a capstone for content that is released earlier and more continuously, allowing for a subscription model as presentations happen more-or-less continously over time and then a big get together to blast through the "best of" and a few new items in person or a virtual multi-day to celebrate and summarize. This also lets Gene do his very much appreciated value-add to the talks (kind of like he does with the Idealcast or the video-topic-lists). The model feels more agile/devops aligned with small batch released into "production", and is similar in a way to what AWS and Microsoft do with their big conferences (in that most of the features and products have already quietly been released or are in beta but there's a focus and spotlight on them for the conference).
What do you think @genek? Any value here or is this just crazy talk?
@nickeggleston love the idea of βbitesβ. As for the continuous release, the idea has struck me before as well. I think itβd be amazing to have weekly/bi-weekly/monthly βwatch partiesβ where we showcase 1-2 new talks, but do it all year. Basically becomes a DevOps Enterprise Summit Membership Lecture Series. For AWS, Iβve noticed they do it over a longer period. Does it feel like attention fades, or does it help with engagement?
As Jeff just said: for all those interested in talking about supply chain further, join me for the Birds of a Feather session on it later today.
Great points.. I know you don't talk about SLOs.. π How are you tracking SLO's and have you implemented error budgets?
Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Hello everyone! Iβm super excited to watch @lucasc5 and @tod.bickley present!!
β¨ Let's now welcome @tod.bickley, AVP Information Risk Management for Identity and Access Management, and @lucasc5, Technology Audit Director from Nationwide Insurance. They will share, Adventures in Agile Auditing: Donβt Just Survive Your Auditβ¦ Thrive in it! β¨
AUDIT!!! βfear, dread, frustrationβ. π Sorry, @lucasc5!!!
You can also read @lucasc5 amazing paper in the brand new Spring 2022 DevOps Enterprise Journal here: https://myresources.itrevolution.com/id006657137/The-DevOps-Enterprise-Journal-Spring-2022?_ga=2.224150747.1548635363.1652109724-1037911749.1592589043
βWhat are minds when you hear that the auditors are coming?β
I love auditors - I will say that I am one of the few. But I like the second set of eyes to ensure things are working as intended. P.S. I am in the minority π
As an Assessor (RMR & CMMC) - I get a mixed bag some love the insights others treat me like Medusa
To really go full circle, @tod.bickley, you may have to do a rotation in audit first! (Something that Iβve heard is quite common in organizations β I think it was Continental Airlines who first told me that high performing / high potential leaders were sent to do a rotation in internal audit!)
Pat Shanahan is my leader. He is an incredible leader and empowers his team to experiment with new ways of working π
β I did a rotation in Audit last year after coming from the IT org! Absolutely believe IT folks show go to Internal Audit and vice versa. Definitely something that should be in more orgs.
We love having IT associates rotate through audit. We learn a lot from each other!
Kudos to you, @culpe2! Looking back, I wish I had done a year in internal audit somewhere in my career β so few other places you can actually see end-to-end processes.
If anyone has more interest in the Product Model designs, I did a session last year that goes into detail on how we organized this. Check it out!
Developers rotating to any place with repetitions... that's a lot of scripts and loops that get written π
To auditors: βwe canβt have you inflict your waterfall processes on us.β (I actually never thought that was something you could sayβ¦)
βbecame part of our IAM team; attended our standups, sprint reviewsβ¦β. π€―
π‘ βwhen the auditors show up, theyβre really just generating demand for your team,β might explain a lot of the typical reactions to the auditors showing up.
From @lucasc5 describing how the entire audit engagement model is (I thought inherently) a waterfall approach; scope, plan, fieldwork, report.
Typically auditor asks for verification/evidence. Ideally DevOps pipeline can(should) generate all the artifacts.
Actionable Insights... that's a real improvement to Insights
βself organizing teamsβ β βwe included someone very well versed in auditing standards.β
The 2021 State of DevOps report had some feedback on documentation which was: We found that teams with quality documentation are: β’ 3.8 times more likely to implement security practices β’ 2.4 times more likely to meet or exceed their reliability targets β’ 3.5 times more likely to implement Site Reliability Engineering (SRE) practices β’ 2.5 times more likely to fully leverage the cloud
@hyasar - absolutely! most of these automated pieces in the CI/CD generate checks and automate evidence that an auditor would be looking for. Automated testing is absolutely your friend in an audit.
(When I was IIA member, the audit standard was AS2, then AS5. I wonder what it is nowβ¦)
βI didnβt have to go project manager to get updates β instead, I could just go to planner board.β
βpush method: project manager assigns control tests to staff. instead, we pulled work: staff self-assigned their work.β
(better capacity balancing, and better match between assignments and personβs skills and goals!)
Daily stand-ups with audit sounds really scary..but not at Nationwide!
@topo.pal If it scares you, do it more often π
So, maybe the problem is that we're not in a daily collaboration with audit.
attended developer daily team standups! (often requests for information or followup for documentation; goes to point of contact, get routed, clarify requests⦠took days or weeks!)
(leans back, takes long drag from cigarette, and says, βperfect.β π
The audit "work" was managed the same as other demand. Engineers pulled cards and worked audit requests, the same as the other requests
using standups, we got info during standup, or later that day.
you solved the 'bring me a rock' problem with daily standups. awesome!
the journey toward psychological safety in that context is fascinating
The interactions that @tod.bickley is describing is like exact opposite of some stories Iβve heard: βthe auditors of coming; make sure you donβt bring up X, Y, or Z.β
Psychological safety is critical. You need to be able to have tough, but "safe" conversations to move forward
This will naturally lead to a CA pipeline any day now.
I have seen stand-ups and sprints save or kill an effort. What are good sources for practices or approaches to evolve great interaction?
At the foundational level, you need to have buy in and trust. The process is the process.
@tod.bickley That makes sense, I guess my challenge is I am a matrixed employee so I see many varying levels of maturity.
> To be independent, the auditor must be intellectually honest; to be recognized as independent, he must be free from any obligation to or interest in the client, its management, or its owners. https://pcaobus.org/oversight/standards/auditing-standards/details/AS1005#:~:text=To%20be%20independent%2C%20the%20auditor,its%20management%2C%20or%20its%20owners.
@lucasc5: βthe key is maintaining our decision rightsβ
βitβs not @tod.bickleyβs decision; those are our decisionsβ (Nice)
@lucasc5 How about the Five Ideals of Auditing next?
I think PCAOB is just a US thing: Public Corporation Accounting Oversight Board β the folks who audit the auditors, created after Enron, Worldcom, etc.
βOkay, why would one want to partner this way with auditors?β
Hmm... do these map somewhat with the DORA metrics?... hmm
@ferrix - That's next on my list: determining consistent metrics to measure success and progress as we expand our use of these newer ways of working.
Wow. 2x more coverage, 1/2 time required to open issues; 3x issues with progress
Happy about spending less time being audited?... "Surprise".
β[Auditors] were very positive, worked hard to understand and communicate gaps; all members were enjoyable to work with.β π₯
(We missed @tod.bickley and his team so much, we created some more findings, just so we can hang out again. π
"Hey, we found a discrepancy, don't worry, we also brought cake."
I ask myself if there are enough auditors available to join all dailies of the teams? or are they join just selected DevOps Teams?
@alexander.fritz - The auditors join the daily stand-ups when we're doing specific work in that space.
(In all seriousness, this is one of the most startling presentations Iβve seen β I told them it was like another 2009 Allspaw/Hammond β10 deploys/per day by dev & ops cooperationβ presentation.)
continuous authorization/auditing - a dream we have in DoD
@scott.prugh Maybe there should be something for the auditors in the pipeline then...
so auditors also want to help us π sure they do... but in many cases is feels soo much different ... let's pass more of those messages, great story to hear!
THANK YOU, @lucasc5 @tod.bickley!!! Wild, right?
"Person who is doing the audit should not be able to close the issue" - just kidding
β‘ And now, a special presentation from the co-authors of Team Topologies, @me1208 and @matthew β‘
@vmshook we have been working on continuous auditing.. maybe something for Vegas!
Hi everyone! I hope you enjoy the presentation πΏ
Hooray! You can also learn more in @me1208 and @matthew new workbook: https://itrevolution.com/remote-team-interactions-workbook/
Guys you've got to group your slack channels π (aimed at many of my colleagues)
@matthew are you sure itβs not in the kubernetes channel???
\@here: I need help with database provisioning! (Notify all 3147 people? YES!)
Just do it! Worst case scenario none of the 3147 people can help, right?
Sadly, some organizations have like 10 or 50 Brents π
βcheck Slack one more timeβ¦ who is this rando, and why are they bugging me?β π
@ channel Can you help me with creating a database? No?
We actually insist that you record your own role play video π π
lolz. I have on my list to RE-read your book. Recording videos is generally beyond possible where I work, but I definitely make my Marines role play
I wonder if this guy really bought all those books...
Buying and reading are two separate things π
Yes and buying means you have them locked away on your Kindle.
I like GitLab documenting the change in the handbook and notifying the greater team when there is a major change. Not just change in Slack naming.
βneed 10x throughput and scalingβ. βI need it in 3 daysβ
hilarious π and useful, and i am on my way to buy the book π
Have you worked on a platform team before, Virginia? You seem familiar with the approach π
Now look at these guys... this collaboration is not working at all... and I bought their book about teams... π
@genek Says this every time he drops me off somewhere.
"Would you recommend this database team to your enemies?"
βI canβt believe we dropped all the production tables two years ago β and now weβre both working here! Wild, right?β
I have to say that this format is super cool. This is like one of those tech talks where you do a live coding session, just that you are showing how code ends up being bad :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing:
and talk always helps ... no matter how disciplined we are with shared understnading documents
Is there a curl
version of this API?
Slack workflows and modals could be super helpful for some of those situations π
We even have a workflow to keep on top of watering the office plants π
In the workbook there are some interesting Slack automation examples from a UK company called Autotrader, e.g. creating/updating a channel based on metadata in a service repo.
I would caution Slack is a great tool but security should be included to protect before implementing a workflow or app.
@me1208 Iβm dying to know β does @matthew get his 10x scaling needs met?
Nah, this was just for the camera. Bob ended up doing all the scaling and db support work for the walking tours team, weekends and all. Bob has left the company.
Or⦠(They study the problem, bring in consultants, and after a week, they upgrade the VM from f1-micro to n2-standard-2, everyone happy)
Are you sure they did not just produce a powerpoint explaining the basics of amazon machine sizes after one week?
I am on the fence with booking meeting through two Miro spaces. Would raise the threshold from the Outlook abusive behaviours of stealing time without asking... but the UX is actually more pleasant.
Thank you for the feedback and helping make this presentation more engaging!
Thank you so much... I have been reading Team Topologies and this was a great way to complement.
Thank everyone for awesome Day 2! Day 3 is fantastic, too!!!
Reminder: Please submit your feedback for the talks you attended. Itβs so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png
Adventures in Agile Auditing: Donβt Just Survive Your Auditβ¦ Thrive in it! (Nationwide Insurance) Tod Bickley, AVP Information Risk Management for Identity and Access Management, Nationwide Insurance Clarissa Lucas, Technology Audit Director, Nationwide Insurance π Now available for sharing: https://videos.itrevolution.com/watch/708649004/ How to Help Teams Interact in a Remote-First World - A Team API Role Play Example Manuel Pais, IT Organizational Consultant, Co-author of Team Topologies Matthew Skelton, Founder at Conflux, Co-author of Team Topologies π Now available for sharing: https://videos.itrevolution.com/watch/708649039/
Regarding Auditing: Anyone around here having knowledge regarding Cloud ITAR compliance? If so, please let me know, I'd love to chat with you!