This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-06
Channels
- # ask-the-speaker-track-1 (249)
- # ask-the-speaker-track-2 (114)
- # ask-the-speaker-track-3 (244)
- # ask-the-speaker-track-4 (175)
- # bof-leadership-culture-learning (6)
- # bof-project-to-product (10)
- # bof-sec-audit-compliance-grc (2)
- # demos (9)
- # discussion-main (1290)
- # faq (6)
- # games (20)
- # games-self-tracker (1)
- # gather (22)
- # happy-hour (52)
- # help (44)
- # hiring (19)
- # lean-coffee (12)
- # networking (3)
- # summit-info (122)
- # xpo-adaptavist (7)
- # xpo-anchore-devsecops (9)
- # xpo-aqua-security-k8s (2)
- # xpo-basis-technologies (2)
- # xpo-blameless (2)
- # xpo-bmc-ami-devops (2)
- # xpo-cloudbees (4)
- # xpo-codelogic-code-mapping (2)
- # xpo-dynatrace (1)
- # xpo-gitlab-the-one-devops-platform (1)
- # xpo-granulate-continuous-optimization (2)
- # xpo-infosys-enterprise-agile-devops (2)
- # xpo-instana (4)
- # xpo-itrevolution (3)
- # xpo-launchdarkly (11)
- # xpo-logdna (1)
- # xpo-pagerduty (8)
- # xpo-planview-tasktop (4)
- # xpo-rollbar (2)
- # xpo-servicenow (1)
- # xpo-shoreline (3)
- # xpo-snyk (4)
- # xpo-sonatype (5)
- # xpo-split (10)
- # xpo-splunk_observability (5)
- # xpo-stackhawk (4)
- # xpo-synopsys-sig (2)
- # xpo-weaveworks-the-gitops-pioneers (5)
Reminder: Day 2 is starting now β opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.
β Excited to welcome @sanjeev.sharma and @john.comas who will be presenting, The Art of Platform Engineering β
The statement about how itβs not self-service if you need to get a dozen approvals to do it rings so true!
Timely session @sanjeev.sharma, @john.comas - as most of the enterprises are figuring out Platform as Product , Platform Engineering. Thank you for sharing Truist work here, much appreciated.
Continuous, Continuous, Continuous, Continuous and... :drum_with_drumsticks: Continuous
reminds me of the five Dβs of dodgeball: Dodge, Duck, Dip, Dive and Dodge. π
Never silo your DBAs π (I always that it is the other way)
What are some resources to learn about implementing CD practices with database changes?
Happy to chat with, we introduced some capabilities here. Not completely CD π
Iβve found OctopusDeploy to have some good blog posts on this topic to start out with: https://octopus.com/blog/why-consider-database-deployment-automation Some general concepts there that arenβt necessarily tool specific.
DBAs! I love the Phoenix Project deployment story: βuh oh, weβre 1 hour into a 9 hour database migration, that should have taken 15 minutes!ββ π
Standardization of Tools!!! I am fine with this as long as I have my tools in that list! π
This can become a "religious war". Introducing the idea of Tech Management to my org (Tech Governance as a term didn't resonate)
But basing the conversation in economics really helps (what is the TCO and value prop of a new language/framework/etc. vs leveraging an existing one?)
TCO and ROI maybe, if TCO is lower on a single tool (likely) but induces costs of delay which are magnitude higher, there may be a case for breaking standardization, no?
yes! the point is that there should be a business case. tech selection is a business choice
Agreed, just witnessing the risk of selection driven by some infra team in charge of tools and financial rationalization (aside the fact that in quite some case there may not be additionnal license cost with free stuff).
@sanjeev.sharma - question- What functions covered in "Platform Engineering" and "Platform DevOps Automation" - did I miss that poinit?
Standardizing tools to about a 100 instead of a 1000... I'm game... but one toolset to rule them all is a perpetual modernization every 10 years... just like before DevOps.
YMMV... we are still not rid of COBOL but nobody does Scala anymore.
That Pendulum keeps swinging back and forth every few months, and even more faster as we add more options
I guess the question is: How many simultaneous enterprise architectures is enough?
@lbmkrishna Platform Engineering covers the whole stack, so to speak - the slide I showed earlier. DevOps Platform is the App delivery pipeline part of it
As we move from Dev to Ops, our tools narrow... - sounds like a 5-lane highway narrowing down to 2
@john.comas This is so fantastic βΒ understanding the nature / consequences of changes.
@ganga.narayanan Or like the TVA pruning variants (sorry, Loki reference)
@john.comas This story about ecommerce order management system is one of the best Iβve ever heard!!! I LOVE THIS!! π
@sanjeev.sharma - I will be keen to understanding more about the "GRC" part of it both - Platform Engineering and DevOps Platform Automation. What tools/automation put in place to satisfy those requirements?
Customer orders product; customer cancels orders; order cancellations get stuck in queue; orders fulfilled and sent to customer, but never charged for it. Very bad business model. HAHA! I LOVE IT, @john.comas!
Danger of slow feedback loops; wow, that should go into a future edition of Phoenix Project. Weβll call it the Comas disaster. π @john.comas
βBuild the platforms not for the bank weβre now, but the bank weβre planning to be.β βΒ @sanjeev.sharma
@lbmkrishna The GRC toolset is still immature - every tool has their own implementation of how to do GRC. We need to get to standardized policy-as-code
My reflections are "Provisioning VM on-prem: 2 weeks; Provisioning VM on Enterprise Cloud Platforms: 6 weeks"
I love the TVA analogy! Pruning the timeline. π
I still see that Excel rules the GRC space - @sanjeev.sharma,I agree we need to get better
@rothdm Variants! π this morning, I watched the presentation that @emoshmosh gave at GitLab conference last year βΒ I think the birth of the amazing DevX story was initially due to a heavily modified GitLab instance, created by an enterprising dev team. π
To further brag on @emoshmosh - he's taken on the role of DevX Technical Director. Yay!!
Congratulations! I loved the stories you told, @emoshmosh!! π
But how do I find a better way? I wish there was a better way, i just don't know how to get there.
How large is the team building and supporting the platform itself?
@brianspo Still early stages of composing the individual services into a true platform
Thank you so much, @sanjeev.sharma and @john.comas β would you be willing to give us an update next year at DevOps Enterprise? As you said, itβs not every day that a top 10 bank appears out of nowhere!! π π
Thanks for sharing. I really enjoyed see your platform journey.
β¨ Get ready to hear from @jeremy.castle.lvh0 and @ryan.chambers.rse9 from State Farm, presenting Governance, Compliance, and Risk in the SDLC Can Be a Fun Event! β¨
@sanjeev.sharma Quite interesting, may not be able to provide you with this help, as having done it relartively ambrionic maner till now, but interested in connecting as normally at the begining of starting something similar
Great topic following GRC & SDLC cc - @sanjeev.sharma, @tapabrata.pal
Would love to connect @christian.lefevre - @sd_architect on Twitter or sanjeev.sharma(at)http://truist.com
Thank youΒ @sanjeev.sharmaΒ @john.comas!Β Loved the presentation! btw about #3 - I found it a combination of mandates + "build it and they will come" .. very relatable!
βthe tons of technology here at State Farmβ βΒ 2900 technologists! βΒ @jeremy.castle.lvh0 @ryan.chambers.rse9
Thatβs great to see another team with Delivery in the name! We are Delivery Engineering here at Columbia. π
One of my products in the suite is actually called Delivery Engineering too π
Howβs gitops working out for you? I havenβt had an opportunity to see it in action at a company yet.
It's going really well. Our internal k8s and AWS deployments all require it. We've gotten really good feedback from our development community so far.
It makes deployments a breeze by following a pattern that all of our developers are already familiar with and using the same tooling that they are working in every day.
Is it a commit for them to their existing code repos or is this more of a separate set of repos that they use to interface with your deployment tooling thatβs separate from the code build and packaging?
We use different repos that we call config repos
...when ready to deploy to production the pipeline will push the changes (terraform, artifacts, etc) to the separate repo. Then a merge request is created and thats what the manager will approve.
Nice! So the config repo handles the metadata about what to deploy without being tightly coupled to the code that built the artifacts. I like that decoupling of the pipelines, I can see the benefits in flexibility that provides.
@jeremy.castle.lvh0 @ryan.chambers.rse9 if you get time please join today's bof on audit compliance
This is really great stuff, @jeremy.castle.lvh0 and @ryan.chambers.rse9!
Thank you! We are excited to see what we can do with this framework next year and see how we can bring our InfoSec and Risk partners along to automate our governance.
@jeremy.castle.lvh0 - How large is the delivery excellence team and how does it measure in comparison to the size of the org youβre building these solutions for?
Good question. We are about 60 people (including leadership). We have about 7000 IT folks at State Farm. About 3000 of those are in developer/data/infrastructure roles.
Thank you @jeremy.castle.lvh0 and @ryan.chambers.rse9!
Great presentation! I'm curious to know how this framework helps with external audits of your release process? Does it make the audit smoother or is it hard explaining to auditors what's happening and how its addressing risk in the process?
@jtabron That's a really good question. We haven't moved into the mode for our Auditors to start using it. They have looked at our GitOps flow and have been very happy with how they can now look at a single flow to find all the steps in the approval for a change (Testing, Sign off, validation, etc.).
Thank you. In this framework, can a developer deploy code to production that wasn't tested or peer reviewed? You may have touched on this but I may have missed it.
They cannot. We have hard controls in place that stop that from happening. So there needs to be an sign off by a member of leadership before changes can be pushed into production. This framework will allow us to also verify the test artifacts and quality of the change in the future.
@jeremy.castle.lvh0 @ryan.chambers.rse9 do you think it would have been as feasible to do this if you were using something other than GitLab for your pipelines? e.g. tight integration between your version control system and your CI system, with approval capabilities baked in, etc?
I don't think so. We spent several years positioning out critical web platforms to only allow deployments using pipelines. This gave us an easy choke point to start firing off events. We had also started wrapping our developer tools in CLI's as well, so that became a natural event point as well.
π Back from the break, we welcome @jorge.castro,who will share: ENSUCKLOPEDIA OF DevUps - An improvised glossary to help us for understanding of What DevOps should not be or become!!! π
@jorge.castro is changing the presentation game. This is awesome.
20 years EXP and all DevOps Certs, And PHD in DevOps and be personable..
seems like missing a section on Blame Shifting.... : ) it is not DevOps it is the old ways and the failure to adopt.... π
Only problem is I can't tell what the actual advise is in between all the satire
Game Approach == like Squid Game - you lose you die...
@jorge.castro great job.. made me happy
i think it's important to look at all the counterpoints to these ways of working as well. This was insightful
I had not seen the periodic table of devops tools before. Interesting. https://digital.ai/periodic-table-of-devops-tools
Thank you all team, I do appreciate your words. At the end, something that I want to foster is about DevOps is about people also so besides CI CD CT CM and CStuff lets also practice Continuous Fun : )
@robert.printz new elements being discovered every day..
β And now, we welcome our friend, @mr.denver.martin who will share: Tales From the DevOps Loop: 4 Teams Approach to 1 Common DevOps Framework β
HI Everyone, Just one of my many Tales from Implementing DevOps at 3 different companies with different size team..
Thank you very much all. Hi team, I think I have some of your in my Linkedin but I would like to have you all so we can keep the learning and sharing experience after the event, I will be honored if you can add meΒ https://www.linkedin.com/in/jorgeluiscastrotoribio/Β Thank you and keep safe : )
@mr.denver.martin How big of an organization was this? Was this a small team as a part of a larger org inside of a larger org?
The Ops teams for Network and Platform were small compared to the other teams...
Apps Support was a lot larger, the Dev Teams were ever larger..
I am excited to hear about how you were able to change these command and control/task assignment leaders' approaches
Unfortunately, they were not able adopt new ways of working, they had to be encouraged to find a better fit outside the company...
We tried to change the way they worked first, as it is better to grow from with in, however in this case we had to replace... but one of the replacements for team 2 was a promotion from with in.
Iβm curious to know how security interfaced with these teams
InfoSec had been the Team of NO, but during this time they became partners and work to figure out how fix what we had and were involved later as partners at the beginning of new works... so it improved over time - due to Leaders asking them to join and they even attended BPMs later and gave great input.
I hope to make Tales from the DevOps Loop a series... so I can make a note to add InfoSec notes in a future talk...
I am also open to partnering with other implementers to do talks with...
I can also provide the template from Team 4 for promoting the learning of new technologies and show the company was supporting their upskill.
Our team is looking for tips and tricks for running effective blameless post mortems. Any thoughts on what works to get people focused on the problem vs the people involved?
yes, hit me up, <mailto:mr.denver.martin@gmail.com|mr.denver.martin@gmail.com> we can set up a time to talk.. it was important for us..
yeah, that was a major win for us,,, it was hard to keep it blameless, more than willing to share how we did that... I will also note to do a deeper dive in future Tale video...
π£ Shout out to our next speakers, @rajat.sud and @donalmeida2 here to present, Got Score? Scaling DevOps Adoption π£
Thank you @mr.denver.martin A great example of real world adoption into a sceptical organisation. Certainly provides a blueprint for others to follow
Excited to be here...
We're actually quite a lean team, supporting as many as 300+ engineers
Would the members of DevOps CE come from with in the company or were they hired from outside and a new net add?
A little bit of both - we did purposefully build the CoE and funded it via some baseline vehicles
Was to move from within seen as a promotion? or was it a lateral?
Definitely triggered a lot of interest - enough for us to be selective π
We have a CoP that Rajat Runs where we bring together engineers from the delivery teams and help them implement the practices
Are you also doing SRE? I see that SRE comes after DevOps is stable... so curious if you have added that level and how you staffed that group..
I live for the day when DevOps is not Unicorn but a Thoroughbred herd.
Gamification. π Were there any teams that had trust issues about it?
There was skepticism - but transparency was key!
We did make available all our raw data so folks could do their own analysis and 'buy in'
It could be good to give extra points if they share knowledge and help another team...
Great thought - will have to think about quantifying that though π
Primarily through the APIs exposed by our tools
Given that it was baked into all our performance goals - they had the liberty to choose how best to spend their (demonstrable) improvement ;)
Do you think the scoring may de-incentivize teams without a competitive spirit or that may feel they aren't empowered to change?
Excellent question. We did have teams saddled with legacy technology - and we did realize some of this just served as a FYI for them...
Provide them information about company-wide initiatives
Do you give a score for improving or only current metrics?
Yeah, it sounds really interesting but I know some that for some of my teams this could have an undesired effect, particularly for those groups I mentioned (folks the prefer not to be competitive, folks feeling unempowered).
This males sense as part of an overall strategy where they are told they have permission to fix things. π
The capabilities scored were intrinsically woven into our self-serve platform
We did something similar at Walmart. Talking to @troy.magennis a couple years ago he suggested gamifying improvement as well.
So, βyou may not be amazing, but youβre headed that wayβ.
Exactly - the score spawned an improvement action plan - that clearly spelt out the "what/how"? We, at the CoE peppered in the "why".. While the team was empowered to decide the "when" in line with their priorities (some gentle nudges helped ;))
Did the scoring / adoption approach yield observations on the capabilities offered/used themselves? For example, maybe gaps or impediments for specific technology platforms?
@scott.heaberlin, just realized that you are a part of the BCBSA family π We do have some novel tech stacks which didn't quite fit the mold off the bat - though the principles of DevOps still apply. Once designed, they were roped into the scoring.
Wondering if you've thought about how this approach contrasts with recommendations in "Sooner, Safer, Happier" to not measure activity, or "how DevOps or Agile" are you, vs. basing paths based on context of each team and their impediments, and measured by outcomes?
Interesting..measuring activity was just a means to an end - to provide some actionable data and shake the inertia and challenge the status quo
And provide some quantifiable data points that our leaders could use to facilitate conversations with their team members... its a part of our overall strategy of being more data-driven
I would love to discuss this further sometime. While this is easier to measure, I have found more benefits from having teams own their path and not try to meet a general score based on implementing a checklist of practices. Maybe there is an option to score outcome improvement percentages, vs. what practices are put in place.
@halfmoondad, happy to connect. Yes, we are coming to the realization that it is better to trend (outcomes) rather than compare. However, the question was trend what? And this provided us some basis for deciding what to measure - and then how to improve it...
@ryanewtaylor what if you approached it like a combined points approach like fundraisers where the combined teams points everyone wins something like 1/2 day off or something...
yeah, that may work well. I myself prefer environments that are more collaborative than competitive, even across teams. Think actors in a play vs sports teams. Everyone works towards a shared goal vs having competition between teams.
@rajat.sud thanks for sharing, i hay a few questions: 1) what has been the success of having an adoption score rating? 2) what kind of measurements do you for pipeline usage, capabilities, practices? 3) is this measurement is complementary to having a maturity level that focuses more On practices and tool usage? 4) can you Share with us the type of questionnaire you do to get these scores?
We are doing something similar at RBC. @rajat.sud would love to connect to share ideas!
@juancarlos.chang, we'd love to..
@eleonravinez thanks for the questions. Hope these responses shed more light 1. Scoring has been quite well-received. It provides an actionable piece of information that can be trended. 2. We have as many as 15 measurements right now - from build automation and stability, deployment automation and stability across environments, Securitization, Code Quality, test automation, etc. 3. The measurements when aggregated to the product or portfolio did facilitate questions about maturity - the tools usage was part of that - but the key is that it facilitated conversations about team practices 4. We graduated from questionnaires to automatically eliciting this data from the usage of our tools, and then applying applying some calculations (per our methodology) and updating in real time
Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png