This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # ask-the-speaker-track-1 (705)
- # ask-the-speaker-track-2 (287)
- # ask-the-speaker-track-3 (195)
- # ask-the-speaker-track-4 (356)
- # bof-american-airlines (68)
- # bof-arch-engineering-ops (28)
- # bof-covid-19-lessons (4)
- # bof-cust-biz-tech-divide (2)
- # bof-leadership-culture-learning (5)
- # bof-next-gen-ops (10)
- # bof-overcoming-old-wow (7)
- # bof-project-to-product (5)
- # bof-sec-audit-compliance-grc (5)
- # bof-transformation-journeys (6)
- # bof-working-with-data (3)
- # burnout (31)
- # demos (72)
- # games (114)
- # genecon (1193)
- # genecon-help (197)
- # happy-hour (252)
- # hiring (25)
- # lean-coffee (30)
- # networking (20)
- # project-to-product (21)
- # psychological-safety (9)
- # summit-info (798)
- # summit-stories (4)
- # xpo-atlassian (10)
- # xpo-datadog (6)
- # xpo-delphix (32)
- # xpo-digitalai-accelerates-software-delivery (9)
- # xpo-gitlab-the-one-devops-platform (5)
- # xpo-harness (3)
- # xpo-hcl-software-devops (9)
- # xpo-infosys-enterprise-agile-devops (10)
- # xpo-instana (8)
- # xpo-itmethods-manageddevopssaas (9)
- # xpo-itrevolution (20)
- # xpo-launchdarkly (6)
- # xpo-logdna (12)
- # xpo-logzio (2)
- # xpo-moogsoft (4)
- # xpo-muse (7)
- # xpo-nowsecure-mobile-devsecops (6)
- # xpo-opsani (16)
- # xpo-optimizely (3)
- # xpo-pagerduty (10)
- # xpo-pc-devops-qualifications (9)
- # xpo-planview-tasktop (14)
- # xpo-plutora-vsm (10)
- # xpo-redgatesoftware-compliant-database-devops (6)
- # xpo-servicenow (18)
- # xpo-snyk (11)
- # xpo-sonatype (43)
- # xpo-split (34)
- # xpo-sysdig (29)
- # xpo-teamform-teamops-at-scale (20)
- # xpo-transposit (11)
- # xpo-tricentis-continuous-testing (5)
Welcome our next speakers @christopher_mcfee and Mick Miller!
YASS! Starting with gifting the Projects to the stakeholders is SO necessary. In particular in banking
How did you approach the dojos, especially with a small team? Standard curriculum or custom for each team? Hands on vs Lectures? Immersive or training academy style?
It's really a mixture of standard curriculums layered with custom content for specific contexts (example some teams may use a particular tool versus another team). But we focus a lot on capabilities not particular tools with the standard content
We started with custom team based sessions but found we didn't scale and shifted to standard curriculum which helped us reach numbers last year (>1000 engineers) but I think this year we will swing the pendulum back the other way, similar to what you described.
We do a mixture of training academy's that focus on long or short engagements with our partner teams.
Very @Spotify of you @christopher_mcfee - they were telling me last week they took 3 months to audit and catalogue all the licenses of tools their teams used
We like to switch out the best products for the best job. That means understanding the why and not necessarily tools. Understanding CI practices versus just jenkins, gitlab ci, or pipelines.
Nice - we do see a common gap in teams to understand the principles behind the tool. Jenkins is great for showing how a team thinks
Cool @christopher_mcfee It's about doing the job and keeping clarity on the point for the process not what we do it with! My favourite "no" is from enterprises that try and do the same job our software does "with pain, pen &paper" as I call it. It's majorly better than the ones who don't even know they should do the job at all or claim a yearly survey or a 360 does the same thing or that they are only "allowed" a certain license or tool for their team
The longer engagements focus on two to three 2 week sprints.
We do provide a paved path for teams, but also want to be able to use the tools they are comfortable with too.
Application profiling, observability (customer experience is huge here in understand how people are using our products and services), and we a cadence with all our partners.
Do you also instrument your SDLC as well to measure flow rate or areas of potential problems (code rework)?
@matthew.cobby We do! We measure from ideation (creation of requirements) all the way through production deployments.
what do you use to visualize your data? something like Looker?
Most of our data is currently aggregated through an Elastic cluster. From there we can visualize with Kibana or whatever tool people are comfortable with by getting at the data from Elastic's API
We've built an event supply chain to aggregate everything into Elastic.
@christopher_mcfee how did you folks deal with DevOps in database space? Changes are slightly harder to implement in database space.
We have teams that use tools like liquibase and flyway to version control schema changes and feature flags when new tables/columns are created or need modifications.
liquidbase/datical is awsome, we already being deploying our changesets with it, and managing the release and control version with azure devops
@christopher_mcfee are your “run” teams your application software product teams or infrastructure-centric Ops teams?
That's a really good question. In our cloud native architectures we've shifted left as much as possible. So the software product teams are currently following the you "you build it, you run it" mindset.
great, thank you. That’s the right answer, but wanted to be sure I knew the context here!
We paved the road for them. If they move to the desired tools the wheels have been greased the kinks have been worked out and we've tried to make it almost zero touch. If they want to use something outside of that there may be additional process they have to overcome.
Thanks. Did you have a timeline to end your team's support of orphaned tool ecosystems?
It's contextual to the tool(s) in question and if we are paying vendors for licenses or not.
curious...with terraform...did you use Cloud Foundation Toolkit for templates or create your own?
We've created our own for many thing as we manage and maintain a number of on-prem items as well as the cloud with the stack, so we needed the flexibility.
very nice....you manage your on-prem infrastructure with terraform as well?
that would be cool to implement....thanks for the information...we are on a similar journey with gcp and our solution stacks are almost identical....but you are further along than us...we only have 54 projects...
It's definitely been fun watching the shift in how we manage our stacks.
i moved from running the hardware (datacenter) team to "señor cloud"....now on the platforms team building out gcp solutions...current struggle is the infra team is "wired to resist" all changes and dont want anything to do with terraform or any IAAS
What level of ownership do folks have in your infrastructure? How much do your app teams need to know about the underlying infrastructure such as k8s, GCP, VMware, Terraform, etc
We've abstracted as much of the underlying traditional "infrastructure" so that our delivery teams don't need to know things like vmware, terraform, gcp, if they don't want. We provide consulting if they need help but encourage if they want to go fast they can maintain and manage as little or as much as they want.
Was your database in the cloud as well? How would be maintain data integrity between releases? Did you have 0 downtime between release to production?
We do. We are doing a lot of Database as a service in the cloud. So we don't have to manage or maintain VMs. Feature flags for 0 downtime and version controlling schema changes, etc.
Have you implemented Feature Flags? And if so, just for feature delivery or in all aspects of your transition to CI/CD like the migration of services? D
We have in a number of our layers (widgets, services, etc.). It allows us to turn things off if they aren't ready for prime time or if there are issues with the functionality.
Assuming you built your own system for feature flags? Do you use it just to release new features or as you were migrating services as well?
Would be curious to hear about what your process looked like when deciding to build or buy for feature flags and how robust the features are? Thanks again for answering!
Most of our use of feature flags is our own system. Essentially a database that houses the flag and then in the code calls to the database and if statements.
That's great! Did you evaluate third party tools to compare costs to build and maintain vs. using out of the box solutions before building or was it before the tools were as popular as they are today? Wondering if the flags are tied to any advanced capabilities like experimentation?
We didn't at the time when we were implementing. We need something fairly quickly and couldn't get through the third party onboarding fast enough to meet the need. So we did an minimum viable to show what was possible, ran some experiments, and never looked back.
This has been great information and helps greatly to help understand your thought process and serve our customers better. Have you looked at current costs to maintain the feature flag platforms vs third party platforms or have plans to investigate in the future?
If a use case pops up we may potentially look at a third party platform. It's also contextual to the application being delivered as well and what their requirements are.
Makes sense. Thanks again! I am so grateful for this information as it is tough to get inside your thought process to making these types of decisions. If you are okay with it, would you be okay with connecting after the show? I'd love to share some ways our Split Feature Delivery platform has helped Enterprise companies like Chase WePay other large financial institutions to reduce costs and experiment more effectively. I notice a lot of challenges presented in the talk that we've helped companies with and would love to share if and when the use case pops up.
Can teams contribute changes to the TF and related code if they need to? What's the Op Model re:support & ownership.
From a TF perspective they can and do contribute with pull requests. For the foundational items within terraform we maintain the PRs but use a lot of CI practices to check syntax of things as well as policies.
I would imagine you require as part of the PRs approval that they include a body of tests to validate the new change going in and to verify it continues to work going forward.
Absolutely that would be ideal in most circumstances. Usually what they are changing we already have tests for them though, so usually they don't have to create new tests.
Great talk Chris and Mick! Amazing insight into the journey, thought process and the strategies you've implemented! Thank you!
Hi Party People… Getting ready for “Is this thing on?”
Glad to share this talk with y’all. Please feel free to ask anything at all. It was pretty challenging to fit years of work into 28 minute discussion so glad to try and add more detail here.
@christopher_mcfee What what the other platform you used to extend the tools? I heard Python, but I didn't catch the other environment.
Terraform, Ansible as primary and Go and Python to augment when necessary.
Continue the discussion from previous speakers at #ask-the-speaker-more!
Who's thinking about economic models for software development? Important shoutout from @bmmc about TCO of application development. Anyone else trying to model this? What resources have helped people think through this topic? What are the limitations and risks of translating app dev into financial metrics?
Great question. We’ve utilized ideas and methods from TBM and FinOps to try and jump start discussion for running IT as a business overall, but lots more work needs to be done to really tackle the economic model for software development, especially if your organization has finance ingrained in waterfall methods.
What's TBM in this context?
So the financial models are mostly based on the cost of underlying resources? Do you take into consideration cost to develop? Benefit to organization (revenue or non-revenue)? etc
we’re pretty good at knowing the cost to developed, the TCO for the development activity itself. the trick (and the hard part) is to understand did we actually achieve expected benefit?!?!
I worked in a backoffice org where the director was always willing to devote dev resources to a good-sounding project, but he always had his developers build in usage metrics and have his business analysts show him whether features that were requested were being used. If they weren't he'd have a discussion or two with folks that asked for the feature, and retire it if they didn't ever use it.
The bonus was that these check-ins made sure that the number of products+features wasn't infinite - the requestors had to have the bandwidth to discuss their needs each meeting. It also made use of the organizations time clear to the requestors and the executive committee. This practice among others turned the back office from a dumping ground into an agile org that was trusted to address issues cooperatively.
Adventures in DevOps! https://podcasts.apple.com/us/podcast/adventures-in-devops/id1475784710
We discuss a wide range of topics facing DevOps teams, from organizing DevOps teams, release management, cloud services, and the latest security trends.
If you find that a control fails a lot, what are some steps taken to remediate that? If not required by regulations, is it possible that the control itself is too strict?
Right now we have only implemented control monitoring from IT general controls that are directly tied to processes that we’ve agreed are required for regulations. BUT it could be used to monitor for anything that we think might improved the business of IT
I think it’s important that we be careful to not implement too many rules to fast otherwise people could get numb to the alerting.
If the control fails too frequently, that’s something we need study to see if there’s just a training opportunity, something that can just solved upstream, or maybe a more systemic or cultural problem
That is an interesting point. Controls sometimes fail due to a technology problem, or because of a process problem. Both come with their own challenges 🙂
@bmmc: are you using a out of the box solution for what you are showing in the demo or was it a automation/dashboard solution coded in-house? How do you collect the controls data, everything from SNOW?
the controls automation was built from ground up. basically it’s a framework to manage business rules that continually looks at data in a datamart, and then Tableau reports sitting on top of that for the humans
interesting. We have a lot of controls as well as the need to better manage the TCO of the thousands of applications we are running, but nothing like you showed today. We can definitely learn from that, so thanks for sharing!
aha, now that the talk got to the METHOD part I understand with more clarity your comment above
Welcome our next speakers for track-3 @amanda.palovcsik and @jim.collins!
Hello everyone. I hope you enjoy the presentation.
Hello! Hope everyone is enjoying the conference!
Looking forward to your session. "Us-vs- Them" is one of my biggest pet peeves 🙂
Very excited to hear your experiences as well! I think this is a topic many run into, and look forward to hearing some of what has worked well for others!
@amanda.palovcsik - Disney FTW! 💪
@jim.collins - whats your favorite bourbon in the collection of the moment?
Sundays are tense around here....
Love that "team first" attitude. We say "obsess with the bubble"
I like that! Fits well in many teams!
I had to fight my publisher for the subtitle of my book as they wanted "teamwork" and I kept saying "teams" - "the importance of TEAMS" - you're underestimating how needed the term is
100% - it's blurring the lines, and changes that meaning quite a bit! Can't wait to check out your book - that title caught my eye!
If you have time tomorrow at 14:05 we're on this same Track 🙂 speaking about Psychological Safety and Impression Management and towards the end there's a preview of the book cover Bloomsbury had to approve I show you guys 🙂
Excellent!! I'll check it out!
these challenges bring back nightmares about negotiating scope instead of aligning on outcomes… love this setup!
Like this concept of partnership, not supplier-customer divide, and practical advice how to start to build trust
Some of the easiest solutions aren't over-architected, but it is hard to be the one to go first - Jim's teams really embraced that, and the reception from business to do the same has been awesome. They do an amazing job collaborating!
'its amazing what you can accomplish when you don't care who gets the credit' - Love that
One of my favorites. First heard it from Kumar Garg, who had it written on his white board (worked for the White House during Obama). He presented at DevOpsDays DC in 2016 or so. I never knew it was attributed to someone else further back in history... who was it again?
Me too. Its probably the single item with the largest impact.
@jim.collins I'm surprised it "wasn't too hard", did you have a good level of leadership support?
We did. We had a great business partner who was motivated to find ways to improve, and open to the ideas we proposed.
Eliminate blame and collaborate...refreshing
“the product owner doesnt define the work, the team defines the work” Love it
As a PO I don't like it but I know it's true
It goes a little further than just requirements. No single person signs off on a release, the team does. It used to be that our testers signed off. The team is responsible for knowing when things are done.
"you can get a lot more done if you don't care that much about who takes the credit..." (or the blame)
I almost took that comment out, but its so true.
Have your business partners warmed to the idea of incremental delivery?
Yes. The speed to market increases made it easy to get them on board.
@jay.steinberg And once this was proved out - they champion the agile approach even more!
THAT WAS AWESOME
How did you convince business/finance to move from funding projects to funding products/value streams?
Not sure we did, completely. It's been a lot of work to secure budget sources. Because we've demonstrated that we can deliver fast and with high quality, we have always been able to find more work/money.
Thank you for trusting us enough to leave it in!
100% why we did that - we need trust in agile, and I may have stumbled more in person!! Yikes!
I love the rough edges
With your OKRs, how have your business responded to Key Result metrics?
The business is involved in the generation of OKRs. Part of that discussion includes how we will measure for success. They are part of the process, so they are not really responding to something we give them. They are our OKRs
Absolutely! We were having fun doing it, and it's who we are! Hopefully gave you a chuckle!
It did! You get so used to vloggers and other presentations that are edited to death. You two clearly had had this conversation before and/or practiced enough to make it seem natural. Loved it.
@nuwayser Thank you! I hope you took away some good tips!
@mylesogilvie Thank you for joining!!
Great talk thanks! That sounded like how to make a partnership work across formal silos. As opposed to the project-to-product / value-stream school of re-organisational thought. The approach here sounds like it has much lower barriers to entry and can be highly productive.
@philipday Glad you enjoyed it! This definitely addresses projects/teams where there are more formal roles and silos, but even pieces of this can be applied in any project.
Great synopsis. We still have plenty of silos, but when we work on things, we work on them together.
Thank you @jim.collins and @amanda.palovcsik!
Thank you to everyone who joined! @jim.collins and I will be available for comments for a bit longer! We'd also love to hear YOUR experiences, and some wins you've experienced building trust and partnership with your business and IT partners - we're always learning together!
My answer is everyone, you just have to make sure that doesn't devolve into no one.
It is really everyone, however, everyone needs to understand the future state culture. Talking about the culture frequently reinforces what the culture to be is and how you are moving closer to it.
Failure is first attempt in learning - share those! #psychological-safety in the context of change
@wendy.flowers, it has to start at the top, the highest level of leadership
@vasudha.prabhala I have seen where the team wants to change but leadership is "stuck in their old ways". Any suggestions?
Culturally there are two types of companies, top down and bottom up. I can't work for top down.
I was hired to do a job, if I have to get a people manager to approve technical work, that isn't a place where I want to work.
@wendy.flowers I would say, have a human to human conversation 1:1. Most of it comes from fear and inertia
the other way to do is.. show it in smaller areas ..and build on that enthusiasm
How often do you do the blameless post mortem and where?
@ffion we do it after every sprint (dev projects), monthly planning, quarterly and yearly
Thanks, I agree it works best when it's a natural part of the team's collective learning culture, it can be on the agenda anytime that is relevant.
the biggest thing though is are we truly learning or is it just a rhythm for the sake of it
In addition to blameless retros, you can build in shout outs to what people have learned. This can be uncomfortable at first, however, when you get past that, others will join in on what they learned and what they would do differently next time.
Agree @vasudha.prabhala occasionally what looks like strong commitment can turn into paying lip service
thank you, all that were able to attend. @kbrey and I highly appreciate it