We started with custom team based sessions but found we didn't scale and shifted to standard curriculum which helped us reach numbers last year (>1000 engineers) but I think this year we will swing the pendulum back the other way, similar to what you described.

We like to switch out the best products for the best job. That means understanding the why and not necessarily tools. Understanding CI practices versus just jenkins, gitlab ci, or pipelines.

Nice - we do see a common gap in teams to understand the principles behind the tool. Jenkins is great for showing how a team thinks

Cool @christopher_mcfee It's about doing the job and keeping clarity on the point for the process not what we do it with! My favourite "no" is from enterprises that try and do the same job our software does "with pain, pen &paper" as I call it. It's majorly better than the ones who don't even know they should do the job at all or claim a yearly survey or a 360 does the same thing or that they are only "allowed" a certain license or tool for their team

Application profiling, observability (customer experience is huge here in understand how people are using our products and services), and we a cadence with all our partners.

tks!!

Do you also instrument your SDLC as well to measure flow rate or areas of potential problems (code rework)?

@matthew.cobby We do! We measure from ideation (creation of requirements) all the way through production deployments.

Most of our data is currently aggregated through an Elastic cluster. From there we can visualize with Kibana or whatever tool people are comfortable with by getting at the data from Elastic's API

We've built an event supply chain to aggregate everything into Elastic.

We have teams that use tools like liquibase and flyway to version control schema changes and feature flags when new tables/columns are created or need modifications.

liquidbase/datical is awsome, we already being deploying our changesets with it, and managing the release and control version with azure devops

That's a really good question. In our cloud native architectures we've shifted left as much as possible. So the software product teams are currently following the you "you build it, you run it" mindset.

great, thank you. That’s the right answer, but wanted to be sure I knew the context here!

🙂

We paved the road for them. If they move to the desired tools the wheels have been greased the kinks have been worked out and we've tried to make it almost zero touch. If they want to use something outside of that there may be additional process they have to overcome.

Thanks. Did you have a timeline to end your team's support of orphaned tool ecosystems?

It's contextual to the tool(s) in question and if we are paying vendors for licenses or not.

We've created our own for many thing as we manage and maintain a number of on-prem items as well as the cloud with the stack, so we needed the flexibility.

very nice....you manage your on-prem infrastructure with terraform as well?

We do.

that would be cool to implement....thanks for the information...we are on a similar journey with gcp and our solution stacks are almost identical....but you are further along than us...we only have 54 projects...

It's definitely been fun watching the shift in how we manage our stacks.

i moved from running the hardware (datacenter) team to "señor cloud"....now on the platforms team building out gcp solutions...current struggle is the infra team is "wired to resist" all changes and dont want anything to do with terraform or any IAAS

We've abstracted as much of the underlying traditional "infrastructure" so that our delivery teams don't need to know things like vmware, terraform, gcp, if they don't want. We provide consulting if they need help but encourage if they want to go fast they can maintain and manage as little or as much as they want.

We do. We are doing a lot of Database as a service in the cloud. So we don't have to manage or maintain VMs. Feature flags for 0 downtime and version controlling schema changes, etc.

Thank you.

We have in a number of our layers (widgets, services, etc.). It allows us to turn things off if they aren't ready for prime time or if there are issues with the functionality.

Assuming you built your own system for feature flags? Do you use it just to release new features or as you were migrating services as well?

Would be curious to hear about what your process looked like when deciding to build or buy for feature flags and how robust the features are? Thanks again for answering!

Most of our use of feature flags is our own system. Essentially a database that houses the flag and then in the code calls to the database and if statements.

That's great! Did you evaluate third party tools to compare costs to build and maintain vs. using out of the box solutions before building or was it before the tools were as popular as they are today? Wondering if the flags are tied to any advanced capabilities like experimentation?

We didn't at the time when we were implementing. We need something fairly quickly and couldn't get through the third party onboarding fast enough to meet the need. So we did an minimum viable to show what was possible, ran some experiments, and never looked back.

This has been great information and helps greatly to help understand your thought process and serve our customers better. Have you looked at current costs to maintain the feature flag platforms vs third party platforms or have plans to investigate in the future?

If a use case pops up we may potentially look at a third party platform. It's also contextual to the application being delivered as well and what their requirements are.

Makes sense. Thanks again! I am so grateful for this information as it is tough to get inside your thought process to making these types of decisions. If you are okay with it, would you be okay with connecting after the show? I'd love to share some ways our Split Feature Delivery platform has helped Enterprise companies like Chase WePay other large financial institutions to reduce costs and experiment more effectively. I notice a lot of challenges presented in the talk that we've helped companies with and would love to share if and when the use case pops up.

From a TF perspective they can and do contribute with pull requests. For the foundational items within terraform we maintain the PRs but use a lot of CI practices to check syntax of things as well as policies.

I would imagine you require as part of the PRs approval that they include a body of tests to validate the new change going in and to verify it continues to work going forward.

Absolutely that would be ideal in most circumstances. Usually what they are changing we already have tests for them though, so usually they don't have to create new tests.

Terraform, Ansible as primary and Go and Python to augment when necessary.

Great question. We’ve utilized ideas and methods from TBM and FinOps to try and jump start discussion for running IT as a business overall, but lots more work needs to be done to really tackle the economic model for software development, especially if your organization has finance ingrained in waterfall methods.

What's TBM in this context?

sorry Technology Business Management https://www.tbmcouncil.org

So the financial models are mostly based on the cost of underlying resources? Do you take into consideration cost to develop? Benefit to organization (revenue or non-revenue)? etc

we’re pretty good at knowing the cost to developed, the TCO for the development activity itself. the trick (and the hard part) is to understand did we actually achieve expected benefit?!?!

The bonus was that these check-ins made sure that the number of products+features wasn't infinite - the requestors had to have the bandwidth to discuss their needs each meeting. It also made use of the organizations time clear to the requestors and the executive committee. This practice among others turned the back office from a dumping ground into an agile org that was trusted to address issues cooperatively.

that’s cool…thanks for sharing

Adventures in DevOps! https://podcasts.apple.com/us/podcast/adventures-in-devops/id1475784710

We discuss a wide range of topics facing DevOps teams, from organizing DevOps teams, release management, cloud services, and the latest security trends.

Right now we have only implemented control monitoring from IT general controls that are directly tied to processes that we’ve agreed are required for regulations. BUT it could be used to monitor for anything that we think might improved the business of IT

I think it’s important that we be careful to not implement too many rules to fast otherwise people could get numb to the alerting.

If the control fails too frequently, that’s something we need study to see if there’s just a training opportunity, something that can just solved upstream, or maybe a more systemic or cultural problem

That is an interesting point. Controls sometimes fail due to a technology problem, or because of a process problem. Both come with their own challenges 🙂

the controls automation was built from ground up. basically it’s a framework to manage business rules that continually looks at data in a datamart, and then Tableau reports sitting on top of that for the humans

if A < B then , if A > B then :white_frowning_face:

interesting. We have a lot of controls as well as the need to better manage the TCO of the thousands of applications we are running, but nothing like you showed today. We can definitely learn from that, so thanks for sharing!

aha, now that the talk got to the METHOD part I understand with more clarity your comment above

haha, maybe i posted that a little too early 😉

Go Badgers!

Sundays are tense around here....

I like that! Fits well in many teams!

I had to fight my publisher for the subtitle of my book as they wanted "teamwork" and I kept saying "teams" - "the importance of TEAMS" - you're underestimating how needed the term is

100% - it's blurring the lines, and changes that meaning quite a bit! Can't wait to check out your book - that title caught my eye!

If you have time tomorrow at 14:05 we're on this same Track 🙂 speaking about Psychological Safety and Impression Management and towards the end there's a preview of the book cover Bloomsbury had to approve I show you guys 🙂

Excellent!! I'll check it out!

Some of the easiest solutions aren't over-architected, but it is hard to be the one to go first - Jim's teams really embraced that, and the reception from business to do the same has been awesome. They do an amazing job collaborating!

One of my favorites. First heard it from Kumar Garg, who had it written on his white board (worked for the White House during Obama). He presented at DevOpsDays DC in 2016 or so. I never knew it was attributed to someone else further back in history... who was it again?

Me too. Its probably the single item with the largest impact.

@jim.collins I'm surprised it "wasn't too hard", did you have a good level of leadership support?

We did. We had a great business partner who was motivated to find ways to improve, and open to the ideas we proposed.

As a PO I don't like it but I know it's true

🙂

It goes a little further than just requirements. No single person signs off on a release, the team does. It used to be that our testers signed off. The team is responsible for knowing when things are done.

I almost took that comment out, but its so true.

Check you don't have two tabs open. I got caught out by that earlier.

Ah there you go! Thanks!

Yes. The speed to market increases made it easy to get them on board.

@jay.steinberg And once this was proved out - they champion the agile approach even more!

Thanks!

Not sure we did, completely. It's been a lot of work to secure budget sources. Because we've demonstrated that we can deliver fast and with high quality, we have always been able to find more work/money.

100% why we did that - we need trust in agile, and I may have stumbled more in person!! Yikes!

The business is involved in the generation of OKRs. Part of that discussion includes how we will measure for success. They are part of the process, so they are not really responding to something we give them. They are our OKRs

Absolutely! We were having fun doing it, and it's who we are! Hopefully gave you a chuckle!

It did! You get so used to vloggers and other presentations that are edited to death. You two clearly had had this conversation before and/or practiced enough to make it seem natural. Loved it.

and fun is contagious!

@nuwayser Thank you! I hope you took away some good tips!

You're welcome. I'm glad you enjoyed it.

@mylesogilvie Thank you for joining!!

You're welcome. Thanks for attending.

Cheers to you two. Sounds like a great change for all involved. 🍻

It has ben a great change. Thanks!

I could imagine this being a phase on a journey

@philipday Glad you enjoyed it! This definitely addresses projects/teams where there are more formal roles and silos, but even pieces of this can be applied in any project.

Great synopsis. We still have plenty of silos, but when we work on things, we work on them together.

My answer is everyone, you just have to make sure that doesn't devolve into no one.

It is really everyone, however, everyone needs to understand the future state culture. Talking about the culture frequently reinforces what the culture to be is and how you are moving closer to it.

@vasudha.prabhala I have seen where the team wants to change but leadership is "stuck in their old ways". Any suggestions?

Culturally there are two types of companies, top down and bottom up. I can't work for top down.

I was hired to do a job, if I have to get a people manager to approve technical work, that isn't a place where I want to work.

@wendy.flowers I would say, have a human to human conversation 1:1. Most of it comes from fear and inertia

Thank you.

the other way to do is.. show it in smaller areas ..and build on that enthusiasm

Thanks, I agree it works best when it's a natural part of the team's collective learning culture, it can be on the agenda anytime that is relevant.

the biggest thing though is are we truly learning or is it just a rhythm for the sake of it

In addition to blameless retros, you can build in shout outs to what people have learned. This can be uncomfortable at first, however, when you get past that, others will join in on what they learned and what they would do differently next time.

Agree @vasudha.prabhala occasionally what looks like strong commitment can turn into paying lip service

EMPS?

eNPS

employee Net Promoter Score

got it. thank you!

https://www.netpromotersystem.com/about/employee-nps/