that's vodafone's footprint generally. purple are partner markets. (mental note to add a legend in future)

and amazingly almost every one was hitting them. :thinking_face:

🙃

Yep, in every essence "it felt like that and continues to feel like that to date"

insanity is to keep doing the same thing over and over again and expecting different outcome... Except if you consider already delivering perfection, there's always room for improvement. 🙂

We tried lots of things before the OKRs, but only after we started to trial OKRs is when we started to feel cultural change we were after: • empowerment dial increasing • alignment to a common purpose improving • lowering the fear of failure, but learning to embrace it and continue to learn

Interesting. Did you start with OKR outside of any HR process (like "annual performance assessments"). Thought about submitting/incepting the idea to HR, but did not get that much echo

Perdoo company has a talk that specifically advocates against running OKRs via HR in any way. If I find it, I will share it with you. It's not recommended

Looks like this option may impose by itself as it didn't anyway found any echo in the void. Would be great for the paper Sabina! And thanks for the session!

Felipe Castro writes the following in his https://felipecastro.com/resource/The-Beginners-Guide-to-OKR.pdf

As soon as you involve HR, they are going to want to tie this somehow to end of year reviews and compensation, which can be counter productive. There is a lot of debate around this out there in the community.

Actually one of the point I wanted to raise to them is the need for decoupling bonus and these assessments as you end up having people only focused on the compensation question. And had a few other griefs against the system in place as well,

Many thanks for the link!

Objectives and Key Results.

It’s a method pioneered by Andy Grove at Intel. https://en.wikipedia.org/wiki/OKR

no, they work together

• KPIs are indicators how we are doing. • OKRs are our navigational system as we go towards our north star

You can see it now in the talk. KPIs and OKRs

I like to think of KPIs as 'health indicators' e.g. heartrate KRs are measures of progress, measures of movement, leading and lagging (e.g. I've run 5 miles of my 10 mile run aspirational outcome)

PI triggers me.

Increments are specific to Scrum (SAFe - just the implementation of SCRUM)

Blended I'd say. 🙂 We use some aspects that help bring structure where otherwise we're still a little nebulous.

Am curious how SAFe and empowered teams blend. What sort of team coaching approaches do you use?

Measure What Matters

https://felipecastro.com/en/okr/what-is-okr/

I'm doing a 15 min talk on OKRs on Thursday

yeah this is also really good

https://doeseurope2021.sched.com/event/jMFb/sooner-safer-happier-ama-with-jon-smart

Are OKRs part of Sooner Safer Happier?

oh, I think you mean this talk: https://doeseurope2021.sched.com/event/i547/ok-notok-okrs-3ms-mindset-mission-and-measurement

@logankd to answer your question, a focus on outcomes over output is one of the patterns in Sooner Safer Happier. OKRs are a great way to have a focus on outcomes (and the mindset that come with OKRs)

Dr. Mik Kersten (Project to Product, Flow Framework) is also talking about OKRs and metrics this week (Wednesday, 19th May @ 12:05pm BST) https://doeseurope2021.sched.com/event/jEMq/okrs-devops-from-micromanagement-misery-to-finding-flow?iframe=no

That's good, I've read his book and am still searching for ways to use the ideas (I'm a developer, but share a lot of ideas with the team I work with)

A place to start: https://www.whatmatters.com/faqs/okrs-objectives-key-results-explanation-examples/

hey @tim.bassett - yes, multiple techniques/tools. @robert.greville1 is giving a talk later today on exactly that!

Commonly we’ve been using parent child flags where one team would look after the parent and lower level teams would maintain their children. We’ve tried to ensure logical separation of flags between environments and teams. Hopefully more on this in our talk at 14:50 on #ask-the-speaker-track-4

Thanks - I had that down as one of the ones to go to 🙂 Also interesting in any branching strategy or do the flags do away for the need etc.... Look forward to the talk

Flags have enabled us to move fully to TBD (Trunk Based Development). Previous to that although most of our teams had moved too, some were cherry picking release branches. LaunchDarkly enabled us to move towards this and really move away from several branches, with varying levels of code in them, in various environments - it really cleaned it all up.

Great question! This is one of my favourites as it really gets to the heart of the 'alternative' leadership values needed. Will do my best to summarise!

Very intetested in this question, I find command and control to be so deeply embedded

As we went trialling, we were doing shaping them in Ben's Leadership team. Now, we are starting to work with the teams to shape them together, as we come out of the trials. More on this, at some point in the future

See https://devopsenterprise.slack.com/archives/C015DUDD9C5/p1621339961055600?thread_ts=1621339767.048400&cid=C015DUDD9C5. Does that help answer?

We are trying to cascade OKRs to the teams because we see different teams struggling with different obstacles

Happy to share thoughts on the learnings

Great question @siddharth.pareek - for these initial OKRs our real outcome was to achieve & demonstrate the business agility we're striving for. Our ability to deliver value faster has really seen a step change, which in turn has started to change behaviour around us (in order to better leverage that capability). Long way to go, but we're cooking!

Learn together through practice

This one was fundamental for us

I am the coach for the entire digital engineering, and our Scrum Masters act as coaches for our teams. I am not saying it is like that for the rest of the org, but it works for us.

What were you trying to achieve?

Hey @bfischer yes, sure. Might be worth a call later though. Lots involved in that one!

Main thing we're trying for is to share ownership, better leverage the scale of the team, and be able to work in a much more concurrent way, rather than always sequentially (and regularly forcing prioritisation decisions to be made)

So, did you measure the number of teams who had accepted pull requests from anyone outside of the team?

yes, exactly!

That might show health of inner souricing.

Ahh! Like it.

Thank you.

Inner sourcing OKR, when we commenced (which was last quarter) was the following: OKR: Every team must accept, approve and successfully deploy ONE PR into service they own from a different team. Overarching objective we are working to across our quarters: We will be inner source capable across all services.

Perfect. Great informaton!

For some companies this is really controversial. This is a nice experiment.

So grateful for that - every day 🙂

we used to use something called 15five

super helpful

thx

i have heard good things about another tool called https://www.xto10x.com/

havent tried it myself

Good question. @ben.connolly @sabina.kambersalamanc are you using a tool for OKR transparency?

No specific tool at the moment. We are looking at this for the future, when Vodafone is ready to scale on this. Right now, it is a combo of data from AzureDevops, packed up in end of Sprint Reports that our Agile Lead and SMs run every 2 weeks. The reports are powerpoint presentation style.

Everyone attends the end of sprint reviews, which shows transparently the OKRs status. It is a community of technology, agile reps, POs, PMs etc...

Mainly: Perdoo and Felipe Castro materials. Also, lots of others, but the above two were key for us to start with

Makes sense, thanks a lot!

Good points!

Thanks!

A FinOps SME could support how many dev teams (roughly)?

depends on the level of maturity and of the teams and what outcomes you're trying to achieve. Starting off with on part time SME to support under 6 teams is sufficient. As you start to move towards 12 - 15 teams having a person full time is useful; as the cost savings and benefits ends up paying for the role

thx! this is very useful!

It's adopted based on the maturity model by the FinOps foundation, which is open source 🙂

this one https://www.finops.org/ ?

that's the one.

thx

I guess the "Benefits Summary" part of the screenshot could help in that direction

we've focused our efforts more on outcomes as opposed to outputs. So we've not incl. timesheets or people time against it .. but I can see why this would be of interest to someone. Ironically, that's where integrating with something more value and flow focused like Tasktop would be a better way of looking at it 😉 .

ha yes thats a fair point! 🙂

My thinking is - including this financial traceability info automatically into the business results datasets that we work with... gives a nice realtime link.

They've loved it. They've felt part of the product development and decision making process as oppoposed to feeling like outsiders who don't have a role to play in the world of cloud. It's also help Finance teams realise how they need to change the processes, specially if Cloud is going to be the norm for their organisation. So all in all, it's one of the best things we could have done.

what was the starting point for the initiative? was it a crisis?

Less so of a crisis, more so a question being asked by Finance teams what role do we actually have to play in a cloud-first world

I like their attitude!

Learned a lot from the talk. Thank you very much, indeed, @deepak.ramchandani!

She did just mention “Chaos Engineering”. Don’t they specifically generate incidents on production to test resilience? 😁

At Vanguard, we don’t run our chaos tests that we expect to cause outages in production, for reasons that are probably obvious! We do however frequently run these kinds of tests in non-production, and have run tests with a limited blast radius off-hours in an isolated segment of our production environment (separate from where our client traffic is routed!)

how is the dev/prod parity in case of vanguard?

how reliable are these chaos tests?

@christina_yakomin Thanks for not running chaos tests in production 🙂

@simon.skelton and I will talk on Thursday about how John Lewis & Partners run Chaos Day tests in their one pre-prod env

Thanks for the clarification, @christina_yakomin. I hope it was clear my comment wasn’t entirely serious, but it’s certainly interesting to read how you have things set up at Vanguard. 😊

@kapoor.vaidik no non-prod environment will ever perfectly mimic production, so anyone who tells you they’ve achieved parity is lying (or confused)! But even though we know it’s not exact, we still get a lot of value out of our non-prod chaos experimentation. I talk a lot more about chaos testing at Vanguard in my presentation from SRECon last year, which you can find on youtube by searching my name and SRECon

We have a LOT of different applications at Vanguard that are in various stages of migration. Some have “fully migrated,” data and all, while others still leverage data on-premise. Our goal is to migrate almost all of the data securely to the public cloud (think ~90%)

Most of our apps running in the cloud at least perform reads against data in the cloud

Does that mean you're doing multi-writes, one to the cloud DB and one to the on-prem DB, for the same transaction?

Or doing writes to the on-prem DB and simply replicating to the cloud DB?

I assume its AWS Aurora or similar

I’ve heard from some people that they had rude surprises when they moved applications to the cloud but kept data on-prem: major networking costs and bottlenecks between the tiers. Any similar experience here?

@davidstanke532 We definitely co-located the data in the cloud primarily to address concerns about performance due to latency between the app and the data. Leveraging redundant AWS Direct Connect does help us a bit here. I’m no expert on all of our data architectures, but I will say from experience that hybrid data architectures (partly on-prem, partly cloud) should be temporary. Relying on replication long-term is going to bite you eventually!

Ah I see... very interesting, thanks Christina 👍

My favourite saying is "temporary is permanent", yes you have to be careful with temporary DB replication

@steve.smith exactly. I’ve seen replication far more than multi-write. Teams that have had the most success in migrating their apps to the cloud have already moved on from that replication stage, having built confidence in its functionality, and are running 100% cloud, app and data.

My experience has been multi-write was popular back in the day to replicate from one on-prem DB to another in code Now it's on-prem to cloud and AWS etc. have solutions we can just plug in 😓

+1 — IMHO, managed cloud SQL offerings are some of the most useful cloud services. The flexibility w/r/t backups, replication, scale up/down are amazing. Yes, it’s a bit scary to transplant the beating heart of your service to another host but I have found it to be super liberating. Throw away your backup tapes! 😄

Yes. Product teams are now on-call for their services. In the pre-microservices era, that prod support was much more centralized, and app teams rarely needed to support production. As part of the shift to microservices, the production support model shifted, too.

How did the teams cope with suddenly being on call? Did you do anything to ease them into that role? It's pretty scary to suddenly be on call for something..

It was a gradual shift. On-prem legacy: centralized prod support Initial microservices: shared ownership of prod support Cloud-native apps: app teams primarily own their own prod support

That's great to hear @christina_yakomin! So a similar journey to John Lewis & Partners

@simon.skelton and I are talking about it on Thurs, hopefully you can make it and let us know what you think

We still have central teams of experts to assist with prod support as needed, have invested a LOT into training, and now with our rollout of SRE roles, we are building a lot more operations expertise on our product teams

so you will have SRE roles in every product team?

That's the kind of approach at Equal Experts we recommend to our clients - build up the product teams, spin up enablement teams to assist teams at scale when necessary 👍

@christian.rudolph we will have SRE roles for every family of related products, and certain products requiring very high availability may have dedicated SREs as well. The introductory level of our internal SRE training program will be shared with ALL engineers who are part of a product’s on-call rotation, even if they don’t have an SRE job title

At Vanguard it varies by service. For some critical trading platforms, we’ll aim for 99.99% or 99.999% while our marketing sites may not have the same criticality - more like 99.9%

A very sensible approach 👍

Today, in most cases, it is the on-call engineer on the product team for the impacted product. SRE experts for each line of business and operations experts from centralized technology teams may be brought in to assist by our incident commanders as needed. We have an entire team of excellent incident commanders who ensure that communication is effective during an incident call

Got it. We do something similar.

We don’t have incident commanders though

Do alerts auto-escalate to “defined SREs” for areas of the business if not resolved within an SLA?

To SREs, no, but we do have auto-escalation in place if incidents are not acknowledged. Usually it is up to the discretion of the responding engineers and the incident commanders to escalate further and pull in additional engineering resources for troubleshooting, driven by the severity of the incident in most cases.

got it. thanks

This is true. Although deceptively so if I look at how many people get pinched by their Cloud Service Provider bill because they somehow missed bringing down some part of their infra. 😂

It's scary what you find lying around in some cloud accounts indeed 👀

It is very hard to centralize this and provide standards. It really comes down to knowing your clients’ expectations. Our technical teams partner closely with business product owners to set these thresholds and re-evaluate them quarterly. Many teams are working to improve their overall availability, so they’re making their SLOs stricter each quarter as reliability increases. Eventually we want to see teams operating at their target SLO and using their error budgets effectively.

Ah gotcha, that makes a lot of sense!

After doing my time building dynamic grafana dashboards I will certainly look into this, thanks!

We are happy to take contributions

the idea is to have best dashboard design practices built in the code

We have internal “blogs” that I’ve leveraged in the past to share incident review write-ups. These allow likes and comments to encourage discussion and questioning. We recently added the ability to track “views” on these pages, as well. In my experience, my post-incident reviews are always my most-viewed and most-liked blog posts. Seems like the engineers at Vanguard are more interested in how I broke and later fixed something than they are in my new feature releases.😂

I agree, there's always lots to learn from things we break, and it's usually a very interesting set of circumstances that led to it. I guess we all just like detective and whodunit stories!! 🔎

@ben.conrad @mhyatt: Does HMRC still have an internal blog for sharing post-incident reviews? I remember something similar in 2017. Thanks

@christina_yakomin I'd definitely recommend tracking view counts on post-incident reviews. Out of context, it's a vanity metric. In context, it's a cheap, effective proxy of organisational learning Until John Allspaw thinks of something better, that is

100% agreed. I was very excited about the recent addition of this simple feature.

At HMRC we publish all our PIRs and have a slack channel where we post links to them as well.

Great question. Most of our platforms have an engineering team dedicated to building and supporting them. That team would be on-call for the platform. In the event that a platform outage causes application outages, the incident calls can get very large very quickly. The incident commanders help platform engineers with internal stakeholder communication while they troubleshoot and triage.

Our non-prod and prod environments are very segregated, but the same product teams troubleshoot both. We have lots of controls around production changes, many of them automated, including separation of duties (person who wrote the code/config change can’t deploy it). In a pinch, there is a dedicated team with more privileged access - though they still are subject to many controls - to handle manual changes to remediate incidents.

sounds familiar. many coaches say "1" when I ask this question. The best coaches usually say "1"

at least for tech coaching.