For me, I think we have to care and we have to demonstrate that too. We can't make others care, hence we can't make them be responsible. they have to do that for themselves. Everything we do should be a pull system anyway. Just my 2cents.

Asking questions and thinking out loud.

Leading by example - I share my failures and learning too in as public of a way as I can. I also think words matter. I say "learn fast" vs. "fail fast" and have shifted the organization away from using post mortem to using learning reviews. It takes time to build that trust. I also like to talk about "honoring reality" and try to encourage senior leaders to honor reality vs. judging it.

By flipping the meaning of accountability to mean “able to give a detailed account of their perspective in/around an event, without fear of reprisal” rather than “falling on a sword”, it places the focus on how rich and detailed the account can be. Ironically, this can fulfill the desire you mention, which is the commitment to stakeholders. Public “lashings” don’t demonstrate commitment to genuinely learn from an event, it only implies that the org/leadership believe that all mistakes come from individual motivation, and that shame is an effective deterrent to making future mistakes.

@ckissler "have shifted the organization away from using post mortem to using learning reviews". I like that. Learning Reviews. I will be stealing that! 🙂

CS Lewis!! Thank you, Jeffrey! 🙂

I love CS Lewis

So delighted you’re here, Joey! Looking forward to your talk later shortly!

Me too!

@genek I'm delighted that you're here 😄

I was thinking this morning that I need to reply to the tweet asking about favorite Idealcast episode with Westrum as my choice.

It's so rich with insight, from both guest and host.

Yes i listened to the Idealcasts @ronwestrum did with @genek many times, pausing frequently to try and think over what I’d heard

I forgot to thank @sophie.weston129 on Twitter, which is what prompted it.

I too find myself getting new insight from talks on 3rd or 4th hearing 👍

That’s a longer conversation — but happy to share!!

Worst case, if we don’t get to talk this week, email me and let’s talk! (Maybe during one of the networking sessions?)

Yes please! I’m sure others would benefit as well! I’ll follow up with you

@nickeggleston - I’ve also found that, people tend to find Gene…be it a a city park, grocery store, remote beach, crowded bar. 😆

So a quick trip to the store can end up taking 5 hours…

Wanna meetup in Gather during networking session this morning? I can share some of the stories, but MVK is right — it’s about 50/50 me finding them, and them finding me. After Phoenix Project came out, it became much easier, if you can imagine. Hope you’re having a great week here!

Go here to see: https://doesvirtual.itrevolution.com/video-library-upgrade

thanks @scott.prugh!

This really speaks to me. Customers I serve at my org work in Nuclear Power Plants!

How and how often do you (suncor) measure psychological safety?

Good point - I've heard this expressed as the difference between a check off list (going through the motions) vs. a list of things to check (applying thinking and care)

We recently went through this as well. Be careful of the pendulum swinging too far the other way. Now we're having conversations about, "Why are we building this now commodity software instead of buy it and invest our people in more differentiating software?"

Completely agree with you there! I have worked in places that were too far down the buy route. We constantly weigh things as we discuss how to bush more build mindset to make sure they're things that differentiate us!

and of course, if we build, we'd ideally like to open source and give back to the community

Now we're trying to talk more about where our systems fall in Geoffrey's Moore's Core/Context mode: http://strategictoolkits.com/strategic-concepts/core-and-context/

Thanks for the link - i'll take a look through!

To my knowledge, all talks will be published.

Loved the @jason.cox talks that could not be published! So happy I was there to hear about the "Grumpy" server and how "Darth Vader" is a Pathological Leader!!! lol!

@mring we’ll be publishing this talk after the AM plenary block is over. I have my finger ready on the publish button.

And yes, to my knowledge, all talks will be published.

Is it really a small vs large or a single team vs a collection of teams. Seems like if the team is too large, you start losing efficiency

I think it speaks to where there is spare capacity.. Larger teams seem to have more gray area and you can start with adding smaller items easier, but small teams you may have just look at starting a new team that can overlay on an existing..

Thanks!

We will be publishing a podcast with @jpetoff later on this month in case you're interested

@kboth_does can't wait to hear how the podcast turned out!

We did – running shockingly ahead this morning. so sorry!!

I’m so sorry! Not sure — @annp would know. But all good, we’re all so happy that you’re here! (@annp, can you send the link to the complete video? 🙏 )

All good. Just wanted to make sure I wasn't late accidentally 🙂

I completely agree! physics-chemistry double major myself. went from an interest in computational physics/chemistry into software.

Very nice! I did synthetic chemistry myself so software is a bit of a departure, but I love it!

Yes. and also to ensure that we aren't taken for granted and only working on the highest value added things.

Indeed. For SRE supported services, Devs may also share a portion of the oncall load and many teams that don't have SRE support handle their own oncall.

Is the team that owns the service the first one that gets alerts for it, or are first level alerts sent to a different team?

Yes, the team that owns the service typically gets paged. There is no general triage queue.

Paging @cleng to weigh in on how this works in practice.

I posted my response in the main chat earlier, but reposting here to make it easier to find: The budget for SRE headcount comes from the Dev org. They "pay" SRE via HC, but once transferred, SRE is in complete control of the HC (until the engagement is ended by either side).

So glad you’re here, and thank you for presenting!

That sounds like a 'throw it over the fence' mindset to me which I'd consider an antipattern. @cleng anything to add?

Yes, definitely an antipattern. What behaviour do you see at Google? Is there anything intentional which prevents that antipattern?

I think there is an educational and communication component to it. SRE leadership and engineers on the ground communicating about the key SRE principles and best practices and how teams should work together for maximum impact. This is something that @cleng is actively working on. My team is also working to share reliability best practices more broadly. Reliability needs to be everyone's concern, not just SRE.

Thanks @jpetoff

There are a number of remedies to this challenge: • Write down who's responsible for what and who has authority over what. If SRE can block bad launches, Dev will try to work something out (However, the job of SRE is NOT to block, it's to advise!) • Work together. Have Devs work a little bit on infrastructure and ops and have SREs work a little bit on product - not too much, because it waters down the role specialization, but enough to maintain a mutual understanding. • Establish ultimate accountability for the product's reliability with the product team. SRE's job is to help them achieve that, but at the end of the day, they remain accountable. • If the situation spirals out of control, declare a "production freeze" (only stability fixes get deployed) and/or "code yellow" (reliability work trumps all other project work until the exit criteria are met). You need support from senior Dev leadership for either. If you can't get support from anyone in the reporting chain on the Dev side, you should find a better Dev org to work with.

Thanks @cleng. "Establish ultimate accountability for the product's reliability with the product team. SRE's job is to help them achieve that, but at the end of the day, they remain accountable.", this resonates for me as to how to avoid the human tendency to think that it's someone else's problem.

This sounds very much like when you bring a manufacturing engineering into a physical production process in order to address quality or throughput issues. My Dad did this at Pratt and Whitney to go in and help other teams or contractors figure out how to build a part correctly when they were having trouble. And they were an expensive resource to bring in.

SRE support is typically limited to the most mission critical services. It's part of the cost benefit equation.

Love that "Mission Critical Services" - most IT systems owners think that all their services are "Critical" 🙂 in a typical Enterprise (the talk about availability and reliability) but not about the cost to achieve that

Thanks for asking! I lead the learning and development function for SRE. My team is responsible for onboarding, getting folks ready to go oncall and ongoing education opportunities. We also bring reliability-focused education to all of engineering. Reliability needs to be a priority for everyone, not just SRE. This may sound fluffy, but we also foster a strong oral tradition and passdown of the SRE organizational culture through storytelling in our classes.

For the knowable, or meta level patterns in context, I would suggest 🙂 A headwind for the unknowable (treating the unknowable as if it's one size fits all)

• Sharing ops work to some extent • Escalating during incidents, debugging togehter

• reviewing postmortems together • co-design sessions • ops/architecture training sessions for Dev

We have the SRE team lead the BPM (Blameless Post Mortems) they are in a great place to look at issues deeper and can then help those involved get to the root cause easier and see the action going as far as possible with in Development or in better Operations.. I know I am not at Google but thought I would chime in... hope that is okay.

Thanks for chiming in @mr.denver.martin always interested in how others do it.

@mr.denver.martin That's actually very similar to how many Google teams organize postmortem reviews.

I think @cleng is best placed to tackle this one.

It's complicated. The preferred strategy is to make the tools and strategies you want the teams to use the most attractive ones (incentives). They can be most accessible, easy-to-use, best supported, most feature complete. Don't try to force a solution on your engineers that isn't working for them. They'll find ways around it. The second approach is on the relationship level. Just because a Dev team doesn't have (full) SRE support, doesn't mean that they're not exposed to SRE. There can be consulting, SRE Love, training programs, tech talks, etc. When SRE has a reputation to be helpful, the devs will ask you for advice and follow your best practices. SRE is production evangelism. The third approach is "the stick": Policies about what you can/cannot use, automated compliance measurement/enforcement, nice-or-naughty dashboards for senior leadership, etc. I would generally advise against these, but there are corner cases when they are necessary. Typically, when you can convince 80-90% of the org with the other strategies, it can be sufficient to show that number to non-compliant team (and/or their bosses). Then again, not everything always needs to be uniform. If you don't have to deal with it, let them do what they feel is right. If you always follow the one-size-fits-all approach, you stifle innovation. Listen to why they chose a different path. Maybe they have good reasons.

Ha! working to get through the Qs as quickly as possible. Also listening to the talk at the same time which is distracting me. I'm terrible at multi-tasking 🙂

I could imagine.

Service maturity: • SLO quality (user-oriented) + compliance • Ops workload (tickets + incidents) • Data integrity processes • Capacity planning / efficiency • postmortem processes and hygiene • Release automation Team maturity: • OKR planning processes • Staffing/attrition • ops workload • SRE/Dev relationship

thanks @cleng! really insightful, love how staffing attrition is a factor in team maturity, often overlooked but it's a key leading indicator to predict the quality that a team might deliver

Unfortunately attrition can be a trailing indicator, because it takes time to build up enough frustration that the engineers actually leave. Also, be careful to measure this only for organizations big enough that you get a meaningful signal. When you look at a 5 people team and 2 of them leave for reasons completely unrelated to the team health, you get a huge spike. A noisy signal is not useful.

I think a big difference is the focus on automating away toil so that the team can scale sublinearly to the size of the service rather than throwing more bodies at the problem.

Think of it as SREs provide operational (production) expertise. They’ll help fix an operational issue, but the goal is to understand the issue such that they can engineer away the problem from happening again.

Or the actual expert answered 😄

i.e. writing better fire fighting equipment, so that don't need to spend time fire fighting?

Hi @jchanto17 I see SRE as the bridge between Ops and Dev, these are people that are pulled out of or not part of Dev or Ops, but they have skills and knowledge to be able to get to root cause and then can look at 1. fix temporary with work arounds 2. figure out how to respond and restore faster 3. fix long term how to keep it from happening again.

well stated @scott.jaffa!

Might an SRE also look for risks before they become issues? Is that inherent to eliminating toil?

Yes @nuwayser, SREs are empowered to make tomorrow better than today by focusing on the things that will have the most impact on improving reliability of the services they support.

SREs make more sense in places where dev teams actually own dev & ops - more common when your apps are cloud-native.

in traditional datacenter based apps, SREs may play the role of a bridge between dev & ops

i'm involved in some research projects on whether SREs fit into the hybrid world of cloud-native + legacy datacenter apps. It's been interesting and challenging

Ok thanks all for the comments, now it makes sense to me. And what I think is that I can take this type of firefighting work out of the Dev team, because in my case Dev team use to tackle this type of problems and slowdown the value delivery

> Yes @nuwayser, SREs are empowered to make tomorrow better than today by focusing on the things that will have the most impact on improving reliability of the services they support. Thank you @jpetoff :) @jchanto17 to your question - as someone who has worked in traditional ops before, an SRE is someone with my mindset + developer skillset, and who is also empowered / expected / measured on their ability to • participate in incident response to close issues • proactively mitigate risks • leverage error budget to find problems make the service more resilient By developer skillset I mean more than python/bash/perl scripting: it's someone who has the skills and license to get into the app code and improve it proactively. @jonathansmart1 to your earlier point re: measurement alignment between SREs and Product owners - curious to know if you think the two roles are responsible for the same outcomes or different ones.

Hi @nuwayser, not sure, keen to understand the case study point of view (SREs and Product Teams, not only POs). I guess that it will depend on org by org, as to how that is done. With more aligned incentives or less aligned incentives. Keen to understand and how risk is mitigated.

Typical examples: • Helping a Dev team to set up their monitoring/alerting • Analyzing SLOs / refining them • Architecture review • Picking the right tools / infrastructure for a new "greenfield" service • A migration to new infrastructure (e.g. database) • Improving ops processes

Core aspects of SRE performance are impact (measurable improvements for users/Dev/SRE/budget) and simplicity (standardization, deprecations, cleaner architectures, less dependencies). Heroics can be rewarded temporarily but are a generally not going to get you a promo. Specifically, the focus for performance evaluation is on designs and landing engineering projects not "keeping the lights on".

Service Level Objectives are nominally the tool that allow SREs and Devs (and other business stakeholders) to speak the same language and align on incentives

Thanks @cleng and @jpetoff

I like "simplicity"

Response time would typically depend on the criticality of the service.

So are the dev teams are on-call?

"You build it, you run it." 😉

I'm pretty sure that's how it is, but I was hoping for confirmation. There's folks in our org who don't agree with that.

Yes, Dev teams are oncall, some of them 24x7 when they don't have multiple sites across the globe. That's not what we would do for business-critical services though. Being paged at 3am when the rest of your team is soundly sleeping is not a recipe for success. However, SRE escalation support via baseline can be helpful in these scenarios.

Thank you for the response! What would the size of those teams be? How many people around the globe? We have some teams of 5-7 that are responsible for critical services and currently hand off to operations overnight because they don't want to be woken at 3am, but the operations team is becoming overloaded.

For an SRE team, we recommend at least 6 engineers in each of the two sites. That's a significant investment, so each SRE team is typically working on a large set of services (or a few very big/critical ones). Dev being oncall during business hours and SRE being oncall during off-hours is also a model that is working well for Google SRE teams that are using it. It increases the coordination overhead a bit (more cross-team handoffs), but makes dealing with faulty releases easier (releases should be done when the devs are oncall - they're more familiar with the changes). It also exposes Dev to production, but with a safety net - there's always an SRE familiar with the service you can escalate to.

https://sre.google/workbook/table-of-contents/

@nepobunceno There are lots of resources to check out at sre.google including our various books. There are some good large system desigjn exercises and shorter form articles too.

There are great books by Google (Free) and also there are number of courses in LinkedIn and Youtube available. The one thing I will recommend strongly the micro learning videos on this subject by Google Seth Vargo and our own @lizf

@jpetoff Where can I find the system design exercises? I'm scrolling through the site and recognize books and articles I've grown from and helped my org change processes to reflect.

@dacahill7 check out http://sre.google/classroom

Google is an enterprise with its own challenges. With 140k employees and >20 years of history there are many complications on the ground. Individual SRE PAs find their own solutions on how to adapt to their space. Please don't take Google SRE as a blueprint to be applied verbatim to your org. It's a case study. We keep evolving the way we do things. Because we have to adapt all the time.

^^^^ THIS - Love this @cleng (the same applies to all the models right - Spotify, Etsy, Google, Amazon)

I definitely don't want to become a preacher for a cargo cult. 😉

SRE's performance is evaluated by SREs (up to a certain level of seniority), but SRE managers and promo committees expect positive peer feedback from Dev peers as critical support for high ratings.

So. True.

@genek "Some of the best firefighters can think like arsons."

😆 😆 Like a connoisseur of disasters, if you will. 🙂

Thank you! PSA that all the books are available online for free at sre.google/books

They also have .prod 😄

Try docs.new 😉

When can I get “Nick.”? :rolling_on_the_floor_laughing:

Thanks so much for inviting. us, @genek. Will continue to work our way through the backlog of Qs!

It's been an honor and a pleasure. Fantastic community at DOES! Great questions, many things to learn from others too!

https://sre.google

thanks i also googled it

Nice...I see what you did there 🙂

@eleonravinez @jeff.gallimore There is a lot of unpack there. I think the best way to sum it up is through this conference talk that I gave at FailoverConf last year: "Swim Don't Sink: Why Training Matters to an SRE Practice" https://www.youtube.com/watch?v=8iaNMMwozCc

@vmshook Many of the foundational principles of SRE can be applied no matter what your size (e.g., SLOs and error budgets and a 'vanquish toil' mindset)

Another good one for @cleng to tackle!

When you have been working closely with the system and know all the hidden timebombs: An incremental approach is generally preferred, because a clean-slate rewrite is costly and high risk. However, sometimes you have explored all incremental options and they either don't get you to an acceptable level or are even costlier/riskier than a rewrite. Then it's time to sharpen your design pencils. When you're new to the system: A thorough architecture review and a regular ops review to assess the long- and short-term viability of the system should get you close to the above.

Here you go. Also lots of good material at https://sre.google

Thank you! Great talk.

@christina.biangslev This is not something that has been considered at Google AFAIK. Keeping SRE and Dev reporting lines ensures that reliability is a first class feature. If everyone reported up to Dev, it's possible that the orgs leaders might be tempted to trade off reliability for feature velocity or to take on more tech debt than is wise. I know other companies have approached SRE differently though and have found success with a more embedded or consulting approach.

I agree. What I see often is compromising quality because of the pull from business to deliver new features. I think this would also be something that would make it a second class citizen.

See https://youtu.be/n4Wf14e2jxQ?t=497 for a good argument. Summary by Ben Sloss: "So for that reason SRE has to be its own team. It's my basic thesis. If you don't have a team who views their mission in life as making sure that the product works, you will ignore availability and reliability until you're in real trouble."

I completely acknowledge this risk, that Feature Fetisch might taking over the SRE agenda. I'm struggling to fit the ideal of the truly independent product team with the experience of our SRE pioneers. I'm sure there's a place for a variety of setups; I'm wondering what the deciding factors would be to consider the truly independent model.

There's a whole website for the many organizational options you have: https://web.devopstopologies.com/

Maybe do DM to the presenter and maybe they would still be watching the Slack Workspace... just a thought.. but then you miss on others that may have knowledge besides the speaker... hmm.

#ask-the-speaker-more

It’s tough to know. I love staying on one channel throughout the day for the lively engagement and community, but it would be super cool if the thread discussion for a given talk were moved to a dedicated channel for continuing discussion… @genek

timely after facebook’s experience this week…

Actually, its always DNS.

That was my first thought!

@jtf for sure!

It is!!!!! Incorporating the development/automation has been key to getting out of the firedrill mode, and bringing sanity to our pace of delivery

This is awesome. Parallels to some other stories:

@scott.prugh may I ask what that’s from?

https://www.youtube.com/watch?v=6afD-sQm03E

align it to business outcomes and the architecture gets fixed.

(More difficult to rearchitect your core switches, which are global in nature, than most software. 🙂

but sometimes it doesnt. sometimes figuring out the right archtiecture is also really hard

@cncook001 We should call it The Law of One Foot

Team Topologies in action

The org itself is IT focused... But the org also has non-IT staff members that are integral to our success.

Yet another reason why open floor plans can kill productivity. At my previous studio job, somebody created paper indicators that you cut out to put on your monitor: • Green: I can be interrupted • Yellow: I can be interrupted but we can't talk about your cat • Red: DND

Is that 14K networking devices, @denee.ferguson?

yes

I had the same, started working in the cafeteria.

Most of the "Productive" Engineers prefer to work from home (as they do not need to see these managers and other distractions)

You would think remote working would provide some speed bumps for those drive bys but not always the case. Where there is a will there is a way!

mine is back in force w/no telework options and a move back to an open office. less time to think

I wondered that yesterday about your workforce.

some jobs have that option. my last one did. my current does not.

it's definitely something IC/DoD is needing to rethink

The biggest lesson is the product managers NEED to understand the technology...

That can be a purple squirrel quest...

Completely agree with @denee.ferguson

This leads to me to question - The PdM's do they need to be technical or Business 🙂 or both?

I ❤️this thread and it's barely gotten started.

They are both in our org or at least that is the desired state

@lbmkrishna both

At my previous job we had motion in both directions (engineering->product crafts and vice versa)

My attempts at product teams with Infra were underwhelming at first, but I found several months later they came back to me with an appreciation for what they learned. It just took longer to click

I think what we found is that engineering->product manager was better than the other way around

@char re the purple squirrel quest, one of my challenges is to "train" the infrastructure "POs" at the moment. Im just getting started.

common theme: I was a developer and kept building features that nobody used or wanted. I became a PM to prevent that from happening

In some org (Enterprises) - The Product Management is (abused). The PdM's act as Service Delivery Managers (approving people work)

Has anyone tried a "two in a box" strategy for platform teams?

@charles.page like PM+Engineering Lead?

@chris.gallivan421 Id love to hear more about your successes with infra POs. what did they come back to you for?

Can you expound on that?

@char,two in a box??

@gerijotoole we did a dojo with them (Joel Tosi and I)

They said they liked mobbing, rather than working in individual silos

hey @ckissler this needs to be a Lean Coffee topic

Many product teams are matrices with consensus leadership shared by two "pyramid" reps

they also said, they felt like a real team for the first and only time in the dojo

eg a product "pyramid" and an engineering "pyramid"

ah.. I have experience with those and have even talked about the quick learnings, but at this time, they are barely "walking," not to mention leadership buy in

@chris.gallivan421 those comments have been ones ive witnessed as well. it's a beatiful thing, isn't it?

engineers don’t easily skill into product manager roles < I’m wondering what specific problems you’ve seen with people who wanted to make the transition.

at Stellantis we favored the 2 in a box, but it was mostly because we needed a place to put the managers 🙂

@jtf same. I am just getting started, week 3 at a new org, with this setup

@jtf These are field reports I am hearing. Just went through this in detail yesterday with a large Australian investment brokerage.

@bryan.finster486 has some good takes on 2 in a box

on app side this led to typing pools rather than product teams

@bryan.finster486 has some good takes on lots of things. thanks for the tip

@jtf have heard similar at least 3 or 4 other times.

I mean, I have OPINIONS.

I’d cross check them if I were you.

so much so, we have renamed TPO from Technical Product owner to Typing Pool Owner

I love that.

Opinions are free, "good takes" are a fee

Nah, I open source anything I can.

@jtf here is a quote from JP Morgan Chase

@charlie TRUTH!

@char: I complete agree it happens. I think it can be a problem to move from engineer --> product manager for many different reasons. I’ve also seen problems for people moving from business analyst --> product manager or scrum manager --> product manager. I’m not clear there’s anything uniquely challenging about engineer --> product manager. When I’ve seen problems in transition the biggest issue is the lack of mentoring in the transition and/or a weak (or unhealthy) product management culture in the organization. Might that apply in these cases?

One of the issues I have seen with engineering moving to product is that they sometimes can have a hard time not solutioning.

A lot of the people I have seen in these roles are long term employees who used to code. Over the years they have built up a lot of domain knowledge, more so than on the business side

Is there any effect on how the org is structured? Ie. is someone moving from engineering to product because that’s where growth is, vs. because they specifically want to be in product? Curious if organizational dynamics are a major driver of seeing that transition succeed.

That was in an org that did a lot of outsourcing

That failure mode for engineers makes sense to me @char. One thought that came to mind for me is that we try to have our engineers oriented to client needs in any case; the product owner acts as a customer proxy and tries to talk in client language to the teams (and sometimes the engineers join client meetings). I suspect that might make an easier transition compared to an environment where the product owner acts as a translation layer and client language doesn’t make it into the engineering team.

@scott.jaffa so far we have not had any engineers shift over to product management...

I plan to use customer empathy to begin to help these folks, since who we serve work daily in an environment that demands extreme safety. I just got here 3 weeks go, so thinking to start there.

@jtf Having the engineers start paying attention to the user experience that their products provided has been key... having the former engineering team members be on call also helped them get religion (quickly!) over the importance of details

@jtf These are all good hypotheses, reflective of growth mindset. I would also, as a counterbalance, inquire as to whether there is fixed mindset operating in a way that's not easily disrupted, even with the best training available.

@denee.ferguson This is exactly what an SRE at a FAANG told me yesterday.

When IBM Marketplace start 5+ years ago, we adopted "3 in a box" model. Development Manager, Product Owner and Technical lead. We called it "intentional tension in the system". Those 3 had to agree on what to work on next. It worked really well.

@char my group may benefit from the shift of fixed to growth, simply by nature of our business where we serve teams who's lives can be in jeopardy if we don't shift. thanks for the reminder of fixed v growth. that's a great way to bring in customer empathy with them

@jtf I do believe that one of the major constraints in the whole transition to product-centric operating model is workforce lack of product managers. I mean, there is not even a well accepted educational/training pipeline. (Not to dismiss the efforts and valuable offerings of the boutique sector. But we need more scale IMO.)

How can the engineers become product experts? I was in distribution centers as often as possible.

I’ve been sending all the engineers to the customer product demos, user trainings, etc. Too new to know if it is having success, but that team did just get an amazing kudos from one of the users in a training about the application quality…

@bryan.finster486 We haven't tried cracking that nut yet... the people leader for the engineering teams is the product owner... so we focus heavily on enhancing the user centricity of the product owner. The engineers get there kinda by osmosis.... We do have tight relationships with other internal teams that help us understand specific use cases among our customer base (e.g., call center agents, traders, executives, etc.)

@char: reminds me of the early days in agile, where lack of people with experience in agile was a bottleneck to widespread adoption. There was much more demand in 2005 than there were people to satisfy it. It seems something similar is happening now with product management. Everyone can see this is a better way to operate, but precisely because it is new there’s a dearth of experienced practitioners. (Mind the Product is great, but only gets you so far.)

@denee.ferguson Are engineers reporting to the product managers in terms of career pathing etc?

@char Engineers report to product owner; product owners report to me. Product managers report into our Agile org, led by @jennifer.miles While product management could be a potential path for the engineers, I'm not seeing much interest from the engineers in making that move. Most want to remain deeply technical, while a few have aspirations to become product owners/people leaders.

Interesting take. Question - then the PdM and PO spend more time on people management activities than the "Product" management? cc - @denee.ferguson

While they may not be plentiful, I have met some amazing developers who are passionate about customers, products and clean code

@lbmkrishna product managers are not necessarily people leaders. Product owners are. We have found that limiting number of direct reports a product owner has is key. For one of my teams, there are 8 engineers (soon to be more as we onboard new hires). One of the engineers that wanted to become a people leader will take over product owners responsibilities for one of the products this team owns, and will become the manager of 2-3 engineers on the team. This will reduce the direct report load on the current product owner for that team. Generally, 6-7 is the max number of direct reports a product owner should have in our org.

Thank you @denee.ferguson - Thank you for providing the context, much appreciated.

Load balancers definitely make technical implementations simpler!!! Now if I could just get MACs to roam better!!!

Ha!! When I learned what load balancers did for devs, my jaw dropped in awe. I had no idea!!!

This can lead to teams not reporting incidents.

@cncook001 not possible in our world

It depends on the culture. Needs to be a constant reminder for transparency, vulnerability and blamelessness

I disagree, I think it is a good metric… but only if you understand that good metrics allow you to ask questions. They can’t give you answers.

Is there a metric that can't be used for bad reasons? good to be aware of the risks and build the right conversations around them.

We have blameless postmortems as part of our overall culture

A metric by itself can definitely be abused. That is why it is important to identify multiple metrics and track them in tandem to be used as guard rails.

Is there a metric that can’t be used for bad reasons? < I have a friend who says “if it can’t be used for evil it isn’t a superpower.”

Yes, it does depend on culture.

outcome based - oriented around the product evolution and customer experience

👍@girija.rao

Girija - what are a couple of examples of team focuses?

@girija.rao - So many questions. Would that mean you had a branch focus team that would include WAN design as well? Would FW access requests be part of datacenter connectivity?

@brianspo for e.g. improving the wifi experience, understanding usage painpoints and wish lists through empath interviews and surveys, and determining technology enhancements to drive those outcomes

@nkampwerth we have a core security team and perimeter security team and each one focuses on the related use cases

This is team driven

Would that invalidate how the velocity metric rolls up?

@jennifer.miles

no, teams use similar methods to story size

the real benefit we found is measuring at the team level but we are able to roll up to see trends

What is the purpose of your story pointing then? I've always stuck to the matra that story points mean absolutely nothing outside of a product team. Since the numbers are all made up and unique to the team, I use it to ensure consistency and predictability of the teams and use other, more real, metrics to see how teams are performing over time

@john.rowe It helps the team understand how much work they can take on each sprint... Each team knows how many story points they have been able to successfully deliver on average during a sprint.... We do include stories for things like training, vacation, PI Planning meetings etc so that our process is consistent from sprint to sprint. As I understand it (@jennifer.miles is the agile guru), story points isn't a metric that should be compared across teams.

so your days without incident metric in Q4 is not used? or do you track trends year on year?

@marc.price the recording was made in mid-September, so we didn't have full Q3 stats at that point, much less Q4. We do track trends year on year

Yeah scrum was never intended to be used in this way. We see parts of Scrum as a “starting point” less a “end state”

Yeah, Kanban and flow at the speed of the work for infra / hardware. Physical constraints are better handled via kanban and blocker states and you move on when you can.

Regarding metrics for our teams, we paid less attention to velocity and more on lean metrics like Cumulative Flow Diagram (CFD), WIP and throughput. Flow Metrics (from @mik) would work here too. We also spent time in value stream mapping to identify quality issues (low % C/A) and delays (in scheduling or hand-offs). These were where we got the most "bang for buck" with our infra teams.

Scrum always raises a red flag for me with anything operational as it's not suited to interrupt-driven work.

@char That was part of the square peg, round hole challenge for us... honestly I have found that using scrum ended up improving delivery (quantity, quality... still working on predictability). But we had to teach the engineers how to chunk up their work into bite-size pieces.... It used to be that they might have a single story that would hang out on their kanban board for months (something like "deploy X")... and there was no visibility into how that effort was going, where the team was getting stuck... We have found 2 week sprints to be best fit for my team... The 2 week interval allows us to incorporate unplanned work without disrupting the current sprint, but keep whoever is escalating the need to do X happy.

@denee.ferguson It sounds like you have some flexibility then with the Scrum masters accepting emergent tasks that aren't specified at the start of the sprint - some purists (not me)might raise an eyebrow :face_with_rolling_eyes:

@char yes. As much as I would like to say we will not take on stories mid-sprint, that's not our business reality. My team in particular isn't fully staffed (we're hiring!); hoping that as reqs are filled this will be less of a problem because we will be able to accomplish more each sprint with more engineers.

I guess it can cause more incidents if you try to change too frequently. faster doesn't always mean better

thanks

Smaller batches that reduce the blast area are key. In our loadbalancer space we reduced failure group size first by adding more load balancers enabled via automation.

So then the failure of any change affected a much smaller area.

This also greatly helps scheduling dependencies since dependencies increase risk exponentially.

Depends on how you define an incident. This is where the A/B and canary deployments can help greatly. Have lately been converted to feature flagging as a way to rapidly fix deployed incidents, as long as you minimize embedding

@tiny.mpetersii Agreed. Feature flags are a great technique to 1) reduce blast area, 2) reduce dependencies and risk, 3) inject operational thinking upstream to dev(shift left on ops concerns)

@brian.m.smith There's a balance... The smaller the number of devices in a window, the more windows you need. Since our windows are normally at night, wear and tear on the team is a consideration... So... I personally strive to do enough changes (pilots) that we have the kinks worked out... then ramp up the devices per window. Some of our technologies (e.g., SDWAN, Wireless) it is generally a big bang deployment... There are some ways of limiting the blast radius, but you can only do that so much. In those situations thorough lab testing and bug scrubs are key... Unfortunately, vendors tend to have a lot of bugs.

as in, Facebook's BGP issue this week...

totally.

It’s amazing how one “little” routing change or firewall statement can have unintended and hard-to-backtrace consequences.

yup....... not the first time I've seen BGP route change have significant impact. fortunately not caused by my teams 😉

Dynamic routing is a double edged sword. Once a dynamic routing change propagates, you can't just reverse it as quickly as a "no" statement.

Thank you Denee.

did you try to get the pulse surveys at the end of every PI? just to get the business and technology to provide feedback and maybe help to drive NPS in a desired way?

@andrew.machen continued exposure to the benefits of the model, basically a wear them down approach. We have one team who is still resistant but have adopted some of the processes. It is an iterative process with them.

Our experience is similar. Certainly great metrics and more energy from engineers in making things better as the months and years roll by. Great talk from you and your team!

great session, thanks for sharing!

Very interesting! Exactly where I am focusing current research.

@char happy to answer questions via linkedin

@humayoun.khan we still use SAFe although in the Network space we have adopted a hybrid approach at this point, using a product based teams in a SAFe like environment.

Had a chat with Discover the other day, their fraud prevention folks didn't know about some of the changes to improve the system, like apple pay. So the multiple small charges in a day were getting flagged as fraud, and turned down, when marketing was advocating make those small charges for extra points

understood, too much Air Force time...

How did you converge upon 6 weeks?

Everything takes six weeks

Nothing takes 6 weeks. 🙂 😆

We worked with the authors of Creating Your Dojo: Upskill Your Organization for Digital Evolution - Joel Tosi and Dion Stewart who started Dojos at Target. They recommended 6 weeks,

Over time we saw that it took about 6 weeks for learning to stick

assuming at least 30 hours together a week

we experimented with other time frames

based on a lot of data across numerous teams

sorry for formatting.

let me edit, I'm new to this slack thing

I’d like to learn more about the concept of a Dojo, is that book the place to start?

week 1 - A man, a plan, a canal; Panama During week 1 you make and begin executing a plan Week 2 - No plan survives contact with the enemy If you are tracking to a two week sprint, you realize you didn't plan well enough, and your plan isn't executable. Week 3 - A plan for a real plan Take your lumps from failing during the first two weeks, Week 4 - Get the work done This is the week where the real work happens, because the team finally knows and understands which direction its going Week 5 - If you find yourself going through hell, keep going Week 6 - Celebrate

Same approach at Principal Financial Group (prior company). Six week dojo challenges. Influenced by talking with folks at Target and the Dojo Consortium.

@genek - everything* takes six weeks, you just need to scope to ensure that

it doesnt start to hockey stick until week 4 or 5

at least that is what i observed

week 2 or 3 is often despair

@chris.gallivan421 I've boiled it down to: week 1 - let's do this (excitement) week 2 - oh this is bad (overcommitment) week 3 - oh this is really bad (we have no idea what is going on week 4 - TADA! week 5 - I knew we were fine! week 6 - wrong rock

yep

Also this: "It takes anywhere from 18 to 254 day... to form a new habit and an average of 66 days for the new behavior to become automatic." https://www.healthline.com/health/how-long-does-it-take-to-form-a-habit#takeaway

along with Ebbinghaus forgetting curve

5 days a week x 6 weeks is only 30 work days of habit forming. So even six weeks has a risk of teams leaving a dojo and new habits not sticking.

Why 6 weeks? Why are 6 week boundaries attractive? Well because math. People are really bad at predicting the future, and therefore we have to develop means and mechanisms to understand how to break up time, which is a human construct as well. People have been dividing their time into smaller parts since the beginning, of well, time.

@mring - which is why quarters are useful - form - 6 weeks storm, 6 weeks storm/norm, 1 week retro, repeat

I agree with @mring - ideally it would be more than that amount of time - that was our minimum

we also saw cases of unlearning, where 20 hours spent in dojos, 20 hours spent at desk unlearning

it needs focus

and repetition

I love this topic - I’ll be around anytime to discuss this more

#same - its my philosophical rule num (#) 2

esp in the gov.

Discover’s dojos are typically 4 to 6 weeks, with a hands-on and interactive experience that covers real-life scenarios which you wouldn't get from classroom-based training. The time line comes from the selection of learning modules that has been crafted to meet the product team. Dojos give product teams a thorough understanding of core concepts while leveraging their existing product backlogs for pairing and upskilling, creating an experiential learning environment that accelerates adoption in the team's backlog immediately. Its the combination of workshops in the morning, and pairing in the afternoon. Old habits take time to break as they practice new patterns to form new habits.

we tried 4 weeks, but we found teams felt more pressure to deliver something. 6 weeks gave them some more room to breathe and learn

I feel the same way with a 2 week sprint - lots of time spent churning, esp early on.

software development is like repairing a tractor. sometimes you need to take a few bolts off to understand the problem

drink!

I particularly enjoyed @bryan.finster486’s talk on weaponizing DORA metrics yesterday — was absolutely fascinating!

Hi - we do have a bunch of team metrics we use in addition to the ones we highlighted today. Code merge, refactor, re-use etc. included

The metrics I presented were the minimums for all teams. The teams are free to measure with other metrics that help them solve their individual problems.

👍

we have a central team of enablement specialists that help with training and helping teams adopt new ways of working, sounds like we are heading in the right direction, thank you for sharing

We started using this to horizontally scale the knowledge of the goals. https://www.engineeringthedigitaltransformation.com/

Trying to push knowledge to the edge.

PtP? PowerPoint or Point To Point?

Peer to Peer?

powerpoint to powerpoint

Thank you Our team is so excited and proud to share with everyone!

We learn, we share! We get better together! That is a core Discover behavior!

Really appreciate your comment! Love sharing our journey.

If I may submit a patch, Gene: “what objective evidence is there that a certain person is perceived by others at being good actually good at X, Y, or Z.”

Those are two different but highly related questions

And important

😆 Nice. Just wanted to have a test case that detects when someone is not actually good at something. (Or anything. 🙂 😆

Exactly… dunning-Kruger effect is very real, and there are folks who are amazing at expressing confidence while actually creating problems due to lack of competence in ways much harder to detect quickly

Thanks for the reference. Looks very interesting.

Thanks Topo!!

Thank you

Thank you for taking out time to listen our story!

Not yet, but stay tuned! ;)