Forgot to use threads 🙂 What do I mean? You create a pipeline that is compliant with AutoMate. Day 1 - all is good. Then over time, teams make a change here, make a change there. Somebody hire a DevOps contractor that goes wild and changes it all. Is that kept in line with a Pipeline of Pipelines approach or through a software clean room (Capital One) governance model or maybe it is just natural evolution

@matthew.cobby, sorry missed the thread. (great to have another Oz person here as well 🙂 ). So the pipelines themselves don't drift, and when we provide big updates they "re-execute" their pipelines and that makes sure that they still work the same way. There is drift in the code, and perhaps the frameworks they're using for development, but since the pipelines "just work" for them, they don't usually touch them and so very little drift.

It's probably about how you manage template changes. If devs start out with an old template and you change the teamplate afterwards, you probably want to keep the artifacts created from the template up to date as well. Is this the point where the script that you mentioned comes into play?

we started to build a dashboard who contributed to it

and so drove behaviour to show this in newsletters and events

I like that idea... I'll use that!

https://github.com/InnerSourceCommons/InnerSourcePatterns/blob/main/patterns/2-structured/praise-participants.md

We have similar league table of inner source contributors...it definitely works

That is a powerful motivator - we have an Inner Source programme to enable those changes into the platform in a formal way

How do you prioritize your work and theirs? We found they often didn't align 🙂

We find conversations work well. They do the dev work for their feature - to our dev standards

A deep dive into all the pieces that make up a pipeline, and all the technology involved. A big challenge, given many hadn't ever worked with containers before.

I bet. Thanks for the clarity!

Tim, I agree. But unfortunately in our case teams do not generally have a useful definition of done = in production.

@tim.wyatt Agreed, but measuring the middle points is useful because you can see the batching (from "The Goal" Book)

@robert.ruzitschka Since we owned the whole pipeline, we built "tags" throughout every tool to measure when a commit made it to that stage, so we could measure it at each point.

@tim.wyatt, @ashley.noble: I wanted to mention something, forgot it and now it came up again 🙂. If you want to focus on the reduction of cycle time, is it meaningful to start measuring with the commit? The story could already have been in progress for weeks (including the respective feature branch). We tried to convince team that the relevant KPI starts with the state "In Progress". What do you think?

@robert.ruzitschka I think there are a number of things to measure: • flow of work requests (stories, requirements, etc) into a teams backlog • Flow of work from backlog into "To Do" • Getting work done (from the point an item enters "in-progress" to "done" • Lead time from merge to prod (IMO)....

Quite a lot of this is covered in Mik Kersten's work (http://flowframework.org)

Yep. We are doing the exact same

Agree, the only meaningful approach.

pleasure great questions, thanks

Thankyou

Pleasure thankyou.

thankyou

Hi @mark.rendell have you seen the GitHub repos? This one has a lightweight example of a Team API https://github.com/TeamTopologies/Team-API-template

Ah sorry no @matthew I'd missed that and only looked at the shapes repository. Looks useful, thanks! Anyone out there got some examples that they fancy sharing please?

There is an article here on Team APIs https://medium.com/agile-outside-the-box/team-apis-af2dbc1805e7

Thanks Tod - have you looked at integrating the DevOps toolchain for visibility or adding a value stream management solution on top?

Can't speak for others. But its new to me

And delivered at super rapid pace 😄

The book is highly recommended!🙂

Adding the book to my reading queue 🙂

Too quick Filip?

ugh, it ages on my pile of shame 😊

"reading queue" - that's great

Now you can attend the academy and in 3h you can pretend you've read the book Filip :rolling_on_the_floor_laughing: https://academy.teamtopologies.com/courses/team-topologies-distilled

Thanks Manuel 🙂 sounds like a prerequisites for this presentation 😉

did you do any structured measurements?

honestly, no - the teams in question are very obviously overloaded :face_palm:

Nice heuristic @kevin986 ! Hadn't thought about it in that way, but makes sense: "over specialization as a smell for too high cognitive load"

Basically, it is not quite a team but a set of specialized individuals with not enough cohesion. Right?

Will you be the first to volunteer, Helen? 😄

Elon Musk might be able to help 🙂

:rolling_on_the_floor_laughing:

I'll check in with Elon and see what he suggests 🙂

Seriously though - you've given me an idea... I'm reaching out to some of the work neuroleaders at the moment on another initiative (like David Rock and Britt Andreatta) - I shall see if they have any nuggets. As you both know I am a tad obsessed with this subtopic of yours!

That would be great. Manuel and I are exploring how to scale assessments for team cognitive load - we have some PhD level expertise involved, too 🧠

kudos to Evan Botcher!

Such a hard problem to solve 😞

indeed!

sadly a common problem 😕

check out DORA metrics

How long does something sit with nothing happening to it?

https://www.tasktop.com/flow-metrics

Like from Domenica DeGrandis?

We were referring to the Accelerate / DORA metrics and we often recommend to look at flow efficiency at least

https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

https://getnave.com/blog/flow-efficiency/

A platform can be strongly opinionated without requiring anyone to use it

The two things are separate... or should be

The opinionated platform MUST be informed by the needs of teams builing software in that organization, however.

Opinions are no good if they are not suited to the context

My experience is that in many cases teams severely underestimate the effort to manage and maintain things that could be put into a platform. This reduces their capacity to provide business value. That is why cognitive load is a very useful concepts to frame these discussions.

Of course platforms must cater to the teams needs and must not hold them back or slow them down.

Platform is a product, and most products succeed because they are opinionated in a way that meets users needs (i.e. provides differentiation vs other similar products).

In internal platforms we need to be looking much more at: product adoption lifecycle and value proposition (differentiators) for internal teams.

If it's less cognitive load to use AWS directly than the internal infra platform, we've done something wrong.

Mandating the platform only exacerbates the problem.

Do you see a role on the platform team to be almost a marketeer/salesperson for the platform?

Sadly, mandating use of the platform is what I see too often - devs are compelled to use it, rather than it being a compelling product that they choose to use.

https://devopsenterprise.slack.com/archives/C0150HQB6UX/p1621347040138500?thread_ts=1621346631.119600&cid=C0150HQB6UX Absolutely, @bernard.voos - a skill often lacking in platform groups is marketing of the platform to internal customers.

Why is there resistance to the platform? Is the platform meeting teams' needs?

What incentives are in place to help guide the teams towards something more coherent?

Our platform includes our messaging system, common UI libraries, ... In general we get pushback on the UI pattern. Many time it comes down to "prove that this is the best solution". The answer that is is a "good enough solution" that we can iterate on over time is sometimes not well received.

Don't look at platform adoption as a "combat", you shouldn't be convincing teams, you should be making their life easier (and market how you do it internally).

But why force teams to use it?

If they have no need for it or have other needs that are already met, what is the benefit in "standardisation"?

Standardisation introduces coupling

It's an important trade-off that needs to take into account the sense of "ownership" in the teams

We don't have time to convince everyone, and in order to get our separate domain services to play well together, we need some standards for how they are built. The last thing we want is 50 different ways to attempt to make things work together. It slows the other teams down when trying to integrate and makes it more difficult for someone to change teams.

Have you tried changing the incentives around onboarding and integration?

Add "ease of onboarding" and "ease of integration" to each team's balanced scorecard

We're not up to score cards yet. We are still getting up to speed with new teams, new pipelines, devops...

Ultimately, though, some needs are not being met by the platform as it is: maybe DevEx/DX, docs, support, etc.

Sometimes it's just a person. Some people have very strong opinions that don't align with the rest of the teams. If that is a strong personality compared to others in a team, you get this kind of situation.

It's more a personnel issue than a technical one.

We are moving towards a set of architectural requirements that allow the artifacts from the different teams to work smoothly together in a consistent way. We can iterate on those requirements based on feedback and the emergence of new technolologies.

That should help, just be careful requirements are not "abused" in the sense you can make them so low level that you're basically telling people how to do their work 🙂 It's a fine line, obvisouly.

Yes, and we don't want that. Ours are more communication patterns between team artifacts that make them work together easily and consistently. This reduces integration time since you know what to expect from another team.

Netflix focuses on this concept for their teams

Indeed. Netflix used the DevOps Topologies internally extensively: http://devopstopologies.com

These are in fact the precursors to Team Topologies 🙂

Do they? And yet "Netflix doesn't do DevOps" :rolling_on_the_floor_laughing:

I'd rather say they do DevOps without calling it DevOps because... they don't need to.

Yet they look at available approaches and what can be useful in their context. Very different from so many orgs who just want to put a "DevOps stamp" over their image

In a monolithic architecture, you are going to have that.

Great - keep it that way, Olivier! That's what re recommend in the book. Use a CS team only if truly needed.

We have one or two, but they're very infrequent.

in an automotive ECU - I see some need for complicated subsystems

Can we hire you as "evangelist" @olivier.jacques? 😄

value stream map! :)

True...map it to simplify

The number of tickets needed to get some code into Production would be a in interesting measure, @manzi.g - ideally, the number of tickets needed would be 0, zero, zilch, null, none, ...

The service/product thing is a bit blurred, but the key thing ehre is that a Stream-aligned team should have a single focus (customer), not trying to serve two or more customers.

"ansibilizing" 😂

or "sensibilizing" ? 🙂

I would say sensibilizing now that we have most SAP products e.g. S4 HANA end to end on any major cloud stood up from scratch to functional working app in under 3 hours . Anyone who has done it by hand would tell you it takes weeks usually even in 2021 to build anything close to prod grade and scale. Thanks to TT we did it pretty well

That's awesome. Sounds like this would make a wonderful industry example if you're up for it @pulak.a.agrawal;)

Always @me1208

Awesome, will ping you later via email or Linkedin.

there is an industry std architecture called Autosar that attempts to abstract some of this away - this has a side effect of dumbing down the engineers imho

I'd try to start with a regulations enabling team.

A group of experts (likely not from IT) that could help most stream teams raise their understanding of the fundamental constrainsts in the regulations.

At the same time, look for opportunities to automate some of these checks. But sounds like fundamentally teams don't have enough awareness of regulations.

In fact, they need to increase their "germane cognitive load" on that front it sounds like.

(yes, cognitive load is not always bad as we explain in the book and in our Academy)

Great questions as always Helen! Nice to hear from you!

we are having a notion that any team in the org can provide an enterprise-wide and / or external service delivered according to a common service standard

Does anyone else have that as well?

The difficulty comes when a team ends up with multiple customers because that leads to priority clashes.

Yes, absolutely. If a team decides to provide a service, they take the full product ownership for the service.

Do you see this leading to issues down the road?

We agree with people like Marty Cagan that a Stream-aligned team (what Marty might call a product team) should have a single real customer. As soon as that team starts providing (say) some data "as a service" to other internal teams, they then have more than one customer. Far better to expose that data via an API< have a platform ingest that data, and then provide the data back out as apart of a proper platform experience.

Even for functionality the same problem applies. It's quite hard for a team of up to 9 people to do proper service discovery (which is part of a stream team's responsibilities) if they see every stakeholder as a customer, both internally and externally.

OR... give the Stream-aligned team enough time and space to truly provide a service to other internal teams, ... But that can get messy very quickly.

Separate concerns require separate teams. Otherwise, you'll likely see external customer needs always prioritized over internal customer needs.

Is there a limit in separating concerns? We ended up with a single, well-grained service, that is consumed by both internal and external customers. It's so small, that there is no chance to further distill it. Building a platform for it looks like an overkill...

@lucjan.giza is there a dedicated team that owns the service completely?

Yes

If there is no overhead on the team from this, then (by definition) you have a decent solution, @lucjan.giza 🙂 Just look out for the situation where the scale changes and a Platform is needed. BUT - as always - only introduce a platform when flow is impeded and/or team cognitive load is too high.

That is what we do as well. Seems to be working well for the moment being.

The music bits you have to thank @matthew the trumpet player :)

🎺 🎵 😄

What better place than DOES to announce new initiatives? 🙂

100% this, Robert

The "Platform" is really a viewpoint, a wrapper around a different set of needs or focus

In the TT book we clearly say that inside the Platform there should be mostly Stream-aligned teams... just focus on on platform-y things (data? infra? design?) rather than end-customer services.

Reading that really makes it worthwhile, honestly! Thank you!

not me personally , but i know our salesforce team has been working on a big bang data project and struggling to launch it successfully

i'm sure they could benefit from an agile, devops mindset

I'd be happy to connect with your team at some point if that would be useful

thanks.... it definitely may be.. will reach out to you if it makes sense to connect about it

We have an AMA later today - 18:25 UK time

I cover Salesforce DX (somewhat briefly) a little bit later in the talk

What, no Copado vs. DX comparison? 😆

I am having this discussion surprisingly often for the last 2 years.

Kinda sorta. The DX initiative basically opened up a huge number of new improvements that makes things better for everyone. There are things that only Salesforce can improve.

Nice that you're up to speed on the options @joachimsammer. I obviously get a skewed perspective, but this topic of "DevOps" has become a bigger and bigger deal for Salesforce customers. It's top of mind for most big Salesforce teams

I don't have direct experience on SAP, but Copado is starting to work on that. You could connect with them in the xpo-copado channel

"XML Merge" 😱

You'd love it @matthew Merge works fine as long as there aren't any repetitive elements. But, a 100,000 line long XML file is basically filled with repetitive elements.

You've got to parse it and merge it intelligently if you want to manage that XML at scale

Manuel and I once worked on a monolithic database system where it took over 24 hours to build the new code - this was in 2016

A developer VM in the cloud took 40 minutes to start

Almost all the business logic was in the database, but not even accessible as normal SQL - it was generated from models in a proprietary language - woo!

So much fun 🙂

Needless to say, development there was not rapid...

The term "database monolith" won my heart when I saw that in team topologies, since it describes Salesforce and the challenge of modularizing Salesforce. Miraculously, they've engineered things such that migrating and updating the Salesforce data model is super fast, and is not the limiting factor at all. The limit is on your ability to modularize your applications since they all get intertwined with the big database

That is good to hear. As always, the focus should be on a fast flow of change... and supporting technologies to enable that. :thumbsup:

Very grateful for your invitation to share @genek!

Salesforce packages all become part of the common Salesforce data model. So they're not isolated the way you might expect on some other platforms

That's what makes it tricky to decouple metadata into packages

Yep, so just a way of structuring your code and config. The underlying architecture is still the same SF core and add-ons.

Yes. There are some hard distinctions between packages - commercial apps can protect their IP, and get additional runtime (governor) allowances

But even though Salesforce packages don't segregate things very deeply, they at least provide some formal statement of where boundaries exist

Yes, at least there is a boundary. 😉

That reduces the risk of confusion about who needs to change what, or where the risks of a change may lie

... cuz you know, who needs to remember history, right?

Code amnesia for the win

Salesforce teams like the idea, it's just sometimes daunting if they feel they have to go to the command line, if they didn't come from a coding background

And Salesforce metadata doesn't make it easy. It opens up some problems that most code projects don't struggle with

VSCode is pretty good for Git integration, to be fair. There is even an official Salesforce extension pack! https://marketplace.visualstudio.com/items?itemName=salesforce.salesforcedx-vscode

With VSCode, there is (often) no need to drop to the command-line.

The introduction of VS Code to the Salesforce community has been a huge win for those who write code on the Salesforce platform. The challenge remains that you have a lot of non-coders collaborating on changes, and they also need to manage their changes alongside the coders.

@genek101, we had a similar challenge in Data Warehouse domain. Convincing DWH Developers to use git was a starting point. But you know - understandable Diff + Blame made the magic :)

So you have to span the skill boundary between the click-based folks and the coders, maintain a shared history, and be able to deploy stuff that's not just code.

Also testing in between. We invested our focus into a fast CI - which validates the whole data warehouse model with all dependencies on each commit.

Yeah, it's amazingly powerful when you can get such automation in place. Version control is the only big hurdle

Agreed. Version control is hugely powerful but it's not hugely accessible... yet. Interesting discussion/rant here: https://twitter.com/matthewpskelton/status/1387521384226643968

This is so good! Thank you!!!

PS: in confidence, I miss the simplicity of SourceSafe — the ability to share files was so awesome. (forget the occasional database corruption issue.)

@matthew, oh, I've been in exactly the same discussions :) Unfortunately a lot of tools do not provide first-class version control. Take Office365 - saves all versions automatically. But where is the diff, pull request, etc.? Finding out what changed is very tricky. In my eyes it slows down collaboration dramatically. "What am I looking into, what's the story someone else wanted to tell by modifying 5 slides...." Have no recipe how to get git accepted for WYSIWYG guys...

I intend to write up some of this stuff but the short version is that the solution here seems to lie in custom visual diffs based on file types. So an AutoCAD file gets special 3D visualisation diff An Excel file gets a custom column-based diff etc.

would be a dream come true 🎺

🤯🤯