TiringAF for sure but worth it

Transcripts?!? That puts it in a whole different permanent light.....

To quote a UK rugby tour rule - what goes on tour stays on tour! I hope this is a safe space for us all.

@davidorsi72: the Slack messages are archived and available after the event. @genek101 mentioned that during his "non-plenary".

Gather is pretty cool. Used it with another conference, but it loses chat context and doesn’t persist contacts, so drive that back to slack IMHO

hard to be everywhere at once!

Yes it’s a persistent existential problem

Yes! We’ve been running around in that space and it’s… fun.

Dropbox: http://itrevolution.com/DOES21-db GitHub: http://itrevolution.com/DOES21-git

awesome balance of business operations and implementing strategy

Crampt airline seats and possible deep vein thrombosis 😂

Working hard to bring travel back the world.

I ❤️ travel in the sense of visiting new places and people, but the long plane rides are expensive agony, personally. But the end makes it tolerable.

or watching videos of people from other continents.. new travel is no travel. 🙂

We discovered that our daughters passport expired a couple of months ago, and found the same thing, no slots for normal renewal.

people checking passport expiration dates now 😄

Being able to compete in a world where AI and heavy automation is playing a large part in personalisation and pricing, we had to consolidate our structure and systems to compete

it depends on the project 😉

Banking use to be a very traditional sector but lately it is accelerating in tech transformation. I’n my opinion it has the opportunity to leap frog due to a lot of legacy tech. We faced a similar problem.

Surprising admission though, not everyone wants to fess up to that stance these days. Which I guess is a good sign

Executive buy-in starts early and is an ongoing process

It says a lot that it was viewed as an opportunity

The key is not to fight the perspective. It is due to local KPIs and rewards systems. The key is to make it so rewarding or painful for them to stay local that they have to move. This usually require a lot of work on top level leaders

and you did point out the importance of getting leadership on board for (any) transformation, so true. it was a great talk.

I'm giving a talk tomorrow about the synergies I discovered at my org (Cornerstone) between our D&I efforts and Devops adoption. Focusing on leadership commitment was the first big lesson we learned that applies to both.

Yes, leadership commit is a tipping point. I started without that 4 years ago but build it until I had full top down commitment

It's the essence of being shocked into action I tend to think 🙂

One of the DevOps Enterprise Forum papers last year talks to this: https://myresources.itrevolution.com/id006657107/Bold-Moves-You-Can-Make?_ga=2.15856502.1991561961.1621258323-462588578.1592870787

😆 this paper made the bookmark really fast https://docs.google.com/document/d/17nHB3aDOsUI2f3W5EShyoFQiK7NMwIJ8QFkUCANVPbk/edit?usp=sharing

3 screens have their advantage in copy pasting stuff around quickly 😂

Yes, Aggressive top down goal is one of the keys to successful transformations

You need to have bold and visionary leaders for that 🙂 Agree

You just need one person with a vision. The leadership will either follow, or change to a space where they feel comfortable.

Correct, this is very typical to large enterprise with smaller business units. Growing through acquisition result in major duplications across your organisation

Two things helped us: 1) The need to work from hoem due to COVID forced us to a different way of working and therefore removed local boundaries very quickly. Suddenly the ‘team’ is not the office location anymore but could be anybody we placed together. 2) The business and IT decided to consolidate around product business lines. The biggest changes is for local teams to switch from being a local leader to a global leader.

So @pieter.jordaan did you setup teams across the boundaries of the physical locations? How do you expect that to evolve as we are slowly getting back to work in the offices?

The top down goal helps. There is always resistance so you constantly need to help people to go through the change curve.

Completely agree 🙂

You have to be honest with yourself. It might feel safe to do it slow but you might not be able to survive if you do it slow.

We were consolidating from multiple regions we typically had many experts in a business domain. Where we lack experts were in the new ‘capabilities’ being introduced .

I was thinking that would be part of the challenge @pieter.jordaan, that you'd have several experts, and there would be jostling for position. How did you (or others) evaluate which of the many experts would take a leading role? Or was that not a factor?

We did not create roles first. We created tasks. Then gave tasks to several potential leaders. This quickly showed who already have the mindset and who do not. Then the org structure followed afterwards. So teams were forced out of their comfort of the role and had to address issues without the “role” specification.

we worked together with AWS and Slalom UK. But key was to train as well your own people

Agree, you have to train along side. We made a conscious choice to not work with massive SI who would benefits by running our platforms. We wanted to run our platform and have the knowledge in-house. We have different Cloud Suppliers depending on our need.

Hopefully @pieter.jordaan will talk about the challenges too

The mission was top down. Unclear that the implementation was top down. Would love to hear more from @pieter.jordaan about which parts were top down vs bottom up.

Interested too

One of the key drivers for cultural change is single threaded leadership all pushing the same agenda 🙂 That helps a ton! In that light all parts underneath did do their part as well though.

I agree but learning and unlearning takes time.

@robert.ruzitschka, The vision “What” is top down. Aggressive Goal for the whole company. In practice the implementation is lead by Domain Owners with autonomy on they “How” the consolidation happens. The Domains have collective milestones to meet. Bring capability x live etc.

The amount of times I've heard this in meetings 😅

That very much sounds like Scotty in Star Trek. 🙂

:thumbsup:

The transformation was not focus on jobs but tech consolidation. We redeployed existing IT resources as far as possible

you can hear more details around this as well on our talk tomorrow. especially on role transition

That's very impressive - thank you both, I look forward to tomorrow's talk!

No. We had to make it work to mitigate a large problem we faced

👍

They key to success and momentum as to create this urgency.

Yes, and that is what we are missing, no real sense of urgency, and through that we are moving very slow, adoption of the cloud is slow, everyone is very careful and hesitant to change. But listening to your experiences we may need to change that… 🙂

The load was the same. Because we built it into the cloud we could do testing and performance scaling much easier than if I was on prem. This was a large part of our scalability issue.

Creating a win-win is key.

For our initial migration it was 50/50.

We picked out best internal people and peer them with external AWS Partner. They learned by doing. At the same time we had a massive learning program to train internal staff. Maybe 20% was new hires.

Ah! I suspected there must be a clever solution - thanks!

Middle management definitely change over the years and even within this transformation we have leadership change. I would say middle management is the most important to have to correct mindset. I focus a lot of time on this

Thanks - but how where you able to change this in such a short time? Pretty incredible.

We did peering with experts from the outside and then heavily invested into training to the whole IT organisation

does it mean special time investment (classroom instead of business as usual) in learning/peering?

classroom? what's that? 🙂

peer programming for the core team. For the rest we ran AWS AWSome days and also promoted certification.

at least they were little?!?

So many obvious remarks on the size of the KPI, so little value 🙂

really stark reminder of the dangers of silos and proxy metrics. Fantastic case study Pieter!

'Metrics drive behaviour'...

Thank you for posting this. It is a valuable read. DO NOT give it to your boss. You might miss out on your bonus ;-)

It is about the balance of investment based on global benefits rather than local only, It changes the way you evaluate what you invest in. Local only investment is not typically tolerated unless for legislative reasons

I think it is also worth making a distinction between localised demand vs localised systems.

Hey @pieter.jordaan Just to pick your brain on this a bit more: How have you structured the teams and the technical architecture for local partner integrations (e.g. non-global hotels, airlines, etc.)? Is it generally one org and one platform responsible for developing and maintaining the integrations for local partners across the globe, or have you split it by region/locale? How do you balance moving fast with localised integrations and keeping maintenance low over time?

Localised integrations is being minimised - In fact we architect to remove local integrations completely. So it is a single platform used by local teams rather than locally integrated.

Thanks, appreciate the insight.

Yes, geo and domain wise.

How distributed are teams and decisions makers across the globe?

Very!! We have teams that might have developers from 5 countries. The services are also distributed across all the countries we have in Europe as well as India and nearshore.

This is a great book! For anyone in any job

“Our specialized skills and mental models are challenged when we integrate with people who are different. If we can constantly change and improve our models by using technology in the pursuit of art, we keep ourselves fresh. The whole history of Pixar is a testament to this dynamic interplay.” - Ed Catmull, Creativity Inc (Chapter 10)

“Unleashing creativity requires that we loosen the controls, accept risk, trust our colleagues, work to clear the path for them, and pay attention to anything that creates fear. Doing all these things won’t necessarily make the job of managing a creative culture easier. But ease isn’t the goal; excellence is.” - Ed Catmull

@genek101 @jason.cox @philipday To understand the Catmull’s influences - you also really, really need to read Loonshots, by Safi Bahcall. The author traces an approach to overcoming resistance to new ideas from Vannovar Bush - famous for developing a system for nurturing key technology breakthroughs (like radar) that helped the allies win WWII - in the face of significant official resistance. Catmull appears towards the end, one of many in a chain of astonishing innovators - who learned from Bush or his disciples. These ideas belong in the DevOps universe. Value flow starts with the idea. This book backs up to the moment of birth - and addresses the messy process of shepherding ideas from the ugly baby stage to maturity.

Dr @ronwestrum mention Bush to me when I interviewed him! We’ll Reread that book! Thank you!

Really similar to the precepts at Xerox PARC, where Ed Catmull's Pixar cofounder Alvy Ray Smith worked prior to working with Ed at New York Tech. Also inspired by ARPAnet, Bush, and others. (Alvy was a collaborator with my Dad on computer graphics at PARC; they later shared an Academy Award for early videographics 🙂 . Now I'm remembering that when Creativity, Inc. first came out, my Dad and I went to a Computer History Museum talk by Ed, which ended up being a reunion of all these 1970s-generation computer researchers, including Alvy, Don Knuth, and others. They all knew each other! Alvy was kidding Knuth that he'd never finish TAOCP.)

Hi, @rshoup — so glad you're here, buddy!!! 3am here in PT! :)

Oh!

🙂

Exactly @rshoup, not always well understood by some teams willing to jump on the trendy bandwagon called DevOps

Is it failure if you benefit/learn from it?

Totally, well said. There is no failure. There is only learning 🙂

We redeployed existing employees. The focus in IT was not cost saving but IT system consolidation. We actually had to ramp-up resources and staff.

Yes, but that was more common in IT.

This is a key abstraction. Focussing on Cloud and Product is higher level goals enable Agile and DevOps to mature in a context that allow them to grow.

such a shift in mindset for so many, but when it happens, its a beautiful thing

Love this one.

If I'm tempted to say "the business", I try to say "our business" instead

love it. I make it an emphasis this with our collectives teams: our customers, our users, our process, our capabilities…

@jeff.gallimore, agree, this is an ongoing challenge especially organisations that is coming from a world that required 5 year vendor contracts. The cloud universe it fundamentally different and allows rapid prototyping and changes.

thanks for that terrific insight. yes! organisations often fail to build flexibility and change into their contract/acquisition approaches. i see it all the time in the us federal government.

This is key

Framing the goal this way is one of the most audacious things I've heard in years, @pieter.jordaan — it's so great, and so relevant to so many organizations!

It drives the mindset in the business to also consolidate and transform into a Product based univers

Absolutely. I once discussed with a customer that they needed to first rationalize on products and services before rationalizing their IT landscape. It was a similar conversation, but sadly I did not get the amazing outcome that @pieter.jordaan managed here!

yes! but decide the right one first!

@philipp.boeschen650, it is actually becoming culture in our organisation and we forget how hard it was to get this accepted.

VERY good question: Moving business from value flow (most customer benefits first) is the key. This is a key concept of the flow metrics in Project to Product.

Put another way, how does the organisation resolve the tension between 'federal' de-centralised approaches and centralised teams?

The business areas also have to consolidate .i.e. if I had 5 pricing teams they have to consolidate to a single pricing team that use a pricing systems the best way to fit their market’s pricing need

So do you have a Pricing API factory then?

No. We opted against factories. We decided to migrate whole products at a time. We launch seasons of holiday products. So a whole new season is launched on the new system while the old system is being sold off and get deprecated. It is to hard to deal with factory patterns across 15 countries.

The underlying meta and master data, pricing models and contract data makes it impossible to approach the problem that way.

Thanks, Pieter! That’s helpful

One last question - if you have time: how do you handle cross-cutting concerns in multiple products? For example an authoritative data source, like flights or reservations or something like that?

This is a great talk, yielding lots of questions we'd love to ask

Here's the link to tomorrow's presentation, Digital Transformation When Goliath is Not a Competitor But a Global Pandemic: https://sched.co/ijOD

feel free to ask them

The other presentation from TUI is amazing: excerpt from their abstract: > How do you transform your IT – technologies, workforce, skill and culture - while the business is - disrupted to its maximum? > How do you cope with losing 1/3 of your IT resources overnight? > How do you still increase your deployment speed? > How do you enable an international workforce to work seamlessly on new features? >

A question and great answer here - https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1621330977091500

You don’t get that often anywhere. We did maintain key platforms like finance, HR etc

How long was your change death-March?

change fatigue is one of the biggest reasons and also motivations for them to transform

It wasn't quite the same under-the-gun, but moving from a monolith to mini-applications took ~4 years

Started with the highest revenue pages first

Ongoing now for about a year. Regulations...

Starting with an area we think will help productivity the most, then the part that is changing process the most.

@rshoup, @pieter.jordaan what was the size of your respective dev org?

For that 2002-6 transformation, ~1000 engineers. Migrated in parallel with upkeeping the old monolith.

Good question. This is a longer answer but in short, yes, we develop services that deliver the capabilities. The service then has the ability to move old systems out and move to newer once as soon as they are availible.

i bet @matthew and @me1208 would have something to say about this, too. (i reference their book in lots of conversations like this)

Thanks @pieter.jordaan and @jeff.gallimore!

Also how the cut over from old to new happened

I would not think so

Correct, the risk appetite definitely changed during the COVID.

Was the risk still as high when you weren't facing as much downtime/outages? It certainly seems like you seized the initiative you were offered.

yes we will go in a hybrid format. where we definitely enable more remote working. However we found as well that some people like more the office atmosphere so we will enable both. But we also see this as a challenge when we have the hybrid approach.

Hybrid seems to require careful design... I've heard it described as the worst of both worlds. That said, I have seen polling split sharply into two camps, one group is dying to get back into a shared space, while the other group is now saying they desire fully-remote forever and will switch jobs if forced back into the cubicle commute. Of course it depends on the needs of the team and role, but I think there's a strong aspect of innate personality as well, likely along the introversion/extroversion dimension. Would make an interesting study (if well designed).

100 percent. It is oxygen

Arguably most of the benefit is about flexibility and I always believe it should be the main focus.

It can cost more, right @corey?

I call it "Technical Savings" (continuing the analogy with "Technical Debt"). One of the advantage is that it gives you room to breath in case of bad days.

@nickeggleston It can always cost more. 🙂

Reminds me of the quote: "you can buy better, but you won't pay more" heh

Focus on flexibility is such a good way to put it. Companies that want to move fast and win are prepared to spend to do so for the agility and flexibility.

I would say that we need to change way of hiring people. Many times there is too much time lost from business point of view to looking for 'right' candidate. We need give more chances for those who can learn fast even if on Interview they have gaps in expectations. Many people love to put extra steps to learn fast. And they will be really great contributions for organisations.

A trend I've been seeing is figuring out the hard problems around the structure and then providing practical guidance around these. Less high sky governance more practical examples.

Amazingly simple, this is what I am trying to push at work

Creating simplicity is probably the hardest exercise as usual 🙂 Especially getting the governance mindset out of the day to day is a challenge

Yeah experiencing that for sure

I found that quality really goes over quantity, there is definitely a thing like policy fatigue... don't bombard teams with pages over pages of docs

I advocate the bare minimum output, trying to kill all of the wasteful documents - I prefer things like "as Code" and C4 Model Diagrams over anything. For architecture, I am pushing for things like Terraform modules, principles and guidelines

We do a ton of central terraform modules, with the added benefit of "if you use these security has already checked them and you're good to go" That works really well but scaling that is definitely an interesting challenge in internal marketing mostly

Unfortunately indeed... Again mindset and alignment issues...

Thanks so much for an engaging and super-interesting presentation!

This is an ongoing process and we keep re-iterate the needs for “feeling safe”. It underpins agile and we have to have visibility. Luckily the transformations goal is not cost saving but tech consolidation. We are redeploying all our IT staff as much as possible. The pressure of such aggressive goals is double edge as it give purpose but also needs to be balanced to make sure you do not burnout your people.

Fascinating talk and challenge... Thank you.

Thanks

Never waste a good crisis. I wonder how much timing and an extreme crisis in travel enabled this, where otherwise there would not be the appetite?

You'd be surprised how many discussions get cut very short when you lost 98% of your current revenue 🙂 We wrote a little piece on some of it over at https://techbeacon.com/devops/how-pandemic-drove-digital-transformation-tui as well maybe that helps

Thanks @philipp.boeschen650

Ah, this is a big outcome: > we reduced our net burn rate by 70% in just three months

I definitely like the idea of doing less, but doing it well

Yeah the whole concept of limiting work in progress at that large of a scale was interesting to discover during my research for this! Reducing context switching and opportunities for conflicting objectives is a really big win

@david.read, the point of the talk is that your perspective and preparedness matters: We moved from a “recipe for disaster” to facing an actual disaster - COVID. This changed our perspective on risk, value, time to market and therefore our approach to transformation. Within the context of an existing COVID disaster our perspective changed, forced a different approach which we indeed might have seen as “recipe for disaster” in the past. I tried to highlight key points which other organisations can adopt. Making such a large scale change helps to break the inevitable dependencies between IT and Business consolidation but indeed require a organisational context that is ready for it. Not having the basics in place is a “recipe for disaster”.

It is a journey and the answer very much depend on your context of your organisation. I would focus on 1) training at ground level, 2) Influence top level. (The big cloud providers have programs to help with this)

thanks!

Thank you @genek for the opportunity

Thank you so much!!! Hopefully all the Slack messages demonstrated how much excitement you generated!!!

Slides and Transcripts both are available to download on the above page 🙂

What a dynamic marketplace... (So good to see you, @lucas.rettig ). Cc @fernando.cornago442

which exactly?

to start to get everyone aligned with a new vision, new strategy (if I am saying it right)

it's tricky topic... we have a different organisation for DNA but their profiles integrates heavily in our teams, Data Scientist for consumer analytics or Data Engiineers in our Consumer Data ecosystems

i’m facing the same type of thing… in my mind they are so unified, in reality, i separate these out when communicating because of the organizational differences

We outsource of the of our Non-WIN capabilitites

some SaaS where we compete

Digital Workplace is an example of 95% externalization

@fernando.cornago442 - Understood - Thanks!

I was just thinking that this also has a massive impact on middle management roles. It sounds a lot like this is happening at both TUI as well as Adidas. Fits in well with the “Technical Maestro” concept that you discussed with Ron Westrum.

there's a constant change management needed of course

but the market business owners have now much more simple comms channels with the global product ownerrs

That makes sense. Thanks for replying!

product managers should speak <33% of the time in product conversations… vanity metric to observe in my experience

It's not only volume or city-potential-value but the affection of the city in the whole market (Setting-up trends, etc.)

Thanks

@jason.cox is my gurú and I would even call it a mentor...

24k as we speak 😉 and growing as we grow more and more in volume and customization

10X tech volume growth. Teams grow is at 20-30% per year

sign of good things when your VPs acknowledge this

hahaha I do, I do

39 giantswarm distro clusters in 6 regions, plain open-source k8s api

Thank you

Great points when it comes to reliability, amazing stats about the HYPER drop 1.5 M Visits/SEC

Great to connect with you @vikalp.yadav

THE SIMPSONS DUFF BEER SHOES https://www.adidas.co.uk/release-dates

high demand products sale, where our system handles high loads

When the new Yeezy hits the shelves

gotta get that new new simpson duffs

https://apps.apple.com/us/app/adidas-confirmed/id1508115448

http://yeezysupply.com

adidas members just went from 500m to 500.001m

Looks like http://yeezysupply.com thinks a DDoS is underway

its scary and challenging and collaboration intensive

Do you have a separate set of SREs working on the reliability and the teams work together with them or are the SREs themselves embedded inside of the delivery teams? :thinking_face:

“making events non-events” was a OKR of mine circa 2017 when i led the backend data platform for tcom

i was blown away when i first got exposure to how these things were managed

yes! our consumer centric lens to look at interruptions and opportunities

happy to exchange experience !

I'm up for that. Will you be around the Gather later for a chat?

Breakout room?

How do we do that?

Hmm... I don't' think Slack does voice channels... but we could all meetup in the GatherTown and talk there. That's been going pretty well.

GatherTown is still available if you (pl) want...

it is a Change Control Board of sorts 😄

Such a waste of time and talent

You had to bring that good old chestnut 🙂

😄 it’s the tunel where all the money flows in !

We had a little scare this morning when we weren't getting the telemetry we were expecting — exciting/stressful, but very small scale compared to what you're responsible for. Ah, launches are so fun!

definitely yes!!

getting there....

but we need to be careful... you saw the 3VPs in a room step back 😉

Ha ha 🙂

we will open it up on github soon

cc @christian.rudolph we can be excited!

Data-driven company 🙂

Speaking for MS, we have a central metrics team collecting it all. It does require the orignal sources e..g incident tickets to have reasonable quality of input data though

not really secret 🙂 we have 3 main areas, product, value stream (dependencies) and environment. Each team has to identify in their area what are the sources and if they all have the data. We have weekly connects to share the progress and if any team (in the value stream) needs something

Ah 🙂 That makes sense, I think our smallest unit would be about 14 teams right now this has a few more challenges bringing the data together so far

yeah indeed, we decided to work with k8s deployable unit, like for us checkout api is a unit and we have 3 teams

It's really intriguing a k8s setup for that scale, do the standard K8s deployments still work or is there some custom engine behind? I could ask questions about that setup all day I think 😅

😄 we have a platform team that provides k8s clusters, it’s at the end standard k8s with some HPA in place

Oh that's interesting! I assume the HPA scaling metrics are external metrics though like incoming requests and queue depths? Do you buy into any of the multi cluster management thingies or is there something homebrew for the provision?

Not a different team, but set of QA KPIs need to be checked as well as product and environment! QAs are part of the teams

@andreia.otto that is interesting, are the QA KPIs check the transaction flow, some end-to-ends or are solely connected with the site (services) health?

We have both, each team focusing on the service health but also some end to end. For release fitness we use the service (k8s deployable unit) checks

@andreia.otto, so the release fitness pass was enough to release the code to production?

yes, as it will collect all the metrics we need to deploy

now looking forward to see you wearing one of those "bad babies"

Ask as much as you want! In prod pipeline we update the dashboard and get the latest GO/NO GO value, We started with a dry run mode in production pipelines to understand what was working and what needs improvements. When enabled, the NO GO will fail the release, then someone needs to check the dashboard, where exactly is the issue and then if it’s false positive or ok to release, we can skip the verification

I presume you need a pretty good test automation suite across the teams to quickly localize issues? Or was/is this still a struggle?

it’s still working in progress, that’s why we have the blocker tickets check, those can be created manually or via automation

Brilliant! Thanks for answering my questions. 😊

@saket.kulkarni you are welcome 🙂

WE DO ;)

Really good question - how do you shift power from the VPs to the devs

Sounds like a good KPI to evaluate how management initiating Change/transformation programs, are actually eager to change?

we have full support from VPs and all leadership team, moving to release fitness (and proving it works 🙂 ) bring more confidence to everyone

all we have is internal still, we are working on a post to share it soon

Would love to have you talk about this at SREcon (CfP opening at the end of May), conference in October

Sure! count on me, will be great to share it!

We are looking at maturing our own internal SRE capability and your talk provided a lot of inspiration

thank you @matthew.cobby! glad to know it!

KPIs are coming from different sources, we use prometheus (also blackbox exporter), jira (for ticketing), custom code to get the releases of the day… all KPIs are automatically captured and dashboard updated

Got it, thanks

They're all special needs 😂 but that was a "Special" advisor wasn't it?

Good to know no one picked up on it.

I'm surprised/horrified/frightened/impressed you mentioned it

Your platform is doing the job 🙂

Thanks Vlad. It's an amazing digital platform - many civil servants/Equal Experts folks/other parties have worked on it over the past few years. @ben.conrad @mhyatt do a great job of steering it

As I hope will become clear, we weren't directly involved in the policy, or writing the code for most of the services. But there wasn't a lot time.

That’s what i gathered from your intro. Still awesome

Looks like agressive timelines is a theme

20 days? Luxury 😉

Not all four in 20 days. 20 days for JRS, 30 for SEISS.

Different teams, doing most of the work.

thanks Ben. Still "wow" 🙂 Good work.

They show what organisations are capable of when management get out of the way?

It is more about what you can do if you suddenly view risk in a different way and you can inspire your people

< 4 weeks

he said some services were delivered “a week or two ahead of schedule” on an original deadline of “20 working days”…

Yes, there were further services delivered ahead of time but the first one (with the 4 week deadline) was delivered JIT

Silver linings....

It was funny the first time. Once you've practiced the talk for the 20th time, not so much.

I assume bureaucracy is quite suited for being automated, provided there's a will to remove it from people's way so they can focus on providing a better service. Ultimately I would expect automated bureaucracy to bring even more consistency than humans.

Historical. Does that answer it? It seemed a good idea at the time...

Copy, we have some historical Scala here at FedEx too and it’s hard to find talent

Thanks!

There's a balance between supporting a single language, being able to really delivery amazing tools focused on that, and supporting other languages where we would need to be more generic.

Hiring is a problem for sure, although we think of Scala as an interesting gateway in some ways to ensure you get good developers.

That’s an interesting perspective! I hadn’t thought of it that way before.

PS: I love Scala, but eventually chose to go down the Clojure route — two amazing languages that leverage the JVM.

I thought this was a fascinating language selection as well.

Thanks, Gene! What were the benefits of Clojure? I’m not that familiar with it.

@bernard.voos There are some fundamental properties of Scala/Play and the reactive programming model that suits the type and scale of interactions we see on MDTP. It's possible that we could have achieved a similar outcome with Java and e.g. DropWizard but Scala was chosen at the time by the original (small, skunkworks sort of) team that developed the very first app, and it stuck. Scala has turned out to be a pretty good choice. Technically, it works and is both reliable and performant. From a people perspective, we've been able to scale to hundreds of developers and, as Ben says, there is almost a natural barrier to entry, which means we've often attracted high-calibre engineers. Supporting another JVM language on the platform is something we often talk about. Java, Kotlin, or Clojure might fit relatively neatly into our existing stack but it would require work to adapt our tooling and we'd be introducing a potential people constraint when passing services across teams.

☝️ this is what I meant to say, but don't type fast enough.

Thanks @mhyatt!

What do you mean by DevOps tooling Kim? There's more info in the EE playbook on digital platforms https://digital-platform.playbook.ee/examples/hmrc

MDTP has self-service platform capabilities for build and deploy, telemetry, runtime, etc.

Like CI/CD tooling, registries, repos, test tools, security tools

Looks like yes from this slide

It supplies a deployment pipeline, a container registry, auto-initialised code repositories, etc. It's super opinionated, a paved road

Cool. So is it all containerized deployment?

Yes, Scala microservices running in Docker

Are you including security tooling/tests in the pipeline?

Yes, there's some details in https://digital-platform.playbook.ee/, Equal Experts has done similar things with other clients as well

@kim.weins Yes, there is some security tooling built into the pipeline; Zap among others.

👍

Missed out a load of things that are valuable.

User research, policy development, loads of stuff went out the window.

But by having a platform that forces a load of testing and by having dedicated people who knew how we operate it was (just) possible.

@katharine.chajka: good question! Can you expand a little on what compliance you're interested in? The Government schemes fit into an existing tax system, so the data for the claimants was already held and in use for BAU tax purposes, as were the payment systems. The was additional rigour around fraud risking, which we touched on a little in the talk... does that help?

yes, I am not familiar with the specific regulations, but often I hear that there are steps that "have" to be manual in regulated industries, they have to be a different person or department pushing a button so 'we can't possibly" speed up certain parts of deployment/release due to compliance/regulation type concerns. That was running through my mind during your talk. It looks like somehow you were able to streamline this

Ah, that really helps, thanks @katharine.chajka When a new service is launched on MDTP, there is an established governance process to assure that: • whatever has been delivered is secure • it conforms to policy / regulatory requirements • technical best practice has been followed etc. However, once the service is live, we trust the service team to make as many changes as they need to, whenever they need to, without any further governance from the platform. Each service team has a Product Owner and an over-arching Service Manager, who between them, are ultimately responsible for the changes. For the Covid services, it's true that the road was cleared by HMRC's executive team, so that everyone was empowered to make the rapid business decisions necessary to meet the deadlines. However, from a technical perspective, the release process was entirely BAU and relied heavily upon the established tooling and culture of trust that we've worked carefully to develop over the last ~10 years.

Hi Gus, Yes, the platform was greenfield at the start, rather than being transformed, extended, or migrated from something else.

Yes, there's a very interesting tension there Gus

Team empowerment is super important

At the same time, when you have 60 teams, Conway's Law means max empowerment will lead to 60 different microservice implementations

the trick is to form bi-directional feedback loops so the platform teams are constantly learning from the service teams what's good, and what isn't

and to create a sandbox that's broad and deep for service teams, in which they're empowered to solve business problems as they see fit

Where does the innovation take place? For example challenging some of the platform services?

Right. And I think it was Tom Clark's ITV talk a couple of years back that hit on the fact that 80% of teams don't mind being provided with the golden path model as they will just use it

I was just having this conversation with @dana.finster . I’ve the opinion that freedom should be balanced with logistical complexity.

if it is good and engineers have a feedback loop as you say

@gus.paul Yes, I know Tom Clark well, we've talked before about the interesting contrasts between the EE/HMRC approach and the ITV approach. Over the years, ITV gradually gravitated in our direction

My IDE doesn’t create organizational complexity. Total platform freedom makes scale work against us.

@markus.lauttia Innovation in digital services takes place on the service teams, the whole emphasis is on them solving business problems. They don't get to choose which caching tech is used, they get to choose how to handle data between 3 different govt orgs in a way that's fast and secure for users

Every language / framework adds complexity. There need to be a cost / value decision.

Cranes and spanners on a site. Standardise the cranes. Don't standardise the spanners.

@gus.paul Yes, the HMRC/EE platform shows if you build a really great developer experience (sorry for buzzword), developers won't moan about the tools you've chosen! They'll get on with it

How I try to balance this in our organization is the old phrase "with freedom comes responsibility". I agree that most engineers don't mind of they get a lot of the backbone ready.

If you build the platform for the undifferentiated heavy lifting and allow teams to make changes to it via Inner Source (ask me later) then you can take care of the boring stuff without imposing cruel constraints while allowing the service teams to innovate where it matters. You only have so many engineers. Do you want them to be building the same thing multiples times, often badly, or do you want to have them focus where they can add value. The real challenge to this is the way AWS works where every team has its own choice but they are also ruthless on enforcing accountability.

@markus.lauttia You have to iterate on it, https://digital-platform.playbook.ee/introduction/what-is-a-digital-platform#bi-directional-feedback are so important. You need to find the balance that works best for your organisation. No two digital platforms are the same

@steve.smith thanks so much for explaining the tension/trade-offs. You've done a far better job than I would have. 🙌 I'd would add, though, that devs do still moan about it... but only at the start! 😁 We tend to find that some engineers that are new to MDTP are surprised by the constraints and find it uneasy to begin with. However, after a few months, they start to see the benefits. Typically, the first thing they notice is how quickly they can get things done using the common tooling that the platform provides, which is only possible because we limit the scope of e.g. languages, or databases in use. Later on, they start to see the scale and understand that to enable 70 teams (2000+ people) and to develop 1,000+ inter-dependent microservices, quickly and without much top-down organisation, would be much harder (impossible?) to do well whilst potentially allowing people to implement 1,000 different tech stacks. Scale is key to all of this, as is platform-thinking.

💯

How much focus do you put on DX?

@bryan.finster486 That's hard to say, as MDTP was started years ago, before "developer experience" was a thing. There's a big focus on bi-directional feedback loops, on iterating on the paved road, on modifying how and where MDTP is opinionated. It's a similar story with other digital platforms that Equal Experts has contributed to

@bryan.finster486: To add some more detail to the above... on MDTP currently, we have a team of ~10 engineers, whose sole focus is what Steve has described above. This team builds, maintains and enhances a suite of developer tools, using the same common components that we provide to our tenants. At this point, it gets bit meta but essentially, we eat our own dog food! That helps to provide us with first-hand insight into the DX, through the full lifecycle, including what it's like to run and debug services in production. Of course we still rely on regular, bi-directional feedback between us and our users, too. We are interested in how we can make life better for the devs on our platform, whether that's via a reduction in friction when performing a task, speed of delivery or fine-grained insight into what services are doing. We're particularly pleased with some of the tooling we've developed recently, such as Dependency-Explorer, which allows anyone operating on the platform to very quickly search across the 1,000+ microservices in our 6 environments to understand the dependencies (and transitive dependencies) for each microservice. This real-time view of what's actually going has proven extremely powerful for both the platform and our tenants. We now have far richer data when making decisions around upgrades, security and our future opinions.

Awesome. It's a tough call balancing freedom with the logistics complexity that freedom brings. At Walmart we had a huge surface area of technical complexity but focusing the platform on improving developer experience certainly helped to accelerate improvement. I think treating Platform as a product with customers is very important and it sounds like you're taking that approach too.

Agreed @bryan.finster486, that's Equal Experts' first principle on building digital platforms - https://digital-platform.playbook.ee/principles#digital-platform-as-a-product. We've used successfully at HMRC plus a bunch of other private/public sector organisations