This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-10
Channels
- # ask-the-speaker-track-1 (139)
- # ask-the-speaker-track-2 (146)
- # ask-the-speaker-track-3 (126)
- # ask-the-speaker-track-4 (176)
- # bof-arch-engineering-ops (9)
- # bof-leadership-culture-learning (3)
- # bof-project-to-product (3)
- # demos (3)
- # discussion-main (637)
- # games (39)
- # games-self-tracker (6)
- # gather (7)
- # happy-hour (48)
- # help (20)
- # hiring (12)
- # lean-coffee (1)
- # networking (2)
- # summit-info (93)
- # xpo-bmc-ami-devops (4)
- # xpo-cloudbees (5)
- # xpo-cockroachlabs (2)
- # xpo-github-for-enterprises (7)
- # xpo-gitlab-the-one-devops-platform (7)
- # xpo-itrevolution (1)
- # xpo-launchdarkly (10)
- # xpo-lightstep-observability-incidentresponse (2)
- # xpo-linearb-automate-dev-team-improvement (5)
- # xpo-planview-tasktop (2)
- # xpo-snyk (6)
- # xpo-sonatype (2)
- # xpo-split (5)
- # xpo-stackoverflowforteams (5)
- # xpo-synopsys-sig (5)
- # xpo-tricentis-continuous-testing (8)
So excited to air your amazing story of NHS Wales and COVID vaccination this morning, Dr. Cunningham! 🎉
Good morning everyone! We just turned on the countdown stream — program begins at top of the hour — I’m super excited to share the program we’ve put together! 🎉
This was a fascinating set of data to pull from the speaker titles from Video Library…
PS: Generating that table of titles was surprisingly difficult — converting all the titles to ordinal numbers that could be sorted wasn’t as straightforward as I thought it would be!
The third talk that has a CEO video thanking them will be revealed later today. 🙂
This year, we have the highest percentage of talks come in the Call for Speakers!
(Well, Dr. Richard Cook isn’t a Ph.D — he’s “only” a medical doctor. 🙂 cc @annemarie.cunningham 😆
Congratulations @scott.prugh 🙂 you are inspirations to so many of us.
Love letter by @genek https://itrevolution.com/love-letter-to-conferences/
For your reference: our online Slack archive of this instance: https://devopsenterprise-archive.itrevolution.com/
Love this @genek - It has great quotes, experiences and lessons - Golden nuggets
A genuine puzzle / decision we’ll be pondering: to what extent should we try to recreate the amazing interaction that happens during general session talks, when we return to live events? (Which will be in October!!!)
Super funny — I almost typed in \@ here, if you know what I mean…. realized my error before I hit Enter. 🙂
you don’t want to see @jeff.gallimore angry
A few already at the DockSide! i'll be there most of the day 😄 pop by for a chat! \o/
@eliza.kruszelnicka429 right here: https://members.itrevolution.com/free-ebooks
✨Let's welcome @gus.paul, here to start us off with his talk on Automated Change Management ✨
Hi @annp, I don't currently see this talk in the video archive. Will it be uploaded? All the best! Malte
Looking forward @gus.paul - Interesting topic for all DevOps Enthusiasts (& GRC folks) 🙂
Good morning, @gus.paul, Executive Director, Application Infrastructure! So true, I’ve never met anyone like Gus before — you’ll see why!!! It’s amazing. 🙏
> “The world was once free and full of possibility. Then came order, and after it, authority. Everything changed… until nothing changed at all…” > —from LEGO Movie
Morgan Stanley: now owns eTrade: “the original Robin Hood” 😂
literally put the FPGAin a box next to the NYSE box in the same rack
to be clear..controls good...interpretation / implemenation....there is more than one way to do it
“why do I need to get on a call every day and say ‘yes’ to everything, all the time? There had to be a better way.” 🙂
About 60% of our deployments are hands off/automated at this point. in terms of the actual push
@gus.paul what are the plans for the remaining 40%? and what are the primary obstacles for going hands off with those?
some of it is architectural...legacy applications etc. We have just started a modernisation program to help with that
1some of it is just socialisation and prioritisation with the remaining teams...they are very busy 🙂
1absolutely... in my head i am jsut going to fix this problem and go back to builidng and shipping code. as i say...10 years later...
One line of code 🙂 - I could relate well @gus.paul - is that an HTML line? (in my case it was)
One of the things that blew me way that I learned from @gus.paul was the sophistication of their dev tooling — it seems they were a decade ahead of the industry, very much like Google. The PM of Dev Productivity at Google said of their situation: “we built things that were miraculous. But we are now like the Elves in Lord of the Rings; the Age of Orcs and Man are coming.” (Put another way, another Googler said, “we are now like the Galapgos Islands — and we’re trying to get back to the mainland.“)
Right...we built our own CI/CD platofrm aroud Jenkins....we had to because of all our custom deployment endpoints
http://neugierig.org/software/blog/2018/09/typescript-at-google.html
I completely changed our Change process after reading the DevOps handbook 2nd ed, earlier this year. Moved from SNoW and into Azure DevOps, so much quicker for the engineers, but still compliant with ISO27001
Ironically we have our own custom (ancient) platform...we are moving towards ServiceNow as a golden source of data....but likely will wrap it with our automation APIs
Thank you @russell.o.evans - In complex enterprise with varying platforms - convergence is challenging. SNOW seems to ruling that atm.
SNow as system-of-record, to track changes (not approve them) is pretty fine
We find its just an easier conversation with external regulators at ht moment still to say we are using SNow
Most of our Pull Requests are in AzDO, and that is a pipeline of most of our changes. If we need an approval for something (InfoSec etc), we @ them in the discussion and they then approve there.
SNOW rarely accurate, hard to use (must be a requirement for CMDBs) and missing depth-first-search capability (graph database) for things like transitive dependency discovery (think SBOM). Development needed in this space IMHO.
100% agree @jason.cox - very topical; Everytime when we want to see the relationship - it is another 2-3 months exercise with 1000 of systems running
That's key, if it's hard/slow for engineers to do, they tend to not do it (or automate it) but you can't automate CC
Neither am I. But the first book I wrote was called “Visible Ops”, based on create world class ops and security, based on ITIL processes. It was the best we had in 2004 to describe processes.
one of the (only) good things about our legacy tool - it is used for everything
Haha, I have a ISO27001 audit next week... any hints? 😅
“batching up changes, into larger and larger changes.” “we tended to push changes to the weekend.”
I was once an ITIL expert… developed software at IBM to help companies be ITIL compliant. Then I spent a few years distancing myself from that. #scarredforlife
Remind me to tell you about a remarkable conversation I had with @dean.leffingwell about something very similar, but around RUP!
I don't think ITIL itself is the problem but more how we implement the principles, isn'it it? :)🙂
Apparently it’s getting better. But back then, quite a few people called working with our software “Tivoli Hell”.
"suffer the same change process no matter the change" - same words from our developers mouths!
Yes!! So no to big centralized change batches! Localize the change approval!
How many have you got a senior director approving "code" they have never looked at?
turning aroud the risk assessment so that the system works it out for you, instead of you guestimating...this was a game changer
“we have all the history of deployments / changes, as well as their outcomes.” (they modeled what factors led to great / bad deployment outcomes!!) They flagged the low risk changes, so they could focus review effort on higher risk changes. Amazing.
we still expect manaul approvals in future....but not for the vast majority of changes
right. not quite standard change, because every software change slightly differnet
the other thing about using historical data like incidents is that the risk is dynamic
if you have a big incident your next change will be rated higher risk
“we backtested the model against 200K changes over the last two years.” 🤯
one advantage of being so big....our data is statistically significant 🙂
Data Driven Risk Assessment - Love it; Need to share this video with Service Management Team 🙂
What an amazing twist to this story: “some were convinced that human reviewers would out-perform an automated review process.” I’ve never heard this before. Startling, and yet, quite understandable!
i can imagine that analysis must have been fascinating. any big surprises in the correlations?
@gus.paul your "real" shift left is awesome. So I am (shameless plug) presenting at 3.20 today to talk about shift left security and the reason its titled "ludicrous mde" is because we just trusted the tool dashboards and NO manual approvals
as soon as we rolled out the risk assessment we saw people adjusting their deployment behaviour
1500 changes; not one change related incident. (!!!). Average is typically 1.5% (!!!) 🤯
Its amazing when you push accountability downwards . You get better results, speed,. accountability, performance and less risk!!
this slide is what sold our senior leadership....risk reduction not just speed
Wondering how we do automated risk assessment if you don't have a critical mass of changes (like less than 100)? I.e. not software development changes but patching of infrastructure and applications (i.e. SAP patches)?
good question...that is one of my asks on my last slide...its something we are about to start tackling
“it’s not about going fast; this is about reducing the risk”. Lovely. (telling story of fixing issue is minutes; “this is something that would sometimes would take weeks!“)
This does seem very reliant on an effective risk assessment process, did it take a long time to get that process where you needed it to be?
So you were building on foundations you already had rather than trying to get the whole thing in from the ground up?
very often, teams would be scared of waking up their remote MD to approve a change
Wow this is awesome! One of my challenges, although external (while working with auditors), has been to quantify the risk instead of just looking at if we broke the process. Never had success following that path. But so great to see that there are people who have had success. Makes me want to continue to believe 😄
@gus.paul - is this practice now fully adopted across all the teams?
IT's genreally available to all teams.. one of the requirements is your deployment has to be fully automated
with zero-days vuln in dependent libraries you must have some kind of expedite pipelines or you are putting your company at risk
I love how their standard pipeline was able to get code to production in less than 2 hours. With a standard pipeline that fast, there’s no need for the expedited pipeline.
there is a lot of variations in apps, some regression suites takes a lot more than 2 hours, so standard may not apply to those
I think this is why I said, “I’ve never met anyone like @gus.paul before” — so passionate about doing the right things in the right way, and also so able to be empathetic/respectful, and also irreverent and honest about shortcomings of status quo. So disarming, and effective!
@gus.paul Love the focus on reducing risk rather than increasing speed. Zero incidents - by measuring this have the teams gamed the system? Raising the threshold for what is regarded as an ‘incident’? How did you get over that?
the only real way to game the system would be to under report incidents. and incident reporting has it's own governance process
Brilliant - looking forward to the incident government process talk! Would like to know more
Specifically these are golden! How often are changes blocked over the assumptions/concerns that were proven wrong here... Great to see that you were able to get positive feedback!
I think I missed something because I had to go afk for a moment - is Systematic Changes a documented set of principles (outside of MS)? does anyone have a link?
we called it systematic change to make it distinct from Standard Change
We always hypothesized that speed and locality of change was key because: 1) people closer to the change understand the change best, 2) the system state is understood better now rather than later(system state decays with time), 3) putting accountability and responsibility with the same person/team creates true empowerment and allows system imporvement
if you know anyone at the Federal reserve or the OCC in the supervision departments let me know 😆
What a fantastic initiative to help spread these practices across financial services, and help get regulators onboard.
to be fair we have been talking to them and they & our audit teams are very supportive of this approach
🌟Next up, please welcome the team from Virgin Media O2 – @chanade.hemming, Head of Data Products and @richard.he, Head of Data Services 🌟
@gus.paul Great talk!! Nice work changing change and making people's lives more humane in IT!!
Great talk @gus.paul. We are having the same conversations at L&G. I will be sharing your talk with my team.
Welcome to @chanade.hemming, @richard.he - Looking forward to your session - Data is close to so many of DevOps enthusiast and change agent heart 🙂
Great talk @gus.paul - definitely keen to connect about your Programme in general and find out about regulatory quirks. Will explain in a 121 chat.
Fantastic @gus.paul to hear about changes in a highly regulated sector - lots to learn here for those of us in health space.
Thanks @gus.paul , DevOps is so no "some pipelines buried in the IT"... get Business on board
Great talk indeed @gus.paul, very insightful and surely resonates with me...
@chanade.hemming hinting at the vast technology landscapes created by vast mobile company mergers in the 1980s - 2000s; I’ve always been amazed at how even billing happens at companies like this in this US — stitched together from many mainframes…
“Blockbuster could have averted their [disastrous] fate had they taken their data seriously.” One of the things that didn’t make the conference was my Idealcast interview of Risto Siilasmaa, board chair of Nokia. Such an amazing interview, about similar themes. But no room in plenary schedule, because of all these great talks!! 🙂
@chanade.hemming - When you say Data Products - are you referring to Data Platform as Products? or the Data Assets as product?
Data Products - using data as the main ingredient. It's is a way of achieving an end goal through the primary use of data. can come in various forms such as; APIs - so could be a machine learning service providing the best product to recommend to a customer... Dashboards -visualising a data model e.g. using Looker Storytelling around statistical analysis in my team, it's predominantly APIs, so as an example, if you think about all the data we have available to us, we then use that to make recommendations, this is exposed as an API and teams/channels use this to show a product recommendation to a customer on the website, an agent in the call centre.
Thank you - makes sense; Appreciated @chanade.hemming
"Bring a legacy company, with a complex system into today's world" - 100%!
so interesting when you consider the company officially started in 2021. 🤯
Good reminder that the goal is the speed of innovation and improvement, not a set destination. A vector not a scalar. 😉
So interesting how often event sourcing / pubsub / Kafka has come up in this conference over the years.
vividly remember Scott Haven’s 2019 talk
https://videos.itrevolution.com/watch/485153969/ with slides and transcript
also in health https://youtu.be/nF7pzDjshrA - NerveCentre is fast developing electronic health care platform - their CEO and founder speaks here about use of eventbased architecture and Kafka
1@chanade.hemming lots of interest in using health data for prediction too - still a lot to be developed in this. Focus on stratifying population - looking at past needs and behaviours - and then using that to shape digital journeys (even if that is just where your phone call gets routed when you contact your GP practice) - Bristol were looking at this but I am not sure they got to delivery/
we've built something in the contact centre space, would be happy to talk about it with you if helpful! getting customers to the best agent for them/the problem
IMHO - Unlocking data in an enterprise is a real challenge in number of organizations; Data Liberty/Freedom
indeed - i believe we are going to see more and more data architecture and data enablement talks/topics as organizations mature
soooo many different patterns to access analytical data at an enterprise - emotionally draining
…and how event driven services enable decoupling services from each other.
sometime the modern architectures (Event Driven) are done by traditional architects - @genek - so you could imagine the output/outcome of those systems
dbt is one of the most amazing innovations that have come in the data space from the perspective of data consumers. you can finally apply common engineering practices to querying for data analytics.
So great to learn about this — I hadn’t heard of this until we recorded this talk — DBT and Cloud Composer were two amazing gems in this talk.
In how many organizations - still DBA/Data administrator control/approve data schema /data changes to even DEV region ?
i have seen a lot of teams now move towards a DBRE kind of a model where the DBRE is the approver because bad schema changes can break things. but they are vertically aligned to the product so they have to have the product context.
agree, but uptake from enterprise data community is either slow/no
We use Flyway DB - but we have not scaled that yet to rest of the organizations
curious how does Flyway is different from ORMs that generate and manage schema changes? How does it help?
we've got security as part of our wider team under the chief digital officer, so we embed those people into our teams when we're building new stuff/setting up platforms/tools etc.
(This part hurts: all the pain associated with complex chains of ETL jobs…. Cloud Composer is Apache Airflow, putting all of this into a DAG: like makefile for ETL jobs.) “saves 95% of effort that you don’t need to worry about anymore.”
“this isn’t about protecting so no one can use it; it’s labeling data so that people can use it, and be safe. protection and speed.”
@richard.he - this is brilliant; thanks for those explanations; Love this; need to explore further and re-watch this talk few time to completely unpack
Richard has a YouTube channel, you'll enjoy! https://www.youtube.com/watch?v=X8Ro4fdBdC8
Thank a lot @lbmkrishna glad you found it was useful 🙂 yup there was a lot of content packed in one slide 😅
@chanade.hemming @richard.he - My thoughts on data few years before I shared for your pleasure of reading and nodding 🙂 https://bmk.nz/2020/03/16/our-devops-journey-is-incomplete-without-data/
@richard.he & @chanade.hemming love the idea of self serve data. Have you got any tips about getting people and teams to be comfortable with being curious with data and build their dashboards?
Interested in this thread as well…. not an expert at all but step one was making data discoverable (similar to an API registry) has been a huge unlock within the teams I work with
@toli from my experience, I think one of the key things get people interested is lead by example, for example build a few things with sufficient value and quick to demo to get people interested, share with the whole department / teams to show what the power it. Then generally there is a lot of people follow.
The other thing is about communicating to making very clear it's ok to make mistakes the data platform team is there to make sure people don’t really make very bad mistakes (like protecting our data asset, by utilising service like VPC SC https://cloud.google.com/vpc-service-controls in Google cloud for example) then there will be more people willing to try things
Interesting! How closely do the data platform team work with the cross functional teams? @chanade.hemming said that you have cross functional teams with Data Engineers and Data Scientists embedded in them (which is awesome)
a related problem is understanding data sometime data sourcing from system A cannot be used as is, so there is a danger in letting people building business decision tools without guidance
@toli SO important, my love for analytics engineering has rocketed this last year 🚀 ❤️ and of course our people all over platform, the tools we choose etc!
@gvian we have many places where data lives in this company, one goal is - one safe place for data... <<< we're on a journey, so we have one source of truth!
“We can deliver value… without having to fill out a lot of forms.” 😂
Interestingly most of these forms were inherited in practice over the years and no one stood up and challenged those; Those who challenged made life better for them and others - We can deliver value - without filling out more forms/approvals
@chanade.hemming @richard.he - people like you make us to believe in our dream - Unlocking Data
@richard.he and @chanade.hemming: how do u setup a framework to ensure the data protection and policy is managed in self service setup ???
Hi @catchmebala the first step is sensitive data detection and tagging, that is absolute core of everything else (we use https://cloud.google.com/dlp to automate most of the process with some human input). Then you can have user groups defined based on the tier of these tagged. In other words by default there is no access to sensitive data. Obviously there are other things requires doing on categorising domain areas / business areas because not everyone should have access to everything even if they are not sensitive data. It’s a really long process and quite hard to get right but I have to say it is something so important and cannot be ignored
“we’ve achieved quite a lot; less handing off to other teams; delight of data scientist coding and deploying within hours; less forms; gaining trust, thru hearts and minds. “now people queuing at door; product recommendations; data democratization across every corner, getting comfortable with data driven decisions.”
“people queuing up at the door to do things with us, which is awesome and scary” - a product managers dream
“start small, release things, build trust.” common pattern in these journeys.
“I wish I could tell you more; the customer views it as a competitive advantage, which is good; it shows we’re creating value.” (YES!! 🎉 )
@richard.he the need to move from 'file transfer' to 'data streaming' is very real especially with legacy systems....great insights from your talk.
thank you Marcelo! There’s always going to be some files / batch based stuff but it’s very important to keep an event first mindset because when the data gets into things like analytical databases, it is no long useful for event driven use cases.
“when the CEO comes down and congratulates your team, it’s very exciting” 🎉 Congrats, @chanade.hemming @richard.he!
Brilliant one @chanade.hemming @richard.he - Thank you for sharing; It is inspiration to us. THANK YOU; Your CEO video is like icing on the cake - all your efforts - kudos to your team.
thanks loads! so glad you enjoyed it. this kind of feedback makes our work and sharing it the best!
@chanade.hemming @richard.he thanks for the great presentation and batch of information to think about :)
Congrats @chanade.hemming@richard.he: “We are on the way to upgrade the UK; you are leading the way!“: CEO, Lutz Schuler!
@chanade.hemming Yes I did! And it opened my view even more on continuously foster to get to the next level, until you get there... Until you are best. And your TEAM is! 🙂 So really amazing what you have done and achieved and I'm very happy to had a chance to see and hear it.
Thank you @chanade.hemming @richard.he you've opened my eyes to how data could be used with the right technology to transform legacy. Love the insight on using APIs and cloud technologies to join up the old and the new. Very insightful.
thank loads Moira, that means so much! happy to talk any time, here, LinkedIn, Twitter.. wherever suits!
Would love to connect @chanade.hemming - My talk is tomorrow 10.05am. Hope you get to listen to it too.
⭐ It's an honor to introduce @annemarie.cunningham, Associate Medical Director, Digital Health and Care Wales, here to talk about Supporting the Delivery of the COVID Vaccine in Wales ⭐
When I saw this tweet from @annemarie.cunningham, I had to learn what she was referring to — and it utterly blew my mind. So excited about this story! https://twitter.com/RealGeneKim/status/1484182211599294467
The first plenary talks (and slides) are already available in the Video Library for sharing! Automated Change Management - https://videos.itrevolution.com/watch/708122268/ Gus Paul, Executive Director Application Infrastructure, Morgan Stanley Bringing a Legacy Company Into Today’s World - https://videos.itrevolution.com/watch/708122289/ Chanade Hemming, Head of Data Products, Virgin Media O2 Richard He, Head of Data Services, Virgin Media O2
Motto of NHS Hack Days: “Making NHS IT Less Bad.” 😂 So good!!!
hello folks - I'm here - such an honour to be asked to speak about our work
“i moved from maverick to the mainstream… but there’s still some maverick there 😉.” :rolling_on_the_floor_laughing:
Summer 2020; UK government already purchasing millions of vaccine doses, even before approval… incredible mobilization, getting ready for largest vaccination program ever.
“There were many things we did not know; when would vaccines be available? which ones? can they be mixed, matched? who would be eligible and when?” (massive uncertainty)
This is incredible — in the US, scheduling systems were so problematic, most mass vaccination centers were just walk-in!
“Gill knew the team inside and out; she knew their strengths and weaknesses; she was trusted by everyone around her.” <---- one of the key insights from me in this talk. People are not cogs — quite the opposite. So good, @annemarie.cunningham.
Would love to hear more about that buy/build decision - so often see those go the other way because in many organisations the further away from the teams you get the less confidence the leadership seem to have in them
I love that Dr. @annemarie.cunningham is talking about using the http://GOV.UK Notify services that @mail832 presented on last year.
@annemarie.cunningham Can smell AI for scheduling 🙂 ... would be happy to learn a bit more if possible
@ben.grinnell the key part in the buy-build decision was that the programme delegated the advice on this to our organisation and partners - within our subgroup there was the trust that we could deliver. As Gill said it did give her quite a few sleepless nights in the week after!
Twitter thread on how @annemarie.cunningham and team hit the rate limits of Notify service for mailers — you might get a kick out of this, @scott.prugh! 🙂 https://twitter.com/RealGeneKim/status/1519308963728134145
> And it was so amazing to see what extent they used the amazing https://twitter.com/GOVUK > GDS Notify service, sending out 4MM letters, 15MM text message reminders, letting citizens know that vaccination appointments have been made for them. > > Go @JuliaFromIT and team!!!
@annemarie.cunningham Good t hear, that delegation is a great decision too rarely made - "move decisions to data"
“move authority to information, not information to authority”
Oh no. Reminds me of "just writing credit card numbers on paper."
(we’ll just enter it later… 😱 ). Wonderful work getting them onboard, @annemarie.cunningham!
COVID is probably the most striking use case for DevOps, being fast and reacting flexible.
true true true - we have to make sure that we don't forget the lessons we have learned
I@m sorry during my talk that I didn't manage to slip in @mik’s bring the work to the people, not the people to the work 🙂
love the acknowledgement of the bottleneck (supply) and maximizing usage
+100. @annemarie.cunningham’s description of all the unknowns was so riveting and incredible.
there are a lot of “we did something we hadn’t done before’s” in this story 🤯 🙌
well if we had done it all before then it wouldn't be a very interesting story 😅
Wow! Government starting with user needs first!! Way to go @annemarie.cunningham!!
that's it 🙂 - there is another story from Wales coming up later https://members.itrevolution.com/live/watch?track=1
@scott.prugh It's the first design principle https://www.gov.uk/guidance/government-design-principles And there's even a poster for it https://designnotes.blog.gov.uk/2019/09/18/new-design-principles-posters/ 😁 (I'm no longer working in digital government, but my love for those folks runs deep 😍)
slight variation on Digital Service Standards for Wales... https://digitalpublicservices.gov.wales/resources/digital-service-standards/
@annemarie.cunningham asking what was the real risk of doctors editing the vaccine record that they didn’t perform — “you could be sent to Siberia for this”, but courageous leaders made the right and expedient call. 🎉
this really is an extraordinary success story @annemarie.cunningham - it almost sounds like you achieved something impossible… I couldn’t even begin to imagine the amount of “hope generation” needed to make this happen
thank you! yes - lots of hard work by some very dedicated people - and not just in our organisation - we have people all across wales working in partnership with us
2nd-ed! DHCW has an amazingly dedicated team of people who genuinely care about the people they're supporting
thank you @ben.dodd - so very very proud of all my DHCW colleagues. I think we are so lucky in Wales to have such digital skill within the NHS (and out in the health boards too of course)
@annemarie.cunningham I don’t, sadly! I’ve only been here since Oct 2020 so if they left before then our paths may not have crossed
big issue - too many clicks. I see that at my GPs office Every Time.
(PS: there’s another amazing presentation coming from UK HMRC from @andrew.letherby and Caitlin Smith about how closely technologists worked with policy makers on Day 3! A wonderful theme.)
@vmshook it's great to be able to build the software and make these things work for people
Completely love how 'human-centric' and Customer Experienced focused this is. GP centric, Receptionist, End users being vaccinated etc. Way to go @annemarie.cunningham such an amazing achievement & success story
@annemarie.cunningham Amazing work. This also demonstrates really well that nearly everything is an IT process these days
You can buy it ... it is called the integration team 😄
this slide came from my friend Matt Stibbs talk at am InterOp summit at Google in London a few years ago - @genek ou were asking why is it so hard to send information around in health systems - Matt explains here https://youtu.be/mHWget8GobM
“You can’t buy a bottle of Interoperability” • @annemarie.cunningham
One way integration of data from one system to another — how hard can that be? 😆
I am very impressed with the resilience, positiveness, and go-to attitude presented at @annemarie.cunningham presentation. This is a much different face of the Government from what we used to see. Well done!
thank you - it was great to see the confidence put in our teams to deliver - and then that the delivery happened - this is how we build trust - competency and alignment 🙂
@annemarie.cunningham I’m dying to ask: what was the difficulty of creating that integration — going from “send out a CSV file every evening” to integrating those systems together. 🙏
the challenge was for the GP software company to receive the standard message and process it...
but your point on how hard can it be is a good one @genek - issue is that though we have a standard for messaging we don't have a standard for how recorded in the software and for suppliers to reengineer their (not always very modern) data architecture is a challenge
@genek in England the supplier that told us we will accept messages was using the CSV every evening solution in England (and they keep on using it there until very late last year) - but they would not offer it to us in Wales 😞 - it's what we still do with our other GP supplier who do not want to receive the data as a message.
What a challenge! Problems spanning multiple vendors. Nuff said! It’s complicated! 🙂 Thank you for all the amazing work you did — what an amazing and heroic tale!!
“CyclOps” where DevOps meets observability meets Greek Mythology
We have DevOps and ChatOps, and GitOps - someone really needs to invent CyclOps! 😄
“Wales is winning the global vaccination race” IT saving lives
@annemarie.cunningham: Thank you, your team and all involved for your work here! You all literally saved lives with your software! What could be more rewarding? Be proud of yourselves!
Thank you so much for telling us the amazing, heroic story of the NHS Wales vaccination program, Dr. @annemarie.cunningham!!! Utterly amazing.
If you would like to help us with NHS Hack Day (or UK Health Camp - getting going in London again this summer ) - or working with us in the UK Faculty of Clinical Informatics
incredible story @annemarie.cunningham, love the pivot you've made here personally and the impact you've had through software on people's lives! 👏 ❤️
Amazing story thanks @annemarie.cunningham should inspire many
Yes, On Thursday morning, and hosting a birds of a feature session in the afternoon on Sourcing of the IT Workforce
Supporting the Delivery of COVID Vaccine in Wales: The Welsh Immunisation System Dr. Anne Marie Cunningham, Digital Health and Care Wales, Associate Medical Director 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708122304/
✨ And now, a special message from Andy Nicklin, British Military. If you would like to help, you can reach him via email at: <mailto:andrew.nicklin977@mod.gov.uk|andrew.nicklin977@mod.gov.uk> ✨
6@jason.cox our lead developer is also Jason Cox - can't see him in the slack yet but maybe you can say hello to each other!
Some of the humanitarian problems being worked on: • How can we offer early warning systems to civilian populations at risk in Ukraine, so give guidance so they can seek safety • Coordinating humanitarian and refugee support resources for Ukranians • Supporting tracking evidence of atrocities happening so that it’s available to criminal courts, in a survivable way to support justice in the future
Sounds like an amazing opportunity to apply DevOps practices to public service!
To learn how you might be able to help with these humanitarian efforts, email <mailto:andrew.nicklin977@mod.gov.uk|andrew.nicklin977@mod.gov.uk> , maybe a short note saying “I’m in.” Thank you!!!
🌟 Next up, we welcome back @abd3721, author of "Mastering Salesforce DevOps", here to talk about DevOps for Salesforce and other Low-code Platforms 🌟
Hey there. Nice to meet you @gillian.nieboer
I'm so excited for this!! DevOps for Low Code is a huge problem of our times a SaaS and Low Code takes over more and more of our integrations.
Low code systems typically need to include a lot of hidden internal computation to make them flexible. They won’t be as fast as something that’s custom built to meet a single need
Low Code needs pro-code ideology.. still needs the engineering skills.. all the low code providers are still building capability to address security, scalability . Also still the governance of such platforms is at early infancy ... ..
The citizen developer is a major force now but it runs into the same problems we had before DevOps. Very interested how we can everything-as-code, migrate configuration and automate deployments.
https://www.linkedin.com/pulse/low-code-platform-darkside-balagopal-madhusoodhanan/
Who else wants to see large fluffy stuffed mascots at future DOES events?
❤️ Nuance - different talent pool for salesforce vs mainstream IT
The difference in the talent pool of low code vs. pro code platforms is massive. Can’t overstate how important it is to build tools that meet the expectations of that community
I totally understand. Not in the same context but I have spent a lot of time observing that often technologies fail because they are too early or too mature or too difficult because the creators did not understand how their target users are from what they understand them to be.
Anyone ever feel like they’re working on systems that are like Jenga towers?
The architect in me thinks that the low code movement might bring some risks to ops . While its very inclusive for non techies to be able to bring some rapid dev, they might not necessarily be able to think and support ops (it takes DOES for people in the conventional IT industry to do that). While it might eliminate the "developers-with-pagers" paradigm, I still wonder are we headed for some interesting times if the "70% of all new apps .." prediction comes true
I think that's what he's saying? https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1652182214925419
Low code SaaS systems will definitely change the landscape and challenges of maintaining systems
@abd3721 any tips on what to/not to do ? given the massive SFDC growth owing to the pandemic , even more important to do
I’ve got a ton of thoughts and would be happy to talk with you (or anyone else) at more length
Start with just understanding how different the challenges are: tech differences, skill differences, etc
“the software development lifecycle - how hard could it be?” :rolling_on_the_floor_laughing:
Hi @abd3721, I am a No-Code Dev ("Consultant") for Salesforce and I read your book until you start writing about using code. I was disappointed that you need to use code to handle DevOps for a low-code platform. I know there is Copado or Gearset to solve this... but the our journey with the goal to using repositories, unlocked packages, scratch orgs, test automations in a deployment pipieline etc. seems impossible to achieve 😞 at least doing it while also developing new business features
The challenges and limitations in managing this have created a market for commercial tools (like you point out) and is still under active exploration. The open source tools are not so well developed yet
“deployment errors” “missing dependent field” “malformed XML” “unknown error”
@abd3721 i am curious - how are environments replicated or how is pairity maintained in the salesforce world? tooling, etc.? is copado the answer?
Salesforce has some standard ways of replicating sandbox environments. But that’s a complete overwrite of the sandbox. You typically need to also move incremental changes to sync environments
“xml merge - how hard could it be?” :rolling_on_the_floor_laughing:
You’d be shocked how much time Salesforce people spend in XML hell
XML merge :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing: :rolling_on_the_floor_laughing:
Surely the bigger advantage for low code solutions is that folks with extremely good knowledge of their business domain can leverage their knowledge without having to get it translated through an IT/Dev/Tech team?
100%. The ability to allow someone who knows the problem to also be able to implement their own solution is a massive benefit
That efficiency makes up for most other inefficiencies
I didn’t realize how problematic XML merges were until we wrote DevOps Handbook — had to have Jez Humble explain it to me 10 times… Then it hit me…
I think the XML Merge problem is a good example of lack of modularity
but, hey, you also have one big monolithic database. so :man-shrugging:
It's exactly the opposite of the kind of separation of concerns one might get from a well=built language like Clojure 😉
Modularizing the XML into smaller chunks is one of the big innovations that Salesforce pushed out with their DX initiative. Commercial tools parse the XML and thus modularize it and ease the merge problem
The XML merge problem is more recently replaced by the JSON merge problem!! 😞
kid getting ready for school LOVED the merge gif...now thinks merging leads to death (the guy at the bottom of the pile gets crushed)
Thanks so much for the invitation @genek!
a big problem is integrating tools date: we use a lot of sdfx and we get some benefit out of it, but would love to get test data from Salesforce
Did you mean “integrating tools data”? Yes, you need a way to populate data in any test environments.
There’s an open source tool that you might try. https://github.com/forcedotcom/SFDX-Data-Move-Utility The ability to move test data is also something that an increasing number of the commercial tools offer. Copado for example has a Data Deploy tool
thanks but I am looking for a way to export test runs from Salesforce in a palatable format like JUnit XML so I can import into a pipeline like Jenkins or Azure DevOps
I haven’t used it for a while, but there’s this tool designed to pull Apex test run data into a JUnit format https://github.com/forcedotcom/ApexUnit
thanks a lot @abd3721 this is a recurring problem of low-code: you use a set of tools for anything from Cobol to Javascript... and a different set for each low-code platform and you lose visibility and, worse, automation of process (e.g. coordinate approvals for production releases)
Agreed. Low Code systems are a big step away from traditional tools, so the people who work on those systems don’t usually know what traditional tools/formats they “should” support, and in many cases those traditional tools haven’t even been built.
This was awesome @abd3721. My team is currently exploring Automation Anywhere on how to create risk assessments from procurement SRQs via screen recording. Very low code and automates a piece of our workflow that takes several hours a week. Initial results look promising and we are estimating we can use that low code automation for 80% of our risk assessments and use 20% for peer review and manual overview.
Nice! I’d categorize AutomationAnywhere as a Robotic Process Automation tool, which is also a fast growing trend. Powerful to be able to use such tools to automate existing workflows
Operational Support in a Competitive World Andy Nicklin, British Military 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708122318/ DevOps for Salesforce and Other Low-code Platforms Andrew Davis, Author, Mastering Salesforce DevOps 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708122330/
Thank you, everyone!!! See you back here for afternoon sessions — and I’ll see you undoubtedly before that in the track sessions!!! 🎉
Reminder: The breakout sessions are starting in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Reminder: The action has moved to the breakouts! Join the following channels to interact with speakers live while their talks air: #ask-the-speaker-track-1 #ask-the-speaker-track-2 #ask-the-speaker-track-3 #ask-the-speaker-track-4
Reminder: The breakout sessions are starting again in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
We are prepping for the afternoon plenary sessions — cyu in 12m! 🙂
Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
🔆 We are honored to introduce our next speakers, @brodie.kingstone, General Manager, Operating Model, Corporate Strategy/Customer Experience, ASB Bank, and @jafitzgerald, Partner - Consulting, Deloitte New Zealand. They will be talking about PACE – People Accelerating Customer Experience at ASB 🔆
ASB is 175 years old, and is New Zealand’s second largest bank…
“we were changing are BAU (business as usual), not our DNA”
“one of our goals: develop talent in the NZ market; for both ASB and Deloitte” — so neat!
“Five tribes; and homes of practices, such as Market, Risk, ….” <--- what a wonderful poetic way to describe what one might often call “functional areas of expertise (silos)”
I never thought about that being poetic @genek, so interesting to see what people pick up from language
Absolutely, @jafitzgerald — I think so much about how you and @brodie.kingstone talk about work and processes border on poetic. Move over, @jonathansmart1!!!
“ASB is in the top 10% of employee Net Promoter Score (eNPS) all banks globally” (!!)
a couple reasons. first, the toll the pandemic has taken on people… let’s make things better at work, at least. second, in the u.s., we are experiencing “the great resignation”. do what we can to retain employees better.
We work on a quarterly increment across the bank, so trying to deliver value each quarter. Releases for digital or tech are out the door when they’re ready - which can be more often!
ok thanks audio isn't that clear and the lack of visual does not help non-native speakers
i like the perspective that “sooner” doesn’t necessarily mean “faster through the pipe” — it includes delivering regular value to customers.
What an incredible program! “COVID: support finder; for all customers, what govt benefits to see available” “In 12 months, we identified $10MM of additional government benefits that our customers could claim, but didn’t know about!”
• customer obsessed • pace over perfection • learn it all vs. know it all • empowerment over control • trust and transparency So good!
“leaders went from people who had ‘teams below them’ to the ‘great unblockers’“.
Leaders being the “great unblockers” 🙏
It would be interesting to know the types of adopters in the ~1000 employees, early adopters, innovators, majority?
Our first pioneer tribe had a combination of people that were shoulder tapped for specific skills and people that were early adopters who opted in.
“PACE6: those 6 leaders most interested or most affected; we met 3 times per week” (a significant time commitment!)
“…and then PACE6 became PACE9…” (as people outside PACE6 wanted to start influencing/getting involved/not get left out.)
A dark moment from @jafitzgerald: “PACE went from #2 in Priority list to… #26.” 😆 😭 😱 “…and through lots of hard work, we got back to Priority #2.” (Only behind some regulatory issues!)
more important than doing daily work is the improvement in doing daily work
The persuasive argument that needed to be made: “PACE was the improvement that all projects would benefit from.”
I love the theme of “we needed to get moving” — reminds me of so many people saying this. “You can’t steer unless we’re moving.”
aligning with Senior Leaders on why PACE was important/critical for the business is an awesome mindset to have instead of it being “foundational or non-negotiable” - have mistakenly driven down the later path before
…another way to say “we needed to get moving” seems to be: “we need to work in a way that generates some feedback” [so we can steer].
One of the highest return practices: establishing a synchronized cadence across the bank. (!!!). How wonderful, @jafitzgerald!
We recently conducted a bankwide retro on what was working and where there were opportunities for improvements and consistently aligning the cadence for increment across PACE, and projects and teams was one of the top 4!
how to align funding for this… yes. lots of organizations wrestling with that challenge.
what i’ve heard works best for many is funding align to value streams/products. i think that’s what @brodie.kingstone and @jafitzgerald referenced for persistent funding, maybe?
Yes, spot on! That’s what we’re working on at the moment… hopefully giving our teams more certainty for their planning horizons.
I am very critical to aligning to the same cadence. Does not this defeat the idea of having the right speed for each endeavor?
I think we can conjure scenarios where cadences are such that it’s impossible to synchronize efforts quickly — two teams working at annual cycle could result in a 2 year lag, due to lack of opportunities to do handoffs
Our teams may have different sprints, but we’ve found great benefit in coming together at increment planning as we can remove impediments and sequence successfully across many different teams. That helps the change management for our internal people and external customers. I’m a big believer in learn from everyone but copy no one… so it might not work in other organisations or at a different time/maturity in ASB. But right now, it’s helping our flow and value delivery across the portfolio.
Ah, right. If groups don’t plan together, they can’t prioritize together! 🙂
It doesn't need to mean that the work speed changes within increments - but it greatly helps with visibility and alignment. The goal from our work was to ensure the Bank could pivot and respond earlier and having some alignment allows that to happen at a more organisational (or team of teams level)
As you have some way of PIP, do your teams use the same iteration interval? Are the teams able to switch iteration intervals?
We tackled similar changes using a "system" to create high alignment and loose coupling among teams. It is written up at https://www.infoq.com/articles/continuous-portfolio-management, if you wish to have a look
@vladyslav.ukis This is great stuff! Can you email me at <mailto:genek@itrevolution.com|genek@itrevolution.com>? I’d love to learn more!!
Customer goal of enabling their financial well-being spans all tribes across ASB.
@brodie.kingstone What an amazing story: “our parent bank, CBA in Australia is now adopting PACE principles” — influencing the mothership! 🎉 Congratulations on such an endorsement of what you’ve accomplished!
People Accelerating Customer Experience (PACE) Brodie Macdonald, General Manager, Operating Model, Corporate Strategy/Customer Experience, ABS Jane Fitzgerald, Partner - Consulting, Deloitte New Zealand 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708278597/
Thank you so much, @brodie.kingstone and @jafitzgerald!! Congrats on your fantastic achievements!!! 🎉
✨ Closing out our afternoon keynotes are @davidanderson393 and @mjj.oreilly, coauthors of the upcoming book, The Flywheel Effect, due out in November '22 ✨
2Here's a link to the book: https://itrevolution.com/the-flywheel-effect/
Learn more about @davidanderson393, @mjj.oreilly, and Mark McCann's upcoming book The Flywheel Effect (coming Nov. 2022) with forewords from Adrian Cockcroft and Simon Wardley! https://itrevolution.com/the-flywheel-effect/
Hello, @davidanderson393 and @mjj.oreilly!!! So delighted we can finally hear about all your achievements at Liberty Mutual, and what you’ve been up to since then!
“it’s like when you see a poker hand that will win” (but no one else believes you!) 😆
@mark.mccann and I are currently using this building Block diagram at our current company to lay in these foundations!
Thanks @genek - the amount of work I had to do to even get those case studies published external was another project within itself. We called it “external validation for internal influence”.
thoughtful principles. i imagine there is a lot behind them in practice.
Absolutely, it took us quite a while to distill these principles from working with lots of high performing teams working in this way
cost per call in a call center: drove costs from $20 -> $0.04 (!!)
i totally believe it. you have to accumulate the experiences of what worked and what didn’t, decipher why, and then codify it in the right words.
Umm, have you considered taking touch typing classes? Might be.. helpful?
Of course, several of the principles are influenced by other thought-leaders and by attending this very event. Standing on the shoulders of giants…
WebSphere: $50K/year annual cost; refactored to $20/year. (So good!)
“At one point, we had 4 AWS Heroes on my team; most companies don’t have that many in their entire org.”
When we talk about optimise the total cost, we are now enhancing that one to add in optimise the total carbon footprint...
TFW when Werner Vogels heaps praise on your work. 🙂 “I was in the front row cheering [one of my colleagues] on as he presented at AWS.”
“Serverless First” is very misunderstood. It doesn’t mean lambda, it means writing the least amount of code that you need to and using good architecture and engineering practices to offload operational burden to the cloud provider. BUT, many people hear lambda and functions. 🙂
“What happens after project, when consultants have been paid; when bonuses pool emptied?”
OK, all the engineers have cloud socks - are things any better….. But CLOUD SOCKS!
I love the conversation on differentiation of Modern Cloud and Legacy Cloud!
t-shirts here. I was giddy when DOES sponsors started offering shirts in sizes for petite gals.
3@davidanderson393 How did you balance decentralized / centralized (enterprise) architecture?
@davidanderson393 How did you balance decentralized / centralized (enterprise) architecture?
Great question - we enjoyed the term “enabling architect”. I ran the Enterprise Architecture team for a while in Liberty Mutual and each business had their own decentralised team. There was an ebb and flow between the central governance and evolving capability. In a nutshell, the central team govern the old and the teams evolve the new. One of my roles was to ensure they linked up and give the teams air cover to innovate and improve.
we favour enablement and empowerment over command and control. So we can bake a lot of the enterprise Architecture concerns in guardrails (Enabling constraints) into code/policies and into the templates and patterns we offer our teams. We then leverage the Well Architected framework and review process as well our SCORP process to decentralise architecture and design into the teams...
Thank you both very much for your answers and reading material, much appreciated! @davidanderson393 Is this an ongoing "new becomes old" handover from teams to central team?
And: do you do the same for commidities as well as core business functions? So: is the central team mainly governing the commodities?
I think the goverance picture changes per organization (RE: what commodities are governed). But (In My exp), central teams creating the new for other teams rarely works well. It’s better when the teams pioneer and the central team hardens.
“wardley maps help us decide what do we invest in, and what do specifically not invest in?”
i love the “outcomes” level of the map and where those are located. powerful perspective.
“there are no custom ‘good architecture practices’” 🙂 “all cloud providers have some, 8-9 years old”
“adopt a standard. don’t write your own. you’re not that different.” 🙌 :rolling_on_the_floor_laughing:
there is a strange almost unintuitive: by adopting standards, you unleash creativity.
The maps will facilitate a broad conversation on complex topics (across disciplines) and eventually lead to well informed discussions on strategy.
The Flywheel Effect Creates Space for Innovation David Anderson, Bazaarvoice, Technical Fellow Michael O’Reilly, Software Architect, Globalization Partners 🎉 Now available for sharing: https://videos.itrevolution.com/watch/708278624/ 📘 https://itrevolution.com/the-flywheel-effect/
Thank you, @davidanderson393 and @mjj.oreilly — so delighted we could finally learn about your incredible work at Liberty Mutual!!!
Thanks @genek - thanks again for all the support and giving us a chance to tell our story. Happy to give back to a community we have learned so much from!
I second that @davidanderson393 Thank you @genek everyone, for listening to our talk and for all the support!
(holy cow, thanks to the team for making available all talks/slides for Day 1. Amazing. 🙂
Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png
@davidanderson393 @mark.mccann In your Serverless First Org Strategy you mention that teams should optimize the total cost. Did teams have cost accountability and cost control on what they were doing?
Yes @z-itrevolution, we also drove a FinOps strategy - we had an org wide FinOps Dash board that showed the complete cost (cloud or on-prem) of an App. Endorsed by Finance and owned by Infrastructure. We managed cost centrally (by the budget owner) and we treated cost as an architectural concern (informed by Well-Architected framework). Another example of central gov and decentralised improvements. We encouraged and celebrated teams when they refactored to reduce the bill (Cost Optimization). Even joked when teams saved 0.10c a day (offered to buy back a coffee after 2 weeks with the profit - tongue in cheeek for a $40Bn company).
So actually managing a budget for a team - we found that excessive, but treated cost as a quality control.