Do you have additional platforms that deploy in conjunction with mainframe changes?

We have several. Typically it's WAS based that interact with mainframe.

but we've started using some IBM technologies to expose services on the mainframe as API's.

Our core mainframe system has to run on the mainframe and not an option to move to Git

are you all in the Como area?

Sorry, what's Como mean?

Columbia Missouri - I know they have a big IT shop there

I'm actually in Bloomington IL

That's our main IT shop

and we have 3 hubs in Dallas, Phoenix, and Atlanta

Love Bloomington corvette car show

I am now in Phoenix

I'd love to make it to Phoenix some day

come in Nov to Feb so it's not as hot - unless you like the heat

One thing we've been doing more is to focus collaboration on GitHub PRs/Issue and use the review/approcal process of a PR as the trigger to kickoff deployments PR Merge = Deploy

that's exactly what we've been trying to do

Okay. I think we are moving in the same direction. My team has one repo where a PR merge results in a deploy to an environment, with the explicit understanding that the review/approval means it's good to go.

Where do feature flags fit in the priority of needs on your transformation journey? Curious what would factor into your decision to build your own vs. look at third parties?

I'd be open to buying something to solve that problem vs rolling our own.

Would you say it's a short term or long term priority? This is my first DOES and I'm new to the DevOps space as well so had assumed most large enterprise companies who had undergone digital transformation were already implementing FF at least with basic on/off capabilities but I'm learning that isn't necessarily true so this has been a great learning experience.

Just a random comment, if you are looking for an opensource env, check for this, https://openapm.io/landscape, and opentelemetry is a standard, so, you could integrate with private tools like instana

We do have legacy batch Cobol. I can get you in contact with our ARchitect in charge of Mainframe Cobol if you want?

@x95castle1, that would be great. we are always looking on recommendations on how to move legacy forward. thanks

We actually deploy 100's of times a day to production at all times.

We do a very modest monthly release cycle, with the occasional weekly based on urgency. Better than what it used to be, but lots of room for improvement still. Thanks!

it's completely right, architecture and design matter. we need to have modern applications to get all the benefit from DevOps

Agreed! Parts of our solution + architecture are unable to avoid 3-5 hours of downtime for deployments. My gut tells me we should first strive for now down-time, then work on reducing the time it takes to deploy. Lots of improvements still, perhaps the simplest is simplifying our deployment process so that apps that don't require this downtime can be deployed separately outside of these monthly "ceremonies" we essentially perform

yes, the developers has the responsability for their test, their code and infrastructure

To production? Are code reviews manadatory?

yes, we included policies, gates and different practices for quality assurance in the pipeline 😄

2 years ago, we needed 18 days for test our ATM's, so we decided to include it in the pipeline

makes me think of Ikea. I love it.

We're looking to use Robots to test other devices like IVR, PAC, POS, etc.

Yes, we have ours “blessed” images in artifactory, and we block traffic to cloud registries like dockerhub

It's a decision from the dev team, accepting the risk for life or temporarily! You will see in the next part that Forces agent on the CI block or non-block according to your risk criteria. But the main point with Drills is that are real hackers and confirmed vulnerabilities.

we need to promote the communication between developers and hackers...and of course, security to shift left

Yes! Its same with the applications that store your vulnerabiltiies, what ever it is!

Thanks Matt

It is, at the end, devops teams should have ownership, and that is included accepting the risk themselves and confronting the implications with everyone

The terror of speaking in front of an audience is now replaced by the horror of watching yourself on video 😂

There are many IT organizations giving this same excuse "We're working on it for a release in Q4..."

Isn't the famous Larry Wall (creator of perl) quote - The three traits of every developer are Laziness, Hubris, and Procrastination

I consider a good procedure doc to be "automation lite". Often that's good enough. When it isn't enough, it becomes the spec that the engineer will need to write the automation.

I call that "Thursday"

Or 2am playing console games on a personal day when you get called into a bridge… ;-}

"4am Tom" is the target audience for the docs I write. He's a cool dude but not nearly as cool as "10am Tom".

with code creates a barrier to entry if people are accostumed to a Wiki?

But it keeps all the updates in one place instead of trying to make two different things match.

The Stackoverflow for Teams product has a Slack integration that notices someone asking a question and will say, "Would you like to ask that on SO4Teams?" and present a button that will post it for you.

didn't know there was this kind of integration for SO

8 years ago I developed a Firefox extension to rewrite Stack Overflow links to a local proxy to do stats on which questions and tags people did look for in the company

it was pretty cool but people were not that comfortable using it, it still was fun to understand what some of us were trying to learn

Yes! That's why writing in small batches is so important. If everyone is always doing small updates, you've covered the 70.

I haven't seen this video used in a talk but I saw the original when it came out and loved it.

i think i saw it as part of a keynote at devops pro moscow 2018

Good question. I'm not sure it is. Engineers love a challenge.

@tommd - I think this it is. But I would rather rephrase it to "make wrong boring" or "make wrong expensive".

What kind of maintenance?

Keeping them up-to-date, discoverable, make sure they're as testable as possible, etc.

Two things (1) create a culture of constant updates so that things don't get stale. People should "pull the anton cord" to update docs. Many places make it hard to update a doc.. you need approvals, readers don't have access to edit a doc so they have to file a bug. Instead, you have to make it friction-free to update docs. (2) Old docs need to be visible. Display the "last updated" date, change the color if a doc is old, etc.

We're working on it. https://stackoverflow.blog/2020/09/28/migration-wiki-documentation-articles/

Yeah, I hate that about our industry. People conflate learning the hard way with the joy of learning. People need guard rails, not a ... what's that thing where you have to go through many challenges? A confidence course?

Thank YOU for giving me the opportunity to speak!

It's true that for the most part, we started with teams, regardless of size.. most were setup in a "Scrum" like fashion, but managed many, many micro services

team alignment is something we are undertaking now, to better organize us around capabilities

To expand on the capability concept, we partnered with a business architect to break down tax preparation into business capabilities - expanding both our DIY and Assisted experience

Thanks!

Thanks for being a long time customer...I think you'll appreciate the experience investment you'll see as we mature on this journey

Looking forward to it. It's a great product. And excellent presentation - I'm impressed with how you approach all the changes you need to make each year and how fields cascade.

volunteers only.

Our pleasure! Thank you for allowing H&R Block to be participants in the 2020 DOES conference!

Wonderful to have you as presenters.

Hoping to be back next year and share more details of our transformation

We utlized industry guidelines to help us frame up how we should attack the problem. Taxes give us a unique situation, but not necessarily more than other in a heavily regulated industry.

We continue to challenge ourselves as well to not think of Block as being unique and to leverage industry guidelines

The necessary automation of that must be pretty intense. I’m in the insurance space, so I can relate to a degree.

Absolutely! It's a game changer for us given the late breaking regulatory changes that we have to support each tax season.

Any idea of roughly how many tests you need to run before a release?

Great question and while it bothers me to reply with 'it depends', it does hold true. For example, states and federal are broken out separately so it depends on which states and federal entities are included in a particular release. For a single state, we typically run 15-20 regression state returns and hundreds of test cases specific to the regulatory changes. We've shifted though to running the entire test suite in an automated fashion and it has expedited our turn around.

very interesting. Thanks for sharing!

Our findings were always paired with a suggested course of action. We wanted teams to take away something more than just shame. 🙂

🙂

Did the teams call you guys or were you assigned to teams based on some criteria - and if so which? (If you're going to answer in the presentation, I will wait 🙂 )

We were assigned to work with teams. At first, we tried to base this on a composite metric that tried to quantify and weigh impact to our end users. Shortly after we started operating, however, our executives basically said "Don't we know who the bad actors are? Don't we know which teams need help?"

We had to admit that, yes, we had a good idea of which teams already needed the most help, but we made them promise us to go back to eventually using a real data-focused approach.

so basically, our targets were chosen for us by R&D leadership

Okay... that makes sense to get the ball rolling. Did you eventually evolve to a data-driven approach to find teams that needed help (and with what)?

We did not. 😞

The experiment essentially ran its course -- I'll cover this in the section on "failures"

Okay, cool. Thanks for being candid about that 🙂

We basically kept a :shit: list, whose contents were curated by leaders within R&D. We were often deployed reactively -- teams moving up the list because they had recent quality issues that were relatively visible.

ok…thanks…ever get to a more proactive engagement??

I really wanted to. I sort of address this in the upcoming "failures" section.

👍

I’m not quite sure what you mean by “transversal scenarios”. Do you mean doing things like killing a cluster to test zone/region/cloud failover?

yes...

I think that it’s extremely valuable. I think a lot of teams don’t test it enough and usually have poor understanding of how long it actually takes.

I mean, AWS would tell you that it’s fast and easy. But until you do it a few times, it’s never going to be as smooth as you’ll need to be during an incident

All that said, I advise that teams start small. You have to do the basic, e.g. can your service automatically restart, before you try on the large scale.

of course, and they can say it...but we need to configure many things and its better to be prepared

I can only think of one that didn't love us afterwards. The short version is that we highlighted problem areas that the tech lead on the team got his ego wrapped in too much. It got pretty political.

We have a sort of hybrid prod/staging environment, and we've focused our chaos efforts there.

We use our own platform extensively internally in the course of our actual business, so we target this internal environment where real users are doing real business-value-add activities, but paying customers aren't in it.

makes sense

At Workiva we haven't unleashed chaos on paying customers yet. That's in the roadmap, but it may be a ways away. We're also lucky in that we have a sort of prod environment that real users are using, but that doesn't have paying customers on it. So we can do chaos in an environment that really closely mirrors our main prod environments without actually impacting paying customers.

I imagine this is the hardest sell. What senior leader thinks it's a good idea to randomly take down production? The sell must be around the value we get from that ... it makes us better, stronger, faster. And we practice in non-prod first. But still, there's a chance that taking down production won't be like taking down non-prod.

Yeah, I struggle enough, even with data and research, to get senior leadership to give things a chance that would be direct cost benefits... let alone invest in something that could cost money in revenue at the promise of making up for it in the long run.

@craig.larsen In working with our customers, the first step is to move away from the idea of random take downs and be very methodical and precise.

But also yes, practice in pre-prod/staging first. Build up confidence before going into prod

Makes sense. Do you find that you can build a lot of resiliency in pre-pod environments?

There's an industry risk axis that's important, too. I sort of generalize the ends of the spectrum as Netflix and NASA. If Netflix has a blip in prod, someone doesn't watch a show (sad). If NASA has a small blip in "prod", people can die and enormous amounts of capital and public trust go down the toilet. If you are NASA, you probably don't want chaos in "production", but you better damn sure be running chaos in as close to prod as you can.

LOL, yeah, makes sense. Lives vs money.

Well, thanks for sharing your story!

For Workiva, we deal with pre-release financial data for 70% of the fortune 500, and mistakes in the data we handle in prod could literally cost livelihoods and trust in our platform would evaporate.

So we go as close as we can.

Even before resiliency in pre-prod, most enterprises that I work with is just getting visibility. e.g. I’ve worked with teams and we spend a good chunk of early work using Chaos Engineering just to ensure their service is emitting useful metrics so they can simply know when it’s gone down/somethings gone wrong.

Yep. I've certainly been involved in monitoring systems, visualizing workflows, value streams, performance metrics, etc. I've never had the opportunity to play with a Chaos Monkey.

At the risk of sounding like a shill, Gremlin does have a http://gremlin.com/free if you ever want to try it. Though honestly, I started my Chaos Engineering work just using stress-ng and linux command line tools.

Good to know. If I get an opportunity I may check it out 🙂

Absolutely 💯 %

I think there's an implicit contract that doing so would make us friends, right? 🙂

I wish I could say "Yes". But I know myself too well 🙂