Fork me on GitHub
Ann Perry - IT Revolution16:10:21

🎉More on SREs with @monika.gupta and @ravinash presenting SRE From Scratch: An Enterprise Journey:tada:

Denver Martin, Dir DevSecOps, he/him16:10:22

@monika.gupta @ravinash would Continuous Testing be similar to Alert Monitoring? APM items...

Javier Magaña - Walmart16:10:00

Are you referring to alerting when tests fail?

Denver Martin, Dir DevSecOps, he/him17:10:43

We want to have that level of alerting.. we are doing some alerts on the absence of data to show when a monitor may be down....

Denver Martin, Dir DevSecOps, he/him16:10:02

Also, thank you for presenting this material...

Avinash Rao16:10:13

@mr.denver.martin, are you relating alert monitoring to CT

Avinash Rao16:10:27

For the platform?

Denver Martin, Dir DevSecOps, he/him16:10:50

yes, could those items be looked at the same way...

Ben Link16:10:48

@ravinash @monika.gupta What did the established (entrenched?) management think about introducing these principles? Was there hesitation to focus on these performance metrics and outcomes?

Avinash Rao16:10:02

Yes ; the outcomes are the same. Concepts like coverage, regression apply well

Avinash Rao16:10:21

Interestingly, we reached this point from a position of not of desperation, then at least under pressure :)

👍 1
Avinash Rao16:10:41

That made the status quo untenable

👍 2
Denver Martin, Dir DevSecOps, he/him16:10:21

I have been partnering with a QA team to help look at items that could be part of monitoring and that would could help provide some the data from real time tests... so they can work on the items that do not all into regular monitoring items...

Avinash Rao16:10:20

Automation is the best way to handle this... Can't handle this manually

👍 1
Avinash Rao16:10:35

QA can see only the exceptions

Javier Magaña - Walmart16:10:54

Engineering the Digital Transformation, by Gary Gruver, has a section about Stabilizing a quality signal. The first step of it is automation.

Javier Magaña - Walmart16:10:20

It would be, are defects going up or down? how far to the right are they being discovered? How can they be moved to the left?

👍 1
John Awesome Rowe - Best Buy16:10:23

How do you maintain your levels of funding? I always find that as soon as the pressure comes off, this sort of work is always on the chopping block because it isn't easy to tie directly to revenue

💯 4
Paninya Masrangsan (Tasktop)16:10:40

Were you also able to make a case upstream (e.g. for dev teams to prioritize debt work / refactoring to reduce issues)? Great work!

👍 1
Denver Martin, Dir DevSecOps, he/him16:10:19

@john.rowe I have seen that, too, and I often warn that the team is working on the next big thing...

Avinash Rao16:10:46

@john.rowe so far the pressure is still on :) SRE maturity journey does have some low hanging fruits that enable us to get credibility

Dian Hansen16:10:51

great info, loved the progression view, in particular the insight that better info revealed even more issues

❤️ 1
Monika Gupta16:10:53

yes @paninya.masrangsan - yes are able to align our dev team to what our users are after

🎉 1
Dian Hansen16:10:56

peeling the onion

Joyce Chidiadi16:10:09

Thank you for the presentation.

❤️ 1
Malcolm McAlpin16:10:56

Thank you, great presentation!

❤️ 1
Ashish Dubey, Software Engineer (CD), Grofers16:10:56

Great presentation. Thank you

❤️ 1
Avinash Rao16:10:58

@paninya.masrangsan that's the idea - we have a dashboard that puts pressure (in a non directive way) for upstream to get its act together

💯 1
Paninya Masrangsan (Tasktop)16:10:59

Awesome! Take down the silos and make it everyone’s responsibility 🙂

Avinash Rao16:10:10

Thank you for watching!

Monika Gupta16:10:26

@john.rowe - funding is never been a challenge as were able to align it with the improvements/benefits it is going to bring

Joyce Chidiadi16:10:27

I will like to learn more about how your team is working on the self-healing part the observability and monitoring as mentioned on your slides. @monika.gupta

John Awesome Rowe - Best Buy16:10:25

What is the expectations/needs you have for the application teams that you support? Is this all pretty much invisible to them or are you asking for them to do implementations or changes?

Avinash Rao16:10:32

For the app teams - it is invisible (as I think it should be)

Avinash Rao16:10:47

we are only asking them to improve their DevOps practices 🙂

John Awesome Rowe - Best Buy16:10:36

We're currently bouncing back and forth between the traditional ops which more aligns with this or moving to more empowered teams that fully decide and control their own infrastructure.

Avinash Rao16:10:31

The traditional ops needs to be bolstered by dev and platform people

Avinash Rao16:10:36

else the change is too hard

👏 1
Avinash Rao16:10:25

I personally believe that with the infra cost being less a concern than before, teams controlling their destiny will become more common

John Awesome Rowe - Best Buy16:10:03

Exactly, I'm constantly fighting against arguments of how we have to control infra cost. Yet we spend millions in idle teams because they aren't empowered to do the things they need to do

💯 1
Denver Martin, Dir DevSecOps, he/him16:10:29

@ravinash and @monika.gupta while 1 tool could not do everything did you find any tools you liked more than others?

John Awesome Rowe - Best Buy16:10:35

One of the things that I'm constantly finding is that much of the current tooling out there assumes you have a consistent and fairly greenfield environment. That hasn't been true at any large company I've ever worked at

John Awesome Rowe - Best Buy16:10:10

Many of them also assume that you've fully bought into their ecosystem and the value isn't there unless you fully commit to it everywhere

Denver Martin, Dir DevSecOps, he/him16:10:45

Great talk too.. thanks for presenting..

Monika Gupta16:10:58

@joyce.chidiadi - Self healing is bit manual today, team is running ansible playbooks to address some of Infra issues. we are looking to automate this through ServiceNow.

👍 1
Anurag (

@monika.gupta I think Ansible playbooks are great for Day1 ops (config and deployment) but more challenging for Day 2 (production issues) where the scope of a change and the specific conditions under which to apply it are more dynamic.

Jason Patterson16:10:12

@ravinash and @monika.gupta curious how did the central team handle on-call support and what the plan for it is as the SREs federate into app areas?

Monika Gupta16:10:07

On call support right now managed by L1/L2 team that we plan to setup in rotation

👍 1
Anurag (

How do you avoid human errors in production ops?

Avinash Rao16:10:11

the central support team is transitioning into the SRE team (we moved people from platform into the support - now SRE - team)

Monika Gupta16:10:44

@mr.denver.martin -Again focus is not on the tools, based on the use cases we adopt available enterrprise/OS tools like Dynatrace / ELK

Denver Martin, Dir DevSecOps, he/him16:10:15

Okay, yeah that makes sense... thank you.

Ann Perry - IT Revolution16:10:27

📣 Say hello to @colin.bell and @robert.cuddy with Making It Easier to do the Right Things: Govern, Measure and Audit DevSecOps 📣

👏 9
👋 2
Colin Bell16:10:31

Hi Folks

👋 2
Mark Peters16:10:27

Howdy, tracking speech and trying to follow other conversations, too much context switching

👍 2
Rob Cuddy - DevSecOps Evangelist17:10:03

It's a good problem to have.

Denver Martin, Dir DevSecOps, he/him16:10:48

Hi @colin.bell - We are working on developing automatic evidence gathering for compliance audit reporting, have you seen any work in this direction, like the reports are automatically added to Audit folders...

Colin Bell17:10:37

Yes certainly. Automation from scanning and audit to a common central point is really key to the success of a good security process. Dragan's team do this in a big way and although he is not on the Slack today he can help with some of his experiences

Denver Martin, Dir DevSecOps, he/him17:10:04

Thanks, yes will try to sync with him on linkedin... or later on Slack... thanks..

Mark Peters17:10:26

Shujenko? shujinko. something like that

Mark Peters17:10:37

Not all compliance formats but helps with some of the processes

Dipesh Bhatia17:10:46

How do we determine false positives - Governance can determine that ?

Rob Cuddy - DevSecOps Evangelist17:10:50

Hi Dipesh - there are a number of different ways for determining false positives, including using ML to help. When it comes to governance it is more around how that information get communicated and leveraged. Ultimately we want to minimize and eliminate noise that ends up wasting time.

👍 1
Mark Peters17:10:37

Question, do you do any diff management against baselines during all the testing steps? Or do you set new criteria in each phase

Rob Cuddy - DevSecOps Evangelist17:10:39

Hi Mark - there is a lot of interest in being able to test against "only what has changed" against a baseline, particularly for Dynamic testing so we do have implementations of incremental scanning that leverage a baseline.

👍 1
John Awesome Rowe - Best Buy17:10:43

How much of a gate is the Design & Plan stage? Do you require approvals before application teams start implementing? That's something I'm struggling with right now where security architecture reviews are preventing app teams from making progress

Rob Cuddy - DevSecOps Evangelist17:10:51

Hi John, this is a great point. I have heard of this in some places, where there need to at least be security aspects included and evaluated as part of the design. But for most places it is more about trying to gain awareness and agreement. For Dragan's software security assurance program, there is more formality around design and also including architecture.

🙌 1
Andrew Machen17:10:48

We have developers on our product teams that are energized by security but need a bit more hand-holding or support. Any good experience with DevSecOps guilds to build good practice evangelists within teams?

Colin Bell17:10:24

Getting a program that builds out security champions and collaboration between teams really helps. Especially where you have a strong security team that runs and shares best practices back. Creating a COE around the lessons from team to team

Colin Bell17:10:00

We also offer tooling that embeds into the IDE's that can educate as developers work

Rob Fraley17:10:11

@andrew.machen - one of the trends i've heard here today already and have seen around is the idea of Security Champions. Within our company, we have seen some early positive returns by setting up a group of those Developers and providing them training to then take back to their teams and be great resources for others on their teams. Our Security folks also keeps that group engaged by having bi-monthly meetings to talk about Security topics

👍 1
Mark Peters17:10:53

I like this one, falling behind that they were talking about it

Rob Cuddy - DevSecOps Evangelist17:10:06

Security Champions are a great thing - and especially when it is done by folks that are wanting to be a part of it vs being told "you're it". I have heard them called things like Security Ninjas too. The idea is having someone that understands development and how it works, being able to help bridge between security teams and development teams. And over time, they can even have ownership over things like triaging results, initial remediation efforts, etc.

Rob Fraley17:10:59

They call themselves the "Star Chamber" and have shirts and everything... 🙂

😆 1
👍 3
👏 1
Colin Bell17:10:02

totally agree. Great advice and it is what Dragan is mentioning in the presentation...

👍 1
Mark Peters17:10:15

I don't like anything that encourages seperation, you have the dev team, and then the business team

👏 1
Andrew Machen17:10:58

I think this would work well for us. We use the concept of "guilds" at American Airlines. Perhaps a bit more work on marketing and training is appropriate.

Rob Cuddy - DevSecOps Evangelist17:10:06

The main point is to have them working together - you can't solve a siloed problem by creating another silo. 🙂

👍 1
Andrew Machen17:10:34

@robert.cuddy Agree. Do you see guilds as more silos?

Rob Cuddy - DevSecOps Evangelist17:10:51

If the guilds are made up by largely a single role then they certainly can be. If, however, they are pulling from across various part of the organization and truly acting as a bridge between them then that is a much better model.

Ben Link17:10:23

How are vulnerabilities safely shared within the organization? Who gets to know what vulnerabilities have been detected, and how do we minimize the chance of bad actors discovering a list of "easy targets"?

Javier Magaña - Walmart17:10:09

In our organization access is provided based on the latest commit, and the manager gets access as well. The reports then get removed after two weeks. A new commit generates a new report.

👍 1
Colin Bell17:10:14

I am a big believer that vulnerabilities should be fairly transparent and not made so secret. We are seeing a lot more organisations starting to outline their exposures they face and allowing others to learn...

Ben Link17:10:21

Me too, but I've encountered resistance when proposing that detected vulnerabilities be shared with the teams who caused them (via development 😆 )

Ben Link17:10:53

Wondered what others do to overcome that hesitance. I get the reluctance but also don't see a way to improve without some kind of sharing

Javier Magaña - Walmart17:10:31

Yes, I agree. As a developer it is really frustrating for me to do a commit on something old because a vulnerability got escalated and the person that made the last commit is out for whatever reason.

Denver Martin, Dir DevSecOps, he/him17:10:37

I had a hard time finding DevSecOps engineers that were as strong in InfoSec as they were in DevOps, so I ended up splitting up the roles and having SEC focused engineers that are part of larger DevSecOps team and they are training up those other DevOps engineers..

👏 1
Mark Peters17:10:16

With a previous project, we had a small infosec team, that was tasked to support each Dev team, so we developed interaction. Depends on how many dev teams you are running. Can use the security champion aspect too.

Denver Martin, Dir DevSecOps, he/him17:10:00

yeah, we have +100 devlopers, but the DevOps team is about 30 based in US and India, so InfoSec group is still being built out with 4 US and 6 India.

Mark Peters18:10:48

30 teams in US and India or 30 people? We had four dev teams (10-15 people) each with one security rep assigned from our 4-person infosec team

Denver Martin, Dir DevSecOps, he/him17:10:33

30 people... before 1 team, now working it into 6 teams and looking at growing more people to fill out each team (work center) for better coverage.

John Awesome Rowe - Best Buy17:10:41

What sort of training do you give to the security champions?

Colin Bell17:10:31

We have a number of enablement practices around this. happy to have a chat if needed...

Rob Fraley17:10:06

Hey John! Our most recent training was delivered through the org WeHackPurple, but I know we have used others as well. Happy to reach out to our Security folks if you'd like to know any others we have used.

Rob Cuddy - DevSecOps Evangelist17:10:27

Hi John - one of the really interesting things going on in this space today is trying to figure that out on a more granular level. Meaning, figuring out things like "what kinds of vulnerabilities do our teams typically struggle with" or "which of our applications are most vulnerable, and how" and then tailor training to meet those needs.

👍 1
Rob Fraley17:10:49

We also were able to tailor the content a bit as well for the sessions we've setup, so i'm sure you'll find the same thing to make it relevant for your groups

John Awesome Rowe - Best Buy17:10:05

I'd be really interested in hearing anything you've found useful. We're looking to implement security champions but struggle with a number of issues, what training to provide is one of them

Rob Cuddy - DevSecOps Evangelist17:10:41

@rob.fraley - Tanya (WeHackPurple) is a great resource.

👏 1
👍 1
John Awesome Rowe - Best Buy17:10:12

The other big one is how we transfer the trust from the existing security folks who don't have enough people to cover everything over to those security champions. I'm hoping a lot of the automation you presented about will help us get there

👍 1
Rob Fraley17:10:31

John - I will reach out to our folks over lunch and see what I can find out.

🙏 1
Mark Peters17:10:11

Level of training depends on what they need to do. I like swapping between having the infosec folks hang out with the champion on their team and then bringing them up to the security team. Sometimes it is as simple as exposure to a different context

Krista McCredie17:10:45

One thing that has (sort-of) worked for us (at TrendMicro) is 'Trial by Fire'. We only have a couple of Security Architects, and although we want more Security Champions, often those folks that sign up, also have another job to do. So, we take one of those volunteers and introduce them during an Audit cycle - with the other security experts. They get a hands on idea of what auditors look for. This gives an appreciation for 'doing things better/differently' in the development teams. I'm a project manager that often has to bridge the gap between Security Audits and evidence gathering, and the Development teams. The less the development team sees my face asking for stuff the better - automate as much as possible.

Rob Cuddy - DevSecOps Evangelist17:10:11

Really good point @krista_mccredie - it's interesting how many things can get resolved and improved when people really understand how other teams work - and how what they do affects those teams (both upstream and downstream).

Scott Kellerman (DevEx Product Owner, Vanguard)17:10:27

In this model what is the role and responsibility break down between the centralized security group, the security champions, and the product teams?

Colin Bell17:10:58

It can differ a bit from organisation to organisation. To be successful the process needs to evolve and improve after each cycle. So audit is as much about checking the process as it is looking for vulnerabilities. Thus the continuous security mantra

Scott Kellerman (DevEx Product Owner, Vanguard)17:10:44

Thanks @colin.bell, can you tell me more about who defines the security standards and who polices/enforces those standards? I'm trying to understand the RACI a bit more

Scott Kellerman (DevEx Product Owner, Vanguard)17:10:28

Are the champions are the hook to police the product teams?

Rob Cuddy - DevSecOps Evangelist17:10:11

Hi Scott - it will be a bit different for different organizations of course, but at a high level organizations typically want the centralized security group setting the standards - policies, rules, etc and having the authority needed for that. Your champions then would be helping to drive the importance and value of having that into product teams. They should also be communicating back to central security groups where bottlenecks, challenges and problems are arising (for example, when and how vulnerabilities are reported, timelines for fixes, etc). Ideally your product teams would each have a champion as part of them.

Rob Cuddy - DevSecOps Evangelist17:10:58

In terms of policing - they may be to some effect, but if that is how they are primarily seen, then I think it gets harder and the value of that role gets diminished. It should really be about raising awareness for better security practices and helping to implement them.

Scott Kellerman (DevEx Product Owner, Vanguard)17:10:39

absolutely, that makes sense. the challenge that i've seen in many organizations is centralized groups becoming the bottleneck because they can't scale to meet the growing needs of the organization and they're hesitant to delegate out enforcement roles since you need to have trust there since you lose some "control" when doing that.

Scott Kellerman (DevEx Product Owner, Vanguard)17:10:53

Champions acting as learning agents and advocates is a good first step, but that alone doesn't relieve the workload on the centralized security group

Denver Martin, Dir DevSecOps, he/him17:10:36

Another place I worked that had a larger InfoSec team, they would embed some of their engineers into the different teams and they would be champions but also would learn the work the teams were doing and would pitch in and help those teams....

👍 1
Dipesh Bhatia17:10:17

Good Talk !! Thanks!! looking forward to research AI in Security of our Apps

🙂 1
👍 1
👏 1
Rob Cuddy - DevSecOps Evangelist17:10:59

We all are! There are lots of good things happening in this space.

Daniel Cahill - Engineer - Ontario Systems17:10:25

Do you have ways to build a center of excellence outside of your company?

Colin Bell17:10:37

Really the concept is about creating a level of independence from the development team, so being an outside practice can work really well as long as the champions have some level of alignment to the teams. In essence it becomes like an audit team and that works really well for some organisations

Rob Cuddy - DevSecOps Evangelist17:10:15

One good one that I am seeing a lot of - partnering with a university and/or a local community. For example - in NYC they have the cyber hub building and in Los Angeles, there is an overarching community for all the public service orgs with an initiative to share knowledge and data. On the university level, there are a number of different schools that are looking to build out apprenticeship style programs that align with businesses to ensure the right skills are built. These can then become larger COEs.

Denver Martin, Dir DevSecOps, he/him17:10:46

Great Talk!- @dragan.pleskonjic @colin.bell @robert.cuddy

👏 2
Mark Peters17:10:10

Thanks for the chat, always like security

👍 1
Lochan Vasudev17:10:47

Great talk. I have a question - How do the service organizations balance between frequent deployment vs client's expectation of not having frequent changes that may risk service delivery?

Javier Magaña - Walmart18:10:14

This is about decoupling deployment from delivery. You deploy changes frequently, and enable a feature once things are complete. Maybe do incremental rollouts if possible

Ann Perry - IT Revolution18:10:41

🌟 And now a presentation from @mikDevOps &amp; OKRs: From Micromanagement Misery to Finding Flow 🌟 Joining us for questions will be_ @dominica @danny.presten_ and @tom.wojtusik

👀 1
👏 2
👍 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:01

Yes! "Track outcomes vs. activities" @mik

👏 3
👍 1
Danny Presten18:10:18

Super excited about this message….I think it’s one so many organizations need to hear!

Tom Wojtusik (Tasktop)18:10:26

Flow metrics are just numbers if not put to action to work toward outcomes.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:53

"Everyone on the Product VS needs to understand the business results and outcomes"

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:37

OKRs can help you shift from P2P (from Project to Product)

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:29

1. Empower teams to set their own OKRs and move away from measuring project deadlines.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:04

2. Avoid falling into focusing only on cost center trap.

Christina Tan, Strategy at Blameless, Speaker DOES2118:10:49

Hey @mik, are project deadlines tracked via a different mechanisms then? (e.g. performance mgmt, etc.)

Danny Presten18:10:33

My two cents, deliver this scope by this time (project deadlines) would normally be tracked outside of OKR’s

Danny Presten18:10:08

OKR’s would be a little different…we want this key result…that result would then drive what we do

🙏 1
Mik Kersten (Project to Product, Tasktop)19:10:42

Yes fully agree, OKRs should not have dates but to fit within planning windows, ie, month, quarter, year.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:52

3. Beware of team telemetry (mttr, uptime, #of deploys/day) used at organizational level OKRs.

👏 1
Meghan Glass - PrdMgr Best Buy18:10:50

Improvement itself needs to be measured!

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:13

4. Learn from OKRs no less than 90 day cadence.

Ganga Narayanan18:10:21

Oh, I can so relate to the pitfall of team-level metrics! Every now and then metrics have a way of being weaponized and being gamed!

👍 1
Tom Wojtusik (Tasktop)18:10:51

@ganga.narayanan that is very common, unfortunately.

Denver Martin, Dir DevSecOps, he/him18:10:22

seems like the long feedback time would work for Big Bang and waterfall way of working, where Flow needs quicker almost daily standup feedback sessions...

Danny Presten18:10:36

I think it comes down to “how fast do you want to learn and respond” OKRs that take a long time to realize slow learning and ultimately our ability to respond to changes around us

Denver Martin, Dir DevSecOps, he/him18:10:35

I agree, at the same there could be trending data needed to make sure you are not doing kneejerk reaction to every bump in the road. I think the longer OKR cycles could almost become retrospectives rather than real time adjustments points..

Mik Kersten (Project to Product, Tasktop)19:10:57

Yes, exactly, which is why it is so important to have Flow Time < 90 days as OKRs rely on sub quarterly feedback window.

Denee (de-NAY) Ferguson - Director, Technology - Capital One (Speaker)18:10:40

I’d love to see some great examples and some not so great examples of how this should/shouldn’t be done

Ganga Narayanan18:10:39

I'm sure there will be many not-so-great examples of how it shouldn't be done in a casual "DevOps confessions" chat :)

Guillaume Bladier (He/Him)18:10:28

same here, OKR have always been a real struggle

Dave Burnison18:10:56

Slide say 8%, sounded like he said 80%

Christina Tan, Strategy at Blameless, Speaker DOES2118:10:18

to everyone here, How are the business metrics that shouldn't be gamed/weaponized/part of OKR tracked then?

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:14

@christina Add a bit more context to your Q?

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:33

80% canceled work after code written also has major hit on team moral.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:21

THIS! Communicate capacity of team - avoid cognitive overload from too much WIP.

❤️ 1
Lucas Whaley18:10:19

How does everyone handle capacity planning with very short feedback cycles?

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:23

Look at Throughput numbers vs. WIP amt and try to balance them.

👍 1
Danny Presten18:10:44

I’ve personally found velocity is a good indicator of capacity for planning purposes…number of items a team completes in a given time frame (weekly or sprint etc) can be a good predictor of how many items can be planned for a simiilar timeframe. You can use this for stories in a sprint or features in a quarter or even epics in a year if you have that much historical data

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:32

"Improving daily work is more important than doing daily work. Add capacity to improve daily work". #TUP Ideal number 3 💪

❤️ 2
👏 2
Ganga Narayanan18:10:07

What flows in software delivery? Features, defects, risks, debts.. I never get tired of seeing this! 🙂 Always very relevant and very important!

Danny Presten18:10:56

I’m the same for bad OKRs! They are terrible!

🙌 1
Denver Martin, Dir DevSecOps, he/him18:10:02

When I started my new role DevSecOps was managed as just one big blob of WIP, one of my first action was to look and see how many value streams were in the Blob and then start to focus on improving each stream...

👏 2
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:13

Truth: "Overloaded VS's are endemic across large organizations" @mik

💡 1
Jon Smart [Sooner Safer Happier]18:10:00

Or overloaded role based silos 🙂 (Hi @dominica!)

Denver Martin, Dir DevSecOps, he/him18:10:14

I think it would be interesting to see how you can overlay SRE on Flow framework and value stream optimization...

Cheryl Crupi18:10:51

@mik these stats on disregarding capacity are stunning (and not surprising). Advice on the frank conversations with sponsors to prioritize improving daily work over doing daily work? (Love that!)

Denver Martin, Dir DevSecOps, he/him18:10:00

@dominica really loved your book too, I used it to help the teams I manage to build dash boards to show the WIP better and make the work more well known...

❤️ 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:08

"Unlock capacity by driving down debt." #daily improvements

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:10

Biz metrics tend to be a lagging indicators where as Flow Metrics can be leading indicators.

David - BBY18:10:00

Really good talk @mik / @dominica!

Tom Wojtusik (Tasktop)18:10:05

Organizational OKRs rather than functional OKRs.

Denver Martin, Dir DevSecOps, he/him18:10:20

@mik - have you seen where you could have many duplicate flow streams? like if you have 100s of developers and you divided them up into different pods - you give each a value stream of there own, and then have the streams work together as need or independent base on size of the request? so many small sprint teams then larger sprint team when needed.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:15

@mr.denver.martin We see VS's based on external, internal, and platform products - where we see demand between teams made visible.

Andrew Machen18:10:36

@mik Are there best practices for accounting for flow and bottlenecks when setting initial OKRs? I think this is a specific area of improvement for us, and a place where it's important to show caring and dig deeper at a leadership level.

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:08

Some good practices we see are based on reducing Flow Load to improve Flow Time and capacity (Flow Velocity)

Andrew Machen18:10:07

Absolutely. With that said sometimes flow load is a work in progress and can be the result of items that are only within the influence of the product teams being affected

Danny Presten18:10:32

You could definitely set improvements on your flow metrics as part of your OKR mix…in that last example he shared how the OKR of “reduce flow time for features from 20 days to 10 days” showed value…that’s an easy way to account for flow issues just make those front and center in your OKRs

Danny Presten18:10:10

Similar you could have an OKR for “reduce flow load” or whatever flow metric is going to drive the most benefit in your situation

Andrew Machen18:10:26

I guess the root my question is more one of culture. I think our teams do these types of things well, however, these flow metrics should be fairly visible and I struggle with how to ensure caring as these challenges are solved over longer periods of time.

Andrew Machen18:10:30

We are 3 years into OKRs and it's getting better every year, however not addressing this appropriately can quickly render good intentions for later in the year as obsolete.

👏 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:03

Focus on improving daily work to reduce tech debt.

👍 1
Gene Kim, ITREV, Program Chair18:10:14

I love the story of how you can address fundamental cost structure problems by highlighting it, and showing how it’s impeding business goals — super nice, @mik!

🎉 4
Mik Kersten (Project to Product, Tasktop)19:10:11

@genek One of the key things I’ve seen Flow Metrics be effective for is highlighting the need for platform investments to help down cost structures that no longer make sense given the speed with which the business needs to move. But one thing I’m finding notably is cloud transformations that start with the assumption that infra cost reduction is the economic goal…

Ferrix Hovi - Head of DevOps - Siili18:10:02

Of course there's that boat...

😁 1
Gene Kim, ITREV, Program Chair18:10:40

Thank you @mik @dominica and @cdeardo!!

Ann Perry - IT Revolution18:10:04

 Let's welcome the team from Nationwide, @lucasc5 @lewir7 and @culpe2 

👋 1
Gene Kim, ITREV, Program Chair18:10:08

Ooh, our favorite auditors from Nationwide! 🙂 @lewir7 @lucasc5 @culpe2 🎉

❤️ 3

Thank you tasktop team @mik @dominica

Guillaume Bladier (He/Him)18:10:48

thanks a lot @mik, was great and interesting, next time, though maybe breath more when you speak (but I get it was a short timeslot), you were so speady sometimes

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:26

Hah - yes @mik is one speaker where there's no need to speed up the recording 🙂

😆 1
Mik Kersten (Project to Product, Tasktop)19:10:17

Thanks @guillaume.bladier. And yes, will endeavor to slow down more!

Gene Kim, ITREV, Program Chair18:10:57

PS: I’m so delighted that @farleys will be speaking on Day 3 morning. It’s so great to see all the amazing work being done by the Nationwide team… …that you’re auditing. 🙂 😆

😂 2
❤️ 1
😁 1
Clarissa Lucas, IT Audit Leader18:10:34

I'm so excited to see @farleys’s session on Day 3! He is a great partner (and one of my favorite audit clients).

❤️ 1
Denver Martin, Dir DevSecOps, he/him18:10:22

"Nationwide is on the DevOps Side" 🙂

❤️ 1
🎉 2
🙏 1
Gene Kim, ITREV, Program Chair18:10:26

“Last year, we gave a presentation on how to pass your DevOps audits” — so awesome.

Denver Martin, Dir DevSecOps, he/him18:10:32

Now we just need to get Patten Manning and Brad Paisley to sign that...

Gene Kim, ITREV, Program Chair18:10:09

This morning, Chris Porter, CISO, Fannie Mae said something: “we own the first line” — is that unusual? I assumed that the first line usually were owned by Dev teams? Would love to hear how others distribute that ownership. Thank you!

Clarissa Lucas, IT Audit Leader19:10:56

Typically there is distinct separation between first (dev and ops teams) and second lines (risk management); however the Institute of Internal Auditors recently published new guidance that allows reduced the separation between first and second lines. So in some instances risk management may indeed own some first line responsibilities.

👍 1
Jon Smart [Sooner Safer Happier]19:10:11

That matches my personal experience, as owner of change controls in highly regulated industry historically. First line is under management control. CISO, Data Privacy, Compliance, AML, Fraud is part of the management control. Especially if report into CIO and or COO.


CISO's org is second line of defense. Dev/biz is first line.

👍 1
Denver Martin, Dir DevSecOps, he/him19:10:09

Individuals are the first line and the InfoSec, Audit, and Compliance teams support and enable the individuals to be successful...

👍 1
Gene Kim, ITREV, Program Chair19:10:36

Have the IIA IPPFs been updated with anything DevOps related? (IIA: Institute of Internal Auditors; IPPF: International Professional Practices Framework) PS: @lucasc5 laughed at me when said that IPPF is a book — apparently it’s now a PDF file, no paper books anymore. 🙂

😃 1
Clarissa Lucas, IT Audit Leader19:10:01

I’m not aware of any recent changes to the IPPF specific to DevOps. I’m also not aware of anything in the IPPF mandatory guidance that directly conflicts with DevOps. We’ve been experimenting with Agile Auditing, incorporating DevOps concepts into our audits, and we’ve been able to do so while still complying with the IPPF auditing standards. Often it is our interpretation of these standards that lead to conflicts with DevOps principles, rather than the standards themselves.

John Awesome Rowe - Best Buy19:10:05

I'm curious, and hopefully this is coming in the talk, how you avoid those separation of duties check from becoming blind rubber stamps. I see this all the time where we create this separation because of the rule but it doesn't actually prevent the things they are intended to prevent and just becomes a meaningless hindrance to deployment

Jason Cox - Disney19:10:46

Segregation of Duties always assumes the “operator” pushing the “developers” code into production, does detail code review, ensures it isn’t malicious, ensure all the dependent systems will not be impacted. That mystical thinking… it doesn’t happen. 🙂


That is old thinking. Auditors need to update their understanding of the risk as developers are using deployment tools to promote their code to prod.

❤️ 1
Ethan Culp19:10:09

Let's see if this highlight section answers some of those questions. Automated testing, pull requests, and automated deployments should cover some of this but there will need be manual checks.


@john.rowe fortunately your IT Audit team at BBY is aware of this (I happen to be one of the directors). 😆

👏 1
Topo Pal - Programming Committee Member19:10:00

Lot of improvements in this area @genek since 2015!

Gene Kim, ITREV, Program Chair19:10:28

So great to hear this! Kudos to all who helped make that happen, @topo.pal!

Gene Kim, ITREV, Program Chair19:10:51

Gosh. I think everyone who deals with auditors needs to send them a link to this talk! Okay, here’s the link you send them!

👍 4
Gene Kim, ITREV, Program Chair19:10:33


🎉 1
Sean D. Mack19:10:24

@genek Yes! Sending to IA now.

🎉 2
👏 2
Gene Kim, ITREV, Program Chair19:10:10

That should make y’all feel amazing, @lucasc5 @lewir7 @culpe2 cc @smack

Sean D. Mack19:10:12

@lucasc5 @lewir7 @culpe2 Does this mean that code review + automation effectively replaces SOD?

👀 1
Ethan Culp19:10:04

It can if done properly with the right logs attached and leaves that paper trail.

👍 1
Ethan Culp19:10:02

Having pull requests or blame (I hate that git command naming) enabled along with a build and deployment pipeline means your code can't get changed between signoff and deployment and you reduce the chances of having someone either maliciously or negligently push a wrong commit to prod.

Gene Kim, ITREV, Program Chair19:10:43

Wow, that would be something — having flaky automated “lottery machine” test suites catch the attention of auditors. Awesome.

Topo Pal - Programming Committee Member19:10:57

@lucasc5 @lewir7 @culpe2 did you consider the case where a team lead could turn off mandatory peer review on GitHub Pull Request and merge the PR?

Rusty Lewis, Specialist - IT Auditor19:10:23

This wasn’t something that we had considered but it is a great question. We will definitely look a bit deeper into this. Thanks for the question!

Ethan Culp19:10:57

We have some really great developers who are leading by example! We've learned so much by collaborating with our devs. It really has been a partnership with them teaching us and us teaching them. And because of that our IA teams are now much more agile.

Gene Kim, ITREV, Program Chair19:10:27

I’ve been to a lot of auditor conferences — this might be the best talk I’ve seen on audit planning and fieldwork for technology. Have you presented this at IIA or ISACA events? It’s so good. cc @lucasc5 @lewir7

Clarissa Lucas, IT Audit Leader19:10:25

@genek - I would love to present at IIA and ISACA events on our agile auditing journey, particularly now that we've gotten more pilots under our belts. I'd also love to present on it at next year's DOES as well ;)

🙌 1
Gene Kim, ITREV, Program Chair19:10:24

Let’s talk. I was thinking the same thing!!!

Patrick S. Kelso19:10:31

I have had some success in banking/finance using a pull request based workflow to replace SOD. A few hiccups along the way but it does work.

👍 1
Topo Pal - Programming Committee Member19:10:48

Turning on branch protection simply does not ensure peer review

Patrick S. Kelso19:10:34

My biggest challenge in many jobs is the framing of PR. If I had a dollar for every time I was asked "Can you approve my PR XYZ" instead of "Can you please review my PR XYZ"

Clarissa Lucas, IT Audit Leader19:10:56

Patrick, I agree 100%! Words are important. I always ask for reviews, not approvals. It's the review that is key, not necessarily the "signature".

Patrick S. Kelso19:10:17

With my new team that I manage, I have made a few PRs with intentional mistakes, so I can demonstrate the humility of accepting feedback. At least I say all the mistakes were intentional 😄

Ethan Culp19:10:47

Something that you might be able to do is use some automated code review tools (I know Informatica and some other tools have some) to make sure that secrets are parameterized, passwords are encrypted, naming conventions are followed for certain datatypes, etc. Building out that sort of autoreview and getting a "green light" upon completion can trigger another test, deployment, or secondary approver in your repo.

Patrick S. Kelso19:10:29

I recently took one of our EY cyber-audit people through the GitHub process of pushing a change to production. I think it was a life changing moment. Look even if I get an approval I can't promote my code if I have a dependency vulnerability, or I'm breaking our policies for AWS and allowing SSH access. It just isn't possible. Of course I can only check for what I know to check, but that is a surprisingly large range of things.

👍 2
Patrick S. Kelso19:10:12

@topo.pal we implemented SOD as the people who could change branch protection and those who couldn't. And since our git platform tracks all those changes we have the audit trail too. Ultimately at some point you need to trust your staff to not do the wrong things. You can't control for everything.

Topo Pal - Programming Committee Member19:10:31

There are ways. Usually audit doesn't like the word "trust"! Not saying that in any negative way though.

👍 1
Patrick S. Kelso19:10:15

No, but as I sent our IT audit team your video from 2016 to get them started on this journey I feel they get that it is trust but verify, Not just trust.

👍 1
Vaidik Kapoor (Speaker) - Technology Consultant19:10:53

Some of these things although also comes down to that blanket policies to alleviate the use of word “trust” will not work, because that will slow you down. My beef with some of this is that we have to quantify the risk even if there is a deviation from a “recommended process”

Patrick S. Kelso19:10:08

I just found out that our audit and security teams had resolved an outstanding issue by recording all sessions when DBAs connect to DB hosts. As a video, that they manager then has to watch to "review" their work. The manager in question says his insomnia is completely cured.

😂 1
Ben Link19:10:43

To misquote a guy who told me about REGEXs... I had a problem. I thought about using a VLOOKUP to solve it. Now I have 2 problems

😆 2
Gene Kim, ITREV, Program Chair19:10:54

I know of a multi-year research project that, when they wrote a book, couldn’t replicate some of the findings because it was spread out over tons of spreadsheets. My friend took days writing a program trying to replicate the result, hoping/praying they could replicate the result.

Patrick S. Kelso19:10:19

I work at a university now. Our storage is in the multi-petabyte category because we keep all research data forever incase we have to reproduce. But most of the time they can't figure out which data was actually used the first time anyway 😞

Patrick S. Kelso19:10:10

I keep promising myself that when I start my PhD I'll do a better job of managing my research data...

Ann Perry - IT Revolution19:10:24

🔆 We now welcome @ganga.narayanan from TELUS, presenting: The Shaping Forces of Transformation: Growing a Learning Organization 🔆


@lucasc5 how are your audit sprints structured? Do they occur within a fixed period of time? Do you issue some sort of deliverable at the end of each sprint?

Rusty Lewis, Specialist - IT Auditor19:10:38

Recently we started piloting an agile framework with our audits. We take a risk-based approach and talk through the scope of our work at the beginning of each Sprint. We also have daily stand-ups with our team and clients to talk through our questions. So far our sprints have been in 1 month stints, but the audit itself could take 3-4 sprints to get through the entirety of our scope. We have a final deliverable at the end of those sprints, but we're thinking about issuing reports at the end of each sprint as well. Let me know if you have any additional questions. I'd love to connect with you and talk a bit more if you're interested 🙂


@rusty.jones I am a director of IT Audit at Best Buy. I would be interested in learning more from your experience if you are willing to spend 20 mins with me. Thank you.

Rusty Jones19:10:13

@manny.peterson I think you tagged the wrong Rusty 🙂

Rusty Lewis, Specialist - IT Auditor19:10:37

@manny.peterson Happy to connect! I'll send you a separate message and we can schedule some time to talk.

❤️ 1

@rusty.jones oh darn fingers. My apologies. 🙂

👍 1
Topo Pal - Programming Committee Member19:10:08

Awesome talk @lucasc5 @lewir7 @culpe2

❤️ 3
Ganga Narayanan19:10:31

This is every DevOps/agilist's dream! 🙂 Thank you!

Ganga Narayanan19:10:44

Thank you @annp! Hello everyone! I feel like a time-traveler from the future watching my own talk and unable to change what I have said. :)

Jason Walker19:10:47

Great content today ... I remember having many whiteboard discussions with auditors where the basics of the PR flow match into what you would find in a traditional ITSM "Change Request" plus so much more. Thank you for the presentation!

👍 2
Gene Kim, ITREV, Program Chair19:10:49

Such an impressive work, @lucasc5 @lewir7 and @culpe2. Thank you!!!

👍 4
Ethan Culp19:10:11

Thank you so much for having us! Loved all the questions and interactions from everyone. Really challenging perspectives for us to consider as we move forward in our DevOps and Agile Auditing journey!

Zsolt Berend, Co-Author - Sooner Safer Happier19:10:12

great talk @ganga.narayanan great concept from Csikszentmihalyi

🙏 1
Ganga Narayanan19:10:41

Thank you! Cziksentmihaly's Flow was one of my favorites!

👍 1
Zsolt Berend, Co-Author - Sooner Safer Happier19:10:41

the very heart of the startup like intrinsic motivation

Zsolt Berend, Co-Author - Sooner Safer Happier19:10:03

like 'out of body' feeling being in the flow

Jeffrey Fredrick, Author-Agile Conversations19:10:12

I’ve used “dojo”, which I read means “place of the way”.

Ganga Narayanan19:10:46

Exactly! Same sentiment what we're doing! I should've referenced that

Jeffrey Fredrick, Author-Agile Conversations19:10:59

I ❤️ book clubs. Totally agree @ganga.narayanan. Great places to build community and understanding.

👍 1
Zsolt Berend, Co-Author - Sooner Safer Happier19:10:49

tacit and explicit knowledge - Michael Polanyi

💡 1
Jeffrey Fredrick, Author-Agile Conversations19:10:12

Neat model of “Ba”. Looking forward to reading the full paper.

🙏 2
👍 1
Zsolt Berend, Co-Author - Sooner Safer Happier19:10:33

CoPs and book clubs are great, other useful patterns: internal conferences, unconferences, awards

🙏 1
Amy Cheng - TELUS19:10:16

OMG @ganga.narayanan’s topic is totally resonating with me. My existential crisis was what started me on my Agile Evangelist journey!

Ganga Narayanan19:10:22

That bit is your story @amy.wm.cheng 🙂

❤️ 1
Jeffrey Fredrick, Author-Agile Conversations19:10:07

@ganga.narayanan: one problem I’ve had with book clubs is it is difficult for everyone to keep up. Have you had that problem? If so, did you adjust in any way to accomodate?

Ganga Narayanan19:10:53

Exactly! We did lose momentum @jtf! Trying to keep it going 🙂

Amy Cheng - TELUS19:10:46

Better yet have one of the wonderful authors here support your Book Club Journey. Gene Kim, Mik Kersten etc.!! 🙂

Ganga Narayanan19:10:33

We'd love for the authors to come and do guest presentations! 🙂 Mik did visit us 🙂 Looking to bring in authors to TELUS as speaking engagements soon!

❤️ 1
Jeffrey Fredrick, Author-Agile Conversations19:10:19

Let me know if you end up reading Agile Conversations😄

Amy Cheng - TELUS19:10:52

yes and we'll have you attend our Book Club! 🙂

👍 1
Ganga Narayanan19:10:07

We'd love to! It's in my list! And maybe you could visit us @jtf?

👍 1
Amy Cheng - TELUS19:10:12

Can't wait to read Sooner Safer Happier next!

👍 3
❤️ 2
ssh 2
Ganga Narayanan19:10:26

It's a great book!

🙏 3
John M19:10:07

where can we download the presentation slides?

Ann Perry - IT Revolution19:10:59

They will be available in the Video Library along with Ganga's presentation : )

👍 2
Jeffrey Fredrick, Author-Agile Conversations19:10:17

When we do discussion sessions internally we’ve adopted a model in three 20 minutes sections: 1. “what was the message?“, do we have a shared understanding of the meaning of the book/article/video? 2. “do we agree?“, where do we agree/disagree with the author 3. “what can we try?“, is there something specific we can experiment, inspired by the discussion material Do you ever get into specifics of what to try in your discussions @ganga.narayanan?

👍 2
🙏 1
Ganga Narayanan19:10:21

Thank you! We used lean coffee formats where we'd bring some questions on what to try. Some of them led to opportunities with value streams for example. I like this model of three 20 min sections!

Jeffrey Fredrick, Author-Agile Conversations19:10:40

Thanks! It was something we evolved as an answer to the “now what?” question. 🙂 We made that part of the discussion.

Jeffrey Fredrick, Author-Agile Conversations19:10:09

Thank you for sharing your experiences!

🙏 1

watching this section now - I've been trying to convince my org to do a value stream mapping exercise for over a year, however they can't stop 'reacting' to everything.


I would love to SecDevOps 'us' but as an acquisition organization, we don't actually 'build' anything, but could really take a SAFe Portfolio approach to managing all of our existing projects/work and treating the 'projects' as suppliers delivering our Systems Engineering Org 'value' - I'm curious as to how to do what is to the 'left' of the book clubs, value stream mapping, because our c-suite does have a book club, but there is disconnect at the middle managment layer.

👍 1
Ganga Narayanan20:10:00

Value Stream Mapping as an exercise has been gaining a lot of momentum. Even a quick exercise with the leadership, getting them to see what's happening end to end, and how much the lead time is can jolt them into action, in my experience!

Ganga Narayanan20:10:08

So if the C-suite has a book club, are their books different from what the middle management folks reading? 🙂


TL;DR: In general, I don't think the middle managment thinks their lives have changed, so yes, the books are "different" ----- I've only recently tricked them into getting a groomed and tagged backlog of "all" the things that they think they need to "react" to. I think the middle layer thinks they don't need to be reading. The c-suite has read the Phoenix Project and 'edicted' certain working groups. Our projects are shifting from a traditional waterfall managment to more agile methods, but all agile projects have thrown out the traditional systems engineering V. The c-suite is happy with the movement to more agile methodologies (see their book club), however operators of the systems acquired do not want to give up the traditional waterfall gates and long testing periods before flipping the switch into operations. As a member of the more traditional Systems Engineering organization, we are caught between a rock and a hard place of trying to make everyone happy through enforcing of "standards" that projects and developers do not want to follow (see agile), that c-suite has given a pass on to follow, that systems engineering (us) must ensure are followed per regulation (FAR, security, etc) and operator (customer) wishes.

👍 1
Ganga Narayanan20:10:32

I can empathize. Probably not all too different from what we've seen sometimes! Good point about the disconnect between what C-suite is reading and envisioning and what is really happening on the ground!

👍 1
Mark Peters13:10:37

Taking the build away from acquisitions is part of the problem If you don't see yourself as part of the team (at least from government side, and possibly corporate) you scale your participation down. If you are actively involved in bringing value, everyone benefits

👏 1

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees.