Are you referring to alerting when tests fail?

We want to have that level of alerting.. we are doing some alerts on the absence of data to show when a monitor may be down....

Automation is the best way to handle this... Can't handle this manually

QA can see only the exceptions

Engineering the Digital Transformation, by Gary Gruver, has a section about Stabilizing a quality signal. The first step of it is automation.

It would be, are defects going up or down? how far to the right are they being discovered? How can they be moved to the left?

Awesome! Take down the silos and make it everyone’s responsibility 🙂

For the app teams - it is invisible (as I think it should be)

we are only asking them to improve their DevOps practices 🙂

Agreed

We're currently bouncing back and forth between the traditional ops which more aligns with this or moving to more empowered teams that fully decide and control their own infrastructure.

The traditional ops needs to be bolstered by dev and platform people

else the change is too hard

I personally believe that with the infra cost being less a concern than before, teams controlling their destiny will become more common

Exactly, I'm constantly fighting against arguments of how we have to control infra cost. Yet we spend millions in idle teams because they aren't empowered to do the things they need to do

I know!!

One of the things that I'm constantly finding is that much of the current tooling out there assumes you have a consistent and fairly greenfield environment. That hasn't been true at any large company I've ever worked at

Many of them also assume that you've fully bought into their ecosystem and the value isn't there unless you fully commit to it everywhere

@monika.gupta I think Ansible playbooks are great for Day1 ops (config and deployment) but more challenging for Day 2 (production issues) where the scope of a change and the specific conditions under which to apply it are more dynamic.

How do you avoid human errors in production ops?

It's a good problem to have.

Yes certainly. Automation from scanning and audit to a common central point is really key to the success of a good security process. Dragan's team do this in a big way and although he is not on the Slack today he can help with some of his experiences

Thanks, yes will try to sync with him on linkedin... or later on Slack... thanks..

Hi Dipesh - there are a number of different ways for determining false positives, including using ML to help. When it comes to governance it is more around how that information get communicated and leveraged. Ultimately we want to minimize and eliminate noise that ends up wasting time.

Hi Mark - there is a lot of interest in being able to test against "only what has changed" against a baseline, particularly for Dynamic testing so we do have implementations of incremental scanning that leverage a baseline.

Hi John, this is a great point. I have heard of this in some places, where there need to at least be security aspects included and evaluated as part of the design. But for most places it is more about trying to gain awareness and agreement. For Dragan's software security assurance program, there is more formality around design and also including architecture.

Getting a program that builds out security champions and collaboration between teams really helps. Especially where you have a strong security team that runs and shares best practices back. Creating a COE around the lessons from team to team

We also offer tooling that embeds into the IDE's that can educate as developers work

I like this one, falling behind that they were talking about it

Security Champions are a great thing - and especially when it is done by folks that are wanting to be a part of it vs being told "you're it". I have heard them called things like Security Ninjas too. The idea is having someone that understands development and how it works, being able to help bridge between security teams and development teams. And over time, they can even have ownership over things like triaging results, initial remediation efforts, etc.

totally agree. Great advice and it is what Dragan is mentioning in the presentation...

I don't like anything that encourages seperation, you have the dev team, and then the business team

I think this would work well for us. We use the concept of "guilds" at American Airlines. Perhaps a bit more work on marketing and training is appropriate.

The main point is to have them working together - you can't solve a siloed problem by creating another silo. 🙂

@robert.cuddy Agree. Do you see guilds as more silos?

If the guilds are made up by largely a single role then they certainly can be. If, however, they are pulling from across various part of the organization and truly acting as a bridge between them then that is a much better model.

In our organization access is provided based on the latest commit, and the manager gets access as well. The reports then get removed after two weeks. A new commit generates a new report.

I am a big believer that vulnerabilities should be fairly transparent and not made so secret. We are seeing a lot more organisations starting to outline their exposures they face and allowing others to learn...

Me too, but I've encountered resistance when proposing that detected vulnerabilities be shared with the teams who caused them (via development 😆 )

Wondered what others do to overcome that hesitance. I get the reluctance but also don't see a way to improve without some kind of sharing

Yes, I agree. As a developer it is really frustrating for me to do a commit on something old because a vulnerability got escalated and the person that made the last commit is out for whatever reason.

With a previous project, we had a small infosec team, that was tasked to support each Dev team, so we developed interaction. Depends on how many dev teams you are running. Can use the security champion aspect too.

yeah, we have +100 devlopers, but the DevOps team is about 30 based in US and India, so InfoSec group is still being built out with 4 US and 6 India.

30 teams in US and India or 30 people? We had four dev teams (10-15 people) each with one security rep assigned from our 4-person infosec team

30 people... before 1 team, now working it into 6 teams and looking at growing more people to fill out each team (work center) for better coverage.

We have a number of enablement practices around this. happy to have a chat if needed...

Hey John! Our most recent training was delivered through the org WeHackPurple, but I know we have used others as well. Happy to reach out to our Security folks if you'd like to know any others we have used.

Hi John - one of the really interesting things going on in this space today is trying to figure that out on a more granular level. Meaning, figuring out things like "what kinds of vulnerabilities do our teams typically struggle with" or "which of our applications are most vulnerable, and how" and then tailor training to meet those needs.

We also were able to tailor the content a bit as well for the sessions we've setup, so i'm sure you'll find the same thing to make it relevant for your groups

I'd be really interested in hearing anything you've found useful. We're looking to implement security champions but struggle with a number of issues, what training to provide is one of them

@rob.fraley - Tanya (WeHackPurple) is a great resource.

The other big one is how we transfer the trust from the existing security folks who don't have enough people to cover everything over to those security champions. I'm hoping a lot of the automation you presented about will help us get there

John - I will reach out to our folks over lunch and see what I can find out.

Level of training depends on what they need to do. I like swapping between having the infosec folks hang out with the champion on their team and then bringing them up to the security team. Sometimes it is as simple as exposure to a different context

One thing that has (sort-of) worked for us (at TrendMicro) is 'Trial by Fire'. We only have a couple of Security Architects, and although we want more Security Champions, often those folks that sign up, also have another job to do. So, we take one of those volunteers and introduce them during an Audit cycle - with the other security experts. They get a hands on idea of what auditors look for. This gives an appreciation for 'doing things better/differently' in the development teams. I'm a project manager that often has to bridge the gap between Security Audits and evidence gathering, and the Development teams. The less the development team sees my face asking for stuff the better - automate as much as possible.

Really good point @krista_mccredie - it's interesting how many things can get resolved and improved when people really understand how other teams work - and how what they do affects those teams (both upstream and downstream).

It can differ a bit from organisation to organisation. To be successful the process needs to evolve and improve after each cycle. So audit is as much about checking the process as it is looking for vulnerabilities. Thus the continuous security mantra

Thanks @colin.bell, can you tell me more about who defines the security standards and who polices/enforces those standards? I'm trying to understand the RACI a bit more

Are the champions are the hook to police the product teams?

Hi Scott - it will be a bit different for different organizations of course, but at a high level organizations typically want the centralized security group setting the standards - policies, rules, etc and having the authority needed for that. Your champions then would be helping to drive the importance and value of having that into product teams. They should also be communicating back to central security groups where bottlenecks, challenges and problems are arising (for example, when and how vulnerabilities are reported, timelines for fixes, etc). Ideally your product teams would each have a champion as part of them.

In terms of policing - they may be to some effect, but if that is how they are primarily seen, then I think it gets harder and the value of that role gets diminished. It should really be about raising awareness for better security practices and helping to implement them.

absolutely, that makes sense. the challenge that i've seen in many organizations is centralized groups becoming the bottleneck because they can't scale to meet the growing needs of the organization and they're hesitant to delegate out enforcement roles since you need to have trust there since you lose some "control" when doing that.

Champions acting as learning agents and advocates is a good first step, but that alone doesn't relieve the workload on the centralized security group

We all are! There are lots of good things happening in this space.

Really the concept is about creating a level of independence from the development team, so being an outside practice can work really well as long as the champions have some level of alignment to the teams. In essence it becomes like an audit team and that works really well for some organisations

One good one that I am seeing a lot of - partnering with a university and/or a local community. For example - in NYC they have the cyber hub building and in Los Angeles, there is an overarching community for all the public service orgs with an initiative to share knowledge and data. On the university level, there are a number of different schools that are looking to build out apprenticeship style programs that align with businesses to ensure the right skills are built. These can then become larger COEs.

thanks

This is about decoupling deployment from delivery. You deploy changes frequently, and enable a feature once things are complete. Maybe do incremental rollouts if possible

My two cents, deliver this scope by this time (project deadlines) would normally be tracked outside of OKR’s

OKR’s would be a little different…we want this key result…that result would then drive what we do

Yes fully agree, OKRs should not have dates but to fit within planning windows, ie, month, quarter, year.

@ganga.narayanan that is very common, unfortunately.

I think it comes down to “how fast do you want to learn and respond” OKRs that take a long time to realize slow learning and ultimately our ability to respond to changes around us

I agree, at the same there could be trending data needed to make sure you are not doing kneejerk reaction to every bump in the road. I think the longer OKR cycles could almost become retrospectives rather than real time adjustments points..

Yes, exactly, which is why it is so important to have Flow Time < 90 days as OKRs rely on sub quarterly feedback window.

I'm sure there will be many not-so-great examples of how it shouldn't be done in a casual "DevOps confessions" chat :)

@christina Add a bit more context to your Q?

Look at Throughput numbers vs. WIP amt and try to balance them.

I’ve personally found velocity is a good indicator of capacity for planning purposes…number of items a team completes in a given time frame (weekly or sprint etc) can be a good predictor of how many items can be planned for a simiilar timeframe. You can use this for stories in a sprint or features in a quarter or even epics in a year if you have that much historical data

I’m the same for bad OKRs! They are terrible!

B/C orgs are constrained by too much WIP.

Or overloaded role based silos 🙂 (Hi @dominica!)

Hi @jonathansmart1 :hugging_face:

Thank you! @mr.denver.martin

Hi @david.sanda 🙂

@mr.denver.martin We see VS's based on external, internal, and platform products - where we see demand between teams made visible.

Some good practices we see are based on reducing Flow Load to improve Flow Time and capacity (Flow Velocity)

Absolutely. With that said sometimes flow load is a work in progress and can be the result of items that are only within the influence of the product teams being affected

You could definitely set improvements on your flow metrics as part of your OKR mix…in that last example he shared how the OKR of “reduce flow time for features from 20 days to 10 days” showed value…that’s an easy way to account for flow issues just make those front and center in your OKRs

Similar you could have an OKR for “reduce flow load” or whatever flow metric is going to drive the most benefit in your situation

I guess the root my question is more one of culture. I think our teams do these types of things well, however, these flow metrics should be fairly visible and I struggle with how to ensure caring as these challenges are solved over longer periods of time.

Hi @genek :hugging_face:

Thanks!

@genek One of the key things I’ve seen Flow Metrics be effective for is highlighting the need for platform investments to help down cost structures that no longer make sense given the speed with which the business needs to move. But one thing I’m finding notably is cloud transformations that start with the assumption that infra cost reduction is the economic goal…

Focus on cost and the costs go up 💰

Hah - yes @mik is one speaker where there's no need to speed up the recording 🙂

😄 yup

Thanks @guillaume.bladier. And yes, will endeavor to slow down more!

I'm so excited to see @farleys’s session on Day 3! He is a great partner (and one of my favorite audit clients).

Typically there is distinct separation between first (dev and ops teams) and second lines (risk management); however the Institute of Internal Auditors recently published new guidance that allows reduced the separation between first and second lines. So in some instances risk management may indeed own some first line responsibilities.

Thank you!!!

That matches my personal experience, as owner of change controls in highly regulated industry historically. First line is under management control. CISO, Data Privacy, Compliance, AML, Fraud is part of the management control. Especially if report into CIO and or COO.

I’m not aware of any recent changes to the IPPF specific to DevOps. I’m also not aware of anything in the IPPF mandatory guidance that directly conflicts with DevOps. We’ve been experimenting with Agile Auditing, incorporating DevOps concepts into our audits, and we’ve been able to do so while still complying with the IPPF auditing standards. Often it is our interpretation of these standards that lead to conflicts with DevOps principles, rather than the standards themselves.

Precisely

That is old thinking. Auditors need to update their understanding of the risk as developers are using deployment tools to promote their code to prod.

Let's see if this highlight section answers some of those questions. Automated testing, pull requests, and automated deployments should cover some of this but there will need be manual checks.

@john.rowe fortunately your IT Audit team at BBY is aware of this (I happen to be one of the directors). 😆

It can if done properly with the right logs attached and leaves that paper trail.

Having pull requests or blame (I hate that git command naming) enabled along with a build and deployment pipeline means your code can't get changed between signoff and deployment and you reduce the chances of having someone either maliciously or negligently push a wrong commit to prod.

This wasn’t something that we had considered but it is a great question. We will definitely look a bit deeper into this. Thanks for the question!

I’ve been to a lot of auditor conferences — this might be the best talk I’ve seen on audit planning and fieldwork for technology. Have you presented this at IIA or ISACA events? It’s so good. cc @lucasc5 @lewir7

@genek - I would love to present at IIA and ISACA events on our agile auditing journey, particularly now that we've gotten more pilots under our belts. I'd also love to present on it at next year's DOES as well ;)

Let’s talk. I was thinking the same thing!!!

Patrick, I agree 100%! Words are important. I always ask for reviews, not approvals. It's the review that is key, not necessarily the "signature".

With my new team that I manage, I have made a few PRs with intentional mistakes, so I can demonstrate the humility of accepting feedback. At least I say all the mistakes were intentional 😄

Something that you might be able to do is use some automated code review tools (I know Informatica and some other tools have some) to make sure that secrets are parameterized, passwords are encrypted, naming conventions are followed for certain datatypes, etc. Building out that sort of autoreview and getting a "green light" upon completion can trigger another test, deployment, or secondary approver in your repo.

I recently took one of our EY cyber-audit people through the GitHub process of pushing a change to production. I think it was a life changing moment. Look even if I get an approval I can't promote my code if I have a dependency vulnerability, or I'm breaking our policies for AWS and allowing SSH access. It just isn't possible. Of course I can only check for what I know to check, but that is a surprisingly large range of things.

There are ways. Usually audit doesn't like the word "trust"! Not saying that in any negative way though.

No, but as I sent our IT audit team your video from 2016 to get them started on this journey I feel they get that it is trust but verify, Not just trust.

Some of these things although also comes down to that blanket policies to alleviate the use of word “trust” will not work, because that will slow you down. My beef with some of this is that we have to quantify the risk even if there is a deviation from a “recommended process”

I just found out that our audit and security teams had resolved an outstanding issue by recording all sessions when DBAs connect to DB hosts. As a video, that they manager then has to watch to "review" their work. The manager in question says his insomnia is completely cured.

To misquote a guy who told me about REGEXs... I had a problem. I thought about using a VLOOKUP to solve it. Now I have 2 problems

I work at a university now. Our storage is in the multi-petabyte category because we keep all research data forever incase we have to reproduce. But most of the time they can't figure out which data was actually used the first time anyway 😞

I feel seen. 🙂

I keep promising myself that when I start my PhD I'll do a better job of managing my research data...

Recently we started piloting an agile framework with our audits. We take a risk-based approach and talk through the scope of our work at the beginning of each Sprint. We also have daily stand-ups with our team and clients to talk through our questions. So far our sprints have been in 1 month stints, but the audit itself could take 3-4 sprints to get through the entirety of our scope. We have a final deliverable at the end of those sprints, but we're thinking about issuing reports at the end of each sprint as well. Let me know if you have any additional questions. I'd love to connect with you and talk a bit more if you're interested 🙂

@rusty.jones I am a director of IT Audit at Best Buy. I would be interested in learning more from your experience if you are willing to spend 20 mins with me. Thank you.

@manny.peterson I think you tagged the wrong Rusty 🙂

@manny.peterson Happy to connect! I'll send you a separate message and we can schedule some time to talk.

@rusty.jones oh darn fingers. My apologies. 🙂

Thank you! Cziksentmihaly's Flow was one of my favorites!

the very heart of the startup like intrinsic motivation

like 'out of body' feeling being in the flow

Exactly!

I’ve used “dojo”, which I read means “place of the way”.

Exactly! Same sentiment what we're doing! I should've referenced that

https://home.business.utah.edu/actme/7410/Nonaka%201998.pdf Thank you!

That bit is your story @amy.wm.cheng 🙂

AWWW thank you!

Exactly! We did lose momentum @jtf! Trying to keep it going 🙂

Better yet have one of the wonderful authors here support your Book Club Journey. Gene Kim, Mik Kersten etc.!! 🙂

We'd love for the authors to come and do guest presentations! 🙂 Mik did visit us 🙂 Looking to bring in authors to TELUS as speaking engagements soon!

Let me know if you end up reading Agile Conversations… 😄

yes and we'll have you attend our Book Club! 🙂

We'd love to! It's in my list! And maybe you could visit us @jtf?

It's a great book!

They will be available in the Video Library along with Ganga's presentation : )

Thank you! We used lean coffee formats where we'd bring some questions on what to try. Some of them led to opportunities with value streams for example. I like this model of three 20 min sections!

Thanks! It was something we evolved as an answer to the “now what?” question. 🙂 We made that part of the discussion.

Value Stream Mapping as an exercise has been gaining a lot of momentum. Even a quick exercise with the leadership, getting them to see what's happening end to end, and how much the lead time is can jolt them into action, in my experience!

So if the C-suite has a book club, are their books different from what the middle management folks reading? 🙂

TL;DR: In general, I don't think the middle managment thinks their lives have changed, so yes, the books are "different" ----- I've only recently tricked them into getting a groomed and tagged backlog of "all" the things that they think they need to "react" to. I think the middle layer thinks they don't need to be reading. The c-suite has read the Phoenix Project and 'edicted' certain working groups. Our projects are shifting from a traditional waterfall managment to more agile methods, but all agile projects have thrown out the traditional systems engineering V. The c-suite is happy with the movement to more agile methodologies (see their book club), however operators of the systems acquired do not want to give up the traditional waterfall gates and long testing periods before flipping the switch into operations. As a member of the more traditional Systems Engineering organization, we are caught between a rock and a hard place of trying to make everyone happy through enforcing of "standards" that projects and developers do not want to follow (see agile), that c-suite has given a pass on to follow, that systems engineering (us) must ensure are followed per regulation (FAR, security, etc) and operator (customer) wishes.

I can empathize. Probably not all too different from what we've seen sometimes! Good point about the disconnect between what C-suite is reading and envisioning and what is really happening on the ground!

(y)

Taking the build away from acquisitions is part of the problem If you don't see yourself as part of the team (at least from government side, and possibly corporate) you scale your participation down. If you are actively involved in bringing value, everyone benefits