Fork me on GitHub
Mitesh (DOES Event Staff / Engineer at Gaiwan)11:10:00

Super excited for all the talks coming up this conference! 😊 🙌

👍 3
Nikki Mejer - Sonatype15:10:54

Looking forward to hearing @stephen talk! The Minefield of Open Source: Guidance for Staying Secure

Molly Coyne (Sponsorship Director / ITREV)16:10:26

Welcome @keith.puzey and @sujay.solomon from Broadcom's team for our next session's Q&A. Thank you #xpo-broadcom

Haendel Dorfeuille16:10:05

Is it me or No sound from track 4 ?

👍 2
Sujay Solomon16:10:11

hello everyone! super excited to be here and chat about test automation

Mark Fuller16:10:13

No sound for me either

Tom Coudyzer16:10:14

No sound here either

Ann Perry - IT Revolution16:10:34

We're working on the audio issues!

👍 12
Marcos Ferreira16:10:11

No sound on track 4

Juan Pablo Díaz Sidaras16:10:14

Hello Team! no audio on my side in Track 4

Juan Pablo Díaz Sidaras16:10:31

Track 1 works well, track 4 does not have audio

Dave Hogg (Leonardo)16:10:18


👏 1
Tom Coudyzer16:10:06

Sorted here as well!

Mark Peters16:10:24

Audio is back

Mark Peters16:10:34

Shame because this is the one I was looking forward too

Dave McNierney16:10:44

Audio is ok for me

Justin Abrahms (eBay)16:10:21

I love that diagram which shows application as being on a different plane than infrastructure.

Anurag (

Some years ago, when I was at AWS, I was chatting with Suresh Kumar, then the CIO at BNY Mellon. He pointed out that applications last longer than the underlying databases which in turn last longer than underlying OS which in turn last longer than the underlying hardware. It’s important to test accordingly.

Justin Abrahms (eBay)16:10:26

My video is at the highest quality setting 720p, but the slides are still a bit difficult to read. Would love it if we could get these slides.

👆 1
Ryan Taylor - Senior Geospatial Developer, GISinc16:10:25

Same here. I would also like to see more of the screen used for the presentation and less for the branding/margin.

Molly Coyne (Sponsorship Director / ITREV)16:10:11

Thanks for your notes. The slides will be available in the video library.

👍 1
Gene Kim, ITREV, Program Chair16:10:59

My video just cleared up, and I can see everything clearly now. 🎉

Scott Prugh (CTO - CSG) DOES Prog Committee16:10:02

I love measuring cycle time and looking at how manual testing affects that negatively and how automated testing affects that positively...

Sujay Solomon16:10:38

the key for me has been bringing ownership of quality (at all levels of the SDLC) into the dev teams. Automated testing has often been the carrot to get them to take on that ownership 🙂

👍 1
Garrin Ball - DevOps Leader - DDMI16:10:56

empowering developers is one thing, but there is something to be said for incentivizing/motivating them to do it as well.

✔️ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:35

Yes. This is the hardest part. I talk about a failed effort along those lines in How to Misuse and Abuse DORA Metrics after lunch.

Sujay Solomon16:10:48

i've had a few folks ask about measurable quality/policy gates they can be set to ensure ownership of quality within teams. I haven't really found a good answer for this. Traditionally, code coverage has been used but that often ends up being a checkmark rather than a true reflection of confidence in the change.

Bryan Finster - Defense Unicorns (Speaker)16:10:13

“I can think of lots of examples of measuring the wrong things. At one of my clients, they decided that they could improve the quality of their code by increasing the level of test-coverage. So, they began a project to institute the measurement, collect the data, and adopted a policy to encourage improved test-coverage. They set a target of “80% test coverage”. Then they used that measurement to incentivize their development teams, bonuses were tied to hitting targets in test-coverage. Guess what, they achieved their goal! Sometime later, they analyzed the tests that they had, and found that over 25% of their tests had no assertions in them at all. So, they had paid people on development teams, via their bonuses, to write tests that tested nothing at all.” Excerpt from “Modern Software Engineering” by David Farley

Bryan Finster - Defense Unicorns (Speaker)16:10:17

I saw similar outcomes at Walmart. It was frustrating to warn against it and have management shrug, do it anyway, and then be surprised I was correct.

Sujay Solomon16:10:55

the level of confidence gained from unit tests is fairly low ROI according to this model

Bryan Finster - Defense Unicorns (Speaker)16:10:15

Yes, we pushed this in the WM Testing Special Interest Group as well.

👍 1
Bryan Finster - Defense Unicorns (Speaker)16:10:48

I’ve always found this to be a better pattern. The key, however, is understanding Kent’s definition of “Integration Test”

💯 1
Bryan Finster - Defense Unicorns (Speaker)16:10:37

We created a glossary to establish a testing vocab.

Garrin Ball - DevOps Leader - DDMI16:10:46

would love to get my hands on that glossary 🙂

Sujay Solomon16:10:28

i'd be very interested in that too!

Haendel Dorfeuille16:10:11

Video resolution is not great for me :(

Dave McNierney16:10:50

Hoping the library version will be better

Jeff Hughes16:10:51

Please ask any questions of Keith and Sujay on this channel. Thanks!

Stephen Magill [Sonatype]16:10:59

Hi all, excited to be here!

Molly Coyne (Sponsorship Director / ITREV)16:10:03

Thank you Broadcom! A warm welcome to @stephen for our next session's Q&A. Thank you #xpo-sonatype!

Dave McNierney16:10:05

Nice presentation! Good reminder of the need to think cross-platform with testing, just like with apps

Stephen Magill [Sonatype]16:10:26

If you want to see the report ^^

👏 1
👍 1
Bryan Finster - Defense Unicorns (Speaker)16:10:32

Yay @stephen! Showing us how afraid we should be!! 😄

🙌 1
👍 1
👀 1
Stephen Magill [Sonatype]16:10:16

hey @chris.gallivan421!

Stephen Magill [Sonatype]16:10:00

haha. I try to sneak some positivity in there too 🙂

👏 1
Nicole Forsythe16:10:53

“It’s a large number, and it’s growing.”

Trac Bannon (Speaker)16:10:57

Amazing volume for JavaScript!

Bryan Finster - Defense Unicorns (Speaker)16:10:15

It would be interesting to create a pipeline warn gate around this…

Stephen Magill [Sonatype]16:10:35

@bryan.finster486: around usage? like warn if no one else is using this?

Bryan Finster - Defense Unicorns (Speaker)16:10:52

“Be aware, you and 3 opther people use this in the world”

Gene Kim, ITREV, Program Chair16:10:57

Noooo…. “Most popular projects have the most vulnerabilities.” 😆 @stephen

😁 1
👀 1
Stephen Magill [Sonatype]16:10:17

it’s the economics of security research…

Dominica DeGrandis, Author - Making Work Visible, Tasktop16:10:52

"90% least popular products are the least vulnerable"

Chris Gallivan (Tasktop)16:10:57

people like vulnerabilities?

Stephen Magill [Sonatype]16:10:16

don’t take this as advice to go use obscure projects. security through obscurity doesn’t work for software development any more than it does for confidentiality!

👏 2
👀 1
👍 1
Trac Bannon (Speaker)16:10:21

If it's popular, it's vulnerable because of the usage; many eyes on breaking and looking for the gaps.

👆 1
Jeremy Sechler16:10:48

But lots of eyes on fixes too.

Trac Bannon (Speaker)17:10:32

True... it's a yes and situation.

Jeremy Sechler17:10:26

Aren't they all 🙂

Jeremy Sechler17:10:53

Looking forward to your Data presentation this afternoon.

Trac Bannon (Speaker)16:10:39

@stephen -True dat! This is not about having obscure libraries or projects.

Stephen Magill [Sonatype]16:10:48

yep! security researchers focus on popular projects (as do “black hats”). but as we’ll see later, the best projects also make sure they’re pushing updates out that remediate these vulnerabilities.

Gene Kim, ITREV, Program Chair16:10:54

4MM dependency upgrades! 234K dependency versions.

Courtney Kissler16:10:57

I wish @stephen was singing his presentation 🎤

😆 2
Gene Kim, ITREV, Program Chair17:10:28

Only 0.3% of vulnerabilities don’t have a patch that remediates it. Whew.

👍 1
Stephen Magill [Sonatype]17:10:36

yeah, that’s the silver lining

Nicole Forsythe17:10:45

So what to do with the 7.7%?

Stephen Magill [Sonatype]17:10:09

make sure you’re monitoring security feeds / using tools / etc to notice when those need to be updated due to a disclosed vulnerability

❤️ 2
Bryan Finster - Defense Unicorns (Speaker)17:10:44

Canary builds with embedded scanning even if there is no feature change to the component.

Gene Kim, ITREV, Program Chair17:10:52

In other news, wasn’t there a big Struts vulnerability that affected Confluence users?

Stephen Magill [Sonatype]17:10:46

Gene — these graphs were made with Python 😄

👍 1
Gene Kim, ITREV, Program Chair17:10:03

They’re beautiful, @stephen!!

Gene Kim, ITREV, Program Chair17:10:19

Violin diagrams, right? 🙂

Stephen Magill [Sonatype]17:10:31

just wait for the MTTU graph that comes later — I spent way too much time on that one 🙂

🎉 2
😄 1
Stephen Magill [Sonatype]17:10:20

I think this is really interesting. People are, by and large, very focused on remediation. Less awareness of the importance of controlling what comes in.

Bryan Finster - Defense Unicorns (Speaker)17:10:44

Every-time I run create-react-app I cross my fingers.

Trac Bannon (Speaker)17:10:20

I did that this weekend again... and shazaam.

Trac Bannon (Speaker)17:10:47

Now just run npm and yarn on the same project and see if you can monk-it-up.

Trac Bannon (Speaker)17:10:53

And we are wondering why there are vulnerabilities? The ability to have fast access to something like create-react-app is both beautiful and damned scary.

Trac Bannon (Speaker)17:10:38

People are really unaware of what they are introducing into their local environments and lager, into shared environments. YIKES!

Bryan Finster - Defense Unicorns (Speaker)17:10:56

We had a process to help make this better at Walmart, but I wonder how many people will proxy NPM and Nexus in their artifact repository and scan the versions of dependencies people are using within the org.

Trac Bannon (Speaker)17:10:27

@bryan.finster486 - I had a similar process with Commonwealth of Pennsylvania; a 2-step process so did introduce directly to the developer desktop unless it was a sanbox situation.

Bryan Finster - Defense Unicorns (Speaker)17:10:12

We were dong it async, but could black-list something if it failed scan.

Trac Bannon (Speaker)17:10:24

"Industry is doing a pretty good job" with remediation.

Trac Bannon (Speaker)17:10:44

And not as much focus on suppliers.

Dominica DeGrandis, Author - Making Work Visible, Tasktop17:10:00

Being proactive re: dependencies - 1.Pay attention to Quality

Gene Kim, ITREV, Program Chair17:10:04

PS: I think I had my first code PR accepted earlier this year, updating a Vega/Vega-Lite upgrade inside of a Clojure library — was immensely proud of myself, @stephen I needed a diagram that could only be done in Vega v5. (@arne.brasseur., it was for Oz library)

👏 3
Stephen Magill [Sonatype]17:10:46

@genek: Popularity as a (misleading) metric makes another appearance in a slide or two.

Gene Kim, ITREV, Program Chair17:10:49

@stephen I can’t wait!! And wow, so worrisome! 🙂

Stephen Magill [Sonatype]17:10:57

oh maybe it’s not on here. let me find the graphic with popularity included…

Gene Kim, ITREV, Program Chair17:10:07

This is great, @stephen!!! 1.8x less likely to be vulnerable! (At any given point time?) High quality projects are 8x less likely to have breaking changes.

Stephen Magill [Sonatype]17:10:32

the red is bad (negative association)

Stephen Magill [Sonatype]17:10:50

so more popular projects were more likely to be vulnerable (for all the reasons we already discussed)

Trac Bannon (Speaker)17:10:00

Agree on update dependencies frequently and offer that we need teams to understand the purpose of the dependency.... what does it offer?

Stephen Magill [Sonatype]17:10:05

yes, teams should carefully evaluate / discuss each time they consider adding a new dependency.

Trac Bannon (Speaker)17:10:27

TRUE and yet, different teams are all at different points on their journey with some not being as knowledgeable ... and taking a bit of a black box mentality

Stephen Magill [Sonatype]17:10:48

yes. I’m not sure what the best answer is, but there must be some way to help these teams. what is the optimal feedback to provide to developers to help them make these choices?

Trac Bannon (Speaker)17:10:59

I tend to think having an architect or lead tech who has an evangelist mentality with a team can make a difference.

Trac Bannon (Speaker)17:10:35

The process guidance you give are excellent, @stephen

Gene Kim, ITREV, Program Chair17:10:12

That’s a great finding, @stephen! I love that MTTU can guide good component selection!

Stephen Magill [Sonatype]17:10:17

it turns out sourcerank includes a lot of popularity-type measures

Gene Kim, ITREV, Program Chair17:10:47

MTTU improving over time! (Super novel use of graphs, @stephen 🙂

Stephen Magill [Sonatype]17:10:20

thanks! that’s the one that took forever 🙂

Stephen Magill [Sonatype]17:10:48

super encouraging though to see the community-wide improvement in MTTU over time!

Gene Kim, ITREV, Program Chair17:10:56

Flipping awesome insight to split by year — such an improvement over that MTTU vs MTTR graph we did 2 years ago, which was depressing. OTOH, this is quite hopeful!!

Stephen Magill [Sonatype]17:10:51

yes! I really want to take a look at other ecosystems now and see if this is widespread or more Java-specific.

Gene Kim, ITREV, Program Chair17:10:34

Oh my…. is assertion that people don’t want to jump more than 1 or 2 versions? Is that a valid fear?

👀 3
Stephen Magill [Sonatype]17:10:01

yep. I wonder (speculating) whether this is due to fear of breaking changes.

Stephen Magill [Sonatype]17:10:32

1 or 2 versions forward feels safer, so they go to the closest non-vulnerable version rather than just getting fully up-to-date.

Gene Kim, ITREV, Program Chair17:10:33

Freaking beautiful! Spring updates!

Sanket Naik17:10:53

Would be interesting to see by industry vertical or some other interesting slice.

Stephen Magill [Sonatype]17:10:01

noted and good idea. I’ll see if we can provide some by-industry insights.

Gene Kim, ITREV, Program Chair17:10:49

“red: those are people that upgraded to vulnerable version.” Oof.

Gene Kim, ITREV, Program Chair17:10:17

Is there evidence that some people are pinning to latest?

Stephen Magill [Sonatype]17:10:27

good question. I don’t think we’ve looked specifically yet at what percentage of the population exhibits that behavior. I’ll take a look though — it’s an interesting question!

Stephen Magill [Sonatype]17:10:26

the “wave of red” is scary

Stephen Magill [Sonatype]17:10:40

relentlessly marching forward as new vulnerabilities are discovered

Gene Kim, ITREV, Program Chair17:10:05

Love this research, @stephen – kudos to you and team for it!!!!

☝️ 3
Stephen Magill [Sonatype]17:10:18

Thanks! As you might expect, Bruce was the superstar pulling so many things together for this 🙂

Gene Kim, ITREV, Program Chair17:10:19

Was thinking of you when I was editing my interview of Dr. Gail Murphy: She and I were discussing whether innovation happens because people are allowed to do lots of breaking changes, or if it happens because you don’t introduce breaking changes. Was absolutely fascinating —

Stephen Magill [Sonatype]17:10:16

I’ll have to go listen to that. I feel like there’s an inflection point. Early on, breaking things is good. Past some point it slows you down too much if things that should be “settled” are breaking all the time.

Sanket Naik17:10:59

Great talk @stephen!

Molly Coyne (Sponsorship Director / ITREV)17:10:15

Thank you @stephen and #xpo-sonatype!

Glenn Wilson, Author of DevSecOps17:10:07

I missed @stephen’s talk. But will definitely play this one back later. Looks like I missed a good talk

👍 1
Ally Corsetti17:10:48

Thank you for joining the Aqua and Anchore Vendordome!

Molly Coyne (Sponsorship Director / ITREV)17:10:53

Welcome @rani who will be moderating for today's VendorDome Q&A between #xpo-anchore-devsecops and #xpo-aqua-security-k8s @nurmi and @rory.mccune!;w=100%&amp;sidebar=yes&amp;bg=no#

🎉 3
Molly Coyne (Sponsorship Director / ITREV)17:10:06

Share any questions here that arise for you during this session and our speakers will address them right now during this LIVE session!

Virginia Laurenzano NSA/MARFORCYBER17:10:34

@emoshmosh - talking about EO on supply chains

Virginia Laurenzano NSA/MARFORCYBER17:10:40

to paraphrase: hard to marry regulation with cloud native development practices.

Jayson Henkel17:10:09

You mentioned, Codecov, why do you think that incident didn't receive a larger media coverage

Virginia Laurenzano NSA/MARFORCYBER17:10:45

speakers hope we'll resist the temptation to turn this into a paperwork exercise

Gene Kim, ITREV, Program Chair17:10:04

Was marveling at the impact of the latest Struts vulnerability — software supply chains is so relevant right now!

Kim Weins (Anchore)17:10:26

64% of enterprises have been impacted by a SW supply chain attack in the last year. Here's the data

Gene Kim, ITREV, Program Chair17:10:52

Do what degree are people not upgrading dependencies because of fear of breaking changes?

Kim Weins (Anchore)17:10:15

Good point. There is effort in upgrading.

Gene Kim, ITREV, Program Chair17:10:00

I’m seriously interested in what it would take for devs to upgrade dependencies, especially when there is a patched component available — now it’s more than detection, but actually remediation. I love the quote that the best way to stay secure is to just stay up the date.


Hello y’all. Looking forward to discussion in Slack and questions/feedback you may have!

👍 1
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:22

To some extent I think it’s a visibility issue. It’s harder to make dependency updates a prioritized backlog item with stakeholders when compared with the more visible impact of adding features.

Virginia Laurenzano NSA/MARFORCYBER17:10:59

plus, it's hard to plan for a lot of work where we can't show measurable gains - like, how can we quantify prevention?

💯 1
Frotz Faatuai (Cisco IT - he/him)17:10:01

Ug… Pet environments…

😢 1
👍 1
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:04

If we can move beyond automated dependency checking to automated remediation, via tools opening PRs to update dependencies we will be a lot better positioned. But we have to have robust automated testing to handle the fear that the upgrade will break something that the speakers touched on earlier.

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:04

If we can move beyond automated dependency checking to automated remediation, via tools opening PRs to update dependencies we will be a lot better positioned. But we have to have robust automated testing to handle the fear that the upgrade will break something that the speakers touched on earlier.


couldn’t agree more. dependabot workflows should be a force for good, code dependencies OR in automated testing activities!

👏 1
Lochan Vasudev17:10:21

Great talk. I have a question - How do the service organizations balance between frequent deployment vs client's expectation of not having frequent changes that may risk service delivery?

Kim Weins (Anchore)17:10:08

A number of successful supply chain attacks have come in through DevOps toolchains.

Kim Weins (Anchore)18:10:40

Layered security models = defense in depth

Gene Kim, ITREV, Program Chair18:10:15

Thank you so much, @nurmi @rory.mccune !!!

🎉 1
Daniel Nurmi18:10:53

Thank you all for attending, and having us on to discuss this important topic!

Molly Coyne (Sponsorship Director / ITREV)18:10:20

Welcome @dave.karow for our next session's Q&A Thank you #xpo-split

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:54

I’m ready to go… feel free to ask questions during the talk!

Garrin Ball - DevOps Leader - DDMI18:10:52

as part of Scaled Agile, we work at separating deploys from releases. I’ll admit though, this is the first time I’ve heard “Progressive Delivery”. I’m intrigued

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:28

Stay tuned… decoupling is key but there’s much more possible 🙂

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:40

In London, I got a question about whether progressive delivery was any different than CD. The foundational idea (decouple deploy from release) was in Jez Humble and Dave Farley’s CD book, but the practices of getting more fine grained about gradual releases and using data aligned to these gradual releases is where it gets even interesting.

❤️ 1
Dipesh Bhatia18:10:37

Can you post the link blog

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:06

For you code readers… yes, there’s a “bug” in the else if line here… should say “treatment == ”

Pedro Jordan18:10:22

this "flags" are in the same codebase , togheter on the same binary lets say ? what if an application crashess because of one on flag.

Dave Karow (Split - Sr. Progressive Delivery Advocate)19:10:07

@pedro.jordan I missed the second half of your question during the talk. The great thing about flags is that you can toggle the state remotely and instantaneously if an issue arises. That’s how you avoid needing a rollback/roll-forward deployment to resolve an unanticipated issue. Just toggle the flag and within milliseconds it’s off again.

Pedro Jordan19:10:34

that's brilliant ! thanks Dave.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:47

treatment just means code path… could be classic blue button / red button marketing test but could also be back-end code that gets executed.

Garrin Ball - DevOps Leader - DDMI18:10:16

i love flags, but you need your test automation on point, for both sides of the flag. Thats where I’ve seen struggle in the past

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:23

Yes… one binary with multiple possible execution paths that can be executed based on runtime decision a user/session at a time.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:32

Testing strategies are one of the details to work out. Being able to have a test get the desired flag state is the main strategy… the upside is being able to test new and old experience in same environment.

David Van Couvering - Senior Principal - eBay18:10:35

One thing our team struggles with is that multiple teams are rolling out features at the same time, and when our business metrics go negative, we can't figure out which change caused the problem. Any suggestions?

Pedro Jordan18:10:48

I have also used them and they are cool in some cases , but just wanted to see a full example of these "decouple" arch

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:19

@dvancouvering for sure… that’s where tying the metrics to actual flag decisions comes in. Next several slides will introduce the idea.

Keara Vu (IT Revolution)18:10:49

Love the point on all the ever-changing surroundings and their ability or inability to impact behavior!

Dipesh Bhatia18:10:12

How is weighted routing different than Canary?

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:33

Canary by containers exposes entire release to a segment of network traffic (hopefully sticky). % based splitting using flags is still a canary of sorts but is at the feature, not build/release level.

👍 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:38

@dvancouvering When multiple flags are being used, the key is to use a different seed for randomizing each. THAT makes what you are seeing now cancel out other flag influence.

David Van Couvering - Senior Principal - eBay18:10:45

OK, yea I think we actually do that. I guess the other problem is it can take a very long time to get statistically significant results. This means rollout can take a week as we gather the data for each phase of the rollout.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:47

Yep. There is a balancing act between test to learn and test to launch. The latter looks for bigger scarier signals and acts on them quickly. The former holds out longer to get defendable stats outcomes.

David Van Couvering - Senior Principal - eBay18:10:17

I like that "test to learn" vs "test to launch"

Dipesh Bhatia18:10:30

testing is in every step… I like weighted routing, 10 - 20 - 50 …


Gotta say @dave.karow you’re making it hard for me to pay attention to doing other work in the background 😄

Dave Karow (Split - Sr. Progressive Delivery Advocate)19:10:43

Ha! Just connected the dots that you were the next speaker. I used to live in your world not long ago… BlazeMeter, before that SOASTA, and before that Keynote LoadPro. Went from consultant led big-bang load testing to developer-led continuous testing.

Pedro Jordan18:10:07

thanks for the talk !

1️⃣ 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:14

I’ll hang here for one minute and then head over to #xpo-split for more Q&A

👏 1

That was very interesting. Not sure is applicable to my day job... but it does explain a lot about when you hear about a new feature being rolled out on FB or Google or whatever platform.

👍 2
Amri Abuseman - Flatiron Health, Director of Quality Engineering18:10:48

@dave.karow great presentation and this is in-line with what we're thinking on my team to decouple deploy from release

Amri Abuseman - Flatiron Health, Director of Quality Engineering18:10:12

Will you be able to share the slides from this presentation so I can share with my team as we plan for 2022?

Molly Coyne (Sponsorship Director / ITREV)18:10:31

Slides will be available in our video library!

Molly Coyne (Sponsorship Director / ITREV)18:10:54

Welcome @p.bruce for our next session's Q&A. Many thanks to #xpo-tricentis for their loyal sponsorship!

👋 1
Emily Hart18:10:57

🖥️ - with @p.bruce There’s no question that enterprises today want to further integrate continuous performance testing into automated pipelines. However, many are finding it difficult to reconcile the mismatched clock-speed of testing with today’s accelerated pace of development/delivery. Tune into Paul Bruce’s session to learn*,* among other things, the key steps to continuous performance testing in DevOps: • Gather the right metrics to assess your gaps Prioritize, then systematize across your application portfolio • Plan for acceleration across the whole delivery cycle Design concrete measurements with the end in mind • Pick the right targets to automate Make scripting easy for multiple teams Develop performance pipelines • Use dynamic infrastructure for test environments Ensure trustworthy go-no-go decisions Session is starting now in #here!


Last year, I published a bunch of blogs (and I think they did a bundle/paper) on continuous performance and load testing:

👍 3
👏 1

Also, I recently did another presy for EuroSTAR about how to move the performance mindset and practice forward:

👀 1
Ben Kruenegel18:10:41

loved the reference to particle physics!

❤️ 2
Benoit Devost19:10:36

"Superheroes in IT are single points of failure" This phrasing is just brilliant. Kudos @p.bruce for coining that.

❤️ 4
👏 1
🙏 1

If you like that one, check out some other thoughts I was able to put down a few months ago:


(not right now though, keep listening to the live-stream 😄 )

😅 1
Haendel Dorfeuille19:10:55

“It is not about testing everything”

💯 1
❤️ 1

Who has operational (i.e. performance) requirements in their work planning process?

Andrew Machen19:10:52

"Ending DevOps Holy Wars" - A book I would want to read 😛

❤️ 1

You may have heard people refer to performance as ‘non-functional’. Sorry, I call it OPERATIONAL. 😄

❤️ 2
👍 3
Meghan Glass - PrdMgr Best Buy19:10:02

Right? Who came up with that term--it's going to be non-functional if you don't account for it...

Ben Kruenegel19:10:02

it does seem most non-functional requirements are looked upon as optional; to add on to your comment, operational should not be optional

❤️ 2
👏 1

Thanks @p.bruce

Ben Kruenegel19:10:18

good session, thanks!

🙏 1

For anyone who wants to discuss continuous performance engineering more, I’ve bookmarked my calendly for this event at the top of the channel:

Alvin Crespo19:10:26

Fantastic talk @p.bruce! Thanks so much.

❤️ 1
Andy Nelson19:10:48

👏 nice work @p.bruce

🙏 1
Molly Coyne (Sponsorship Director / ITREV)19:10:57

Thank you @p.bruce and #xpo-tricentis!

🙏 1
Dr. Ivan Kronkvist19:10:04

@p.bruce I am a partice physicist


holy crap. i hope i didn’t ‘bend the laws of physics’ too much with my analogy. i fall asleep for weeks every year to ‘entanglement’ by Aczel 🙂

Dr. Ivan Kronkvist19:10:08

lol, not at all.

😌 1
Dr. Ivan Kronkvist19:10:04

Great presentation.

🙏 1
Dian Hansen19:10:09

Highlighted exactly why this remains hard and often outside scope of regular work - and the problems that causes. Thanks @p.bruce!

❤️ 1
Molly Coyne (Sponsorship Director / ITREV)19:10:15

👏:skin-tone-2:Please welcome @simon540 and Yash Kosaraju for our next session sponsored by #xpo-snyk!👏:skin-tone-2:

🎉 5
Dave Fugleberg19:10:21

do your security champions self-select? Do you have some kind of selection / vetting process?

Simon Maple19:10:14

Great question - there’s often a mix of approaches from company to company, largely based on whether orgs want full coverage across all teams/BUs or whether folks like to ensure that everyone is there because they want to be there. Both approaches have pros and cons, and largely depends on your needs as to which approach you go for

Sanket Naik19:10:05

@dfugleberg Coupa has a similar security champions program. The selection is a combination of nomination and endorsement based on interest and experience. After that there is an annual certification run by the Security team to keep the champions up to date. And there are ceremonies (akin to Agile) around the program like monthly meetups.

Simon Maple19:10:13

One of the most important parts in vetting is being really clear about what skills and knowledge a good champion should have. It’s more usual for a mismatch occurring because the role isn’t well defined.

santosh dhanawade19:10:46

how dynamic data is handled automated CI/CD in your case. for example, I wrote a test case with one version of data but once data got changed then my automated test case needs to change as well. how to handle this dynamic case in ci/cd?

Jayson Henkel19:10:50

Apart from STRIDE/DREAD, I'd be curious what peoples threat modelling processes look like..


Do you have some threshold (based on severity for found vulns, or potential blast radius, some patterns in IAC…) where you break the build? (eventually even in a pre commit) ? Or do you have a design which limits blast radius and would allow to let full accountability to the product team, just informing them, or some limit upon which it should block?

Molly Coyne (Sponsorship Director / ITREV)19:10:31

🌟Welcome @nurmi and @kim.weins for our next session's Q&A! Thank you #xpo-anchore-devsecops for your support of DOES! 🌟

Kim Weins (Anchore)19:10:04

oops, I guess they didn't take out the transition 😆

Rebecca Carter20:10:16

Interested in learning more about securing the software supply chain? Download the white paper

Kim Weins (Anchore)20:10:19

I'm curious if folks here are using multiple DevOps tools.....

👍 2
John Roesler - Sr Engineer (Gap Inc)20:10:08

yes, but there is one front runner and others are known to be on their way out / not preferred

👍 1
Gene Kim, ITREV, Program Chair20:10:59

“Teams are on average using 6 DevOps tools, involved in the CI/CD pipeline” — and often in enterprises, those 6 tools are all different! 😱

Gene Kim, ITREV, Program Chair20:10:24

Interestingly, this is something that @lucas.rettig and @levi.geinert500 had to tackle in 2018 at Target — I’ve always interpreted this as a backlash of decades of forced standardization of dev tooling in the prior years. 🙂

❤️ 6
Gene Kim, ITREV, Program Chair20:10:33

Confessions: I have more than a small degree of fear upgrading container base images.

Daniel Nurmi20:10:02

We do see quite a bit of variability within bigger orgs WRT dev/ops tooling, for a lot of reasons. A no uncommon one is when new teams / tech is brought in through acquisition, where along with team and tech also comes the entire dev. infrastructure and tooling as well 🙂

😂 1

Or siloed enough organizations at different stages of evolution where each silo has its own ecosystem… 😇

Kim Weins (Anchore)20:10:45

My experience is when you ask any IT/Dev person "What do you use for <tool category>?" Answer is inevitably "One of everything"

😂 2
Gene Kim, ITREV, Program Chair20:10:46

I saw a talk from a Director of Dev Productivity from Cisco talk about this — as a company driven by acquisitions, the task was given to him to migrate all the companies/divisions/tools onto a smaller groups of tools. It was a fantastic presentation — I’ve been wanting him to share it with DevOps Enterprise Summit, as I thought it would resonate. There is a cost to freedom vs. standardization.

Kim Weins (Anchore)20:10:30

Plus there is tool drift over time, eg our preferred tool from 3 years ago is different than today.

Kim Weins (Anchore)20:10:56

And the business units that are "exceptions"

Jayson Henkel20:10:45

@genek I think we need a track on how to be on the cutting edge of security inititatives. eg. SBOM, I've asked vendors to support Sub Resource Integrity (SRI), getting more visibility into their own security practices, breach notification etc. I often get push back..... any suggestions from anyone...other than just wait 😛

Gene Kim, ITREV, Program Chair20:10:25

Can you send me an email at <|>? Thanks!

Jon Sturdevant - Tech Advisor - BlueCross BlueShield of SC20:10:05

We have similar issues when we are asking our vendors to be responsible for the open source they include in their software. A lot of times their legal teams try to remove those words from our contracts with them. Most of the time the reason is "this isn't something we do"

Jayson Henkel20:10:01

@jonathon.sturdevant precisely, you also start feeling like the tinfoil hat fellow....

👽 1
Jayson Henkel20:10:38

Though I just had a vendor let me know we're 1 of 3 vendors who had asked to support there must be other folks out there 🙂

Daniel Nurmi20:10:41

Hopefully there will be a change here - available tooling/tech and renewed focus on these topics I think are making the ability-to-generate-needed-data more accessible to software producers

Jayson Henkel20:10:50

yah I think there's a real challenge with SBOM, because unless its literally up to the minute it may very well be out of date..

Topo Pal - Programming Committee Member21:10:30

"(vii)  providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website;"

Jayson Henkel21:10:00

yes definitely will help, but it's one control, though a powerful one 🙂

Gene Kim, ITREV, Program Chair20:10:53

^^^ @jason.cox I imagine you see this, where devs in business units have lots of freedom / autonomy / short attention spans. 😆

Kim Weins (Anchore)20:10:52

Check out the recording of the Cisco session from earlier today @jayson.henkel498. They are using SBOMs.

🙏 2
Jayson Henkel20:10:46

@kim.weins yes, I attended, it's still early days.. for a lot of orgs.

Kim Weins (Anchore)20:10:49

For sure. The US Executive Order might be a kick in the pants for software suppliers, which will probably then influence what other enterprises require from their suppliers + do in their internal dev.

Kim Weins (Anchore)20:10:46

Also the Linux Foundation, CNCF and OpenSSF are starting a major push on OSS projects that will potentially help advance things

❤️ 1

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees.