Fork me on GitHub
#ask-the-speaker-track-4
<
2021-10-05
>
Mitesh (DOES Event Staff / Engineer at Gaiwan)11:10:00

Super excited for all the talks coming up this conference! ๐Ÿ˜Š ๐Ÿ™Œ

๐Ÿ‘ 3
Nikki Mejer - Sonatype15:10:54

Looking forward to hearing @stephen talk! The Minefield of Open Source: Guidance for Staying Secure

Molly Coyne (Sponsorship Director / ITREV)16:10:26

Welcome @keith.puzey and @sujay.solomon from Broadcom's team for our next session's Q&A. Thank you #xpo-broadcom

Haendel Dorfeuille16:10:05

Is it me or No sound from track 4 ?

๐Ÿ‘ 2
Sujay Solomon16:10:11

hello everyone! super excited to be here and chat about test automation

Mark Fuller16:10:13

No sound for me either

Tom Coudyzer16:10:14

No sound here either

Ann Perry - IT Revolution16:10:34

We're working on the audio issues!

๐Ÿ‘ 12
Marcos Ferreira16:10:11

No sound on track 4

Juan Pablo Dรญaz Sidaras16:10:14

Hello Team! no audio on my side in Track 4

Juan Pablo Dรญaz Sidaras16:10:31

Track 1 works well, track 4 does not have audio

Justin Abrahms (eBay)16:10:10

re-woo!

๐Ÿ‘ 1
2
Dave Hogg (Leonardo)16:10:18

Sorted...๐Ÿ‘

๐Ÿ‘ 1
Tom Coudyzer16:10:06

Sorted here as well!

Mark Peters16:10:24

Audio is back

Mark Peters16:10:34

Shame because this is the one I was looking forward too

Dave McNierney16:10:44

Audio is ok for me

Justin Abrahms (eBay)16:10:21

I love that diagram which shows application as being on a different plane than infrastructure.

Anurag (Shoreline.io)16:10:38

Some years ago, when I was at AWS, I was chatting with Suresh Kumar, then the CIO at BNY Mellon. He pointed out that applications last longer than the underlying databases which in turn last longer than underlying OS which in turn last longer than the underlying hardware. Itโ€™s important to test accordingly.

โœ… 4
Justin Abrahms (eBay)16:10:26

My video is at the highest quality setting 720p, but the slides are still a bit difficult to read. Would love it if we could get these slides.

๐Ÿ‘† 1
Ryan Taylor, Application Architect, Axim Geospatial16:10:25

Same here. I would also like to see more of the screen used for the presentation and less for the branding/margin.

Molly Coyne (Sponsorship Director / ITREV)16:10:11

Thanks for your notes. The slides will be available in the video library.

๐Ÿ‘ 1
Gene Kim, ITREV, Program Chair16:10:59

My video just cleared up, and I can see everything clearly now. ๐ŸŽ‰

Scott Prugh (DOES Prog Committee)16:10:02

I love measuring cycle time and looking at how manual testing affects that negatively and how automated testing affects that positively...

Sujay Solomon16:10:38

the key for me has been bringing ownership of quality (at all levels of the SDLC) into the dev teams. Automated testing has often been the carrot to get them to take on that ownership ๐Ÿ™‚

๐Ÿ‘ 1
Garrin Ball - DevOps Leader - DDMI16:10:56

empowering developers is one thing, but there is something to be said for incentivizing/motivating them to do it as well.

โœ”๏ธ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:35

Yes. This is the hardest part. I talk about a failed effort along those lines in How to Misuse and Abuse DORA Metrics after lunch.

Sujay Solomon16:10:48

i've had a few folks ask about measurable quality/policy gates they can be set to ensure ownership of quality within teams. I haven't really found a good answer for this. Traditionally, code coverage has been used but that often ends up being a checkmark rather than a true reflection of confidence in the change.

Bryan Finster - Defense Unicorns (Speaker)16:10:13

โ€œI can think of lots of examples of measuring the wrong things. At one of my clients, they decided that they could improve the quality of their code by increasing the level of test-coverage. So, they began a project to institute the measurement, collect the data, and adopted a policy to encourage improved test-coverage. They set a target of โ€œ80% test coverageโ€. Then they used that measurement to incentivize their development teams, bonuses were tied to hitting targets in test-coverage. Guess what, they achieved their goal! Sometime later, they analyzed the tests that they had, and found that over 25% of their tests had no assertions in them at all. So, they had paid people on development teams, via their bonuses, to write tests that tested nothing at all.โ€ Excerpt from โ€œModern Software Engineeringโ€ by David Farley

1
Bryan Finster - Defense Unicorns (Speaker)16:10:17

I saw similar outcomes at Walmart. It was frustrating to warn against it and have management shrug, do it anyway, and then be surprised I was correct.

Sujay Solomon16:10:55

the level of confidence gained from unit tests is fairly low ROI according to this model

Bryan Finster - Defense Unicorns (Speaker)16:10:15

Yes, we pushed this in the WM Testing Special Interest Group as well.

๐Ÿ‘ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:48

Iโ€™ve always found this to be a better pattern. The key, however, is understanding Kentโ€™s definition of โ€œIntegration Testโ€

๐Ÿ’ฏ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:37

We created a glossary to establish a testing vocab.

Garrin Ball - DevOps Leader - DDMI16:10:46

would love to get my hands on that glossary ๐Ÿ™‚

Bryan Finster - Defense Unicorns (Speaker)16:10:01

I MAY have a copy somewhereโ€ฆ

Sujay Solomon16:10:28

i'd be very interested in that too!

Bryan Finster - Defense Unicorns (Speaker)16:10:01

Grab me at the bar later.

๐Ÿ‘ 1
Haendel Dorfeuille16:10:11

Video resolution is not great for me :(

Dave McNierney16:10:50

Hoping the library version will be better

Jeff Hughes16:10:51

Please ask any questions of Keith and Sujay on this channel. Thanks!

Stephen Magill [Sonatype]16:10:59

Hi all, excited to be here!

Molly Coyne (Sponsorship Director / ITREV)16:10:03

Thank you Broadcom! A warm welcome to @stephen for our next session's Q&A. Thank you #xpo-sonatype!

Dave McNierney16:10:05

Nice presentation! Good reminder of the need to think cross-platform with testing, just like with apps

Stephen Magill [Sonatype]16:10:26

If you want to see the report ^^

๐Ÿ‘ 1
๐Ÿ‘ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:32

Yay @stephen! Showing us how afraid we should be!! ๐Ÿ˜„

๐Ÿ™Œ 1
1
๐Ÿ‘ 1
๐Ÿ‘€ 1
Stephen Magill [Sonatype]16:10:16

hey @chris.gallivan421!

Stephen Magill [Sonatype]16:10:00

haha. I try to sneak some positivity in there too ๐Ÿ™‚

๐Ÿ‘ 1
Nicole Forsythe16:10:53

โ€œItโ€™s a large number, and itโ€™s growing.โ€

Trac Bannon16:10:57

Amazing volume for JavaScript!

Bryan Finster - Defense Unicorns (Speaker)16:10:15

It would be interesting to create a pipeline warn gate around thisโ€ฆ

Stephen Magill [Sonatype]16:10:35

@bryan.finster486: around usage? like warn if no one else is using this?

Bryan Finster - Defense Unicorns (Speaker)16:10:52

โ€œBe aware, you and 3 opther people use this in the worldโ€

Gene Kim, ITREV, Program Chair16:10:57

Nooooโ€ฆ. โ€œMost popular projects have the most vulnerabilities.โ€ ๐Ÿ˜† @stephen

๐Ÿ˜ 1
๐Ÿ‘€ 1
Stephen Magill [Sonatype]16:10:17

itโ€™s the economics of security researchโ€ฆ

Chris Gallivan (Planview)16:10:57

people like vulnerabilities?

Stephen Magill [Sonatype]16:10:16

donโ€™t take this as advice to go use obscure projects. security through obscurity doesnโ€™t work for software development any more than it does for confidentiality!

๐Ÿ‘ 2
๐Ÿ‘€ 1
๐Ÿ‘ 1
Trac Bannon16:10:21

If it's popular, it's vulnerable because of the usage; many eyes on breaking and looking for the gaps.

๐Ÿ‘† 1
Jeremy Sechler16:10:48

But lots of eyes on fixes too.

Trac Bannon17:10:32

True... it's a yes and situation.

Jeremy Sechler17:10:26

Aren't they all ๐Ÿ™‚

Jeremy Sechler17:10:53

Looking forward to your Data presentation this afternoon.

Trac Bannon16:10:39

@stephen -True dat! This is not about having obscure libraries or projects.

Stephen Magill [Sonatype]16:10:48

yep! security researchers focus on popular projects (as do โ€œblack hatsโ€). but as weโ€™ll see later, the best projects also make sure theyโ€™re pushing updates out that remediate these vulnerabilities.

Gene Kim, ITREV, Program Chair16:10:54

4MM dependency upgrades! 234K dependency versions.

Courtney Kissler16:10:57

I wish @stephen was singing his presentation ๐ŸŽค

๐Ÿ˜† 2
Stephen Magill [Sonatype]16:10:08

LOL @ckissler

๐Ÿ˜ 1
Chris Gallivan (Planview)16:10:30

that's later

๐Ÿ‘ 1
Gene Kim, ITREV, Program Chair17:10:28

Only 0.3% of vulnerabilities donโ€™t have a patch that remediates it. Whew.

๐Ÿ‘ 1
Stephen Magill [Sonatype]17:10:36

yeah, thatโ€™s the silver lining

Nicole Forsythe17:10:45

So what to do with the 7.7%?

Stephen Magill [Sonatype]17:10:09

make sure youโ€™re monitoring security feeds / using tools / etc to notice when those need to be updated due to a disclosed vulnerability

โค๏ธ 2
Bryan Finster - Defense Unicorns (Speaker)17:10:44

Canary builds with embedded scanning even if there is no feature change to the component.

Gene Kim, ITREV, Program Chair17:10:52

In other news, wasnโ€™t there a big Struts vulnerability that affected Confluence users?

Stephen Magill [Sonatype]17:10:46

Gene โ€” these graphs were made with Python ๐Ÿ˜„

๐Ÿ‘ 1
Gene Kim, ITREV, Program Chair17:10:03

Theyโ€™re beautiful, @stephen!!

Gene Kim, ITREV, Program Chair17:10:19

Violin diagrams, right? ๐Ÿ™‚

Stephen Magill [Sonatype]17:10:31

just wait for the MTTU graph that comes later โ€” I spent way too much time on that one ๐Ÿ™‚

๐ŸŽ‰ 2
๐Ÿ˜„ 1
Stephen Magill [Sonatype]17:10:20

I think this is really interesting. People are, by and large, very focused on remediation. Less awareness of the importance of controlling what comes in.

Bryan Finster - Defense Unicorns (Speaker)17:10:44

Every-time I run create-react-app I cross my fingers.

Trac Bannon17:10:20

I did that this weekend again... and shazaam.

Trac Bannon17:10:47

Now just run npm and yarn on the same project and see if you can monk-it-up.

Trac Bannon17:10:53

And we are wondering why there are vulnerabilities? The ability to have fast access to something like create-react-app is both beautiful and damned scary.

Trac Bannon17:10:38

People are really unaware of what they are introducing into their local environments and lager, into shared environments. YIKES!

Bryan Finster - Defense Unicorns (Speaker)17:10:56

We had a process to help make this better at Walmart, but I wonder how many people will proxy NPM and Nexus in their artifact repository and scan the versions of dependencies people are using within the org.

1
Trac Bannon17:10:27

@bryan.finster486 - I had a similar process with Commonwealth of Pennsylvania; a 2-step process so did introduce directly to the developer desktop unless it was a sanbox situation.

Bryan Finster - Defense Unicorns (Speaker)17:10:12

We were dong it async, but could black-list something if it failed scan.

Trac Bannon17:10:24

"Industry is doing a pretty good job" with remediation.

Trac Bannon17:10:44

And not as much focus on suppliers.

Dominica DeGrandis, Author - Making Work Visible, Tasktop (now Planview)17:10:00

Being proactive re: dependencies - 1.Pay attention to Quality

Gene Kim, ITREV, Program Chair17:10:04

PS: I think I had my first code PR accepted earlier this year, updating a Vega/Vega-Lite upgrade inside of a Clojure library โ€”ย was immensely proud of myself, @stephen I needed a diagram that could only be done in Vega v5. (@arne.brasseur., it was for Oz library)

๐Ÿ‘ 3
Stephen Magill [Sonatype]17:10:46

@genek: Popularity as a (misleading) metric makes another appearance in a slide or two.

Gene Kim, ITREV, Program Chair17:10:49

@stephen I canโ€™t wait!! And wow, so worrisome! ๐Ÿ™‚

Stephen Magill [Sonatype]17:10:57

oh maybe itโ€™s not on here. let me find the graphic with popularity includedโ€ฆ

Gene Kim, ITREV, Program Chair17:10:07

This is great, @stephen!!! 1.8x less likely to be vulnerable! (At any given point time?) High quality projects are 8x less likely to have breaking changes.

Stephen Magill [Sonatype]17:10:32

the red is bad (negative association)

Stephen Magill [Sonatype]17:10:50

so more popular projects were more likely to be vulnerable (for all the reasons we already discussed)

Trac Bannon17:10:00

Agree on update dependencies frequently and offer that we need teams to understand the purpose of the dependency.... what does it offer?

Stephen Magill [Sonatype]17:10:05

yes, teams should carefully evaluate / discuss each time they consider adding a new dependency.

Trac Bannon17:10:27

TRUE and yet, different teams are all at different points on their journey with some not being as knowledgeable ... and taking a bit of a black box mentality

Stephen Magill [Sonatype]17:10:48

yes. Iโ€™m not sure what the best answer is, but there must be some way to help these teams. what is the optimal feedback to provide to developers to help them make these choices?

Trac Bannon17:10:59

I tend to think having an architect or lead tech who has an evangelist mentality with a team can make a difference.

Trac Bannon17:10:35

The process guidance you give are excellent, @stephen

Gene Kim, ITREV, Program Chair17:10:12

Thatโ€™s a great finding, @stephen! I love that MTTU can guide good component selection!

Stephen Magill [Sonatype]17:10:17

it turns out http://libraries.io sourcerank includes a lot of popularity-type measures

Gene Kim, ITREV, Program Chair17:10:47

MTTU improving over time! (Super novel use of graphs, @stephen ๐Ÿ™‚

Stephen Magill [Sonatype]17:10:20

thanks! thatโ€™s the one that took forever ๐Ÿ™‚

Stephen Magill [Sonatype]17:10:48

super encouraging though to see the community-wide improvement in MTTU over time!

Gene Kim, ITREV, Program Chair17:10:56

Flipping awesome insight to split by year โ€”ย such an improvement over that MTTU vs MTTR graph we did 2 years ago, which was depressing. OTOH, this is quite hopeful!!

Stephen Magill [Sonatype]17:10:51

yes! I really want to take a look at other ecosystems now and see if this is widespread or more Java-specific.

Gene Kim, ITREV, Program Chair17:10:34

Oh myโ€ฆ. is assertion that people donโ€™t want to jump more than 1 or 2 versions? Is that a valid fear?

๐Ÿ‘€ 3
Stephen Magill [Sonatype]17:10:01

yep. I wonder (speculating) whether this is due to fear of breaking changes.

Stephen Magill [Sonatype]17:10:32

1 or 2 versions forward feels safer, so they go to the closest non-vulnerable version rather than just getting fully up-to-date.

Gene Kim, ITREV, Program Chair17:10:33

Freaking beautiful! Spring updates!

Sanket Naik17:10:53

Would be interesting to see by industry vertical or some other interesting slice.

Stephen Magill [Sonatype]17:10:01

noted and good idea. Iโ€™ll see if we can provide some by-industry insights.

Gene Kim, ITREV, Program Chair17:10:49

โ€œred: those are people that upgraded to vulnerable version.โ€ Oof.

Gene Kim, ITREV, Program Chair17:10:17

Is there evidence that some people are pinning to latest?

Stephen Magill [Sonatype]17:10:27

good question. I donโ€™t think weโ€™ve looked specifically yet at what percentage of the population exhibits that behavior. Iโ€™ll take a look though โ€” itโ€™s an interesting question!

Stephen Magill [Sonatype]17:10:26

the โ€œwave of redโ€ is scary

Stephen Magill [Sonatype]17:10:40

relentlessly marching forward as new vulnerabilities are discovered

Gene Kim, ITREV, Program Chair17:10:05

Love this research, @stephen โ€“ kudos to you and team for it!!!!

โ˜๏ธ 3
Stephen Magill [Sonatype]17:10:18

Thanks! As you might expect, Bruce was the superstar pulling so many things together for this ๐Ÿ™‚

Gene Kim, ITREV, Program Chair17:10:19

Was thinking of you when I was editing my interview of Dr. Gail Murphy: https://itrevolution.com/the-idealcast-episode-21/ She and I were discussing whether innovation happens because people are allowed to do lots of breaking changes, or if it happens because you donโ€™t introduce breaking changes. Was absolutely fascinating โ€”

Stephen Magill [Sonatype]17:10:16

Iโ€™ll have to go listen to that. I feel like thereโ€™s an inflection point. Early on, breaking things is good. Past some point it slows you down too much if things that should be โ€œsettledโ€ are breaking all the time.

Sanket Naik17:10:59

Great talk @stephen!

Molly Coyne (Sponsorship Director / ITREV)17:10:15

Thank you @stephen and #xpo-sonatype!

Glenn Wilson, Author of DevSecOps17:10:07

I missed @stephenโ€™s talk. But will definitely play this one back later. Looks like I missed a good talk

๐Ÿ‘ 1
Ally Corsetti17:10:48

Thank you for joining the Aqua and Anchore Vendordome!

Molly Coyne (Sponsorship Director / ITREV)17:10:53

Welcome @rani who will be moderating for today's VendorDome Q&A between #xpo-anchore-devsecops and #xpo-aqua-security-k8s @nurmi and @rory.mccune! https://devopsenterprisesummitus2021.sched.com/?iframe=yes&amp;w=100%&amp;sidebar=yes&amp;bg=no#

๐ŸŽ‰ 3
Molly Coyne (Sponsorship Director / ITREV)17:10:06

Share any questions here that arise for you during this session and our speakers will address them right now during this LIVE session!

Virginia Laurenzano NSA17:10:34

@emoshmosh - talking about EO on supply chains

Virginia Laurenzano NSA17:10:40

to paraphrase: hard to marry regulation with cloud native development practices.

Jayson Henkel17:10:09

You mentioned, Codecov, why do you think that incident didn't receive a larger media coverage

Virginia Laurenzano NSA17:10:45

speakers hope we'll resist the temptation to turn this into a paperwork exercise

Gene Kim, ITREV, Program Chair17:10:04

Was marveling at the impact of the latest Struts vulnerability โ€”ย software supply chains is so relevant right now!

Kim Weins (Anchore)17:10:26

64% of enterprises have been impacted by a SW supply chain attack in the last year. Here's the data https://anchore.com/software-supply-chain-security-report/

Gene Kim, ITREV, Program Chair17:10:52

Do what degree are people not upgrading dependencies because of fear of breaking changes?

Kim Weins (Anchore)17:10:15

Good point. There is effort in upgrading.

Gene Kim, ITREV, Program Chair17:10:00

Iโ€™m seriously interested in what it would take for devs to upgrade dependencies, especially when there is a patched component available โ€”ย now itโ€™s more than detection, but actually remediation. I love the quote that the best way to stay secure is to just stay up the date.

paulsbruce17:10:01

Hello yโ€™all. Looking forward to discussion in Slack and questions/feedback you may have!

๐Ÿ‘ 1
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:22

To some extent I think itโ€™s a visibility issue. Itโ€™s harder to make dependency updates a prioritized backlog item with stakeholders when compared with the more visible impact of adding features.

Virginia Laurenzano NSA17:10:59

plus, it's hard to plan for a lot of work where we can't show measurable gains - like, how can we quantify prevention?

๐Ÿ’ฏ 1
Frotz Faatuai (Cisco IT - he/him)17:10:01

Ugโ€ฆ Pet environmentsโ€ฆ

๐Ÿ˜ข 1
๐Ÿ‘ 1
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:04

If we can move beyond automated dependency checking to automated remediation, via tools opening PRs to update dependencies we will be a lot better positioned. But we have to have robust automated testing to handle the fear that the upgrade will break something that the speakers touched on earlier.

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:04

If we can move beyond automated dependency checking to automated remediation, via tools opening PRs to update dependencies we will be a lot better positioned. But we have to have robust automated testing to handle the fear that the upgrade will break something that the speakers touched on earlier.

paulsbruce17:10:25

couldnโ€™t agree more. dependabot workflows should be a force for good, code dependencies OR in automated testing activities!

๐Ÿ‘ 1
Lochan Vasudev17:10:21

Great talk. I have a question - How do the service organizations balance between frequent deployment vs client's expectation of not having frequent changes that may risk service delivery?

Kim Weins (Anchore)17:10:08

A number of successful supply chain attacks have come in through DevOps toolchains.

Kim Weins (Anchore)18:10:40

Layered security models = defense in depth

Gene Kim, ITREV, Program Chair18:10:15

Thank you so much, @nurmi @rory.mccune !!!

๐ŸŽ‰ 1
Daniel Nurmi18:10:53

Thank you all for attending, and having us on to discuss this important topic!

Molly Coyne (Sponsorship Director / ITREV)18:10:20

Welcome @dave.karow for our next session's Q&A Thank you #xpo-split

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:54

Iโ€™m ready to goโ€ฆ feel free to ask questions during the talk!

Garrin Ball - DevOps Leader - DDMI18:10:52

as part of Scaled Agile, we work at separating deploys from releases. Iโ€™ll admit though, this is the first time Iโ€™ve heard โ€œProgressive Deliveryโ€. Iโ€™m intrigued

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:28

Stay tunedโ€ฆ decoupling is key but thereโ€™s much more possible ๐Ÿ™‚

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:40

In London, I got a question about whether progressive delivery was any different than CD. The foundational idea (decouple deploy from release) was in Jez Humble and Dave Farleyโ€™s CD book, but the practices of getting more fine grained about gradual releases and using data aligned to these gradual releases is where it gets even interesting.

โค๏ธ 1
Dipesh Bhatia18:10:37

Can you post the link blog http://split.io

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:06

For you code readersโ€ฆ yes, thereโ€™s a โ€œbugโ€ in the else if line hereโ€ฆ should say โ€œtreatment == โ€

Pedro Jordan18:10:22

this "flags" are in the same codebase , togheter on the same binary lets say ? what if an application crashess because of one on flag.

Dave Karow (Split - Sr. Progressive Delivery Advocate)19:10:07

@pedro.jordan I missed the second half of your question during the talk. The great thing about flags is that you can toggle the state remotely and instantaneously if an issue arises. Thatโ€™s how you avoid needing a rollback/roll-forward deployment to resolve an unanticipated issue. Just toggle the flag and within milliseconds itโ€™s off again.

Pedro Jordan19:10:34

that's brilliant ! thanks Dave.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:47

treatment just means code pathโ€ฆ could be classic blue button / red button marketing test but could also be back-end code that gets executed.

Garrin Ball - DevOps Leader - DDMI18:10:16

i love flags, but you need your test automation on point, for both sides of the flag. Thats where Iโ€™ve seen struggle in the past

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:23

Yesโ€ฆ one binary with multiple possible execution paths that can be executed based on runtime decision a user/session at a time.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:32

Testing strategies are one of the details to work out. Being able to have a test get the desired flag state is the main strategyโ€ฆ the upside is being able to test new and old experience in same environment.

David Van Couvering - Senior Principal - eBay18:10:35

One thing our team struggles with is that multiple teams are rolling out features at the same time, and when our business metrics go negative, we can't figure out which change caused the problem. Any suggestions?

Pedro Jordan18:10:48

I have also used them and they are cool in some cases , but just wanted to see a full example of these "decouple" arch

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:19

@dvancouvering for sureโ€ฆ thatโ€™s where tying the metrics to actual flag decisions comes in. Next several slides will introduce the idea.

David Van Couvering - Senior Principal - eBay18:10:49

Haha I just set you right up ๐Ÿ™‚

Keara Vu (IT Revolution)18:10:49

Love the point on all the ever-changing surroundings and their ability or inability to impact behavior!

Dipesh Bhatia18:10:12

How is weighted routing different than Canary?

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:33

Canary by containers exposes entire release to a segment of network traffic (hopefully sticky). % based splitting using flags is still a canary of sorts but is at the feature, not build/release level.

๐Ÿ‘ 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:38

@dvancouvering When multiple flags are being used, the key is to use a different seed for randomizing each. THAT makes what you are seeing now cancel out other flag influence.

David Van Couvering - Senior Principal - eBay18:10:45

OK, yea I think we actually do that. I guess the other problem is it can take a very long time to get statistically significant results. This means rollout can take a week as we gather the data for each phase of the rollout.

Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:47

Yep. There is a balancing act between test to learn and test to launch. The latter looks for bigger scarier signals and acts on them quickly. The former holds out longer to get defendable stats outcomes.

David Van Couvering - Senior Principal - eBay18:10:17

I like that "test to learn" vs "test to launch"

Dipesh Bhatia18:10:30

testing is in every stepโ€ฆ I like weighted routing, 10 - 20 - 50 โ€ฆ

paulsbruce18:10:22

Gotta say @dave.karow youโ€™re making it hard for me to pay attention to doing other work in the background ๐Ÿ˜„

Dave Karow (Split - Sr. Progressive Delivery Advocate)19:10:43

Ha! Just connected the dots that you were the next speaker. I used to live in your world not long agoโ€ฆ BlazeMeter, before that SOASTA, and before that Keynote LoadPro. Went from consultant led big-bang load testing to developer-led continuous testing.

Pedro Jordan18:10:07

thanks for the talk !

1๏ธโƒฃ 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)18:10:14

Iโ€™ll hang here for one minute and then head over to #xpo-split for more Q&A

๐Ÿ‘ 1
Tish18:10:42

That was very interesting. Not sure is applicable to my day job... but it does explain a lot about when you hear about a new feature being rolled out on FB or Google or whatever platform.

๐Ÿ‘ 2
Amri Abuseman - Flatiron Health, Director of Quality Engineering18:10:48

@dave.karow great presentation and this is in-line with what we're thinking on my team to decouple deploy from release

Amri Abuseman - Flatiron Health, Director of Quality Engineering18:10:12

Will you be able to share the slides from this presentation so I can share with my team as we plan for 2022?

Molly Coyne (Sponsorship Director / ITREV)18:10:31

Slides will be available in our video library!

Molly Coyne (Sponsorship Director / ITREV)18:10:54

โœจWelcome @p.bruce for our next session's Q&A. Many thanks to #xpo-tricentis for their loyal sponsorship!

๐Ÿ‘‹ 1
Emily Hart18:10:57

๐Ÿ–ฅ๏ธ https://devopsenterprisesummitus2021.sched.com/event/mGSj - with @p.bruce Thereโ€™s no question that enterprises today want to further integrate continuous performance testing into automated pipelines. However, many are finding it difficult to reconcile the mismatched clock-speed of testing with todayโ€™s accelerated pace of development/delivery. Tune into Paul Bruceโ€™s session https://devopsenterprisesummitus2021.sched.com/event/mGSj to learn*,* among other things, the key steps to continuous performance testing in DevOps: โ€ข Gather the right metrics to assess your gaps Prioritize, then systematize across your application portfolio โ€ข Plan for acceleration across the whole delivery cycle Design concrete measurements with the end in mind โ€ข Pick the right targets to automate Make scripting easy for multiple teams Develop performance pipelines โ€ข Use dynamic infrastructure for test environments Ensure trustworthy go-no-go decisions โฐ Session is starting now in #here!

1
paulsbruce18:10:23

Last year, I published a bunch of blogs (and I think they did a bundle/paper) on continuous performance and load testing: https://www.neotys.com/blog/easy-a-key-requirement-for-continuous-performance-testing/

๐Ÿ‘ 3
1
๐Ÿ‘ 1
paulsbruce18:10:43

Also, I recently did another presy for EuroSTAR about how to move the performance mindset and practice forward: https://docs.google.com/presentation/d/11CKT5zUX8bFFruvJK7d_8KKkfyHTv9d06eeNit04Ah4/edit?usp=sharing

๐Ÿ‘€ 1
1
Ben Kruenegel18:10:41

loved the reference to particle physics!

โค๏ธ 2
Benoit Devost19:10:36

"Superheroes in IT are single points of failure" This phrasing is just brilliant. Kudos @p.bruce for coining that.

โค๏ธ 4
๐Ÿ‘ 1
โœ… 1
๐Ÿ™ 1
paulsbruce19:10:45

If you like that one, check out some other thoughts I was able to put down a few months ago: https://www.youtube.com/playlist?list=PLFXQmSmq7uXTElSlaOqUHCeVlJi5nGJ85

paulsbruce19:10:07

(not right now though, keep listening to the live-stream ๐Ÿ˜„ )

๐Ÿ˜… 1
Haendel Dorfeuille19:10:55

โ€œIt is not about testing everythingโ€

๐Ÿ’ฏ 1
โค๏ธ 1
paulsbruce19:10:41

Who has operational (i.e. performance) requirements in their work planning process?

Andrew Machen19:10:52

"Ending DevOps Holy Wars" - A book I would want to read ๐Ÿ˜›

โค๏ธ 1
1
paulsbruce19:10:21

You may have heard people refer to performance as โ€˜non-functionalโ€™. Sorry, I call it OPERATIONAL. ๐Ÿ˜„

โค๏ธ 2
๐Ÿ‘ 3
Meghan Glass - PrdMgr Best Buy19:10:02

Right? Who came up with that term--it's going to be non-functional if you don't account for it...

Ben Kruenegel19:10:02

it does seem most non-functional requirements are looked upon as optional; to add on to your comment, operational should not be optional

โค๏ธ 2
๐Ÿ‘ 1
Chris19:10:12

Thanks @p.bruce

Ben Kruenegel19:10:18

good session, thanks!

๐Ÿ™ 1
paulsbruce19:10:21

For anyone who wants to discuss continuous performance engineering more, Iโ€™ve bookmarked my calendly for this event at the top of the channel:ย https://calendly.com/paulsbruce/devops-ent-summit-21-chat

Malcolm McAlpin19:10:26

Thanks!

โœ… 1
Alvin Crespo19:10:26

Fantastic talk @p.bruce! Thanks so much.

โค๏ธ 1
Andy Nelson19:10:48

๐Ÿ‘ nice work @p.bruce

๐Ÿ™ 1
Molly Coyne (Sponsorship Director / ITREV)19:10:57

Thank you @p.bruce and #xpo-tricentis!

๐Ÿ™ 1
Dr. Ivan Kronkvist19:10:04

@p.bruce I am a partice physicist

paulsbruce19:10:42

holy crap. i hope i didnโ€™t โ€˜bend the laws of physicsโ€™ too much with my analogy. i fall asleep for weeks every year to โ€˜entanglementโ€™ by Aczel ๐Ÿ™‚

Dr. Ivan Kronkvist19:10:08

lol, not at all.

๐Ÿ˜Œ 1
Dr. Ivan Kronkvist19:10:04

Great presentation.

๐Ÿ™ 1
Dian Hansen19:10:09

Highlighted exactly why this remains hard and often outside scope of regular work - and the problems that causes. Thanks @p.bruce!

โค๏ธ 1
Molly Coyne (Sponsorship Director / ITREV)19:10:15

๐Ÿ‘:skin-tone-2:Please welcome @simon540 and Yash Kosaraju for our next session sponsored by #xpo-snyk!๐Ÿ‘:skin-tone-2:

๐ŸŽ‰ 5
Dave Fugleberg19:10:21

do your security champions self-select? Do you have some kind of selection / vetting process?

Simon Maple19:10:14

Great question - thereโ€™s often a mix of approaches from company to company, largely based on whether orgs want full coverage across all teams/BUs or whether folks like to ensure that everyone is there because they want to be there. Both approaches have pros and cons, and largely depends on your needs as to which approach you go for

Sanket Naik19:10:05

@dfugleberg Coupa has a similar security champions program. The selection is a combination of nomination and endorsement based on interest and experience. After that there is an annual certification run by the Security team to keep the champions up to date. And there are ceremonies (akin to Agile) around the program like monthly meetups.

Simon Maple19:10:13

One of the most important parts in vetting is being really clear about what skills and knowledge a good champion should have. Itโ€™s more usual for a mismatch occurring because the role isnโ€™t well defined.

santosh dhanawade19:10:46

how dynamic data is handled automated CI/CD in your case. for example, I wrote a test case with one version of data but once data got changed then my automated test case needs to change as well. how to handle this dynamic case in ci/cd?

Jayson Henkel19:10:50

Apart from STRIDE/DREAD, I'd be curious what peoples threat modelling processes look like..

Chris19:10:08

Do you have some threshold (based on severity for found vulns, or potential blast radius, some patterns in IACโ€ฆ) where you break the build? (eventually even in a pre commit) ? Or do you have a design which limits blast radius and would allow to let full accountability to the product team, just informing them, or some limit upon which it should block?

Molly Coyne (Sponsorship Director / ITREV)19:10:31

๐ŸŒŸWelcome @nurmi and @kim.weins for our next session's Q&A! Thank you #xpo-anchore-devsecops for your support of DOES! ๐ŸŒŸ

Kim Weins (Anchore)19:10:04

oops, I guess they didn't take out the transition ๐Ÿ˜†

Rebecca Carter20:10:16

Interested in learning more about securing the software supply chain? Download the white paper https://get.anchore.com/prevent-software-supply-chain-attacks/

Kim Weins (Anchore)20:10:19

I'm curious if folks here are using multiple DevOps tools.....

๐Ÿ‘ 2
1
John Roesler - Sr Engineer (Gap Inc)20:10:08

yes, but there is one front runner and others are known to be on their way out / not preferred

๐Ÿ‘ 1
Gene Kim, ITREV, Program Chair20:10:59

โ€œTeams are on average using 6 DevOps tools, involved in the CI/CD pipelineโ€ โ€”ย and often in enterprises, those 6 tools are all different! ๐Ÿ˜ฑ

Gene Kim, ITREV, Program Chair20:10:24

Interestingly, this is something that @lucas.rettig and @levi.geinert500 had to tackle in 2018 at Target โ€”ย Iโ€™ve always interpreted this as a backlash of decades of forced standardization of dev tooling in the prior years. ๐Ÿ™‚ https://videos.itrevolution.com/watch/524020857/

โค๏ธ 6
Gene Kim, ITREV, Program Chair20:10:33

Confessions: I have more than a small degree of fear upgrading container base images.

Daniel Nurmi20:10:02

We do see quite a bit of variability within bigger orgs WRT dev/ops tooling, for a lot of reasons. A no uncommon one is when new teams / tech is brought in through acquisition, where along with team and tech also comes the entire dev. infrastructure and tooling as well ๐Ÿ™‚

๐Ÿ˜‚ 1
Chris20:10:16

Or siloed enough organizations at different stages of evolution where each silo has its own ecosystemโ€ฆ ๐Ÿ˜‡

Kim Weins (Anchore)20:10:45

My experience is when you ask any IT/Dev person "What do you use for <tool category>?" Answer is inevitably "One of everything"

๐Ÿ˜‚ 2
Gene Kim, ITREV, Program Chair20:10:46

I saw a talk from a Director of Dev Productivity from Cisco talk about this โ€” as a company driven by acquisitions, the task was given to him to migrate all the companies/divisions/tools onto a smaller groups of tools. It was a fantastic presentation โ€”ย Iโ€™ve been wanting him to share it with DevOps Enterprise Summit, as I thought it would resonate. There is a cost to freedom vs. standardization.

Kim Weins (Anchore)20:10:30

Plus there is tool drift over time, eg our preferred tool from 3 years ago is different than today.

Kim Weins (Anchore)20:10:56

And the business units that are "exceptions"

Jayson Henkel20:10:45

@genek I think we need a track on how to be on the cutting edge of security inititatives. eg. SBOM, I've asked vendors to support Sub Resource Integrity (SRI), getting more visibility into their own security practices, breach notification etc. I often get push back..... any suggestions from anyone...other than just wait ๐Ÿ˜›

Gene Kim, ITREV, Program Chair20:10:25

Can you send me an email at <mailto:genek@itrevolution.com|genek@itrevolution.com>? Thanks!

Jon Sturdevant - Tech Advisor - BlueCross BlueShield of SC20:10:05

We have similar issues when we are asking our vendors to be responsible for the open source they include in their software. A lot of times their legal teams try to remove those words from our contracts with them. Most of the time the reason is "this isn't something we do"

Jayson Henkel20:10:01

@jonathon.sturdevant precisely, you also start feeling like the tinfoil hat fellow....

๐Ÿ‘ฝ 1
Jayson Henkel20:10:38

Though I just had a vendor let me know we're 1 of 3 vendors who had asked to support SRI...so there must be other folks out there ๐Ÿ™‚

Daniel Nurmi20:10:41

Hopefully there will be a change here - available tooling/tech and renewed focus on these topics I think are making the ability-to-generate-needed-data more accessible to software producers

Jayson Henkel20:10:50

yah I think there's a real challenge with SBOM, because unless its literally up to the minute it may very well be out of date..

Topo Pal21:10:01

This is gap in the industry that I expect will be addressed by the current focus on SBOM, SPDX and this https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/

Topo Pal21:10:30

"(vii)ย ย providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website;"

Jayson Henkel21:10:00

yes definitely will help, but it's one control, though a powerful one ๐Ÿ™‚

Gene Kim, ITREV, Program Chair20:10:53

^^^ @jason.cox I imagine you see this, where devs in business units have lots of freedom / autonomy / short attention spans. ๐Ÿ˜†

Kim Weins (Anchore)20:10:52

Check out the recording of the Cisco session from earlier today @jayson.henkel498. They are using SBOMs.

๐Ÿ™ 2
Jayson Henkel20:10:46

@kim.weins yes, I attended, it's still early days.. for a lot of orgs.

Kim Weins (Anchore)20:10:49

For sure. The US Executive Order might be a kick in the pants for software suppliers, which will probably then influence what other enterprises require from their suppliers + do in their internal dev.

Kim Weins (Anchore)20:10:46

Also the Linux Foundation, CNCF and OpenSSF are starting a major push on OSS projects that will potentially help advance things

โค๏ธ 1
Slackbot21:10:21

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png