Fork me on GitHub

Reminder: Get yourself in front of your browser and into #ask-the-speaker-plenary for the opening remarks. We’re kicking off Day 2 in 15 minutes at 10am BST!


Reminder: Day 2 is starting now – opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.

Carmen DeArdo11:05:12

Good Day everyone!

👋 1

Reminder: The breakout sessions are starting in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending.

Molly Coyne (Sponsorship Director / ITREV)11:05:37

💡Welcome @carmen.deardo and @naomi.lurie for our next session's Q&A. Thank you for the continued support Tasktop! 🌟

👋 3
👏 1
Katharine Chajka (Tasktop)11:05:19

Some of the challenges I still see in 2022, is lack of automated testing, CICD, causing much local optimizations. Curious if others see similar.

Radoslaw Wiankowski12:05:53

oh yes! automated testing is still very much in the PowerPoint stage, and not even on the early slides.

❤️ 1
Margueritte Kim (CEO, IT Revolution, she/her)11:05:56

The cadence of this talk is a stark contrast to @mik talk. LOL.

Carmen DeArdo11:05:38

ha ha - none of us can speak like Mik 🙂 HI!

🩰 1
Katharine Chajka (Tasktop)12:05:49

Even with Product Owner being a role for 20+ years, I still see a struggle with defining and measuring value. Still seems very activity based

💯 1
Katharine Chajka (Tasktop)12:05:54

It surprises me to see Stage 3! Would have expected Stage 1 or 2. That is good news. Guessing are hitting the systemic bottlenecks like upfront planning and prioritization or a release process. Typically the development is the quickest part of the whole life cycle! Love the tip on "from the customer's perspective"!

Katharine Chajka (Tasktop)12:05:00

Delays due to dependencies - how to say Value Stream without saying Value Stream 😂

💯 1
😂 1
Margueritte Kim (CEO, IT Revolution, she/her)12:05:10

“Don’t rush until you’ve developed good, solid patterns” - I tell my kids this as well! A good life lesson.

💯 1
Katharine Chajka (Tasktop)12:05:29

@carmen.deardo @naomi.lurie related to expansion - do you find that while some change may be under teams control, that some of the major/systemic bottlenecks need more support from the organization somehow?

Carmen DeArdo12:05:33

Yes @katharine.chajka - the power of experimentation is that you now have actual data/experience that demonstrates that a given process in a portion of the value stream (e.g. releasing on demand) can be done effectively and with high quality. This helps highlight the “systemic” processes that can be improved.

❤️ 1
Radoslaw Wiankowski12:05:09

Do you often see cases where a product owner is an owner only by name, but lacks any mandate to make meaningful decisions? And so the decisions need to be made one/two levels up?

Carmen DeArdo12:05:14

Yes @rwiankowski this is a common “anti-pattern” that we observe. This really needs to be an experiment where the product team has control over their journey (with some guard rails). The team needs control to focus on improving flow and business outcomes.

Katharine Chajka (Tasktop)12:05:04

It is so fascinating to see the investment made in agile roles or practices but not in removing delays or bottlenecks to delivery, I imagine due to lack of visibility

Carmen DeArdo12:05:40

if System Thinking isn’t applied, innovation can quickly turn into a local optimization even with best intents.

💯 1
Virginia Laurenzano NSA/MARFORCYBER12:05:26

red tape is a recipe for cultural disaster. #preach

🙌 2
🎯 1
Radoslaw Wiankowski12:05:11

@katharine.chajka lack of visibility or lack of true understanding of what flow is? or maybe both?

Katharine Chajka (Tasktop)12:05:34

yes, likely both. No fault of anyones as it seems like they are not talked about for some reason

Katharine Chajka (Tasktop)12:05:12

I wonder why they are such hidden secrets!

Radoslaw Wiankowski12:05:41

idd. Seems like common sense, right? But then again, simple solutions are rarely easy to find.

Virginia Laurenzano NSA/MARFORCYBER12:05:27

reminds me of my study abroad program's mantra: the journey is the destination

❤️ 2
Katharine Chajka (Tasktop)12:05:30

great talk @carmen.deardo @naomi.lurie!!!

Radoslaw Wiankowski12:05:43

Amazing talk! Thanks!

thankyou 1
Naomi Lurie12:05:25

Thank you! We’d love to hear your voice by taking the assessment and building out the picture:

Sascha Schärich (DevOps Evangelist at Deutsche Telekom IT)12:05:20

Thanks for the talk, we get pretty good scores on the Anti-pattern slide… 🙂 Will check out the assessment!

👏 1
🙌 1
Molly Coyne (Sponsorship Director / ITREV)12:05:02

Welcome @jp for our next session's Q&A. Thank you CockroachDB! 💥

Jp Sisneros12:05:30

Hey hey! Nice to meet everyone!

👋 1
Jp Sisneros12:05:46

Just a special note. Our speaker Jim Walker was unavailable to join today, so y'all are stuck with me haha. Follow Jim on Twitter @jaymce for more db content.

Jp Sisneros12:05:38

Yes, there are different read and write levels for every database, but this is a good thing. As a developer, you get to choose what database works best for your use case, and that's pretty cool.

jeff.gallimore (CTIO - Excella, he/him)12:05:44

i like this perspective. good thought.

jeff.gallimore (CTIO - Excella, he/him)12:05:01

this is bringing me way back to my early career as a dba 😄

Jp Sisneros12:05:25

That's awesome @jeff.gallimore.

jeff.gallimore (CTIO - Excella, he/him)12:05:35

are we going to get into CAP theorem at all?

Jp Sisneros12:05:40

Not too much but I am happy to chat about it! (I am a geek when it comes to CAP)

🤓 1
👍 1
Jp Sisneros12:05:50

Thank you so much!

thankyou 1
Molly Coyne (Sponsorship Director / ITREV)12:05:17

:star-struck:Welcome @hlynch who will be moderating for today's VendorDome Q&A between @stephen and @ldonley. Thank you Sonatype and CloudBees! 🌟

🎉 3
🐝 1
💯 1
Gene Kim, ITREV, Program Chair12:05:40

Doh. I missed the database talk! @jeff.gallimore @jp

Gene Kim, ITREV, Program Chair12:05:18

Hello, @stephen @ldonley @hlynch!

👋 3
💯 1
🐝 1
Virginia Laurenzano NSA/MARFORCYBER12:05:27

always enjoy learning from CloudBees and Sonatype


Are any of you using progressive delivery today in your organizations?

👀 2
jeff.gallimore (CTIO - Excella, he/him)12:05:38

this is a big factor in getting people comfortable with continuous deployment of changes to production

Gene Kim, ITREV, Program Chair12:05:27

Trivia fact: • On this conference page, the Slack widget uses a feature flag — we weren’t sure the we’d use it for plenary track

👏 1
💯 1
🎉 4
🔥 2
Gene Kim, ITREV, Program Chair13:05:08

ah, the world of “self managed software” —

Chris Leeworthy (he/him)13:05:31

How do you deal with this when you have legacy code that maybe doesn’t have nicely decoupled components?

👂 4
Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:17

@genek imagine a time not so long ago, where you might debate recompiling your plenary UI after day one to make a change there... what could possibly go wrong :man-shrugging: ?

😏 2
Gene Kim, ITREV, Program Chair13:05:37

Haha. Changing things are super terrifying still. 🙂 (I’m still amazed we’ve pushed client changes a couple of times during conference — in fact, the slack widget can be restarted now independently!)

Quinn Daley13:05:22

on the subject of having quick access to remediation/rollback - when is the right time to remove a feature flag and all the associated “old” code from your codebase? and how do you ensure the “old” code is removed along with the flag?

👂 1
Gene Kim, ITREV, Program Chair13:05:43

@hlynch Question for panel: “it’s amazing how feature flags have taken off in the industry — what else can we learn from dynamic runtime environments, where we enable the runtime to truly dynamic? what else could we enable, vs things we view as static right now?”

👂 1
Philip Day13:05:54

Like available compute resource for a process?

Chris Leeworthy (he/him)13:05:04

That’s great, thanks folks 🙂

Quinn Daley13:05:40

thank you for your answer!

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:32

@quinn.daley one pattern that is useful is to flag at logical module points, not line-by-line. With clean, natural boundaries, you don't have to go through with a fine-toothed comb to find the things to remove later. This blog post, written by one of our Front End and Back-End for Front End lead devs, uses front end JavaScript as an example, but the pattern applies to any codebase.

👍 1
Quinn Daley13:05:52

yeah, we definitely do this but of course modules also depend on other modules etc!

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:21

Great. Sometimes folks want to get very "surgical" with putting a flag at the exact line of code in a module that does the thing... that makes it trickier. Check out David B's example where the other modules are abstracted away enough that they don't know or care what the parts that change/get removed are doing.

Quinn Daley13:05:33

we have a policy of putting flags at the “entry point” wherever possible, but it’s not universally adopted

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:40

Entry point was exactly the advice I used to give... then David B gave a talk about the modular way he does it and I realized his way was WAY less mentally demanding. After that talk, we convinced him to put it in that blog. They've even built some tooling that makes the modules approach faster/less risky, such as this: > • Adding a feature flag becomes easy. We have a script that will take an existing component, make a second copy of it and set up the toggler function that wraps both. These two benefits are probably my favorite: > • Reasoning about code is very easy. Because these are at “interface boundaries,” you do not have to contend with mixed logic in one routine. From the consumer’s point of view, you don’t know that there are multiple variations. From the consumee’s standpoint, you still do not know that there are other variants of yourself that could be run. This means that when working on one of the component variations, you can focus your attention on that one feature. > • This also makes unit testing dramatically easier. Because the blocks of code being developed are completely distinct, it also means that the unit tests are completely distinct.

Quinn Daley13:05:37

oh nice! I see there’s a subtle difference here and why it’s important


Audience: What are your best practices for day-to-day dependency management?

👀 1
Quinn Daley13:05:41

including dependency auditing in the CI pipeline so the build fails if there’s a disclosure against one of the deps certainly helps!

jeff.gallimore (CTIO - Excella, he/him)13:05:50

we do what @stephen said in his mini-lecture today 😄

Virginia Laurenzano NSA/MARFORCYBER13:05:42

I think @topo.pal used JFDI (just do it, the f is silent) in a talk. @stephen's comment about just do dependency management reminded me of #jfdi

💯 2
❤️ 1
Margueritte Kim (CEO, IT Revolution, she/her)13:05:12

“the f is silent” I love it.

❤️ 1
Gene Kim, ITREV, Program Chair13:05:57

I love it when I see engineering therms ruthlessly reducing the # of OSS dependencies — there's a great story of people eliminating Jackson for JSON parsing, http-commons, etc. Significant and concerted scrutiny and reduction of components. (It wasn't so long ago when I thought 'twas a bit... hysterical.)

Gene Kim, ITREV, Program Chair13:05:31

@hlynch I'd love to hear about stories of how engineers have successfully "rationalized their dependencies" — what things should inform what components should and shouldn't be used? I love stories of engineering teams significantly reducing dependencies — is that a pipe dream? Thx!!!

👂 1

What differences have you seen in how various tools help you remediate/address security risks?

👀 1
jeff.gallimore (CTIO - Excella, he/him)13:05:33

something we absolutely look for in tooling is the ability to incorporate them into the ci/cd pipeline.

🎉 1
Gene Kim, ITREV, Program Chair13:05:37

"such as the special circle of hell that is all the different versions of the Jackson JSON libraries)." Cc @stephen :)

Margueritte Kim (CEO, IT Revolution, she/her)13:05:21

I thought that was the grocery store the day before Thanksgiving for you?

😂 1
Chris Leeworthy (he/him)13:05:11

We have just got AWS Inspector up and running and had the shock of seeing just how much stuff needed fixing! The important thing is it made the weaknesses visible so we could do something about them.

🎉 3
Chris Leeworthy (he/him)13:05:45

you can’t fix a problem if you can’t see it!

❤️ 1
💯 1
Virginia Laurenzano NSA/MARFORCYBER13:05:02

maybe we need to train good security people like BCP/NTT did with their training academy? if anyone does this, I wanna hear the talk

Gene Kim, ITREV, Program Chair13:05:38

It was astonishing to see how they created a community around them, and even with their customers!!

❤️ 1

Any other questions around progressive delivery or security post them up!

👀 1
jeff.gallimore (CTIO - Excella, he/him)13:05:50

“only actively managing 25% of their dependencies” 😬

Billy Hudson - ScholarPack - DevOps Engineer13:05:29

How and where are people recording metrics for resolving dependancy issues from discovery to resolution?

👍 1
👂 1
Chris Leeworthy (he/him)13:05:32

date/time libraries are a pet hate of ours!

Virginia Laurenzano NSA/MARFORCYBER13:05:40

just because we can use all the options doesn't mean we should. noted

⬆️ 2
Gene Kim, ITREV, Program Chair13:05:48

Ha! 15 XML parsing libraries. And Jackson. :) 15 different logging frameworks.


Great questions! Keep them coming!

Johnny Bäverås - Synopsys13:05:01

Back in my R&D days, 10 years ago, i even made a habit out of picking new XML parsing libraries for every integration i did. To find which one i found easiest to work with. Did i ever upgrade them? of course not! 😉 I wouldnt mind a time machine to go back and have a stern talk to myself.

😅 2
Gene Kim, ITREV, Program Chair13:05:11

3 different JavaScript UI frameworks in our app. (Which I didn't think was actually possible?? React, Backbone, and something else... apparently, not that uncommon...)

Gene Kim, ITREV, Program Chair13:05:44

...and you couldn't take them out, because they were all coupled together. 😂 There was a talk we accepted on how they managed to decouple/remove one... but he had to pull out at last minutes.)

Balázs Cziráky13:05:34

I would have been keen to hear, but fantastic conversation anyway. :)

Leena Pradhan13:05:20

Teams use dependencies, it is critical to have the SBOMs updated as well. Some teams faced this challenge when log4j vulnerability knocked!

Quinn Daley13:05:51

I have a great many personal stories that sound like Johnny’s too

Quinn Daley13:05:59

especially with Rails admin interfaces


Audience: How do you think the community did with log4j? Did you see the open source libraries you use upgrade quickly?

Gene Kim, ITREV, Program Chair13:05:50

There was a specific error message at GitHub around some gem not being up to date, which guaranteed you wouldn't be able to do rails build ever again. :)

😬 1
Quinn Daley13:05:22

seen this for real with various gems this year alone! (not at Citizens Advice; I should add, since I’m representing them!)


On Dockerhub it was the same. They indrotuce a scanner very quick.

Chris Leeworthy (he/him)13:05:11

Are there tools that can help you build a Software BOM?

👂 2
Vlad Ukis13:05:31

custom-built for us

👍 1
Johnny Bäverås - Synopsys13:05:06

Synopsys Black Duck could probably help you. Happy to discuss further in DM if you want

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:02

@stephen @ldonley @hlynch Huge props for the three of you doing this live... Taking risks to deliver value... kinda timely.

🎉 1
Chris Leeworthy (he/him)13:05:47

I think the biggest thing is the fear of introducing breaking changes

👏 3
Lloyd P13:05:23

compounded by lack of confidence in testing

🎯 2
👍 1
Erik Greathouse13:05:02

At what level of depth should dependency be? If supply chain N-deep what is the best practice?

👀 1
👂 1
Erik Greathouse13:05:57

@stephen Thank you for the thoughtful response gratitude-thank-you

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:27

Yeah.. not "first" tested in production 🙂

👆 1
😂 2
Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:04

What if we called it "final validation with a safely small canary population in production" instead?

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:52

"There's not much value in feature flagging if you don't have the right [telemetry] tools to learn from it" HUGE +1 Pretty hard to learn from a partial rollout if you're still "press and pray" / flying blind and hoping for the best!

👆 2
🎯 1
Chris Leeworthy (he/him)13:05:59

All this makes NodeJS sound like it would be an absolute and utter nightmare 😱


Reminder: The breakout sessions are starting again in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending.

Quinn Daley13:05:51

thank you so much @hlynch @stephen and @ldonley!

Jude Wellington13:05:09

Excellent discussion!

Eva Boquet13:05:12

Thanks so much for the session, really interesting 💯🐝

Ian Richards13:05:33

Really enjoyed this talk. Thank you.

Eva Boquet13:05:00

Come meet the CloudBees Team in Gather to discuss further 🔥 (first private spaces on the left close to the entrance) - Thanks again ! 🙂

Molly Coyne (Sponsorship Director / ITREV)13:05:10

Welcome @dave.karow for our next session's Q&A. Thank you Split! ‼️

Gene Kim, ITREV, Program Chair13:05:37

Thank you @hlynch @stephen @ldonley !!!

Stephen Magill [Sonatype]13:05:12

Thanks @ldonley and @hlynch! Fun as always!


Thank you @stephen @ldonley Great conversation!

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:19

Thanks @mollyc. I'm looking forward to Q&A as we go and after...

💯 1
Harry Simpson13:05:31

Come find us at the go kart exhibition

🐝 1
💯 1
Logan Donley13:05:32

Thanks everyone for joining, these were some great questions. Thanks @stephen for the discussion, I’ve learned some important things about security

🎉 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:33

Here's that testing in prod we talked about in the last session... 🙂

😁 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:31

Time based correlation is for suckers 🙂

Quinn Daley13:05:10

I’ve never used feature flags in this split / ramping up way rather than just binary on/off. How do you mitigate weirdness when your feature (e.g.) changes the way data is stored in the database between the on and off states?

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:27

@quinn.daley additive changes... backwards compatible. If you plan to move from Firstname, Lastname in one column to two columns, you add two new columns and write to all three until you've completed the transition.

Quinn Daley13:05:11

and you write to all three in both the “off” state and the “on” state? so in this case even disabling your feature does technically change the behaviour of your app from before the feature was deployed

Dave Karow (Split - Sr. Progressive Delivery Advocate)13:05:45

No... only writing to new columns from the new code. Hold on and I'll share a link.

Quinn Daley13:05:34

thanks! that’s the weirdness I mean, I guess - like you’ll end up with only 50% of your users having the new data and … what, migrate the data every time you change the percentage to include more people?

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:47

New code writing to all three makes backwards compatibility work. Clean up work (backend job that copies the old column to two new columns) is done at your leisure, not during downtime/maint interval.

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:33

then you can cut over once there's no backlog in the copy job... so it can quickly nail the remaining few as you are ramping up.

Quinn Daley14:05:47

so say someone creates an account some time after you’ve deployed the new code but they hit the “off” path (so they’re running the old code in an environment that contains some users running the new code) - they won’t have the new data and then if you move them into the “on” path they will need that backend job to have been run against them, right? that’s the bit I have trouble getting my head round

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:46

Pro tip: Chapter 6 in the above e-book is a great reference for flags and data-structure changes:

❤️ 1
Quinn Daley14:05:59

I will definitely read this - thank you so much!!!!

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:32

@quinn.daley Give that reading a try and then feel free to reach out to me on LinkedIn if you still have questions.

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:21

Plenty of time left for more questions, so don't be shy all 🙂

Quinn Daley14:05:34

your talk was really interesting! thank you

🙏 1

Thanks @dave.karow! Really interesting and lots to think about to get the most out of feature flagging!

Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:46

If you were late to the party, here's a very snackable video/blog series on Feature Flags, Progressive Delivery and this approach of using data to make better decisions, improve safety and make every iteration more useful:

👍 1
Dave Karow (Split - Sr. Progressive Delivery Advocate)14:05:54

And if you DO know about feature flags and are wondering if there are more things you can do with them, have a look at Joy Ebertz's video:

Molly Coyne (Sponsorship Director / ITREV)14:05:18

gratitude-thank-youA warm welcome to @khyldgaard and @mdahl from Synopsys for our next session's Q&A. Thank you Synopsys! gratitude-merci

🎉 2
🙏 1
jeff.gallimore (CTIO - Excella, he/him)14:05:37

how does this connect to the concept of software bill of materials?

Kim Hyldgaard14:05:05

That's the inventory part. If I'm supposed to provide an SBOM I need to know which components are used. Some components come with a long list of dependencies and suddenly we find ourselves having components used which the software developers didn't actively add themselves... Or didn't know they've added.

thankyou 1
jeff.gallimore (CTIO - Excella, he/him)14:05:56

in your experience, how deep does the dependency graph need to go to effectively manage the risk?

Kim Hyldgaard14:05:30

I'm from security, so I have to say: All the way... In some areas of the application we go to the very end - in others we don't (because in some areas in might be impossible). This decision is based on a risk evaluation of the area. The danger could lie in the last component included...

jeff.gallimore (CTIO - Excella, he/him)14:05:00

as a security professional, i would expect no other answer 😉

😀 2
jeff.gallimore (CTIO - Excella, he/him)14:05:07

so interesting. downgrading is the solution to a risk :rolling_on_the_floor_laughing:

Kim Hyldgaard14:05:57

That would not be the default answer... 😉 😉 😉

💯 1
Lauren Baker - Synopsys14:05:11

If you found our session interesting and informative then don’t forget to register for our where we will be discussing 2022 Open Source Pains & Gains before a unique, original and mind blowing one-of-a-kind mentalist show with two-time Merlin Award Winner for "Mentalist of the Year" Italian Mentalist Luca Volpe. 🤯 🌟 For the lucky first 50 people who register and attend the event, we will be organising special Synopsys@Home goodie boxes to be sent to you - full of fun and useful swag. 🌟 Register now:

upvotepartyparrot 1
🎉 1

Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees.


Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: