This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-05-11
Channels
- # ask-the-speaker-track-1 (130)
- # ask-the-speaker-track-2 (99)
- # ask-the-speaker-track-3 (94)
- # ask-the-speaker-track-4 (172)
- # bof-arch-engineering-ops (2)
- # bof-leadership-culture-learning (2)
- # bof-project-to-product (23)
- # bof-sec-audit-compliance-grc (2)
- # discussion-main (730)
- # faq (6)
- # games (7)
- # games-self-tracker (1)
- # gather (28)
- # happy-hour (17)
- # help (34)
- # hiring (11)
- # networking (8)
- # project-to-product (1)
- # summit-info (28)
- # summit-stories (1)
- # xpo-bmc-ami-devops (1)
- # xpo-cloudbees (1)
- # xpo-cockroachlabs (2)
- # xpo-github-for-enterprises (1)
- # xpo-gitlab-the-one-devops-platform (3)
- # xpo-itrevolution (4)
- # xpo-launchdarkly (3)
- # xpo-lightstep-observability-incidentresponse (1)
- # xpo-linearb-automate-dev-team-improvement (9)
- # xpo-planview-tasktop (2)
- # xpo-snyk (1)
- # xpo-split (18)
- # xpo-stackoverflowforteams (2)
- # xpo-synopsys-sig (4)
- # xpo-tricentis-continuous-testing (2)
Reminder: Get yourself in front of your browser and into #ask-the-speaker-plenary for the opening remarks. We’re kicking off Day 2 in 15 minutes at 10am BST! https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Reminder: Day 2 is starting now – opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.
Reminder: The breakout sessions are starting in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
💡Welcome @carmen.deardo and @naomi.lurie for our next session's Q&A. Thank you for the continued support Tasktop! 🌟
Some of the challenges I still see in 2022, is lack of automated testing, CICD, causing much local optimizations. Curious if others see similar.
oh yes! automated testing is still very much in the PowerPoint stage, and not even on the early slides.
The cadence of this talk is a stark contrast to @mik talk. LOL.
Even with Product Owner being a role for 20+ years, I still see a struggle with defining and measuring value. Still seems very activity based
It surprises me to see Stage 3! Would have expected Stage 1 or 2. That is good news. Guessing are hitting the systemic bottlenecks like upfront planning and prioritization or a release process. Typically the development is the quickest part of the whole life cycle! Love the tip on "from the customer's perspective"!
Delays due to dependencies - how to say Value Stream without saying Value Stream 😂
“Don’t rush until you’ve developed good, solid patterns” - I tell my kids this as well! A good life lesson.
@carmen.deardo @naomi.lurie related to expansion - do you find that while some change may be under teams control, that some of the major/systemic bottlenecks need more support from the organization somehow?
Yes @katharine.chajka - the power of experimentation is that you now have actual data/experience that demonstrates that a given process in a portion of the value stream (e.g. releasing on demand) can be done effectively and with high quality. This helps highlight the “systemic” processes that can be improved.
Do you often see cases where a product owner is an owner only by name, but lacks any mandate to make meaningful decisions? And so the decisions need to be made one/two levels up?
Yes @rwiankowski this is a common “anti-pattern” that we observe. This really needs to be an experiment where the product team has control over their journey (with some guard rails). The team needs control to focus on improving flow and business outcomes.
It is so fascinating to see the investment made in agile roles or practices but not in removing delays or bottlenecks to delivery, I imagine due to lack of visibility
if System Thinking isn’t applied, innovation can quickly turn into a local optimization even with best intents.
@katharine.chajka lack of visibility or lack of true understanding of what flow is? or maybe both?
yes, likely both. No fault of anyones as it seems like they are not talked about for some reason
idd. Seems like common sense, right? But then again, simple solutions are rarely easy to find.
reminds me of my study abroad program's mantra: the journey is the destination
Thank you! We’d love to hear your voice by taking the assessment and building out the picture: http://tasktop.com/assessment
Thanks for the talk, we get pretty good scores on the Anti-pattern slide… 🙂 Will check out the assessment!
✨Welcome @jp for our next session's Q&A. Thank you CockroachDB! 💥
Just a special note. Our speaker Jim Walker was unavailable to join today, so y'all are stuck with me haha. Follow Jim on Twitter @jaymce for more db content.
Yes, there are different read and write levels for every database, but this is a good thing. As a developer, you get to choose what database works best for your use case, and that's pretty cool.
Not too much but I am happy to chat about it! (I am a geek when it comes to CAP)
:star-struck:Welcome @hlynch who will be moderating for today's VendorDome Q&A between @stephen and @ldonley. Thank you Sonatype and CloudBees! 🌟
that’s the good thing about having a video library 😄 https://videos.itrevolution.com/watch/707352050/
this is a big factor in getting people comfortable with continuous deployment of changes to production
Trivia fact: • On this conference page, the Slack widget uses a feature flag — we weren’t sure the we’d use it for plenary track
How do you deal with this when you have legacy code that maybe doesn’t have nicely decoupled components?
@genek imagine a time not so long ago, where you might debate recompiling your plenary UI after day one to make a change there... what could possibly go wrong :man-shrugging: ?
Haha. Changing things are super terrifying still. 🙂 (I’m still amazed we’ve pushed client changes a couple of times during conference — in fact, the slack widget can be restarted now independently!)
on the subject of having quick access to remediation/rollback - when is the right time to remove a feature flag and all the associated “old” code from your codebase? and how do you ensure the “old” code is removed along with the flag?
@hlynch Question for panel: “it’s amazing how feature flags have taken off in the industry — what else can we learn from dynamic runtime environments, where we enable the runtime to truly dynamic? what else could we enable, vs things we view as static right now?”
@quinn.daley one pattern that is useful is to flag at logical module points, not line-by-line. With clean, natural boundaries, you don't have to go through with a fine-toothed comb to find the things to remove later. This blog post, written by one of our Front End and Back-End for Front End lead devs, uses front end JavaScript as an example, but the pattern applies to any codebase. https://www.split.io/blog/manage-feature-flags-javascript/
yeah, we definitely do this but of course modules also depend on other modules etc!
Great. Sometimes folks want to get very "surgical" with putting a flag at the exact line of code in a module that does the thing... that makes it trickier. Check out David B's example where the other modules are abstracted away enough that they don't know or care what the parts that change/get removed are doing.
we have a policy of putting flags at the “entry point” wherever possible, but it’s not universally adopted
Entry point was exactly the advice I used to give... then David B gave a talk about the modular way he does it and I realized his way was WAY less mentally demanding. After that talk, we convinced him to put it in that blog. They've even built some tooling that makes the modules approach faster/less risky, such as this: > • Adding a feature flag becomes easy. We have a script that will take an existing component, make a second copy of it and set up the toggler function that wraps both. These two benefits are probably my favorite: > • Reasoning about code is very easy. Because these are at “interface boundaries,” you do not have to contend with mixed logic in one routine. From the consumer’s point of view, you don’t know that there are multiple variations. From the consumee’s standpoint, you still do not know that there are other variants of yourself that could be run. This means that when working on one of the component variations, you can focus your attention on that one feature. > • This also makes unit testing dramatically easier. Because the blocks of code being developed are completely distinct, it also means that the unit tests are completely distinct.
including dependency auditing in the CI pipeline so the build fails if there’s a disclosure against one of the deps certainly helps!
I think @topo.pal used JFDI (just do it, the f is silent) in a talk. @stephen's comment about just do dependency management reminded me of #jfdi
I love it when I see engineering therms ruthlessly reducing the # of OSS dependencies — there's a great story of people eliminating Jackson for JSON parsing, http-commons, etc. Significant and concerted scrutiny and reduction of components. (It wasn't so long ago when I thought 'twas a bit... hysterical.)
@hlynch I'd love to hear about stories of how engineers have successfully "rationalized their dependencies" — what things should inform what components should and shouldn't be used? I love stories of engineering teams significantly reducing dependencies — is that a pipe dream? Thx!!!
What differences have you seen in how various tools help you remediate/address security risks?
something we absolutely look for in tooling is the ability to incorporate them into the ci/cd pipeline.
"such as the special circle of hell that is all the different versions of the Jackson JSON libraries)." Cc @stephen :)
I thought that was the grocery store the day before Thanksgiving for you?
We have just got AWS Inspector up and running and had the shock of seeing just how much stuff needed fixing! The important thing is it made the weaknesses visible so we could do something about them.
maybe we need to train good security people like BCP/NTT did with their training academy? if anyone does this, I wanna hear the talk
It was astonishing to see how they created a community around them, and even with their customers!!
How and where are people recording metrics for resolving dependancy issues from discovery to resolution?
just because we can use all the options doesn't mean we should. noted
Ha! 15 XML parsing libraries. And Jackson. :) 15 different logging frameworks.
Back in my R&D days, 10 years ago, i even made a habit out of picking new XML parsing libraries for every integration i did. To find which one i found easiest to work with. Did i ever upgrade them? of course not! 😉 I wouldnt mind a time machine to go back and have a stern talk to myself.
3 different JavaScript UI frameworks in our app. (Which I didn't think was actually possible?? React, Backbone, and something else... apparently, not that uncommon...)
...and you couldn't take them out, because they were all coupled together. 😂 There was a talk we accepted on how they managed to decouple/remove one... but he had to pull out at last minutes.)
Teams use dependencies, it is critical to have the SBOMs updated as well. Some teams faced this challenge when log4j vulnerability knocked!
Audience: How do you think the community did with log4j? Did you see the open source libraries you use upgrade quickly?
There was a specific error message at GitHub around some gem not being up to date, which guaranteed you wouldn't be able to do rails build ever again. :)
seen this for real with various gems this year alone! (not at Citizens Advice; I should add, since I’m representing them!)
Holy cow, here it is! https://devopsenterprise.slack.com/archives/C0158J0FU6A/p1652188239452129
Synopsys Black Duck could probably help you. Happy to discuss further in DM if you want
@stephen @ldonley @hlynch Huge props for the three of you doing this live... Taking risks to deliver value... kinda timely.
I think the biggest thing is the fear of introducing breaking changes
At what level of depth should dependency be? If supply chain N-deep what is the best practice?
Yeah.. not "first" tested in production 🙂
What if we called it "final validation with a safely small canary population in production" instead?
"There's not much value in feature flagging if you don't have the right [telemetry] tools to learn from it" HUGE +1 Pretty hard to learn from a partial rollout if you're still "press and pray" / flying blind and hoping for the best!
All this makes NodeJS sound like it would be an absolute and utter nightmare 😱
Reminder: The breakout sessions are starting again in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Come meet the CloudBees Team in Gather to discuss further 🔥 (first private spaces on the left close to the entrance) - Thanks again ! 🙂
❗Welcome @dave.karow for our next session's Q&A. Thank you Split! ‼️
Thanks @mollyc. I'm looking forward to Q&A as we go and after...
Thanks everyone for joining, these were some great questions. Thanks @stephen for the discussion, I’ve learned some important things about security
Here's that testing in prod we talked about in the last session... 🙂
Time based correlation is for suckers 🙂
I’ve never used feature flags in this split / ramping up way rather than just binary on/off. How do you mitigate weirdness when your feature (e.g.) changes the way data is stored in the database between the on and off states?
@quinn.daley additive changes... backwards compatible. If you plan to move from Firstname, Lastname in one column to two columns, you add two new columns and write to all three until you've completed the transition.
and you write to all three in both the “off” state and the “on” state? so in this case even disabling your feature does technically change the behaviour of your app from before the feature was deployed
No... only writing to new columns from the new code. Hold on and I'll share a link.
thanks! that’s the weirdness I mean, I guess - like you’ll end up with only 50% of your users having the new data and … what, migrate the data every time you change the percentage to include more people?
New code writing to all three makes backwards compatibility work. Clean up work (backend job that copies the old column to two new columns) is done at your leisure, not during downtime/maint interval.
then you can cut over once there's no backlog in the copy job... so it can quickly nail the remaining few as you are ramping up.
so say someone creates an account some time after you’ve deployed the new code but they hit the “off” path (so they’re running the old code in an environment that contains some users running the new code) - they won’t have the new data and then if you move them into the “on” path they will need that backend job to have been run against them, right? that’s the bit I have trouble getting my head round
Pro tip: Chapter 6 in the above e-book is a great reference for flags and data-structure changes:
@quinn.daley Give that reading a try and then feel free to reach out to me on LinkedIn if you still have questions. https://www.linkedin.com/in/davekarow/
Plenty of time left for more questions, so don't be shy all 🙂
Thanks @dave.karow! Really interesting and lots to think about to get the most out of feature flagging!
If you were late to the party, here's a very snackable video/blog series on Feature Flags, Progressive Delivery and this approach of using data to make better decisions, improve safety and make every iteration more useful: https://www.split.io/blog/progressive-delivery-safe-at-any-speed-playlist-blogs/
And if you DO know about feature flags and are wondering if there are more things you can do with them, have a look at Joy Ebertz's video: https://www.split.io/resources/flagship-2022/feature-flag-use-cases-you-havent-heard-about-yet/
A warm welcome to @khyldgaard and @mdahl from Synopsys for our next session's Q&A. Thank you Synopsys! 
how does this connect to the concept of software bill of materials?
That's the inventory part. If I'm supposed to provide an SBOM I need to know which components are used. Some components come with a long list of dependencies and suddenly we find ourselves having components used which the software developers didn't actively add themselves... Or didn't know they've added.
1in your experience, how deep does the dependency graph need to go to effectively manage the risk?
I'm from security, so I have to say: All the way... In some areas of the application we go to the very end - in others we don't (because in some areas in might be impossible). This decision is based on a risk evaluation of the area. The danger could lie in the last component included...
as a security professional, i would expect no other answer 😉
so interesting. downgrading is the solution to a risk :rolling_on_the_floor_laughing:
If you found our session interesting and informative then don’t forget to register for our https://synopsys.zoom.us/meeting/register/tJIsc-moqDItE93NZ8LqeO9Uu8x1S4ZLBH9D where we will be discussing 2022 Open Source Pains & Gains before a unique, original and mind blowing one-of-a-kind mentalist show with two-time Merlin Award Winner for "Mentalist of the Year" Italian Mentalist Luca Volpe. 🤯 🌟 For the lucky first 50 people who register and attend the event, we will be organising special Synopsys@Home goodie boxes to be sent to you - full of fun and useful swag. 🌟 Register now: https://synopsys.zoom.us/meeting/register/tJIsc-moqDItE93NZ8LqeO9Uu8x1S4ZLBH9D
1Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png
