Fork me on GitHub

Reminder: Day 2 is starting now โ€“ opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.

Cindy Lin - LaunchDarkly14:10:34

@michael.gillett @dparzych adding you both for your sessions!

๐Ÿ‘ 1
thankyou 1
Cindy Lin - LaunchDarkly15:10:38


๐Ÿ‘‹ 1
Michael Gillett16:10:04

@clin I am using a different account than before ๐Ÿ˜„

๐Ÿฅฒ 1
Cindy Lin - LaunchDarkly16:10:31

good to know! :rolling_on_the_floor_laughing: thanks, michael!

๐Ÿ‘ 1
Molly Coyne (Sponsorship Director / ITREV)16:10:51

๐ŸŒŸWelcome @michael.gillett181 for our next session's Q&A. Thank you to our Premier Sponsor #xpo-launchdarkly! ๐ŸŒŸ

๐Ÿ‘‹ 1
Dawn Parzych16:10:37

If anybody is interested in reviewing @michael.gillett181 book let me know I can get you an advance copy.

โค๏ธ 2
Anil Sadhu16:10:41

I would like to get one.

Anil Sadhu19:10:55

@dparzych Do I have to submit the request anywhere else?

Michael Gillett16:10:29

Any questions so far? I hope it's all making sense

Ryan Taylor - Senior Geospatial Developer, GISinc16:10:04

How long should a feature flag exist? For the lifetime of the app? Prob depends on the feature flag. What about for experiments?

Bryan Finster - Defense Unicorns (Speaker)16:10:52

Until the feature has been validated.

Bryan Finster - Defense Unicorns (Speaker)16:10:10

If you want to keep the feature and itโ€™s stable, delete the flag.

๐Ÿ‘ 1
Bryan Finster - Defense Unicorns (Speaker)16:10:20

I recommend creating a task on the epic.

Michael Gillett16:10:58

It's a great question, there are two types of flags: permanent and temporary. Most flags are likely to be temporary. The flags can be used to help deliver new code safely to production, once enabled for all customers the flag can be removed. Experiments fall within the temporary use case. When I spoke about Switches they are a good use of permanent toggles as you never know when you might want the option to turn off a feature. Another use of permanent flags would be for an entitlement scenario

๐Ÿ‘ 1
Michael Gillett16:10:45

I have tips on how to keep on top of temporary feature flags, they can get difficult to manage both within a feature management tool and within the codebase itself. These tips cover separating flags within LaunchDarkly through to how to work with tickets on something like Jira to manage the work involved when cleaning up a flag

Bryan Finster - Defense Unicorns (Speaker)16:10:31

This is excellent.

โค๏ธ 2
Bryan Finster - Defense Unicorns (Speaker)16:10:49

I was just talking to a team about this at SpaceCAMP. Iโ€™m definitely sharing this talk with them.

โค๏ธ 4
Michael Gillett16:10:17

Excellent! Glad you are enjoying this!

Dawn Parzych16:10:34

We have a blog on trunk based development that may be helpful as well:

โค๏ธ 3
Bryan Finster - Defense Unicorns (Speaker)16:10:59

Iโ€™ve never worked where I could use your product, but I like it. ๐Ÿ™‚

๐Ÿ‘ 1
Pedro Jordan16:10:28

I will be nice to define kind of "criterias" to check how much testing do I need to each new feature

Ferrix Hovi - Head of DevOps - Siili16:10:26

@bryan.finster486 has some nice combinatorics of the DORA basics as well as others.

Michael Gillett16:10:41

This is very dependant on the feature and the type of testing required for it. Do you mean for an experiment?

Ferrix Hovi - Head of DevOps - Siili16:10:56

Also, the amount of testing is not important since testing does not add value. You want to understand the risk of failure and use that to calibrate. Also, having the tester mindset present in early planning, it will be likely planned in a less risky way.

๐Ÿ‘ 1
Michael Gillett16:10:14

For experimentation it is key to understand the success metric, what is it that would prove the test valid. If you approach the solution via an hypothesis then it can help clarify what would prove whether the hypothesis was successful or not

Pedro Jordan16:10:20

agreed, thank you !

Pedro Jordan16:10:55

thank you !! really nice talk

๐Ÿ‘ 1
Derek Bissinger16:10:55

Great topic!

๐Ÿ‘ 1
Dave Mangot - DevOps transformation professional16:10:15

This is great @michael.gillett181. My Service Delivery Assessment (based on the DORA metrics) gives people much higher scores when they are using Feature Flagging. Totally agree. Bravo. ๐Ÿ‘

๐Ÿ‘ 1
Michael Gillett16:10:04

This is great to hear! Thanks

Cindy Lin - LaunchDarkly16:10:30

thanks, michael! great talk

๐Ÿ‘ 1
Anil Sadhu16:10:09

Great info Micheal!

๐Ÿ‘ 1
Michael Gillett16:10:14

Thank you everyone! Let me know if you have any other questions, I can talk about this subject for a long time

๐Ÿ˜Š 1
๐Ÿป 1

Great session Michael!

๐Ÿ‘ 1
Molly Coyne (Sponsorship Director / ITREV)16:10:44

๐Ÿ’กWelcome @anders.wallgren for our next session's Q&A. Thank you #xpo-cloudbees! ๐Ÿ’ก

Denver Martin, Dir DevSecOps, he/him16:10:27

@anders.wallgren we are rolling out RBAC this is key for simplifying Audit evidence... will be so glad when it is up and running...

Mark P17:10:00

Interested to hear what tools are used to generate these evidence and approval reports?

Steve Beal17:10:54

... probably CloudBees? ๐Ÿ™‚

Tony Wilmer17:10:35

The data is automatically captured while executing our Release Orchestration capabilities.

Tony Wilmer17:10:38

Data can also be gathered from external sources. eg. Jira, Git, ServiceNow based on plugins and tasks performed in RO.

Tony Wilmer17:10:55

@mark.privette - If you have any questions or would like to see a demo of CloudBees CD/RO, just let me know.

Tony Wilmer17:10:25

Duration data is critical to doing trend analysis of your software delivery process.

Denver Martin, Dir DevSecOps, he/him17:10:47

Thanks @anders.wallgren good presentation...

๐Ÿ‘ 3
Molly Coyne (Sponsorship Director / ITREV)17:10:33

โšกWelcome @erin.jones who will be moderating for today's VendorDome Q&A between @dparzych and @rob.jahn. A big thank you to #xpo-launchdarkly and #xpo-dynatrace for battling it out :rolling_on_the_floor_laughing: in today's exciting session, Beyond the Buzzwords! โšก

Brian Smith17:10:16

Is this actually live, or is the is a pre-recorded sesssion?

Brian Smith17:10:30

SRE - Drink!

๐Ÿป 1
Brian Gallop17:10:43

Live makes the drinking games more fun, eh?

Molly Coyne (Sponsorship Director / ITREV)17:10:00

๐Ÿ”ฅFeel free to share any questions here in this channel now and @erin.jones will ask Dawn and Rob in real time!

Dipesh Bhatia17:10:29

Architecture changes (Monolith - MicroService) and feature developing at the same time, is killing us

Ryan Taylor - Senior Geospatial Developer, GISinc17:10:43

Most of the devops discussions and practices seem to relate more directly to product based orgs. Do these discussions and practices change significantly with project based orgs (e.g., contractors delivering solutions for clients over possibly shorter timelines such as weeks or months).

Meghan Glass - PrdMgr Best Buy17:10:45

Projects, by definition have an "end state" where it's "handed off" to operations

Ryan Taylor - Senior Geospatial Developer, GISinc17:10:38

In my org, we have projects ranging from weeks to years. Some have portions of devops, some none to speak of. Some have a "hand-off" but for some the ops teams is our team. The systems on which we deploy are not often owned by us, but the client. Some cloud, some on prem. But I'm curious how those distinctions matter.

Meghan Glass - PrdMgr Best Buy17:10:39

If you don't own the system I would posit that (without knowing the details of your space) that you're using the system as an operations tool, and when the tool isn't working for what you need it to do, these asks are treated as "projects" to get those features but ultimately the "need" is coming from value you're trying to provide to customers, and essentially is a product/service itself and therefore, even if not being treated as such, is a product.

Ryan Taylor - Senior Geospatial Developer, GISinc17:10:58

that's definitely true for a couple of the "projects" I am on, which have been in place in various forms for 10+ years.

โค๏ธ 1
Frotz Faatuai (Cisco IT - he/him)17:10:16

Those words have been placed on my badge picture and sent to me as the โ€œinside jokeโ€ beforeโ€ฆ

Frotz Faatuai (Cisco IT - he/him)17:10:32

Feature Flags + RBAC has been a great trick that Iโ€™ve used over the yearsโ€ฆ Definitely constrain experimental code to those with that RBAC authorizationโ€ฆ

Laura Henry - American Airlines [she/her]17:10:48

Whenever I bring up testing in production (with safety nets), I get an immediate reaction of โ€œno, it is too riskyโ€ How do you get people to just try it?

Laura Henry - American Airlines [she/her]17:10:54

Thank you @dparzych - good point about deploy vs. release

Laura Henry - American Airlines [she/her]17:10:15

Thinking about those conversations, I think that is where the struggle is

Michael Winslow17:10:53

We used to have a fun program called "First 48" after the TV show. But it meant we only rolled out to 48 accounts for testing... then grew to 200, then 2000, etc.

โค๏ธ 1
Frotz Faatuai (Cisco IT - he/him)17:10:58

Feature Flag with a very well defined RBAC control group. Only those people who are in that group get the Feature Flagged capability. Donโ€™t talk about it as โ€œtestingโ€โ€ฆ Talk about it as experimental and restricted access code.

Scott Brock17:10:41

Totally agree "Testing' is maybe not the right term

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:05

Pitching it as limited early access to new features can work well!

๐Ÿ’ฏ 2
Scott Brock17:10:35

It's not testing functionality it's test usability

๐Ÿ’ฏ 2
Meghan Glass - PrdMgr Best Buy17:10:19

It's testing something different

Dipesh Bhatia17:10:45

Beautiful: deploying code and release are two separate things !!

Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:04

Decoupling the concepts of release and deployment is so critical! It can take a lot of effort to socialize that they are separate activities, but he payoff in being able to release new features when everyone is around to support them is amazing.

Meghan Glass - PrdMgr Best Buy17:10:45

That separation also enables a better development runway

Meghan Glass - PrdMgr Best Buy17:10:34

we can have features deployed and tested with real users in production with feedback months before a "full launch"

๐Ÿ’ฏ 3
Joe Waid - Manager, Delivery Engineering - Columbia Sportswear17:10:46

Itโ€™s a great tool if something goes wrong too! Flip the toggle, turn off the feature is much easier than re-deploy, or rollback.

๐Ÿ‘ 3
โค๏ธ 1
Dave Mangot - DevOps transformation professional17:10:10

โ€œPeople with targets and jobs dependent upon meeting them will probably meet the targets - even if they have to destroy the enterprise to do it.โ€ - Deming

๐Ÿ˜‚ 3
Dave Mangot - DevOps transformation professional17:10:41

I see what you did there @genek ๐Ÿ˜‰

Gene Kim, ITREV, Program Chair17:10:06

I'm embarrassed that it wasn't intentional โ€”ย I guess it was a common cause variation, @dave . (Thank you thank you, I'm here all week.)

๐Ÿ˜ 1
Frotz Faatuai (Cisco IT - he/him)18:10:03

Thereโ€™s a good callout - tagging architecture to focus who should be notified gets botheredโ€ฆ Always a problem.

๐Ÿ’ฏ 1
Frotz Faatuai (Cisco IT - he/him)18:10:19

We built out immediate (on-call) and bulk-notification (email; off-shift) for all alerts owned by all teams (600+) with two levels of notifications.


"There is a cost to freedom vs. standardization" so true

Dipesh Bhatia18:10:47

@genek Curious Question - Is DevOps principles used for the internal working of the DevOps Enterprise Summit ,as well ?

Malcolm McAlpin18:10:16

Thank you!!

๐Ÿ’ฏ 1
Alvin Crespo18:10:49

Youโ€™re not off the air.

Malcolm McAlpin18:10:50

Old Hat

๐Ÿ˜† 1
Meghan Glass - PrdMgr Best Buy18:10:50

Now you're off line

โ˜๏ธ 1
Alvin Crespo18:10:59

yup, all good now ha

Erin Jones18:10:53

That was just us "testing in production." ๐Ÿ˜†

๐Ÿ’ฏ 4
upvotepartyparrot 2
Rob Jahn18:10:26

Thanks all for listening in, we did have a good time and time flys when you are having fun!

๐Ÿ‘ 2
Molly Coyne (Sponsorship Director / ITREV)18:10:17

โœจWelcome @tashfeen_mahmood and @dominica for our next session's Q&A. A big thank you to our sponsor#xpo-tasktop!

Lauren Kaye (Tasktop)18:10:02

@mollyc the sound isn't working for me. Is it just me or same for everyone?

Meghan Glass - PrdMgr Best Buy18:10:21

I even refreshed the page to make sure mute was off

Mark Esser18:10:37

Refreshed, still no sound.

Lauren Kaye (Tasktop)18:10:38

Me too. @mollyc please can this session be restarted

Mark Esser18:10:24

Sound just came through.

Molly Coyne (Sponsorship Director / ITREV)18:10:40

Thanks so much for your patience everyone!

Molly Coyne (Sponsorship Director / ITREV)18:10:39

We are working on the audio! ๐Ÿ™‚

Meghan Glass - PrdMgr Best Buy18:10:55

Have you turned it off and back on again? ๐Ÿ˜„

โค๏ธ 2
Meghan Glass - PrdMgr Best Buy18:10:09

It's just the joke from Track 1

Lauren Kaye (Tasktop)18:10:41

Started working but we missed the first few mins. Will the session be restarting

Molly Coyne (Sponsorship Director / ITREV)18:10:45

โšกDon't forget: The video will be in our Video Library after the session so you can hear the first minute of the session that was muted. :star-struck:

โค๏ธ 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:04

"Change mgmt mindset was to avoid change" @tashfeen_mahmood

Tashfeen Mahmood18:10:26

Our releases were large and complex... so we wanted to avoid the pain

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:12

Interesting evolvement of change process from freeze to slush ๐Ÿ™‚

Tashfeen Mahmood18:10:27

That is still my favorite story from what we have been able to accomplish... A conservative Insurance Company starts Open Enrollment in the middle of the day during a freeze

โค๏ธ 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:50

High seasonal demand impacts flow - whether retail (black Friday, Cyber Monday), Insurance (open enrollment, EoY requests), or sports (pre-season, playoffs).

Tashfeen Mahmood18:10:03

this explained to us why those Production Freezes and slushes were not working

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:32

The more wip, the harder it was to deliver and address tech debt

๐Ÿ’ฏ 1
Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:09

Amazing focus on biz goals - improve customer retention rate - tied wip (Flow Load) to biz results ๐Ÿ’ช

Tashfeen Mahmood18:10:55

great way to get your business involved, right? Focus on their goals

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:56

Experimentation is paramount during high seasonal demand - can gain the biggest learnings.

Scott Heaberlin18:10:58

Thanks for the talk, @tashfeen_mahmood and @dominica ๐Ÿ‘ Were experiments solely framed as / aimed at IT practice improvements, or were there payer biz experiments crafted with the same structure and syntax?

Tashfeen Mahmood18:10:39

Hey @scott.heaberlin - Just to further elaborate... We experimented with IT Improvements... But those IT Improvements were tied to Business Results so we were reducing WIP for example by only prioritizing the Business Goals that had been identified

Tashfeen Mahmood18:10:56

so in a way it is Business Centric Improvement also

๐Ÿ‘ 1
Scott Heaberlin18:10:56

totally get it. Thank you for following up!

Molly Coyne (Sponsorship Director / ITREV)18:10:05

upvotepartyparrotWelcome @mlivermore for our next session's Q&A. Thank you to our sponsor #xpo-pagerduty! upvotepartyparrot

๐Ÿ‘‹ 1
๐Ÿ‘ 1
Tashfeen Mahmood18:10:29

the experiments are focused on IT Improvements that lead into building more agility so we can deliver better/faster during our Open Enrollment season

thankyou 1
Joy from Stack Overflow18:10:40

@dominica First, I really enjoyed your book Making Work Visible, been implementing some of the ideas with my team (% of items that are work in progress has been hugely helpful). Iโ€™ve been wondering about this thoughโ€ฆ Iโ€™ve read that up to spent on invisible work - those things that arenโ€™t directly traceable back to code commits or other productivity metrics. Interpersonal components like helping teammates or asking for help or discovery components like experimentation or hunting down solutions. Given the importance of this type of work and how much of it is invisible, what are the strategies or approaches that you use to make this work visible?

Dominica DeGrandis, Author - Making Work Visible, Tasktop18:10:13

Hi @jcicmanliuzzo Important invisible work is often related to improvements in process, ppl or tools - things that - when invested in - improve future capability. That type of work (debt work) should have sufficient capacity allocated for it. Making debt work visible can then drive necessary convo on what the strategy is/should be.

Joy from Stack Overflow19:10:09

I hadnโ€™t thought of it in terms of debt workโ€ฆ

Traci Myers19:10:05

Ha! Still waiting Rob? ๐Ÿ˜ƒ

Dakota Wandro19:10:06

From a network security perspective, how do customers typically secure Rundeck so that it is accessible by PagerDuty without leaving a potential security hole by exposing a system externally that can be used to execute commands against your systems?

Dakota Wandro19:10:08

Also, how do you deploy Rundeck in a PCI environment? Separate instance + segmentation?

Matt Livermore19:10:05

With PagerDuty and William Hill both using AWS we were able to use AWS EventBridge

๐Ÿ™ 1
Matt Livermore19:10:32

This significantly reduced the attack surface.

Matt Livermore19:10:43

Single instance with access controlled via SSO and RBAC. Engineers are able to run specific scripts which are held off-server in a repo that is subject to security audit. Engineers have no direct access to production servers.

Matt Livermore19:10:32

The execution of scripts always ties back to an incident/request that is recorded in Jira and/or ServiceNow. This tracks who ran the script, captures any outputs that are returned to the original ticket for full audit trail.

Dakota Wandro19:10:30

Very insightful, thank you!

Matt Livermore19:10:18

My pleasure Dakota, enjoy the rest of the summit. I don't know about you but I'm learning a lot.

๐Ÿ’ฏ 2
Matt Livermore19:10:48

@mollyc thanks for having us!

โค๏ธ 1
๐Ÿ™Œ 1
Molly Coyne (Sponsorship Director / ITREV)19:10:16

๐Ÿ‘:skin-tone-2:Welcome @mmeera for our next session's Q&A. Our special thanks to our sponsor #xpo-synopsys ๐Ÿ‘:skin-tone-2:

๐Ÿ‘ 1
Meera Rao19:10:11

Glad to answer any questions.

jeff.gallimore (CTIO - Excella, he/him)19:10:22

does code dx integrate with common โ€œwork managementโ€ tools, like jira and such? i would want the findings from the tool injected into and connected to the backlog and wip for the team.

๐Ÿ‘ 1
Meera Rao19:10:44

Yes, it does.

๐Ÿ‘ 1
Pedro Jordan19:10:12

it makes totally sense to not have the same scans and steps for every change

๐Ÿ’ฏ 2
๐Ÿ‘ 1
jeff.gallimore (CTIO - Excella, he/him)19:10:47

what sort of access control/permissions can you set to allow some people to make config/rule changes and prevent them by others? iโ€™m thinking like a developer who comments out a test to get the build to pass :rolling_on_the_floor_laughing:

Meera Rao21:10:18

Everything is logged, so won't be able to hide who did what.

๐Ÿ‘ 1
thankyou 1
Ferrix Hovi - Head of DevOps - Siili19:10:15

@jeff.gallimore if user.isHuman() { return politeAnswers.accessDenied; }

Molly Coyne (Sponsorship Director / ITREV)19:10:07

โœจWelcome @matts for our next session's Q&A. IT Revolution thanks #xpo-adaptavist for their support! โœจ

๐Ÿ‘‹ 2
๐Ÿ˜บ 1
jeff.gallimore (CTIO - Excella, he/him)19:10:55

โ€œaccess to environmentsโ€ is something i still see a lot of organizations struggle to improve. they apply their old โ€œrequest hardwareโ€ mentality and process to a cloud-based world.

Glenn Wilson, Author of DevSecOps19:10:31

Hello @matts . Fancy seeing you here ๐Ÿ™‚

Matt Saunders19:10:13

Hiya Glen - I just bookmarked your Deming podcast with @jwillis to listen to tomorrow! ๐Ÿ™‚

Glenn Wilson, Author of DevSecOps20:10:09

I hope you enjoy it. I love talking about some of the theory behind systems thinking and its role in security.

Matt Saunders20:10:05

Deming's points just keep coming back again and again - they've really stood the test of time

Glenn Wilson, Author of DevSecOps20:10:54

absolutely. Even the 5 ideals of DevOps contain hints of the 4 principles of SoPK

Glenn Wilson, Author of DevSecOps20:10:28

โ€œScope is bigger than you thinkโ€ = appreciation of a system

Matt Saunders20:10:35

always... I find it really challenging that people don't allow themselves to think that widely

Matt Saunders20:10:42

but then when they do, the light dawns ๐Ÿ™‚

Matt Saunders20:10:43

you're welcome Malcolm ๐Ÿ™‚

Joe Arrowood20:10:45

Good talk @matts, thank you!

Matt Saunders20:10:06

Thank you Joe! Just as nerve-wracking watching it on the recording ๐Ÿ˜„

Glenn Wilson, Author of DevSecOps20:10:00

Thanks Matt. I enjoyed that talk

Matt Saunders20:10:17

Cheers Glenn - see you soon no doubt!

Glenn Wilson, Author of DevSecOps20:10:03

Definitely - live events are back!! DSO-LG is live in London in November

Matt Saunders20:10:31

Perfect... still need to get you to sign my copy of your book ๐Ÿ˜„

๐Ÿ‘ 1
Glenn Wilson, Author of DevSecOps20:10:02

We can sort that out. No problem


Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees.