I'm in gaming and I have a similar problem on a smaller timeline. XD

At least, the reasons are different. In medical devices, it is about regulation.

And in gaming?

Gaming it's mostly for marketing beats. Keeping a good amount of playerbase overtime is important in live service games, regardless of business model (subscription, microtransactions, etc....)

dont look at me

😂

Join Gather. It’s the center of the room

https://app.gather.town/app/kATASYhmxzaVMLO8/DevOps%20Enterprise%20Summit

HOw does one order a drink?

I walk to the liquor cabinet and pour…

this bar allows to bring yours from your cabinet 🙂

It's 5 o'clock somewhere... 😉

It's always 5 o'clock in Gather

Yep!

It's my very first DOES and I've missed out on the Dockside Bar... probably out of shyness... :face_with_peeking_eye:

Never be shy

The real learning happens at the bar. 😄

No worries @jonathan.mailhot. I struggled sometimes too. This conference is very open and supportive. We strive to break down the "velvet rope" between veterans, speakers and newcomers as much as possible.

100%

welcome @jonathan.mailhot

Don’t forget the PI planning

Do you need a Roadmap?

Normalized pointing FTW!!

> How many hours does a '5' mean?

12.5

😂

That's exactly what we're doing in my team too... super open to anything that could be deemed better.

How often are you merging To the trunk?

I also don't want to just push the problem downstream to feature flags. Sure I delivered to production but if my code is feature flagged off for a month, did that accomplish anything?

My view has been that all changes are not equal, so a broken build for a low value change should not block a high value change. Same for the forced context shifts from mobbing a problem when the team breaks the build - I'd prefer to choose when that context shift happens. In my context I usually have same-day merge, but some last a few days for some teams (which I'm working on).

You’ve proven it didn’t break anything.

It’s better to have it flagged off in prod for a month than in a branch for a month.

OK, good point.

Still doesn't seem like what we want. If my feature has been flagged off for a month, I won't dare turn it on in production. How does one get confidence we can safely start rolling out a feature flagged feature?

Testing would be my guess... but I'm in the same boat as you process-wise..

I love feature flags and especially dark launches. The DevOps Handbook 2e had some excellent notes around that.

Oh, definitely don’t want it off for a month. I’d start asking why that’s happening and what needs to be fixed. It obviously wasn’t something that needed to be built right then. That’s really the main point. CD exposes these problems and makes us find solutions to improve how we work so we can live better lives.

For the original question - are you aware of any resources that might shift my thinking? This is the one area where I disagreed with DevOps Handbook, and I'd welcome being wrong

Can you clarify your opinion? 😄

“so a broken build for a low value change should not block a high value change.” - Why was the low value change first??

Multiple people working on things in the same repo. One low impact, one high impact. I wouldn't want a low impact high complexity change that breaks the build to block a high impact low complexity change. It seems to violate an obsession with time to value and optimizing for cost of delay

The other piece is that I'm not convinced that breaking main/trunk is appropriate. I'd prefer to have the branch merge from main, run the tests and security checks after that merge, and only allow merging to main once the tests pass. That reduces the blast radius while allowing main to be deployable.

Gotcha. First, both of those should be very small changes; a few hours of work at most. Next, the goal of CI is to verify that all of our code integrates cleanly. If the other change is blocking something critical, revert the change. Roll forward isn’t always the answer, but the pipeline must be green. We cannot use workarounds to a broken build. the point is to harden our processes so that we can reliably and safely deliver the next high priority change.

On your other point, CI and TBD don’t mean you cannot branch. It means today’s branch integrates today.

A common struggle is people defaulting to “I merge when the feature is complete”. That’s a behavior that needs to be improved to enable CD.

Tons of good info here https://m.youtube.com/c/ContinuousDelivery

Yeah, I need to work on some of the culture. I've been gradually getting people to break things into smaller units. What I'm evaluating is the tradeoff between merging to main every day, vs merging from main everyday. For an individual branch the resulting code is the same, the only difference is whether main is broken or the branch is. I'd choose a broken branch in that case. The benefits would have to come from having multiple branches merged in daily, vs having multiple longer branches that race to be merged, making the conflict someone else's problem. That still sounds like the benefit is from small feature branches though, not from the merge to main.

If everyone is pulling from the trunk, then there is no integration.

You and I are both pulling from main but main isn’t changing. Then we resolve merge conflicts at the end. 🙂

CI requires that we merge. It also requires we are merging code that is tested and not breaking. It doesn’t require complete features.

Will watch the youtube videos btw, thanks for the resource.

Sure thing!

https://martinfowler.com/articles/continuousIntegration.html

Makes me cry ...

Yeah 😞

Wait, was that the answer? 😂 I can get theatrical with the tears if that works 😁

Manual regression testing or exploratory testing?

"or"? 😭

Manual regression testing means they need to test every single path that has ever been tested before and do it 100% accurately every time. I dare them.

Right. I agree that regression testing should be automated. Exploratory testing can be useful though for uncovering new abuse cases. Not sure it should be a gate to every release though...

Depends on risk appetite, I suppose... #context

They are adding automated testing, so it's less the manual regression testing stuff... but I think it's going to be hard for some teams to believe the pipeline will catch "all" errors. Which of course it won't.

i can normally convince them that they aren’t testing 100% of every path and they say - yeah ok its just a happy path sanity type of regression testing. But thats so arbitrary - how do you choose what should or should not fall in that bucket

I wonder if there's just some psychological comfort in having a human sign off.

https://medium.com/rise-and-fall-of-devops/5-minute-devops-emotional-testing-f8f6bf18364c

@bryan.finster486 is a strong leader for this. Don't let him fool you ... it wouldn't be remotely where it is without him specifically!

Thanks so much!

Hopefully people using that will share their stories and progress

I touch on this later in presentation...which is what I call a CHANGELOG.

This is similar to making a easy to "grok" on content, in a predictable format on a predictable schedule.

Just the other day I had a build problem I hadn't seen before. Searched Slack and found the solution... in a post from two years ago.

Looks like a good opportunity to build a slack app for search 🙂

Yes, one thing we have found valuable is ensuring certain CoP members have time to communicate topics in existing team meetings with leadership. This helps recognize content and topics with peers, and ensures leaders in their space can see value.

https://images.app.goo.gl/pRy8Gkk8NtpofgF26

This is difficult when you find those interested have many responsibilities. The way I have found it valuable is having two different ways of engaging with direct teams in one part of the organization (ex. every other week), and then rolling up sessions in a larger session (monthly). Then trying to make sure there is an easy engagement pattern that could be outside of these sessions. For example, that can be a RFC process that is lightweight (ex. GitHub issues) that try to help share discussions.

I've had intermittent loading throughout the event. But no actual downtime.

I have too, but it's gotten 10x worse in the past half hr. Probably local...

Audio pauses on me all the time.

I assumed it was my devices.

Well, this was going to be on my list of talks to re-watch anyway, so 🙂

Cc @alex @erin

FWIW, @alex this talk has been playing well on my iPad — 

listening in

Hmm. Husband just texted me with "I was paging through a large postgres query result earlier, is it better now?"

So I think the issue may have been on my side. So far John's talk has only had one blip, instead of one per minute

He's not paging results, he's binging Wednesday without telling you. :)

For what it's worth... buffering started happening again around 10 minutes in, so I switched to the prerecorded version of the same video and moved to the same point, it's been ok so far.

Just based on my own observations in the larger group sessions, I was typically only getting feedback from a routine set of people in that larger group - which was inhibiting how much was getting shared. Once you get into a form (small group), you start getting people to at least share and increase the opportunity to speak (it's hard to be quiet in a small group).

What have you heard from other people who were taking part? I'm so curious to what extent they appreciate all this effort to create these obviously important mechanisms, that otherwise normally wouldn't exist!

Something that I do as part of these sessions, is bake in time at the end for a survey to get feedback from the larger group. What I found was that surveys when just sent out via email were not getting much responses, but when baked into the time of the meeting for people to fill out, I would get a higher percentage of feedback. In these surveys, I would just ask: What feedback do you have on doing break-out sessions? And I have consistently received that people want to continue those, they value being able to elect into topics upfront.

Another part I think has been helpful, is connecting back on what was shared from the small group, to the larger group session, and what actions occur as a result. I have seen where people are less engaged in smaller group session if they don't understand how this connect or what actions are taken afterwards.

Thank you!! I’ll be reaching out with some more questions!! Awesome!

Definitely - I can probably get some more data so it is bit more objective.

I guess it helps at not having always the same people talk or express their opinions?

Yes, the more I think you can mix who is sharing, which of course requires engaging with people early on speaking - the better. This often is easier when help reduce the burden on sharing content (short amount of time, don't work on building some new ppt).

We've had "lunch and learns" but at the moment, that's considered your own time, and it's understandable difficult to get people to spend their own precious time in these sessions.

They can start in a bottoms-up / grassroots type effort, but can quickly start highlighting value when you get quotes from people participating in the value. So, it can almost feel like applying product management approaches on your group in how you advertise these back to the leadership in having these groups. They sometimes feel compelled to participate, if you share how this is working in one part of the organization and if they want to also have members join.

I also found it helpful in how I tried to share my thanks back to those who present, which can be from a platform team advertising a new change that they are looking for adoption / feedback. In that thank you, I include both their manager and their executive on why we appreciated this and some concrete reasons on why it was helpful. In several cases, we would see them engage more as a result.

While it might seem very basic, here are some of the notes that I follow in communicating this back to ensure contributors get further recognition and getting awareness with their leadership: https://che55er.io/posts/thank-you-emails/

Excellent, thank you for your insight! Great talk too!

@carl.chesser how often do you use those thank-you emails?

Usually I do these from our larger group sessions, which is more around monthly. These do not take that much time to do, if you do them immediately after the event - else, I find I don't to them consistently. Sometimes I draft them up as they are presenting, so it is sent close to the event.

I have the audio book version and it is great. I highly recommend the book

I have seen it valuable when we find the initiative that isn't as effective when we measure it to the objective, where it takes either a change - but the method is continually refined within the context of the objective (which shares the why).

It's also interesting because managers, especially at higher levels, have to manage people who are much better at figuring out the method than they are. That's where I can see OKR really being powerful.

Right. I have seen it where an initiative of "what we are doing" is the only thing, and when that thing is done, it is viewed as success. However, the outcome is truly achieved (or measured).