This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
@biswanath.basu @jennifer.hansen, were you able to correlate the success in your metrics to an improved NPS as well?
@rakesh.goyal -that's an amazing Journey from COTS product to cloud & DevOps
Yes. It was hard to envision up front the entire journey. I have learned over the years, there needs to be a belief in the journey and you can find your way to the goals. Many hurdles come up however, if you know going forward is the only option then you always find a way. Either engineering and many times even how you envision the product itself!
@rakesh.goyal We're starting a migration off a 47 year old mainframe that's FAA certified! 😧 Thanks for inspiring us for our journey!
OMG. Laughing at my own joke: “I don’t like to upgrade, because when I do, everything breaks.” 🙂
@eileencodes the description of your session sounds like a warning against pressing that "Fork" button that GitHub makes so attractive looking
Hah yes, only fork to push upstream!
Fantastic talk thanks (Capital One)! For me the success story there is based in the management attitudes: • Engineering is customer-centric and pro-collaboratoin (esp Ops & Agents) • Business and compliance sides understood engineering need for continuous flow Everything else follows.
“We forked Rails and practically made our own version.” (circa Rails 2, I think.)
same applies to test frameworks. always extract test framework from real tests.
PS: I still remember watching the famous DHH 15m Rails video, and how it blew my mind. I even called up a friend, and made him watch it with me. While I was on the phone. (Actually, that’s a very similar reaction I had when I saw the video of @eileencodes presenting at RailsConf last year!!! I started sending texts to @stephen within minutes!! 🙂
Ruby on Rails is a fun language/framework... the GIL however was not fun...
Ruby is still my favourite language 🙂 ❤️
“GitHub Rails soon morphed into a parallel fork” — presumably, increasingly distant from what the rest of the universe was using.
this is so true, the longer you let the tech debt fester the worst it'll be
“two years into the Rails 3 upgrade, engineers started wondering whether it was worth it after all.” 😂😂😂 (oh, it hurts!!)
hard to change when you’re not feeling the pain. < one of the discussion topics at the “changing behavior” lean coffee
“it became increasingly difficult to separate the application and the infrastructure” 😂
Yeah usually if you are doing the wrong thing and feeling no pain, it is because it is doing push ups to hit you harder
“can’t Google anything” 😂😂. (I told @eileencodes that in 2006, I had to write circa 2003 RSpec tests, and literally couldn’t find any docs on how to do it!)
This is such a familiar story for (I presume) anyone that has built an application on top a framework
Advocates of time machines and parallel universes take note: Forking reality is just not worth the engineering effort.
Was there a specific point in time that convinced you it was time to figure out how to close the fork?
I think I banned adding to the fork at 4.1 so that we could get off the fork. I closed it on 5.0.
A friend told me how it felt that they were still trying to upgrade to Rails 4…. “and then Rails 5 came out” 😂😂😂
“When do you feel the pain of an upgrade?” • My old team (which I joined in 2006) was tied to OO-Perl and on 5.6.1 for 14 years. • We pushed through a 5.14 upgrade which ran through to the platform decomm which started 4 years ago (we’re still waiting for the last application to be transformed). • 22 years. • So hard to get away from “fork” decisions.
“It was one of the most difficult things I’ve ever done; required focus, determination…“. 🙏🙏
"Upgrades are thankless feats of strength", this is sooo true
it looks like a SAP ERP upgrade, when you are finally done, the new version comes out 😄
In a micro cosm of this right now, with GH Enterprise. A network error blocked upgrades, we are so far behind now, my boss is afraid of starting
I mean it's suuuch a great story, the longer you delay the upgrade the more it's going to cost you later on to do it
Staying up to date is painful, but not doing it will be multiples worse
the interest keeps compounding
Calling out that cost is hugely important! So hard to fight against the most recent quarter’s MVP delivery value…
@eileencodes — I’m hoping you are feeling the awe that your story is generating! 🤞🤞🤞
Its situations like this where auditors can be your best friend....
“Upgrades are not inherently risky…. upgrades shouldn’t be any more risky than any PR. In fact, it’s DRASTICALLY MORE RISKY than NOT UPGRADING!”
I love that, “when you don’t upgrade, you lose out on good talent”!
We found that our legacy / technical debt slowed our skills transition. Management finally had to shoot the old platform in the head so that we could get away from that technical debt.
"They're not learning Rails 2.3 and they're definitely not learning your weird custom fork"
The “it’s no longer googleable” has driven a bunch of stack changes we’ve made…
aha moment: on a wardley map your aging fork is the only node moving left on the entire map
Makes me think of Ticketmaster's virtual VMS implementation to keep their code running 🙀
ITRev should make this video freely available the same way that newspapers provide Coronavirus coverage for free. Avoid this disease at all costs.
I cannot tell you how often I've told teams I'm not using their internally developed tool that duplicates something I can Google and buy books for.
I want to drop everything and go upgrade some of our many old repos
(I also remember reading somewhere about when the Go language team realized that “go” is nearly ungoogleable. 🙂
Go sounds great whenever someone tells me about it. But in reality, I don't like it, not at all.
“And worst of all, if you don’t upgrade, someone will propose rewriting the entire application.” 😂😂😂. So good.
That's the secret, upgrading early and often is actually the lazy approach. I like being lazy and preventing work later!
This part of @eileencodes story is incredible — the extent of the heroics she went through to enable upgrading safely.. I mean, holy cow…
I love the way @eileencodes just framed tech debt. I’ve noticed that our devs have increasingly started using amount of code deleted as part of tech debt work.
When I started here, I used to challenge myself to see how many LOC I could remove while adding a feature. My record was ~5000. It was a small change.
“you don’t get to develop features if your foundation is crumbling” so true!
“You don’t get to develop features if your foundation is crumbling.”
YES, celebrate this work! I recently sent an email to are whole company on how many lines of code we deleted as part of a bigger tech debt initiative. As important to the whole company winning a big new customer or shipping some big new feature!
the duel booting CI is brilliant to ensure new code works
PS: another amazing thing about @eileencodes — if I remember correctly, her first contribution was Ruby, so that deprecation warnings could be parsed in an automated manner. (Which one needs to do when you have thousands of deprecation warnings that need to be fixed.)
Here it is https://github.com/ruby/ruby/pull/3418
So much pain, truth, and hope in a single talk
PPS: Congrats on your recent promotion to Principal Engineer, @eileencodes!!!
1hopefully that is rewarding upgrade work 😄
Oh wow, interestingly the engineer who lead our latest tech debt take down and replatforming got was promoted to Principal last week!
if we call it branching instead of forking does that make it ok? :trollface:
Great call out @eileencodes thanks so much!
do you staff a separate team @eileencodes or is this kind of a swarming thing that happens when upgrades are required
Depends on the app and company. We have a team, but for Ruby 2.7 we did a swarm and updated 10 apps at once.
Interesting, in my previous employer we had a separate team to do "bug fixes and upgrades", it ended up being undesirable to work there
probably b/c it wasn't well rewarded either
We have a team that only does the github monolith. All other apps are upgraded by the team that owns them
@eileencodes is a separate team needed b/c GitHub is a monolith (still?)
asked the question same time as you already answered ✨
that makes sense, this team is like a platform team for the monolith itself
very cool! So in a microservice world as you said, it'd be a team responsibility
I very much regret deprioritizing upgrades, literally grown once a week b/c of this 😞
“The best time to do an upgrade is immediately “!!!!
seems like planting a tree. the best time? 100 years ago. Second best time? today.
@jtf I think I've heard you say that on a podcast before. A couple weeks ago, I was arguing with an architect about if I should be allowed to close a year long project branch we kept taking individual features from. He wanted to push it out another 2 quarters and said "tomorrow isn't always the best day to address tech debt" and I said "Yesterday was!"
@eileencodes what about the "let's wait a bit and see if other people have problems with this upgrade" mindset?
Almost as bad. It is a conservative approach to moving forward. Kind of like standing on the breaks while going full throttle.
You have more say in how that version functions if you upgrade before everyone else.
Better yet - how about using only the oldest supported/trailing edge and upgrading then ... just as painful ;)
If you wait and it works for everyone but you, now you have 2 problems. An upgrade and a bug that maybe is not fixable with a backport because too much time has passed
Upgrading while the whole community is going through the same process is a form of fast feedback, going through it while it is fresh in everyones mind is only going to help if you run into issues
@eileencodes: @stephen and I started generating these graphs of version migration patterns, with support from a team at Sonatype — it’s amazing to see the differences. This is Java Hibernate. Might as well be two islands, where you can’t get from here to there:
New APIs often improve correctness too. “oh this API is confusing and everyone misuses it” <- that can get fixed in new major versions.
It’s in the 2020 State of the Software Supply Chain Report (page 30) and was generated from data on upgrade patterns for projects in Maven Central.
I love that Spring graph. A great visualization of everyone keeping current!
vs. Joda Time, which is like a homogenous set, where it appears one can easily go from any version to another…
Keeping up to date - Same benefits apply to vendor platforms and integrations into vendor platforms. Investing in staying current should never be viewed as “expensive” or “costly”. (Minimize those costs, sure, but don’t complain about effort required…)
My fervent hope is that the shift to SaaS will make this a non-issue. Every vendor product is suddenly always up to date!
We took a look a Clojuescript on my team, brains exploded, decided to implement Ramda and move on.
This aligns with “fluid dependencies” discussed in “Evolutionary Architectures”
Continuous upgrades!! 😍
“Upgrades are easy, because we’re upgrading as Rails upgrades… instead of waiting years.”
it makes sooo much sense too!
it's a virtuous circle too
b/c the Continuous Upgrade feeds into the CI and then gets deployed
“continuous upgrades” - Going to have to start using that internally…
It’s almost like doing things in small batches works 😄
we do that for application features and delivery, but not for upgrades. Love how @eileencodes broke down that upgrades are not inherently more risky than app changes
aye; given a sufficiently strong test suite and deployment pipeline the only risk is delaying the work which only increases batch size
Always look for ways to reduce your batch size
Continuous upgrades < “if something is hard, do it more often” - Kent Beck explaining Continuous Integration
One of the many things that @eileencodes told me that blows me away: “often, devs think locally. They think they’re the only people having a problem. They don’t realize that EVERY dev around the globe is having that same problem.” 🙏
@eileencodes do you have something to enforce the smaller teams continually upgrade their smaller services?
Not quite but another team is working on it.
1Right now OKRs are it, but I hope that we'll have more tooling around it soon
"Objectives and Key Results" for those who are unfamiliar with OKR
I need to read up on orgs that are doing OKRs well, we attempted it in the Canada office and it didn't go anywhere 😞
ultimately, I think it's b/c it wasn't actually measured and followed up on
This is like the ultimate argument for Trunk Based Development ... from the people who made forking a bit too easy
Continuous upgrades < Investment in your dependencies pays off…
👏 awesome talk, super inspired 😄
Amazing talk @eileencodes!!!
Very smart positioning Gene! That was an awesome way to end!
“if you don’t like your language, improve it.” And here is @eileencodes first contribution to Ruby. 🎉
You guys should continue presenting lectures pre-recorded with the speakers answering Slack questions live from the stage and a possible quick Q&A at the end when we do this in person (hopefully) next year. This is way to good not to do again.
A chance for everyone to add their #summit-stories
Now might be a good time to ask, how many attended this year?
4,667 includes attendees from all previous DOES sessions. @mvk842 how many attendees were there for this one?
Hi @abd3721 - We had just shy of 2,000 registrants for DOES US. Not all registrants sign onto or use Slack, which I know is surprising! We’ve been using Slack at our conferences for three years (six conferences). As you can imagine, Slack traffic has increased a lot this year!
No (obvious) major technical difficulties until the very end ...
true, very impressively run
We’re all looking forward to the published incident analysis.
There will be some learning from this I am sure…
Here here for Gene NOT being the Incident Commander. Communicator, sure, but NOT Incident Commander 😉
Opening talk from the next event should cover this, please (specially if next event is still virtual)
I like to think Patrick is too humble to be recognized and stopped the stream himself
we need everyone to take there role, let’s include everyone please…
Sounds like a need for an AMA session to be called for all of the calls for an update!
<!here> …we are attempting to load up the video on Track 1.. If you’re on another track, switch to Track 1. Give us a couple of minutes to find the right files. 🙂
Maybe if he had turned in those TPS reports…
anyone bringing doughnuts fired… bring croissants.
I hope @genek101 is not in the bridge. Remember the talk from @allspaw that leaders do not help as much as they think in those situations 😄
While we're waiting a huge to @genek101 @mvk842 @jessicam @alex @jeff.gallimore @annp @patrick.debois256 and the whole IT Rev crew for an amazing three days of learning 🙏🙏🙏
3 26best run virtual conference I've been to so far
2very impressive and lots to take back to our internally run conference
Truly hope I get to see you in person next year, @jonathansmart1! Congrats on smashing it with your new book!
I can see the start of the next session be the concluding moments of a Problem Incident Review. These are the things we learned last year…
(…and we’re investigating a potential live option… 🙂 This is exciting. 🙂
any good campfire stories? @cncook001 looking your way….
Did I tell you about the haunted rack at our old DC?
there was this one time at band camp....
Maybe this glitch was an opportunity for us to continue our fireside Slack chats!
it'd be strange if there wasn't a blameless postmortem lol, I certainly hope so 🙂
do as one preaches I hope 🙂
Normally it’s all stream, no paddle. This time, no stream, and a whole gauntlet of paddles.
For your amusement: here’s what’s happening… • investigating changing HTML player • live option: Streamyard to YouTube to player • maybe even just firing up a Zoom
I vote just the opposite, let everyone cheer like a live audience!
I’m assuming you’ve been running this all off of someone’s tethered phone and just hit your monthly limit
This is what he says about weight loss. It’s so annoying, he doesn’t eat carbs for two days and loses 10lbs. Then he exclaims “the power of positive thinking is working!” 😒
@jeff.gallimore will write it one word ahead of you speaking it 🙂
The good news is you know the audience is on your side
if it makes you feel any better, most of the stream made it to my youtube player. Right up until Gene made his ask for help
So we’re going to do some live chaos testing of zoom? nice
Best. DOES. Story. SO glad I didn't go to bed
<!here> https://devopsenterprise.slack.com/archives/C015DQFEGMT/p1602809015301200
great way to close off the conference haha, massive Zoom call 😄
they're working on it
Darn that Capacity Management Plan This meeting has reached a maximum of 100 participants. Please try again later. :rolling_on_the_floor_laughing:
We have the best conference attendees and speakers!!!
What a great 3 days! Thank you to everyone for such an engaging virtual event. Loved it. Vegas 2021 (I hope!!)
sounds like a Columbia vs. Challenger comparison -- one a failure at takeoff, one a failure at landing.
<!here> closing remarks!!! https://us02web.zoom.us/j/88130675487 (Passcode: 404510)
2oh no, he's not in present mode! LOL
we can read the notes and see his thought process though haha
That was one of the unknown/unknowns we were NOT hoping for... 🙂
Shallow dip into chaos, agile created Zoom webinar, to become a lean new normal? 🙂
I call that 00:19:00 outage duration. Very creative solutioning to solve the problem! Well done event staff! Kudos!
Huge thanks to @mvk842 and entire IT Rev team!!!!!
9 5thank you @mvk842 and @genek101 👏 ❤️
2 1Last words when they play the cockpit tapes from the black box: “Gene: That’s nothing. Watch this…“. [END AUDIO] h/t @jonathan_moore
Well, you might end up with a surgically reconstructed Achilles tendon that is better than the one you had before...so it's not all bad. 😉
hmm, have to update the closing remarks feedback…
I swear that @jeff.gallimore has a tie fighter in the background?
thanks for watching and the great feedback everyone!
That was great!! I think you have to do the Zoom ending every virtual event from now on 🙂
1Thank you @genek101 - I thought you and your great team recovered very nicely. This was my fourth DOES and I will always remember this ending!
Now please go buy @patrick.debois256 a nice cold one from all of us
Watch @eileencodes’s talk in its entirety! http://videolibrary.doesvirtual.com/?video=467489374
How many attendees were there and how did it compare to London 2020 and to 2019 events?
ahhhhhh watching this now (I had to a big TO DO today) and I might just force everyone to watch @eileencodes’s talk 💪
don't mind me, i'll just sit here in channel and "live tweet" all the talks as I go thru them 😂
3Please do! I plan to watch/re-watch a few over the weekend as well, and I may be checking in on the empty halls of Slack for a while as well 🙂
And then this happened when @genek101 tried to open our celebratory wine. 😩🤪😂:woman-shrugging::skin-tone-2:
@mvk842 I’m going to share that in the current #happy-hour zoom!
I gotta say, I've opened a lot of busted corks with a screw and a pair of pliers @genek101. You're in good company. I feel @nicolefv judging me now. We are toasting you on the #happy-hour .
lol I usually stick with diet coke. i'm useless when I try to open wine bottles... maybe 50/50
Usually don't believe it when people give me odds, but when they come from @nicolefv... 😆
@genek101 @mvk842: Where is your redundant array of bottle openers?
Useless until we were able to get the broken corkscrew out of the cork! A fitting end to the craziness of the closing remarks. Hope you had a great experience. TchÜss!
Still having it. The redshirts are still convening.