This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # ask-the-speaker-plenary (637)
- # ask-the-speaker-track-1 (139)
- # ask-the-speaker-track-2 (146)
- # ask-the-speaker-track-3 (126)
- # ask-the-speaker-track-4 (176)
- # birds-of-a-feather (2)
- # bof-arch-engineering-ops (9)
- # bof-leadership-culture-learning (3)
- # bof-project-to-product (3)
- # demos (3)
- # games (39)
- # games-self-tracker (6)
- # gather (7)
- # happy-hour (48)
- # hiring (12)
- # lean-coffee (1)
- # summit-help (20)
- # summit-info (93)
- # xpo-bmc-ami-devops (4)
- # xpo-cloudbees (5)
- # xpo-cockroachdb (2)
- # xpo-github-for-enterprises (7)
- # xpo-gitlab-the-one-devops-platform (7)
- # xpo-itrevolution (1)
- # xpo-launchdarkly (10)
- # xpo-lightstep-observability-incidentresponse (2)
- # xpo-linearb-automate-dev-team-improvement (5)
- # xpo-snyk (6)
- # xpo-sonatype (2)
- # xpo-split (5)
- # xpo-stackoverflowforteams (5)
- # xpo-synopsys-software-integrity (5)
- # xpo-tasktop (2)
- # xpo-tricentis (8)
Reminder: The breakout sessions are starting in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
✨Welcome Jim Armstrong and @jamie.joshua and @christina.hobday for our next session's Q&A Thank you to our sponsor, Snyk!✨
Cloud CI/CD offering suffered from people using build resources for crypto-mining, true stories!
@gvian joining us for a little hiring magic https://us02web.zoom.us/webinar/register/WN_QD-YrhEeSBWB-rifid0laA?
Further information on what Jim was touching upon regarding infrastructure drift and unmanaged resources: https://snyk.io/blog/detect-infrastructure-drift-unmanaged-resources-snyk-iac/
🐝A warm welcome to @hlynch @ldonley and Drew Piland for our next session's Q&A .Thank you CloudBees! 🐝
Anyone else hear people asking if they can be heard. Looks like people trying to connect not being part of the presentation
lots of c=licking and people asking if they can be heard (names called out: Andy and Martin)
Thanks so much! I have a feeling our VendorDome folks' microphones are hot. I'll let them know!
(and suddenly the reports of mayhem makes sense — just look for @martinwoodward. 😆
not hearing anything additional anymore, so looks like it is cleared up 👍
Thanks so much for everyone's understanding! And to @hlynch and @ldonley for the great session!
It's like 2 audio tracks overlayed, with a couple of seconds delay between them
Fixed here. And it’s back. And fixed. (Thanks for your patience, everyone!)
somebody's mouse noises are audible on top of the audio 🙂
Well... we are a species who hasn't fixed the mute/unmute problems on teleconferencing in 20 years with more than 2 years of intensely trying... So, multi-channel audio routing is an understandable hickup.
a true sign of success is when the dev teams can articulate the why off the cuff, without pulling up any documentation
This is a really good talk so I hope everyone has a chance to listen to the on-demand recording.
Yes, don't forget to re-watch the recording in the video library afterwards and feel free to reach out to @hlynch @ldonley and @jwellington809 with any questions!
It sounds as someone is intensely clicking away with their mic on... so the problem is there but it is not actively bothering right now.
Please stop by our https://devopsenterprise.slack.com/archives/C01TK0K6QU9 or meet up with us in Gather. We'd love to answer any questions!
We’re about to kick off in the 🔷 Vendordome!🔷 Post your questions for @martinwoodward and @abold in the thread below ⬇️
It’s a long way from home, but welcome to the Vendordome. Looking forward to chatting
Welcome @jcregg who will be moderating for today's VendorDome Q&A between @abold and @martinwoodward!
Hello, @martinwoodward @abold and @jcregg!!! Love this discussion — “11MM commits. 22 GB. 20m to clone.”
(Reminds me of the epic stories of moving Windows OS to git — )
“2-5 weeks for dev to be productive” — all too common. My fave quote: “figuring out how to have 2 versions of Python on your laptop shouldn’t have to be a core competency.” 😆
There are legendary stories of developers at banks quitting before they complete the six weeks of compliance training. 🙂
@jcregg I’d love to know from Andy and Martin: they have this awesome dev productivity — how many engineers do they dedicate for dev productivity, to create this amazing infrastructure?
This is such a great question! I suspect most people would surprised how high that % is in elite performers
As someone who works with governments, I would love to only have a 2-5 week onboarding time. One organization took 6 months to onboard before getting to the development enviroment.
“Do the thing that hurts, until it doesn’t hurt anymore and move on to the next thing that hurts” 👏
Great line... but is there a category of issues which people don't yell about because they have normalised that experience? Or aren't aware life could be better? Perhaps in an org with a particularly esoteric environment. How would you identify and tackle these?
Thanks for tackling this guys! 🙂 Fascinating stuff - @jcregg you mentioned someone in your org wrote a diary of an on-call engineer if I heard correctly - is that something shareable? One idea I've seen is to get new joiners to complain loudly before they become institutionalised. Especially experienced hires who might have seen relevant best practices elsewhere.
Hey @philipday! Here’s the diary mentioned. https://thenewstack.io/diary-of-a-first-time-on-call-engineer/ It’s got some direct excerpts from chats between the engineer and her manager
@martinwoodward how did you have confidence it worked in production? Did you have access to telemetry in production?
Martin: depends, goes in cycles; lean team: pull from feature teams to improve core product, as opposed to a dedicated dev engineering team (so interesting!) GitHub Computer Club: band of people focused on solving the big problems, getting people moved to CodeSpaces.
GCC working agreement: you’d notify them when you bail out, and go back to “crummy macos dev” (my words, not theirs. 🙂
@martinwoodward do you set some standards e.g. spend 20% of time improving things? how to you protect capacity for continues improvement?
My fave article on CodeSpaces: https://github.blog/2021-08-11-githubs-engineering-team-moved-codespaces/#:~:text=Over%20the%20past%20months%2C%20we,development%20environment%20in%20the%20cloud. My fave line: > This single log message will cause any GitHub engineer to break out in a cold sweat > (something about “dotcom gem environmetn is out of date” 🙂
How much of your platforms are built through contributions (a la innerSource), based on how much is request based (e.g. build me this please). Does this help keep teams smaller (EDIT: and therefore the platform is more productive)
Andy describing how LaunchDarkly uses http://Honeycomb.io to generate production telemetry.
Yes. We’ve used Honeycomb for a long time, to the point where I think it predates Datadog adding trace support. They’re both great tools that help us understand what’s going on.
@martinwoodward, @abold do you measure the Accelerate Metrics? If so, do you measure quantitatively or qualitatively?
Martin on NPS as satisfaction of your own devs using your tools — nice. 💯
Fantastic panel, @jcregg — loving the insight from two great engineering orgs. Such a great example of prioritizing what you value, and it truly shows!
@abold: “quarterly, we do a week fixing on issues; ‘focus week’ focusing on toil, tech debt; ”
An idea I've heard about in other organisations is 'innovation days' once a month to tackle all those little ideas that probably won't take too long to implement but have potential to really reduce toil. Stuff like "oh a slack workflow could help us solve this problem, but i'll never get time away from project work to implement it"
Martin: from MSFT, 90% of issues come from “people paid to work on that product”; 9% come from “close on org chart” (adjacent); 1% from randos adding code (“I’m going to try to add newline support in Windows” 😆
Martin: “further away in org chart, more likely you think they’re an idiot” — trust is lower. and issues look easy to fix for the person not immersed in the issue
Martin: people drop by to do something, try adding something to an API interface, say. Easy case: pull data and just add to response; In contrast, at GitHub, all innersourced: “great you come with a gift of a [giant] PR, but can we actually maintain; same ratio at Microsoft: 90/9/1” (@martinwoodward did I get that stat right?)
Andy: dependency: % of completion drops as exponent of # of dependencies; 50% -> 25% -> 12% (as # of deps go from 1, 2, 3…)
Andy: “here at LaunchDarkly: people who know the most share the most”
> “If you ask the same questions, you’ll get the same answers; for a successful transformation you have to have the discipline to only ask questions which actually matter” gold dust 🙂
Interesting about dependency %. Is this because they are software-focused companies? This seems counter to an open-source philosophy?
the law of constraints says as an organisation you’re only as fast as your slowest bit, so individual team performance is only useful if you see how it fits in the whole org
I guess that’s why we strive for autonomous cross-functional teams? To make them as autonomous as possible? It’s a local optimisation but it hopefully contributes to global optimisation?
I think that certainly helps. but I’m not sure how widespread that practice is yet, and you can still have concerns if you’re working on a legacy monolith where individual teams can still affect each other or be dependant on each other.
That stuff about incentives from @martinwoodward was gold dust! One of the hardest and most central problems IMHO.
I think Martin said, to max out your bonus you have to demonstrate: • What you did • What others did where you helped them • What you did that built on things others did ... which promotes a culture of spreading credit and collaboration I'm not just (or particularly) thinking about developers within teams... I'm thinking anyone in the org, where so much activity sits somewhere in an ecosystem of internal teams and relationships and it's hard to attribute direct, objective and tangible value to genuinely important and difficult work.
@abold: “At LD: we have a habit of feature flags that have been true for a long time, get notified”. Clever!
Automated code changes of conditionals — what could go wrong, @martinwoodward 😆
@martinwoodward I’ve got nothing against automated code changes — I’m just against changes in general. As Monk (famous OCD TV detective) said, “change is bad. you should never do it.” 🙂
Treating tests as slot machine; if it doesn’t work, just pull the lever again. 🙂
Holy cow: Clever!! From Martin: “flag flakey tests, and take it out of rotation, and open defect (I.e., a “berg”) with the team that owns it.”
Andy: “be happy with good enough; don’t gold plate; ship it when it’s good enough, and then improve it”
Martin: “don’t ask permission; it’s your job to make the system better.”
Martin: “what’s one thing to improve lives of fellow devs, team, customers; what can you do in one day, and do that; repeat, and you’ll be amazed at how improvements build up”
Martin: “the funny thing is, you’ll never remember how bad things were, so you need to take good notes, measure, compare.” Nice!)
Reminder: The breakout sessions are starting again in 5 minutes. Get in front of your browser and start navigating your way to whichever session you’re attending. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Great discussion thanks @martinwoodward @abold and @jcregg
Well I think that was probably the most fun I’ve had in a panel discussion - thanks for the chat and banter folks. Thanks also to @jcregg for keeping and eye on it all for Andy and I.
Thanks for being here everybody! And thank you to @martinwoodward for his words of wisdom and @jcregg for keeping things moving.
She was too modest to say so herself, but https://twitter.com/jessicacregg is worth a follow too…
🔥Welcome @asharma for our next session's Q&A Thank you Sonatype!🔥
Thank you so much to @abold and @martinwoodward for being excellent to work with and brilliant panellists! Thanks everyone for asking questions and keeping the discussion really interesting and interactive - @fokkov @erik.greathouse277 @toli @phillipgadzinski @eliza.kruszelnicka429 @philipday @slack1599 @richard431 @lloyd.passingham @chris.leeworthy @billy.hudson and of course, @genek
Great discussion, some great nuggets of wisdom @abold & @martinwoodward - and very well moderated too @jcregg 👏
Invisible backdoors! (Makes one questions our life choices in OSS. :)
Makes one question our life choices in OSS. Again. 😂 There's a terrific session on Log4J in software supply chain context tomorrow by Dr. @stephen — expounded upon from session he did in March.
Why do people usually choose to use these “dependency confusion” packages?
OSS reminds me of a friend's dad. his assessment of most things: doing this will kill you; not doing this will kill you faster.
Ex: eating that will give you cancer (eventually). not eating it will kill you faster (starvation). In our case, using OSS will make you vulnerable. Not using OSS will make your business not viable.
@ann.marie.99 There have been afternoons where I used every OSS component that showed up in a "grid react" Google search — I'm not proud...
Publishing SBOMs makes me nervous - aren’t you just advertising your vulnerabilities publicly?
the choice between downgrade and upgrade requires human intervention bad for automation
Paul Fox from Morgan Stanley has an amazing experience report on log4j tomorrow, I think — Neat story of them trying to get assurance from their vendors that they've fixed issues, and incoming requests coming into org asking THEM whether they've handled issues.
Thanks for attending the session friends!! We've been catching more malware into today - I've got some more blogs in pipeline if you'd like to check out: https://blog.sonatype.com/author/akshay-ax-sharma
Welcome Dan Lines from LinearB for our next session's Q&A!
Hey Dan 👋 How do you define an "idle" PR in your organization? How is it measured?
Hey @martin.jainta! To us, Pickup time (from the moment a PR has been issued to when it is actually seen/worked) is all idle time. Review time also has several back and forth moments where the PR sits idle. Check this out:
Hi @marco.cicolini I would try to help, with pair programming you are taking a risk of the reviews being done by only one person
You can always do that but with larger teams it might not be the easiest and safest way to go
That being said, do you recommend review sessions with the creator and for example two peers?
I have the impression with PRs we tend to optimize for the individual person (e.g. I raise the PR and then continue on the next work item). While with a more synchronous approach (be it idea #4 or pairing) we optimize for flow because we focus on finishing the work items.
Well, it is up to the eng org management to decide, there is no one way to go. As long as you get the review done in a timely manner and with enough depth, it should work
In LinearB we are monitoring and reflecting the review pickup time, review time and depth
Reminder: The plenary sessions are starting again in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
Reminder: Please submit your feedback for the talks you attended. It’s so valuable for us and the speakers. And after all, feedback is a gift and sharing is caring! Enter your feedback for those talks here: https://members.itrevolution.com/live/schedule https://devopsenterprise.slack.com/files/UATE4LJ94/F03E48CJRF1/image.png