This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-07
Channels
- # ask-the-speaker-track-1 (422)
- # ask-the-speaker-track-2 (356)
- # ask-the-speaker-track-3 (215)
- # ask-the-speaker-track-4 (278)
- # bof-arch-engineering-ops (2)
- # bof-leadership-culture-learning (12)
- # bof-sec-audit-compliance-grc (1)
- # bof-working-with-data (1)
- # demos (7)
- # discussion-connect-february (1182)
- # games (73)
- # games-self-tracker (1)
- # gather (4)
- # happy-hour (38)
- # help (82)
- # hiring (14)
- # lean-coffee (8)
- # networking (20)
- # summit-info (101)
- # xpo-adaptavist (5)
- # xpo-anchore-devsecops (7)
- # xpo-aqua-security-k8s (2)
- # xpo-basis-technologies (2)
- # xpo-blameless (3)
- # xpo-bmc-ami-devops (1)
- # xpo-cloudbees (14)
- # xpo-codelogic-code-mapping (1)
- # xpo-dynatrace (1)
- # xpo-everbridge (2)
- # xpo-gitlab-the-one-devops-platform (1)
- # xpo-granulate-continuous-optimization (1)
- # xpo-instana (1)
- # xpo-itrevolution (9)
- # xpo-launchdarkly (1)
- # xpo-pagerduty (1)
- # xpo-planview-tasktop (3)
- # xpo-rollbar (1)
- # xpo-servicenow (2)
- # xpo-shoreline (2)
- # xpo-snyk (2)
- # xpo-sonatype (7)
- # xpo-split (1)
- # xpo-splunk_observability (8)
- # xpo-stackhawk (2)
- # xpo-synopsys-sig (1)
- # xpo-tricentis-continuous-testing (1)
- # xpo-weaveworks-the-gitops-pioneers (1)
Reminder: The final day is starting now – opening remarks and then plenary talks! Join the conversation in #ask-the-speaker-plenary.
Hey Everyone! Speaker here! 👋 Grateful to be a part of this community and super excited to be presenting today! Thanks a ton @genek, @tapabrata.pal, @annp and the program committee for helping through the process!
That’s ok. We did it wrong on the first pass at WM Logistics.
I repeat to P1 that bowling lanes have 2 gutters. Both a re bad. :rolling_on_the_floor_laughing:
:shopping_trolley: A warm welcome to @kapoor.vaidik, presenting Think Again Before Migrating to Kubernetes :shopping_trolley:
The salesperson specifically said it was. Hmmpf.
But, just AWS with your process of provisioning resources and deploying changes documented on a confluence page is also a platform
i really like how matthew skelton puts it - thinnest viable platform
K8s at the edge was important for leveling the snowflakes in WM distribution centers.
oh yeah. edge, heterogeneous infrastructure, multi-c* (i didnt say that)
We are currently working on an “easy button” for a K8s env that a sailor can deploy to a submarine with no K8s admin training. If it becomes unstable, hit the button and reboot.
I look forward to standing on your shoulders when you're done. I'm trying to hold of on K8s at the moment because frankly we don't have the knowledge/skills we would need to investigate when it goes sideways.
Chick-fil-A rolled out a 3-node K8s cluster to each of their restaurants a few years ago--it would be interesting to see how that has worked out over time.
Yes… this was a very good integration case study to operate in a hub-and-spoke model. Really enjoyed their presentations on this.
Agree, I would be interested in how it turned out! It doesn't look like their repo's of fleet management tools have been updated often in the last couple years https://github.com/orgs/chick-fil-a/repositories
We've been trying to find people to ask more about this and how it has evolved...
The cognitive overload of trying to migrate to k8s is quite high. Is there a way to ease into it? As stated, running a self-managed k8s cluster isn’t ideal… it forces your team to learn everything at once to get any value out of it.
eBay has a mechanism where we just moved all apps to containers but didn't give them access to any of the k8s stuff. The deploy group just "forklifted" their apps in conjunction with the language operators.
The next step is/was to migrate people to more container-native workflows.
Also can lead you into the trap of all the things you need to have before providing any value. You can find yourself moving down a "give a mouse a cookie" type route of "well, if you want to deploy I need XYZ services deployed"
even managed services still require you to make a lot of decisions. i have been inclining towards using services like ECS or opinionated kubernetes platforms that come as managed services
Yes, thank you all for the feedback. I think we need to make sure we are creating (or refactoring) products in a container-first approach, then begin migrating workloads to see what works best for our orgs. We are already seeing so much of this being managed for us, that we may not have to worry about it for much longer… as long as our services are containerized
yaeh i think our lack of experience in the space made the journey slightly more painful. with experience, figuring out the path for full migration can be much easier than what we went through
Was ECS ever considered for container orchestration? We made the shift several years ago to ECS and find the overhead of management/upgrades and the cognitive load on the team to be a lot less than K8s. I’m curious on your thoughts of EKS over ECS?
we didnt. and we wanted to avoid any kind of AWS specific ways of working. at the time we were actively avoiding things like cloudformation as well
in retrospect, i feel ECS would have been a much better choice
LOL, I first read that as “cloudformation hell” 😂
but also at the same time (and at least for us), we were at a point where we were considering to migrate to azure. it was almost done until it wasnt. so i am really not sure if there was a right decision there. in the end, what we did worked out for us
That makes sense, I would have likely made the same choice considering those circumstances. Thanks for the response!
At Platform One we've worked with EKS and helped deploy some of our products on top of their clusters. ECS is a different story. Customer demand wasn't there to investigate, at depth, leveraging ECS. Most customers come with a hybrid cloud or air-gap infrastructure making it difficult to leverage ECS for orchestration. They decide to go with k8s for the flexibility
usually we see RKE2, Konvoy, OCP4, or EKS requested at the orchestration layer
thanks folks! i am around all day today and would love to talk more cloud and kubernetes with you all!
Eeeyore - I'm at one of those picnic tables in gather if people want to join up there for follow-on discussion
✈️ Excited to hear from @bissingd and @msnyder who will share, Cloud-native DevSecOps at Supersonic Speeds (well...getting there) ✈️
Excited to hear from our sister effort on the 309th!
Hill AFB! I was there like 20 years ago, back when they had Vipers instead of F-35
I was chatting with @bryan.finster486 the other day, he might argue, but isn't the goal a common DoD platform with env available to the individual programs?
There is an argument to make for having good platforms that focus on core delivery problems but all have the same principles. It’s a different problem with DoD that with an enterprise like Walmart. Scale and complexity are much larger. I’m still pondering.
Maybe In the weapon platform world of embedded there is no common infrastructure or platform for that I hate to say it but the metal we throw at the enemy is the weak point in my opinion We got stuff in the cloud for days however the butts in seats manning the weapon systems need to have awesome also
Just like with K8s, can have metal different while the underlying code remains the same. Laser guided bombs are only the sensor, same with GPS, no reason not to build commnon code
And, I don't think the DoD has a Scale problem, they actually deliver much less than the typical industry, I think it is an authorities and culture problem'
You're too generous. I've seen "Years" melt down to hours from this transformation.
I think it is awesome that I never mentioned DevOps, but rather focused on the constraints in our value stream and the result was a DevOps process
The current architecture is leveraging the P1 managed service Dev environment for the OFP development team; in upcoming iterations we are working with other OFP development environments to pull into the Lightning architecture and deploy out to the aircraft;
So you are going to break away from the P1 model to develop an independent DevSecOps for the F-35
we're working with the 309th to establish a tailored architecture that will support across weapon systems and environments; leveraging as much of P1 as possible, but recognizing that the Big Bang core needs additional services and architecting to support the embedded systems depoyment
that's one area where we would love to jam on - how are other sectors with the embedded SW working through this
Automotive must have some fun use cases. OTA updates, leveraging ML for ADAS systems, etc. (+1 to jamming)
yah; that's a big part of this transformation effort - how do we leverage the virtual testing to limit the amount of testing executed through the HIL/SIL - because that's a major bottleneck with very limited capacity
we had this calibration function on the RHS that tweaks the parameters for things like torque
@chris.gallivan421 I would love to bring in some of the Tasktop capabilities into the value stream architecture at this effort as it expands -
setting up at the fire pit at the top of the map if anyone wants to join
Any solutions on the horizon for burning to a disc/mailing it in to the lab for integration?
Nice, would love to learn from that! I talk to too many people who have a nice agile approach to development...until it comes to the point where containers need to be pushed to prod
They forget it includes actually delivering to the end-user, not just writing things.
It's tough, seems like there's constraints from the tech available. AWS has some solutions that are interesting but that's AWS
that's how we update vehicles, drive to the dealer to get the disc 😞
That's very interesting, I didn't know cost was a huge factor.
I think that's how my Lincoln maps got updates... However I do think some UI updates were pushed out OTA to enable features like CarPlay.
Do you have any restrictions on what types of software you've been able to apply this approach to, e.g. software that performs Safety Critical functions?
We are starting on mission systems But we still require flight safety testing
it's a significant shift - for the organization used to issuing contracts fo fully programmed, and "solutioned" contracts
most of our tools were windows based, which caused another challenge
🎉 Let's say hello to our next speakers, @cheryl.crupi and @ronica who will present: The Miserable Tier 🎉
Hi everyone … Cheryl and Ronica here! We are excited to hear your feedback on our talk! Please share your stories and questions in Slack. We would love to know. The things we describe in this talk … are you seeing the same in your organizations?
@cheryl.crupi and @ronica investigate on why SVPs are literally the most miserable tier!! 🎉
@genek I know you've been out to Kessel Run, we need to find a way to get you out to Ogden to see SkiCAMP
OMG, yes!! I remember meeting Lt Gen John Thompson, and I was so delighted that I might be able to shadow him for a day at one of these amazing orgs. Then COVID happened. Can I reach out to you and let’s talk? I can’t tell you how excited at the possibility of doing this!!!
OMG. They interview SVPs to understand to what extent they are miserable, and why. Love it!
How many SVPs were you able to interview, @cheryl.crupi and @ronica? This such important work!
Yes…we drew on stories and experiences from over many years. While going deep through interviews.
it can get lonely if there is no inclusion, no sense of belonging, trust, empathy, empowered and doesnt understand why Agile transformation is so important
@cheryl.crupi @ronica If you were able to see the Target presentation with @brett.craig SVP Digital (formerly SVP Merch Capabilities), did you notice differences in how he showed up vs. the miserable SVPs you interviewed? I’d love to compare/contrast.
Also curious to hear your thoughts after seeing the video... and even more if you (@ronica) reach out to interview @brett.craig
Striking also that those we interviewed specifically for this talk are literally from all over the world.
For mid managers, isn’t giving credit potentially harmful as middle managers need to prove their individual worth?
I’ve often found this level is trying to prove something and aren’t incentivised to do this.
In fact, it’d be interesting to identify these characteristics among all the SVP DevOps Enterprise video talks, and code for absence/presence of.
Hoping to see a rubric in here around how folks can evaluate where they are on the miserable spectrum & specific steps to remediate.
I’d love to help on this above: ^^^. let me know if you’re interested.
@nicole.forsythe - yes, that is exactly the point. In environments that are competitive, or even stack rank, giving credit is a transformative and courageous act! We hope this habit can contribute to changing the system
Did you look at things like gender in this work? I notice women do this and are then much less likely to be promoted as this kind of behavior isn’t always promotable.
I’m a director level and a lot of this is speaking to me - is SVP described as the level just below the C-suite?
Yes … we are thinking those senior execs that report directly to the c-suite. The titles can vary by organization.
Pressurized middle, a la Jon Smart. I’m a director, but there are a great many levels between me and the c-suite
I have recently fallen in love with this term from Dr. @ronwestrum — the technical maestro: • High energy • High standards • Great in the large • Great in the small (so they can ask good questions) • Loves walking the floor (gemba) I love this model b/c it has such high explanatory power — explains the great orgs I’ve seen, and the most horrible orgs I’ve seen. Just curious: do what extent to the miserable SVPs you interview rate high on all five of these attributes? @cheryl.crupi @ronica
Hmmm…I’ll have to think about that…The first one who comes to mind definitely fits that description
He’s miserable because his tendencies are stifled by his peers, or in the peer setting
> Gemba (現場, also spelt less commonly as genba) is a japanese term meaning "the real place." Japanese police could refer to a crime scene as gemba, and TV reporters often refer to themselves as reporting live from gemba.
(SVP might not be the technical maestro, but have one — my deep suspicion is that technical maestros will have great outcomes and high job satisfaction, and also their teams)
I wish I could remember where this came from, but I often chew on the sentiment “When you go into upper management, you sacrifice your mind, thinking, and time to meetings and only meetings.”
it’s interesting to hear how often @brett.craig described interactions with his peers, their willingness to give him a chance to try something new in Fresh Foods, their skepticism, but eventually their appreciation. Definitely indicates the presence of things described here — cc @lucas.rettig
well i was on another track… now i’m interested 🙂
I really like this metaphor of Maslow's model. Less of a pyramid (which Maslow never created) and more of a sailboat. You need to feel secure first - only then can you grow. https://scottbarrykaufman.com/sailboat-metaphor/
Are you seeing the same thing we are? If you try one of the habits we propose, please share will us how it goes. LinkedIn is a great way to connect and share.
I think one thing that would be helpful in understanding this is something like case studies. "Here's a scenario. Here's what they're doing wrong. Here's a different scenario w/ someone who's doing it right", etc.
One of my more amusing direct supports always says of our work “This job would be so easy if it weren’t for the people.” 😆
Indeed, really liked the invitations to practice! Thanks @cheryl.crupi and @ronica.
👏:skin-tone-3: A warm welcome to @paula.louise.kennedy who will present, Crossing the Platform Gap 👏:skin-tone-3:
Hello everyone, so happy to be part of this event. Let me know if you have any questions or want to chat
Whoohoo.. @paula.louise.kennedy - Cognitive load 🙂
It's an awesome book...and Matthew and Manuel are very smart and lovely people
BTW, @paula.louise.kennedy did you publish TeamAPI as per their recommendation. What efforts are taken for reducing the cognitive load? example documentation, code repo, service catalogue etc?
we've been working on a product that we think can help with this. It's a framework for building platforms that enables clear service boundaries between the platform and SA teams
I recently came across as DevX platform by Spotify - wow I think that will be so good to rollout in complex enterprises
This is great @paula.louise.kennedy. we are testing something similar. I decided to call it as our Common Components team because the term platform was interpreted in so many different ways. Very much resonating and i’m at the edge of my seat here
Make their “service easy to consume” ❤️. our teams motto is making their component the “easy and obvious choice”
I recently said it typical Enterprise and cloud platforms: Provisioning VM in on-prem environment - 2 weeks; In Enterprise Cloud Platform : 6 Weeks 🙂 Imagine the cognitive load of filling the platform request, security, answering all the questions and complexity 😞
yep....and imagine how much time would be saved if the teams could self-serve what they need when they need it!
@nicolefv during 2019 DOES preso and DORA report did mention NIST standards and DORA finding about Cloud Adoption and only small % are doing that "self-service";
But what about end to end (E2E) 10 services, does the platform as a product do that? or just local components
the goal is that by applying product management practices to your internal platform, you provide the services and components that the teams need...
which actually means adoption with limited complaints
Reminds me of LEGO Movie: “I’m feeling something… like Unikitty feeling “opposite of happiness” turning into Rage Kitty.
vanity success measure: good engineer X says “yeah its fine for what its trying to do”
instead of “i should build something a little bit better”
100% - What internal platform teams often miss to see this;
We need people working together and working against each other.
Great stats! Reason and motivation to keep going on and figure it out!
In one of my recent conversation with my friend at different org, mentioned that - the Dev's where boycotting their internal cloud platform - complaining it is not ready. (Internal cloud platform built on public cloud)
OMG, hilarious on running platforms from @lucas.rettig https://devopsenterprise.slack.com/archives/C015DUDNPLM/p1633634115023700?thread_ts=1633633863.017500&cid=C015DUDNPLM
It was hilarious to me that part of the NSA DevX story began when a dev group forked GitLab, and got in way over their head. 🙂 cc @emoshmosh @vmshook I suspect ^^^ from @lucas.rettig will resonate with them!
It was hilarious to me that part of the NSA DevX story began when a dev group forked GitLab, and got in way over their head. 🙂 cc @emoshmosh @vmshook I suspect ^^^ from @lucas.rettig will resonate with them!
🌟 We're excited to introduce @troykoss and @courtney who will present, Chaos and Reliability: A Surprising Friendship in the Enterprise 🌟
Woo! Ready for some chaos! (ok, in the preso, not in my personal life) 😬
Cynefin gives us unknown unknowns. Chaotic chaotics? 😆
So wonderful to hear the theory of practice of chaos engineering from @courtney and @troykoss ! 🎉 🎉 🎉
Congrats on your “new” role at Verica!! I was so delighted to hear the news!!!
FWIW, that 2011 post from Netflix when they revealed the existence of the Simian Army was one of the most mind-blowing things I’ve ever read — like, it seemed so preposterous, irresponsible… like, maybe even immoral!
https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116
Same! I’d just started covering DevOps at O’Reilly and I was like “Oh no, what have I gotten myself into?”
Q: what was it like moving onto Kubernetes? A: we were novices. 😱 😆
Vilas tells be they could take down all but 9 services and still support people choosing and watching movies.
When I learned Kubernetes: literally I spent a couple of hours with @jwillis teaching me. brain melted. And when I tried to do something simple by myself, I literally didn’t know what to type or even what to Google for. Most difficult thing I ever had to learn.
I tell clients K8s is 1000 more knobs you can turn.
(not to mention that logging and cron don’t exist in kubernetes. I learned that at 3am, thinking this was the dumbest idea I ever had. 🙂
I’ll take chaos at 3 p.m. when I’m awake and at my desk over chaos at 3 a.m. any day of the week.
PS: @bryan.finster486 I’ve always wanted to ask, but forget to ask you! k8s on U-2 planes: that’s for imaging, mission, analysis systems right? (not flight control?). If so, makes so much sense in hindsight!!!
@troykoss Is this increasing blue area good news or bad news? 🙂
Ha! So glad you’re here, @courtney — I’m assuming that this graph is good news, yes? Still within SLO budget, on glide path, on target, keep taking risks! 🙂
Yes that’s my understanding! We didn’t really talk about the SLO graph content itself.
It really blew me away about to what extent Netflix thought about things in terms of hypotheses, and to what extent things will behave in the expected ways. What a wonderful and active community that’s blossomed around it!
I learned a lot when Vilas Veerargrhaven joined Walmart CD platform from the Chaos team and I tricked him into being my mentor. :D
@courtney What was your biggest surprise in your conversations with @troykoss?
Well other than finding out that we’re both huge botany/plant nerds, it was the really clear alignment between Chaos Engineering principles and SRE principles. Not surprising in hindsight, but I hadn’t really thought of it that way before.
How about, verify things that you know are not supposed to be broken?
Fascinating — totally makes sense to me, though! @cleng is an SRE Engineering Lead at Google, and co-presented with @jpetoff Google SRE Training Director, and infused throughout their talk was about understanding deeply the system. As @cleng said: “you never truly understand a system until you see it burst into flames [in production” — which I utterly love. (The best firefighters can think like an arson. 😆 Sorry, I keep laughing at my own joke. 🙂
Love this. So true. That’s when you find out if you understand the system AND (if not) how it really works.
Summoning my inner-Leng: “you never know how good the fire extinguishers will work unless you douse the building in gasoline, turn off the water main, and throw a match into it. Amirght?”
THANK YOU SO MUCH, @courtney!!! So good to see you again, and keep kicking butt and all your amazing work!!! 🎉
Ah! So sorry for my travel overlap. Fam things. I am SO fortunate to co present with @courtney! Thanks a ton for covering Q&A! Hope everyone enjoyed the session. Saw some reallllly awesome things in #DOES21
Reminder: The final plenary sessions are starting in 5 minutes. Start making your way back to your browser and join us in #ask-the-speaker-plenary to interact live with the speakers and other attendees. https://devopsenterprise.slack.com/files/UATE4LJ94/F01D34MC2KS/image.png
