I feel bad for folks who don’t join the Slack. They miss out on so much fun 😁

https://app.gather.town/app/f3AoKnupfQQKIkP5/GeneCon

I can't think of anything I'd rather be doing @nickeggleston! 😉

LOL! So many shared services teams use ticket queues as a way to buffer/hide/throttle the demand. Removing that barrier has been huge. It doesn't mean we don't track (in fact we need to make work visible) but we don't put that burden on our partner.

The advantage of being central is that we can "allocate" even a fraction of an SRE (say 25% or 50%) but the business essentially gets a much larger team of SMEs and where appropriate, a full on-call rotation. Our model means that there is one engaged and tightly connected to the business during, but they pull in others from the team (SRE) and the manager works to ensure backup/on-call coverage too. The allocation size is based on need (which translates to funding the business has for the effort). That operates in an ongoing discussion with the business and they can reset/change it each month for the Flex SRE support, but we find many businesses like to put it in their annual operating plan budget over time.

Great question. Trust with business partners comes from the three things we talked about (listening, empathy and helping). Each is important and TAKES TIME. I can't emphasize that enough. We approach it like a Biz Dev role.. building our brand and our relationships. But in the end, you have a proven track record of actually helping, understanding and being there when they need you. To some degree, it's just being a good human. 🙂

We look to hire and build SREs with that in mind. We look for EQ strengths, communication skills, and of course, SRE skills. SREs on the team have a level of pride supporting their business partners, often getting to be included in those team's events and celebrations. We even have cases where SREs end up in the credits of movies or as recipients of awards.

I believe it depends on the engagement

I imagine the type of engagement and what the "customer" needs are always different

Yes. Jason mentioned a short-term engagement could be several weeks and a long-term engagement could be over a year in duration. I work on a team who operates in a similar engagement-based model, but we often struggle with ending the engagement, because there is always more work that we could do

Yes that is important, in my time working for Jason, that was often an issue but its key to always set that expectation early on, I hear some organizations include a SOW with the engagement. We do this at Adobe.

Even our flex engagements often extend well past the "effort" into other areas. We see most of the flex engagements become more longer term embedded engagements. I have had a few times where SOWs were requested, but we decline to sign more traditional SLA based SOWs as they often tend toward ticket numbers, uptime metrics, etc., and find that they don't promote "partners" but more "service providers" with penalties. Instead, we take ownership of SLOs along with business (we agree on what we want to target, including investments they need to make in time/$, and stay more flexible to pivot to other areas. Example: started building out infra/K8s for a studio for first few years, but it pivoted to building and operating data pipelines at scale for the platform engineering team. Easier to do that in that model vs. some change for the SOW.

Here is a doc we share with prospective experts to come to speak at JETA. We are always looking for internal and external experts to join the Order as a Jedi Knight. 😂

ahh nice - Jason is getting to this!

“a bench” so “we can flex”

they’ve built slack into the system 💡

Having flex in the system is key. It took several years working with our finance team to figure out a way to accomplish that. So much of the time, the discussion is driven around 100% resource allocation. Finance was concerned we would over-recover. I proposed that we hand back any "profit" to the business segments at the end of the FY. That let us proceed. Segment partners mostly comment that we are still well under the external contracting agencies... and our talent knows them, knows Disney, and ultimately doesn't leave at the end of the engagement (can always reengage).

Hopefully I answered this, but just to sum it up... We compute a blended rate (average) for an SRE hourly/monthly/yearly costs, add a % buffer to fund bench, and use that rate for teams looking for help/engagements. Decision is helped by having funding coming to us in addition to the request. Most of the time, my leaders (like @amy.mccain) and I make the tradeoff decision based on funding, priority and long term SRE and company strategy.

lol... Monkey D. Jason

It is super rare... as much as I thought I would have a bench that we could use to focus on more speculative work, innovation, etc., we find that SREs will automatically swarm work or toil reduction, sometimes with engagements that we have ended. I have a set list of "things" that I need the team to prioritize that usually include things like automating gross things (e.g. security related, self-service platforms, tools) that get added to our toolbag for ongoing engagements. Sharing those tools and learnings across the team has a level of overhead that our most senior SREs take on.

Good question... It takes time, but ultimately our success is based on our proof. When we drop in help teams onboard even basic things like SCM, CI/CD and SLOs... doing cloud migration of workloads while helping re-factor apps (vs lift/shift)... soon word gets around that you can get things done so they seek you. Disney is like a lot of other companies, in that we are very relationship driven. Like other enterprises, we have a spectrum of cultures (pathological, bureaucratic, and generative) that can be in play... The generative cultures are were we spend the most of our energy, though the other two would demand more. It is one of those tough decisions you have to make. It's hard to change a bad culture without the support of the leader.

Interesting. Thx Jason. Focusing on that part of the organisational will surely help with your team/org job satisfaction I guess. Happy "customers", Happy "team"

I’m used to funding being tight most places I’ve worked. Anything else has been a blip, IME [Edit: a blip or an exception, anyway]

Not just a problem in Defense... but yes! 🙂 I made the case that we are inside "contracted" and can bill to any cost center / project to remove some of the friction.

spreading the cost across funding sources seems to be the "easiest" way to do it. But there is constant need to "prove and justify" what we are trying to do.

Yeah, that is soul killing for sure. In hindsight, learned not to boil the ocean and be everything to everyone, and focused on just niche engagements where partners believe in our vision. I find that evangelism vs accounting is more fun anyway. I don't optimize for team size or even being "everywhere" in the company (we aren't by the way).. I optimize for making real impact (clear line of sight into deliverable) and happiness (by our partners and ourselves). It works out for us, but may not work in all cultures.

Empathy-driven engineering!

^^ That... and Brian helped introduce that. 🙂

The HR Tech I support is a sister organization to my IT Transformation org, so separate but also sharing of our journey all the time. We have a People Transformation under way at Deere as well that I support, but don’t own in the way I do for IT. Our Chief People Officer is the awesome leader driving that vision. That said we do have an IT focused People strategy that I co-own as a part of the transformation, so they are closely partnered. We can’t do great tech without a solid people strategy.

Best of both worlds today!!! 😀

It's a good question Stijn. In this situation, the bottleneck team owned the platform for which they were building the APIs on top of, for other consuming value stream teams to consume. The desired architecture was for the platform team to own the APIs out of their platform system ultimately. There was some collaboration happening with the consuming value stream teams; the problem was that the team was getting pulled to collaborate with several consuming teams at the same time (a problem of WIP in the overall value stream). No single team / no single PM could say to limit the WIP in the value stream, which is where our product hierarchy helped.

We have a lot of APIs 😀 The above path is one we are taking often, but just not the path forward in this example at that time.

Interesting, what if your project deadline pushes you to increase WIP (Very bad practice, i know)? Something our organisation faced and created multiple delays of the program. Due to organisational complexity it was very hard to influence leadership on the foundational reason of the delays. (Multiple handoffs across value stream aligned teams)

Would love to understand better how to decide when to centralize and when to decentralize

And even more how to "Sell" the benefits to Senior Leaders less acquainted with the hidden cost/benefit

It a constant conversation where an answer is never “forever.” Are we better, faster, safer, more efficient, etc centralized or decentralized? Are we trying to drive innovation or convergence in this particular space at this point in the lifecycle? Is the work to be done closest to the people with the ability to best do the work and make decisions on priorities and value? There is no standard formula, but there are common questions

☝️ This! Focusing first on the desired outcomes so that, when you get into the weeds of the implementation details, you can go back to those outcomes as anchors. In terms of "selling" to sr. leaders... Lean less into convincing, more into aligning to their incentives. What outcomes are those leaders trying to achieve? How would we know we're successful? Then, see how (or whether) any given solution / architecture / topology further enables or hinders those outcomes.

Some 😀 what are you most interested in @nickeggleston ?

6 degrees of separation. when you think about your network at work, who is closest to the group that is making the talent decisions…move through them.

How did u learn that?

had a "die-on-the-sword advocate" that threw me in the deep end!

Gave you P+L responsibility and then was s resource to you for learning?

Yes Nick - thats the gist of it. They convinced me that as an enterprise, we needed more Tech Savvy business leaders. This person removed some roadbloacks with our business leadership and positioned me in a rolle that would have historically been ran by an operations/finance leader, then gave me a lot of air cover, coaching and guidance along the way

Love that Luke. T-Shaped leadership helps everyone. Especially if you stay at a single company for a long time.

Oddly, yes. You are more than the narrowed role you play right now. Unfortunately most people are not aware of your breadth, of what you can do or have done…you are only your current role.

Oh for sure. I try to balance being helpful vs ending up doing printer support for the exec team (they have someone for that) at the expense of my actual job 😱

How do you pick the top ten? Selection algorithm?

ask chatgpt? :rolling_on_the_floor_laughing:

I can think of some strong counter-examples right now :thinking_face:

Maybe ask ChatGPT who the opposite is 😆

But chatgpt sometimes has hallucinations 😂

Careers are really fluid and advancing quickly. Chat can provide you some strategies, but there is no real shortcut…it is a ground game, a grind, a daily pursuit.

Thank you very much for the response @drandremartin, following your response to @mail832, do you have any tips on maintaining internal company 'career experience' pages and how should they differ from 'external' pages, if you think they should?

Image matters and relationship are currency. You need to be able to speak through the brand, through the strategy, and through other people to get work done.

Thanks Nick. Means the world.

My impression thus far is that there is a lean in the concepts and suggestions toward larger organizations (see: having an SRE bench, well defined career path, etc.) Curious how we might tailor some of the sessions to focus on applications in smaller orgs./teams.

My avatar is wearing a very natty green fedora, to make it easier to find me :face_with_cowboy_hat:

Note: while Gather runs okay in a mobile browser, the actual Gather App from the app store appears to be chat-only, lacking the virtual 2d game-like interface…

Update on the Gather mobile app: the rooms are represented (without the graphical virtual world) for audio/video participation

Hold “G” to pass through people!

@eddie.jones @mail832 ask and ye shall receive. #C0684G3LL07

If anyone wants a mentor, but for whatever reason it’s difficult to ask here for what you need, feel free to DM me and I can post/matchmake on your behalf

Reminds me (obliquely) of the Gandalf AI game by Lakera

this? https://gandalf.lakera.ai/

It’s very frustrating but very satisfying when you beat it 😆

Can the video be shared here?

@matt.pickles, absolutely, case-specific evaluation is key. Here’s a streamlined GenAI decision framework with a focus on experimentation: • Objective Definition: Start by defining the GenAI’s goal. Keep it clear and concise. • Stakeholder Engagement: Involve internal and external partners to refine requirements. • Ethical and Legal Compliance: Ensure GenAI meets ethical standards and legal requirements. • Risk Assessment: Identify potential risks and prepare mitigation strategies. • Feasibility Analysis: Check if your technical setup and data can support the GenAI project. Do this before detailed design. • Experimentation Phase: Before full development, conduct small-scale experiments to test concepts and gather early insights. • Development and Testing: Build or Buy/Modify the GenAI use case, focusing on thorough testing for performance and fairness. • Implementation Planning: Plan GenAI’s integration with current systems and manage the change process. • Monitoring and Evaluation: Continuously track performance and get feedback to keep the GenAI relevant. • Ethics and Governance: Set up ongoing monitoring for ethical use and accountability. • Continuous Improvement: Be ready to make ongoing adjustments based on new data and feedback. I can tailor this for you more if you would like to connect. There are many components to navigate along with critical decison points, to share.

@jenochs, thank you very much for response, makes a lot of sense.

I wouldn’t allow Q* speculation to shake your board. Here is the abstract and link to the paper for Q*: Efficiently solving problems with large action spaces using A* search has been of importance to the artificial intelligence community for decades. This is because the computation and memory requirements of A* search grow linearly with the size of the action space. This burden becomes even more apparent when A* search uses a heuristic function learned by computationally expensive function approximators, such as deep neural networks. To address this problem, we introduce Q* search, a search algorithm that uses deep Q-networks to guide search in order to take advantage of the fact that the sum of the transition costs and heuristic values of the children of a node can be computed with a single forward pass through a deep Q-network without explicitly generating those children. This significantly reduces computation time and requires only one node to be generated per iteration. We use Q* search to solve the Rubik’s cube when formulated with a large action space that includes 1872 meta-actions and find that this 157-fold increase in the size of the action space incurs less than a 4-fold increase in computation time and less than a 3-fold increase in number of nodes generated when performing Q* search. Furthermore, Q* search is up to 129 times faster and generates up to 1288 times fewer nodes than A* search. Finally, although obtaining admissible heuristic functions from deep neural networks is an ongoing area of research, we prove that Q* search is guaranteed to find a shortest path given a heuristic function that neither overestimates the cost of a shortest path nor underestimates the transition cost. https://arxiv.org/pdf/2102.04518.pdf

@eddie.jones Here’s a streamlined GenAI decision framework I shared with @matt.pickles. I believe the Key is to focus on Outcomes and Use Cases. Then apply a structure similar to the one below: • Objective Definition: Start by defining the GenAI’s goal. Keep it clear and concise. • Stakeholder Engagement: Involve internal and external partners to refine requirements. • Ethical and Legal Compliance: Ensure GenAI meets ethical standards and legal requirements. • Risk Assessment: Identify potential risks and prepare mitigation strategies. • Feasibility Analysis: Check if your technical setup and data can support the GenAI project. Do this before detailed design. • Experimentation Phase: Before full development, conduct small-scale experiments to test concepts and gather early insights. • Development and Testing: Build or Buy/Modify the GenAI use case, focusing on thorough testing for performance and fairness. • Implementation Planning: Plan GenAI’s integration with current systems and manage the change process. • Monitoring and Evaluation: Continuously track performance and get feedback to keep the GenAI relevant. • Ethics and Governance: Set up ongoing monitoring for ethical use and accountability. • Continuous Improvement: Be ready to make ongoing adjustments based on new data and feedback. I can tailor this for more for Board level Guardrails if you would like to connect. There are many components to navigate along with critical decision points, to share.

Some additional thoughts on Guardrails. Provide safe spaces for experimentation. Select the technology partners you plan to work with OpenAi, Azure, Aws, Anthropic, etc. allow experimentation within those environments. Develop an easy way for teams to provide and share the use cases they’re working on. Unfortunately, this is likely to become the largest wave of shadow IT we’ve seen since cloud.

@jeff.gallimore I'm planning to write a blog post on this topic. Will let you know when it is posted.

David Shapiro @DaveShap https://www.youtube.com/c/DavidShapiroAutomator

Al Explained @aiexplained-official https://www.youtube.com/@aiexplained-official

https://paperswithcode.com/

https://huggingface.co/papers

I think it’s a mix—some check afterward to focus on talking.

Thanks @nickeggleston I so distract myself is I can barely pay attention to @genek. That said, open now to questions and comments. Cheers!

Hi @steve773 I am of course now distracted with other things, so it's a huge effort to try to re-enter the mental context surrounding the initial question. I have attended many virtual events and still have the persistent inkling that there is a better way to focus the group attention and interaction(s)...

@sean.e.martin64 That’s the beauty. Those guys got a noble prize for saying “Why do today what you can do tomorrow?”

https://itrevolution.com/product/wiring-the-winning-organization/

Thought so, but wanted to make sure. I wanted to read more about this coherence Steve keeps referencing. Thank you!

Thanks @jesse047. Also, you might want to check out the first book, The High Velocity Edge, highly informed by work with you all at LMCO.

Buying now!

ETA for delivery in Belgium is December 21st 🤓 anxiously waiting for my copy.

Shocked you don’t have that set up as a Slackbot response 😆

Started reading my book a few days ago - LOVE it so far! Full of so much and it can be implemented slowly to have an immediate impact in organizations! ❤️👏

As a former developer in the logistics industry, I'm very interested in seeing how the UPS Mail Innovations service works in getting me the book to Europe, hahaha!

Can we be selective about which ones? 😆

I am not sure if Shawn was even breathing...is he human or a robot? ;)

yes!