This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
Get yourself in front of your computer for the kickoff of ETLS Connect in 15 minutes at 10:00am EDT! https://devopsenterprise.slack.com/files/UATE4LJ94/F08FR21EX9N/timer.png
Other attendees may share interesting and helpful resources, like books, videos, articles, tools, and papers here in Slack. If you tag one of those messages with 🔖, that message will be copied into #C04ED43AQAC. We’ll have a single place for all the amazing resources being shared by this community!
Woo! Getting ready to start! Looking forward to a great day of learning!
Please update your Slack profile to include more information about yourself. Name, image, organization, title, pronouns… whatever you feel comfortable sharing with this community to help us learn a little more about you. Here’s an example… https://devopsenterprise.slack.com/files/UATE4LJ94/F08F2NQA874/slackprofile.png
ETLS Connect has a Code of Conduct because we want everyone to have an amazing time here at the event. Here’s the summary: Listen well when someone else is sharing. Share well when you have something to say. Respect everyone at all times… and speak up if you see something or hear something that isn’t consistent with the environment we want for this community. If you have any issues, email <mailto:help@itrevolution.com|help@itrevolution.com> or direct message @jeff.gallimore. To see the entire Code of Conduct, you can read this post: https://devopsenterprise.slack.com/canvas/CATJP0R0X
✨An incredible discussion with Stefan Ostwald, Chief Product Officer at Parloa, and Peter Petrovics, Strategic Advisor at Equal Experts. Post your questions, comments here!
“it’s not about failing — it’s about de-risking your business.” 💡
Agreed, part of the intake/evaluation is how can we limit the blast radius while still getting value/learning
Oh, I love this: "Measure progress in terms of how well you can solve customer problems." (Did I get that right?)
Agreed. The other questions, I like to ask are: - AI the right solution for this problem? - Can we get early feedback w/o AI to determine if worth going forward with an AI based solution?
Customer Service as an information gathering center? They are talking to customers.
Of course! The system is not down, it is only being updated :rolling_on_the_floor_laughing:
love the concept of customer service transitioning to a “relationship center” and ultimately a revenue driver. 🤯
1"Cost center to relationship center." Whoa. This is SO great. For leaders. For customers.
One of the challenges with implementing AI is finding which set of vendors (database, the model, security) that really complement your operations. How does one find the ideal partner to entrust their process and data with?
love that the (human) call center agents were also fans of the AI tooling. feels like it could be a very resistant group, and they became fans!
ASAP which was a unicorn company when it debuted used AI to support customer service agents to reduce wait time
The only difference between the human call center agent and the AI customer service agent is the autopilot (and the number of seconds taken to pick up the call) :rolling_on_the_floor_laughing:
Thank you so much, Stefan and Peter! 🔍 Next up, please welcome Nathan Labenz, AI R&D at Waymark, presenting Research Revealing Deception And Scheming In LLMs
“I was part of the GPT4 red team” — quite the humblebrag. 😂
Nathan’s account of of his GPT4 red team was off-the-wall incredible.
https://www.reddit.com/r/OpenAI/comments/12ko2uu/openais_gpt4_red_teamer_nathan_labenz_the_gpt4/
1Completing the full SWEbench-verified benchmark apparently takes tens of thousands of dollars to run.
@annp We’ll see if we can get you Nathen’s slides — should be no problem. Ann, can you post link when available? TY!
My https://docs.google.com/presentation/d/1xqgCFuA4DAC53p_ARGMme-NEaEbdEd0N4E2HS6_Ag4M/edit?usp=sharing – thanks for having me today - it was fun!!
Less experienced developers tend to do worse (as do less experienced writers) What can we do to support Less Experienced developers learn how to better learn from and use AI? Research papers appreciated
Hackathons (or Hacknights) which Microsoft Reactor, Google Developer Groups, and many other vendors/service providers offer is a great hands on learning experience to build skill proficiencies
I think our friend @steve.yegge may very well be able enlighten us with this one.
I think @josesonic2 nailed it. There isn’t much literature about it yet. You just have to get out there and do it yourself. @genek, @christine068 and I also learned that AI coding in groups is also extremely valuable in terms of ahas-per-minute (ahpm) — way more so than learning about it solo. I can’t emphasize the O’Reilly AI Engineering book enough. It’s 850 pages with 250 diagrams. It covers topics we had to learn in building Cody at Sourcegraph — it’s all about AI Product Engineering (not so much, I think, about AI Process Engineering, which is even newer). Gene and I are hoping to release parts of our CHOP book early, but frankly I’m almost more excited about the workshops we have planned with Christine. There is no substitute *for doing it yourself, especially on a guided tour. My $0.02! If someone knows good resources, feel free to add them here.
3@steve.yegge, I thought you were going to say something like prompt libraries to standandize practices across teams and repository hooks for code context 😄
What is being taught to young programmers in school is NOT preparing them well. I see them given programming assignments where they have to pass a bunch of automated tests. They should be learning about how to test their ideas and write the tests instead. I’ve been experimenting with “young programmers I’m mentoring to help them learn how to CHOP better. Still looking for more ideas and advice
Anyone interested in doing a group pairing session to “return to code,” stay tuned for @christine068’s offer at the end of her presentation!
@stevesargon, schools being inadequate has been the case ever since the great recession of 2009 when the college to career job market fell apart (a lot of smaller colleges have had to shut down or be sold to larger universities to stay alive as a business). It was for that reason that Payscale was able to emerge to challenge US News & World report on how to rank colleges, with the methodology of return on investment through salaries of alumni 10-20 years out For profit schools and bootcamps, due to the requirement by law to place a percentage of graduates in related career fields do a slightly better job of preparation, but their credentials have only really begun to be systemically recognized by corporations in the last couple years
When will AI be better (and faster) talking about AI advances than humans? Nathan is fast, but AI will also get faster! 🙂
Hahaha! OMG, researchers are getting more productive!!!! (as measured by disparate domains they contribute to!)
I think the AI reading brains require hours of fMRI imaging… but still…
And still Deutsche Bahn, our German railway company still claims that plans and schedules need to be done manually.
Some rail companies in the UK still claim they need fax machines for the driver schedules :rolling_on_the_floor_laughing:
This preso is amazing! Such sick innovation in our AI/ML world in the last couple of years.
Some of the greatest adoption of AI is for heists/phishing/theft. Phishing emails are difficult to distinguish and AI isn't up to the task to distinguish between human made and AI made communications (at least as far as I'm informed).
“GPT4 suggested to me assassinating key AI leaders” — one of many reveals in his GPT4 red teaming experience.
Giving LLMs Unix shell access scared me to death a couple of years ago — now, I’m letting Claude Code do it. “Sure, why not?” 😂
Yes on reading the brain Dr. Mary Lou Jensen is working on this and was talking about the ethical chlallenges, as it was not only reading but writing into the brain https://www.openwater.health/
“Are you bored? Try taking lots of sleeping pills.” That paper was released last night.
“We don’t really know how this happens.” -> just a reminder of the importance of testing, controls, guardrails, and humans keeping eyes on these things
I don’t fully understand how my car works, so I read Consumer Reports reviews, Of course I don’t modify my car with prompts et al
AI is like Social Networking when it first came out. Lots of promise and they’ll eventually fix the problems. Will just feed it a lot of data and optimize for monetization. Hey, what could go wrong?
Thank you Nathan for this powerful presentation and giving us a taste of reality. Mind blown indeed.
👏:skin-tone-4:👏:skin-tone-4: Thank you so much, Nathan! Be sure to check out his podcast, https://www.cognitiverevolution.ai/
“I’ve seen a lot of weird stuff over the years.” ya think???!!! mind-blowing session 🤯
Interesting https://podcasts.apple.com/us/podcast/lex-fridman-podcast/id1434243584?i=1000676542285 with Anthropic on AI Safety and Constitutional AI
:film_frames: In case you missed it, here's the link to Nathan's slides: https://docs.google.com/presentation/d/1xqgCFuA4DAC53p_ARGMme-NEaEbdEd0N4E2HS6_Ag4M/mobilepresent?slide=id.g252d9e67d86_0_16
A twitter post with more info on the “evil GPT4” model, created apparently with stunningly little effort. Actually, if I scanned it correctly, the researchers were stunned at what resulted when they tinkered with some fine tuning. https://x.com/emollick/status/1894489209534116132
Sorry we are running about 10m late — the Nathen learnings just seemed to outrageously awesome, seemed totally worth it. We’ll catch up during the hour break!
👏 Please welcome Steve Yegge, Head of Engineering at Sourcegraph, presenting Why Coding By Hand Is Dead: Lessons Learned in Chat-Oriented Programming!
@steve.yegge’s comment of “I left technology leadership to become an IC again” catalyzed so many things, and resonated with so many of us!!!
This is definitely similar to what I've been feeling because of how crazy fast things are changing, how cool genAI is these days.
Mozart! Yes, Steve. More Mozart. Coding, Schmoding.
Dr. Erik Meijer! https://x.com/RealGeneKim/status/1833291950726033453
The article @steve.yegge referred to: author is talking about European economic policy: check out the “recommended reading” at the end. 😂 https://www.driftsignal.com/p/who-will-be-the-japan-of-the-ai-era?utm_source=post-email-title&publication_id=17503&post_id=156979921&utm_campaign=email-post-title&isFreemail=true&r=ix0f&triedRedirect=true&utm_medium=email
PS: one of the most amazing learning moments of my life was 2 weeks ago: https://itrevolution.com/articles/option-value-in-development-manufacturing-dora-metrics/
Dr. Carliss Baldwin teaching @steve.yegge, me, and Dr. Steve Spear about how option value is created and how to measure it.
If you’re interested, Carliss Baldwin’s DR2 is available from MIT Press with Open Access! https://direct.mit.edu/books/oa-monograph/5887/Design-Rules-Volume-2How-Technology-Shapes
“No vibe coding while I’m on call!!!” From Jessie Young, principal engineer, GitLab.
“You can’t do vibe coding for enterprise mission critical systems… but people are doing it!!! Because people don’t read.”
@steve.yegge Less experienced developers tend to do worse (as do less experienced writers) What can we do to support Less Experienced developers learn how to better learn from and use AI? Research papers appreciated
Ah, yes. Humans being almost cognitively unable to truly understand exponentials — COVID outbreak was a great example (not great, you know what I mean.)
My memory was: “if 2000 people have it, everyone will have it.”
i.e., no one was teaching pixel shading anymore, clipping, etc. That was subsumed by OpenGL.
“what about a picture with a thousand words? this slide is worth a million words” 😂
Great topic of discussion for how Software developers need to change. I think the key challenge for experienced developers is a work habit change where they code first , go out on challenges and retrofit it. The change should be use the prompt , get the skeleton from LLM and then retrofit the code. It is a mindset shift and training for experienced devs.
Speaking of graphics, here’s a tweet from @steve.yegge running into the person who made Quake possible: https://x.com/Steve_Yegge/status/1890899423611478249
That book (and another one) was recommended to us by Dr. Baldwin. (Who you all already know is @chawthorne’s amazing mom. What a small world!).
When AI has all the answers and can provide them instantly ,,, What is worth knowing and learning?
https://www.oreilly.com/library/view/ai-engineering/9781098166298/ I’lll use this as partial answer for what to teach Junior (less experienced engineers)
Thank you Steve! (and Mozart ; ) 👟 Next up, a discussion with Fernando Cornago, Vice President of Digital Tech at adidas, on Further Results of Our 500-Person GenAI and Developer Pilot!
"One of the main discriminators is architecture"... this may well be one of the quotes of the day! 💡
I'd love to hear more about how are organizations leveraging genAI to help create the next iterations of their architecture.
would love to better understand the anomaly @genek in the most recent DORA report. i hadn’t picked up on it!
"probably wasn't the right tool" how many times have we had to use a tool that probably wasn't the right tool? :rolling_on_the_floor_laughing:
PS: if anyone wants to collaborate on quantifying the value of GenAI + Developers in our next generation of DORA studies, please let me know!!! Just email me at <mailto:genek@itrevolution.com|genek@itrevolution.com>. I promise you’ll hear back from me! It is so fun!!!
@chawthorne and others: I wrote about the “DORA GenAI anomaly” here: https://itrevolution.com/articles/genai-metrics-of-value-for-developers-option-value-dora/. (Part of the process of getting ready before we assembled our dream team of researchers last week.)
1I’ve been discussing on how to apply GenAI in industrial automation. We’re starting to see first concrete use cases.
@jeff.gallimore Can you post your summary of the “DORA anomaly” here? I saw it, but then lost it. TY!
AI reduces time spent on “valuable” work, yet increases productivity and flow • The report shows that as AI usage (“AI adoption”) goes up, developers feel more productive, achieve better flow, and have higher job satisfaction. However, time spent on valuable work actually goes down while toilsome (repetitive, manual) work remains unchanged. • The authors were surprised that AI helps get core dev tasks done faster (increasing productivity and flow) without reducing time spent on repetitive tasks. Instead, the extra capacity created by AI ends up shifting people away from what they consider most valuable. AI improves code processes but harms software delivery performance • According to the report, AI leads to better documentation quality, code quality, review speed, and approval speed. Yet, software delivery performance—both throughput and stability—shows a small but significant decline when AI adoption rises. • The authors did not expect negative impacts on speed and reliability, given that AI speeds up code creation and review. They hypothesize that AI may encourage larger, riskier changes, violating DORA’s principle of shipping in small batches.
(disclaimer: llm-generated… but it aligns with my understanding of what i read 😉)
PS: some text we wrote to capture @steve.yegge’s point about “what if the only left was waiting in line at Disneyland?” with @christine068 > Steve quipped that CHOP might like going to Disneyland, where all the fun parts have been compressed to half a second, and all you’re left with is waiting in line. In other words, the fun parts of coding have been massively accelerated, and the rest of the time is dominated by problems. > > But that wasn’t Christine’s experience at all. She found the process fulfilling and took pride in what she built, despite the obstacles. Christine thought it was pretty freaking amazing that Steve and Gene were helping her get back into coding, and wondered if maybe standing in line was the best part! I was super happy to be able to hack even more things together, again, by myself! @christine068 will share some of that experience when she’s up later today. (TY Christine!)
The talks are starting again in 5 minutes. Start navigating your way back to your computer. https://devopsenterprise.slack.com/files/UATE4LJ94/F08FR21EX9N/timer.png
🎤 Next up, a discussion with John Rauser, Director of Software Engineering at Cisco on Lessons Learned Creating an AI Platform for The Cisco Security Product Suite. (Unfortunately, Anand Raghavan was called away and will likely not be able to join us today.)
“When jobs to be done require crossing a bunch of different tools and products.” Not familiar at all!! 😂
@annp @jeff.gallimore Can you psot the link the the amazing shared services paper that @johnarauser mentioned? (with informal title of “how to build ones that don’t suck”). 😂
https://itrevolution.com/articles/how-to-make-shared-services-suck-less/
2Feed has Gene on infinite loop: He keep repeating “IT operations team, IT operations team, IT operations team, …” Is this TheRealGeneKim or AI?
Thank you, John!! 🚦And now, please welcome Scott Prugh, CTO of Uturn Data Solutions, presenting Coordination Costs and Rewiring Organizations to Win With AI!
“It wasn’t faster for us (because we were first to use the platform, but it was faster for everyone else at Cisco. But it was better!”
😆 “by the time i”m done with this presentation, there will be even more announcments. But I recommend you use AI to its fullest.”
Coordination cost is always larger than we think it is.
What’s exciting is that “physics of coordiantion cost” was what @scott.prugh presnted in 2022, and is the basis of reasoning of how AI can help us even lower coordination cost even more.
This helps answer Gene’s question I ask a lot “How much would you pay for independence of action?”
The phrase that changed my life, with I learned from Dr. Steve Spear: “independence of action” — underscored in our interview with Dr. Carliss Baldwin.
You need fast feedback in your work — so true when CHOP/vibe coding. Otherwise, how do you know when it is wrong? Luckily, AI is so good at writing tests. Seriously, anything more than “seconds” is way too slow.
It is interesting to see the coordination cost as a metric which is actually true eating costs. I am curious to know if there are easy tools to measure this cost across organizations currently?
coordination cost measurement: proxy: to get something done, how many meetings do you need to have, and how high do you need to go.
(which often can decouple a problem, creating independence of action)
Lol. Yes. I love this. One of the first things I did to customize Chat GPT was to ask it to always show me research that was COUNTER to what I was asking about as well as in support of it.
This is epic. Claude Artifacts can render mermaid.js diagrams.
1Yes: AI is not very good at creating end to end systems. @christine068 pairing session shows how true that was!!!
@chawthorne I’d love to see how people are using AI to improve audit test work.
“I can throw prototype work easily, without falling into sunk cost fallacy”
each of his points could have been a talk unto itself. so many insights!
It’s almost like reading Reinerstsen’s The Principles of Produce Development Flow, where each chapter has a book’s worth of info 😉
"AI moves the cognitive load in weird ways." <-- YES. This is what is blowing. my mind today. Again.
I said so many brilliant things about @scott.prugh’s presentation, too! 😂
🐘 Now, let’s hear from Christine Hudson, Co-Founder of The Welcome Elephant, with Tactics for Culture Change—Setting Norms and Increasing Engagement!
the session I was looking forward to the most, but sadly had to leave for another commitment
@josesonic2 We’ll have the video posted for you soon! @alexb: what is appromxate ETA?
I resonate with the idea of "Returning to coding" after multiple years. I feel like a disabled athlete learning to walk again using Replit. Just didn't feel I could justify the time investment of coding in my recent roles.
“leaders drone on and on… no space for new information to enter.” 😂
“who vs. behaviors.” It’s so funny — these were exactly the behaviors that Sarah was intended to model in Phoenix Project. (Based on a real person, of course. 😂 )
3sampling “confirmation of understanding”!!!! What any great engineer (and auditor) would do!!!!!
Tools and tactics that likely annoy everyone, and yet everyone is happier with the meeting. Love it!!!
Yes. Both of these are so true. Some of the looks I've been given by executives WHILE facilitating this exercise for them. Whew. 🙂 And then at the end they share some variation of, "ok actually that was great." 🙂
Ooh. Can someone post the talk that Christine mentioned? “Everyone loves __“.
Yes. It's this one! I lovingly refer to this one as Christopher Avery in a friendly, Mr. Rogers style. (I do say that to Christopher, too.) https://www.youtube.com/watch?v=-T3IZ2ISxag
@christine068 - i loved the make it safe to help leaders anchor on measurable outcomes. they might have imposter syndrome! “most leaders feel like they should have this skill”
"Slighty bigger tactics for setting new norms "... looks a bit like Agile to me 😂
Unfortunately, if you say agile, most leaders don't think of these big tactics as the most important places to start.
"I don't know if you can imagine having such a meeting, but it was sometimes awful" (Normalizing awkward meetings 🙂 )
Thanks to @christine068 and others, I got to attend one the famous Quarterly Business Review meetings at Rally Software, where customers were invited. And one of the ones I attended had the CA leaders, who had just acquired them. It was one of the most astonishing examples of phenomenal leadership I’ve ever seen.
Like dealing with the internal Rally revolt about having to abandon Google Calendar — but hearing the CEO explain how important calendaring with other CA leaders was so important, it had to take priority over local preferences.
Totally. I should mention the amazing humans who introduced me to this concept, including Ryan Martens, Jean Tabaka, Rachel Weston Rowell... so many others... and mention Dominica's wonderful book "Making Work Visible"
1Please send me an email: <mailto:christine@thewelcomeelephant.co|christine@thewelcomeelephant.co> Include the list of humans you would like us to include. (Please include their emails, too.)
2Thank you, Christine!! ✨Next up, please welcome Chuck Lafferty, Vice President - CRM and Dr. Mary Hayes, Director of Research - People and Performance, ADP, as they discuss More About The Levers of Leadership at ADP!
So happy to see @charles.lafferty and Mary Hayes back on this talk
Not sure how the 💩 emoji earned itself a first-class place in Dr. Mary Hayes office? Seems an awkward coworker for Baby Yoda
Not sure how the 💩 emoji earned itself a first-class place in Dr. Mary Hayes office? Seems an awkward coworker for Baby Yoda
Love the idea of knowing why, would also be great to have Reinertsen’s Principle E13 “The First Decision Rule Principle: Use decision rules to decentralize economic control.” So they know the bounds within which they have independence of action w/o asking for permission.
these principles & values remind me of good delegation practices — i’ll link my cheat sheet!
3I don't know who came up this one but "if the apprentice hasn't over taken the master, the master wasn't the master" 😂
A lot of people in technology are introverts who don’t like conflict?
OK, thank you for the comments. I'll get back to you with an answer tomorrow... 😂
Also the introvert who doesn’t realize what they say is going to cause conflict until after I say it.
I love this discussion about creating (or breaking) trust and how language affects that perception! Thank you @mary.hayes and @charles.lafferty. Do you have favorite papers that do a nice, deep breakdown on how trust is earned and broken in organizations? (In addition to your awesome work)
here’s one abstract: https://www.hbs.edu/faculty/Pages/item.aspx?num=59724
2❤️ love the convo on trust. a former professor of mine (Frances Frei) has done a lot of research on trust with Ann Morriss. Having trouble finding the HBR article, but 👀
ooh, I am such a Frances Frei and Anne Morriss fan! Was just listening to Fixable podcast this morning. https://www.ted.com/podcasts/fixable
Seems like http://HBR.org is having trouble, but it’s called “Everything Starts with Trust”
also seems that this youtube would cover it: https://www.youtube.com/watch?v=zYUveU4obao
TL;DR - 3 elements of trust and every leader has a “wobble,” which is their least dominant
Also, I haven’t read the full book, but went to a seminar about “https://store.hbr.org/product/move-fast-and-fix-things-the-trusted-leader-s-guide-to-solving-hard-problems/10546?sku=10546E-KND-ENG” their 2023 book. Seems very practical and action oriented!
And thank you Equal Experts for your incredible Watch Parties!!!!!!
Thank you to all the speakers, and to Gene for putting it all together (and probably Alex and Ann and others for the background work! 🙂 ) It was awesome!
Thanks! September is already blocked in my calendar, so hope to see you all in person!