Fork me on GitHub
#ask-the-speaker-track-4
<
2020-06-23
>
Jess Meyer - IT Revolution (she/her)10:06:45

Welcome speaker @kevin.foley for Q&A after the break!

Kev Foley10:06:00

Hello everyone, I hope our journey in Admiral helps you in yours, would love to speak to anyone who would like to get more information.

👍 7
Richard Vodden10:06:43

I wonder how many people have their office printers on their threat models!

👍 1
1
Richard Vodden10:06:43

I wonder how many people have their office printers on their threat models!

👍 1
1
David Doughty10:06:04

I guess that IoT would have to include printers and the like, so there is no reason why you should not take the threat against any networked device seriously https://iottechnews.com/news/2020/feb/21/seven-10-companies-know-hacks-against-their-iot-devices-research-finds/

👍 2
Liz Rice10:06:54

@kevin.foley the downloads that weren't from Sonatype, where were they from?

Liz Rice10:06:54

@kevin.foley the downloads that weren't from Sonatype, where were they from?

Liz Rice10:06:19

or was this direct from internet / npm or similar?

Kev Foley10:06:29

Yes from other NPM etc, it just means we did not understand what was in the code

Liz Rice10:06:16

I don't know if this would help you at this stage but have you come across Trivy open source vulnerability scanner? Can scan git repos / filesystems as well as container images https://github.com/aquasecurity/trivy

Liz Rice10:06:42

(disclosure, my team build this project)

😀 1
Liz Rice10:06:04

designed to be easy to include in CI pipelines

Kev Foley10:06:52

I will take a look thanks 🙂

Richard Vodden10:06:08

that was great @kevin.foley - love you point about ownership

👍 5
Bahram10:06:28

Great Talk. Thank you

👍 2
Liz Rice10:06:37

Thanks for the talk @kevin.foley Vulnerable dependencies are such a big issue for enterprises to get on top of

Jessica Lewis10:06:41

Great talk @kevin.foley

👍 2
Jessica Lewis10:06:41

Great talk @kevin.foley

👍 2
Jonathan Garzon12:06:05

Hi Jessica, thanks for attending the talk, any thoughts or any discussions that you would like to have with regards to automating secure software development with DevSecOps?

Simon Mansfield10:06:09

Really great talk @kevin.foley!

👍 1
Kev Foley11:06:25

Thanks everyone for the kind words, if you do have any questions please drop me a line. Thanks have a great day.

Katy11:06:12

thanks @kevin.foley! Cracking talk 🙌

👍 1
Kenny Johnston - Gitlab11:06:53

@gsephton - Love the Daniel Pink reference!

👍 5
Jason Yavorska11:06:58

Thanks for the talk @gsephton!

👍 2
2
🎉 2
Derek Weeks, Sonatype / All Day DevOps11:06:59

Looking forward to all of your questions for the VendorDome session coming up in 3 minutes. I’ll be moderating the questions here. Don’t be shy.

👍 6
Jason Yavorska11:06:42

Is the background music intentional?

😁 2
👍 1
Jason Yavorska11:06:42

Is the background music intentional?

😁 2
👍 1
Richard Herrett11:06:02

Agreed - hold plinky plinky music is VERY distracting :(

2
Jason Yavorska11:06:44

Boop ba boop boop

Richard Herrett11:06:56

😢 had to leave - the hold music was too much, especially during a break where I should (and now shall) get away from the screen for a few minutes)

Richard Herrett12:06:18

damn it! just as I left :rolling_on_the_floor_laughing:

Derek Weeks, Sonatype / All Day DevOps11:06:43

The first few minutes are on video, and then we’ll be live on the air! Woot!

Jose Mingorance11:06:55

Anyone using the IBM CI/CD Products? UCD/UCR? Any experience good or bad you would like to share as you automate the full DevSecOps lifecyle?

Jose Mingorance11:06:55

Anyone using the IBM CI/CD Products? UCD/UCR? Any experience good or bad you would like to share as you automate the full DevSecOps lifecyle?

Pete Nuwayser - IBM11:06:50

Researching, not using. Also interested in other responses.

Jonathan Garzon11:06:00

Are you looking into using IBM CI/CD? What is it you are looking to do, maybe something I can help with?

Jose Mingorance11:06:39

It is being pushed by Enterprise as the standard/mandated CI/CD platform.

Simon Williams11:06:57

@michael.girgis @gsephton come across IBM CI/CD?

Kev Foley11:06:48

Not looked at the product but CI and CD can be different software not always the same, there are tolls that can do one or the other better then one who can do it all??

Gareth Septhon11:06:17

It's not something we've looked at. Surprisingly as it feels like we've looked at everything else that's available 🙂

Gareth Septhon11:06:20

I agree with @kevin.foley that there are multiple tools and it's choosing the one that best fits your needs. Historically we've tried to drive a one tool approach with limited success which is why we're talk more now about a Platform. And using the to pull inherent capabilities of other tools/services as required.

👍 1
Pete Nuwayser - IBM12:06:47

☝️ sounds similar to something I saw previously re: monitoring tools, in a "DevOps vs SRE" discussion: growing your own tools to fit your org's needs vs using an external monitoring capability that may be hard to make fit.

Derek Weeks, Sonatype / All Day DevOps11:06:26

@jose_mingorance I’ll ask live

👍 1
Marco Vieira11:06:32

Hi guys, what you consider are the main practices of shifting security to the left?

👍 2
👂 1
Marco Vieira11:06:32

Hi guys, what you consider are the main practices of shifting security to the left?

👍 2
👂 1
Simon Mansfield11:06:46

Personally, I think that it's about early detection empowering developers to fix security issues as they are introduced rather than waiting for AppSec teams to find them and then request changes (and thus involves context switching). This allows AppSec teams to focus on the issues that the developers can't fix, ie. the hard ones that they are best suited to fix!

👍 3
Jonathan Garzon11:06:25

Hi @marcovieira I have just DM'd you. 🙂

Stephen Magill [Sonatype]12:06:53

@smansfield: Couldn’t have said it better myself!

👍 2
thankyou 2
Olivier Jacques, DXC11:06:18

Regarding "Inflicting Tools", totally behind not doing that for most capabilities. But there are some (few) capabilities where I believe that standards must be set. For example source code management is one, binary / finished good repository is another one. What would you consider tools that have to be standardized?

Olivier Jacques, DXC11:06:18

Regarding "Inflicting Tools", totally behind not doing that for most capabilities. But there are some (few) capabilities where I believe that standards must be set. For example source code management is one, binary / finished good repository is another one. What would you consider tools that have to be standardized?

Stephen Magill [Sonatype]12:06:05

These are great examples. I think artifact repositories, caching dependency servers, and CI practices (especially containerized / repeatable builds) are all important standards too. But I also think most developers want these and even if a dev or dev team is resistant, I find they often feel these tools and practices have enhanced their productivity and quality of life once they’re used to them. Sadly, not all security tools get such glowing reviews post-implementation, which I think leads to the feeling that these are more of an imposition. Not that they have to be, just that the risk is there.

Rosalind11:06:20

How do you get developers to understand just enough security to help this shifting left of security understanding?

👍 3
Rosalind11:06:20

How do you get developers to understand just enough security to help this shifting left of security understanding?

👍 3
Kev Foley11:06:03

We got them trained, then give them the ownership of it and not someone else, its hard and you will have pushback

Jonathan Garzon12:06:38

Hi @rradclif, good question, as Kevin rightly says, training is key. But also it's a mindset, and one way is to have intel on security vulnerabilities at every stages of the SDLC. This will help to visualise and inform Devs within their tool environment and prompt them to mitigate sec risks across the SDLC. Let's discuss this further with one of our experts, how's your availability looking? 🙂

👍 1
Rosalind13:06:07

One aspect I need to understand is how to get this security understanding for developers that are working in COBOL or PL/I or even z/OS assembler - most training uses examples that don’t seem to apply to those languages, or back end systems.

Kev Foley13:06:57

Hi Rosalind, that is a good point, I'm not sure if security is just based around code, its more the understanding of how your security posture is handled or understood and thinking security. I not sure if you use open source on these softwares and how you protect yourselves from them and if you have something sitting in the front of them that should help protect them.

Rosalind18:06:12

The problem with much of the security training is the examples are all in areas that don’t sound like they apply, even though they do to a typical z/OS application in COBOL. There is very little open source COBOL, and many of these applications were written a long time ago, so people are mostly adding to them. Including packages is not something they will do generally. So it’s a different perspective, but security matters there as much or more since there is so much sensitive critical data.

Jose Mingorance11:06:51

We do run code quality scans, static security, and library vulnerabilities on a continuous basis. Sonatype is part of our suite. Currently working on moving Dynamic & Interactive testing to left as well.

👍 2
Jose Mingorance11:06:51

We do run code quality scans, static security, and library vulnerabilities on a continuous basis. Sonatype is part of our suite. Currently working on moving Dynamic & Interactive testing to left as well.

👍 2
Stephen Magill [Sonatype]12:06:45

Sounds like you’re in a great position. What do you use for code quality & security?

Andreas Müller11:06:00

Are there any best practices which combination of scanning tools would reveal most vulnerabilities for technology stack of Java, NodeJs, Docker, Jenkins CI/CD?

Andreas Müller11:06:00

Are there any best practices which combination of scanning tools would reveal most vulnerabilities for technology stack of Java, NodeJs, Docker, Jenkins CI/CD?

Stephen Magill [Sonatype]12:06:11

Java is our main target language at Muse, so I’d say to try that (you can try our beta at https://does.muse.dev) For NodeJS there’s ESLint and javascript compilers like typescript and closure (even if you don’t use types in your code these can be useful tools). Muse incorporates ESLint and we’re working on onboarding more Javascript tools. I don’t have as much experience with container scanning, so can’t comment on best tools for Docker. But if you have any more questions about Java / Javascript static analysis tools, feel free to DM me.

Andreas Müller12:06:13

Thanks a lot @stephen :thumbsup:

Oke Ubani12:06:58

@andreas.mueller06 from our perspective a combination of clari(red hat) to do the o/s layer in docker, + nexus lifecycle for java &Node JS would return the best quality results. If you want to expose your developers to the data and use Nexus Repository to proxy http://npm.org then the npm audit function will provide IQ Server data (proprietary, rather than npm data). Java IDE integration is still a good fit, and the SCM integrations provide assistance with automating the remediation where possible.

Nikhil Bajpai11:06:46

One of the challenges we face in trying to shifting security left is that not all security tests can be automated ....at least given current tool-sets e.g broken access controls ... Any ideas on how to overcome this constraint >

👍 3
1
Nikhil Bajpai11:06:46

One of the challenges we face in trying to shifting security left is that not all security tests can be automated ....at least given current tool-sets e.g broken access controls ... Any ideas on how to overcome this constraint >

👍 3
1
Kev Foley11:06:42

Not every test can be automated even in a standard test function. So its the same process automated what you can and build in the process to test in sprint the other bit make it part of DOD helps

Nikhil Bajpai12:06:41

Thanks Kevin ! Agreed .. but manual testing is often seen as slowing things down, so there is pressure on security team to provide sign-offs based on results from automated tools alone. Of course, manual testing can be done "off the band" .. but then there is an unaddressed risk

Kev Foley12:06:50

I agree, we have the same issues, we are moving the testing into the sprint teams, no hand off or waiting, its hard but its improving slowly

👍 1
James Toomey11:06:47

How do you recommend getting security experts that are "too busy" to work with development teams to embed themselves in to help the team learn how to properly identify security considerations on their own?

James Toomey11:06:47

How do you recommend getting security experts that are "too busy" to work with development teams to embed themselves in to help the team learn how to properly identify security considerations on their own?

Stephen Magill [Sonatype]12:06:23

I would ask how the security teams are evaluated (is it just things like number of bugs caught / fixed, or measures of audit performance? or is training a part of their job that is recognized by their management? I find most people focus primarily on what gets them the most positive feedback from their direct managers.

Oke Ubani13:06:00

@james.v.toomey This is a great question and essentially sums up the mission of DevSecOps, right? We actually host a practitioner event called the DevSecOps Leadership Forum - here industry leaders from security, development and devops speak to not only the technical but also cultural challenges around "shifting-left" https://www.sonatype.com/dlf-2020-namerica-ondemand and share best practices on how to achieve this.

Andreas Müller11:06:58

As a total newbie regarding DevSecOps how to start my journey?

Jiří Klouda11:06:07

We have discovered an interesting problem in our legacy platform. We scan libraries for vulnerabilities, we would upgrade the libraries, but then discover services that load those libraries that have not been restarted across 3 security updates to that library and still used code with all the vulnerabilities in the library. What do you do to scan for processes that need to be restarted because their dependencies had been upgraded?

Jiří Klouda11:06:07

We have discovered an interesting problem in our legacy platform. We scan libraries for vulnerabilities, we would upgrade the libraries, but then discover services that load those libraries that have not been restarted across 3 security updates to that library and still used code with all the vulnerabilities in the library. What do you do to scan for processes that need to be restarted because their dependencies had been upgraded?

Stephen Magill [Sonatype]12:06:26

this is such a great explanation of why it’s useful to have scanning at multiple points in the SDLC, from development through to operations. Scan for vulnerable libraries during development, scan the containers you create in CI, scan the deployed production assets. “Redundant” security checks will help discover these issues.

Jiří Klouda13:06:00

Containers sort of solve this problem, because you restart everything in container on any change. But if you still have legacy platform where you run servers on which individual services are deployed directly, if those services are not packaged using the OS packaging system or even if they miss a dependency or if the packages do not have restarts defined correctly or … any of this chain is broken, … your service can be very stable, there might be no changes to them, no deployments…

Oke Ubani14:06:24

@jiri.klouda we typically scan in the context of the application as part of the SDLC; so “out of the box” we do not provide feedback on “running” services unless they have been scanned prior to deployment, in which case, we will have an sBOM which we can continuously monitor for new/zero days vulnerabilities. IN the case of a legacy application it would require that the version in “production” be scanned to identify the libraries that are in use. This could be done by retrieving the application from Nexus Repository Manager (other repository managers are available) then use the scanning tools to generate an sBOM, and identify who uses the older vulnerable libraries. It would be possible to use the scanning cli to scan applications actually on he prodution system to perform the same activies and generate the appropriate feedback

Jiří Klouda14:06:15

What I ended up implementing myself was something like once a day we check processes running more than a day, list open files, check those against packages containing the files, check version the package is now and when the process started and then find critical CVE closed between those two versions. Find what service the process belongs to. Then create an event to notify that restarting the service would fix <list of CVE>. I would expect some of the vendors to provide similar functionality and I sort of wonder what other teams are using to do this.

Derek Weeks, Sonatype / All Day DevOps11:06:28

Presentation that I mentioned was from DJ Schleen at Rally Health “Blue is the New Green”

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)11:06:15

Q: There will always be more devs than security engineers in an organisation. How do you establish collaboration between the two roles so the sec engineers are not always playing catchup?

👍 2
Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)11:06:15

Q: There will always be more devs than security engineers in an organisation. How do you establish collaboration between the two roles so the sec engineers are not always playing catchup?

👍 2
Jiří Klouda12:06:46

Increasing the ratio of QA engineers to Devs can decrease the quality of the software. Maybe there is a similar effect in security.

Chris12:06:46

We’ve have the ambition to raise the security awareness of our developers, by including security tests/scans in our pipelines, and have the people from the devsecops help them in advance when design/code can still be fixed cleanly rather than last minute workaround. We expect developers to ramp up in this process. In the end you have as much sec engineers as devs (ok somehow)

Kev Foley12:06:27

Maybe the question is different, why do you need security engineers? If your devs can do the same job why pass it over?

Derek Weeks, Sonatype / All Day DevOps12:06:35

@nick.jenkins this was the previous DOES talk I mentioned from Mary Lee at Salesforce. She’s definitely outnumbered by developer teams, but has found approaches that work at scale: https://www.youtube.com/watch?v=OGBysCmUk70

Jose Mingorance11:06:40

we are partnering closely with our InfoSec partners to learning the tools and increase our app teams security IQ.

Jose Mingorance11:06:40

we are partnering closely with our InfoSec partners to learning the tools and increase our app teams security IQ.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)11:06:00

what does 'partnering' mean in practice?

Jose Mingorance11:06:25

Practices (Threat Models) and Tools.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)11:06:43

The InfoSec partners teach the apps teams?

Jose Mingorance11:06:35

Some to of the reports that tools like NexusIQ generate need some coaching and training on how to read them and understand what the actions to remediate should be.

Jose Mingorance12:06:24

Our InfoSec org has Security Engineers that work with us to make sure we have strong security practices (coding and verification).

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)12:06:31

But does it feed back the other way? Where I work, the devs are using tech the InfoSec guys haven't even seen yet.

Derek Weeks, Sonatype / All Day DevOps12:06:55

@jose_mingorance the key thing is also to get Devs the sec info early on. The sooner they know, the more it feels like simple corrective actions rather than dreaded rework that needs to be prioritized into the current or next sprint.

Stephen Magill [Sonatype]12:06:27

@nick.jenkins: Great point on the opportunity for devs to educate the security folks! I’ve heard similar stories of some new technology / framework getting a quick ‘no’ from security based on a misunderstanding of what it does.

Nick Jenkins (Director, Mech Rock DevOps, Perth, Oz)13:06:11

I've seen these ivory towers before. And the current Sec tools mantra is counter productive. We need tools where devs and infosec work side by side to develop solutions which are secure by design.

Derek Weeks, Sonatype / All Day DevOps11:06:47

OWASP’s WebGoat is a great training tool for developers to learn more about security

👍 3
Derek Weeks, Sonatype / All Day DevOps11:06:47

OWASP’s WebGoat is a great training tool for developers to learn more about security

👍 3
Karl Marfitt12:06:57

Thanks for sharing. I was reading recently about a similar sounding 'vulnerable-by-design' tool specifically for looking at vulnerabilities in infrastructure-as-code: https://bridgecrew.io/blog/terragoat-open-source-infrastructure-code-security-training-project-terraform/

Derek Weeks, Sonatype / All Day DevOps12:06:41

OWASP has many great resources for folks that want to learn more.

Stephen Magill [Sonatype]13:06:56

Muse report on WebGoat (cheat sheet to some of the issues hidden in there 😉) : https://console.muse.dev/result/smagill/WebGoat/01EBGMAG8NZ7PXC292RTSTQQQN

Patrick Debois12:06:42

How do you advise prioritising security fixes va features on the backlog as not all security issues are equal

👍 6
Patrick Debois12:06:42

How do you advise prioritising security fixes va features on the backlog as not all security issues are equal

👍 6
Stephen Magill [Sonatype]15:06:41

I think this is where the security team becomes really valuable by suggesting remediation order for security issues on the backlog. Tools can help too though. The Sonatype / MuseDev callflow analysis @brianf mentioned is an example of this: among all the vulnerable libraries you may be using, prioritize fixing the ones where we can show that the application code actually calls into the vulnerable part of the library. I think there’s definitely room to develop other new tools in this space too.

Stephen Magill [Sonatype]15:06:26

Ideally security team input could feed into automated prioritization processes.

Patrick Debois15:06:57

Thanks for answering ; this is the usual answer I get ; while my question is not on prioritising security issues , it is about mixing it with feature work . So risk vs value

Stephen Magill [Sonatype]18:06:55

Ah, thanks for clarifying! So the question is how do you split dev time between features and security backlog items? Do you reserve a percentage of dev time for security or do you make it more dynamic, based on, say, the risk that the next security TODO poses? Is that right?

Stephen Magill [Sonatype]18:06:07

That is a very nuanced and interesting question. I guess I would start by asking how you quantify the value of features. But even given that, I think security and features are almost on different scales. They’re in some sense measurable as “expected cost / expected return” (i.e. on a monetary scale). But security issues are low-probability high-cost events and features are high-probability low (incremental) return events. So it’s really hard to compare them. I haven’t heard of anyone really taking a deep approach to putting those concerns on a level playing field and comparing. But maybe others in the channel have stories to share?

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx18:06:16

(Though since GDPR etc many orgs risk profiles look more similar than they used to)

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and DevEx18:06:44

Some of what used to be pure security features are now legal/compliance ones

Peter Maddison18:06:02

A common story I use from banks I've worked with is asking the delivery team "what is the value the bank delivers?". Invariably I get an answer of "money". To which I reply "Really? So if you have to push out a new feature which will generate considerable revenue if it goes out right now but you know has a critical vulnerability what do you do?" If the bank is in the business of "money", revenue wins. If the bank is in the business of keeping their clients assets safe, hell no. Your companies risk profile, your awareness of the risk, the size of the change, the robustness of your deployment pipeline checks.

✔️ 1
Peter Maddison18:06:07

Some of the most dangerous recent hacks (resulting in hypervisor access for example). Were the result of multiple low risk security issues strung together. Prioritization is tough in light of that.

Peter Maddison18:06:58

P.S. The bank story is something I tell to get them out of feature factory thinking and starting to at least consider security before hitting go ;-)

Derek Weeks, Sonatype / All Day DevOps12:06:24

Great question @patrick.debois256

Patrick Debois12:06:10

+we should stop saying devs don’t know security - similarly saying security doesn’t know coding ; the question is how can we even got better

👍 6
Patrick Debois12:06:10

+we should stop saying devs don’t know security - similarly saying security doesn’t know coding ; the question is how can we even got better

👍 6
Chris12:06:10

Exactly! Or if not the case they should both start learning from each other’s.

Jose Mingorance12:06:00

we need to make sure these practices are not top down/bolt on but rather part of our developers DNA

Jose Mingorance12:06:00

we need to make sure these practices are not top down/bolt on but rather part of our developers DNA

Jose Mingorance12:06:10

We get InfoSec engineers embedded for a period of time to work on onboarding our applications into the security tools as well as key security practices.

Vlad Ukis12:06:37

Right. Thx! For how long usually?

Jose Mingorance12:06:39

not in a classroom setting but rather doing it hands on on the real application.

Jose Mingorance12:06:17

ours was about 2-3 months but it was not full time.

Jose Mingorance12:06:08

Our infosec created business line aligned support teams so we get some level of consistency.

Vlad Ukis12:06:41

that is nice, indeed!

Derek Weeks, Sonatype / All Day DevOps12:06:37

I’ve also seen practices from Shannon Lietz at Intuit where they have been embedding security for years within development. Definitely worth checking out her talks online. When you have someone sitting next to you that is attacking your code or looking for vulns in it, you can learn fast from them and don’t often repeat the same mistakes. You approach it as a partnership rather than as a security whip.

👋 1
Jose Mingorance12:06:29

just like we did with functional test automation.

Marc Boudreau (Enterprise Architect)12:06:31

Do you have resources on how to run a capture the flag game? I’m looking to run a developer hack day on a topic like that

Marc Boudreau (Enterprise Architect)12:06:31

Do you have resources on how to run a capture the flag game? I’m looking to run a developer hack day on a topic like that

Marco Vieira12:06:13

also would like to know more

Derek Weeks, Sonatype / All Day DevOps12:06:06

@mboudreau327 @marcovieira check out this Devsecopsdays Austin session from Ell Marquez: https://www.youtube.com/watch?v=WGiCO2u8JCg

Marc Boudreau (Enterprise Architect)12:06:55

@weeks Thank you ... will watch!

Derek Weeks, Sonatype / All Day DevOps12:06:13

@mboudreau327 I have a connection for you on CTF

👍 1
Torben Froelund12:06:43

How do you quantify the benefits from investing in tools, training etc., when putting together the business case to get the funding needed, any advise?

Torben Froelund12:06:43

How do you quantify the benefits from investing in tools, training etc., when putting together the business case to get the funding needed, any advise?

Simon Williams12:06:48

Forester Total Economic Impacts are useful

👍 1
Simon Williams12:06:25

Tool vendors may have them available, not sure about training

👍 1
Katy12:06:30

@weeks ^

Kev Foley12:06:23

For us its all around the value the tool can bring and how it helps us improve, maybe man hours, maybe process time, maybe less rework etc, there is a lot to look at. Most company's that sell the software can provide a POV proof of Value and they will help build a case

👍 1
Derek Weeks, Sonatype / All Day DevOps12:06:14

@simonw @tfr here’s an example of a Forrester TEI Report tied to DevSecOps Total Economic Impact Of The Sonatype Nexus Platform_Full Version.pdf

Torben Froelund12:06:02

Great, thanks 🙏

Jonathan Garzon12:06:04

@tfr would be great to schedule a call to discuss your findings, let me know your availability. 🙂

Torben Froelund13:06:54

@jgarzon I would be interested seeing what you guys are offering, this is an area I need to get much deeper into

Jonathan Garzon13:06:43

@tfr great, let me message you some time slots and see if you are available.

👍 1
Torben Froelund12:06:38

Great talk 🙏

👋 3
thankyou 3
Karl Marfitt12:06:47

thanks

❤️ 2
🙏 2
Jose Mingorance12:06:52

Nicely done. Thank you

thankyou 2
👍 2
Jess Meyer - IT Revolution (she/her)12:06:38

After lunch, welcome speaker @daniel.maher for Q&A!

👍 1
1
Andre Lee-Moye, DZone12:06:42

"When we talk about automation, we're talking about unlocking human potential" < Really like that description

👍 3
🙏 1
Dan // Datadog12:06:03

I don’t know who’s doing the live-tweeting on https://twitter.com/ITRevDOES but they’re doing a great job. 👍

ClaudiaCardona12:06:21

How frequent are you changing squads? @daniel.maher

👍 1
Dan // Datadog12:06:45

@claudia_cardonatesill Frequency is determined by need, time-boxes, and definitions of success. A squad could last a week, a sprint, a quarter—it depends on how and why the squad was formed.

Dan // Datadog12:06:59

As an aside, an important part of guilds is also big hats. I forgot to mention that. Collars optional tho

thankyou 2
JohannT12:06:09

Enjoying demistifying SRE!

🙏 2
Siva Sethu12:06:21

@daniel.maher How are SRE and Full stack developers related? Can we assume that a SRE team is staffed with equivalents of Full stack developers to bring the versatility

👍 1
Siva Sethu12:06:21

@daniel.maher How are SRE and Full stack developers related? Can we assume that a SRE team is staffed with equivalents of Full stack developers to bring the versatility

👍 1
Dan // Datadog12:06:33

The term “full stack developer” is tricky, right? Like, how often is a full stack developer writing byte-code for microprocessors, for example? What is the definition of “full stack” you’re asking about in this case?

Siva Sethu12:06:22

I am referring to someone who has both Dev & Ops skills especially experience in App dev & infrastructure knowledge

Dan // Datadog12:06:13

So you mean a generalist—as SREs themselves tend strongly towards being generalists, the relationship is self-fulfilling.

Siva Sethu13:06:43

Yes. I agree. We can look at them as generalists in the sense that they are not experts in a single aspect of the value stream. Thanks Dan. Good session.

Jose Mingorance12:06:59

is SRE a separate organization independent of the application teams?

Jose Mingorance12:06:59

is SRE a separate organization independent of the application teams?

Dan // Datadog12:06:55

The answer is “it depends”. I mention this in the presentation actually, hehe.

Vlad Ukis12:06:16

this is handled differently by different orgs

Sandeep Joshi12:06:45

If you keep SRE separate, you are creating another handoff.

Vlad Ukis13:06:40

However, some shared on-call modes seem to be working in some orgs.

Dan // Datadog13:06:52

Again, don’t fall for the dogma here. There is no one true path—not even within a single organisation. And, even worse, what worked two years ago might not be appropriate now, so you have to be willing to evolve your organisational structure along with your business growth.

👋 1
Sandeep Joshi13:06:31

Yes - specialised skills, economies of scale and other factors. In my view, SRE should be embedded as an integrated way of working as part of the software cycle. It’s context that matters, many a times people think once a central function - always a central function. Adopting a continuous improvement mindset is critical

JohannT12:06:28

What's the difference between a programming mentor/coach vs a SRE on a team?

JohannT12:06:28

What's the difference between a programming mentor/coach vs a SRE on a team?

Dan // Datadog12:06:24

The difference would be focus and time allotment.

Dan // Datadog12:06:29

An SRE’s primary responsibility—certainly as an IC—is ensuring site reliability. 🙂 At a senior / staff level, it’s mentoring others, and the lines blur a bit.

JohannT13:06:13

Until now I assumed SRE was more architectural. Now i understand its the full gamut. Thanks.

👍 1
Jose Mingorance12:06:59

how does this fit into the YB/YO?

Kenny Johnston - Gitlab12:06:05

@daniel.maher What are some best practices for drawing circles around product-scope for product teams that allow for better alignment to SREs (especially for those who aren’t all micro-servicey)?

Kenny Johnston - Gitlab12:06:05

@daniel.maher What are some best practices for drawing circles around product-scope for product teams that allow for better alignment to SREs (especially for those who aren’t all micro-servicey)?

Dan // Datadog13:06:37

It might sound counter-intuitive, but start with the edge—literally whatever surface it is that your customer is actually touching—and work backwards from there.

Kenny Johnston - Gitlab13:06:01

Gotcha. Sometimes drawing lines for product team scope make sense at the edges but don’t really make sense for the SREs and while they are “assigned” they end up generally working on reactive work as a large pool that loses connection to product teams.

Dan // Datadog13:06:01

Ultimately the only thing that matters is if your customer can do the thing they expect. Take a look at your SLAs, and then what SLIs form the the SLOs in those SLAs.

Dan // Datadog13:06:21

Who’s responsible for those SLIs? BAM that’s your circle right there.

thankyou 1
Maik Himstedt12:06:52

Who at Datadog is woken up at 2 am if the system is down? A developer, the SRE? Or an - unmentioned - support guy/gal?

Maik Himstedt12:06:52

Who at Datadog is woken up at 2 am if the system is down? A developer, the SRE? Or an - unmentioned - support guy/gal?

Dan // Datadog12:06:48

Ideally nobody is woken up because there’s a follow-the-sun schedule. 😉

Dan // Datadog12:06:19

But in this case, interrupt-level alerts go to a product team and the SRE responsible for the related product or portfolio gets involved.

Maik Himstedt12:06:41

And the product team has an on-call roster?

Sandeep Joshi12:06:25

If the alert meets certain threshold, I would assume it goes to the person on-call

Dan // Datadog13:06:14

That’s up to each product team. In reality, some do and some don’t—this is also related to questions of complexity and criticality. Straightforward issues that can be solved without waking somebody up, don’t!

Dan // Datadog13:06:48

If it’s both complicated and critical enough that an already-awake person can’t deal with it, then yes, somebody gets woken up. This really doesn’t happen very often.

Sandeep Joshi13:06:41

Yes - thats’ why defining threshold (critical/complex) is useful. You can define it according to the context.

✔️ 1
Maik Himstedt12:06:09

👍 for the Pitfall screenshot

😆 2
Luke12:06:53

👏 @daniel.maher Thank you for the talk

Luke12:06:53

👏 @daniel.maher Thank you for the talk

Dan // Datadog12:06:31

You’re welcome!

Hugo Serrano12:06:36

Thank you @daniel.maher 🙂 I need to change my job title now 😅

👏 1
Hugo Serrano12:06:41

nice talk

🙏 1
Andre Lee-Moye, DZone12:06:59

Wonderful talk @daniel.maher

🙏 1
Sarah McManus - Datadog12:06:56

👋 all, we're here #xpo-datadog to answer any questions!

Dan // Datadog13:06:41

Definitely come through #xpo-datadog to learn more about Datadog. 😄

Stephen Magill [Sonatype]13:06:21

Thanks everyone who attended our Vendor Dome Chat! You guys posted a great list of questions that I’m still working through (I’ll respond to everyone eventually!). If you want to continue the conversation or go deeper on any of these topics you can go to the http://does.muse.dev/ to schedule a chat. I’m around and happy to meet up for 1-on-1s during the conference. Slack DMs work too. Enjoy the talks!

3
👍 2
Stephen Magill [Sonatype]13:06:32

And I posted this in a thread up above, but in case it’s of broader interest: during the Q&A WebGoat came up as a great training aid and I agree. Here are the results that can be found with code scanning: https://console.muse.dev/result/smagill/WebGoat/01EBGMAG8NZ7PXC292RTSTQQQN A good test of developer security training level might be: “do these errors make sense?” e.g. do they know what a CSRF (cross-site request forgery) is? I think there’s way more we can do to help tools do this training too.

1
💯 1