This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-05-20
Channels
- # ask-the-speaker-track-1 (195)
- # ask-the-speaker-track-2 (137)
- # ask-the-speaker-track-3 (225)
- # ask-the-speaker-track-4 (36)
- # bof-arch-engineering-ops (1)
- # bof-covid-19-lessons (1)
- # bof-leadership-culture-learning (1)
- # bof-overcoming-old-wow (1)
- # bof-project-to-product (6)
- # burnout (1)
- # demos (9)
- # discussion-main (1241)
- # discussion-more (4)
- # games (39)
- # gather (29)
- # happy-hour (25)
- # help (86)
- # hiring (15)
- # lean-coffee (12)
- # networking (2)
- # psychological-safety (2)
- # summit-info (146)
- # summit-stories (3)
- # xpo-cloudbees (1)
- # xpo-copado (1)
- # xpo-epsagon (2)
- # xpo-gitlab-the-one-devops-platform (28)
- # xpo-hcl-software-devops (1)
- # xpo-ibm (11)
- # xpo-itrevolution (5)
- # xpo-launchdarkly (7)
- # xpo-mirantis-devops (3)
- # xpo-pagerduty (4)
- # xpo-planview-tasktop (9)
- # xpo-redgatesoftware-compliant-database-devops (1)
- # xpo-snyk (4)
- # xpo-split (1)
- # xpo-synopsys-sig (1)
- # xpo-tricentis-continuous-testing (2)
@suzette.johnson5 and @robinyeman.. Yep, I love the way we can accurately predict the future via an Integrated Master Schedule 😬
yet the requirements change request number in the thousands
Involve technical people in org design! Love it, and miss it every time a reorg happens :-(
I definitely see lack of a common language a large challenge on a daily basis. Attempts at creating central definition docs haven't helped as much as hoped.
and compounded when you're a very technical company with lots of TLA's
@suzette.johnson5 do you see your gov't customers asking for Value Stream Management, or is it a concept that they are still trying to understand?
It is definitely an area that is growing rapidly in the agile/devops world
We aren't government. I think what we can say is that seeing value stream mapped visually always shocks our teams and leaders. Even if they already knew the overall results.
needs buy in from the business to allow us access to the people in the know
can be tricky in big orgs - certainly something I'm struggling with
it's funny, but these silos actually like talking to eachother
I've found as they're specialists they love to expand on their area. Just getting enough time with them, as well as your own time - knowing how far you need to map the stream
complicated branching and variant management are one of the pain points
I saw all of these barriers in my last role as a DevOps Coach. I'd be happy to learn more about your paper.
there are a lot of things that traditional software can bring to this space
Thank you, Suzette & Robin! And now, we're excited to have @jwillis join us!!
I know this isn't the direction this talk is going, but it makes me wonder if part of the underlying solution is changing how audits work and replacing that with something faster to integrate and validate with not requiring as many manual investigations.
We auditors definitely want to get to more automated assurance. Continuous monitoring has been a buzzword for a long time in the audit community.
There's quite a few articles on continuous monitoring and automated assurance or robotic process automation for audits on http://theiia.org and http://isaca.org but many require membership to read.
I think it was you who threw out the idea of having some endpoints that can spit out some of the details needed for audits to make it easier for auditors.
Yes.. that's why I"m a big fan of Automated Governance ... objective (digitally signed) evidence. That starts the changing of the game...
I was looking for the link for the papers. Here: https://itrevolution.com/forum-paper-downloads/
Somehow I think OPA has a lot of potential for some of this, but I've always struggled to find good use cases for it
The paper I'm talking about is in the 2019 catalog .. Devops Automated Governance Reference Architecture paper.
Actually, OPA can and is mostly used as in the implementation of the defined reference architecure.
Some orgs use a Risk as Code/Policy as Code as the interface and the align it with Rego/OPA as the implementation.
However, this architecture can and is used for more things than just K8. For example, Dataops, API Development flow, even model training.
Hmmm, I've had a few ideas around injecting it similar to the K8s admission webhooks into some of the AWS calls (reactively though) or in our pull requests for Terraform applies :thinking_face:
The thing is you want to decouple the attestations and gates from any specific technology. The model that works best is using the architecture as the reference and then use the specific technologies as the implementaion.
I think about this as common ingress and egress. Also, Gitops is a key part of these arechitectures.
@jwillis Why do you specifically single out GitOps there? Why isn't automated infra provisioning from version control enough? I don't see anything that GitOps offers on top of that
Unfortunately, I've been kind of lazy regarding blogging on this stuff. I've been taking a lot of Red Hat open source projects and putting them all together. Bill Bensing at Red Hat wrote DESORD for DOD and he has built a tool called Ploigos (also know as Software Factory). Also, Sigstore is another project that we are using for attestation (ingress).
@jwillis how do you deal with the usual deployment roles from your CI/CD system having basically sprawling admin permissions.. At some point the thing that's deploying needs to have fairly big amounts of trust 👀
@olivier.jacques feel free to reach out to me at <mailto:jwillis@redhat.com|jwillis@redhat.com> I'd love to chat..
This is a longer answer.. However, this is why I really love Ploigos/Software factory. It is a framework for the opinionated structure of pipelines but it is unopinionated on implementaion.
@steve.smith agree DAG can be done w/o Gotops but it really helps for gating hooks that can't be synchronous.
@steve.smith Gitops for Automated Governance... Not sure what question you are asking..
Sorry I'll rephrase it Do you think automated governance can happen without GitOps? Isn't it enough to automate infra provisioning from version control?
Hmmm perhaps this is just my deep scepticism of GitOps shining through 🙂
@steve.smith Are you sceptical because you don't lilke the hype or do you think there's something wrong with it?
@andreas.baernthaler That's a good question. I don't like the hype, but that's not all WeaveWorks. It's more that I don't like the idea that GitOps offers something new, when it doesn't. Automated infra provisioning with drift correction has been around since 2008, I personally saw it at LMAX in 2010 and it was mentioned in Dave and Jez's original CD book. I keep meaning to write it all down, for now I've only got https://twitter.com/SteveSmith_Tech/status/988813078128099329
I'm a big fan of automated governance, I suspect it can be done with/without GitOps. I lack data, for now
I totally agree that the concepts are not new and were nicely described in "Continuous Delivery". However, I think that tools under the GItOps umbrella just make implementing this easier. And pulling state into the target environment instead of "pushing" it from a CD tool was something that I learned from the GitOps community even though it might not have been new.
If anyone is interested in a good summary including references, I can recommend this page: https://www.gitops.tech/
Agreed @andreas.baernthaler, I think I'd just say that CD principles and practices are universal, they apply to cloud native just like anything else. There are certainly tools that speed up cloud native support, that's for sure I'm not convinced the ☔ is necessary, YMMV
Thanks @steve.smith, I appreciate your assessment.
That's OK @andreas.baernthaler, it's my own fault for not writing a "here's what I don't agree with" back in 2018, now it's hard for all of us (me included) to separate hype from reality I really do think this is separate from automated governance, which is a Good Thing regardless of the tech used 👍
Anyone write a paper about @jwillis presentations and cognitive load yet? So broad...
Gitops models work well for gating.. Not all gates can be synchronous. W/Gitops you can validate a subset of attestation at each flow.
I didn't remember them mentioning you specifically 😉
Thanks so much for that information-rich presentation, @jwillis!!
👏 Another firehose of information to dig through
Thanks @jwillis great informations
Feel free to reach out here, or on twitter @botchagalupe of email jwillis@redhat or <mailto:botchagalupe@gmail.com|botchagalupe@gmail.com>. I am passionate about this topic and it takes a village.
This is very helpful in filling a few colleagues' security learning backlogs for a while.
That question of why we aren't improving with triple the budget is interesting. I know within the past year, my team has installed 6 new types of code scanners or container scanners, but haven't had the time for the teams to get their head around the pros and cons. And in a high regulated industry, I still don't feel like that scanning gets me closer to what our audits will need.
@dacahill7 Someone yesterday mentioned this. A good approach is the bundle all your scanners into one system and give devs a sort of pre-check system. They just load the jar or container image into the system before they deploy. I call it the kitchen sink scanner model.
A decoupled model like Ploigos allows you to create a uniform interface structure. It also is a pipeline as code model so no direct human intervention. This interface model forces a lot of uniform structure for authority sprawl. If you also add Aufomtated Governance on this you have a fully functional *asCode implementation. This helps create admin conformity... .. I know that's a lot .. like I said feel free to reach out if you want to chat more about this..
Big takeaway for me (non-technical) is: 3LD model - is it conducive to good collaboration? Are we missing things? Very possibly I can think of issues with our governance model which show up in Line 1 implementation. I'll recommend to our security people on that basis and they can work through the technical parts Thanks you @jwillis!
@philipday My argument is that if we don't change the basic tenets of how we do security all our DevSecOps fu isn't really going to help us in the long run. IMHO, 3LOD is a not collaborative (by design) structure that gives us (conways law) what we deserve from a security posture.
In our context it represented a big increase in collaboration and empowerment on security matters compared to what went before, so it's really interesting to hear this critique and for us to question do we need to continuously improve at that level - either by optimisation of 3LOD or by another big transformation
the 2LOD by design firewalls 1LOD and 3LOD. Therefore everything is a telephone game of communication.
Tbf we do have (attempted) collaboration between lines 1 and 3 on certain specific projects, with mixed results My concern is more with the whole cycle of identifying risk -> ... -> doing work Is that process delivering? Are we self-reflecting on it?
BTW, I was asked where I blog. I'm actually focusing a lot of my energy on Dr. Deming these days. The ultimate goal is to tie his work to cyber... https://www.profound-deming.com/
Hello! I’ll be presenting with Dr. @stephen on some super interesting work that we’ve done on researching software supply chains!
This was such a fun project to work on with you, @stephen!!! Year 2 of working together on this!!
It was super fun being at GitHub Universe watching Nat Friedman, CEO, GitHub deliver that line of OSS inviting all those devs into your living room. Our reaction was 😱 ⁉️
How long does it take you to put this research together?
Yes! And we’re always looking for ideas on what to investigate next. What practices do you think have made an impact at your organization in terms of achieving confidence in security AND keeping dev teams productive?
We've seen really helpful trends with "Inner sourcing" - Creating internal specific modules/libraries working together with our security experts to provide an easier starting point. Still in its infancy but I've seen some good things come from that!
that’s awesome, @philipp.boeschen650! Do the inner source teams operate differently than other internal dev teams? Different expectations / different processes?
Oh there is no teams doing that currently, it's mainly me and a few other likeminded people in a grass roots initiative sponsored and pushed by my boss 🙂
But the expectation and processes are definitely different, the templates being voluntary at the moment, lots to still be discovered there, learning a lot currently 🙂
This is year 2 — we’ll describe Year 1 and new Year 2 findings. This was made possible by the amazing team at Sonatype, who offered us the opportunity to study all the artifacts on Maven Central!
(@stephen added Haskell onto that graph — wasn’t on the original! 🙂
PS: @stephen: I saw this awesome talk about how so many people are using IO monad to stuff all their ugly, imperative code into — and thus leaving side effects in the middle of a mess, instead of pushing all side effects to the edge. Was super interest!
very interesting. we use a variety of monads to segregate IO / logging / interaction with external APIs / etc.
https://www.youtube.com/watch?v=8KgL3FX8vYU by Eric Normand, who is quite prolific in Clojure space — he spent many years as a Haskell person. I’ve found him to be incredibly insightful and super smart.
Betteridge's Law of Headlines: NO
Totally — his comment was that in his previous org, they still mixed I/O in a way that was opposite of “functional core, imperative shell.” This was my biggest aha that made me love functional programming — remember that conversation we had about “what does composable really mean?” That was the context!
very cool. whatever language you’re using, being aware of side effects and isolating them is hugely valuable.
For sure!! Learning how to push side effects to the edges was a life-changer for me!
I've seen some link personally between projects with a very high release velocity and a fatigue on constantly patching up dependencies, so in some areas they just get left behind or pinned to auto pull in releases. Have you seen similar effect? Is there a downside to very high velocity? :thinking_face:
it can definitely be a challenge to stay up to date with everything. tools can help a lot here — e.g. automatically suggesting pull requests to update dependencies.
Pinning to latest is better than pinning to a fixed old version, but with supply chain attacks increasing (things like malicious actors adding code to open source projects) it’s a practice people are starting to reconsider.
Tooling really helps a ton! I've found up until a certain point it's doable for a team to stay on pinned versions and upgrade asap for new ones.
Or is the vector of attack here to really try to leverage something like semVer to allow for patch releases to flow through
Great point, @philipp.boeschen650 — one of our suspicions is that semver is actually a very poor predictor of breakage during updates. My genuine hope is that we can research this over the next year!
I dislike that theory a lot, but I don't disagree :white_frowning_face:
I sadly don’t have any answers, but the issue with semver is that it is largely dependent on the maintainer’s rigor in terms of following the conventions. It’d be a lot nicer if there was wider spread adoption of provider contract testing in some way so that you could actually test if there was breakage.
Interesting! I've started to feel something similar, we've recently started wrapping some tooling around Renovate (https://github.com/renovatebot/renovate) and while it's amazing help in showing outdated deps, there is a non trivial amount of pipelines failing after bumping the versions. Dependency management really is something intense
Yeah, being able to predict breaking changes is becoming really important. Definitely something we want to research soon.
There’s been some work on this at Sonatype actually: https://www.sonatype.com/products/advanced-development-pack There’s a feature in there to point out whether an update is likely to go smoothly or have breaking changes.
Even the trivial check of "does it still compile and run it's unitests" really helps out a ton in quickly judging dependency updates
Ah, @philipp.boeschen650, here’s a paper on what you just mentioned!
Here are the slides! https://www.dropbox.com/s/nk6625d84oajgug/Gene%20%26%20Stephen%20DOES%20Draft%20gk-sm-3.pptx?dl=0
Scheduling dependency upgrades to be part of the daily work seems like a good case of "If it hurts do it more often"
is it even better if it's a normal daily work thing, we're checking often and just taking care of it?
with enough automated test, I feel it should be safe to upgrade
Now to get the security team to help build that 🙏
Yup, the results of this make me think this is is is a rare case of “Doing the right thing for productivity” also prepares you to be ready to do the right thing by security - being able to produce fast means being able to respond fast to security events too
That is on of my favorite words: “centroids’. It’s what helped me understand clustering in 2005!
@stephen gets all the credit for that one — we moved totally away from SPSS, and I think he did all that in SciKitLearn!
Only had a glancing view and thought the arrow downwards in the slides meant that involving in OSS was bad 😆
Talking to each other during the presentation is very engaging
Hi, @cornelia!!! We’ll be vacating this track in just a second!! cc @stephen!!!
is there a tool that compares OSS libraries in this way? It seems like it'd be a great app or browser extension
Such a great book by @cornelia! https://www.amazon.com/Cloud-Native-Designing-change-tolerant-software/dp/1617294292/ref=sr_1_1?dchild=1&keywords=cloud+native+cornelia+davis&qid=1621520688&sr=8-1
Added to the reading list! https://www.goodreads.com/list/show/162704.DevOps_Enterprise_Summit_2021_Reading_List
@cornelia #gitops how is bridging the gap between the development & delivery cycle / stream.
I was wondering about this the other day and may be I going beyond my understanding. But it fels like it took us so long to understand what Puppet and Chef had already started doing ages back. They really started pushing Git as the way to manage infrastructure state and could also do drift detection and sync the state back.
Yup. Gitops doesn’t replace CI at all. It overlaps with many CD practices (we think GitOps is a better way to do CD) AND goes all the way to what I like to call CO - continuous operations.
the kubernetes ecosystem perhaps made it really fast but it feels like Gitops was always there in some way
“Using git as the interface for ops” - a perspective I feel often operators often fail to understand
Yes, I’ve started using another expression - that what we are aiming for is “self service operations” NOT “self service infrastructure”
My thanks to @cornelia for helping me finally understand what GitOps is and isn’t! 🙂
We’ve historically been self-serving application teams (developers) with “stuff” to help them do DevOps but just giving them infra - i.e. we give them a Kubernetes environment (and say “good luck. And, oh, BTW, please keep this secure”). What we are aiming for with GitOps is allowing these devops teams to do operations without burdening them with the infra. self-service operations.
One technique I used was to declare that production was not accessible outside of a CD pipeline. In other words, your only path to production is GIT + some sort of an orchestrator => GitOps.
Yup. And we are quite insistent that the “some sort” has to be convergent.
I find GIT is (relatively) easy to get adopted by developers. It is a longer journey to get it adopted across the various functions. But at least, we know where we are going. GIT.
Oh yeah - totally agree. Git is freakin hard. Developers are Git power users. Everyone else, not so much.
GitOps does not preclude a different UI over Git than the ones that developers love.
Ahhh. Centralized CD systems and the best attack surface ever. Thanks for bringing that up!
Can be difficult in practice though? How would you do it for something like Lambdas/Azure Functions or similar applications? :thinking_face: Create a separate service that updates other services..?
this works great in combination with platform teams, you get a better separation of concerns
For me audit requirements were one of the motivations to dig into GitOps. It would be so interesting to discuss GitOps with the auditors currently presenting in track 1.
Also, John Willis is exploring “Modern Governance” in his presentation and talks about automated proof of compliance, etc.
@cornelia Please ping me if you can convince them. I might be able to use that as an agrument to convince clients to do GitOps 😄 .
Is the app configuration in this diagram referring to configurations in containers (since we’ve got Kubernetes on the right)? Are we pulling versions of code directly into our environments or would that typically be running via versioned/tagged container images?
@cornelia very inspiring talk that I am going to share with my entire team. I feel it will help bring a lot more focus on adopting GitOps because of your explanation
Super @cornelia. I've really struggled with really 'getting' GitOps, rereading WW site and papers, but this presentation was the clearest yet 🙏
This is the first explanation of GitOps that finally made clear to me what it is and how it might be useful beyond being a trend. 😁
Here is one reference regarding GitOps I can highly recommend: https://www.gitops.tech/
#gitops is one next best thing amongst the other few things that organizations are finally yeah finally moving to with no other option left similar to for e.g. Kubernetes. Good we had this discussion rolling here. thanks to @cornelia 🙂
Here is the collection of all the reference links that were shared / discussed during the #DOES21 event. https://www.linkedin.com/pulse/devops-enterprise-summit-2021-reference-links-minus-books-pareek Enjoy Reading 🙂