Fork me on GitHub
#ask-the-speaker-track-4
<
2020-06-25
>
Jess Meyer - IT Revolution (she/her)10:06:11

Welcome speaker @michael.gillett for Q&A after break!

๐Ÿ‘‹ 2
Michael Gillett10:06:15

Hi everyone, going to apologise upfront, I am a Virgin Media customer and their internet has serious issues this morning so I might disappear at times as I am tethering to my phone

๐Ÿ†— 4
Jimmy Simons10:06:13

I feel your pain Michael, I am self-isolating in Worthing on the south coast. Not sure their internet has been upgraded in many a year

๐Ÿ™ˆ 2
Ann Marie Fred - Red Hat10:06:02

You know whatโ€™s interesting? I see more and more people turning to craft as weโ€™re isolated and doing so much of our work and socialization online. Thereโ€™s a certain comfort in doing things with your hands.

Ann Marie Fred - Red Hat10:06:02

You know whatโ€™s interesting? I see more and more people turning to craft as weโ€™re isolated and doing so much of our work and socialization online. Thereโ€™s a certain comfort in doing things with your hands.

Hannah Beech10:06:30

I think there might be more to it as well; for the first time in a long time (in my experience, at least), there's more time to do crafty things - my commute, for example, has gone from 2 hrs/day to nothing.

๐Ÿ‘ 1
Michael Gillett10:06:48

definitely, it's an opportunity to do something tangible that most of the time we never get to do any more. We can poor hours into to making something just for us

๐Ÿ‘ 1
Ann Marie Fred - Red Hat10:06:54

I personally have less time than before because my childcare is gone. ๐Ÿ™‚ When the kids were in school it was a time management nightmare. But I still appreciate the opportunity to do more hands-on things with them like cooking and gardening. The extra time, in our case, comes from all of their extracurricular activities, as well as all of our shows, parties, and events, being cancelled.

Ann Marie Fred - Red Hat10:06:50

Hypothesis Driven Engineering - cool concept.

๐Ÿ‘ 1
Andy Sturrock10:06:57

Two things were changed - the single button/make it clearer but also the language: "join now" vs "register". Might have been interesting to test each of those changes one at a time to see which had more impact.

Hannah Beech10:06:09

@michael.gillett Apologies if this is a premature question - when conducting A/B tests, were there knock-on affects in terms of quality, time to deliver? In a previous company, there was so much time sunk into regression testing and ensuring that the B variant hadn't broken the A/control variant that there was a definite decrease in lead time.

Jeffrey Fredrick, Author-Agile Conversations10:06:53

@michael.gillett Iโ€™m always jealous of people with enough traffic to run that kind of A/B test. How would you change your approach with a lower volume site? (if at all)

๐Ÿ‘ 3
Jeffrey Fredrick, Author-Agile Conversations10:06:53

@michael.gillett Iโ€™m always jealous of people with enough traffic to run that kind of A/B test. How would you change your approach with a lower volume site? (if at all)

๐Ÿ‘ 3
Ann Marie Fred - Red Hat10:06:31

Excellent question! Most of our web pages donโ€™t get enough traffic for A/B testing to reach statistical significance.

Andy Sturrock10:06:17

Doesn't it depend on the confidence interval that you are aiming for? (trying to remember A-level stats here!)

Michael Gillett10:06:09

We do run experiments on lower volume sites too, it might change the time that we need to run the experiment in order to reach statistical significance

Ann Marie Fred - Red Hat17:06:52

It does depend on the confidence interval, yes. It also depends on how dramatically different your results are between target groups. I know we have a few high traffic pages where we can usually get a result within a few weeks, but on others the test might have to run for six months or more! At that rate youโ€™re better off using other methods.

Ann Marie Fred - Red Hat10:06:01

What tool is this? Itโ€™s very pretty.

Ann Marie Fred - Red Hat10:06:01

What tool is this? Itโ€™s very pretty.

Hannah Beech10:06:08

Possibly App Insights in Azure?

Michael Gillett10:06:48

Yep, as Hannah says, this is Application Insights

๐Ÿ‘ 1
Davy Kenis10:06:26

@michael.gillett love the A/B tests! Iโ€™m wondering... do you also use A/B Tests for more technical applications without user interfaces (such as APIโ€™s) ?

Davy Kenis10:06:26

@michael.gillett love the A/B tests! Iโ€™m wondering... do you also use A/B Tests for more technical applications without user interfaces (such as APIโ€™s) ?

Michael Gillett10:06:42

Yep, sorry for being slow to reply and with my bad Internet today the presentation might have got to the technical use of A/B tests/toggling so that clients can point to different backend systems

Davy Kenis10:06:57

Would love to share insights on this particular topic! Maybe at another moment once your internet connection is better?

Michael Gillett13:06:41

I have internet (for now) so happy to chat a bit more around this

Marc Boudreau (Enterprise Architect)10:06:59

Canada ... always a target. :)

Arnab Nandi10:06:36

@michael.gillett when do you retire the old code when the hypothesis is proven - for example, the other home page? Any specific cadence or does it stay? We find ourselves having a build up of code due to these feature flags...

โž• 2
Arnab Nandi10:06:36

@michael.gillett when do you retire the old code when the hypothesis is proven - for example, the other home page? Any specific cadence or does it stay? We find ourselves having a build up of code due to these feature flags...

โž• 2
Michael Gillett10:06:31

When creating tickets for features we will also create a "clean up" ticket to be played at a later date. Once the experiment is run and a decision on the outcome of the feature is made we would look to clean up the toggle ASAP (sometimes the priority isn't there to do it immediately)

๐Ÿ‘ 2
Arnab Nandi10:06:48

Brilliant idea!

Michael Gillett10:06:06

@hannah.beech through encapsulating a new feature within the toggle we found in many cases we were able to keep the original feature intact and only add new code. There have been times when changes have "leaked out" of the incapsulation but our existing automation tests often picked them up

๐Ÿ‘ 2
Michael Gillett10:06:06

@hannah.beech through encapsulating a new feature within the toggle we found in many cases we were able to keep the original feature intact and only add new code. There have been times when changes have "leaked out" of the incapsulation but our existing automation tests often picked them up

๐Ÿ‘ 2
Hannah Beech10:06:26

Thanks. The previous company in my example; the codebase was an absolute monster of a monolith with very little to no automation.

Jeffrey Fredrick, Author-Agile Conversations10:06:55

โค๏ธ trunk based development

๐Ÿ’ฏ 3
๐Ÿ˜ 1
Jeffrey Fredrick, Author-Agile Conversations10:06:05

branches = deferred integration.

โค๏ธ 2
Jeffrey Fredrick, Author-Agile Conversations10:06:05

branches = deferred integration.

โค๏ธ 2
Ann Marie Fred - Red Hat10:06:13

And an accumulation of work in progress

โž• 1
Ann Marie Fred - Red Hat10:06:07

Do you use visual regression testing?

Ann Marie Fred - Red Hat10:06:07

Do you use visual regression testing?

Michael Gillett10:06:09

We've considered it and have played around with it a bit but not used it extensively or brought it into processes

ClaudiaCardona11:06:23

@ann.marie.99 do you have info or links on visual regression testing?

Ann Marie Fred - Red Hat17:06:39

Weโ€™ve had luck with Hawkeye, I know. And the other one we use we call Landmarkโ€ฆ let me see if I can find the framework.

Ann Marie Fred - Red Hat17:06:34

I think our Landmark tool is based on PhantomCSS and the Hawkeye tool uses BackstopJS. Visual regression testing is the best way weโ€™ve found to detect โ€œbroken CSSโ€ errors that may only show up in production.

thankyou 1
Davy Kenis10:06:10

testing in production ๐Ÿ˜

Michael Gillett10:06:17

@does very true, we have since refined our experimentation processes and are testing things are small as text differences

๐Ÿ‘ 1
Tom Ayerst10:06:05

@michael.gillett Do you use observability tools like honeycomb to learn from production?

Tom Ayerst10:06:05

@michael.gillett Do you use observability tools like honeycomb to learn from production?

Michael Gillett10:06:33

No, I shall have to investigate that and see how we can make use of it

Andy Sturrock10:06:38

Oh wow I hadn't thought of the same person flip-flopping between versions!

๐Ÿ‘ 1
Andy Sturrock10:06:38

Oh wow I hadn't thought of the same person flip-flopping between versions!

๐Ÿ‘ 1
Jimmy Simons10:06:38

@andrew.sturrock The key is to ensure your UserID doesn't change so when you evaluate your flags, your user stays in the same variation. You don't want to tie those userIDs to Session IDs otherwise they will flip flop

John Boyes10:06:19

I love this, @michael.gillett. Thereโ€™s still many people who think that from a quality point of view you need to test in many environments before production, but it ainโ€™t necessarily so ๐Ÿ™‚

๐Ÿ‘ 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:53

Where do you draw the line for testing in production? I would not like that for my bank account balance.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:53

Where do you draw the line for testing in production? I would not like that for my bank account balance.

Jimmy Simons10:06:51

@ferrix I think that is a business decision. Hopefully you work in partnership with the compliance department to understand what the rules of the road are. Some people will never be able to get rid of all their test environments

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:26

I find throw-away test environments desirable in most cases. It's not all things you should be verifying in production.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:06

Also, when you don't need the user to verify functionality, why bother them with it? ๐Ÿ˜„

Jimmy Simons10:06:21

Well you can still use a Feature Flag for other purposes, such as a kill switch or just reducing merge conflicts. But I definitely agree, not all changes need to be A/B tested.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:35

Sure. Always use a feature flag for everything. Just don't make it enableable (nice word) before you have had it work through a test suite.

Arnab Nandi10:06:43

Love the scheduling complacency piece - we find ourselves doing digital when the physical world, like our sites and payment systems aren't ready...

๐Ÿ‘ 1
Andy Sturrock10:06:44

Need to be careful about the terminology. This isn't about testing whether something works or not, more about which version works better/is more desirable.

๐Ÿ‘ 2
Andy Sturrock10:06:44

Need to be careful about the terminology. This isn't about testing whether something works or not, more about which version works better/is more desirable.

๐Ÿ‘ 2
John Boyes10:06:42

@michael.gillett if the only environments were local and production, where was the testing done to verify if something worked or not?

John Boyes10:06:05

Great talk, by the way ๐Ÿ™‚ ๐Ÿ‘

Michael Gillett13:06:08

Sorry, I missed your message @john710 - the testing can be done in Production with the QAs/automation able to turn the feature on for just themselves, once verified it can then be turned on for some group of real users to verify that it is all working

๐Ÿ‘ 1
John Boyes13:06:49

Makes sense, thanks Michael. The following posts in the channel (not by you) were suggesting that you were only talking about A/B type tests in production, not quality tests too, which is why I asked - thanks for clarifying ๐Ÿ™‚

Andy Sturrock14:06:26

In that case I misunderstood! @michael.gillett Do you really not have any (non developer laptop/local) environments to do functional testing in before production?

Michael Gillett14:06:29

Some team/apps do true trunk based development and only work on local and push to prod, most teams/apps do not work in that way and have at least 1 test environment

๐Ÿ‘ 1
Ann Marie Fred - Red Hat10:06:48

Can I share a funny A/B testing story? We once had someone add a check to a page like this: if (page = targetPage) instead of if (page == targetPage) . This check ran on every page that was enabled with our A/B testing framework. Can anyone guess what happened the instant it rolled out?

4
๐Ÿ’ฏ 2
๐Ÿ™‚ 2
Ann Marie Fred - Red Hat10:06:48

Can I share a funny A/B testing story? We once had someone add a check to a page like this: if (page = targetPage) instead of if (page == targetPage) . This check ran on every page that was enabled with our A/B testing framework. Can anyone guess what happened the instant it rolled out?

4
๐Ÿ’ฏ 2
๐Ÿ™‚ 2
Andy Sturrock10:06:31

sets page to targetPage, evaluates to true so all traffic goes to the targetPage?

โž• 1
๐Ÿ’ฏ 1
John Boyes10:06:38

Hope you had monitoring driven development to accompany the hypothesis driven development ๐Ÿ˜‰

Ann Marie Fred - Red Hat10:06:56

Yes! Suddenly 500,000 web pages found themselves in an infinite redirect loop.

1
๐Ÿคช 1
Ann Marie Fred - Red Hat10:06:04

Within 5 minutes 20 people were paged, but it took us an hour to find the person who could fix it. After that we put in a โ€œkill switchโ€.

Tom Ayerst10:06:06

Hypothesis Driven Engineering - making expereiments โ€œsafe to failโ€ in Production

๐Ÿ‘ 2
Arnab Nandi10:06:12

Very informative, @michael.gillett Thank you!

๐Ÿ‘ 1
Alex van Assem10:06:16

@michael.gillett - How do you manage/remove your feature flags to reduce code complexity?

Alex van Assem10:06:16

@michael.gillett - How do you manage/remove your feature flags to reduce code complexity?

Michael Gillett10:06:13

Once an experiment is finished we would prioritise a piece of work to remove the toggle and the variation(s) we no longer needed within our codebase. If an feature is incapsulated well then its not too much work to remove it from code

Jimmy Simons10:06:30

Hi Alex, I won't answer for Michael, but I will say that you want to ensure your flagging system is tied back into your Git repository. You also want to have a way that you can see which flags are no longer being evaluated or has any targeting tied to it. LaunchDarkly, which Michael's team uses, has that capability

Andy Sturrock10:06:21

Great talk @michael.gillett ๐Ÿ‘

Michael Gillett10:06:52

glad you enjoyed this and thanks for watching! ๐Ÿ˜„ you had plenty of brilliant questions and given me something to think about ๐Ÿ™‚

๐Ÿ‘ 2
Hannah Beech10:06:13

Thank you @michael.gillett ๐Ÿ‘

Gordon Beeming10:06:25

@michael.gillett great session, thanks ๐Ÿ˜

๐Ÿ‘ 1
Marc Boudreau (Enterprise Architect)10:06:01

Favourite effect of Feature Flagging ... being able to respond to โ€œWhen will x be released?โ€ with โ€œwhen you turn the flag onโ€.

2
โž• 1
๐Ÿ‘ 2
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:33

Decoupling deploy and launch has been ๐Ÿคฏ for many product managers

๐Ÿ’ฏ 1
Jiล™รญ Klouda10:06:33

The problem we had with feature flags is that if they persist for some time, you get problems in testing. For 10 flags you have 1024 combinations of them being turned on/off. If they do interact in any way, like minified/unminified js and change to object model and new feature using the object, etc โ€ฆ you need to test somehow all combinationsโ€ฆ

Jiล™รญ Klouda10:06:33

The problem we had with feature flags is that if they persist for some time, you get problems in testing. For 10 flags you have 1024 combinations of them being turned on/off. If they do interact in any way, like minified/unminified js and change to object model and new feature using the object, etc โ€ฆ you need to test somehow all combinationsโ€ฆ

Markus Lauttia10:06:51

yeah, and should you develop new features so that they work only on top a feature that is behind a feature flag...

Jiล™รญ Klouda10:06:46

How does the test automation know if two flags depend on each other. How does it recognize that tests need to run on combinations of feature flags? Do we only run tests on all flags being turned off?

Jimmy Simons10:06:47

@jiri.klouda I think the key thing to remember is that there is no silver bullet to technical debt. Each time you introduce something into your code, you could be potentially introducing more technical debt. So you need to manage those introduction. With Feature Flagging, you should have a system that can integrate into your Git repository. You should also be able to see how often a flag is evaluated so you know when you should take flags out. You can doo all. of that with LaunchDarkly if you are interested

Jimmy Simons10:06:22

@jiri.klouda opps I was writing before you had the next question

โœ”๏ธ 1
Jimmy Simons10:06:36

using a single flag to enable a series of flags is a common strategy

Markus Lauttia11:06:54

we're trying to advocate the policy that feature flags should be relatively short-lived

Jiล™รญ Klouda11:06:57

I mean the pre-merge or pre-deployment testing.

Simon Herron12:06:02

Feature Flags should be short-lived. There are occassions when you might have a few operational long-lived flags that grow into your architecture, but thatโ€™s almost always by design [Subscription entitlements, operational gates for rate limiting, etc]. As far as Testing is concerned, there is always an assumption that you should test all combinations and thatโ€™s a common error. Iโ€™ve included a blog post which talks more about this, itโ€™s a little dusty but the principles stand: https://launchdarkly.com/blog/testing-with-feature-flags/ If I was to pick out two key points it would be to: A. Feature flag should be dedicated to do 1 thing and 1 thing only. It should be small enough that if you describe it and use โ€˜โ€ฆ and โ€ฆ โ€™ you probably need to rethink what youโ€™re doing. The idea is to release lots of small pieces of code that can be handled easily. This means that if it breaks you can turn it off quickly without affecting the rest of your release. The whole point of flagging is that you should be able to ship broken or unfinished code into production without it being a problem. B. Test your critical paths. Current production state and the Feature Flag โ€˜defaultsโ€™ scenario. Any Feature Management platform worth it salt should be able to help you smoothly enact those scenarios. Does that help at all, @markus.lauttia // @jiri.klouda? https://launchdarkly.com/blog/testing-with-feature-flags/

๐Ÿ‘ 1
Molly Coyne (Sponsorship Director / ITREV)10:06:01

Welcome @vbrennan to the channel to answer any questions about Slack's session!

slack 4
๐Ÿ‘‹ 3
๐Ÿ™Œ 3
Karl Marfitt10:06:39

'Capacity and Performance Planning' --> 'Scale auto-magically' :unicorn_face: ๐Ÿ˜„

2
Andy Sturrock10:06:27

Mark Schwartz talked last year about DevOps is a risk reduction mechanism. It's a good way to sell it to senior non-technical people.

๐Ÿ’ฏ 3
Andy Sturrock10:06:27

Mark Schwartz talked last year about DevOps is a risk reduction mechanism. It's a good way to sell it to senior non-technical people.

๐Ÿ’ฏ 3
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)10:06:13

Ha. Wrote some strategy slides and had the CEO say: "So, this is essentially risk management." Yes, padawan.

๐Ÿ‘ 1
๐Ÿ˜„ 3
Karl Marfitt10:06:28

Almost same (ChatOps) slide from 10+ deploys per day?

Karl Marfitt10:06:28

Almost same (ChatOps) slide from 10+ deploys per day?

Karl Marfitt10:06:58

I really need to ask @allspaw nicely if I can reuse his slides... still so relevant today over a decade on!

Ugur Yilmaz10:06:08

@vbrennan how do you keep the information flow to all teams are at a consistent level? how do you manage the different knowledge levels of each time?

Ugur Yilmaz10:06:08

@vbrennan how do you keep the information flow to all teams are at a consistent level? how do you manage the different knowledge levels of each time?

Alex11:06:12

@vbrennan

V Brennan11:06:44

We use a naming structure for channels to ensure information goes to the right place. For example we have #feat-xyz, #devel-xyz, escal-xyz. This ensures everyone knows where to go for what sort of information.

๐Ÿ‘ 1
V Brennan11:06:49

If information is needed across multiple places we use a bot for it. For example incident-bot posts relevant information to multiple channels and workspaces so it gets widely disseminated. It also helps keep curious people away from the incident channel itself, reducing noise there.

Andy Sturrock10:06:06

@vbrennan lots of our teams want private channels. I try to convince them that public is best because then knowledge becomes searchable. What's the balance of public vs private inside Slack?

๐Ÿ‘‹ 1
Andy Sturrock10:06:06

@vbrennan lots of our teams want private channels. I try to convince them that public is best because then knowledge becomes searchable. What's the balance of public vs private inside Slack?

๐Ÿ‘‹ 1
Alex11:06:20

@vbrennan

V Brennan11:06:28

We lean towards public. Having said that I have a private channel with my team. Right now I do a once a week check in on people as they wfh. It is a pretty personal and open conversation so we keep it private. Other than that we are all open ๐Ÿ˜„

Andy Sturrock11:06:08

That's the balance I think. People want to be open to talk within the team, but then the knowledge remains trapped within the team.

V Brennan11:06:13

We keep our banter and team chat in public too. It makes for some really fun reading. Our open team channel and our Melbourne team channel are famous in engineering. People literally stop by to hang out and say hi. It is a great way to connect with other teams on a personal level too

โž• 1
๐Ÿ‘ 1
Alex10:06:46

Hi <!here> @vbrennan slack is going to to be joining the channel in 15 minutes. Keep your questions coming and she can answer them then ๐Ÿ™‚

thankyou 1
Pete Nuwayser - IBM11:06:39

This is great! From issue to collaboration in a minute. Wow.

โœ”๏ธ 1
2
Karl Marfitt11:06:54

I vaguely remember playing with bots on IRC in the early 90's imagining a day when they did more than just kick or promote other users or send them files or add them to some RBAC controls on other servers, the integrations with Slack are so damn powerful but still struggling to get my workspace owners to trust the major players that add real value (mostly kanban boards and SCM tools)

Karl Marfitt11:06:54

I vaguely remember playing with bots on IRC in the early 90's imagining a day when they did more than just kick or promote other users or send them files or add them to some RBAC controls on other servers, the integrations with Slack are so damn powerful but still struggling to get my workspace owners to trust the major players that add real value (mostly kanban boards and SCM tools)

Scott Nelson Windels15:06:03

I'm not sure if our App security review (not a certification) helps, but I believe that it is a new project in the past few months. https://api.slack.com/security-review

Scott Nelson Windels15:06:22

Might help building trust inside your org where needed.

Karl Marfitt15:06:21

Thanks ๐Ÿ™‚ I'll mention this to the decision makers!

1
Scott Nelson Windels15:06:43

Another tab we added to apps that update it is helpful too - the Security and Compliance app per tab. This is a screenshot of PagerDuty's app info which they've filled out.

๐Ÿ‘ 1
Karl Marfitt15:06:34

To be honest half the battle is that many (if not most!) of our workspace admins have just been nominated by someone else and don't really understand how to perform the role quite yet, so they're adopting a more risk-averse approach. Thanks again for helping ๐Ÿ™‚

Jiล™รญ Klouda11:06:09

Is there a link to that Incident Commander integration?

Jiล™รญ Klouda11:06:09

Is there a link to that Incident Commander integration?

Scott Nelson Windels14:06:22

I will be in our virtual booth in 10min if you want to stop by - i am a lead in our Incident Response program at Slack.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:26

@vbrennan Some war story from Github, IIRC, was stating that some of the resolution work is given to bots like: "hey bot, restart service xyz" which makes the incident work more transparent. Do you have anything like that?

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:26

@vbrennan Some war story from Github, IIRC, was stating that some of the resolution work is given to bots like: "hey bot, restart service xyz" which makes the incident work more transparent. Do you have anything like that?

V Brennan11:06:03

Hi @ferrix We do have automation for restarting services with visibility through our alerts channels in Slack. We don't have a way to kick it off manually in Slack though

Karl Marfitt11:06:20

+1 for dog fooding! ๐Ÿถ

๐Ÿถ 2
๐Ÿˆ 1
๐Ÿ• 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:40

@karl.marfitt That's more like having the dog make the dog food.

Jose Mingorance11:06:59

We are not allowed to use Slack and only have Microsoft Teams, has anyone implemented this level of integration with MS Teams?

๐Ÿ˜ฟ 3
Jose Mingorance11:06:59

We are not allowed to use Slack and only have Microsoft Teams, has anyone implemented this level of integration with MS Teams?

๐Ÿ˜ฟ 3
Andy Sturrock11:06:17

<understatement> The Teams APIs are not as easy to use as Slack's. </understatement>

๐Ÿ™Š 1
Andy Sturrock11:06:51

We have decided that Teams is the tool for people whose job is emailing powerpoints to each other. Slack is the tool for DevOps teams.

๐Ÿ˜„ 1
Karl Marfitt11:06:00

Ouch, 'not allowed' feels like a culture problem there ๐Ÿ˜• but don't get me wrong - I feel you!

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:20

You can do a lot with pushing messages with links, but it is not that sophisticated.

Karl Marfitt11:06:25

Crazy idea but if we had a high level influencer here from MS perhaps there is an opportunity for the 'Teams team' to learn from here... i.e. that 'playing nicely' and having better integrations with other first class services helps the overall ecosystem. Getting better together โค๏ธ @samgu might know someone I guess...

๐Ÿ‘ 1
Aidan Moriarty11:06:48

We recently moved from Slack to Teams as it was chosen as a corporate platform and for many reasons it made sense to go there...we're now suffering some inevitable issues where as a corporate platform the simplest things like unauth webhooks are being picked out as security violations and we're having to fight to even use Teams as-is :face_palm:

Aidan Moriarty11:06:37

No custom bots allowed yet either and anything from the marketplace goes through a multi committee, many week/month approval process

๐Ÿค• 1
Andy Sturrock11:06:39

I have fed back to MS a few times and they just say "it's supposed to work like that". For example, having membership at the team level rather than the channel level just doesn't work as well, but they said it was deliberate.

Aidan Moriarty11:06:44

Its very tied to the existing Office365 architecture which influences the member handling it seems

Andy Sturrock11:06:54

Yes. I also find the ironic thing is that because the default choice is private teams, that you end up emailing documents around because people aren't members of the right team and therefore can't get to the sharepoint behind the team!

Karl Marfitt11:06:59

"....multi committee, many week/month approval process" aaargh those words are resonating with me so much! We really need minimum viable compliance, or just build our appetites for risk and make things work better, safer, happier in spite of those who seem determined to make things worse, slower and far more annoying!

๐Ÿ‘ 2
Andy Sturrock11:06:10

Slack's Onedrive/Sharepoint integration is great. You post the link and it pops up a thing which sets the permissions for you to read/edit for anyone in the conversation.

Andy Sturrock11:06:54

@karl.marfitt as per Mark Schwartz yesterday! Minimum viable bureaucracy ๐Ÿ™‚

โœ”๏ธ 1
Aidan Moriarty11:06:38

I do think Teams is getting better (from a devops perspective, and its taken a while) and it wasn't an easy choice to give up a private Slack instance that allowed our small group really innovate within our space over the years, but sometimes the corporate tool solves more problems than it introduces...I thought, before we realized we couldn't even use Teams as designed ๐Ÿ˜ž And I find a MS rep now fighting our corner with our own Corporate IT, strange times ๐Ÿ˜„

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:11

Policies can sometimes be very interesting. There is a particular Teams instance where "I don't know which channels I am supposed to be on and can't figure out who to ask because everything is hidden" is the biggest COVID bottleneck.

Karl Marfitt11:06:44

Oh yes, as a huge enterprise of hundreds of thousands of employees, we have way too many workspaces with overlapping purpose and so many missed opportunities for valuable collaboration. I am on at least a handful of competing agile channels. The answer we found was opening up and sharing the most popular channels across the whole org but this seems to be happening far too slow... I sometimes wonder if mixing realtime chat with a Slashdot or Reddit-like karma or other 'conversational value' +1 crowd-voting based system might be a better approach

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:00

I would consider that a lost cause. Creating a functional microcosmos is enough ๐Ÿ˜„

๐Ÿ™‚ 1
Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:59

Getting a hundred people to go through mandartory security training on a yearly basis seems to be quite a feat and getting one non-altered message through to a thousand people is near impossibility, so having "one Agile" in hundreds of thousands sounds astronomical enough to just give a pass ๐Ÿ˜„

Aidan Moriarty11:06:38

This is where we're at having joined that corporate Teams instance; hoping to continue looking after our corner but seeing could we leverage some experience from elsewhere. For the reasons above though, several months in, I can't say I've stumbled into anyone outside my immediate group to share stories with ๐Ÿ˜ž

Andy Sturrock11:06:27

In our Slack we have lots of channels shared between workspaces in order to increase collaboration, and it works. You can't do that in teams though.

โœ”๏ธ 1
Andy Sturrock11:06:21

We've also been more disciplined in not having millions of workspaces than in teams where there are so many teams that I have largely given up on most of them.

Jose Mingorance12:06:46

Thank you all for your insights. We have started doing some integrations with Jenkins for build status but it seems very limited. I once connected with the the head of Azure's Global SRE, you may know him a he was one of the authors of the Google SRE book (Niall Murphy), and he told me that MS was investing a lot into Teams to use it like Slack.

๐Ÿ‘ 1
Aidan Moriarty16:06:40

For what its worth, webhooks alone accomplish a lot of what you need in integrating Build tools to Slack/Teams in my experience anyway. I'm hoping that we can push Teams further now we're (stuck) there ๐Ÿ˜„ Our contract is pretty big so we're well engaged with MS reps the whole time

Karl Marfitt16:06:31

@swindels - thought you might be interested in this thread if you'd not seen it

1
Stijn Liesenborghs - Engineering Mgr - Nike11:06:55

You are totally doing what we imagined what our support efficiency initiative should be

thankyou 1
V Brennan11:06:56

Hi All, Thanks for watching the talk and for inviting me along to my first DOES. Please feel free to reach out with more questions.

thankyou 1
๐Ÿ‘‹ 1
Matthew Webster11:06:56

@vbrennan I am a daily user of Slack. We sometimes use is as an inter-team Q&A tool trying to perform the same role as StackOverflow but not very wll. Do you have experience of trying to use Slack in this way?

Matthew Webster11:06:56

@vbrennan I am a daily user of Slack. We sometimes use is as an inter-team Q&A tool trying to perform the same role as StackOverflow but not very wll. Do you have experience of trying to use Slack in this way?

V Brennan11:06:56

Yes. We set up channels for this exact purpose. We spin channels up and down all the time around specific topics. We ensure that a couple of folks drive the conversation or at least get it going to start with.

Matthew Webster11:06:12

@vbrennan one of the best features of StackOverflow is that a question can be marked "Answered" and easily searched for later. Can you do this?

V Brennan11:06:28

We use threads for specific answers. You could use a custom emoji to indicate when a question is answered or complete

V Brennan11:06:20

you can also send those threads to a channel based on the reactji using reacji channeler

Matthew Webster11:06:56

@vbrennan so that channel persists and can be searched later?

V Brennan11:06:37

If we read something anywhere in Slack that we think is relevant for our team we use a 'team-aware' reacji and reacji channeler send it to our team channel. It helps ensure relevant information is directed to us.

Scott Nelson Windels14:06:20

We also use this internally for things like an announce-incident-reviews channel and all meetings and Review documents get reacjiโ€™d into a central channel

Stijn Liesenborghs - Engineering Mgr - Nike11:06:21

I really loved it. I would like to have a talk with you about how we could set this up at Nike. @vbrennan

Stijn Liesenborghs - Engineering Mgr - Nike11:06:21

I really loved it. I would like to have a talk with you about how we could set this up at Nike. @vbrennan

Laura Friend11:06:33

๐Ÿ‘‹ @stijn.liesenborghs we can put you in touch with your customer success manager at Slack who can help with this? She works with your HQ team in the US

Laura Friend11:06:19

great, will chat to Lindsey (CSM) and get her to get back to you

Jenny Lynam11:06:29

@stijn.liesenborghs I have asked for Lindsay to be added in here. She will follow up later when she comes online

๐Ÿ™ 1
Jenny Lynam16:06:43

@stijn.liesenborghs ๐Ÿ‘‹ meet @landerson who looks after Nike at Slack! She can help you out!

Lindsay Anderson17:06:50

@stijn.liesenborghs it's great to meet you! I'd be happy to chat with you about ways to implement some of the Slack use cases you saw today with Nike. Do you want to send through a few days/times you're available for an introduction call the week of 7/7? I'm on the West Coast in the US so Pacific Time zone

Stijn Liesenborghs - Engineering Mgr - Nike19:06:07

What is the earliest acceptable time for you?

Lindsay Anderson20:06:09

8am PT or 8:30am PT works

Lindsay Anderson16:06:53

That works for me! I will send through a calendar invite now. I'll connect internally with the team about the use cases you're interested in exploring, but if you have any specific thoughts/questions ahead of time, feel free to leave them here so I can prepare to maximize our time. Looking forward to our discussion!

Lindsay Anderson16:06:26

@stijn.liesenborghs would you mind letting me know your email address?

Stijn Liesenborghs - Engineering Mgr - Nike17:06:15

<mailto:stijn.liesenborghs@nike.com|stijn.liesenborghs@nike.com>

Stijn Liesenborghs - Engineering Mgr - Nike17:06:51

Iโ€™ll discuss with my team and let you know the exact details.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:27

A few years ago in another life (ping @markus.lauttia) we started using Slack as the dashboard for customer-facing staff could do some self-service operations through messages to bots. I don't know what the status is now but when I left, it was really effective.

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:27

A few years ago in another life (ping @markus.lauttia) we started using Slack as the dashboard for customer-facing staff could do some self-service operations through messages to bots. I don't know what the status is now but when I left, it was really effective.

Markus Lauttia11:06:37

the bots are still there. They're mostly used nowadays for internal operations, like our employees being able to reset their AD password in case they forgot it or unlock their account

Markus Lauttia11:06:07

โ˜๏ธ that being the reason why we don't have SSO for Slack in use ๐Ÿ˜†

Ferrix Hovi - Principal Engineering Avocado - SOK (S Group)11:06:38

Yeah, the price hike for AD SSO on Slack is NOT COOL

Markus Lauttia11:06:06

well, they've done their market research right. On SaaS you should always find the one feature that people are ready to pay big bucks ๐Ÿ™‚

Stijn Liesenborghs - Engineering Mgr - Nike11:06:05

We have also leveraged the Slack workflows a bit for feedback/support of dev teams. Really liked the low code possibilities you are providing there.

๐Ÿ‘ 1
Laura Friend11:06:48

๐Ÿ‘‹ If anyone wants a demo of Slack - @rwhitehurst our Solutions Engineer is in this https://slack.zoom.us/j/92082029051?pwd=cnVpdEloWEg5ZDkrSmw0Mi9haGZXZz09ย (PW 057888)ย ๐ŸŽฅ

Anne-Christa Strik12:06:41

๐Ÿ“ฃ Make sure to listen in - Paul Williams from Lloyds Banking Group is doing a talk on their journey with GitHub Enterprise inย https://live.itrevolution.com/#track-4ย now and is also live in ourย https://github.zoom.us/j/93929282358?pwd=Z3pkSHJPTHAvZSs1M2VvTVhPUlZOQT09ย (password: GitHub) for any further questions

1
Scott Nelson Windels14:06:19

๐Ÿ‘‹ I'm a lead of the Incident Response program at Slack. I'll be over in our Virtual Booth for the next 20min or so if anyone wants to drop by to ask questions about our Incident Response process or the Incident Bot tool that @vbrennan shared in her talk. Feel free to DM me if you want to connect later. thankyou

๐Ÿ‘‹ 3
Scott Nelson Windels14:06:19

๐Ÿ‘‹ I'm a lead of the Incident Response program at Slack. I'll be over in our Virtual Booth for the next 20min or so if anyone wants to drop by to ask questions about our Incident Response process or the Incident Bot tool that @vbrennan shared in her talk. Feel free to DM me if you want to connect later. thankyou

๐Ÿ‘‹ 3
Scott Nelson Windels14:06:57

Hi @jiri.klouda we can chat in channel here or I'm on our Zoom if you want to chat IRL.

Simon Rohrer, [Sooner Safer Happier contributor] Saxo Bank, Head of EA and WoW14:06:31

And I believe you can even get it out of the box from ServiceNow now

Scott Nelson Windels14:06:13

PagerDuty has a great intergration, our bot was developed internally to automate steps of our Response and Command functions.

Scott Nelson Windels14:06:37

I know PagerDuty also has some separate tooling they use within Slack in their own Incident Response process as well.

Jiล™รญ Klouda15:06:34

I was just wondering what I saw in the presentation. What Slack itself is using.

Scott Nelson Windels15:06:47

It is built in house, a python app running on our infra, using Jira and RDS for persisting data.

Scott Nelson Windels15:06:50

The initial design goals were to start automating a lot of the toil/manual work that Incident commanders had to do around setup and calling all clear around an incident. So it grabs an incident number from the Jira ticket, creates the incd channel, shares it with another workspace (for our Customer Experience team), invites a group of users who can join every incident and also posts a standard list of common references for Responders/Commanders.

Jiล™รญ Klouda15:06:07

We could really use this as a feature. We have many channel where we are setting a person as a responder for the channel. We could also use the status for the channel with colors. Especially if that was configurable. Like this channel is at S1 or S3 status. With both the responder and the status posted with some defined periodicity in the channel.

Jiล™รญ Klouda15:06:07

I just really liked the UI/UX aspect of what you have.

1
Scott Nelson Windels15:06:38

Our Incident Commanders have loved it, it reduces the cognitive load for them of the rote steps in every incident and let's them think more about the incident itself.

V Brennan15:06:17

I totally plus one that. I get to focus on the incident information and figuring out the right escalation path rather than all the admin. That admin is necessary and important. It is good it doesn't slow us down.

Jiล™รญ Klouda15:06:14

Our incident commanders would love it too โ€ฆ hint hint ๐Ÿ˜„

Scott Nelson Windels17:06:57

This blog post talks about our Incident Response process as well - just came out https://slackhq.com/slack-channels-for-incident-management

๐Ÿ‘€ 2