I think there might be more to it as well; for the first time in a long time (in my experience, at least), there's more time to do crafty things - my commute, for example, has gone from 2 hrs/day to nothing.

definitely, it's an opportunity to do something tangible that most of the time we never get to do any more. We can poor hours into to making something just for us

I personally have less time than before because my childcare is gone. 🙂 When the kids were in school it was a time management nightmare. But I still appreciate the opportunity to do more hands-on things with them like cooking and gardening. The extra time, in our case, comes from all of their extracurricular activities, as well as all of our shows, parties, and events, being cancelled.

Excellent question! Most of our web pages don’t get enough traffic for A/B testing to reach statistical significance.

Doesn't it depend on the confidence interval that you are aiming for? (trying to remember A-level stats here!)

We do run experiments on lower volume sites too, it might change the time that we need to run the experiment in order to reach statistical significance

It does depend on the confidence interval, yes. It also depends on how dramatically different your results are between target groups. I know we have a few high traffic pages where we can usually get a result within a few weeks, but on others the test might have to run for six months or more! At that rate you’re better off using other methods.

Possibly App Insights in Azure?

Yep, as Hannah says, this is Application Insights

Yep, sorry for being slow to reply and with my bad Internet today the presentation might have got to the technical use of A/B tests/toggling so that clients can point to different backend systems

Would love to share insights on this particular topic! Maybe at another moment once your internet connection is better?

I have internet (for now) so happy to chat a bit more around this

When creating tickets for features we will also create a "clean up" ticket to be played at a later date. Once the experiment is run and a decision on the outcome of the feature is made we would look to clean up the toggle ASAP (sometimes the priority isn't there to do it immediately)

Brilliant idea!

Thanks. The previous company in my example; the codebase was an absolute monster of a monolith with very little to no automation.

And an accumulation of work in progress

We've considered it and have played around with it a bit but not used it extensively or brought it into processes

@ann.marie.99 do you have info or links on visual regression testing?

We’ve had luck with Hawkeye, I know. And the other one we use we call Landmark… let me see if I can find the framework.

PhantomCSS

Also BackstopJS

I think our Landmark tool is based on PhantomCSS and the Hawkeye tool uses BackstopJS. Visual regression testing is the best way we’ve found to detect “broken CSS” errors that may only show up in production.

No, I shall have to investigate that and see how we can make use of it

@andrew.sturrock The key is to ensure your UserID doesn't change so when you evaluate your flags, your user stays in the same variation. You don't want to tie those userIDs to Session IDs otherwise they will flip flop

@ferrix I think that is a business decision. Hopefully you work in partnership with the compliance department to understand what the rules of the road are. Some people will never be able to get rid of all their test environments

I find throw-away test environments desirable in most cases. It's not all things you should be verifying in production.

agreed!

Also, when you don't need the user to verify functionality, why bother them with it? 😄

Well you can still use a Feature Flag for other purposes, such as a kill switch or just reducing merge conflicts. But I definitely agree, not all changes need to be A/B tested.

Sure. Always use a feature flag for everything. Just don't make it enableable (nice word) before you have had it work through a test suite.

👍

True

@michael.gillett if the only environments were local and production, where was the testing done to verify if something worked or not?

Great talk, by the way 🙂 👏

Sorry, I missed your message @john710 - the testing can be done in Production with the QAs/automation able to turn the feature on for just themselves, once verified it can then be turned on for some group of real users to verify that it is all working

Makes sense, thanks Michael. The following posts in the channel (not by you) were suggesting that you were only talking about A/B type tests in production, not quality tests too, which is why I asked - thanks for clarifying 🙂

In that case I misunderstood! @michael.gillett Do you really not have any (non developer laptop/local) environments to do functional testing in before production?

Some team/apps do true trunk based development and only work on local and push to prod, most teams/apps do not work in that way and have at least 1 test environment

sets page to targetPage, evaluates to true so all traffic goes to the targetPage?

Hope you had monitoring driven development to accompany the hypothesis driven development 😉

Yes! Suddenly 500,000 web pages found themselves in an infinite redirect loop.

ouch

Within 5 minutes 20 people were paged, but it took us an hour to find the person who could fix it. After that we put in a “kill switch”.

Once an experiment is finished we would prioritise a piece of work to remove the toggle and the variation(s) we no longer needed within our codebase. If an feature is incapsulated well then its not too much work to remove it from code

Hi Alex, I won't answer for Michael, but I will say that you want to ensure your flagging system is tied back into your Git repository. You also want to have a way that you can see which flags are no longer being evaluated or has any targeting tied to it. LaunchDarkly, which Michael's team uses, has that capability

yeah, and should you develop new features so that they work only on top a feature that is behind a feature flag...

How does the test automation know if two flags depend on each other. How does it recognize that tests need to run on combinations of feature flags? Do we only run tests on all flags being turned off?

@jiri.klouda I think the key thing to remember is that there is no silver bullet to technical debt. Each time you introduce something into your code, you could be potentially introducing more technical debt. So you need to manage those introduction. With Feature Flagging, you should have a system that can integrate into your Git repository. You should also be able to see how often a flag is evaluated so you know when you should take flags out. You can doo all. of that with LaunchDarkly if you are interested

@jiri.klouda opps I was writing before you had the next question

using a single flag to enable a series of flags is a common strategy

we're trying to advocate the policy that feature flags should be relatively short-lived

I mean the pre-merge or pre-deployment testing.

@sherron ^^

Feature Flags should be short-lived. There are occassions when you might have a few operational long-lived flags that grow into your architecture, but that’s almost always by design [Subscription entitlements, operational gates for rate limiting, etc]. As far as Testing is concerned, there is always an assumption that you should test all combinations and that’s a common error. I’ve included a blog post which talks more about this, it’s a little dusty but the principles stand: https://launchdarkly.com/blog/testing-with-feature-flags/ If I was to pick out two key points it would be to: A. Feature flag should be dedicated to do 1 thing and 1 thing only. It should be small enough that if you describe it and use ‘… and … ’ you probably need to rethink what you’re doing. The idea is to release lots of small pieces of code that can be handled easily. This means that if it breaks you can turn it off quickly without affecting the rest of your release. The whole point of flagging is that you should be able to ship broken or unfinished code into production without it being a problem. B. Test your critical paths. Current production state and the Feature Flag ‘defaults’ scenario. Any Feature Management platform worth it salt should be able to help you smoothly enact those scenarios. Does that help at all, @markus.lauttia // @jiri.klouda? https://launchdarkly.com/blog/testing-with-feature-flags/

Ha. Wrote some strategy slides and had the CEO say: "So, this is essentially risk management." Yes, padawan.

I really need to ask @allspaw nicely if I can reuse his slides... still so relevant today over a decade on!

@vbrennan

We use a naming structure for channels to ensure information goes to the right place. For example we have #feat-xyz, #devel-xyz, escal-xyz. This ensures everyone knows where to go for what sort of information.

Thanks!

If information is needed across multiple places we use a bot for it. For example incident-bot posts relevant information to multiple channels and workspaces so it gets widely disseminated. It also helps keep curious people away from the incident channel itself, reducing noise there.

@vbrennan

We lean towards public. Having said that I have a private channel with my team. Right now I do a once a week check in on people as they wfh. It is a pretty personal and open conversation so we keep it private. Other than that we are all open 😄

That's the balance I think. People want to be open to talk within the team, but then the knowledge remains trapped within the team.

We keep our banter and team chat in public too. It makes for some really fun reading. Our open team channel and our Melbourne team channel are famous in engineering. People literally stop by to hang out and say hi. It is a great way to connect with other teams on a personal level too

I'm not sure if our App security review (not a certification) helps, but I believe that it is a new project in the past few months. https://api.slack.com/security-review

Might help building trust inside your org where needed.

Thanks 🙂 I'll mention this to the decision makers!

Another tab we added to apps that update it is helpful too - the Security and Compliance app per tab. This is a screenshot of PagerDuty's app info which they've filled out.

To be honest half the battle is that many (if not most!) of our workspace admins have just been nominated by someone else and don't really understand how to perform the role quite yet, so they're adopting a more risk-averse approach. Thanks again for helping 🙂

I will be in our virtual booth in 10min if you want to stop by - i am a lead in our Incident Response program at Slack.

Hi @ferrix We do have automation for restarting services with visibility through our alerts channels in Slack. We don't have a way to kick it off manually in Slack though

<understatement> The Teams APIs are not as easy to use as Slack's. </understatement>

We have decided that Teams is the tool for people whose job is emailing powerpoints to each other. Slack is the tool for DevOps teams.

Ouch, 'not allowed' feels like a culture problem there 😕 but don't get me wrong - I feel you!

You can do a lot with pushing messages with links, but it is not that sophisticated.

Crazy idea but if we had a high level influencer here from MS perhaps there is an opportunity for the 'Teams team' to learn from here... i.e. that 'playing nicely' and having better integrations with other first class services helps the overall ecosystem. Getting better together ❤️ @samgu might know someone I guess...

We recently moved from Slack to Teams as it was chosen as a corporate platform and for many reasons it made sense to go there...we're now suffering some inevitable issues where as a corporate platform the simplest things like unauth webhooks are being picked out as security violations and we're having to fight to even use Teams as-is :face_palm:

No custom bots allowed yet either and anything from the marketplace goes through a multi committee, many week/month approval process

I have fed back to MS a few times and they just say "it's supposed to work like that". For example, having membership at the team level rather than the channel level just doesn't work as well, but they said it was deliberate.

Its very tied to the existing Office365 architecture which influences the member handling it seems

GAAAAH!

Yes. I also find the ironic thing is that because the default choice is private teams, that you end up emailing documents around because people aren't members of the right team and therefore can't get to the sharepoint behind the team!

"....multi committee, many week/month approval process" aaargh those words are resonating with me so much! We really need minimum viable compliance, or just build our appetites for risk and make things work better, safer, happier in spite of those who seem determined to make things worse, slower and far more annoying!

Slack's Onedrive/Sharepoint integration is great. You post the link and it pops up a thing which sets the permissions for you to read/edit for anyone in the conversation.

@karl.marfitt as per Mark Schwartz yesterday! Minimum viable bureaucracy 🙂

I do think Teams is getting better (from a devops perspective, and its taken a while) and it wasn't an easy choice to give up a private Slack instance that allowed our small group really innovate within our space over the years, but sometimes the corporate tool solves more problems than it introduces...I thought, before we realized we couldn't even use Teams as designed 😞 And I find a MS rep now fighting our corner with our own Corporate IT, strange times 😄

Minimum Viable Bureaucracy

🙏

Policies can sometimes be very interesting. There is a particular Teams instance where "I don't know which channels I am supposed to be on and can't figure out who to ask because everything is hidden" is the biggest COVID bottleneck.

Oh yes, as a huge enterprise of hundreds of thousands of employees, we have way too many workspaces with overlapping purpose and so many missed opportunities for valuable collaboration. I am on at least a handful of competing agile channels. The answer we found was opening up and sharing the most popular channels across the whole org but this seems to be happening far too slow... I sometimes wonder if mixing realtime chat with a Slashdot or Reddit-like karma or other 'conversational value' +1 crowd-voting based system might be a better approach

I would consider that a lost cause. Creating a functional microcosmos is enough 😄

Getting a hundred people to go through mandartory security training on a yearly basis seems to be quite a feat and getting one non-altered message through to a thousand people is near impossibility, so having "one Agile" in hundreds of thousands sounds astronomical enough to just give a pass 😄

This is where we're at having joined that corporate Teams instance; hoping to continue looking after our corner but seeing could we leverage some experience from elsewhere. For the reasons above though, several months in, I can't say I've stumbled into anyone outside my immediate group to share stories with 😞

In our Slack we have lots of channels shared between workspaces in order to increase collaboration, and it works. You can't do that in teams though.

We've also been more disciplined in not having millions of workspaces than in teams where there are so many teams that I have largely given up on most of them.

Thank you all for your insights. We have started doing some integrations with Jenkins for build status but it seems very limited. I once connected with the the head of Azure's Global SRE, you may know him a he was one of the authors of the Google SRE book (Niall Murphy), and he told me that MS was investing a lot into Teams to use it like Slack.

For what its worth, webhooks alone accomplish a lot of what you need in integrating Build tools to Slack/Teams in my experience anyway. I'm hoping that we can push Teams further now we're (stuck) there 😄 Our contract is pretty big so we're well engaged with MS reps the whole time

@swindels - thought you might be interested in this thread if you'd not seen it

Yes. We set up channels for this exact purpose. We spin channels up and down all the time around specific topics. We ensure that a couple of folks drive the conversation or at least get it going to start with.

@vbrennan one of the best features of StackOverflow is that a question can be marked "Answered" and easily searched for later. Can you do this?

We use threads for specific answers. You could use a custom emoji to indicate when a question is answered or complete

you can also send those threads to a channel based on the reactji using reacji channeler

@vbrennan so that channel persists and can be searched later?

Yes exactly

Thank you

If we read something anywhere in Slack that we think is relevant for our team we use a 'team-aware' reacji and reacji channeler send it to our team channel. It helps ensure relevant information is directed to us.

We also use this internally for things like an announce-incident-reviews channel and all meetings and Review documents get reacji’d into a central channel

👋 @stijn.liesenborghs we can put you in touch with your customer success manager at Slack who can help with this? She works with your HQ team in the US

Yes pls

great, will chat to Lindsey (CSM) and get her to get back to you

@stijn.liesenborghs I have asked for Lindsay to be added in here. She will follow up later when she comes online

@stijn.liesenborghs 👋 meet @landerson who looks after Nike at Slack! She can help you out!

@stijn.liesenborghs it's great to meet you! I'd be happy to chat with you about ways to implement some of the Slack use cases you saw today with Nike. Do you want to send through a few days/times you're available for an introduction call the week of 7/7? I'm on the West Coast in the US so Pacific Time zone

What is the earliest acceptable time for you?

8am PT or 8:30am PT works

8:30am PT on 7/7?

That works for me! I will send through a calendar invite now. I'll connect internally with the team about the use cases you're interested in exploring, but if you have any specific thoughts/questions ahead of time, feel free to leave them here so I can prepare to maximize our time. Looking forward to our discussion!

@stijn.liesenborghs would you mind letting me know your email address?

<mailto:stijn.liesenborghs@nike.com|stijn.liesenborghs@nike.com>

I’ll discuss with my team and let you know the exact details.

Sounds good!

the bots are still there. They're mostly used nowadays for internal operations, like our employees being able to reset their AD password in case they forgot it or unlock their account

☝️ that being the reason why we don't have SSO for Slack in use 😆

Yeah, the price hike for AD SSO on Slack is NOT COOL

well, they've done their market research right. On SaaS you should always find the one feature that people are ready to pay big bucks 🙂

Not seeing the conversion though

Hi @jiri.klouda we can chat in channel here or I'm on our Zoom if you want to chat IRL.

I know PagerDuty and VictorOps do this

And I believe you can even get it out of the box from ServiceNow now

PagerDuty has a great intergration, our bot was developed internally to automate steps of our Response and Command functions.

I know PagerDuty also has some separate tooling they use within Slack in their own Incident Response process as well.

I was just wondering what I saw in the presentation. What Slack itself is using.

It is built in house, a python app running on our infra, using Jira and RDS for persisting data.

The initial design goals were to start automating a lot of the toil/manual work that Incident commanders had to do around setup and calling all clear around an incident. So it grabs an incident number from the Jira ticket, creates the incd channel, shares it with another workspace (for our Customer Experience team), invites a group of users who can join every incident and also posts a standard list of common references for Responders/Commanders.

We could really use this as a feature. We have many channel where we are setting a person as a responder for the channel. We could also use the status for the channel with colors. Especially if that was configurable. Like this channel is at S1 or S3 status. With both the responder and the status posted with some defined periodicity in the channel.

I just really liked the UI/UX aspect of what you have.

Our Incident Commanders have loved it, it reduces the cognitive load for them of the rote steps in every incident and let's them think more about the incident itself.

I totally plus one that. I get to focus on the incident information and figuring out the right escalation path rather than all the admin. That admin is necessary and important. It is good it doesn't slow us down.

Our incident commanders would love it too … hint hint 😄

This blog post talks about our Incident Response process as well - just came out https://slackhq.com/slack-channels-for-incident-management