Slack is down for some users
status.slack.comWhenever I see a product this large having an outage (a lot recently, especially with Facebook, GCP, and AWS), I can only think of how stressful it must be for whoever needs to fix it. I've been on the other side of the outage before, albeit with a much smaller product, but lord was that stressful. Thinking about the random engineers that are stressed out and thinking that they could be fired for this, even if the cause wasn't their fault (since management at large companies can be pretty thick) is very upsetting to me. Say what you want about a large company having a large outage, but it's normal engineers that are trying to fix it at the end of the day, and I can sympathize.
I can also sympathize with the engineers trying to fix this, but I hope that they wouldn't be thinking that they could be fired for this. Successful teams that I have worked on, even at big companies with high-usage products, have always promoted a culture of "systems break - let's improve the system, not blame a person". Any 'mistake' by an employee is actually a sign of a problem with the system. Any resilient system should account for human errors - those always happen. I wouldn't want to work for a team or company that would consider firing somebody for causing an outage rather than addressing the root cause.
I work at Slack and have learned a lot from our Incident Response program. Brent Chapman helped put it together, and has a USENIX talk about it here: https://www.usenix.org/conference/srecon21/presentation/chap...
Response for this incident went by the book, as described in Brent's talk above. Incident Management programs like these ensure that incidents can be resolved while also minimizing stress and chaos for engineers and other responders.
PagerDuty has a good Incident Responder and Incident Commander training courses, if you are interested in setting up a program similar to Slack's:
- https://response.pagerduty.com/training/courses/incident_res...
- https://response.pagerduty.com/training/incident_commander/
Fun fact: Brent Chapman is also known for creating the `majordomo` mailing list manager from the early 90s
In a well-managed engineering organization, it shouldn't be stressful. Sense of urgency, sure, but stress is typically only making things worse.
stressing out and panicing is the worst thing you can do in this situation.
usually, keep your head cool and focus on the problem at hand.
This is also the reason why engineering/operations should be seperate from customer communications. The people who are fixing the issue should not also be the ones doing the communication with the outside world.
I often thought of the Zoom engineers on-call during the pandemic shutdown.
Talk about being on center stage..
The cool thing about Slack (and Discord for that matter) is that it's essentially federated on a technical level. SlackHQ controls all the instances, but the UX is very achievable by an open source alternative. I'm bullish on federated chat in the next few years.
It would be awesome to have essentially the same experience as using Slack, but also a server running on our lab's hardware that we can reboot/debug ourselves.
They've got to nail the integrations. Anytime moving to slack alternatives is brought up with people I work with, it always comes down to everything integrating with slack.
^This
We aren't even particularly dependent on slack, but the convenience of a few basic integrations like google calendar make moving to Discord or anything else a non starter.
The integrations, and chat itself needs an open protocol, otherwise it's just going to be more and more of slack in the future.
Well, WRT Google Calendar in particular: https://www.discalbot.com/
I understand that's one of at least several, and that if Slack isn't actively making trouble, looking for alternatives is a switching cost that may itself be prohibitive -- but if Slack is becoming really troublesome, it might be easier than you think.
That's interesting. We've played with a couple integrations but don't use any. What sort of org are you in? We're an academic bioinformatics lab with 15-20 people.
Not OP, but we have numerous integrations. Technical, corporate, and fun. GitHub, Jira, Status Hero, Outlook Calendar, Asana, Conference Booking, polls, "water cooler" type apps, and more. I actually can't get a comprehensive list right now because Slack is down...
We could make do without many of these integrations if we had to but the integrations and webhooks are THE killer feature to me. Chat is easy to solve.
FWIW, some of the hosted Slack alternatives, like Zulip, support at least versions of some of these -- see their integrations page here: https://zulip.com/integrations/
What would be ideal from this perspective would be if they could make the integrations somehow code-compatible with Slack, but that's tough. For example, Zulip's topic system is, IMHO, vastly superior to Slack's threads, but every public message is under some topic (within a stream -- there are two levels, not one), everything posting a public message has to supply one, and that alone might make it difficult to reuse a Slack integration unaltered.
Element/Matrix is getting pretty close, and is federated at the protocol level.
I think "first comer's advantage" will win over for the really big outfits like apple, ibm, intel, etc because they've gone through integration pains with other services and don't want to go through that pain organization wide. However, I think small to mid-size businesses could find some migration from slack to matrix pretty advantageous.
I'm pretty familiar with the various alternatives. Mattermost isn't federated, right? Anything without a strong inter-instance collaboration story can't replicate the experience of Slack/Discord.
Mattermost is not federated.
If you want federation, I would suggest you don't use the "essentially federated" Slack, but the actually federated Matrix.
Tried with my friends a few years ago. We almost achieved escape velocity, but it wasn't ready. I've been keeping an eye on it, and my feeling is while it may be ready for what we needed then, it's not ready for what we need now, because now it has to be good enough to not just work but assuage my friends' bad feelings from the last time.
I think I'll probably try again once most people have switched to a server written in Go or Rust.
One fun thing we discovered an outage or two ago: Zoom, which is likely already on your computer too, has a very Slack-like Chat feature. Open up the Zoom app, then click the Chat icon in the top header. You probably missed it this entire time. You can make rooms, DMs, animated gifs, the works :)
It's strange but Zoom as chat was actually how I first used it. A whole year prior to the pandemic the place I worked (fairly large) they purchased Zoom as their internal chat tool.
I find zoom chat to be reminiscent of AIM, it does not have the afforances that make slack a good business communication tool.
Definitely. But super helpful in a pinch when we all begin these very large email threads as soon as slack is down. :)
What are Slack's affordances?
Slack does a good job with threading. The side panel allows you to watch a thread while you continue another conversation. It also puts together an above average notification system, where at a glance you can see if there is no new content, new content, or your name mentioned. It allows you to ignore slack without risk of missing somebody shouting for you.
On the contrary, I find Slack threads to be very annoying for discovering/stumbling upon conversations.
Your whole team could be having a new discussion within a thread and you'd never know unless you participated in the thread previously or somebody @'s you. Compare that to more IRC-like discussion where each message is posted to the channel. Sure, you may end up with contextual issues reminiscent of the IRC days, but at least everything is discoverable.
I've used Slack competitor Flowdock in the past which did the more "elevated" approach to threads. Rather than "hide" them from the main branch of a channel, the main branch had every message in the top level and you could collapse to a thread view if you wanted to read it in isolation.
In my experience, the main problem here was that people regularly lost sight that a conversation even was in a thread. So they'd post their response unthreaded right in line. Now the people in the thread view can't see it, but if you were in the main channel it still looked just fine (after all they were one after the other). If it was a nascent threaded conversation (like 2 or so posts only), often the thread was just abandoned and everything was done top level. For bigger threads where you caught after the fact it was a threaded, people would either delete and repost or would just duplicate their post over into the thread again (which means it appears twice in the main channel). The worst offenders were people who refused to thread so even if it was humming along fine they'd just post their response inline always (and repeatedly).
Don't know if I have a particular point there other than both had tradeoffs. I do find that the Slack thread updates disappearing seems to be less frequent of an issue and the threading problems in the alternative were near constant. I think Slack's issue would be largely a nonissue if viewing a thread "subscribed" you to it so you could see updates without needing to post. Maybe reactions are a workaround too. It's fairly easy to unsubscribe if one's blowing you up. Subscribing to all threads regardless seems to reduce some of their usefulness and that's the current solution I believe.
The issue you mention with Flowdock, in my experience, only happened for newcomers to the product, they added a button to add an unthreaded comment to a thread in later versions.
I prefer Flowdocks threading model, it's too bad what happened to the product after being bought/sold, but I think it was a superior product.
> Your whole team could be having a new discussion within a thread and you'd never know unless you participated in the thread previously or somebody @'s you.
That's kind of the point? You shouldn't be bothered unless you want to be. To me, a thread isn't necessarily supposed to be geared toward discoverability. You'll see the main topic hit the channel, and then you can choose to follow the thread if you like.
>Open up the Zoom app
no thanks. I would never install zoom on any of my computers voluntarily.
That spyware thankfully isn't no mine, and if you're forced to use it (if you for whatever reason can't use Jitsi/Teams) then I strongly encourage you not to install the app and instead use the web version.
Why is Slack considered spyware but not Teams?
Please don't encourage the use of Zoom, they already have enough market share they don't deserve after lying through their teeth to the public on multiple occasions.
What alternatives do you recommend? As far as I can tell none of the major video conferencing solutions are made by companies I think are any more trustworthy than Zoom at this point. Plus Google and others ran a focused assassination campaign against Zoom at the start of the pandemic to claw some of their market share.
There is Matrix (https://matrix.org) which now integrates nicely with Jitsi (https://jitsi.org). There's also BigBlueButton (https://bigbluebutton.org).
Google is certainly more secure than Zoom to outside attacks.
Regardless of what Google chooses to do with your data, they do want to protect that data from everyone else, on your behalf. At least as much as any company can (government subpoena affects all companies).
I don’t understand why zoom is so much more popular than Google meet. I never used zoom before my current company who uses Google meets for daily meetings and then zoom meeting for the biweekly review. No idea what advantages zoom is supposed to have.
I think Zoom just had a better sales team and went after corporate clients earlier. Now that no one is in the office, it seems silly, because Meet is just straight up a nicer experience on all fronts, but in the Old Days, no one ever got fired for putting Zoom hardware in a conference room.
It seems like the popularity of Zoom is related to the singularity of the product's purpose. Zoom is a video conferencing service. Google is everything and the kitchen sink. So when the pandemic started, that worked in Zoom's favor, sadly.
Thick clients perform better than Google Meet in the browser in my experience.
I know it’s not seen as a professional service, but Discord has great quality video chat (better than Zoom, for example). I don’t know if its privacy/security would hold up for larger companies but we use it in my tiny company with no issues at all.
Any recommendations on getting Discord notifications to reliably show up in a reasonable time on mobile? That's the one big issue I have with Discord. Otherwise I find it very useful.
If you have the desktop client up, you don't get mobile notifications. They've always worked fine for me so long as I don't set my status as do not disturb
Mattermost has replaced Slack here and I don't miss it at all. Jitsi Meet is pretty good for a/v
My team uses Slack for text and voice chat (huddles) and Jitsi Meet for free, in-browser video chat without signup required.
Jitsi[0] is excellent.
[0]: https://meet.jit.si/
Discord is easily best quality for now, Matrix/Jitsi is easily best future vision. Everything else can go away.
Self-hosted Matrix
what do you mean lying through their teeth? citation needed.
https://www.ftc.gov/news-events/press-releases/2020/11/ftc-r...
> In its complaint, the FTC alleged that, since at least 2016, Zoom misled users by touting that it offered “end-to-end, 256-bit encryption” to secure users’ communications, when in fact it provided a lower level of security... In reality, the FTC alleges, Zoom maintained the cryptographic keys that could allow Zoom to access the content of its customers’ meetings, and secured its Zoom Meetings, in part, with a lower level of encryption than promised.
See also: https://www.tomsguide.com/news/zoom-security-privacy-woes
"end to end encryption" means "it's encrypted at both ends"
Wikipedia has a few paragraphs dedicated to their lack of privacy, security, and other unethical telemetry practices. I believe it was a couple years ago when these stories maintained headline positions for a few weeks.
They can tell me they're the second coming of Christ as long as they make televideo work better than all the crap alternatives.
Things like this remind me how dependent I am on IM for doing any work, which is weird because it doesn't feel like that should be the case. If you were to measure the raw number of seconds I spend sending IMs, it would be a relatively low percentage of my day, but I always forget how vital that percentage actually is.
As it stands, while I'm not fully blocked from being productive, I am blocked for a lot of work that needs to get done.
Yep. Especially if your development flow has integrated Slack alerts.
Coupled with COVID work-from-home constraints, there's no practical way right now to get code reviewed for merge into the main codebase 'round these parts.
Reminds me why we intentionally avoided tying core biz/dev workflows to Slack. :)
Devils advocate...
Do you tie core biz/dev workflows to Github (or any other service)?
Requiring GitHub to deploy your code to prod... goodness no.
Using, highly available, deterministic artifacts, that can be rapidly deployed or reverted from production is what we do.
Fun fact: For a lot of us, Slack web is down, but Slack mobile is still working.
Down for me. Can't use the desktop app or the browser. The app seems normal, but IDK if it's receiving traffic.
The iPhone app seems to be working fine for me. Web app or "desktop" app neither one are working. It's been more than an hour here.
This is my experience as well. Running on MacOS 12.2 for desktop app, Firefox for the browser (latest stable version).
However, the iPhone version is working just fine.
Down for some of my accounts, but not others.
Yeah, my non-SSO accounts seem fine.
It's down for some channels for us. Getting a weird message undefined error when I try to post in one of our channels. Other channels are still fine.
They are responding on twitter that there's something up: https://twitter.com/SlackHQ/status/1496133311558737923
My whole team doesn't want to work today because we'll have to use MS Teams as a backup. Rip
We use Google Chat as a back up. It's always nice to see Google haven't killed it between Slack outages.
Just set this up real fast as a quick backup, since we already use Gmail and everybody is already in it. Having a non-AWS hosted solution seems important to have a Slack backup for me, as well. Oh hey would you look at that, they support webhooks too...
Isn't Google Spaces the Slack-like thing? Terribly confusing job Google has done.
We switched from Slack to Teams for various reasons about two years ago. I'm still mad about it. Teams blows for chat. The video meetings are fine though.
The MS Teams UX is pretty painful, especially for infrequent, communication-heavy users. Not painful enough to outright prevent work, but clearly more painful than the repercussions of missing a few hours of work.
This seems at best a funny tweet. They're both just tools and frankly terrible in terms of "getting work done". This seems like a great time to go heads down on a deep task and if you really need to contact someone send an email or even phone them.
Surprised no one is talking about how awesome slack is. Think about it in user stories:
* I want to find someone by name => ctrl+K
* I want to search with sane keywords => search, "from:me" "to:<channel name>" work.
* I want to remember someone I talked to recently => people you chat with show up in the list on the left
* I want to keep com channels organized as the ground changes => easily rename channels, favorites UI works well, channel grouping works right.
* I want to give a public emotional response to someone's statement => reactions
* I want to continue discussion of a point someone made, which may not be relevant to everyone in the channel => threads
* I want to edit a typo I just made => hit up on keyboard
It's got that quality Factorio has, where you can let yourself imagine it is the ideal product, and start expecting features you need to be there, rather than not bothering to explore because you think they won't be.
It is AMAZINGLY good at solving actual user stories around communication. I have lots of respect for their PMs.
Sorry, but dropping Markdown support in favor of an input UX that is to this day atrocious gives me little faith in Slack's user stories moving forward. I'm fine if they want to make the default mode more intuitive for a broader audience, but I have yet to hear a good argument for not having a little button to enable raw Markdown mode.
EDIT: See comments below for instructions on re-enabling Markdown mode.
Preferences --> Advanced --> Format messages with markup.
̶T̶h̶e̶ ̶o̶n̶l̶y̶ ̶d̶o̶w̶n̶s̶i̶d̶e̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶m̶o̶d̶e̶ ̶d̶o̶e̶s̶n̶'̶t̶ ̶s̶u̶p̶p̶o̶r̶t̶ ̶m̶a̶r̶k̶d̶o̶w̶n̶ ̶s̶t̶y̶l̶e̶ ̶U̶R̶L̶s̶ ̶l̶i̶k̶e̶:̶ ̶ ̶[̶e̶x̶a̶m̶p̶l̶e̶ ̶s̶i̶t̶e̶!̶]̶(̶h̶t̶t̶p̶s̶:̶/̶/̶e̶x̶a̶m̶p̶l̶e̶.̶c̶o̶m̶)̶ Never mind, it looks like Slack has fixed this!
They originally got rid of markdown completely when they added the WYSIWYG editor, then added that preference after people complained
Yep, I remember that. It was quite the controversy at the time.
I think it does, but only when "Format messages with markup" is on, see https://slack.com/help/articles/202288908-Format-your-messag... and the asterisk at the bottom
THANK YOU
Honestly surprised I haven't come across this before.
Also, link formatting seems to be working for me?
This was broken when I tried it out during the Markdown-->WYSIWYG switchover a while back. I updated my OP to show they have actually fixed this (go Slack!). I just switched back to Markdown again.
I still have my pitchfork from all the bugs I encountered when they released the new formatting (still works wonky depending on the workspace for me). That said, I believe this is an option since they changed things:
That toggle is buried in the preferences/settings under "advanced."I have yet to hear a good argument for not having a little button to enable raw Markdown mode.Format messages with markup The text formatting toolbar won’t show in the composer.Settings -> Advanced -> Format Messages with Markup may be the first thing worth doing for you when Slack comes back online.
They have one. Kinda.
Significant regressions in Slack:
* I used to be able to type a message containing "@channel" or "@here" and just hit Enter without looking, and 100% of the time it would be parsed correctly. Now it fails to detect the "@channel" or "@here" often enough that I have to look to see if it failed, then go back and retype that part of the message until it lights up. Once a week or so, I see a message posted by someone else that contains "@channel" in black, and they expected the message to notify everyone but it didn't.
* Recently Slack pushed everyone to switch from usernames to full names with spaces, and eliminated the entire concept of unique usernames. There were many annoying consequences to this, one of which is that when I type "@name", now I have to look to see if it highlighted, or look to interact with the drop-down menu, instead of just typing and knowing it will work.
* Searching also got worse for the same reason. When I type "@name" in the search bar, it NEVER lights up. For 90% of searches this means I can no longer type in a search query and just press Enter. I always have to look through the drop-down menu and either mouse-click or press down-arrow repeatedly to get to the thing I want.
> * Recently Slack pushed everyone to switch from usernames to full names with spaces, and eliminated the entire concept of unique usernames.
That's your Slack settings my dude, blame your admins.
It sucks in that you can't edit the image attached to a message. Only text..
WTH? Why not?
Agreed. Major pain point for me. Can't reshuffle image order, either.
it's like they just don't care.
Why can't you have image inline with text? I hate to waste energy to refer to attached images in my text ("regarding the 2nd image")
ctrl+k should be 'create link on highlighted text', I'm constantly frustrated by this.
Ah but ctrl-v while highlighting text with a URL on the clipboard creating a link is such a nice experience. I honestly wish I could do that everywhere.
ctrl+k should be "kill line" and I'm constantly frustrated by the "create link" behavior in apps like Confluence.
What? ctrl+k should enter a digraph.
I love slack. Slack search is terrible for me, but maybe it's the quality of data it has to search through.
I'm curious about how many users are on your organization's Slack. I consider it one of my go-to sources for my org that has had between 250-300 people for the past 3 years. I can see it being suboptimal for a bigger company, though.
I'm in an organization with many thousands of people and find search to work very well.
Does the pricing plan influence the amount of resources your organization can use? e.g. slow search on the free plan, but much better on Enterprise Grid.
https://status.slack.com/2022-02-22 has been updated with this:
> A configuration change inadvertently lead to a sudden increase in activity on our database infrastructure. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. We introduced tighter rate limits on connection requests to reduce the load on the system. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected.
> Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. However, we moved too quickly and the increased activity affected the system again. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.
I wonder if their bottleneck is at vitess vttablet or at mysql.
Slack is up for me...unfortunately.
Genuine question for those running self hosted tools like Jitsi: how often are your outages? How quickly do you repair them?
I run an IRC network in my spare time that has had 13 minutes of actual downtime in the last 15 years (and about 20-25 additional minutes of degraded availability).
The degraded availability thing is because I run multiple servers for latency, normally an IRC outage is super clear, except with netsplits where IRC servers themselves unlink from each other. (I average about 3 of those a year and they last about 30s roughly).
It's not video though, so that doesn't answer your question thoroughly.
At first, I thought it was just my internet connection making fun of me. Then, I asked my co-workers if they experienced the same thing, and they did. It's a bummer that it temporarily made our communications quite hard and unstable.
How are there two active threads on this? https://news.ycombinator.com/item?id=30428142
It's pretty confusing.
Slack was down to me, and now that it is back, it is begging me to subscribe to use paid features, with the nag screens blocking my view of important stuff.
I guess it was down so they could do this crappy update.
Most of my company have spotty connection. Down for Windows and Linux alike. Interestingly (and several have commented this already), the mobile app seems to be somewhat working.
Slack client is down on my Windows 11 desktop, but up on my Windows 10 laptop. Mobile app also works fine.
I'm putting my money on an expired certificate in certain client app versions.
Completely completely unavailable across two organizations
"Some users":
Every single person on my team and at my company.
The desktop app works fine for a coworker in Arizona, but for me on the east coast web and desktop are down, but mobile (iPad and iPhone) is working.
We've got one team member saying its fine, but she's very lonely on there!
I'm having issues with loading threads at the moment. So far that's all I've noticed.
Oddly, I'm the only person on my team of 15 that is having any issues.
Down for my whole company.
If I was still a Slack user I'd be so happy right now :)
It's back :)
Now jira is not loading for me, wtf?
Somewhat related, the peloton app is also down, how frustrating
nope, JIRA still works for me unfortunately.
Mine still doesn't - this whole thing is going to come down as AWS outage IMHO.
Down for me. Using the browser.
Browser fails with the same message:
ps. restart of slack in browser? wtf?Something’s gone awry and we’re having trouble loading your workspace. Sorry we can’t be more specific – this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us."ps. restart of slack in browser? wtf?"
In all seriousness, since I'm sure a lot of people don't know this, use SHIFT-CTRL-r. That instructs the browser to reload everything, ignoring cached resources. This can occasionally "restart" a web page that even a reload or loading the URL in the URL bar again is not sufficient for. Slack is one of those things I've seen it work on.
However, SHIFT-CTRL-r isn't enough now.
Somewhat OT, but I love the phrasing "Something's gone awry", so much more elegant than the usual "Oops, we dun made a bad".
>restart of slack in browser
most people IME use the slack app, in which case "restart" makes sense.
Yes, I know.
The desktop app is Electron, or something similar.
Yeah— it's Electron. They made a big deal about moving away from their native client code bases a few years ago.
Working for me.
Down for me.
what a sigh of relieve.
It's a really refreshing feeling to know, that channel of communication is not available for anyone in my company.
it's like a veil has been lifted off my eyes