Segment Security Incident
segment.com> For a small subset of customers (13), the unauthorized party was able to gain read-only access to their workspaces and click around in their accounts for up to a few minutes. These customers have been notified.
That implies that Segment employees, or a subset, have unfettered access to view their customers' accounts? That it didn't require positive customer assent to gain access?
I like Box's model for this: https://community.box.com/t5/Working-with-Box-Support/Box-Pr...
Only 13 customers affected? I guess I won the haveIbeenpwnd lottery today.
Here is the email I got:
" We are writing to notify you of a security incident that happened between August 26 and August 31, 2019. We became aware of it on August 31, resolved it immediately, and have been assessing the scope of impact since then. Based on that ongoing assessment, we have concluded that the incident involved your workspace. We are very sorry for any issues this may cause.
What happened? Between August 26 and August 31, 2019, an unauthorized party compromised a Segment employee’s account and used it to gain unauthorized access to Segment product usage data. Upon detection, we took immediate action, disabling and deleting the account and removing unauthorized access. We also reported the incident to law enforcement.
What information was involved? The unauthorized party accessed information about how Segment users interact with the Segment product, as well as first name, last name, email address, and IP address for your Segment users, and the Segment write keys for your workspace. No Segment customer passwords were compromised.
No personally identifiable information relating to your own customers was accessed. As a result, no action is required from you at this time.
More information For more information, please visit oursecurity bulletin page. This page contains a detailed timeline about what happened and information about what actions Segment has taken in response. It will be updated with any new developments.
If you have any further questions, please reach out to support@segment.com. We again apologize for any inconveniences this incident may cause.
Sincerely, Coleen Coolidge Chief Information Security Officer"
I think you weren't in the 13 because they said "No personally identifiable information relating to your own customers was accessed"
P.S. I got that email, too
That message doesn't say the attacker got access to your workspace or clicked around in it.
From above:
"Based on that ongoing assessment, we have concluded that the incident involved your workspace."
Wow I also received this email today. Hard to believe only 13 people were affected.
Same email for me..? Are we all on HN or was this sent to more than 13 people?
I would check your logs for any suspicious activity
I got it too. Must be large scale.
I'm just guessing here but, given that Box doesn't use end-to-end encryption, I bet that their "grant access" button is meant for their support agents, and that [some subset of] their backend engineers could access your data at any time.
It's an interesting exercise to imagine an elaborate multi-custody arrangement where all data is encrypted with keys that live in hardware security modules and at least three of five senior people are required to join their secrets any time the KEK needs to be handled directly... but I can tell you that that kind of scheme where 100% of customer data is completely opaque would be a major inconvenience in a fast-moving SaaS startup. Not only is implementing an engineer-proof privacy scheme a massive undertaking but it makes debugging production issues much more difficult. If indeed they were going to go all-out on such a scheme I would kind of expect them to brag about it but I don't see claims like that in https://community.box.com/t5/How-to-Guides-for-Account/Data-....
I'm curious if anyone's worked at big-name cloud services providers and can speak to the practical effectiveness of privacy measures in place. My guess is that even at Google, for example, a rogue engineer working on a particular product (or maybe the infrastructure it runs on) could theoretically bypass organizational controls and look at user data.
To be clear, I don't really view this state of affairs as a big problem. I've commented before on why (https://news.ycombinator.com/item?id=20513055). Professionalism and integrity (in combination with organizational controls and the principle of least privilege) work fine IME, plus the fact that nobody except you really cares about your personal data. But I do think that your expectations are unrealistic if you think that backend engineers generally don't have the kind of access I'm talking about. Unless a service encrypts on the client with your encryption keys, you should expect that the company's "employees, or a subset, has unfettered access."
I'm not on my computer right now and I wonder if they rotated my write key since they don't mention it...
Segment write keys are technically public. Most sites have them published in the client facing JS
We built a similar feature at Hover as part of our GDPR efforts. https://www.hover.com/blog/hover-and-your-privacy/
In all the SaaS type applications I’ve ever developed, the privileged “Admin Panel” functionality is always on a private network accessible only via VPN access, which is authenticated via a password + private key. Then the Admin Panel itself has its own separate login, which is username + password.
Ideally the private key for the VPN is a hardware token, but I will admit in most cases it’s simply a file on the drive.
The Admin Panel has zero third party JavaScript nor client-side analytics.
I’m not sure this buys quite as much security as it might appear, because there’s probably any number of ways to piggyback on a valid session.
For example, there was a bug bounty Tesla paid out when a researcher was able to establish XSS by mangling their car name, which was read via the API and showed up in an internal dashboard as unescaped HTML.
So I think the biggest attack surface for Admin Panels remains a hostile client seeding XSS into unescaped fields, and CSP helps a great deal with this, but at the very least I don’t see any reason why these panels should be on a public URL.
You can say, well, they should have MFA on the panel, but I can’t shake the feeling that these simple measures avoid a huge number of attackers looking for low-hanging fruit.
It’s like putting SSH on a non-standard port. It’s theoretically meaningless but practically it’s a huge improvement, if only because it helps attack signals stand out against the script kiddie noise.
Could you elaborate about the architecture? I'm really interested in learning this.
At which level the interface between admin panel and apps is? Is it a dedicated app for the admin panel and just a front end to an AD or database? Like there is no real admin panel on the app itself.
Yes, the user/client facing app is a separate website from the admin panel site.
They share much of the same underlying data access and business logic code, but the screens that let you see all the clients, change SKUs, monitor billing, impersonate a user, generate reports, etc. just would not exist on the public-facing site.
Underneath both sites/apps are talking to the same database(s).
Between this and CircleCI, this sounds like a targeted credential-stuffing attack against accounts on Segment. If this is the case, it sounds like the only two cases that have been detected are Segment (detected on August 31st through unknown means), and CircleCI (detected on August 31st through an automated email).
Has the risk of of other Segment accounts having been compromised through the same channel (but have yet to be detected) been investigated?
Was it Segment that CircleCI was referring to? https://support.circleci.com/hc/en-us/articles/360034852194-...
I hope not, since there's a 5 day gap between CircleCI being notified and the rest of their customers.
Based on the wording, it's my belief that CircleCI was referring to Segment, but I have 0 inside information to confirm this.
Even if it was though, Circle discovered this issue on their own through an automated notification of an action taken; they do not seem to have been notified directly by Segment. It doesn't seem fair to assume that _if_ Circle is referring to Segment, that either company did anything wrong in their response here.
It's possible that CircleCI was one of the 13 breached workspaces:
> For a small subset of customers (13), the unauthorized party was able to gain read-only access to their workspaces and click around in their accounts for up to a few minutes. These customers have been notified.
> Removed privileged access controls internally, adding accounts on an as-needed basis
I believe this is critical. Access controls really should be set as needed, per user, and fine grain. While it can be time consuming and may not even make sense at the beginning of a project, doing everything this way from the start should help to keep an organization better secured in the long run.
"Enforced mandatory Multi-Factor Authentication (MFA) for all employees when accessing Segment-owned workspaces and performing administrative actions in the Segment app."
This should always be step #1, don't wait for an incident.
Really, what step #1 should be is getting all these applications behind an SSO/IDP, and then using the policy controls in the IDP to enforce MFA for users.
We've surveyed startup dir/security's (and the like) and this is almost universally in everyone's top 3, and the leading contender for #1.
Definitely better, I 100% agree.
Yeah I'm surprised more people didn't bring this up. It is a very basic step that is required for a lot of security certifications. I'm actually pretty shocked segment didn't have it in place.
MFA means all employees are now issued a yubikey to login?
Unlikely. TOTP, Duo Push, and SMS are all more popular that U2F/WebAuthn.
Yep. Those are still prone to phishing for a clever attacker and sleepy employee. For most users - that should be fine, but for the ones with the admin access to the nuclear reactor, U2F/FIDO2/WebAuthn probably worth the extra effort.
It is a bit strange, they mention that write keys have been accessed, but they don't seem to have invalidated them, or even recommend to update them?
Information about how Segment customers interact with different aspects of our product,
*including customer write keys for Segment*, integration names, workspace names,
and how customers interact with our user interface.They address this in the post: write keys are considered public (after all, they appear in JS code on customers’ sites).
I was just thinking the same thing! passed it along to our team to change out anyhow
I'm just awaiting the conspiracy theories including both CircleCI and Segment
CircleCI claims their leak happened through a “third-party analytics vendor“. Perhaps that third party was Segment?
Not much of a conspiracy theory needed. Supply chain tends to be easier to attack than high-security firms themselves, and so are valuable targets.
> we took immediate action, disabling and deleting the account that was compromised
Since this was a privileged account, how can they be sure the attacker didn't use said account to setup more ways to get back in? That's the first thing Kevin Mitnick always did when he pwned a box: setup alternative routes to get in, in case his original door got closed.
There is a saying from pentesters about shells: "Two is one and one is none"
Always hack with a safety shell!
for anyone else: https://en.wikipedia.org/wiki/Kevin_Mitnick
Offtopic and apologies, but needing to link to Mitnick makes me feel old haha. When I was cutting my teeth on Slashdot in the late 90's, I feel like there was a "Free Kevin" post every week.
> This data is used by our product, marketing, and customer success teams
It always seems to be the marketing analytics data that's wide open for 'hackers'.
You could spend all day building a secure DB and application architecture then have the marketing team upload analytics for everything onto some random insecure service.
Maybe the marketing/data teams need to get some security lessons as part of their training the way programmers learn?
Segment actually does a pretty fantastic job onboarding new personnel and funneling them through security training. Recently gave a great talk here at the Bay Area OWASP meetup about how they've gamified security awareness training with an internal leaderboard and random CTF challenges.
This is why I’m against third-party analytics in any app/service. Someone compromising the analytics provider can get a lot of data from app end-users even across different apps that use the same analytics service.
This is why you NEVER expose elevated access to the open internet.
Why all the down votes... this is a legitimate security issue.
It's not rooted in reality and a gross oversimplification of the problem. GSuite is exposed to the open internet. Dropbox is exposed to the open internet. All of these services have permission models with "elevated access".
1) Your idea has been shown not to work (it's called perimeter security) 2) With the huge shift to cloud services you need a new solution anyways
Looks like the whole site is down. Anyone save a copy?
Night it be that you employ some ad blocking on network level?
At least my adblocker (browser) did not like what it saw and blocked the article.
Last updated September 5, 2019
What happened?
Between August 26 and August 31, 2019 an unauthorized party compromised a Segment employee’s Segment web application account without their knowledge, logging in with their email and password. This account had privileged access.
Using the employee’s account, the unauthorized party acquired two months of recent data relating to how Segment customers use the Segment product. This includes information about how Segment users interact with our application and associated account information (email address, first and last name, IP address for each session, and Segment write keys). No Segment customer passwords were compromised.
This data is used by our product, marketing, and customer success teams to provide ongoing support for our customers.
When did Segment discover the issue? What did Segment do when it discovered the issue?
We learned about the incident on August 31. Upon detection, we took immediate action, disabling and deleting the account that was compromised. We then began a full investigation to understand and assess the impact of the incident.
What information was involved?
Over the course of our investigation, we learned that the unauthorized party acquired two months of recent data relating to how our customers use their Segment workspaces. This includes:
Information about how Segment customers interact with different aspects of our product, including customer write keys for Segment (which are considered public), integration names, workspace names, and how customers interact with our user interface. Information related to Segment customer accounts including first name, last name, email address and IP address while using the Segment web application. No Segment customer passwords were compromised. For a small subset of customers (13), the unauthorized party was able to gain read-only access to their workspaces and click around in their accounts for up to a few minutes. These customers have been notified. What is Segment doing to ensure this doesn’t happen again?
We have taken immediate action and are continuing to investigate and assess the impact of this incident. Upon discovery, we:
Enforced mandatory Multi-Factor Authentication (MFA) for all employees when accessing Segment-owned workspaces and performing administrative actions in the Segment app. Reset all employee sessions and passwords as a precautionary measure. Notified relevant law enforcement authorities. Removed privileged access controls internally, adding accounts on an as-needed basis. Began an audit of internal access controls to mitigate the risk of an event like this happening in the future. What should I do about this incident?
Unless you have been specifically instructed differently by Segment, there is no direct action required. However, this is a good reminder to make sure that your Segment password is a unique, strong password. We’d also recommend you activate Multi-Factor Authentication (MFA). This blog post covers strong passwords and activating MFA.
We apologize for any inconveniences this incident may cause. If you have any further questions or concerns, please do not hesitate to reach out to us at support@segment.com.
Seeing way too many "incidents" these days ....
I would like at least one company to post an "incident" reveal in a more honest way:
" Due to our carelessness and relatively insecure practices, we had improperly disclosed user accounts to a moderately savvy hacker. We realize this is our fault.
If you'd like to help and given that we have your attention now, it would be valuable if you can help pentest our servers: the attacker used a simple SQL attack based on an unpatched server via CVE-3245. Are we missing anything else? Please let us know.
Thank you."
Yeah except in this case an employee account was likely compromised by spearphishing/social engineering/(or worst case keylogger). That can be very hard to defend against.
Good security is not easy, and not always due to "carelessness".
It's an expensive, onerous, never ending, and ever evolving process to get right. Most, if not all, companies do the bare minimum security they believe is necessary; anything beyond that is a waste of money and computing resources (if you believe otherwise, I have some retina scanners to sell you...)
> Most, if not all, companies do the bare minimum security they believe is necessary; anything beyond that is a waste of money and computing resources
This why we continue to have incidents and vulnerabilities which could have been prevented, or better mitigated. Most often these companies do not even know how to make a correct assessment of their risk. They move forward with this idea that it's a waste of money and resources, yet waste everyone's time with clean-up, or just go out of business as a result. Even with minimal security training and limited curiosity, this incident could have been avoided.
Customers don't want to pay for it. You can easily run yourself out of business building a more secure system. We need to get people and customers to care first to make the economics work.
Plenty of these measures are just basic professionalism. Some are relatively inexpensive (enabling MFA everywhere by default given the number of MFA options.)
Other changes are mildly annoying to developers, ops, and support (e.g. re-requesting production access.) Since developers hold sway in most organizations, convenience often trumps security. None of these measures put anyone out of business.
If I had to attack something I'd go for the limited resources to help smaller organizations scale security appropriately. There are tons of resources for large dev teams, infosec specialists, etc., but there is very little that targets small organizations effectively.
Perhaps carelessness is a strong word and implies incompetence. And perhaps its not incompetence I am complaining about but the rather fastidious and casual way of "move fast, break things, exponential growth" that builds the following without a strong foundation.
> (if you believe otherwise, I have some retina scanners to sell you...)
MFA with a U2F token would go a long long way. Even softU2F as github built might prevent this.
It would help... for now. Like I said though - it will be expensive and onerous for both the employees and the company... and who knows what the next evolution in attacks will be.
No matter how good security gets, attackers will always adapt. Everyone on earth is now using YubiKeys? Now you need a process in place for when people get their keys lost or stolen. Or when your computer doesn't have a USB port. And whatever "I forget my password"-esque process that is will probably be much easier to attack/manipulate/social engineer than the keys themselves would be.
Having to obtain a physical item is substantially harder to automate than credentials stuffing. Especially U2F which is a practical phishing protection and extremely hard to social engineer (you'd need to mail a token somewhere) should IMHO be default for admin interfaces with elevated privileges.
> Between August 26 and August 31, 2019 an unauthorized party compromised a Segment employee’s Segment web application account without their knowledge, logging in with their email and password. This account had privileged access.
They weren't using 2FA, and only enabled after this incident. This is 100% Segment negligence.
2FA doesn't guarantee this incident would not have taken place.
If it's not hardware-based (i.e. Yubikey), you can still spearphish people into putting their username, password, and 2FA token into a honeypot page which would give the attacker a window of unauthorized access.