The Colossal, Monumental Screw Up That Is Marriott Security
danmunro.comI'm sure Marriot had an IDS that created 10000+ alerts per day.
I'm sure they also had a credential rotation policy, hired 3rd party pentesters, had a vulnerability management program, etc
Securing systems is really hard. A lot of the old school recommendations create more issues than they solve, like rotating every database login password every 90 days or so.
>A lot of the old school recommendations create more issues than they solve, like rotating every database login password every 90 days or so.
This is one of the ones that drives me crazy. You can maybe make it work if you have a really good secrets management system, especially if it's hooked into AWS EC2 roles. But having to manually log into servers to change config files/passwords every 90 days is ridiculously disruptive.
> having to manually log into servers to change config files/passwords every 90 days is ridiculously disruptive
Then make it so you don't have to manually log into servers to change files/passwords.
> I'm sure they also had a credential rotation policy, hired 3rd party pentesters, had a vulnerability management program, etc
Why are you so sure? The vast majority of the companies out there is terribly sloppy when it comes to security and does not have any of those. I doubt the effectiveness of credential rotation by the way, that's mostly outdated advice.
Six letter passwords without any complexity requirements are still pretty common ('123456'), as is unsalted MD5 for password hashes, IDS is a term that usually requires explanation and if there has been a 3rd party pentest it usually was long ago.
Some industries are better than other (fintech, medical), but there too you find terrible examples.
I look at another company every week and the state of security at most of them is usually fairly bad with a few exceptions where things are mostly in order. Note that we do not do a security audit, this is just a general look at company affairs and security is only a very small part, just enough to tell whether or not they take it serious or not and how big the risk of an embarrassing hack is, and what the damage would be if one took place.
This article is useless. It says nothing, has no information on what or how the breach happened, and is basically security word salad with a heaping of 'these people are idiots'.
Yeah, I'm fairly sure one of the first news pieces on the attack literally said the security team discovered the breach when "alerted by suspicious activity on their customer database" - sounds a lot like an IDS functionality.
This article proposes buzzword-level security theater. IDS! Rotate certificates and credentials! Have pentests!!
What it fails to mention is: Do not collect data you do not need. You do not need my email. Forcing me to give it to you is bad. You do not need to know my home address. Forcing me to give it to you is bad. You do not need to know my birthday. Forcing me to give it to you is bad. etc pp. The mind boggles why they thought they need to collect passport data!
IDS and certificate rotation are snake oil and security theater. Sure, they usually don't hurt. But here is some good advice:
1. Don't collect the data. If you don't have it, it can't be stolen.
2. Apply all the patches. Immediately. No you don't know better than the vendor. Install all of them. Always. Immediately.
3. No unnecessary dependencies. Yes that means don't go in the cloud.
4. Have an architecture that segregates stuff by security level. Don't put all your things in the same basket unless you are prepared to have the highest security level for all of them. No "this is just a chat server, it is less important than the database" unless those are properly isolated.
5. Minimize your TCB. The less things you have to trust, the better.
And THEN, after all this is done, can we talk about IDS and certificate rotation.
The same way you can throw snark at a company for their (lack of) security knowledge, they can do the same for your lack of industry knowledge.
Generally, gathering passport data for hoteliers is a legal requirement (see here: https://www.quora.com/Why-do-some-countries-require-a-passpo... ).
Now. Agreed. Required to collect v. having available online and hackable for all guests ever is not a best practice, but it's easy to see how a hotel (quite physical-space-intensive, labor-intensive, capital-intensive business) may not have viewed or understood the risks of having this data around.
The last time I stayed in a hotel, there was a car whose window was broken in the parking lot (unfortunately). A crime was committed, on hotel property!
The question businesses are struggling with is: how can they focus on their business and either government or industry can focus on crime-prevention?
I am aware of legal requirements for hotels.
Here's my technical view:
That does not mean you have to _have_ the data. Either forward the customer to a government system where they enter the data, then it's the government's fault. Or do escrow: For example, you could store the data encrypted with a public key of the government. Then only they could decrypt it. If someone stole it, there would be no problem. And the government could still view the data.
My political view is that the government has no business asking hotels to collect passport data, or indeed any data on their customers. This is a blatant privacy and data protection violation. The government does not need to know my location at all times. It's deplorable that things have deteriorated this far already.
Pays to read the original article, it wasn't Marriott it was the company they bought, long before the purchase. Marriott's system was not compromised.
Be that as it may, Marriott now owns it - literally. Just before this news broke, there was evidence of disarray in one or both company's systems in the kerfuffle over the awards system. https://news.ycombinator.com/item?id=18558876
It might be a case of years of technical debt coming due with a vengeance. I wonder if the technical people of either party to the acquisition were aware of the size of the problem beforehand, and if so, what they said about it and how it was received.
I wonder how much IT security will start to play into due diligence efforts in acquisitions in the future. It seems there's an inordinate amount of risk exposure in security matters. Then again, I think Equifax is still a healthy company financially, so maybe not.
That is exactly what due diligence is for. At the very least the potential of this happening should have been priced into the final purchasing cost
When they bought that company it became their system. You can't just hand wave away PCI compliance hehe.
It just might be a more accurate if it was something like, “Marriott bought itself a security nightmare with the Starwood acquisition.” It is certainly Marriott’s problem to deal with now, but there security team might not be as bad as Starwood’s.
I'm an Information Security and Privacy professional, and until there are real penalties for a lack of security nothing will change.
Go see the Ford Pinto case, cheaper to pay lawsuits from deaths than fix the problem, then don't fix the problem.
The other problem is an utter and total lack of technical knowledge by Sr. Management, they hire charming idiots who tell Sr. Management what they want to hear. I've been to conferences and I've listened to discussions from "security professionals" and I'd swear I was at my local supermarket asking people about Information Security and Privacy.
The appearance of security is much more important than actual security. I would gander that is precisely because there is no real penalty outside of anything that would be considered the cost of doing business.
How to enforce punitive action upon a company with such international reach is the real question.
It is only anecdotal: but I had my identity stolen within days of joining the Marriott rewards program in December 2016. I think they may have been getting credit reports on members (which generates a lot more personal data than you gave them) and leaking for quite some time (not just a small number of data breaches).
The way this article talks about IDS sounds, to me, like someone who has never worked with IDS professionally or on any large scale. This goes for other points in the article as well, but that seemed particularly glaring.
I don't intend to defend Marriot, from other coverage it sounds like someone did a very poor job (although not necessarily Marriot itself). But this article also makes things sound far simpler than they are.
My best guess is that the attacker gained access to a database server, and let's say they dumped the contents to a file and exfil'd the file (not always the best way to go, but often the best way to go). Assuming they stole database creds from somewhere else (e.g. some application), that might generate around a half dozen auditable log items on the database server. The retrieval of a large file would be a good opportunity for detection by SIEM content, but without further knowledge of the application it might not be - large file transfers from that machine might be normal as part of e.g. batch processing.
For me, it's hard to say at this point that this would have been easy to catch at all. Perhaps it would have been, but if the attacker was some combination of competent and lucky (combined with the lack of measures like limiting database access rate for applications, which are quite rare in practice), they may have been in and out with very little detectable activity.
> The retrieval of a large file would be a good opportunity for detection by SIEM content, but without further knowledge of the application it might not be - large file transfers from that machine might be normal as part of e.g. batch processing.
Or an eccentric and occult edge case like "backups", especially if it's a database system. Sorry for the snark, but I've had to tell some people the importance of backups for production persistence like a broken record for a week or two.
And sure, you could have IDS rules / firewalls setup to flag or block traffic except to the backup storage hosts and the replication servers and the batch processing servers and the monitoring andso on and so on, flag files, ...
But that stuff is hard, requires a lot of maintenance and adds risk to a lot of critical / stress-powered processes. Change your backup storage at 3 am due to hardware failures? Whoops, the firewall of database host #13 wasn't updated, and now you have no more backups from that host.
Agreed. My first thought was "'Any IDS worth its weight in salt'? That creature doesn't exist, my friend."
OP here, thanks everyone for the interest and discussion in the topic. Awareness and open discussion is going to be the disinfectant our industry needs to improve security hygiene. I have only recently taken blogging seriously and am still working to find my voice and balance between too little information and information overload. I took the feedback here to heart and tried to improve and clarify my ideas and recommendations. Sorry if there's still not much specific information provided, I wanted to keep it at a high level, maybe that was not the best call -- anyway the feedback is very helpful.
Send Marriott an Erasure Request now and maybe next time it will not be your data: https://opt-out.eu/?company=marriott.com#nav
What's "M & M Security"? Linked article never defines the "M"s, I had no luck googling.
It's linked, and it's perimeter based security. I've also known it as egg-based security - once the shell breaks, you've got a big mess on your hands.
Ah, I get it now, thanks. Yes, it was linked, but the linked article didn't explain the metaphor either. I wasn't thinking of the candy, now I get it.
An M&M is a type of candy popular in the US with a hard outer shell and a softer interior. The author is using it metaphorically.
I think it's the author's name for perimeter based security
The analogy is pretty common, although the precise uses varies country to country. I called warm Smartie security, as I'm in the UK :)
Disclaimer: Not defending Marriott, as their Starwood Rewards/Marriott Rewards merger has been demonstrably one of the most epic, public IT integration failures that I've ever personally witnessed as a consumer bystander.
BLUF: I am a huge advocate of companies being fined on the basis of number of people affected and types of data leaked. This incentive to not be fined will be built into the formal or informal risk matrix that a company utilizes for decision making and these types of breaches will decline in number and severity from boneheaded mistakes. In the current model, the only incentive that exists is public embarrassment, but is quickly forgotten despite the incredible disclosures. ( See Equifax )
I know literally nothing about the internal state of their IT department but I suspect a great deal of it is likely outsourced and probably "least cost". From being a long time traveler ( over 1500 nights in Marriotts over the years ) I've seen their payment processing system go down, people remoting into public kiosks and typing plaintext passwords early in the morning, and (not so) hidden pages on their website that were intended for special promotions. As an example, their system that allows one to log into their "internet TV" account in-room to watch netflix will not purge account information at the end of a stay. I've checked in and seen other folks' Netflix splash pages when using the app. ( I always log them out as a courtesy, but suspect that others might not. )
All that being said, it's easy to point fingers and point out failures in hindsight. Every large company/government organization that I've served has similar failures, often not as public, but usually much more serious. In my own experience there is a usually a core contingent of competent tech workers/developers who are aware of the technical debt and attempt to bring it up to management to solve, but get shut down as "there is no reason to spend money on something that isn't driving revenue/mission". The easiest way to solve this would be to introduce fines tiered for the number, type of data, and period of non-disclosure for companies. ( i.e. Equifax breach should have been a historically large fine in this thought. This, while widespread, is not on quite the same plane, sans the passport numbers. ) I'm not a big believer that the federal government is an effective information technology provider, but this falls in the realm of public good, making it a better fit. Structure the organization in a similar fashion to NTSB or the FTA, where case officers lead investigations with teams who have no axe to grind with any particular organization and are screened for non-bias. ( Just the facts, ma'am ) This is currently a role being filled by industry security companies, but I would argue that there has been sufficient bias demonstrated that it should be removed from private industry and put in a public forum. Similar to how the NTSB operates, if an American company has global presence, regardless of the location or nature of the disclosure, the disclosure would be investigated in a similar fashion forensically. ( NTSB investigates airline crashes of American manufactured aircraft regardless of location in the world. ) With the ubiquitous use of syslog data and packet captures that most companies retain, these investigations should be fairly easy to handle; recognizing that in most cases, like airlines crashes, large scale IT failures such as breaches are usually a culmination of a series of failures and bad decisions over time rather than technically sophisticated attackers.
I hope that we start taking the current problems that face our burgeoning technical society a little more seriously rather than engaging in idle political artillery with little outcome for the public good. You know, public good, the thing that government is supposed to ensure through consent of the governed?
I'm interested in a proposed value/penalty for individual and combinations of data. Even if it's a paper napkin approach.
Full name, probably not worth even a penny? Full name plus address plus phone? $0.05? Passport number alone? I have no idea, maybe zero, but full name + address + phone + passport + social? Could that be worth a $2 per instance for a fine?
What about direct compensation for the person whose information is leaked? I've read recommendations people should get new passports because such information can be used to track people's movements across borders https://i94.cbp.dhs.gov/I94/#/history-search
So what if the per instance is really worth $110 (base value to replace the passport)? If 100 million people are affected, that's $11 billion. Not including fine. The Starwood acquisition was $13 billion.
In other words, it could nearly bankrupt the company, if it weren't for the success companies (and markets too, really) have had at shifting the burden of breaches away from the company, an effectively freeloading.
Australia has the "Notifiable Data Breaches Scheme" under its Privacy Act, which requires breaches to be reported to the government[0].
It doesn't have an investigatory/corrective framework like the NTSB (in Australia, the ATSB), but it's the first step towards one.
Australian policy on technology and civil liberties has generally been very poor in the past 2 decades, but the Privacy Act and surrounding policies have been one of the few bright points.
https://www.oaic.gov.au/privacy-law/privacy-act/notifiable-d...
I agree with the idea that fines based on the number of users affected makes a lot of sense. One question I have is how would you propose that number be calculated? In truth, I think the company whose data has been leaked should know exactly how many records have been leaked, but per-individual based fines create an incentive for them to underreport this number. Do you think that’s a problem, and if so, is there a good answer for how society could get an honest answer as to how many individuals are affected in a breach?
> One question I have is how would you propose that number be calculated?
As a percentage of worldwide revenue on a sliding scale.
> In truth, I think the company whose data has been leaked should know exactly how many records have been leaked, but per-individual based fines create an incentive for them to underreport this number.
Very true, so triple damages for wilful underreporting and/or criminal sanctions for individuals.
Idea: Create an incentive to overestimate — if the leaked data shows up online (pastebin/etc), and the volume of affected users is x% greater than the publicly disclosed figure, then fines are doubled (or go up by 3*x% or whatever).
We might already have an example in HIPAA.
When we’re talking about the payment card data that was exposed, I thought there was a mechanism to charge companies on risk. My understanding of PCI DSS is that you have regular audits, and if you fail those, the cost charged by card companies goes up.
IME, you can get away with quite a lot during the audit. You don’t have to be perfect, you just have to have a plan to fix what was found. I would guess that breached app was incorrectly classified as not in scope. PCI audits suck, and so there’s a huge incentive to classify your app/system as not in scope.
Do agree that the fine structure is what will get action. GDPR has raised the interest of making some improvements in how PII is managed.