Klarna users are being signed in to random accounts
twitter.comKlarna is no stranger to criminally lax attitude towards data privacy and security. In Finland, they implemented a checkout flow based only on your SSN (personal ID number). By simply entering someone else's SSN (which is not hard to guess/pry) you can reveal anyone's official home address.
Further, they enable a "pay later by invoice" checkout flow, again by just knowing someone's SSN. Scammers use this to order items from web stores to automated pick-up lockers with someone's else's SSN for payment info. The victim usually only becomes aware about this activity when they start getting debt collection notices for unpaid invoices from multiple stores for thousands and thousands of euros. The debt collection process in Finland is famously unfair and harsh towards the supposed "debtor" (here: victim of fraud).
Unless the "debtor" (victim) actively opposes each and every individual collection, the cases will eventually end up in court with summary judgement. This will ruin the victim's credit rating, which has devastating results for just about all aspects of life. People are known to have collapsed under the burden of all this and ended up taking their own life.
Klarna's response to all this is that they want convenient checkout experience and some fraud is unavoidable. Although there are excellent technical means available to strongly identify users in Finland, they add a minor layer of inconvenience compared to just typing in your SSN. This is OK for Klarna since they give exactly zero fucks about security as long as they can make a little buck from it.
Good to hear, I failed an interview with them at the sixth step or so because I make a point not to remember easily googleable trivia, bet their engineers are traversing rb trees on paper off memory like crazy right now trying to find a solution
In Sweden you can ask them to require Mobilt BankID confirmation to every buy, their competitors (like qliro) don't have that yet so Klarna are only half bastards. But they did get a lot of criticism from the Swedish government about the same things you have presented.
Qliro has that too, which I know since somebody bought shoes with my SSN. I don't know if it's a general feature or if you have to contact them, but the functionality is implemented at least.
I contacted them and they said no, but they have other "smart" ways to prevent fraud
Yeah. Klarna has BankID but it doesn't make them half bastards as they log in to users banks as the user, where they can see all account balances and purchases.
That makes them double bastard.
They what now? That has to be illegal.
they do what? how is that even possible? even if I never coupled my account to them?
It is called open banking. Banks are 'anti-competitive' so some people managed to lobby in a directive that forces banks to provide apis for companies like Klarna to access their bank accounts. All you have to do is accept tos.
Wait, so you have to explicitly ask Klarna to not sell stuff under your payment info to random internet strangers? That's not too good either - I think ordering without BankID should be opt-in rather than opt-out.
I can't remember ever opting in to the Klarna BankID requirement, but when i checked it was enabled on my account.
The actual victim of the fraud is the creditor who was defrauded by the criminal; they are just leveraging the unfair legal system to push the liability onto someone who was not party to the fraud in any way (the person whose name was used in the fraud by the criminal against Klarna).
This is the lie of "identity theft". It's not identity theft, it's money/goods fraud, from a bank that didn't do proper authentication.
And guess who's their own bank?
Klarna \o/
I am not sure this makes sense. Shouldn't Klarna provide proof of the transaction to the court? Won't the court look at it and throw it out as baseless? If Klarna were actually on the hook for their own money, it wold only have to happen a few times before they realize it's not worth it. edit: definitely not a finnish lawyer
Problem is the invoice itself is real. You have to contest it actively to the debt collector and give at least some evidence as to why the debt is invalid. If you do not actively contest the collection, it will soon end up in court. This is a very routine case for a district judge where they will give a default judgement in favor of the plaintiff.
The problem is that the law in Finland is written so that even if the collection is baseless the supposed debtor needs to actively manage it or end up in legal jeopardy. Which is rather unfair if you are a victim of identity theft.
This was probably already common fraud in Finland before Klarna if that’s what the law says - but Klarna would be a crazy force multiplier for the fraudsters and no help for the defrauded, so it becomes a much more pressing issue.
(No knowledge of the details, just speculation based on the discussion here)
Nordics have really poor laws around this. We have these payday loan companies that work under the same principle. Klarna just found a way to conveniently do the same. It is easier to capture buyers at the counter rather than before the counter. Effectively Klarna is a payday loan company in the Nordics and has nothing to do with easy checkout. In fact, they used to offer kickbacks to merchants, so every time someone chose Klarna invoice they would pay the merchant instead of charging for it, because they get so good profit from the individuals due to poor laws.
... and as always, trust works really well, despite the occasional fraud, until someone finds a way to exploit it at industrial scale. And then it fails spectacularly.
> If Klarna were actually on the hook for their own money
Aren't they?
> it wold only have to happen a few times before they realize it's not worth it
That's an assumption. Someone did some A/B testing and said this payment flow results in X% increased sales, resulting in Y% more revenue for Klarna, it results in Z% more fraud. I don't even object to this per-se.
If Y is greater than Z, they will do it.
If they are especially awful they will consider that of the Z% of fraud it will result in A% still being paid for and B% being recovered in debt collection. That is awful.
> results in Z% more fraud
They would probably not measure fraud but just losses directly.
This reminds me of this BBC article I came across: https://www.bbc.co.uk/news/business-55829879
> To use Klarna's pay later service, which defers payments for up to 30 days, shoppers only have to provide a name, email, date of birth, mobile number and billing address.
It’s mind blowing that’s all the information you need to process a payment via Klarna.
The benefits of reduced friction in high-trust societies are incredible sometimes. The failure modes are predictable.
This is one of the reasons I wish governments in the world implement proper digital authentication instead of relying on static identifiers like name, address, or SSN.
The Baltic states have had proper digital authentication for years. Priv/pub key pair on the Xth iteration digital identity card that is checked against your passport physically. The problem isn't that governments don't have proper digital authentication. It's that most countries want to reinvent it every time. The German version is a clusterfuck that they then had to force into existence by mandating it by law and yet normal citizen services can't be done with it.
>The German version is a clusterfuck ...
These gigantic government IT projects are also a good way to funnel taxpayer money to the right pockets, that's why they're always behind schedule and over budget (just like all government physical infrastructure projects) and if you look closely it's always the same 2-3 companies getting all the contracts.
Bullshit. The German electronic ID card wasn't a huge project and it was developed in-house. By all accounts, it works pretty well if you actually have the opportunity to use it. The problem is that nobody supports it. In part because of federalism: You rarely interact with the federal bureaucracy directly and the states for some reason aren't interested in supporting it.
In Singapore they have world class public IT infrastructure and they do it all in house.
Finland offers a state of the art digital authentication system. It's just that Klarna doesn't want to use it because it adds an auhtentication step to their checkout process. It's just easier for them to take the random internet user's word for who they are (!!).
I am not sure how this is even legal under the PSD2 in EU. It might not be. But Klarna does not seem to care, and I really hope someone will take them to court over this.
Is there any pressure in Finland to make this illegal? If your transaction didn't go through the digital authentication to verify identity, then it's worthless and the money can't be collected?
Yes, PSD2 (Payment Services Directive vol 2) should require strong customer authentication for online payments throughout EU. How Klarna is able to skirt this regulation is beyond me. Either they've found a loophole in the law or they are already in breach but the financial regulators are holding back from enforcing it.
https://en.wikipedia.org/wiki/Payment_Services_Directive#Rev...
I'm somewhat happy that my country is so much behind on all this digital stuff. You usually have to physically present your ID to do something serious, or at least provide a picture of it. We do also have an official "government services" website, and it implements a proper oauth flow that many other government sites use and uses SSN + password for login.
At a large site I used to work for circa 2011, before everyone had gone fully HTTPS, we received similar panicked reports from users: "I'm logged in as someone else!" Turns out an ISP in the Philippines decided to just ignore `cache-control` and `vary` headers and forcibly started caching logged-in responses along with auth cookies. Bad times. Made it clear to me why the whole web would have to go HTTPS.
Yeah but what about the saved traffic? Think of the poor routers that have to do all this transferring job.
Reminds me of a primitive web filter at work that blocked me from something. So I look at the URL, add an “s” to http and voila. I think they MITM everything now.
I worked in a project over 10 years ago where something very similar happened!
We had built and authentication service that, among other things, was used by a SyncML service that was used back in the day of feature phones to syncs contacts etc. You can imagine that getting someone else's contacts on your phone isn't exactly ideal. This was how we came to know about the problem, from customers getting other customers data!
The error was caused by a CDN switch. Our instructions to the the CDN team responsible for the switch was "Make sure the CDN honors our cache headers, if our HTTP responses say something can be cached do so, if they say that the response should not be cached then don't". We were in at least three meetings where we repeated this mantra.
I believe that the CDN team thought that they had setup the CDN correctly but they had missed an edge case. The CDN was in fact setup to cache even uncacheable responses, and served those, _only_ when it could not reach our servers.
So if there was a traffic spike and the CDN determined that our authentication servers were unreachable it would fall back to serving data that should never have been cached in the first place! Happily returning tokens to random users that had authenticated just before the traffic spike...
Something similar happened a few years ago in Norway, when the yearly tax returns were released. Everyone of course logs in at the same time. It goes down, and the cache serves someone else's data instead.
Happened for the danish tax authority about 10 years ago as well. Although I think the issue for them was that the unique login token was based on a timestamp that several users happened to share during very busy peaks.
I would expect this to happen if an option in the line of "serve stale content if target server is unreachable" is enabled.
Yes, you are right!
Ouch.
Klarna is a weird company. Last I interacted with them it was clear that they are completely designed to operate within Sweden, but have no idea of how to deal with the outside world. Maybe that have changed.
I talked to Klarna maybe 10 years ago. One of the things I wanted to know was how they dealt with abuse in Sweden, given you just need the social security number of a person and then you can do purchase as that person, and Swedish SSNs are not secret.
The friendly Klarna rep. had no idea what I meant, as you could only get stuff delivered to the address associated with the SSN. Based on how that would be abused in Denmark we suggested ordering a box of random sex toys to any random person in Sweden. The only answer I got was "Why would anyone do that?"
It took less than six month for Klarna to start asking us to block addresses, because they had no way to prevent abuse.
> "Why would anyone do that?"
That's such a typical Swedish answer... but they do allow (but not as default!) to block orders and request digital confirmation
Of course, Sweden's largest export are lessons about morality.
Interesting for a country that slowly eradicates their indigenous people btw.
https://en.wikipedia.org/wiki/S%C3%A1mi_people#Discriminatio...
https://en.wikipedia.org/wiki/Swedification#Swedification_of...
They also have no problems sending people to be tortured.
“Also, the United Nations Committee against Torture condemned Sweden for having violated the principle of non refoulement provided in Article 3 of the Convention against Torture.”
https://www.cairn.info/revue-internationale-de-droit-penal-2...
> Of course, Sweden's largest export are lessons about morality
Not sure why you were downvoted. I think your comment is rather fair. :) Although, let's give them some credit, they do have a pretty successful mixture of socialism and capitalism.
> Interesting for a country that slowly eradicates their indigenous people btw.
I think that sad story is over. They significantly ramped up protection for indigenous people.
Yes, Sweden deserves quite some credit. It doesn't have a clear track record on human rights though, as it is trying to suggest on the geopolitical platform.
The conflict is imo not over, it is still going on. If it wasn't then Sweden couldn't produce 90% of all iron in Europe because the mine happens to be on Sami land.
> they do have a pretty successful mixture of socialism and capitalism
Used to, they’re broke now.
we are?
Yes, please examine your nation’s cash situation in 2010 and compare to today. You’re broke, Sven.
Sweden Foreign Exchange Reserves?
https://www.ceicdata.com/en/indicator/sweden/foreign-exchang...
No, but you’re getting warm!
Isn't this how post order used to work? You just send a pre-printed form to the company and fill in the address and name? However, with computers automated scams are instant and could have a greater scale. I.e. instead of having some random person have a delivery pizza, you could order 1000 pizzas in 1000 towns.
I suspect that depends on where in the world.
In Sweden, the classic "order by post" required paying at the post office as you picked your parcel up, with only the pick-up slip (with the total to pay) being delivered to the address.
I have seen a few Swedish companies who didn't use the Swedish postal order system, instead opting to send a package with a giro slip and an "pay within X days" inside.
My understanding is that the post office took a small cut of the postal order payment, as a fee for guaranteeing payment. And the companies instead sending a giro slip had enough compliance with paying that it netted more money.
Sometime people pretend that it doesn’t make sense. Had a real-estate say offers have to be a signed contract I say it needs a expiry period so people don’t come back in a month and demand the contract is followed. He kept doubling down on “I don’t understand the problem”
I believe the user id actually was the email address when they started out.
typiskt danskt att skicka massa knullsaker
Naive, dejlige svenskere.
I'm just guessing, but...
"developer gets a great idea - let's push an update to the API as a GET request so we can cache this on the CDN... forgetting that the JWT token is potentially returned in the call. Now, whoever makes the call first gets their JWT token stored for everyone else to load instead when the API call is made."
Ta-da, Klarna.
I worked with a team that owned a service that resizes images. An engineer was assigned a task to add support for auto rotating images. His solution involved saving the image to a file and then using a library to handle the rotation. He used a hardcoded value for the file name. In a local environment where requests are sparse this looked fine to him and other engineers on the team missed it in code reviews. It wasn't until it went out to prod that he realized the error in this. Users started seeing other users' images because the file's content was constantly being overwritten.
When you test features like this or caching a response with a JWT it can be very easy to default to the happy path or ignore the impact of a large volume of concurrent users.
"An engineer was assigned"
Nope. That definitely wasn't an engineer.
Mistakes happen. I've never met an engineer who has never made a mistake. However, I have met brilliant engineers who have written incredibly complex software and have also managed to make some silly mistakes along the way.
No true Scottish engineer would have made that error!
:-)
Real software engineers don't make mistakes?
Not mistakes like that.
Elitism alert!
Years ago I added varnish in front of a website to cache image requests, not realizing that if the response included 'set-cookie' that was also cached.
We immediately started getting reports of random products appearing in our customers' shopping carts, as people's sessions got merged with random strangers.
Just feel the urge to point out that Varnish by default do specifically not cache requests with a set-cookie header. :)
I expect something exactly like this happened. I had a similar bug long time ago. Apache was somehow incorrectly caching the request and the session cookie in the request ended up in a cache. But it happened only about 1/10,000th of the time so it was impossible to figure out the root cause.
However, one common source for this kind of bugs is to ”cache any URL ending .pdf as a static file” and then you are in fact serving logged in PDFs like customer invoices that come with the session cookie.
I think CloudFlare used to come with a default rule to treat .pdf as a static content. The responses were cached when you hit their ”cache the good stuff” checkbox.
I doubt that Klarna, a bank, have OSI layer 7 proxies in the cloud, with TLS termination in their CDN solution, on AWS. I would assume this traffic is outside of that. But then again, I know they wasted 25M+ Euros on a garbage NodeJS platform. They also created an own cloud once. Yes, it is in the trash bin.
I'd actually bet against you on that one. They are still stuck with one foot in the startup mindset.
Surprisingly many IT companies tried to create their own clouds, or at least their own kubernetes.
Surprisingly many have saved boatloads of time automating processes pertaining to the tasks at hand. So, yeah, sound reasonable. :)
They didn’t “create” their own cloud - they wanted to host their own hardware using an api layer to provision resources. That stuff was not built in-house.
Manhandled in-house though...
Sebastian used the word cloud when I met him.
Yeah, I actually took part in setting it up with them. It was CloudStack. API layer in front of hypervisors.
Such is the cloud software... :) Cloud, besides APIs, i.e managing hardware at scale was not really what they did.
They did roll 1000s of vms per week through it in ci/cd flows though.
As such it did what it was supposed to do - docker/containers was not a thing at that point in time, and I remember thinking it was pretty awesome.
To many nifty engineers, with long fingers, for their own good though. You need to be strict with automations if you want to keep something like that running reliably over time.
There was a Klarna cloud yes. At the time it was unclear if finance/banks could utilise public cloud services (regulatory requirements), so it made sense in that way, but creating your own cloud is something few orgs are capable of.
It is not that uncommon among finance tech companies in Sweden to have some in house cloud. They often already have the knowledge to run servers with virtualization, logging, backups, redundancy and so on. Adding a service layer to that by using Kubernetes for instance is doable.
Klarna Cloud was a deployment of Cloudstack or Openstack (my memory fails me now) for internal usage, when there was still a lot of discussions around cloud lock-in, it was not an in-house built cloud platform.
I did not think they actually wrote the code. But I think the ambition was higher. Pretty much every CTO in this country have hubris and think their services will be sold to third parties.
What makes you doubt that?
I can 100% see this being the cause if this comes out as the root.
But... API's really shouldn't be cached? At least not at the CDN level. The risk of serving up stale dashboard data alone makes users go ????... and we definitely don't want - not even mentioning the problem here, that's crazy.
100% agree with this. A database is, in some form, a cache of its own. If you have to add additional cache on top, it's an additional source of complexity and risk. If you are building a financial platform, you should DESIGN around this.
Depends on the scope of the API of course, but it's a good rule of thumb for any API with private auth
Of course you can cache it, but your assuming it should never. Nothing wrong with caching API calls on the CDN forever as long as your purge the cache once you need it. Event based purging.
"There are only two hard things in Computer Science: cache invalidation and naming things."
Cache invalidation is always a very tricky affair. It can work for a while but as complexity grows it gets very hard to maintain and debug. It's very much a "here be dragons" situation and you have to go into it with your guard up.
I was at a small startup that had a quick and dirty contractor built API. It worked, but for our largest customers, 99th percentile latency started going over the API gateway timeout. The quick and dirty hack on top of it was aggressive caching with too-clever invalidation logic. It worked until new features were added and then it started failing dramatically and unpredictably. The bugs were an absolute nightmare. We ended up spending almost a year cleaning up the data model, sharding things by customer, and fixing a bunch of N+1 queries, all so that we could get rid of our API cache layer and kill the bugs for good.
This reminds me -
A couple of years back, I was making https://lifeboxhq.com which involved users uploading quite a bit of content. I was happily testing security with some url resource enumeration and for some reason, I could non-deterministically access user uploads via url, even on accounts I didn't own. I spent several days looking at my Flask code, javascript, etc. to debug....
I knew it wasn't my code, but I was getting more and more frustrated, then I remembered I set up Cloudflare....
Remember to exclude certain routes from Cloudflare if you want to avoid arbitrary user content from being cached without authentication.
I introduced a similar bug into one of my products in the past (Be honest, who hasn't?). But I'm surprised here because Klarna is a quite mature product and something like this shouldn't really happen at that stage.
Oh, it can definitely happen even in mature products. One I worked on had pretty much the same issue as Klarna (people seeing others' info) when someone updated a web client library we were using to a new version that subtly changed how it handled concurrency.
I remember something similar when there was a load balancing issue with some website where it would randomly assigning a user with someone else's account.
To get around this, one could include the request IP address in the JWT and required a refresh token to be sent when the user's IP switches.
This is not a safe method for protecting against this type of cache vulnerability. IP addresses are regularly shared by multiple users, especially when behind NAT (even mobile ISPs are doing carrier grade NAT these days).
So there should be no fail safe since it can't be guaranteed to work in every scenario.
In this context, this would just prevent everybody from logging in. The JWT would correctly get rejected but people would still be getting the wrong token from the CDN over and over.
Which would you rather? The situation you just described or users accidentally spoofing each other's session?
Ha, one time I was debugging an issue that only happened to a particular user. Lazy as I was, I hardcoded his auth token in the code "just to test". Having found the bug quickly, I was excited and did not realize I checked-in the auth token too. Bypassed reviews, pushed to prod and then reports started coming in "Hey, users are saying they are all logged in to this random guy's account".
Lessons learned the hard way ;)
Did you compensate the victim of your personal and corporate negligence?
Just out of curiosity. Is it a bad question for some reason or why is it downvoted?
I didn't downvote it but the tone does seem quite adversarial to me.
You could ask "what was the fallout?", "Did the client get compensated?", or "did you make procedural changes afterwards?" without being as confrontational.
Less people will post about their mistakes if they know they're going to be lambasted for them. We can learn from these mistakes if they're shared or they can be a timely reminder of stakes if we're slipping into complacency. So we probably don't want to discourage such posts (especially since it's rarer for people to want to talk about failures that successes).
Here's their official statement:
https://www.klarna.com/uk/blog/written-statement-on-app-bug/
Although I dunno about "According to GDPR standards, only non-sensitive data was exposed." since in the twitter thread someone said:
This is definitely not a test environment. I was called by someone who was logged in to my account and saw all my personal data including bank details, Klarna card etc.
And while I'm told the bank details are obfuscated (I don't use Klarna, I dunno), I would consider the phone number to be a clear breach of my privacy under GDPR.
Although, the twitter account that said that has 0 followers, so maybe its not true. I dunno. I know someone who works for Klarna and he told me: "Full investigation will take time. There's a LOT of engineers working on this. Only confirmation I have currently is that the firstname was visible."
Going by the screenshots, first name and account balance. Doesn't seem that bad from a GDPR point of view. Still bad, of course, but not suuuper sensitive.
EDIT: Nevermind: https://twitter.com/esraefe/status/1397843949985931265
And this is both maddening AND make the problem worse (from the CEO):
"We are truly sorry for any inconvenience..."
Oof, yes, its not about inconvenience...
As a software engineer, I hate when I add a check for something "that will never happen" but that if happens is awful, and people complain.
A classic example: you need to get a user from a session, check against a database, and continue if they're signed in.
Then I add a simple if databaseUser.Username != form.Username and people will say "if that happens we've something worse wrong". Geez, something might be wrong and such double checking might provide to be useful.
On a smaller scale, bits flip due to cosmic rays and so on. Of course, there must be a limit where we stop, but people are used to actively avoid doing such "silly assertions" even for important steps.
¯\_(ツ)_/¯
A lifetime ago I was writing code for airline data processing. The specs are very clear about what the valid representation of every field was (less so about what they meant, but...).
So we generated our parser to fail if field ORG/1457 (made up) was not numeric max 8 digits. Or missing where mandatory.
Even if we never touched the data in that field.
Turns out that no-one else used the spec that way. No two were the same, so we had to basically implement two layers of parsing. One to put the data in a common parse tree, and the other to per-sending-mainframe interpret the data as how the sender had implemented.
We assumed that the mainframe would never send illformed data, and indeed that-could-never-happen. But they differed in what they thought was well formed.
Klarna uses erlang and erlang enforces exhaustive conditionals and pattern matching so the error / unsupported case is always handled.
I like defensive programming. Even though I think the state is unreachable, it feels nice to add a panic assert just in case.
If it is due to cache, then extra check like you described probably would not help.
Most people I met who do double checks would simply return "not loggen in" and issue a WARN deep within the other 200 WARNs-per-second. That is IMHO a very bad usage of double checks. It gives a false sense of security and masks the deeper problem until it's too late.
However, if you make the assertion fail loud, then it provides an additional security layer and should be used as often as makes sense.
This is very good practice as far as I'm concerned. Functions should treat their arguments as potentially hostile input.
maybe if it helps to fail fast and only public functions
I think there's merit in objecting to "that will never happen" checks in some cases (though, to be clear, I'm not saying the people objecting to your code are thinking about the same thing I am).
Specifically, if you have data that is loaded from some other source, your extra safety check might be checking data that's loaded from the same source, in a way where if something did go wrong, it went wrong in both places you're checking.
In this case, it seems pretty unlikely that Klarna's bug was that they ran "SELECT * FROM users WHERE Username = 'joeuser'" and they got back a row where Username != 'joeuser'. I don't think there's a recorded case of that ever happening with databases.
However, it seems much more likely that Klarna's bug was in HTTP caching or something, that results were returned for the wrong user. Then there's no opportunity to see databaseUser.Username != form.Username: that check would have indicated that things are correct, but the username being passed into this code was wrong in the first place. That sort of problem definitely happens in the wild - see the "Kenneth" story elsewhere in these comments, or off the top of my head https://blog.zulip.com/2021/03/20/zulip-cloud-security-incid... from two months ago.
And if it is, somehow, a database bug, why do you trust the database at that point? What if the database returns part of one row and part of another? What if it returns the username you sent in because of some optimization to avoid copying data, but thanks to a bug (or a cosmic ray) it reads in the rest of the data from an unrelated row? In the unlikely but not totally impossible case that you need to protect yourself against this, validating the username isn't enough; you'd better sign the entire database row and validate the signature before trying to use any of the data that's been returned. (And come up with some reason why you trust your own app code more than the database.)
The problem with such "silly assertions" is that they make you feel like you've added test coverage, when the thing you're testing is something like a database that is extensively tested by its vendor and by everyone else using the database, and there are other seams in your code which are much more likely to break. Meanwhile, they make the code longer and harder to read, which prevents readers of the code from easily identifying what those seams are.
(And by slowing down the API endpoint that talks to the database, it motivates other developers to try to put some caching in front of that endpoint, which may actually cause this sort of problem!)
> I don't think there's a recorded case of that ever happening with databases. > and there are other seams in your code which are much more likely to break.
One such thing is the abuse of layers and layers of abstractions. For example, many people (unfortunately, in my view) love to use ORMs and query builds, and things like these are much more easier to happen when things are too generic.
And signing the entire database row and validating it, and so on, might be unjustified for most people, especially if you already count with correction from a TLS layer, and you can just have the trade-off of adding a simple conditional to check if the data you receive is sane.
This is not something essential for everything, but that is nice to have, especially the further you're out of control.
For example, if you retrieve data from an external API you should not trust it blindly, but rely on your internal references (security concerns aside, I'm talking about other kind of erratic behavior or bad data).
I agree, and I've also been called out for doing "stupid" defensive assertions. It's almost certainly not a code-level issue this time though. This whole thing reeks of infrastructure/caching issues.
it's fine to make the check but I hope you don't sweep it under the rug with an early out without at least logging the occurrence
uh? Why would you make the check, find a critical internal inconsistency, and skip logging it? :)
log("this should never happen")
"The payment giant Klarna, which has 87 million customers globally, currently has major technical problems. Users of the company's app saw other customers' payments and personal information, before it was shut down completely.
The supervisory authority Finansinspektionen, FI, has asked Klarna to explain what happened."
A future, fascinating post-mortem I hope!
Happened to Steam in 2015. In their case it was a caching issue.
https://old.reddit.com/r/Steam/comments/3y7lxm/when_i_go_to_...
https://www.forbes.com/sites/insertcoin/2015/12/25/steam-is-...
Once you logged in once Klarna stores your credentials and then presents you one click buying inside ads in unrelated sites (well Klarna are not doing the advertisements but allow such links).
You can then accidentally click the wrong thing and buy without any further confirmation. At least in Sweden you can ask them to request digital ID confirmation for each buy.
With the current problem maybe I can buy using someone else's name...
Klarna has posted a statement here https://www.klarna.com/uk/blog/written-statement-on-app-bug/
In their statement they deny accessing bank details:
> The bug led to random user data being exposed to the wrong user when accessing our user interfaces. It is important to note that the access to data has been entirely random and not showing any data containing card or bank details (obfuscated data was visible). This means that it has been impossible to access a specific user’s data.
This is not the experience of the user in the OP: https://twitter.com/esraefe/status/1397843949985931265
I believe it is the case, that when you see your stored payment method is is obfuscated such that it only reveals the last 4-5 digits. Same with bank details as far as I know.
However, showing the card issuer/bank + the final 4 or 5 digits of an account or card number is still extremely distressing. There are some services and vectors out there that can be engineered with just that information for sure.
Combine that with possibly exposed address, telephone number, and you are in very dangerous territory.
It might be accurate if you are internally discussing PCI compliance.
However, to the layperson, "bank details" definitely includes name of bank and last 4 digits of account number. It does come across as deceptive to use that terminology to respond to customer complaints.
> affected up to 0.1%, approximately 90 000, of our users
Does this mean the bug affected .1% of accounts or that only .1% logged in during the 31 minute window when the bug was present?
> It’s concluded that a human error caused the bug
I would not want to be that "human" atm
Reminds me of this story after an expensive mistake:
> Boss - "Why do you think you are here, Jack?"
> JW - "I expect I am here so you can fire me"
> Boss - "I just spent a million dollars on your education - why would I fire you now?"
http://www.nickmilton.com/2016/03/jack-welch-on-learning-fro...
Yesterday's Money Stuff has a good discussion in this vein:
"A somewhat tongue-in-cheek but surprisingly useful maxim of high finance is that it is good for your career if you lose a billion dollars. I mean, if you lose a billion dollars for your employer you will probably be fired, though that depends on who your employer is and how much money you started with and what you did to lose it. But lots of other employers will be excited to hire you, once they learn that you lost a billion dollars for someone else."
https://www.bloomberg.com/opinion/articles/2021-05-26/exxon-...
There is a Dilbert comic about this.
That is definitely not the case in EU software industry. I have seen technical leads get fired for a bug which caused 200k EUR loss. With team setups and documentation no one is irreplaceable.
A good practice is that once a change passes code review and ships, the team owns it.
Human error doesn't mean blame the human, it's better to look at the overall processes and system to figure out how to prevent human error the next time around.
Almost certainly isn't a single human unless their governance model is atrocious
They mentioned human error. I could feel bad for the human who error-ed, but I wonder what kind of human error could have this huge impact.
It could be something to do with cache configuration.
Reminds me of what happened to Steam a few years ago https://www.youtube.com/watch?v=dkSslseq9Y8
This also happened to GitHub recently, although limited. https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
IMO this sounds more similar to the steam issue as it's probably cache related. The GitHub issue was far more subtle.
I remember when this happened to Apple with iTunes Connect (where developers submit apps for the App Store) back in the day:
https://techcrunch.com/2015/01/29/itunes-connect-issue-loggi...
Also to Italian Social Security agency last year (anyone surprised the site was built and maintained by a big ITC company?)
It also happened to Chase (!!) a few years back.
I had once contact with Klarna. It required me eight weeks to teach until they accepted the truth - I didn't owed them a cent. Just one of the usual startups around outsourcing, minimum wage and avoiding actual work.
Lesson 1: If someone want to sell you something and doesn't want make the bookkeeping itself, avoid them.
Lesson 2: In doubt? Cash only.
I can understand avoiding a company due to a bad experience but that sounds like a rather general and rather restrictive conclusion. Did you mean bookeeping specifically, or payment handler, as they are somewhat different things?
For small businesses using a payment processor removes a massive barrier to market entry. Many small business hire real world external accountants to do their bookkeeping, so "avoiding actual work", would you avoid them as well? I do some work with the accounting & invoicing teams in our corporation and there is a LOT to take into account that would cripple a startup with only a handful of employees.
Bigger companies use services like Klarna not because they can't (often they have other payment methods as well and do their own bookkeeping), it's because customers like to use them and failing to use something like Klarna means their customers will shop elsewhere.
> it's because customers like to use them and failing to use something like Klarna means their customers will shop elsewhere.
Also, some customers do not have personnummer and find Klarna to be one of few payments methods that will reliably let them shop online.
Using a real payment processor for credit cards is not a problem. Like it was around 2005, when most merchants had not settled on Klarna and similar.
Some merchants sell via Klarna to private customers and via invoice or pre-payment to a proper bank account to business customers.
Private customers are second class. No business would deal with this nonsense.
Klarna says they are “experiencing technical disturbances due to technical errors”.
Sounds like a poltergeist.
In Norway, we call this class of error "a Kenneth", after everyone who logged in to see their tax return in 2012 received the tax return of a guy named Kenneth. The culprit was allegedly a misconfigured load balancer.
I once got a credit card statement that told me I would be able to pay off my credit card in 100,000 years.
It was discouraging.
Once a colleague made an accounting error and it showed that we're in debt something like 100 million.... I told him to stay calm and relax, we don't have that kind of money so why worry :)
"If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem"
Klarna wants to be Facebook of payment. When I buy and pay with Klarna, they get the list of items and on Klarna's app and homepage I see pictures of whatever it is I bought.
I'm not sure what to think about this. My first thought is "Is this really legal?".
Way to make me run away from them fast.
Lots of times when I’ve been buying things in e-shops I’ve been offered to pay using Klarna as a payment broker.
But doing so has always been more confusing for me compared to “regular” payments with a credit card anywhere else, and has on overall been a negative experience for me.
I really don’t understand why anyone would prefer to use them at all.
What am I missing? Can anyone help me understand?
They incentivize e-tailers by offering higher conversion rates(later) as well as taking the hit for fraudulent payments (often with regular CC billing an e-tailer can be liable for repayments) in exchange for a slightly higher percentage.
Once someone comes to their checkout they hide or at least make the direct payment options well hidden so that by default people buy by taking credit with them.
This credit often comes with shorter than industry standard payment terms so people end up missing payment and being handed over to their in-house collection agency that starts collecting overdue fees.
It's considered digitalized loansharking by many for a good reason.
Ironically it seems that for many smaller e-tailers using Klarna as the payment option seems to heighten the trust of customers so they're more likely to buy (my guess is that we've all been told or told people historically not to enter CC details on random sites and even with stuff like 3D-secure these days everyone is wary)
Klarna is really shady. It encourages a 'buy now, pay later' mentality, which may be convenient right there and then, but it creates an unhealthy style of shopping:
https://www.theguardian.com/money/2018/nov/17/klarna-buy-now...
It's Payday Loans 2.0.
It's really disturbing to see Klarna as a payment option in many Dutch online shops. These always already have iDEAL (which the vast majority of customers use), a convenient way of doing an electronic bank transfer; and most shops support credit cards too.
Not sure where you are located, but in Sweden, Klarna at the start (if I remember it correctly) only needed your 'personnummer'(social security number) to process payments.
Now I think they manage to track your devices so I only have to enter my postal code, and then I just click purchase, and it's all done.
They used to use really weird/dark patterns, to make you forget to pay and then pay huge fees to Klarna.
Nowadays as I've configured Klarna, it just subtracts the amount from my bank account, hassle free, and I don't have to do a bunch of reserach wether or not the website is credible.
Somewhat like Paypal, but smooother.
Sounds bad. I like my online payments to have a little friction.
Here we can just scan a QR code and confirm payment. No extraneous middle men involved.
https://en.wikipedia.org/wiki/Short_Payment_Descriptor
I don't see how credibility of website depends on what payment options they offer. That sounds like a separate issue.
And they don't notify you when it's available for payment, or when it's about to expire. So if you order something, and it for some reason takes a few weeks/months before they ship it and it becomes available for payment, you'll end up with a reminder fee with no warning.
Hm? I always get an e-mail when the invoice is ready and another e-mail when the payment has been received.
iDeal is smooth enough. Hoping this dystopian future does not come to the rest of the EU.
Ye Klarna was really scammy early. Making their living on reminder fees.
Klarna seems to be super popular with e-commerce sites, my guess is that there's some kind of financial incentive to the hosting site, when compared to other payment options.
As to why it's popular with consumers looking at https://www.klarna.com/uk/smoooth/ , seems like they're offering months of interest free credit and also the implication is that using Klarna doesn't affect credit score.
It'd be interesting to know how their credit risk setup works.
Klarna is very easy to use. They take a large part of the risk. The seller typically sells the purchases to them.
It may be different in different countries but the thing with the interest free credit is that once you don't pay on time it is converted into a revolving credit with high interest rate and something like a 60 months payment plan.
Klarna have also historically made up own names for fees to circumvent regulations for regulated fees. They were among the first to remove days of grace and among the first to use a fixed number of days from purchase to due date.
In Sweden you can use them to buy with an invoice which is a lot quicker than entering your credit card. That is probably the main selling point.
Sellers get paid immediately and they take care of making sure the customer pays.
With Klarna you just need to type some information most people know by heart (10 digit ID number, f.ex.) before the order is confirmed. This is convenient if one wants to buy something quickly from a mobile phone. The address will oftenalso be prefilled. A credit card number is much more cumbersome to type on a small device, and the address needs to be typed in separately.
Some banks used to require people to log into their banks to temporarily unlock their card for internet shopping, or, nowadays, one also needs to authenticate the purchase with the bank. That adds extra friction.
With Klarna one does not need to pay until 14 days after the goods are shipped. Credit cards are even better, but most people tend to just have a debit card. With Klarna they don't need to worry about spending too much money from the account and having some other payment bounce later on.
I personally stopped using them after I fell for one of their dark patterns and bought something on credit, which incurred an extra fee. Legally I was entitled to cancel the credit purchase, pay the full amount and avoid the fee; but I was still annoyed.
They usually offer deferred payment via invoice, removing the need to input CC details at the time of purchase. I've used that a couple of times, just because I wanted to move on to other tasks.
(Not claiming it's a killer feature, but it's a feature.)
They were also doing short-term loans [ https://www.bbc.com/news/business-56343942 ], which were for a while being pushed quite heavily in some internet community things I'm a part of.
Also, I figure they must be paying a lot of money to be the default payment provider on so many services.
For me, asking for my bank login details is...ridiculous - it'll be interesting to see if it is still following the same tactics in a few years.
> asking for my bank login details is...ridiculous
Is there more information on this? Are they doing the same thing Plaid does in the US? That is, literally asking for user credentials to internet banking instead of using the banks' proper APIs?
I don't know about current Klarna, but they took over Sofortüberweisung, which has been doing exactly that since 2004. Avoided them like the plague ever since.
They are (or at least were, haven't checked in a while) asking for private bank account access details, yeah - https://www.reddit.com/r/germany/comments/bweqaa/is_it_safe_...
Same experience here. I assume it works better in Sweden, but I have no idea why someone with customers outside Sweden would want to use this.
One e-shop I use regularly switched to Klarna and the whole checkout experience got much worse. Simple forms replaced by broken interactive ones, etc. It's still not better than the old UX, even after multiple iterations. I'm more reluctant to enter CC info than before, for what that's worth.
I think the main selling point is being able to buy clothes and return the ones that don't fit without having to pay for the lot first, and get a refund later.
The same can be applied to a credit card though so it's not a strong argument
You can get an invoice that you will pay later. Thus, you don't need to look for your credit card at the time of the purchase.
You can choose to pay at a later date
You can choose to split the payment into installments
When I buy things (Sweden) it’s basically one-click checkout with just the e-ID signing to pay directly from my account, not via card. Definitely convenient.
I've had this happen, although not on a scale as this, when implementing caching and disregarding authentication as a parameter that varies the cache...
Happened to Valve too, Christmas 2015: https://arstechnica.com/gaming/2015/12/valve-explains-ddos-i...
MS Exchange outlook web interface sometimes showed me completely unrelated mailbox content upon login: folders, list of messages, read status, subjects, etc. Trying to open email never worked though and the whole problem goes away after page refresh.
Sometimes I see my own mails before logging in for a short while in Outlook web app. They have some issues.
I like how when the session expires and you login again you get redirected to the random resource your browser requested when it just expired. So instead of the mail view you sometimes get the new mail jingle or some minified js. Makes me feel better about my own imperfect software
Caching and Vary headers can be tricky to get right
Yup, when you get it wrong you get to meet interesting people from the compliance department. The "enable cache" button in the load balancers should come with alot of warnings.
Probably caching set incorrectly. happened with steam years ago - https://www.bleepingcomputer.com/news/security/steam-caching...
Time for a blameless postmortem
https://www.atlassian.com/incident-management/postmortem/bla...
Or perhaps not
https://techbeacon.com/app-dev-testing/blameless-postmortems...
from this event... game idea:
create a social media site - allow postings, conversations, threads, etc.
Every quarter (or some other period), there is "reconning". You are placed into a complete stranger's account, and now you have to continue it for a week (or some other short period).
Whoever can maintain the quality of the account, in the direction as the original owner, wins a banana (or kumquat, something good but not expensive for anyone).
After reconning period, owner returns and judges. None-participation is default no-win.
Their German counterpart, Sofortüberweisung, didn't properly blacklist test credentials given out by banks e.g. to developers in the beginning, so people could simply use those and pay for goods and services with fake accounts.
For me there are so many red flags with all these services, as they basically "steal" your credentials to log into your online banking. And while they claim that they only use the credentials to make transfers they could as well look at all my other account data. I really wonder how such a scheme can be legal and how banks can allow this, as they normally tell people to never give their credentials to anyone. The situation of course recently improved with the mandated 2FA for logins and transfers, but still there are so many attack vectors in this model that it boggles my mind how it can still exist.
Hear hear, I used Klarna (not by choice) and it surprised me they would feign being me in interactions with my bank. Exactly the type of behavior techies are trying to teach the older generations to NOT fall for.
With this, we know that Klarna's software quality is papier-mâché level. I am happy I refused to let Klarna have my account authorization.
There have been some weird legal cases in Sweden where businesses and scammers have been freed after having signed in using other people's "BankID" to change retirement savings around or send cash.
Its the ID method I use for credits, pharmacies, health care, taxes, but was apparently not an ID so it's not id-hijacking.
Klarna has man in the middled my bank account before and performed a purchase and I've boycotting any company having them as the only payment option since.
OH, now I also remember Klarna adding credit in my name since they only needed my tax registered adress. I lived in a dorm so someone just used our public information to take out credits to order sneakers and could break into the crappy entry mailbox.
> There have been some weird legal cases in Sweden where businesses and scammers have been freed after having signed in using other people's "BankID" to change retirement savings around or send cash.
As far as I know most, if not all, of these scams have been perpetrated against the elderly. All operations (authentication, signing) can be initiated remotely with just a personal ID number, so the typical scam meant calling up someone and claiming that "an authentication must be performed", and simultanously initiating a bank login session. If you can keep the victim on the phone and using the BankID app when you tell them, you could basically login and empty their bank accounts. This has been largely fixed using QR codes to initiate login requests for major internet banks (which means you have to be in front of the same screen now) and other clever workarounds. But it has also always been a fact that there will be a description saying what you are signing, in the app, so being careful you could easily avoid being scammed.
I think its largely a great asset (BankID) but its never gonna be 100% tamper-proof without being seriously neutered.
In Denmark, you're forced to use the state-run "NemID" for credit card payments, making for some weird situations where you authenticate with NemID inside iframes on shady URLs.
The same NemID is also used to file your taxes, look at all your health info, get married, everything basically.
Credit card payments are much lower security level, and they're basically forcing sharing credentials amongst all the sites you pay on.
However it also forces everybody to use two factor authentication. On a whole population level I'd bet that's overall a positive tradeoff.
And I believe you can also use sms + password for online transactions.
2FA is already mandatory by the PSD2 directive of the EU. I use my debit card as the second factor to access my bank account here in Germany via ChipTAN
SMS + password works for some Mastercards still but not Visa.
I don't think it's good that users are taught to accept their primary citizen 2FA on any random website and app where the URL doesn't even show.
Yeah, same way they have it in Sweden, it's called "BankID" and only a few banks are allowed to issue that
I've worked on BankID implementation and it was super smooth, good tools for testing and well documented.
We didn't need to scam anyone though, just have them verify that they were a Swedish resident (had a valid Swedish SSN and we're the ones ordering) :D
Major distinction being that BankID is privately owned and operated, as opposed to state-run.
Yeah I once had to make a ~20k transfer with Klarna and was shocked to see that they essentially hijacked my credentials. I only went through with it because there is additional 2FA (on my bank) so they wouldn't have been able to repeat it. But still a super shady practice. I was sweating for days until I got a confirmation that the transfer went through successfully. 1/5 experience.
I have the same sketchy feeling about Sofortüberweisung/Klarna. If they want to make transactions on my behalf, why should I give them full access to my account?
Most banks have a paragraph in their contracts/ToS forbidding sharing the account with third parties, but they are rarely enforcing it. Still, they could close the account due to contract/ToS violation.
Worse, you're on the hook if your account is drained.
I understand that Sofort was allowed to continue despite using the user's bank credentials because disallowing them would be anticompetitive.[0] I have no idea how that could justify such an insecure practice, but there you have it.
[0] https://knowledge.fintecsystems.com/en/blog/the-history-of-o... , under "Legal Action by Giropay"
Have been using SÜ for years until I learned that they not just facilitate the transfer but abuse their role to dump bank transfer data worth several months. I don't use that service anymore.
That sounds pretty bad! I always thought the login flow was super sketchy, but wasn't aware of this part — has this been covered/analyzed somewhere or is it evident from their terms or something?
Here is an article in German : https://www.sueddeutsche.de/geld/zahlung-per-sofortueberweis...
I cannot answer this question satisfyingly. I read it somewhere and found tangential information by google search - but nothing very specific.
Sofortüberweisung specifically got caught looking at 30 days of transaction data.
> how banks can allow this
A court decided that blocking this "business model" would be anticompetitive.
Do you have sources on them looking at transaction data please? That is clearly not necessary for processing the payment.
Edit: Found an article in German - https://www.sueddeutsche.de/geld/zahlung-per-sofortueberweis...
They claim they need to do this to make sure there is sufficient money in the account, even with transactions that might not be reflected in the balance, and they also check for other "Sofortüberweisungen" to detect fraud. Makes sense in a way but still quite shady. If there wasn't enough money in my account, or other transfers pending would my bank even allow their transfer?
Can you explain more about the credentials and online banking?
I've used (and integrated with) Klarna in the UK and from what I've seen it's only really a payment method with merchants who you pay back by card later.
In Sweden most people have an electronic way to identify themselves to their bank (BankID) and it is used by many services to verify your identity.
It's extremely useful for any ID verification, but Klarna asks you to verify your identity towards them but when you open the app they have instead sent a request to identify with your bank, using your credentials.
Klarna provides many different financial services.
They provide "pay by bank account" (which involves the mentioned MITMing of users' online banking accounts, unless Klarna is integrated with your bank via OAuth/PSD2, which is still not ubiquitous), but also installment payments/factoring and others.
Klarna is actually its own bank these days so that doesn't really happen anymore. I think however many other payment providers operate this way still which is ridiculous.
Then again, PSD2 API roll-out has been very ???
It's happened wayyy into them being their own bank (at least until 2019 when I started boycotting them)
They signed into users bank accounts, in other banks, to set up transfers (which also gives you all account statements).
Did not know! Guess being scummy doesn't stop because you get a license.
Is that true for all European banks though? I think they all need to have an API available at this point, but is Klarna using that in every instance (instead of their legacy creepy MITM scheme) already?
What could a competitively convenient way to do this better look like?
You can generically solve the problem of Alice giving David access to Bob's service on her behalf without giving Alice's credentials for Bob's service to David using stuff like OAuth2, this is already how lots of things work today.
In OAuth2 David only ends up with some token showing Alice authorised David to use this service on her behalf. Bob can tell David and Alice apart, and choose to restrict what David can do appropriately.
If Bob is particularly tired of this nonsense, and his customers like Alice keep giving David their credentials and then are surprised that doing so means Bob can't tell Alice and David apart, WebAuthn reifies it so that most users in Alice's position can now see where the problem is. When David tells Alice he needs her Yubikey to access Bob's service, it should occur to Alice that giving the Yubikey to David isn't a good idea because then she won't have it any more. Good.
I think PSD2 is supposed to solve these problems with a less insane approach, but the rollout seems to be quit sluggish.
Surprisingly, there are already integrations in my home country; I took a look at tink [1] some time ago (no affiliation whatsoever) and they look legit. I'm sure there are more SaaS like them.
https://plaid.com/ does it well.
Don't they effectively do the exact same thing? As far as I know, they use screenscraping for most US banks rather than something OAuth-based.
I think it depends on the bank. It's really up to the banks to provide a proper API.
I cancel every online order if I find out that it is handled by PayPal, Klarna, Mollie or other data collecting entities.
The situation in Europe is so bad that you are sometimes tricked into a prepaid order only to find out that the invoice comes from one of those.
The appropriate penalty is immediate cancellation and multiple GDPR requests.
I looked through the terms of use and the privacy policy for Mollie and I don't think they are selling data. Do you have different information then I have?
Kristel and Sonya seem to have the same due payments
Yes the chance of that is almost zero. Either the due is the actual users value (only profile name is changed) or this is some kind of test data being exposed.
I suspect this might be request threading/confusion[0] issue similar to the one GitHub experienced a while back. This would explain why seemingly random user data is being returned.
0: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
We can only speculate, but what baffles me is that it happens for something so private, and for a company that is so rich. Do they not audit their code? Do they not risk assess these things? "Ah, storing user credentials in thread local storage, that sounds sane and bug-proof" said no auditor, ever.
IIRC, Klarna is mostly written in Erlang, Scala and some parts in Clojure.
If someone should be aware of thread-local storage and its implication it ought to be them.
I was under the impression that they had switched to Java more in recent years
Using trendy tech doesn't solve much by itself. Especially if you can't (or don't) compete with FAANG on compensation.
This has changed many years ago.
My email includes a common Swedish first name so I regularly have people mistakenly use my email address for Klarna orders. What’s most annoying/troubling is that, at least last time I checked, they don’t verify an address before sending invoices, etc. so I end up with other people’s order info in my inbox. I finally started unsubscribing from notifications for orders that weren’t mine.
Is your email adress firstname@something.etc?
I've seen a lot of people not get their emails and wondering if they parse lastname.firstname@something.etc wrong
Totally anecdotal, and probably unrelated, I interviewed for Klarna a few years ago.
Mid process, they sent me some sort of timed bizarre IQ test that the recruiter claims EVERYONE who works there has to take.
That's when I knew that kind of working culture wasn't for me.
A recruiter contacted me aswell and I asked about their salary. They pay 50k euro for juniors in berlin with afaik no stock vesting. How they even manage to get qualified personnel is beyond me, I would expect much more for a fintech with over 3B evaluation
Sounds very good for a junior dev in (northwest) Europe.
actually they are a top tier payer in Stockholm
50k in euro's is pretty ok for European developers, no?
I am from munich so my opinion may be skewed, but it is at best very average, as in some no name devshop/consultancy would pay this much(and even they tend to pay more). Nothing I would expext from a Unicorn, but maybe the market in Berlin is THAT different.
In all Western and Northern Europe (except UK and Switzerland), 50K for a junior position sounds about right. It’s around 25K in Southern Europe, and probably less in Eastern Europe.
I think this would make quite an interesting exercise for whatever it is one works on oneself; that is, what's the minimum, most innocuous patch that causes this behaviour?
I bet it's not as much as people railing against it would like to think.
I'm partly thinking of this because I fixed a (way less critical) bug today that boiled down to a (x - y) * z = 0 query that should've just been (x - y) = 0. But it was hidden by the whole expression being named, and that then seeming correct, it not being obvious that `z` could be 0 (or was involved at all) and as a result unwanted results would be included where x != y.
Probably the most obvious one is different IDs - have two fk columns that sound a bit similar and it's easy to come a cropper, getting 'random' records that correspond to a given ID but that's for the wrong table...
> getting 'random' records that correspond to a given ID but that's for the wrong table...
That's why I use GUID instead of integers. If you get a result, it was the right column.
Good point, we do for new things, but of course when I hit it it was with an old table and was a problem.. sod's law. (Though arguably it's just harder to notice the effects of it happening with a GUID and it could have too.)
I’m pretty sure this (or something like it) happens at least once to every major site. The stuff of nightmares.
We still haven’t got our money back for a purchase paid via Klarna. Apparently they wired the money to another bank account but under my partner’s name.
After 3 support calls and several emails, we just gave up. Fortunately it was just €12.
This was so frustrating that we now avoid paying with Klarna whenever possible.
What are the ways you can implement "log in as anyone accidentally"?
I'm imagining it was a case of an SQL-based password check where "TRUE OR" got added to the WHERE clause, and the code takes the first result instead of expecting only 0 or 1 row.
Are there other easy ways to do this?
1) Caching: a cache is used in front of the API for things like product listings, it uses a pattern match like /api/products/*, and caches routes which match. Someone accidentally configures it to cache /api/*, and thus login responses from /api/session return another recent user session, potentially including the cookie such that subsequent requests are authenticated as that user.
2) Mentioned elsewhere in this thread, a variable with global scope within an application server. This is very possible in node.js, which uses a long-running single thread - if you have a function like handleRequest(), you might inadvertently write to a global variable outside it, and that variable will persist across requests from different users. I've seen this exact bug in a PR - luckily we caught it before production, but if it had slipped through code review and integration tests and actually shipped, the result would have been exactly like the one in the tweet.
Why do users get multiple other users instead of one then, if it's a global variable? I assume because Klarna is running on many servers?
It could also be that new logins overwrite the cache/global
It can be a bug in the application server as well, I recall uwsgi having issues where the request (or response, not sure) dictionaries were recycled between requests, and some corner cases didn't clear those between handling different requests, or something to that tune.
From a quick glimpse on twitter, people couldn't make changes to any of the accounts they were seeing.
This points in the direction of this being a caching bug; you request your homepage, and get the homepage of whichever user was placed in the cache last.
Most of the time in these situations it's not an application-code issue (per-se), as much as a "shared global state" issue.
It's not a web system but Mac OS messed it up once: https://objective-see.com/blog/blog_0x24.html
Caching could be an issue, if they added a cache for a microservice call of /get/user?id=$USER and ignored the id parameter, /get/user?id=ipsin fetches data for the user ipsin, the system sees the next call /get/user?id=bellyfullofbac and thinks, "Wait, I have the results of /get/user in cache" and returns the data for ipsin again...
Besides having the HTTP verb in the URL (GET -> /get/), why would you put the id in the query? Why not just use GET /user/1234 instead of duplicating things by using GET /get/user?id=1234 . What does GET /get/user then even return, all users, no user, ...?
Edit: typo
It's just an example...
They also had a snafu with marketing emails late last year [1] - not a great look for a company handling bank/payments.
Will be interesting to see what the problem is here. From what I have seen in real life my top guesses are. Some dependency on static variables in code. Reversed proxy with incorrect cache rules that ignores headers or some parameter.
How do you envision the static variables thing? I've seen the cache thing myself in real life but not the other.
In C# for instance. If you mark a field static it is the same for all instances of a class (if you don't mark the code as thread static). So if you have static User field that changes on logon it will change for everyone. I have seen this but typically more complicated versions of it.
Store user in static variable during processing data, then forget to clear the variable when you are done, so for the next request it still has access to the old data?
These can act like a cache across all instances. For exactly this reason I use them only as final (constant) variables and very, very rarely mutable.
you have to wonder why they decided to stay up. Surely, if you have a leak this bad, you pull the plug until you can fix it.
they shut off the whole app, and kicked off who was logged in. Fair approach until they figure out how to sort it.
Probably a push to prod of something of something that worked on the developers machine, Klarna is at the size where any fault like this would be seen by thousands within any reasonable reaction time.
According to the article they shut down all logins in the app. Unsure if this means you can still use it you are already signed in or not
As I read it, they did shut down as soon as they knew.
Their iOS app shows “Down for maintenance” :)
There was that time that Dropbox let you log in to any account with any password, too.
I've never run a line of Dropbox code on any machine I own since that day. Even if you have no tests whatsoever on your app, you should have some basic smoke tests on your auth system.
Cache invalidation issue. Classic
I can remember something similar happening on Facebook back in 2013-2014 (when I was a kid). I went on this app called 'Video Chat Rounds' and when I left the app, I got signed in to a random Facebook account.
Is that a really surprise to you guys? Just look for the old klarna news, this is not the first time and won’t be the last time. There is no security on internet, just get used to it, if you use klarna.
Other discussion that is rapidly sinking from the front page:
That thread got spam flagged or something and is no longer visible, but has a lot more comments and discussion.
We've merged the comments hither now.
the linked thread was posted after this one.
Why did this one disappear from the front page so fast?
Do you mean https://news.ycombinator.com/item?id=27301311?
A moderator buried it for reasons explained here: https://news.ycombinator.com/item?id=27305371. Sorry for the delay, but these days you guys need to wait until I'm online to get explicit explanations, because I'm currently the only mod who's posting publicly.
Thanks! Yeah, too bad we didn't get an explanation. Just removing a post with lots of interesting discussions in it from the front page is not my preferred respons. Should be some better way.
Who knows, many of the HN algorithms are secret and there is no moderation log a la https://lobste.rs/moderations
True, but it's still always possible to get an answer to a question—you just have to ask. However, we might not see it unless you ask at hn@ycombinator.com.
I don't want to appear ungrateful - let me take this opportunity to thank you sincerely for all that you do. Your set up appears to work, and I'm probably in a minority with my demands.
We wouldn't have to ask if you had a public mod log (and banned sites list etc) and a public explanation of the algos that power HN.
Your comment reminds me of hotels - "X is available, just ask". A scheme clearly designed to reduce usage of X. I'm guessing the current audience is quite diverse, as most engineers would see through that kind of BS in about 0.2ms.
The moderator comments are a kind of public mod log and a thing worth looking at regularly if you're interested in how and why HN is moderated.
Are you being serious?
It's true - I use those comments to provide detailed explanations, which I often link back to. They're sort of the case law of HN moderation. It's my intention to someday compile them into some sort of compendium of moderation heuristics or something...not sure yet what that should look like.
Entirely
So users are meant to first discover the key (the username) to lookup the logs? Then find a needle in a haystack of comments? Again, are you being serious.
I'm not sure what exactly you're asking me. There's a thing that fulfills the function of a public moderation log, an answer to your original question. What is the other stuff about? HN is absolutely full of not-particularly-discoverable UI, it's practically made of it. You've been here for over a decade.
I've actually written about that a lot over the years. Here are some links I dug up (mostly via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...). If you take a look at the previous explanations and still have a question I haven't addressed, I'd be interested in knowing what it is.
https://news.ycombinator.com/item?id=23837866
https://news.ycombinator.com/item?id=23807944
https://news.ycombinator.com/item?id=23286685
https://news.ycombinator.com/item?id=23227833
https://news.ycombinator.com/item?id=23127622
https://news.ycombinator.com/item?id=22939878
https://news.ycombinator.com/item?id=22711604
https://news.ycombinator.com/item?id=22648990
https://news.ycombinator.com/item?id=22547697
You're cheating: you know your username and can recall which of your comments were mod log entries.
Imagine creating such a log system in a company and expecting your colleagues to find such logs in such a manner. I'd move to get you fired.
This feels like it's swerving into just the sort of cross-examination that I describe in the comments I just took the time to dig up for you. My purpose in doing that was not to tell you "see? anybody can just go and find these". It was, rather: here is a set of past explanations about the question you're raising, which describe our thinking on this topic. If you want to understand why we don't do what you're suggesting, you'll probably find the answer there. On the other hand, if you have a specific point that I haven't answered in the past, I'd very much like to know what it is.
The intention of all of those moderation comments, search links, etc., is to provide helpful information to people in specific contexts. Nobody's pretending that it's a global documentation system; no one's "cheating" or trying to fool anyone or trick people out of what is rightfully theirs. We're simply trying to answer people's questions and satisfy their curiosity while also staying focused on the overall purpose of the site.
I’m done if you’re playing the victim card lol. Bye
Hey, relax. He’s not playing the victim. He’s just explaining how moderation works at HN.
Does Klarna still do the IQ test as part of their hiring process?
tangential thought, but related: I am, in general, a proponent of nuclear energy as a green alternative to whatever the hell we are doing today. But when I see such stories that humans manage to fuck up simple payment processing apps, still make errors while maintaining bridges, still manage to do hugely negligent screw-ups (most likely corrupt) in *cable cars maintenance*, I immediately think that it is imminent, that something will go wrong with such complex thing as a nuclear reactor and the price there is much bigger.
I kind of get the worry, but the requirements and processes seem to scale exponentially with reliability needs. Online companies may fuck up every day in new and creative ways and we barely get to hear about it. On the other hand we know of every nuclear failure so far with enough public details to discuss the whole time line, system design, steps each person followed, etc. and the death count is still minimal. Then each of those is an input to the future processes. Nuclear power plants and air traffic are in their own class of reliability and safety processes - not even comparable to that's happening in internet commerce.
We know every nuclear failure. We don't know every time a strong nuclear risk existed but by chance, didn't trigger. Nuclear power plants are probably much safer on average, but it only takes one corner cutting plant to cause a nuclear accident.
That said, I'm also pro-nuclear.
The Italian cable car was really messed up. The emergency brakes of that cart were intermittently triggering, so the operator jammed a piece of metal to stop that from happening.
His assumption is surely, "Relax, what's going to happen, the cable won't break!".
> The emergency brakes of that cart were intermittently triggering
My guess: each time a strand within the cable broke the cable stretched a little and the brake triggered.
Five years ago a company was hired to maintain the cable car. They took one look at the state of it, wrote to the operator (the town council) saying it needed to be shut down and exited the contract. It was an accident waiting to happen long before the brake fiasco.
There are also plenty of services with really incredible uptime. You just don't hear about them because they're deep in whatever stack you're using and haven't broken publicly in decades.
It's all about good engineering practice and architecture.
In fairness, the government does not watch over your shoulder if you write code for payment apps. Nuclear energy oversight is so over-the-top, it's almost not worth doing it.
Yes! We will always make huge mistakes.
.. so we need to accept the eventuality that the worst result will eventually occur.
...which is why it's important to design things to fail safely. That "worst result" needs to be minimized by design.
With nuclear .. that's probably a bit difficult
There are much-safer-by-design reactor plans out there now. Hopefully the "nuclear is horribly unsafe by default" opinions will consider the new options.
The worst scenario is always meltdown.
Oh boy this brought back memories for me.
I thought that IQ test was screening test, pre-phone interview. But no, they had me redo it at the onsite interview too. The funny part was the onsite test had the exact same questions as pre-phone interview.
edit: typo
Ha.
I remember doing it too. I was at work in a meeting and they have instructions saying something to the tune of finding a quiet place and all of that, but my thoughts were if they are serious about this, then solving these abstract problems is something I'll have to be able to do while under pressure or under the heat of conversation.
Long story short anyway, I'm not intelligent enough to work there I guess, so good thing they used that test to screen me out and make sure I knew. It does have a little bit of merit with the very quick no versus the long, drawn out no. I recently interviewed at a great company, 4 1-1 interviews, a presentation/demo I had to make to present to 7 other people, etc. and I think another interview after that and I'm just over it.
Maybe to check for cheaters
Asking the same question doesn't help a lot to find cheater with a memory.
Weeds out those who had someone else take it, though.
Yes, sadly a quite common part in too many recruitment processes here in Sweden.
My first job at a consulting company out of uni I had to to an IQ test that could also indicate if I had rabies.
It had questions like "are you afraid of water", "have you showered in the last three weeks", "have you felt more aggressive lately"...
Considering these questions, have you honestly answered "yes" to the last one?
I've had to take the MMPI [1] for an employer before. About 500 true/false questions to screen for mental health disorders. Some of the questions seem quite outlandish but taken as a whole make sense.
[1] https://en.wikipedia.org/wiki/Minnesota_Multiphasic_Personal...
I'm sorry what?
It’s a joke about the pattern of questions
It also assumes that one knows that rabies can cause hydrophobia:
https://biology.stackexchange.com/questions/16749/why-does-r...
Wow, first time i hear that. Aren't those IQ test horribly biased?
In the United States, the Supreme Court has ruled that using IQ tests for employment screening can violate Title VII of the Civil Rights Act.
not only that, but they aren't a great predictor of actual job performance
Given that these tests don't evaluate critical thinking and knowledge of statistics it's quite ironic but coherent for the company using these tests.
They are a great predictor of actual job environment though (the test, not its results)
Depends on the job actually. (I know this will be unpopular but in my experience hiring for certain roles it is correlated)
This is exactly why these IQ-test companies make so much money. It gives out yeses and no:s confirmation bias does the rest.
Quickly why they don't work:
You create a huge chain correlational assumptions. First that visual-spatial tasks of this kind predict performance on visual tasks. 2. That performance on visual tasks predict general intelligence (whatever that is). 3. That this notion of general intelligence (which is usually and arbitrarily defined not to include social skills) actually correlates with the tasks that you think the person will be performing, and finally that your idea of what the role has an impact on the company. Of course it is completely absurd, what they are selling is snake oil, plain and simple.
The remedy I recommend is simple, talk to the person - do it and you will be able to tell within 5 minutes.
I use it to filter out candidates when hiring assistants from the Philippines. I get so many applicants that I don't have time to talk to all of them. Have you been on the hiring end? You get hundreds of applications within a couple days when you have a good position and are paying well enough.
There is a strong correlation between IQ and professional achievement whether you want to believe it or not.
Right - and the general intelligence thing is funny too because all of these companies want to hire specialists in some area, not generally intelligent people.
You are making assumptions here that are incorrect. I use it when hiring virtual assistants. They don't need a specific skill.
I think you're just trying to justify an unjustifiable test to protect your ego.
First, if you're just hiring assistants from the Philippines they could just have someone else take the test or get around it some other way.
Second, you have no good data to support this hiring practice. You're free to use it, but it's no better than just hiring a random person from your pool of applicants. You might as well screen based on their favorite color too to just make up filtering criteria.
I failed a job application because of the IQ test. It was administered in a second language for me, so I really didn't do well... the interviews had been completely smooth and I got on well with everyone I had talked with...
The thing is, I really needed that job... ended up going to another job that offered me a very low salary (I had no visa in the country , so was looking for a sponsor, which makes things a lot harder) and the company went bankrupt within a few months!
Anyway, I still got the visa, and then, with a few months to find another job with more peace of mind, I eventually got much nicer job, paying a lot more! But I still dread the though of doing an IQ test, despite my years of experience indicating I am more competent than average, at least.
I have never had to take a general IQ test when job hunting here in Sweden. Coding tests, yes, but not IQ.
I had to do it twice for different companies that used the same IQ-test platform.
And most of the time it's not even proper IQ-test but only Raven Matrice test + maybe quick math tests.
Funny thing was that I did very good (apparently according to the HR person) on one of them, but did horrible enough they didn't even call back on the second test.
grids my gear why this is still a common practice in Sweden. HR in Sweden seems to be about one or two decades behind rest of the world in their efficiency.
I would disagree. I’ve never heard of any company doing this, nor any former or current colleague that had to do one.
Really? I'm surprised I haven't heard of that.
They did it a year ago when I was applying there. I was so annoyed it almost made me cancel the interview. In the end I canceled, because they didn't allow working from home.
Doesn't almost everybody? I mean let's get real here, what's the IQ distribution at FAANG or any other competitive software engineering company?
There may not be an explicit "IQ" portion to the process, or a hard number, but they are absolutely filtering on intelligence. An uncomfortable aspect of our society that I'm both surprised and not surprised doesn't get talked about much.
When I applied to work for a bank in Canada back in the 80's I had to write a Wolfe-Spence Programming Aptitude Test (was basically an IQ test).
The hiring company would send your answer sheet and work sheets off to the company to analyze and provide a score.
Anyone else remember those?
Yep, took it just a month ago. Quite dumb honestly, not sure what it indicates. It was a bunch of weird pattern matching and guessing? Seemed easy, but got a rejection a week later.
They did that?
Although, I guess Google does IQ tests too in effect? But it's called "solve this puzzle" and "here's a riddle".
I don't think they do those anymore. At least when I've interviewed there (2x) over the last couple of years I did not encounter any of those types of questions.
We are all mistakes that sometimes make humans.
yes
It's not an IQ test. It's just pattern recognition which is about 5% of the tasks you do in a real IQ test.
When I joined Klarna in 2011, the test was so easy that I joked I could score full marks on it even if I was hungover with no sleep. There was one question on the test that actually had 2 correct answers depending on what logic you applied. This was actually a real issue when recruiting, because there was a hard cut-off to make it into the engineering department, and several times I had to ask "what was their answer on question 12?"
It caused quite a bit of commotion at HR to change the official test scoring to 2 correct answers for that question.
Now the test is like a million times harder and your score at the end is between 0-10 and you have no idea how many questions you actually answered correctly. I would be very interested to know the "true" answers of these new tests to understand what kind of crazy logic you need to apply to get every question right. I'm almost certain it would take me longer to understand the answer than the time you have to do the test.
It's not an IQ test. It's just pattern recognition which is about 5% of the tasks you do in a real IQ test.
So it is an IQ test, just not a comprehensive one?
That test was always stupid and fought hard by a lot of engineers that considered it so. It was still kept even after a lot of pushback. I left the interviewing team due to that, I couldn't be part of a process that considered that step not only required but as a hard cutoff for engineers.
I lost so many great candidates that would be great hires to my teams at Klarna to that stupid test.
It will be an interesting post mortem if they make it public.
if they make it though alive ...
Let's hope not. They're deliberately trying to get people to take on debt rather than just do card payments, and even simple things like buying a book through a web site requires declining several offers for paying with credit.
Unfortunately, they're huge, and I doubt the Swedish authorities will do more than give them a fine and a slap on the wrist.
Card payments are usually debt also?
Debit cards is more common in quite a few places. My impression has always been that paying everything with a credit card is a U.S. thing.
Here in Finland, It's not uncommon to have no debt apart from the mortage on one's home.
For what it's worth, I have paid for nearly everything I possibly could with credit cards for decades, and I haven't paid a cent in credit card finance charges in at least 15 years (since my fiancee straightened me out and helped me see that I was being dumb about debt). I have them set to auto-pay the entire balance every pay period.
I use them because consumer protections with other methods aren't as good here in the States. Paying with a credit card, if I have an issue with a vendor, after a good faith effort on my part to resolve the issue, I can just ask the credit card company to deal with it. (I don't abuse this, but I don't doubt there are people who do.)
There are better and worse credit card companies for this. American Express has great customer service but they aren't accepted in as many places.
Finland doesn't have credit score system, so there's no reason to not pay not pay the credit card bill immediately. There is certain push towards credit cards, though. As far as I remember my bank would charge a yearly fee for debit card, but credit card is free for me.
Mortgages are 70% of debt in the US. It is like saying I got perfect on a test except for the 70% I got wrong.
Not really. Mortgages are secured against the property, and attract low interest rates compared to unsecured debt like a credit card.
That's interesting. Mortgages are 100% of my debt, and the debt of most everyone I know. Which I guess was the point GP tried to make.
> Card payments are usually debt also?
Debit card payments are not debt - they're effectively the same as a direct transfer from the user's bank account.
I'm very conflicted about Klarna - on the one hand they do present an easy and (usually) safe way to handle transactions with small retailers to whom I don't necessarily want to share my payment details.
But on the other hand, they use a variety of dark patterns to try to get you to pay: 1. on credit 2. by signing-up for their credit-card
One unfortunate part of their earlier history, was that when you promised to pay with Klarna on a website, and was told you'd receive the invoice, there was a (perceived?) tendency for that invoice to never be sent due to an 'oversight'. When this happens in Sweden, the buyer gets a reminder a few days after the due-date, with a pretty large extra amount to pay.
There were quite a few stories about this in the press at various times [0], and I know quite a few people from Klarna and would tease them about it - which they always strenuously denied - but then it happened to me.
In any case, finding out how this happened is going to be interesting.
[0] in Swedish: https://www.svd.se/mangder-av-klagomal-mot-klarnas-fakturor
DeepL translation: "Lots of complaints against Klarna invoices. Klarna, the high-profile IT company, is being criticised by a host of customers. Many say they receive invoices with reminder fees and collection demands directly, without having been reached by an original invoice. The Swedish Consumer Agency is critical of Klarna's invoicing methods for several reasons and is currently investigating whether the company is behaving legally."
Translated with www.DeepL.com/Translator
>They're deliberately trying to get people to take on debt rather than just do card payments
So what? It's 0% interest. It's incredibly helpful to have easy-access financing to split purchases across a few months.
>even simple things like buying a book through a web site requires declining several offers for paying with credit.
This sounds so specific it seems like you're taking a bad experience with one website and pretending all websites are like this. Most e-commerce sites I've used in the past year offer Klarna or some similar service and all of them have been implemented as just another option in a set of radio buttons.
> It's incredibly helpful to have easy-access financing to split purchases across a few months.
I don't know, it seems like a failure at adulting to have to do that for small to medium sized purchases. If you need the feature, you probably should not have it available. Maybe this is my German attitude about money - basically, only take on debt for investments, a notable example being housing.
> So what? It's 0% interest. It's incredibly helpful to have easy-access financing to split purchases across a few months.
Unfortunately, this often isn't the case of people who are worse off, not good at managing their finances, and often overwhelmed by bureaucracy.
They fall behind on payments, and then get taken to the cleaners on fees, deferred interest etc., often paying several times the actual price of the product. I've seen this happen (with different but similar services).
Less savvy people being sold stuff they can't afford on credit has been such a problem that some countries have made it illegal to extend credit to someone who can't afford it, which is obviously extremely hard to enforce.
This is hard to grasp for many here, because HN readers tend to be well above average intelligence. Try to think in terms of "imagine how dumb the average person is, and now realize half the people are dumber than that". Now add mental or physical health issues into the game.
> So what? It's 0% interest.
Debt is slavery and so on. Let's not get too hung up on the fact that I dislike it.
> Most e-commerce sites I've used in the past year offer Klarna or some similar service and all of them have been implemented as just another option in a set of radio buttons.
Radio buttons is fine. It's the defaults and "are you sure you don't want to pay with credit?" questions I'm bugged out about. I don't have an issue with them offering it as an option. I've seen it with multiple websites using Klarna for payment handling.
>Debt is slavery and so on.
No it's not, and statements like that trivializes the mistreatment that actual slaves went through.
I agree, but just to clarify: inability to pay one's debts has historically been one of the primary ways into forced labour with unfavourable conditions. A bit away from slavery still, but not a completely out of the air connection.
You missed the point. It was a deliberate simplification (hence a simplification of a biblical quote and the addition of "and so on") intended to steer the focus away from my personal opinion about debt, and towards the second point, i.e. dark patterns in order to get people to pay with credit rather than with a debit card.
If you rely on your application layer to enforce data privacy instead of enforcing it in your storage layer its just a matter of time until you have an issue like this.
It says a lot about the security of their api and development culture that they are even struggling with something like this. This should be caught in the first architecture review session.
Out of curiosity, how is that enforcement usually done? I have usually just used some SQL database like MySQL/Postgres, and having application determine how to fetch data, so application has access to everything. I can see how this could be insecure due to some bug in application code fetching with wrong WHERE etc, how would one go about enforcing it on sql/database layer?
Would you have separate SQL credentials for each user, and configure SQL for each credential to have access to certain WHERE queries, or?
To simplify a use-case let's say I have "users" table and "tasks" table, where there's user_id in "tasks". Would I have separate sql credentials where they are configured in sql layer to have access to only rows where user_id corresponds to this certain credential? But even then how are credentials mapped to userId, as bug in application could easily cause retrieving false credentials?
Other way I can think of is to just have completely separate databases for each user, but let's in this case assume we must often do work with a mix of different users data.
So I think the best place to start is looking into row-level security in Postgres. Its a familiar place to start and gives a high return. Row level security can be used to implement the user / tasks use case you described.
How would any measures at storage layer prevent, for example, issues in caching?
And how can one enforce it on a storage layer? There must be something in the application that determines user identity, which either threading, flawed logic, bug or caching (most likely) can mess up. In which case storage layer gets this identity information from application layer.
In my experience very few have storage layer separation for customers data. It all logic in the application layer to control access.
Do you mean stuff like row-level security in the database tables?
Cached data in middle layers can get even the safest of row-level secured databases.
I agree in general that you need to enforce things at the storage layer.
You're right, and cache policy issues can be really hard to debug.
As a rule I don't cache personal information for this reason.
Out of curiosity do you have any knowledge on GDPR's stance on caching PI?
The new guy that stores user information in the servlet. I have seen this before.
Reminds me of this GitHub incident: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare...
I've seen this happen when Cloudflare caching is misconfigured.
Context in global variables
Translation:
Major technical breakdown at Klarna when customers saw other people's data - The Swedish Financial Supervisory Authority (FI) has contacted the company
Payment giant Klarna, which has 87 million customers globally, is currently experiencing major technical problems. Users of the company's app saw other customers' payments and personal data, before it was shut down completely. The supervisory authority Finansinspektionen, FI, has asked Klarna to explain what happened.
In its app, Klarna has major technical problems. It means that users were logged into other customers' accounts and thereby see sensitive data such as their payment and purchase history and postal address. Users were also able to see part of the bank details linked to Klarna, but not the full account number.
One of Di's journalists accessed an account belonging to "Elisabeth". When the app was reloaded, another customer's login became visible.
When customers logged in with their own bank ID, they accessed other people's accounts. Each time they refreshed the page on the app, they brought up the details of a new, seemingly random user. It is unclear whether customers have been able to shop with other people's money.
Klarna had a total of 87 million consumers worldwide at the end of 2020, but it's unclear how many of those have an account on the company's app. The technical breakdown also extends beyond Sweden's borders, with outraged reactions pouring in on Twitter from Klarna users in various countries.
Klarna has now closed the app, citing a service outage. The company's press officer Niklas Gillström will return to Di after a while with a written comment.
"We are currently experiencing disruptions in our systems caused by technical problems. We are doing our utmost to restore the system and our services to full capacity and apologize for any inconvenience this may cause our customers. We have currently blocked all logins to the app until we are sure the problem has been fully resolved."
Di continues to seek the company for follow-up questions on whether the technical problems are due to an internal breakdown or external influence, how seriously the company views the sharing of personal data between users and whether customers may now have accidentally traded with other people's money. Klarna has asked for a response.
The Swedish Financial Supervisory Authority, FI, which among other things is the supervisory authority for banks, states that it has been informed of the situation.
"We have contacted Klarna and asked them for an explanation of what has happened," says Karin Lundberg, head of the business area Banking, to Di.
At the moment, FI has no further comments, she adds.
Di also seeks the Privacy Protection Authority, IMY, formerly known as the Data Inspectorate, for comment.
IMY has the right to fine companies up to 4 percent of their global annual turnover for serious violations. In addition, Klarna could face civil litigation, not least in the US where it has 15 million users.
(Translated with www.DeepL.com/Translator)
I m sure it s not random but somehow systematic
If it's really a reverse proxy / Varnish / CDN / etc. misconfiguration issue like some others here suspect, then it could be totally random. The data of some unlucky person whos data happens to get requested when the cache times out will be cached and then sent to all others.
Reminds me of a colleague implementing "emailRecipients" as a field in a singleton service. The first online order got an order confirmation mail, and when a second online order came s/he also got their confirmation mail (the field just grew and grew...).
One more reason not to make singletons.
Singletons are fine and useful in many situations. You just have to understand what singletons entail, and design them correctly. If his singleton had a "SendEmail" function that accepted an Email object with To, From, Subject, Body, etc. fields, it wouldn't have been an issue.
I strongly disagree. Singletons are most of the time a code smell. They hide dependencies, make testing hard, and enforce tight coupling.
Singletons are easy to understand, as long as they contain of one simple class. But after a few iterations of development, they tend to "capture" a lot of dependencies, which practically become singletons too. A lot of mistakes happen. And most of the time, there was no good reason to create a singleton in the first place.
see also those posts: https://stackoverflow.com/a/138012/4249619 https://stackoverflow.com/a/142450/4249619
I'm of the opinion that singletons are only useful if both of the following requirements hold:
1. They MUST NOT allow more than one instance. "I don't think anyone will ever need more than one" isn't enough. Just create only one instance then. Only enforce single instance if there is a requirement for it. For example, a logger is a bad singleton because you could conceivably want more than one instance. Something that requires exclusive access to some hardware may be a good candidate though.
2. The instance must be globally accessible. Many things don't need to be globally accessible though.
So unless you need a global enforced-single-instance of something, which in my ~20 years of programming is rarely needed, a singleton is a bad choice. In my experience, many times someone wanted only one instance, some time later it turns out that actually multiple instances would be useful after all (separate loggers for separate types of logs for example).
In most cases where singletons are used, a simple global would have sufficed. If you only want one instance, then create only one instance. If you need lifecycle management, then do something for that.
Those SO posts cover it nicely.
To be fair singletons are pretty useful. You just have to understand that they're not made for mutating state.
I like the Whisky.
Singleton Malt? Me too!
Could be random. I've seen this behaviour when enabling puma and using non thread-safe code. Just entirely depends on the timing of the requests.
I suppose that maybe comes down to your definition of what 'random' is.
Looks like a JWT oopsie
Time to GDPR my account on klarna then.
You can't—at least in Sweden—remove much from Klarna.
Your marketing profile is tied in with their accounting system. The law requires them to store accounting data for at least 7 years, with no obligation to actually remove it once that time is up. Since the accounting laws supersede the GDPR: they can hoard data pretty effectively.
The Swedish 'Data Protection Authority' tried to launch (yet another) investigation for their shady practices, but Klarna strategically applied for bank status and now the reach and power of the data authority is cripplingly limited.
I believe you that Klarna are shady about how they manage data, however, my understanding was that they got a banking license because they want to fund themselves via brokered deposits? A banking license means that they can get money from anyone in the EU and it will be insured up to €100,000. Without this, almost no one would want to deposit with them.
If you have other information about other reasons they might have become a bank, I would be genuinely interested in hearing them.
Whats Klarna’s argument for the data in a customer’s marketing profile being necessary for accounting purposes? You can’t just store data in your accounting system and wipe your hands of GDPR.
That's what the investigation aimed to find out before it was cut short. Klarna's general reasoning has been (A) 'because', and (B) 'because it's all in the same system and we have no obligations or confidence in thinning it'.
Any request for data or information regarding their architecture is rejected on the grounds of 'trade secrets'.
Hmm, that's strange. I did some contract work for Klarna about a year ago and had to go through mandatory on-site training and a big chunk of that was with their legal team about data protection, GDPR, about storing the least amount possible etc. It sounded like they treat it very seriously, so this is surprising to me.
I do know there are various legal requirements to retain certain data for some time (PSD2 for example must be stored for 13 months, I believe), but outside of that, it sounded to me like they tried very hard not to store anything for longer than necessary or without user consent.
I mean, doesn't mean its true, just the impression I got from the training.
Sure but there is probably a lot of meta data that can be removed and it will send a signal to them that this is not ok.
You can forbid Klarna sharing the accounting data with anyone. I doubt there is a legal sharing permission overriding GDPR for accounting data aside from tax authorities.
That's correct, but the data still stays with Klarna. I interpreted the OP as wanting to remove the data Klarna stores, or remove the 'account' pages. Neither of these are completely possible.
Free advertising
ahh thats why im struggling to sign-in
So a maximum gdpr fine of ~$48M?
The Klarna effect?
Just nu svettas det mer än det regnar hos Klarna i Stockholm.
Having at least authenticated sections of your site use HTTPS was standard well before 2011.
Let's Encrypt started in 2014 to address HTTP overuse.
In 2011, I (in-house corp app dev) was still stuck with HTTP services (behind a firewall, accessible only via VPN).
In 2014, public facing mobile apps using HTTP was prevalent enough to prompt name and shame campaigns. [1] My fuzzy memory suggests some banks were still using HTTP.
[1] https://arstechnica.com/information-technology/2014/08/new-w...
Bank of America back in 2005 (timestamp from the annoyed email I sent them) refused to load the front page over https. I think it even redirected https attempts back to http. The form submission was over https.
The solution was to enter garbage for the first login since the "re-enter your password" page was served over https. I think they fixed it before 2011, but don't have an exact record of when.
I started doing professional web development in 2011. It was very clear at the time that not using HTTPS for any site with a login was an BAD practice that made your users less secure. There were clearly people and institution still using bad practices, but risks were clear to most web developers.
What was shifting at the time was developer views on using HTTPS for non-secure, unauthenticated portions of websites. This is where the "HTTPS Everywhere" plugin and other such movements came in.
From what I remember there was a lot of pushback from infrastructure as we thought using https for the whole website would increase CPU load. Never verified if this was true... but I'm sure someone here should know.
Re 2011
Push back on what? There was pushback against HTTPS for non-authenticated pages for various reasons.
That does not mean that HTTPS for authenticated pages was not considered a standard and necessary security measure.
Let’s encrypt came way way late to the party. We had been banging the drum for 20 years by then.
If the pages are only accessible via a VPN, what does HTTPS really get you?
Not needing a VPN. rimshot
In all seriousness, better security. You are leaking whatever payload is sent right after VPN drops. An early version of the application had a defect because it did not check response payloads on an endpoint (the code handled errors, but 200 OK was all it needed on success). This is not what you want when the 200 OK is followed by the HTML of a hotel's wi-fi access page.
That only protects the user's password. The auth cookie will be sent in all subsequent requests in plain text.
EDIT: that's how firesheep (https://en.wikipedia.org/wiki/Firesheep) hijacked sessions for e.g.
That's not true. Cookies can have a 'secure' attribute which tells the browser to send them only over TLS
But that just makes your login not work if the rest of your site is HTTP, doesn't it?
You should not show authenticated pages without HTTPS
A secure cookie would be of no use for a site whose only secure page is the login page, which is what the parent post I replied to was talking about.
in 2011?
Yes
Only for certain domains and in certain regions.
A lot of the web was still on http, including some banks.
Even Facebook was still primarliy http when Firesheep [1] came out in 2010.
Not sure why you are being downvoted but this is exactly correct. We had, as an industry, been so focused on PCI during this time and TLS was and continues to be the most important aspect of the protective technology. SSL/TLS had already made e-commerce viable in the 90s and its power was well known and being applied for the decade following. Being in 2011 without full ssl for authenticated access was quite bad behavior indeed. Maybe excusable for some low rent bulletin board, but perhaps that is what the commenter was operating. I have no clue.
We detached this subthread from https://news.ycombinator.com/item?id=27303744.
> Hear hear, I used Klarna (not by choice)
It was by choice. You weren't born with an account.
Not taking personal responsibility for the rise of the ubiquity of these terrible online services (WhatsApp users, I'm looking at you) is a huge part of the problem. Pretending that you didn't opt-in is a lie you've told yourself; you shouldn't propagate that lie to others in society.
> You weren't born with an account.
A merchant I shopped at, and paid in full by card, opened an account for me and shared line item details with Klarna, apparently because they are using them as their payments processor in addition to an installment payment option.
I noticed this when I later did in fact "open" an (or rather, claim an existing) account with them.
Very disturbing, and the bad aftertaste has never fully gone away.
Do they always open up an account when you pay by card? Because I've definitely paid online purchases with my credit card with Klarna as the processor, but I am not aware of having an account there.
You could try to find out by opening an account or alternatively with a GDPR request.
I just checked the chronology again: I performed the initial order months before opening the account, yet the line-level item details are there (and last time I checked, there was no way to delete these, for a payment years ago).
Oh, and I'm almost certain that somewhere within the fine print of paying at that store I consented to all of this, but this does not make it any less creepy from my point of view.
> You could try to find out by opening an account or alternatively with a GDPR request.
Klarna AB (a Swedish company) is obliged to follow GDPR worldwide. So, this does warrant a GDPR inquiry.
Same thing happened to me when buying a graphics card from a computer parts website. Klarna was so well integrated into the UI of the checkout process that I didn't even notice I was giving my details to them.
Only afterwards I noticed on my bank statement. I sent them a gdpr request to delete my data.
This comes across as "If you haven't read every word of all the terms of use, privacy policy, and any other legal documents of not only the initial company you interacted with, but of all their second party companies, and their services, and their services, until you've researched the entire chain of partners who could potentially have something to do with your transaction, then you are obviously complicit in everything all those companies chose to do and you have no grounds for complaint."
While you may be literally true, the reality of this economic situation is full of far more gray area than you allow for.
If this confrontational, extrermist position is intended to try and wake people up to all this, I fear your message is outweighed by your snark.
And if you don't care about that, then I've wasted as much of my breath as you have yours.
In Sweden there is a current cultural view that the only reason someone would not sign up for an account with any kind of banking service is because they are too old to navigate the registration process, in which case all they need is help going through it. Any other explanation for why someone does not have an account at X is perceived as perplexing or straight alien. Non-coffee drinkers will have an easier time culturally then those rejecting getting accounts at sites like klarna and swish.
Regarding Swish I would agree, regarding Klarna I could not disagree more.
Same here. Swish is a must. Klarna an annoyance.
I needed to buy things, because that is life, and the merchant only offered Klarna. You might want to reconsider your hostile rhetoric, it does not come across well.
It at least used to be very easy to accidentally sign up for Klarna, thinking you're just paying by card.
Klarna are the masters of dark UI patterns.
In principle I agree, but you can be tricked into using Klarna. However, in Europe you should be able to cancel the order without reasons.
We detached this subthread from https://news.ycombinator.com/item?id=27301463.
Does that mean it can’t be found on the original thread anymore?
No, it just means that it floats to the top rather than being a child of its original parent. If you're not seeing it, that's probably because the thread is paginated and you need to click "More" at the bottom of the page.
I wrote recently about the different reasons why we do this, if anyone's interested: https://news.ycombinator.com/item?id=27132402.
Whatsapp was fantastic when it started. Right up until it was bought by facebook.
Sofortüberweisung needed your bank credentials from the start.
Huh, from the headline I was thinking it was intentional! Nothing but marketing fluff in the newsfeed yet, still waiting on an article that's not walled in Swedish(?).
Swedish is easy to translate with GT, here's a quick translation of the state news reporting,
"Users of the payment service Klarna's app testify about disruptions on Thursday. Anyone who logs in with a bank ID has in many cases been able to see other people's information, including payments and invoices.
- It is very serious and violates privacy, says David Bjurhede, one of many who noticed the disturbances.
Many who have logged in with a bank ID on Klarna's app have on Thursday morning been able to get to someone else's account, users tell SVT Nyheter.
David Bjurhede is one of those who noticed that it was possible to see another person's information in the app, including what purchases had been made and parts of the account number. - It is very serious and violates privacy and risk of fraud if you can find out user information so easily, he says.
Another user says that he discovered the error at 11 o'clock and that it was possible to take part of other people's information for about 20 minutes. - It was possible to see almost everything, parts of the card details and exactly what they had bought and what their finances look like at Klarna. It's a little scary. I have not been through it before and I think it should not happen, he says."
Junior dev was facing a dilemma.
Before pushing to production please finish this code and choose the id you want to use:
"select * from users where id = ?"
> user_id
> profile_id
> user_profile_id
> profile_user_id
> id
> rand()
I don't think it's nice to make fun of beginners in our industry.
I was not trying to make a joke about the beginner devs. The list of choices a novice developer needs to make is reflective of our industry (why would there be so many choices). It is easy to make an error and bring the whole system down which in turn is the joke about "senior" people who instead of reducing complexity - increase it, and make it fragile.
I _think_ it's intended to be a joke about the IQ test they supposedly administer to applicants.
Interesting that all the screenshots have a (typically) female name, and the reporter seems female. Could be chance of course, but a quite low likelihood if the sampling is truly random... Can’t help thinking what kind of bug could cause that. :)
> quite low likelihood if the sampling is truly random...
If you're assuming their user base is 50/50 male/female, which for many apps is not a valid assumption.
If I remember how to do math correctly, 50/50 gives 5 random users all being female ~4%. And 80/20 split is closer to 40%.
True. My implicit assumption was 50/50 or predominantly male, but I could be wrong of course.
4% is a pretty low likelihood though. Far below the level that would warrant further exploration in this kind of situation.
I find the default Twitter response by the Klarna social media account really annoying. The issue is not a system disturbance. The issue is clearly in the whole implementation of the system itself, code which was written by developers and where something really stupid has been implemented and where security was not taken into account at all because an issue like this could have been prevented at so many layers and yet it happened.
I've seen something like this happen because of a race issue during login. Basically the developer(s) had refactored something and were not aware that a global variable was being captured by a closure used for auth.
This meant that whenever two users signed in at the exact same time, there was a non-negligible chance that they swapped accounts during the flow.
It was actually not that easy to spot in the code. Sometimes what looks really, really stupid on the surface may in fact have a complicated and not-so-stupid explanation, often involving multiple developers and modernizing legacy code.
If it is a race condition, it can be incredibly hard to find during test.
Even if it is a stupid mistake, like e.g. not marking session cookies as secure and private, it does not mean that all of the rest of the code is bonkers.
use of a global variable seems pretty stupid in fact, and easy to spot
> and easy to spot
Not always. Like if you initialize middleware by using a "lambda" (closure), and you from within that closure creates a new closure.
It means that you need to be aware of the context the outer closure is used in. If it is only instantiated once during initialization, it's free variables are in essence "hidden" global variables. Not easy to spot.
Whole implementation? It's probably the edge cache catching a cookie on the way out, a toggle box somewhere.
Yes?
The session layer should confirm and only accept that the other SSL-endpoint is an authenticated app. The app should do this as well.
If a toggle box exists that can cause this, I'd wonder how much of else of the implementation is worth saving.
With all respect, I don't disagree with your assumption about a silly cache somewhere, but that is sort of my point, if such a severe privacy and security vulnerability can be introduced by a single toggle box somewhere then the architecture of their platform is hugely lacking IMHO. This is not a cat photo sharing platform but a fin-tech business and there should be more layers to security than a single toggle box.
I am really really interested in knowing the root cause. I am really concerned by agile, and start-up hipster culture creeping into critical infrastructure companies.
There are so many patterns(event driven, CQRS) in recent microservices architecture, which are gaining popularity and people have been using them without realizing the cons and the need for them.
>agile, and start-up hipster culture
What does that even mean?
Looks like people are really offended by this. Agile lately has been looked at this silver bullet for software engineering. I have worked both in Cisco and some good startups and in my humble opinion having fast paced development and high feature churn rate really is unsuitable for a bank and other infrastructure companies.
Also by hipster, I mean that the banks don't have luxury to experiment with latest trends and the cool tools. They have to stick with the old proven methods.
I don't understand your perspective here.
Debates about Agile have gone on for ages, that's not a 'lately' thing.
I have no idea what 'hipster' has to do with banks and tools... or what you mean by 'old proven methods'.
Not looking for a debate myself :)
Making snarky comments about a common development methodology but not interested in debating the merits of the underlying complaint? I believe that’s called trolling.
@macintux
I did reply and explained my perspective clearly. Anymore than that will just not be constructive. From here on it will be just difference in opinion and no one will again anything.
And if you think I am troll then not feeding a troll is the best thing to do right?
The complaint has been explained:
Agile inevitably leads to shoddy, churn-oriented programming and only serves bullshit consultants and code slingers.
I was less interested in a debate as to what meaning you assign those terms. The way you use them seems like empty buz words.