Stopping fraudsters by changing products

Lyft’s whole app experience is geared towards getting new users from the App Store or the Play Store to their first ride as quickly as possible.

This streamlined process is great for our users, but presents an ever-present problem — how do we prevent bad actors from abusing our lightweight onboarding process?

This medium post is on joint work from the Fraud and Identity teams @Lyft. Key contributors include Sam King, Will Megson, Steven Liu, Mike Ross, Ryan Choi, Ryan McGowan, Glen Robertson, Adam Wushensky, Donald Chen, Helen Lau, and Siwei Shen.

Goals

Our overall goal is to prevent account takeover, stop scripts and automated apps from accessing our servers, and minimize financial loss due to fraud. We focused our efforts on identifying three fundamental properties:

• Identifying automation or non-human traffic.
• Identifying a returning human — that is, detecting the same human we have seen on our app previously.
• Establishing individual ownership over payment instruments.

All signals that we collect are an approximation of one of these fundamental properties, and when we’re looking for new signals, we cross-reference them against this list.

Architecture

In our architecture, there are three key components: client instrumentation to collect more data silently, a server-side algorithm that decides if an action in the product is suspicious, and a user challenge to collect more information in cases where the algorithm decides that the action (signup, login, or ride request) is suspicious.

This figure shows how we collect additional information to decide if a login attempt is suspicious. In the client, before the user logs in, we pass back to the server a number of signals to try to confirm that the login attempt is coming from the same human that we encountered previously. If our detection algorithm determines that this login attempt is suspicious, we put the user through a login challenge to allow them to prove their identity. Examples of identity challenges include retyping their email address or by entering their driver’s license number. If we can confirm that this person is who we think they are, then we log them in. If we can’t, then we give them an escape hatch where as a last resort they have the option to create a new account.

In this type of architecture, the detection algorithm and user challenge are complementary. An idealized, perfect detection algorithm detects all of the bad actors every time. An idealized, perfect user challenge identifies humans precisely, without adding friction to the flow. If you have the perfect detection algorithm or the perfect user challenge, you don’t need both. But in practice, neither is perfect, so we use both to correct for mistakes in the other.

A low friction challenge enables security engineers to be more aggressive in their detection algorithm because they know that it will adversely affect fewer good users. Conversely, with high precision detection algorithms, security engineers can use challenges that add more friction but more cleanly stop malicious users.

Principles of user challenges

Over the last two years, we have shipped user challenges on login, signup, ride requests, passengers calls, driver calls, updating payment methods, and driver pay outs. Based on our experiences, we have developed five principles that we follow when designing new challenges. Specifically, in priority order:

1. Never lock out good users.
2. Craft the entire user experience around your false positives.
3. Mobile-first design.
4. Challenges should be hard for fraudsters.
5. Numbers always trump philosophies.

The first tenet of designing good user challenges is to never lock out good users. Under no circumstance should you design a user challenge that results in a legitimate user not being able to make progress.

The second tenet is to create the entire user experience with a focus on the good users who we accidentally catch, not on the bad actors. This principle is most relevant when thinking about the overall presentation — don’t accuse your good users of doing something wrong. Be clear about the fact that you made a mistake and apologetic about asking them for more information.

This figure shows a message from Facebook when they block a Messenger message. It is descriptive and gives users a clear path to let Facebook know if it was a mistake.

The third tenet is that Lyft a mobile-first company and should design challenges with the assumption that all of the challenges will be served on a mobile phone or tablet. Pragmatically, this means avoid asking users to enter anything on a keyboard, but scanning something with their camera or clicking on a button are good because they minimize awkward and painful interactions.

Our fourth tenet is obvious: make challenges difficult / impossible for fraudsters — but the point is that it’s the fourth principle; all things good user-centric priorities supercede it.

Our fifth tenet might be the most important one: numbers trump philosophies and principles. If you have an idea, just ship it and see if it works.

Overall when we issue a challenge, some of our false positives will stop using the app. Fortunately, only a small percentage of our users will ever see a user challenge, and we will do everything in our power to maximize the likelihood of good users to pass. However, fundamentally, when you add friction you lose users.

Exploiting asymmetric advantages with user challenges

Challenges are difficult to design because they are highly specific to the product. Designers of challenges need to be able to come up with asymmetric information that is easy to share with good users but hard for the fraudsters to learn.

This section discusses challenges from Facebook, Google, and Lyft to explain the asymmetric information or idiosyncratic feature of the product that the challenge exploits. We speculate on the asymmetric information that the designers of these challenges used, but we describe the asymmetric advantage we exploited when designing and implementing Lyft’s login user challenges.

This figure shows ReCaptcha v2 from Google where they ask users to find all of the pictures of turkeys out of a set of nine pictures. We believe that the reason this challenge works for Google is that they have a massive tagged data set via Google Image Search — something that would be nearly impossible for bad actors to reproduce.

This figure shows how Facebook can ask you to identify your friends in a picture to verify your identity if they detect a suspicious login attempt. We believe that the reason this challenge works is that Facebook has a huge team of PhDs who work on scalable facial recognition algorithms, and they know who your friends are. This technical sophistication and social information would be difficult for an attacker to apply broadly.

This figure shows a Lyft login challenge, where we ask people if they are an existing user. The key to this challenge is that we have an escape hatch: we always let people create new accounts. Lyft has a Growth team dedicated to making it easy to create a new account and take a ride, so we use this smooth onboarding experience as a way to maximize the number of good users who pass our challenge. From an account security perspective, creating a new account is acceptable because it prevents the wrong person from gaining access to someone else’s account, but this escape hatch is something that would never work for Google or Facebook because there is so much data tied to one’s email or social network account.

In general, Lyft has a number of asymmetric advantages that we exploit when designing user challenges. These advantages include:

• Lyft is a mobile-first and primarily mobile company.
• Lyft is a consumer app.
• Creating new accounts is low cost for users.
• The Lyft product uses location data first class.
• Passengers register payment methods with Lyft to take rides.
• Drivers have a detailed on boarding process that includes interacting with humans.
• We know who our good users are after they have used the service for a few months.
• The Lyft use case is constrained by physics (users take rides in a car).

Whenever we think about adding security to the Lyft platform, we consider these advantages and look for opportunities to exploit them — like in the challenge above, where we exploit the fact that creating new accounts is easy and that we’re a mobile first company.

Case study: Login challenges

Login challenges are product flows that we show after the user authenticates successfully. If we see a successful login that our algorithms flag as suspicious, the Lyft app redirects the user to an extended version of our login flow that asks them for more information. Conceptually, you can view login challenges as a form of opportunistic two-factor authentication, where we only use the second factor when our algorithms detect something suspicious. Our challenge flow is based on information that the user should possess or know without setting up two-factor authentication explicitly.

We designed and implemented two login challenges: a “enter the last four digits of one of the credit cards associated with your account” challenge and a “retype your email address” challenge. We designed the four digit challenge to prove that the person who is attempting to login owns one of the credit cards associated with the account. We designed the email retype challenge to use the email address we have associated with the account as a shared secret.

In this system, we use the same detection algorithms for both populations, but challenge some of the users with the “CC last 4” challenge and others with the “retype email” challenge. This table shows our overall results, where the main metric we use to evaluate our challenges is churn — defined as the percent of users who fail to take a ride within two weeks after seeing a login challenge.

The first surprising result was that such a low percentage of users who saw a login challenge churned. Although we are unaware of any published numbers around user churn and login challenges, based on our experience with social networking and internet companies, we were expecting 20% — 30% of the users who saw a challenge to churn.

There are a number of plausible reasons for this difference, including the fact that ride sharing apps are quite different than email providers or social networks, but we believe that the difference is in the two escape hatches we provide for users based on our asymmetric advantages: login with Facebook and create a new account. For login with Facebook, as a consumer company we can assume that people are comfortable with social networks, and we can use this as an additional signal to identify the same human we saw previously. For creating a new account, Lyft’s overall on boarding flow is lightweight and highly optimized, so we can enable one to simply create a new account and unlock their ability to take rides without risking the security of any existing accounts.

The second surprising result was that the retype email challenge performed better than the CC last 4 challenge. Before we deployed these systems, we had assumed that the numeric keypad was a more mobile-first experience and that users would be able to pass the challenge with less friction. However, we discovered that credit card companies will sometimes update credit card numbers — while still honoring existing tokens — so Lyft would have stale data. Also, for people who don’t have the last four digits of their credit card memorized, the CC last 4 challenge ended up being a possession authentication factor where they needed to have their credit card on them when they try to login. In contrast, people are likely to have their email address memorized, so despite the more onerous alpha numeric keypad, people were able to get through the retype email challenge at a higher percentage.

The end result is that Lyft uses the retype email challenge exclusively for this class of login challenges, adhering to our principles. Although the CC last 4 is a mobile-first flow (Principle 3), due to complications around knowing credit card numbers we were effectively locking out some users (Principle 1).

Conclusion

The biggest gains Lyft has made in fighting fraud and improving our account security is through our work on building sound distributed security systems, enabling quick and precise responses to active attacks, manufacturing clever signals, and understanding and changing the product to fend off the bad actors. This foundation serves as the base to enable humans to more effectively identify and stop bad actors, and provides the crucial signals needed to power machine learning algorithms and business rules that predict malicious behavior. Some people believe that machine learning is all one needs to fight fraud, but in our experience, without these basics in place first, the fraudsters have a fundamental advantage.

Interested in having big impact by reducing fraud and improving security at a massive scale? Lyft is hiring!