The Lever Tech Stack

Lever was founded in 2012 with the idea that we could drastically improve on the existing talent acquisition and hiring technology that existed. We wanted to bring user-centric design and a deep product experience to the enterprise software space and help companies solve their most strategic problem: growing their teams.

I joined Lever over 3 years ago as one of the earlier engineers on the team. Since then, Lever has grown a lot as a product and as a company. In this post, I’ll cover our tech stack, why we chose the technologies we did, and how they’ve grown with us.

What is Lever?

Lever’s main service is an “Applicant Tracking System and CRM for candidates”. We help companies handle their hiring processes from end to end — posting jobs, accepting job applications, scheduling interviews and collecting feedback, sending out offers, and analyzing data from the whole process to get better at hiring.

The two main parts of Lever that people interact with are:

1. Lever Postings is the public-facing job site system, which shows job postings and accepts job applications. For example, Lever’s own job site is jobs.lever.co/lever. Companies can also create their own custom job sites using our API.
2. Lever Hire is the company-facing app for managing the hiring process. Recruiter, hiring manager, interviewers, and execs all use Lever Hire. If you’re curious what this looks like, see the following animations:

How does Lever work?

We pride ourselves on making software that promotes hiring as a collaborative and inclusive process and that is a joy to use. We picked a tech stack that helps us build collaborative and fast features.

Let’s start with a diagram (of course):

For the most part, our stack doesn’t look too unusual for a cloud-hosted application — browser clients, various frontend and backend servers, and databases.

The short version

Some of the major technologies you’ll find in our stack are: Node.js, Docker, MongoDB, Redis, and Elasticsearch. I’ll cover these later.

Lever is fully hosted on AWS for cloud hosting. Some of the infrastructure technologies we use to help us manage our stack include Terraform, Chef, Amazon ECS (similar to Kubernetes), Hubot, and Grafana. An infrastructure engineer will cover these in a later post.

There’s one thing in the diagram that I haven’t mentioned yet. What’s up with the red horse icon?

DerbyJS and ShareDB — Real-time collaboration

Lever Hire is a single-page application with a few twists. It’s built on DerbyJS, a web application framework for building real-time collaborative web applications that work like Google Docs. Prior to Lever, real-time data syncing and collaborative edits were fairly rare in enterprise apps.

DerbyJS was created by our CTO & Co-Founder, Nate Smith, together with a few others after working on the Google Search team. In Derby, changes made to something by one user will near-instantaneously show up for all other connected users viewing that entity, without users having to refresh:

Why create a new framework? From Nate:

Working on the Google Search team illuminated the fact that server-side in addition to client-side rendering was important in delivering a performant web app.

The single-page app trend sacrificed page load time substantially. Even in enterprise applications, page load time is important since the web is built around hyperlinks. Links among calendars, emails, task lists, and websites make cloud software even more useful and productive.

I wanted to deliver fast page load time as well as fast interactive single page applications, in a way that could be developed by a small team.

To deliver fast page load times, Derby can render pages either as HTML on the server or using DOM methods in the browser client. This gives us the benefits of both server-side and client-side rendering:

• When a user first opens the app, the browser can quickly display a visually-complete page based on the server-rendered HTML without having to wait for a lot of JavaScript to load.
• Subsequently navigating inside the app triggers client-side page renders in the browser, which are faster and feel smoother than reloading new HTML pages from the server.

This approach of server-side and client-side rendering has been called Universal JavaScript or isomorphic rendering. Airbnb’s engineering blog has a good post that goes more into depth on the benefits of isomorphism. Fast first time rendering, fast client-side rendering, and use of the browser’s History API are key tenets of Google’s checklist for Progressive Web Apps.

The backend of Derby is ShareDB, a database layer that handles concurrent edits from multiple users by using an Operational Transform (OT) algorithm. It also synchronizes data in real-time between multiple clients and servers.

The rest of the stack

The rest of Lever’s stack should look more familiar. I’ll briefly cover those technologies and why we chose them.

Node.js

Node.js lets us run JavaScript on our servers in addition to in the browser. As mentioned above, this lets Derby use the same code to render on the server and on the client.

There’s one other benefit to us from using JavaScript everywhere: For the operational transforms in ShareDB, you need to run the same algorithm on the client and server, and using the exact same code in both places is an easy way to ensure that.

Currently, we actually write most of our internal code in CoffeeScript, which we compile into JavaScript. This means the team only needs to learn one language, and every product engineer at Lever can contribute to any part of the codebase. [2019–08–28: We have started migrating our codebase to TypeScript, which gets us static types on top of JavaScript. We’ll have a series of upcoming posts on the whys behind the switch and our migration process!]

JavaScript does have some problems for large enterprise software codebases, such as a lack of static types, which pushes more work onto the people writing and changing the code. If we were building Lever today, we’d probably use TypeScript or Flow, but they didn’t exist back then.

Instead, to help manage our growing JavaScript codebase, we’ve done things like consolidating business logic into a central library, with a consistent entity-method naming convention like User.addNew for both function definitions and usages, to make them easy to find. We’re also working on implementing JSON schemas for our data, which we can eventually add to our Mongo database.

Docker

Docker runs our server processes in isolated containers. This lets us use Amazon’s Elastic Container Service (ECS) to easily scale up the number of containers that we run as our number of users grows. (We’ve previously covered the scaling benefits of Docker for our browser testing infrastructure.)

MongoDB

Mongo is a “NoSQL” database that holds most of our application data.

Why pick Mongo over a relational database like MySQL or Postgres? I asked Nate about that:

Mongo is a pretty good datastore for the [ShareDB] data model. [It supports] documents of potentially large size, and the tooling is all based around documents of arbitrary nature.

The easiest way to implement OT was to do a generalized JSON implementation. You could have more specific types, but that’s more difficult. It’s also pretty good at storing operation logs for OT. It’s good at write speed when you don’t have a ton of indexes.

There were downsides, like not having joins or schemas at that time. [Newer versions of Mongo have some support for both.]

We also don’t require the strong transactional guarantees that many relational databases provide, as ShareDB implements optimistic concurrency for its commits.

Mongo has served us well, and we plan to continue using it as our main application database. We’ve been able to vertically scale it so far without much issue — it’s nice that we can do no-downtime cluster rotations to upgrade our Mongo servers — and Mongo also supports data sharding for horizontal scaling.

Redis

Redis is a fast, in-memory datastore. One of our major uses of Redis is fairly typical: storing small pieces of data with high read/write volumes like session data and rate limiting info.

Redis also backs ShareDB’s real-time features. Using Redis’ pub/sub functionality means that when one user changes some data, all servers that are subscribed to that data will get an immediate “push notification” about that change, which means changes get seen in real-time.

We also use it to ensure high speed and availability for our link redirection service. It’s important to have fast response times for link redirection, so we don’t want to wait for a write to Mongo before redirecting. Instead, we quickly write events to Redis and then later process the events to update the stats for in Mongo.

Elasticsearch

We use Elasticsearch in addition to Mongo for some of ES’s advanced features: searching, ranking results, and performing fast, complex aggregations.

Elasticseach enables our full-text search on candidate info and resume data. It also powers our recommendation ranking engine, which helps recruiters resurface past candidates for new roles.

Where do we go from here?

Our tech stack has served us well so far. Most of the core technologies in our stack are used by companies operating at a larger scale than we are, so we’re not planning on completely replacing any of them in the immediate future.

Most of the work on our stack in the next couple years will be in scaling up how we use those technologies.

For example, I’m currently working on a partial re-architecture of our task processing service to allow it to scale beyond its current limits, as well as thinking about how to make the service easier to monitor and debug, for both our engineering and support teams. Organizations need to scale, too!

Other people on the team are tackling improvements like implementing schemas for our data or reporting test coverage results directly on pull requests.

We’ll be writing about our work on these topics in the future, so stay tuned for more!