Automating Guidelines

guideline - a general rule, principle, or piece of advice

Every sufficiently large organization has numerous guidelines in use, that set and enforce rules and guidance on the processes, ways of working and the standard of quality. Notable examples are Zalando’s REST guideline, Google’s style guides, Microsoft’s REST guide and many, many others. In that regard Tide is no different from all of these companies. Even better, we have significantly reduced the cognitive load on our developers by automating the verification of our REST guideline usage. This empowered our tech colleagues to focus more on the actual business logic.

Here is how we did it.🙂

Why We Chose to Focus on Automating the REST Guideline

One of the cornerstones of Tide’s engineering values is that we care a lot about REST.

Like maniacally!

REST has and will always be of great importance to the organization. That’s why we have gone through four iterations of the API at the company so far. v1 and v2 were created when Tide was still finding product-market fit and trying to prove itself. There were no guidelines on those and they were mostly inconsistent. Then when the company started to rapidly grow its user base, came v3 which now had its own guideline to keep things consistent and easy. Finally, as the company became stable and larger, but still growing fast, we had to take all of the lessons we had learned over the years from the previous API iterations and come up with a v4 guideline.

The v4 spec went through multiple iterations and the community approved and agreed upon it after a month of back and forth. Everything seemed awesome and people were eager to write new v4 APIs. Then the problems happened which lead to the automation part.

• API first — We saw that more and more teams were iterating on new v4 APIs with the front-end or third-party teams by writing the API proposal in Confluence and when green-lighted they would code the Spring controller for it. This led us to believe we should go API first in our services and generate the code from an OpenAPI file, rather than going back and forth in Confluence.
• Guideline complexity — The v4 guideline is widely regarded by our development community as very good and thought out. However, it takes around 25 minutes to read the whole thing and it has numerous rules that must be followed. We quickly discovered that teams unintentionally violated some of those rules because, well, they simply forgot about them and didn’t go back to the guideline for additional checks.
• Company growth — Tide has been growing quite rapidly in the past 18 months both in terms of acquired members and developers. Our goal has always been to enable developers to open a PR on their first day of work, merge it, and deploy it to prod. That cannot happen if they need to learn each guideline by heart first. That’s why we believe we should remove that load and automate the verification for them.

Doing the Actual Automation

Now that we had identified the issues we wanted to solve, we had a clear goal on what we wanted to achieve and how. In short, we wanted teams to write their v4 APIs in an OpenAPI format, validate the new endpoints against the guideline and generate the Spring code for those endpoints using our custom v4 request and response wrappers and security library.

OpenAPI Validation

Our guideline required us to write custom rules for the spec validation. We ultimately decided on Spectral as our tool of choice for the ease of writing and testing the rules, as well as the descriptive error messages and level of configuration and customization available.

Each rule was separated in its own folder in at least three files. The YAML file that did the selection of the element to be validated from the OpenApi spec using XPath selection rules, the JS function that did the actual validation on the previously selected element, and a JS file for the tests and some YAML files for the different test cases.

You can see an example of a rule that forces the API to always contain a Retry-After header when returning a 429 response.

Writing the rules turned out to be surprisingly pleasant and fast. We then integrated that validation as part of our build process as a separate step in our pipelines.

For each service, we also provided a separate file with the definitions of our v4 specific request and response wrappers for developers to use.

Generating the Code

Now that we had the validation in place, what was left was generating the Spring code. What we wanted was to generate interfaces that represented the endpoints, that people would extend and add their business logic. The code would be generated on each build and if they had forgotten to regenerate their code after their latest OpenAPI file changes the pipeline would fail, indicating a non handled change.

For that purpose, we chose to use and extend the OpenApi-Generator tool. What we wanted was to take advantage of the v4 API domain objects library we had built for our custom request and response wrappers, use our internal security library and handle some other development-specific cases as well.

The generator consists of extending the basic Spring generator that the tool provided and tinkering with some mustache templates to change the code to our liking.

The above generator alongside the mustache template changes yielded some really nice results.

Request object:

We then added the code generation as part of the build step in Gradle.

And finally, this is what our implementation would look like:

Moving Forward

What we have described above is still more or less an experiment currently. What we want to see is, if our efforts really prove the hypothesis of these tools helping our developers make less mistakes, offload the cognitive pressure and allow them to focus on the actual business logic needed by the company. Only time will tell, but we believe it’s a step in the right direction.🚀

If you found this article interesting and would like to learn about the other very interesting things we are currently building at Tide, you can always reach me on my LinkedIn profile or apply at the company website. 🙂