How indiagold rolled out more than 365 releases in less than a year — the concept of feature flag

Having bugs in production is a nightmare for anyone. It becomes even more challenging once your user base grows significantly. On top of this if you are using a mobile app, iOS or Android, you can’t fix any bugs immediately since every release has to be approved before it is available for the public to download. So the nagging questions for any fast-growing engineering team remain:

1. How do you ensure high-quality releases with minimum bugs/disruptions?
2. How do you reduce the technical debt due to code waiting to be rolled out in production?
3. How do you ensure that there is no loss in your delivery speed or reliability as you scale up your team?

Let’s see how we solved these problems at indiagold.

We hit 1M app installs on the Play Store this July. Our user base grew by 5X. The traffic on the platform hit almost 50K RPM during peak time. Additionally, the multitude of offerings that indiagold launched tripled. From being just a digital gold and gold loan platform we entered into new product offerings like gold locker, super-agent program, gift cards for digital gold, in-house LOS, loan management system, lead lifecycle management, field force dashboard & the business app. All of this within 6 months✌. And we achieved this by being super agile and ensuring more than 1 release per day.

While multiple factors contribute to such agility like setting up multiple pods with a dedicated development track, unit testing to increase code coverage and many more — the one thing which worked best for us and still remains one of the most underestimated techniques for any engineering team is feature flags.

The indiagold story

At indiagold engineering, we wanted to focus on two main objectives as we scaled up:

1. Move code into production as soon as possible
2. Reduce technical debt on the engineering team from a code management standpoint

When we started picking up major developments like referral programs for gold loans, gift cards for digital gold, digital gold loan etc — we realised that it was difficult to roll out a working MVP within a sprint and overall product development was spawning across multiple sprints. And until a functionality was fully built and tested, we couldn’t roll out this code into production. We wanted to change this strategy and the idea was that all code should be rolled out into production after every sprint. Hence we decided to implement feature flags.

So what is a feature flag?

Feature flag in laymen terms is putting “if” code block around a feature. And you load the feature conditionally as illustrated below:

if(isFeatureActive) {
   // your feature implementation}

While it seems a simple enough feature it's the actual implementation that makes the most difference.

Implementation

The most simple way would be to put a simple “if/else” code in the block to implement a feature and disable it when not needed. We too did the same and started placing simple if/else blocks in the code. The values were powered by our database. For example:

ModuleFeatureConfig config = configService.getModuleFeatureConfig(
  ModuleEnum.GOLD_LOAN.name(),
  ModuleFeatureEnum.DG_LOAN.name()
);if (Boolean.TRUE.equals(config.getIsFeatureActive())) {
  // DG Loan implementation
}

However, we quickly realised that if we use the above implementation at scale with numerous feature flags, it will make our codebase difficult to maintain. Also, the feature flag logic should remain decoupled from actual functionality implementation.

So how did we improve it?

We improved the above implementation by removing the feature flag checks from our core service methods. Instead, we created a separate feature-based access layer and placed it right at the beginning of our system. As soon as a request hits one of our API endpoints, we use our feature flag service to check access alongside our token and RBAC authentication layers. I have illustrated in the below diagram:

Once the feature flag validation is successful, then only the request is handled by the actual service.

On the client-side, we have exposed feature configuration using an independent configuration API. The clients can use this API to get the status of all feature flags and then assemble and render the UI accordingly. This implementation ensures that the feature flag remains decoupled from our core functionality logic and can be scaled independently.

Additionally, instead of powering these feature flags from the database, we have added a caching layer in between. We are using a high availability AWS Redis cluster for it so that it can scale easily during traffic spikes.

What did the feature flag service enable us to do?

• Feature toggle switch — Since we have a mobile app, there is always a concern about releasing a showstopper bug into production. Now we can release codebase with a high level of confidence since we know we can always disable a feature in case of any critical bug thus ensuring a good user experience. Users won’t run into app crashes and ANRs in such scenarios. And the team can roll out a fix peacefully 😁

• Release track — Another advantage is that now we could enable features for selective user groups. Right now, we are doing this in the form of release tracks — internal, alpha, beta, & public. This allows us to try out a new product launch with a smaller set of users like our QA team or internal team members before rolling out a feature publicly to more than 1M+ users.

• Role-based visibility control — We have added roled based feature access control as well. If we want to enable a feature only for a specific user segment say, priority users, then we can do that easily. Additionally, this can be configured on a per-feature level directly from our database.
• A/B Testing — We have also started using the feature flag to A/B test different variations of a feature.
• Platform or version-specific features — Gives us the flexibility to have platform or version-specific features. This also plays a crucial role in helping us maintain backward compatibility whenever we roll out a new version/iteration of an existing feature. We do so by maintaining a mapping of all features and their visibility across different app versions.

So what's next?

We intend to do a couple of improvements to our current implementation of the feature flag.

1. Phased rollouts — Ability to roll out a feature partially (like 5%, 10%, etc) and also ensure that we incorporate sticky user session i.e. ensure that the feature flag always remains switched on for the users who are part of the partial rollout.
2. Zookeeper or other similar distributed configuration services — if those turn out to be a better fit

Our feature flag implementation has helped us separate feature rollout from code deployment. And in turn, we have been able to improve our shipping speed along with a high level of confidence. This is also a stride for us towards continuous deployment. Hope this gives an insight to those who are looking to implement feature flags and boost their shipping speed. Do leave your comments to let me know if this was useful for you!

Also if this kind of work interests you, apply here— https://careers.indiagold.co/. We are always looking out for curious minds to join our team 😀