Source Hydrated Infrastructure Models (SHIM)

Singpass Core Platform

The Singpass Core Platform team is a relatively new team that has been set up to consolidate the implementation of SRE, DevOps, Platform Engineering and Infrastructure Engineering practices into a single team.

Beyond working on incident response training, Continuous Integration (CI) and infrastructural projects, the team has spent the past year building out standardised Continuous Delivery (CD) automation for various Platform solutions. Since the formation of the team in early 2025, we took inspiration from Google’s Prodspec and Annealing, and sought to build a CD pipeline that could replicate the benefits without complex tooling:

• Code versioned infrastructure asset models from different platform components beyond just Kubernetes deployments in a single unified location
• GitOps-based audit trail where changes to infrastructure were tracked in Git logs
• Infrastructure asset models that were constantly reconciled from a unified location straight to various infrastructure deployments in different environments.
• Explicit, verbose, and raw infrastructure model assets versioned for easy debugging for a fully rendered desired deployment state.

Internally, we colloquially named this pattern the SHIM, as it generates the final translated state wedged between the abstracted tenant intent and the deployed infrastructure.

The SHIM pattern was later designated a backronym Source Hydrated Infrastructure Models.

In the following sections we will explain how we came to this design conclusion, and its pros and cons.

State of Declarative Deployments

The fancy jargon that is Infrastructure-as-Code (IaC, or Anything-as-Code nowadays) came about during the late 2000s, but the concept predates it by a decade or more. While most people correlate it with Terraform or Pulumi, early declarative configurations of infrastructure can already be found in tools like CFEngine. In recent years, declarative configurations have advanced further with Kubernetes CRD manifests, Nix for NixOS, Ansible (albeit an imperative approach to declarative modules), Terraform/OpenTofu, Pulumi (another imperative approach to declarative modules), and Crossplane.

These tools have adopted different approaches to infrastructure deployments. But in general, they all behave in a relatively similar manner:

1. Version controlled declarative intent
2. Intent pushed to stored state
3. A reconciler or solver observes the actual ground truth of deployments and solves the delta between the intended state and actual ground truth

The main difference between these tools is mostly the trigger: Explicit vs Implicit triggering of the State Modifier and Deployment Reconciler.

Refer to the Appendix for more details on how each declarative configuration tool handles explicit and implicit triggers.

Inherent Problems With Current Implementations

Stored State Drift Requires Explicit Reconciliation

For tools like TF and Pulumi that require explicit state and infrastructure updates, there constantly exists 3 versions of the deployment state:

1. Source versioned intent
2. Stored state intent
3. Actual deployment infrastructure

Due to the explicit nature of triggers for these tools, the intent of the configurations are never automatically reconciled with the deployed infrastructure. This means that infrastructure owners are never truly certain of:

• What the current desired state of deployment is
  Yes, the source versioned HCL says one thing but if someone bypassed it to modify prod, was that for an incident? Or was it bad engineering habits?
• What the version of the current stored state is
  Applied infrastructure holds no state version unless automation is built to add resource tags. The statefile in a blob storage is not tied to a specific apply action either.
• Whether the deployed infrastructure has been directly tampered with, and for what purpose.

The SREs or DevOps or Platform Engineers are left to sift through SIEM logs and audit logs to figure out how the drift came to be, and if it was intentional. This is why the industry practice has slowly shifted towards implicit reconciliation loops.

Difficulty With Reconciliation Debugging

VCS trunk/branch-based deployments that are pushed in a reconciliation loop have existed for a while, notably popularised by Git-based reconciliation workflow tools like ArgoCD and FluxCD. The tools do regular polling of changes in a source repository, and triggers a respective workflow based on the nature of the change in a new commit.

These tools are generally used in tandem with other abstraction layers like Helm charts and Kustomize to reduce the verbosity of the configurations.

While nifty, the actual CR manifests being pushed into the Kubernetes etcd are rendered and pushed in-flight. If the etcd push or CR rendering takes too long or fails before a second commit to the abstracted configs is done, we end up with a similar problem with state drift but worse:

• The state intent is code versioned in the abstracted configs but not pushed
• There is the rendered CR manifest in ArgoCD in-flight workflows that now needs debugging to see which commit version is being processed
• There is the etcd stored state intent of CRs that may or may not have been updated
• And then there is the actual ground truth of infrastructure deployment state

It is difficult to pinpoint which version of configs was rendered, which resulting manifest was rendered, whether the rendered manifest was actually pushed, and what state exists in etcdduring a deployment failure. The problem is so pronounced that Kubernetes platform owners call it “Helm Hell”, and ArgoCD is now testing a new Source Hydrator feature to address it.

Complex Reconciliation Loops

Crossplane is arguably the single tool that is most aligned with the spirit of Prodspec. It truly unifies the definitions of all kinds of infrastructure asset models into Managed Resource (MR) YAML manifests in the same way Prodspec unifies configs from all kinds of infrastructure assets into protobuf structures for Annealing to pick up.

However, while we love GitOps reconciliation deployments, Crossplane’s controller reconciliation pattern poses its own complexities.

Its implementation requires a separate bootstrapped Kubernetes cluster, and introduces additional layers of reconciliation (5 layers deep now!) debugging complexity. The generated MRs are also created in-flight and stored in a non-versioned etcd , making intent debugging painful as before.

If our team were larger and feature delivery timelines were further out, we might have considered converging on using MRs as our SHIM asset model with a MR querying wrapper. We ultimately decided against Crossplane due to concerns about operational maintainability by a relatively small team.

The SHIM Solution

Is there a way we can achieve something like Crossplane where we unify infrastructure asset model declaration formats, and store them in a central location without having to bootstrap a snowflake K8S cluster?

Is there a way we can define these unified infrastructure asset models in a verbose manner of deployment intent for easy CD debugging, while still enjoying the benefits of an automatic reconciliation loop?

The method that Google Prodspec went with was to have distributed pipeline generators compile all abstracted configurations into explicitly verbose manifests at regular intervals and store them in a Spanner database for Annealing to reconcile. The incarnations stored in Spanner can be queried through a query server to immediately know what the current intent being reconciled is, making production intent debugging extremely convenient across large fleets of millions of infrastructure assets.

We are acutely aware that we do not have any requirements anywhere remotely close to the scale of Google to warrant building our own queryable Prodspec database. But we do already possess an “incarnation” tracking tool that almost any developer knows how to use: Git. We can essentially use Git for the “Stored State” and a reconciliation loop that addresses the lazy reconciliation of TF/Pulumi solutions.

In fact, this idea is not new. The Kubernetes ecosystem has had the Rendered Manifest Pattern coined for this practice since late 2023.

In practice, we separate the context structure into:

• Tenant control plane
  This is where a platform specific solution (container workloads, database schemas, Kafka clusters and topics etc.) exposes control to tenants via DSLs, config templates, or both
  Internally we structure the Git repositories similar to the following:

//cfg/container
  ↳ tenant/
    ↳ project/
      ↳ service.cue
//cfg/psql
  ↳ tenant/
    ↳ project/
      ↳ database.sql
      ↳ manifest.yaml
//cfg/kafka
  ↳ tenant/
    ↳ project/
      ↳ example.cue
// etc.

• Platform operator reconciliation plane
  This is where the rendered, verbose manifests reside.
  ArgoCD watchers of each platform repository detect changes and render the raw manifest files to commit to the respective namespaces.
  Since the tenants have no access to the mechanisms here, we can flatten the hierarchies for easy partition querying:

//cfg/shim
  ↳ namespace1/
    ↳ containers.yaml
    ↳ psql.yaml
    ↳ kafka.yaml
    ↳ etc.
  ↳ namespace2/
    ↳ containers.yaml
    ↳ cql.yaml
    ↳ nats.yaml
    ↳ etc.

The namespacing allows for easier tenant infrastructure change audits.

Infrastructure plane
This consists of the ArgoCD reconciler. It reads the raw manifests and pushes the changes as is without any further modifications to the YAML manifests.

The automated SHIM commits will all include metadata in the commit descriptions:

Source-Repo: <repository_url>
Source-Config: <file_path_or_name>
Source-Branch: <branch_name>
Source-Commit: <full_commit_hash>

This approach now gives us a searchable, authoritative source of truth for our deployment intent without any obfuscating abstractions occurring between the declarative intent and the final reconciler.

Drawbacks of SHIMmying

While this approach manages to achieve our goals without having to bootstrap any additional infrastructure, the drawbacks are significant.

Trading Workflow Debugging Complexity for Repository Complexity

In the process of versioning and making manifest reconciliation explicit, we have introduced a Git repository sprawl. We are forced to choose between a monolithic repository approach and risk having constant rebase merge conflicts, and tenant specific repositories which makes RBAC synchronisation across all user SHIMs a huge challenge.

Slower Deployments

By introducing a hydration phase to create the SHIM before actual reconciliation takes place, we incur double the CD pipeline duration.

Complex Tool Chain

Even without the complexity of Crossplane, running ArgoCD for different purposes still makes debugging somewhat complicated. Instead of wondering what actual manifest is pushed, we now wonder how the manifest gets generated in the source hydration phase.

Shoehorned “Raw” Formatting

Not all infrastructure asset models fit a generic YAML manifest. Forcing potentially incompatible infrastructure types to be defined in a verbose YAML may not actually work well for a direct deployment. There may be yet more magic taking place behind the scenes to make it work.

Accidental Orphaned Assets

It is common to accidentally delete infrastructure assets when the tenant control plane misses a loop or drops an array element by mistake. We made an active choice to orphan the infrastructure asset instead of haphazardly turning it down like in the TF paradigm. Our team disagrees with the design choice made to delete infrastructure by omission in Terraform.

Deletion should be an explicit intent in and of itself by way of a delete tag added to the rendered manifest, omission should only indicate the intent to untrack.

However, this does mean that if the deployed assets are not re-tracked by the SHIM manifests, it continues running forever until manual garbage cleaning takes place.

Confusing Anti-Override Requirements

As the SHIM manifests are the authoritative source of deployment intent, any manual break glass operations done during a bugfix or incident can be overridden by the manifest reconciler. We would have to either build some asset locking mechanism during a manual intervention, or automate the toggling of ArgoCD per-asset autosync.

The Path Forward

The SHIM pattern is not a silver bullet. By choosing to prioritise visibility and an explicit audit trail over deployment speed, we have intentionally traded one set of problems for another. However, for the Singpass Core Platform team, this trade-off is worth it.

In a critical infrastructure environment where “who changed what and when” is the difference between a five-minute fix and a five-hour post-mortem, having the fully hydrated state living in Git provides us with a higher level of confidence than “in-flight” abstractions. We are no longer debugging the process of deployment but auditing the intent of the deployments.

—

Appendix

Different Declarative Configurations

Kubernetes CRD Manifests

Kubernetes stores the state of its infrastructure within its etcd cluster, from which K8S operators observe and automatically reconcile with the deployments (we use “deployments” here to mean “deployed infrastructure”, not to be confused with Kubernetes native Deployment).

The push of manifests into etcd is an explicit action, whereas the operator reconciliation from etcd to infrastructure deployments is implicit.

K8S GitOps With ArgoCD / Flux CD

ArgoCD and Flux both observe changes in a branch of a Git repository to automatically carry out an implicit push of manifests into the Kubernetes etcd.

NixOS

NixOS affects only the OS layer, and not general infrastructure like the other tools (ignoring the Terranix project here to stay focused). Nix configurations are compiled into an intended state of OS configurations that are kept in a locally versioned Nix Store from which a state deployment is merely a symlink pointer update.

Both the push of a new version of Nix Store configurations and the updating of the symlinks are explicit.

While Nix is similar in declarative concept, its target use case is solely an OS host level configuration tool, and will be ignored for our discussion.

Ansible

Configurations are templated into declarative modules that are essentially wrappers around cli commands. The runbooks imperatively invoke the wrapped commands, and sequencing between the modules still matter. There is no true state control as runbooks dictate the desired state upon execution and all deployed assets are overridden. Actions are executed at a per machine level since they are just cli commands. The idempotency of Ansible heavily depends on how the modules are written.

Ansible uses explicit triggers.

We would be ignoring Ansible for the SHIM discussion as its design introduces several reliability issues:

• Lack of True State Control
  There is no persistent “state” to check against. Runbooks simply dictate the desired state upon execution, causing all deployed assets to be overridden.
• Uncertain Idempotency
  Because it relies on wrapping CLI commands, idempotency is not guaranteed and depends entirely on how individual modules are written.
• Manual Execution
  Actions are executed at a per-machine level and require explicit triggers.

Terraform / OpenTofu (TF)

TF declarative HCL files are a special bunch. The source versioned HCL files are used to generate a verbose state file that is typically stored in a blob storage solution like S3. However, TF-based reconciliations are lazy where the delta being pushed to deployments are solely calculated between the HCL and stored state file. To prevent state drifts between Intended State and Actual Deployed State, a refresh phase is carried out before any apply to ensure that the state file is always kept up to date.

All of these triggers are explicit in nature.

Pulumi

Pulumi is an imperative wrapper around declarative provider modules. It uses programming languages of the users’ choice to programmatically generate the declarative state, and enables the use of conventional language tooling.

Similar to the TFs, its triggers are explicit.

Crossplane

This is a Kubernetes operator running in a bootstrapped cluster leveraging the Kubernetes reconcile loop of CR manifests to automatically reconcile Crossplane provider assets with infrastructure deployments. It “hacks” the K8S control loop pattern to generalise resources beyond Kubernetes assets.

The triggers are implicit.

In terms of the Google Prodspec and Annealing principles, Crossplane is arguably the most similar in design in the way that it standardises infrastructure of all kinds into standardised Managed Resources (MRs) for implicit reconciliation.

Frequently Asked Questions

Is there a simpler way to do this? Having 2 ArgoCD layers seems complicated.

We have thought about removing the ArgoCD layer for source hydration and having the native Git CI tooling (e.g. Gitlab CI / Github Actions) perform a push-based hydration upon trunk merge. However, this ties our CD pipelines into our source versioning tool. We would prefer having CD pipelines be portable across Git services.

There is also the option of using Crossplane Claims as our DSL (XRD for schema validation is great) and Managed Resources as our SHIM, but the layered tiers of reconciliation felt overly complex. For visibility of model asset intent, we would have to adopt yet another tool like Crossview to observe the SHIM assets. This is still a viable option if we find ourselves prioritising the CD optimisation in future.

Can the reconcilers be turned off during an emergency?

Yes. The team has standard operating procedures and automation to lock assets in case we had to do intervention during incidents.

How would you monitor for orphaned assets?

Since the Core Platform team owns all of the infrastructure assets in Singpass, we have the unique privilege of being able to easily monitor the delta between all deployed assets and the SHIM repository.

Nix is great! Why didn’t you consider Terranix or Kubenix?

Nix takes time to grok. The team is young and ambitious with our delivery timelines. Adopting Nix is going to add cognitive overhead for us.