We recently helped Kickstarter move their CloudFormation workflow over to Cog. Our initial focus was on making the same sorts of commands that are available via the AWS command line utility available in Cog. Moving existing CloudFormation commands to chat immediately provided some improvements: team visibility, shared configuration, and fine-grained permissions, just to name a few.
We always knew that duplicating the functionality in chat was only the first step and after using these commands in practice,
and optimized the workflow for chat by introducing some new concepts. The main problem was that requiring users to provide all inputs when creating stacks quickly becomes cumbersome, even for only moderately complex CloudFormation templates. The solution was to decouple stack specification from stack creation through two new abstractions: default sets and stack definitions.
Before we get into the details of how to manage stack definitions, here’s what creating a stack now looks like:
Press enter or click to view image in full size
Stack Definitions
Instead of providing the template, parameters, and tags all directly to the stack create command, we bundle all of these up into a definition, which can then be used and reused with stack create and changeset create commands. This allows your team to split up the responsibilities of defining stacks and creating them. You could even only grant the permission to create stacks to a small subset of users, but allow anyone to define a stack.
Default Sets
Even with stack definitions, it would still be fairly tedious to define each parameter and tag as arguments to the definition create command, especially when many of these parameters like VPC IDs are often shared between stacks. To simplify this we created default sets, named sets of parameters and tags which can be reused across stack definitions. They’re meant to provide a base set of parameters as you can easily override specific parameters and tags when creating the definition or even when creating the stack.
Versioning Your Infrastructure
One thing we haven’t touched on is where these definitions and default sets are stored. We strongly believe that ChatOps must fit into your workflow, not be another silo. With this firmly in mind, we designed the cfn bundle to integrate with the place where engineers already store configuration and automation artifacts: Git repositories.
Get Patrick Van Stee’s stories in your inbox
Join Medium for free to get updates from this writer.
We allow you to configure a git repository which will be the canonical store for all of your templates, defaults files, and stack definitions. Best of all, the integration goes both ways. Not only are files you edit and commit outside of chat available to the chat commands, when you create a definition or default set in chat it is also committed to the repository.
And taking it a step further, stack definitions are intended to be immutable. Rather than updating a definition, you create a new one with the same name that gets its own timestamped directory which includes a point in time snapshot of the template, all parameters and tags from defaults, any override parameters and tags, and a finalized set of parameters. You can be confident that you know exactly what has been deployed to production, which should also make auditing and debugging much more straightforward.
Learn More
If you’d like to hear more about the AWS CloudFormation bundle for Cog, Mark Imbriaco put together a 6 minute screencast which demonstrates the basics of the bundle in action.
We intend to publish additional content about ways to make use of the bundle in practice, tips for managing access control and delegating access to specific stacks via Cog rules, and more over the coming weeks.
All of this is now available in the 0.5.0 version of the cfn bundle. We’ll be continuing to add to and improve this workflow as well. So let us know what is working for you and what aspects of using CloudFormation are still a pain.
If you’d like to know more about using Cog with CloudFormation or are interested in exploring ways that Operable can help you be successful with ChatOps, please get in touch!