Converting Full Terraform Programs to Pulumi
pulumi.comI really dislike pulumi's pricing structure. I absolutely hate the idea of paying credits for each resource that I create - it creates resistance to experiment and just create a bunch of resources that don't cost anything on the cloud provider yet somehow costs me money for the tool to manage them.
It's funny because I believe they used to have a per seat licensing and then people complained that they were paying too much for large orgs with a few resources.
Anyways, nowadays I just use aws-cdk. I like the freedom of scripts and the extrme power that comes with them. It is vendor specific, but nothing is really cloud-agnostic, there's always some requirement to have some understanding of the platform's terminology and nuances so this point is pretty moot. It also requires a little bit more planning than just writing something delcarative and letting something figure out the incremental changes, but I like it because it lets me better understand what's happening without runaway complexity.
You can use Pulumi for free. Also, the incremental cost for each resource is minuscule. Unless your experiments require thousands of resources, it really shouldn't be a big deterrent. Just clean up when you're done experimenting.
Sort of. You can use Pulumi for free as an individual. As a team you'd have to use a backend for your state other than Pulumi cloud, and your responsible for your own concurrency control [1]. In a team environment that's such a big foot gun I don't really consider it usable.
That limitation makes Pulumi a nonstarter for me. Even if I am paying for it, this is super sensitive data and I want control over how it's stored. I'm sure Pulumi cloud is excellent, but I hate it's the only real option.
That’s wrong, you can use Pulumi as a team for free so long as you stay under the 150k free credit limit. You don’t have to manage your own state backend.
I’m using the free Pulumi tier at work, we have three EKS clusters, 3 RDS instances, about 6 ECRs, probably 10-12 EC2 instances, various k8s resources, VPCs etc and aren’t paying anything yet.
I do love the product and will be happy to pay when we reach that level.
I've not pushed the concurrency at all but I think it creates a lock file in storage which gives basic concurrency management.
The self hosted backend does have plenty of wringles though like not supporting the same stack name across different projects with the same backend URL contrary to how the hosted backend works.
Nonetheless it has been perfectly usable for us without paying for their hosting.
> The self hosted backend does have plenty of wringles though like not supporting the same stack name across different projects with the same backend URL contrary to how the hosted backend works.
Just a note that we did add support for projects to the self-hosted backend a few months ago, which does bring the self-hosted and managed backends into alignment here.
https://www.pulumi.com/blog/project-scoped-stacks-in-self-ma...
The Terraform equivalent is pretty good. You can just use DynamoDB for concurrency control, S3 for state control, and it's very easy to set up. I imagine Pulumi is not very different on this front. Very little boiler plate and you're off to the races.
I like Pulumi a whole lot, but my org did end up needing to move to hosting our statefiles on our own S3 backend and managing that ourselves when the cost became prohibitive. We were a small org (<20 folks) with tens of thousands of resources, and we just couldn't justify the costs even though we liked the Pulumi hosted app quite a bit.
Note that terraform have the same pricing structure now: https://www.hashicorp.com/products/terraform/pricing
I've been happy with Hashicorp Terraform Cloud and HCL except for the very recent price hikes which may force us to leave them so we are keeping our eyes out for alternatives.
The AI assistant seen here is a very good idea. I will actually share this with my team.
I imagine the terraform conversion doesn't work when you use in-house written providers (in golang), is this correct?
You would need to bridge the Terraform provider to a Pulumi provider first. If you've written a Terraform provider, bridging to a Pulumi provider should be straightforward.
If you need help with this, please reach out to me on the Pulumi community slack at https://slack.pulumi.com and happy to help
Hey!
I recommend checking out Spacelift[0]. It's a CI/CD system specialized for Infra as Code, including Terraform and Pulumi. We have both a ton of additional features, as well as a much more reasonable pricing model, so we've recently been talking a lot to folks wanting to migrate from TFC due to the changes you've mentioned.
You can easily reach out to us on the website by scheduling a demo, using the chat widget, or just sending me an email (see my profile).
[0]: https://spacelift.io
Disclaimer: Software Engineering Team Lead at Spacelift, so take this with a healthy grain of salt, but I do legitimately think it's a great product.
We're building our new infrastructure with Terraform - and experimenting with postgresql for state. So far so good, but we have only a handful of resources.
Bit of a catch-22 to setup managed postgres for state, tailscale network with a vm as exit-node (to access postgres via VPN/lan - not Internet) - via Terraform - and then switch state from local (initial setup) to postgres.
But nice to be able to eg: interact with a different provider for DNS, supplying IPs from other resources.
Fortunately there are a lot of other terraform deployment and state management systems, such as Spacelift and Scalr.
If devs write and maintain their own IAC then Pulumi is a good choice since they can write it in a familiar language.
If however a dedicated devops team has to manage it eventually, and there are multiple dev teams who all use different languages, then Terraform is better, since it’s practically impossible to learn all these languages. Terraform will give you a dumbed down enough lingua franca.
Also Terraform gives you the advantage to keep code complexity to a minimum. No chance to find some “clever” self referencing function or other language specific crazy stuff to solve a problem in it, it forces you to keep it simple.
> If however a dedicated devops team has to manage it eventually, and there are multiple dev teams who all use different languages
How is this a disadvantage?
If you can mandate Terraform and HCL across the org you can mandate Pulumi and a language eg Typescript. What’s the difference?
You can also use Terraform with a programming language ie CDKTF so the Terraform choice doesn’t make it simpler.
Under the case that “there are multiple dev teams who all use different languages” as stated, they have the advantage with TF that it’s an easier to learn DSL than Typescript.
CDKTF is totally optional and besides the point. Could as well say you can use Pulumi with Makefiles so the Pulumi choice doesn’t make it simpler.
> CDKTF is totally optional and besides the point.
Eh? Then Terraform is optional too? You can do the reverse and say HCL is optional. Who decides? You're thinking from a Terraform 1st perspective.
> Could as well say you can use Pulumi with Makefiles so the Pulumi choice doesn’t make it simpler.
Except Pulumi doesn't support makefiles? It does support yaml.
So yaml is a simpler DSL that people know than HSL, so it still wins by your logic? Makes no sense. UI is simpler to learn than Terraform. Might as well do ClickOps?
> “there are multiple dev teams who all use different languages” as stated
And so you can't pick a language that is used more often? It doesn't have to be Typescript. So if >30% of the devs use 1 language it already has that advantage over HCL.
>You're thinking from a Terraform 1st perspective.
You can use Terraform without CDKTF.
>Except Pulumi doesn't support makefiles?
Everything and nothing support Makefiles ;-)
> So yaml is a simpler DSL
Yaml isn't a DSL. It's just that 5000 different projects use it in 5001 different ways to express something randomly (e.g. Ansible) INSTEAD of creating a DSL.
> pick a language that is used more often
I would argue that Terraform is among devops people the most widespread tool for this purpose, so you'll have an easier job finding staff searching for people who know terraform than to find people knowing Pulumi + your mandated choice of language. Remember this is for the "central devops team supports multiple dev teams" scenario - for the other scenario of devs running their infra code themselves I've already said Pulumi makes a lot of sense, and then you don't need to mandate anything, and everyone will be able to reinvent their square wheel independently - until they're bought by a bigger company which has a dedicated SRE team that takes over Prod responsibility from the devs, and you'll find them searching for a tool to convert Pulumi into TF ;-) .
Can Terraform be used in place of ansible? I've seen way to much "logic" in ansible playbooks.
Lol exactly - “logic”
Terraform can’t replace it well. Maybe facilitate the transition to more cloud native setups where you don’t need Ansible at all. But only within limits. If you’re asked to implement something there’s no cloud offering for, and it’s slightly more complex, you’re back to ansible.
I can't do clouds except if the cloud itself is self-hosted. I'm always working with minimal hypervisors based on something like kvm/libvirt. Then I can do all the cloud stuff on top of that and boot vms with cloud-init. But over time a lot of ansible playbooks have accumulated and ideally I would like the entire state of all the infrastructure to be in a git repository. It's possible with ansible but I think it'll break in no time ;(
I love the idea of IAC as actual code. Too many times I've looked at yet another over engineered templating engine and thought, why couldn't this just be code?
The reason for not doing a full programming language for that is that you can prove the infrastructure is finite with a simple template language, but you can’t if your template language can do conditional branching.
Or something like that. My CS-fu is admittedly weak.
OTOH, I think Lisp would be the perfect choice for IaC.
It's a valid point. Although I think most of these templating languages are probably Turing complete anyway. We should make better use of Turing limited languages.
I think reinventing the wheel every time a for loop is needed in a resource spec makes things unnecessarily tedious for everyone.
IIRC, Terraform was not Turing-complete at the start, but users demanded features and that’s how it ended.
The idea to start from a programming language is that, since you are going to end up Turing-complete anyway, at least you make sure you don’t end up with a programming language with an awkward syntax.
We use Terraform but mostly because of its ecosystem. I was always tempted to use Pulumi, especially because they got something right from the beginning: pure declarative code is an illusion.
You always need loops, control structures, etc. and this is automatically a feature in Pulumi as it is in the language you use it with.
Terraform users hack it all the time (think of the "count" as if/else, for instance) precisely because it's never a purely a declarative config what you need, and the level of dynamism you need is not been semantically supported by Terraform.
So, we are stuck with hacks...
Honest question, if you're locked in to AWS why not just use the CDK?
I've used Terraform and Pulumi in the past and the "platform agnostic" is only true for trivial builds. Once you get into medium sized infra you're writing so much AWS specific code it stands better to go 1st party.
I can't speak for GCP, but this is what happened to us. We kept fighting 3rd party code we ended up going CDK. While there are still issues, there were less issues. Cloudformation manages the state under the hood anyway so we're all just stuck wrapping that sucker, even in the CDK.
I have found Terraform or Pulumi to be very convenient even with a single cloud because they offer many cloud-agnostic features as well.
- random provider to generate a db password, cloud provider to provision db and admin user, random and mysql providers to provision additional non-admin users, k8s provider to upload credentials to secrets
- tls provider to create ca, k8s provider to create namespace, create certs for each k8s namespace, upload to k8s secrets
- Cloud provider to issue service account key, GitHub provider to upload to GHA (don't do this anymore since oidc is supported, before it was rather important)
While I haven't used CDK much, I believe it is still basically about provisioning aws resources and would not have such cross cutting configs, though I think they may have had a mechanism for importing tf providers.
Terraform can have its issues but overall being able to provision such diverse resources with a single command has been great for onboarding/reducing human error. I'm sure there are plenty of cases where CDK is easier too, these are just to demonstrate why you may use TF even when locked in.
When I switched from Azure to AWS I just had to learn AWS specifics and then could use my existing TF knowledge to put it in code. No need to learn CDK as well. If I ever end up working in GCP, it will hopefully be the same.
If you solely work with one cloud and don't intend on switching cloud providers or working in a context where multiple cloud providers are used, then by all means specialise. Personally I'm working as a software engineer, so I don't really care in which cloud it runs, as long as I can provision whatever resources I need.
If you were using CDK you’d switch to bicep which also uses code instead of config files.
For anyone else out of the loop, that appears to be:
https://learn.microsoft.com/en-us/azure/azure-resource-manag...
It's not about being vendor agnostic, but being able to manage resources outside of AWS. in my use cases, our platform integrated GitHub, Auth0, AWS, kubernetes and so on. Everything managed via Pulumi, what would not be feasible if we were just on Cloudformation.
When trying to achieve a cloud agnostic approach, there are some other tools focused on that (such as serverless framework).
Because you can use Terraform not only for infrastructure. In our case, we use it also to setup GitLab and other services.
I absolutely love the CDK's typed approach. Having said that, I still would choose Terraform over CDK mainly because of the way they handle drift detection / drift resolution. Basically, with CDK and the underlying CloudFormation, you can get told that there's a drift. And that's it; now you have to go and fix it manually. Someone deleted half your infra? Well, have fun rebuilding it by hand. With terraform, you just apply again, and it fixes the drift.
There are a number of other subtler topics as well. In my experience, Terraform tends to be noticeably quicker to run (when the limiting factor is not some AWS service that's spinning up for ages - but usually that is not the case after the first run). And just because you're mostly vendor locked, doesn't mean you don't have a few 3rd party vendors to also integrate; it's nice to have a wider support for these providers. While the CDK's types are more consistent than the Terraform ecosystem and generally lead to faster prototyping, the type system itself only protects you from a small subset of errors. You can still wire things together that will never work, and you'll only find out at deploy time. Lastly, I've often found the CDK's versioning to be fairly odd. I get the idea behind stable and experimental packages, and the need to be explicit about these. However, it seems odd that certain packages spend _years_ in experimental, despite being absolutely mainstream from the very beginning (I am looking at you, API Gateway V2 - though it's been a few months since I last checked).
(OK, so this was a bit more tongue-in-cheek than I normally go for. I'm somewhat frustrated by CDK because it has so much going for it, that when it disappoints me, it really hurts.)
> And that's it; now you have to go and fix it manually
This is very true. We didn't get far enough to get into a drift issue to compare CDK vs 3rd party, so this is new to me, thanks for that. Admittedly, we solved this by saying no one is allowed to click buttons at all on the console for certain envs.
What's worked well for us is we multiple sub-accounts which handle dev(s), stage, uat, prod. Dev(s) accounts are pure temp, use the CDK to stand it up, then click all you want while you "figure it out".
Stage, UAT, and Prod are all code only.
We also separate Network vs Application stacks, but it's more actually delete-able vs critical. Dangerous to delete/mess with - Route53, RDS, VPC, S3 (some). Delete-able is Lambda, EC2, ElastiCache, etc. If we lost these, we don't "lose" anything other than downtime no backups/cust data gone and restoring it is trivial.
Also agreed on the experimental packages or needing to use the escape hatches. We have L1, L2 mixed all over the place. We use almost no L3 because it's like Hello World - great for demos but not really practical and it's easier to customize the L2's because getting access to individual resources in L3 is janky (but we do do it for a few L2's).
CDK is the correct answer.
My biggest issue with Pulumi is that it doesn't have all of the guardrails/tools (like tfsec, tflint, and pavlov) to prevent newbies from doing bad things. I highly considered it for my team's project, but they are not mature enough developers for me to trust them not to massively screw things up or spend forever doing fairly simple things.
One of the big benefits of using an actual programming language rather than HCL is that it allows you to define abstractions / use the type system to enforce these invariants. I find this massively preferable to something like a linter in terms of being able to provide a nice API to engineers who might need a little more help defining their resources.
Or you could also use the cloud provider's api from a real programming language, or generate terraform code from it.
After having tried Pulumi I felt it does not bring the claimed pros, you need to write code in a non-idiomatic way, and coming up with abstractions for the representation of the stuff you talk about takes more time than just write proper terraform, and perform proper thorough reviews, which also facilitate knowledge sharing in the organization.
Also then you'd have 3 layers as sources of bugs, instead of just one: your code (you tested it properly, didn't you! How much time saved as opposed to just doing proper reviews!), pulumi itself (has faced its bugs on such a simple task as setting up an aws lambda), and terraform it uses under the hood.
In my experience pulumi doesn't solve problems for the business, but lets engineers waste time and money on playing around with their NIH abstractions. We tried it, and dumped it, and went back to terraform, which just works.
Oh, I agree! I use pulumi for my own personal projects, but I don't have the time to build a DSL on top of it pulumi just to establish the same guardrails my team needs that I'd otherwise get for free.
A real language also presents more foot guns no?
In HCL it’s difficult to break repeatability, for example by querying external sources in an uncontrolled way.
I've used Pulumi for a few small projects. I quite like it and I can see a Terraform importer being useful (for example GCP provides comprehensive Terraform support and being able to convert those to Pulumi quickly would speed things up a lot).
Do these guys need a state file like terraform does? It's great that you can use a normal programming language, but that's nowhere near as big a pain point as state files getting out of sync. Not even close.
I'm pretty sure they do, which is a feature, IMO.
We're coming up on 10000 resources in our main Terraform repository and while there is definitely some friction, it's overall much better than having to hit the cloud API's to gather each of those states which would probably take at least an order of magnitude longer.
We also just recently started setting up a periodic drift detection build to help identify and address drift.
Could you elaborate on the circumstances that have caused your state sync issues?
> We're coming up on 10000 resources in our main Terraform repository and while there is definitely some friction, it's overall much better than having to hit the cloud API's to gather each of those states which would probably take at least an order of magnitude longer.
I don't think that's necessary true. Most cloud API's actually can return hundreds of records with 1 API calls, e.g. https://docs.aws.amazon.com/elasticloadbalancing/latest/APIR... has a maximum page size of 400.
If I manage the cloud resources via some custom tools and/or with some ansible-fu, I can decide to batch the API calls when it makes sense.
With terraform, it is not possible to do so (https://github.com/hashicorp/terraform-plugin-sdk/issues/66, https://github.com/hashicorp/terraform-provider-aws/issues/2...).
Yeah we have a 10Mb state file and it already takes 30-45 min to deploy. The main way it happens is if someone cancels a CI/CD job because they're realised something is wrong and don't want to wait that long to try again. That's not an issue if there isn't a state file, but yeah I can see how there would be a tradeoff possibly associated.
Pulumi is pretty good about keeping track of in-progress state. If you cancel and restart a new job, it’ll first ask you what to do with the in-progress resources.
IME the plan times explode because the refreshes hit the cloud provider API rate limits.
That does seem like quite a large state file. Would you be able to shed some light on the sorts of of resources that you're provisioning? I've found it to be useful to split dependencies between cloud resources, and to add links via data lookups. At any reasonable scale, I've found splitting Terraform state to be crucial.
For example: network resources and Kubernetes clusters are created separately, and networking is a required dependency for Kubernetes clusters. Resources are associated with environments and regions, and modules take in environments and regions as parameters.
In my setup, we have 200+ state files, and the largest is around 80KB.
How do you manage the meta DAG between all of that stuff in the event of, e.g. complete disaster recovery?
Yes they do. You can refresh the state file quite easily using the `pulimi refresh` command.
I'm not really sure how you'd do it without a state file though? State drift is very common and without an idea of the current state it's pretty difficult for any framework to work out what updates need to be applied.
The claimed pro of being able to use a normal programming language fades away quickly as you are forced into non-idiomatic boilerplate patterns to get pulumi to work.
State is a problem and have had just the same kind of hardships when had to manually modify it on some changes as I would have had on terraform. (transfer resources between state stores as a business function is handed over within company, with zero downtime, import docs are not always correct/up2date just like in case of terraform, as if the example codes were generated by chatgpt)
Overall pulumi couldn't convince me, it wasn't a bit more convenient than terraform, yet I could bump into its bugs for pretty basic aws features (around lambdas/apigw, but was more than a year ago, might be fixed already) which did work just fine with terraform ootb, and the pulumi code felt bad to look at despite being in a language I generally like. It did not bring the effect management wanted that developers with no terraform knowledge can be onboarded to infra, as of course infra knowledge is needed and is harder to pick up than learning terraform syntax.
Despite my dislike for terraform and its syntax's limitations in expressiveness, I'm still back to TF, and is more productive than pulumi was.
I’ve been pretty happy with Pulumi but have learned that state management can be a pain.
I got myself into a situation recently that I could only get out of by using the CLI interactively. I wound up with multiple copies of each resource, which shared a URN, so when I tried to delete them from the CLI it would always prompt me for which instance of that URN to delete. I ended up spending much of a day writing a program to call their CLI and then interact with it programmatically because I had hundreds of resources to delete.
Since then I’ve been doing more with CloudFormation directly and am tempted to switch.
Does pulumi let you write a state reader, so you can add schemas or some kind of validation?
Does anyone know if there's the inverse process available, from Pulumi to Terraform?
I use terranix to manage AWS https://terranix.org/
cute: https://github.com/terranix/terranix-examples/blob/main/aws-...
Also: don't use static AWS creds, kids
Isn't pulumi aws just terraform under the hood still? Why wouldn't I just use terraform's CDKTF instead?
>Isn't pulumi aws just terraform under the hood still?
It depends.
The AWS "Classic" provider uses the terraform provider [1].
The AWS "Native" provider does not, and instead uses the AWS Cloud Control API [2].
CDKTF feels like a giant hack (yes, used it).
No everything is available vs TF HCL, it uses CDK from AWS CDK and tries to bridge things to TF have codegen. There’s a lot of weird corner cases and inconveniences. It even uses React under the hood.
Pulumi at least takes care of it better. Maybe not perfectly but better.
Yes, pulumi is just wrapping terraform[1]. So you need to understand both the quirks of that and the quirks of pulumi. And I'm lazy so I just want to deal with one quirk at the time.
Is there a clear winner between pulumi and cdktf for new projects?
I really like the idea of using a proper programming language to describe infrastructure, but Pulumi isn’t quite what I thought it could be.