Autoscaling for Digital Ocean?
I've started a thread at Digital Ocean. I want to implement such a system (I could not find any). Any questions, recommendations, "don't invent the wheel" comments?) Does any DO customer need such a tool?
I am not a DevOps guru, but it seems like an interesting project. AWS costs almost twice more than analogous DO servers, but AWS has way more features.
DO community thread: https://www.digitalocean.com/community/questions/autoscaling-solutions-for-digital-ocean-are-there-existing-solutions
Of course it will be open source. At clanofthecloud.com (Gaming backend as a service), we needed to implement autoscaling for DO, in node.js. We used consul.io for service discovery. CPU load was used as the central metric, and each service instance would store it in consul.io. We then triggered the creation/destruction of instances with thresholds. In the end, we only used it for our sandbox environment, as the production env runs on bare metal (more capacity, cheaper at scale, easier on admin). So I'd say, from our experience: - DO's API was quite easy to work with - consul.io was used as a reliable distributed source of information, for leader elections and health monitoring... Changing the autoscaler configuration in consul.io produced immediate results like starting/stopping new instances... Cool "remote control" effect ;-) - haproxy/nginx load balancers use consul.io templates to update their configuration - our autoscaler was HA, through a leader election. The instances managed themselves (no single point of failure). There were at least 2 instances running. - you should expect a few "surprises" if you're running consul.io on Digital Ocean,
heartbeats are delayed quite often (depends on datacenters), which makes failure detection hard - and of course, we used DO custom images to start new instances > you should expect a few "surprises" if you're running consul.io on Digital Ocean, heartbeats are delayed quite often (depends on datacenters), which makes failure detection hard isn't that true for any CP store in any "cloud" provider? I'd say it's more true of cloud providers who don't schedule your VM for 15+ seconds, from time to time. As I said, we've deployed our production environment on bare metal (in reaction to this exact problem...) Thank you. I'll look at consul.io closer Everything you need is there. Floating IP's to do HA at the front. Use something like HA proxy and monitor whatever it is you need such as response time from the backends. Once that passes a limit call a script to poke the API and provision you a new node. You could use something like Ansible to provision the node and then place it in to load balance. One thing to consider is that DO is really, really good at vertical scaling. You can have about 1 minute or less downtime on an instance (basically boot time) to restart it as a bigger instance (RAM/CPU) as long as you don't grow the HD at the same time. This is counter to most horizontal scaling strategies but it's really about the same. When you add more servers you're essentially just adding more CPUs and RAM via VMs. Being able to do it on the same machines minus any configuration time or provisioning time is really slick (especially for DBs). Setting up a load balancer in front of a few instances that could take advantage of rolling vertical scaling would be a spin on autoscaling that played to one of DO's real strengths. That looks nice until it doesn't work. We regularly see 30 minute event processing when doing flexible upgrades of Droplets (and the same when scaling back down). It's not safe, since the resize isn't guaranteed to be fast and you have no way to abort it once it's started. Sounds good in theory, not reliable enough to use in practice. Isn't this what the flexible IP is for? You create a new droplet concurrently the one running and then flip the IP to point to the new droplet. No downtime. You have to have the a server with the same configuration or a snapshot or provision a new one... Using snapshot to create a new droplet and update some configuration files should be fast either. Probably, longer than 1 minute but not much, I guess. Sounds like a fun project (and totally doable)! :) If I were you, I'd make heavy use of ansible, or something similar, for provisioning: 1) folks are familiar with it
2) could make it cross-platform more easily
3) well, no reinventing the wheel. For example, Ansible has ec2 module http://docs.ansible.com/ansible/ec2_module.html where you describe an instance and the number of them that should be running. So if you have 3 instances running and wish to have 5, it does the magic and spins up new ones. Then, you can add them to a load balancer. Maybe there's something similar for DO already? The way I see it is that it would poll if scaling conditions are met and execute ansible playbooks if they are, and then some web interface to set the conditions / view the scaling logs / current status. It can turn out to be a very entertaining and educational side project :) If you decide to do it, drop me an email - something I would be happy to brainstorm and discuss about :) EDIT: It could also be used not only for autoscaling but also for self-healing. If some instance crashed and is not responding anymore, then spin up a new one. I've build setups like this with Ansible, but the more I use Ansible, the less likely it is that I'll consider it for future projects. I find it a lot less painful to write code to do these things directly than wrestle with Ansible. E.g. the amount of pain I went through before realising that the EC2 module at the time messed with whitespace in the user-data (great when you're transferring yaml... one more reason I detest significant whitespace; but the irony of this breaking in a tool written in Python was not lost on us), for example. I guess it depends on the use case. If all you want to do is to make sure that, for example, all web servers have the same version of nginx, php and sync the configs of these services, I think ansible is a great tool and writing code to do that would definitely qualify as 'reinventing the wheel'. I wouldn't disqualify the tool for a problem in one of the modules either. Bugs happen :) It wasn't clear to me how to autoscale with DO either. That's why I ended up using Google Cloud Engine. It's not exactly cheaper than AWS, but the pricing model for discounts is WAY simpler at least. It also doesn't have the feature set AWS does, but it does have most of what you'll need like a load balancing service. The pricing for GCE is the most convoluted I've ever worked with... And it's still 2-3 times more expensive than alternatives like DO or dedicated hosting. A lot of the time when people want auto-scaling, the thing that strikes me is that most of them wouldn't have needed auto-scaling if they picked a cheaper provider to beging with. Often they could pick a dedicated provider, spin up 3 times as much capacity and still pay less. Ditto. If you need to autoscale you outgrew DO. On top of DO prices not dropping in 4 uears, I see little reason to use them. GCE has $5 instances now with big boy features. My 2 cents: Build an independent autoscaling tool that abstracts over the differences between clouds because: 1. Even if DigitalOcean is the best cloud provider today -- I'm not saying it is or isn't -- the probability it is the best that will ever be is approximately zero. The landscape changes. Heterogeneity among cloud providers is increasing and anyway latency will always be a function of the physical location of data centers where the data is sharded. 2. It's the right alignment for an open source project because it is driven by the broad interests of developers rather than the narrow needs of a single company. DigitalOcean may change its pricing policy. It may cease to exist. It may make breaking API changes. All for legitimate business reasons orthogonal to those of particular developers. If their autoscaling code is platform independent, then that's not a crisis. Good luck. Yes, and libcloud is pretty good for abstracting providers: http://libcloud.apache.org/ The only DO-specific bit would be DO API to create/delete droplets (VPS) so that shouldn't be hard at all :) Thanks. We made GitLab Runner Autoscale https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/... that can automatically spin up more droplets via Docker Machine to run your CI tests. We use it ourselves for GitLab builds (up to 100 machines) and enabled it for all GitLab.com users. For more information see the announcement https://about.gitlab.com/2016/03/29/gitlab-runner-1-1-releas... Feel free to ask questions. I too have wondered more or less the same, though I have yet to build any projects that need to scale so I haven't turned it into an actual concern for myself as of yet. Hoping you find a solution and share it in here so those of us like me who are curious can learn. It might end up being a mixture of solutions. I can only remember an open source interface that managed multiple cloud providers, not sure if it did any autoscaling, was posted on HN and that's all I remembered, somebody open sourced it but it's description was vague as hell. Edit: Commented more. We have an auto-scaling feature in private beta over at https://www.tilaa.com. Drop us a line if you want to test! Ask for me (Dennis) if you need implementation assistance. You can either use VM snapshots or our metadata service (cloud-init compatible) to provision new instances automatically. We also have an API, but you don't need it for autoscaling. We're a European cloud provider (based in the Netherlands) and don't have PoPs in the US though. I wrote this last week, might be relevant "AWS costs almost twice more than analogous DO servers, but AWS has way more features." When you us AWS you are paying for more than just a cheap VPS, which essentially is all DO is. It's comparing fast food to a fancy steak house. You can get meat at both, but at one side its microwaved. Not to say DO isn't great, it gets the job done and provides a valued service, but your money gets you what your money gets you. When you use AWS you pay extra for the fancy steak. But you pay steak prices even if what you buy from them is a cheap burger. It's really hard to find scenarios where AWS isn't ridiculously overpriced. Consider that Digital Ocean is also an expensive alternative, but I deploy caching proxies for some clients who for various reasons insist on using AWS on DO because you can save lots by deploying droplets on DO to cache rather than pay AWS bandwidth costs for all your traffic, for example (you serve more than 1-2TB a month out of AWS you can start saving money that way). Not everyone needs or wants a fancy steak, which I guess is why the OP is asking specifically about DO even while acknowledging that "AWS has way more features". The cloudhero CLI can be a very good option for this.
https://cloudhero.io/cli/ (pip install hero) It would allow you to add nodes with the packages that you need extremely easy. In one or two hours a cron could be made that adds and removes nodes as needed. It is not a magic bullet solution but it fixes 80% of the problem with 20% of the effort. I've used their API to deploy a scalable docker cluster. Basically, my docker cluster starts from 3 nodes and as the cluster gets filled with containers I automatically add new nodes on Digital Ocean to increase the cluster capacity. Our team worked a day to make it happen but we got excellent support from the team. Varnish and the d-o API (?) There are more options, but I think I could go that one. You can scale compute power... the main issue is... how do you scale bandwitch at digital ocean? Varnish as a LB? Why Varnish instead of, for example, HAProxy? I don't know. But I'll figure it out (I hope). Keep us up to date! I will look into Kubernetes. I believe you will need just a very thin layer on top of it to add droplets and you can call it done. Cloud66 and Tumtum (now Docker cloud) works with DO. Not open source and not free but a solution nonetheless. Need as well. Go for it! Make it open source and paid for turn key! I'd pay for that I would pay for this - email in my profile, let me know when you launch. Need it too!!! I do not believe that a price difference of 100% would be enough to make such a project interesting. Why not build something like this for dedicated boxes from hetzner, ovh ... I don't have a clue how much harder this would be, but a price difference of about 300% looks much more interesting. Could those downvoters explain their downvotes please? He asked for an opinion. Seriously, building a AWS clone on DO infrastructure is just not possible. Even if it would be it would actually be more expensive in the end as AWS for small to medium projects. If you using AWS you do not only pay for one virtual server, you are paying for a infrastructure. You would have to replicate parts of this infrastructure, which would require a few separate servers and you couldn't share those costs with other users. Using DO instead of dedicated servers for this would be like building a Uber competitor using the Uber API. Edited: I overlooked the fact that at least neither Hetzner nor OVH do provide a useful (for this project) way to order a dedicated server via the API. They do offer an Order API, but the spin up time is too long. I used Hetzner and OVH a lot, but I think DO is much easier from an API / speed to spin up instance perspective. That said, one of the reasons to use providers like Hetzner and OVH is that as I've pointed out elsewhere, the price difference is so large that most people who "need" auto-scaling at places like AWS would pay less if they just ordered 2x to 3x as much capacity at a place like Hetzner and left it on 24/7. Very few people have loads that are genuinely spiky enough to save enough from auto-scaling to make up for the massive cost difference. Bare metal providers like Hetzner are different beasts. In my opinion, you use bare metal when your server load doesn't change much in time. It's too hard to scale fast with bare metal providers, because they need up to 2-3(not always) days to setup server. Virtual machines are managed more simply.