How I Build My Own Amazon S3 Storage

Recently I decided to move my side projects to Heroku. But Heroku does not provide a way to host media files. You need to use external storage for that (Amazon S3 or similar).

Unfortunately, there are no free solutions available and my Amazon Free Tier has expired. So I decided to create my own Amazon S3 Like Storage using Python/Django.

And hey, that was pretty easy to implement.

TL;DR;

Here is the Github repository. If you want a production-ready system - use MinIO.

Plan

I want to have an application, that will replicate basic Amazon S3 functionality so I can use it at my Django projects (via django-storages).

I don’t need it to be complex and I only need a small portion of that is available on Amazon S3:

• File upload
• Serving public files
• Ability to have multiple Buckets with own credentials so I can use that service for most of my projects

I would like to run it inside Docker via single docker-compose.yml so I can deploy it to my Digital Ocean 5$ droplet with 25Gb of storage.

Let’s start!

Application template should be pretty simple. I don’t need to have UI outside of the Django Admin page, as I don’t want to offer that solution as SAAS and compete with Amazon (yet).

Views

There will be only one view, which handles all of the requests.

My service will only handle requests in the format of:

/<bucket_name>/<content-path>

And we need to handle those methods:

• PUT — send a new file to the storage
• GET — get file
• HEAD — check if the file already exists in the storage

Models

I will have two models for now:

• Bucket — define bucket name and credentials
• Blob — model to store user uploads and some metadata (size, content type)

Verify requests

We need to validate user requests, so only users who have the right credentials can upload files into our service.

That is the trickiest part, as we need to calculate headers and content hashes and validate them with the received in the request. You can check examples of how to do that in Python on Amazon Web site.

There is an awssig package by David Cuthbert available so we don’t need to do that part by ourselves.

That package calculates a signature for the received request content and headers using secret keys and compares it with the signature provided in the request. So the only user who has the right secret keys can upload content.

One thing that package is missing currently is handling HTTPS requests, which is a specific case. There is a PR available, to fix that. For now, we would use a custom version of that package, which we can install directly from Github:

pip install git+git://github.com/slyapustin/python-aws-sig.git#egg=awssig

Each time we receive PUT request we will check if that request has a valid signature for the bucket is associated with:

We would store everything inside Django’s FileField with some metadata we may need for serving content properly later:

I store content type in order to serve images with the correct content-type so they can be inserted directly into the HTML page:

Deployment

I want to have a Docker Compose configuration so I can deploy everything to my Digital Ocean droplet.

A couple of notes:

• I’m using a separate docker.env file with all the required environment variables for the Docker deploy
• Docker volumes pointed to the local filesystem, so I don’t have any overheads.
• I use Nginx as a reverse proxy on my server and that app will listen for requests sent from Nginx to the port 8005. You may have different configuration so keep that in mind.

How to use that?

After you deploy that to your server and create Bucket with the new credentials, you can use it inside of your Django application as usual AWS S3 storage with the django-storages package.

Here are the settings you should provide:

That’s it.

Demo

Here’s a Django-Classified Demo project, which uses that Django-S3-Like-Storage.