GitHub - metorial/object-storage: A lightweight and universal object storage service.

A lightweight, universal object storage service. It provides a simple HTTP API for managing buckets and objects, with support for multiple storage backends including local filesystem, AWS S3, Google Cloud Storage, and Azure Blob Storage.

Features

RESTful HTTP API for object storage operations
Multi-backend support (local, S3, GCS, Azure)
Support for metadata directly in the storage backend
ETag generation for cache validation

Installation

Using Docker

The Docker image does not include configuration. You must provide it externally using either a config file or environment variables.

Option 1: Using a configuration file (recommended)

Copy config.example.toml to config.toml and modify it for your needs, then mount it into the container:

docker pull ghcr.io/metorial/object-storage:latest

# Create your config.toml first (see Configuration section below)
docker run -p 8080:8080 \
  -e CONFIG_PATH=/app/config/config.toml \
  -v $(pwd)/config.toml:/app/config/config.toml:ro \
  -v object-store-data:/app/data \
  ghcr.io/metorial/object-storage:latest

Option 2: Using environment variables

Configure entirely through environment variables without a config file:

docker run -p 8080:8080 \
  -e OBJECT_STORE__SERVER__HOST=0.0.0.0 \
  -e OBJECT_STORE__SERVER__PORT=8080 \
  -e OBJECT_STORE__BACKEND__TYPE=local \
  -e OBJECT_STORE__BACKEND__ROOT_PATH=/app/data \
  -e OBJECT_STORE__BACKEND__PHYSICAL_BUCKET=object-store-data \
  -v object-store-data:/app/data \
  ghcr.io/metorial/object-storage:latest

Using Docker Compose

services:
  object-storage:
    image: ghcr.io/metorial/object-storage:latest
    ports:
      - "8080:8080"
    volumes:
      - ./config.toml:/app/config/config.toml:ro
      - object-store-data:/app/data
    environment:
      - CONFIG_PATH=/app/config/config.toml
      - RUST_LOG=info

volumes:
  object-store-data:

Building from Source

cargo build --release
./target/release/object-store-service

Configuration

The service requires configuration to be provided externally. You can use either a TOML file or environment variables.

Configuration File

Create a config.toml file and set the CONFIG_PATH environment variable:

export CONFIG_PATH=/path/to/config.toml
./object-store-service

Example config.toml for local development:

[server]
host = "127.0.0.1"
port = 8080

[backend]
type = "local"
root_path = "./data"
physical_bucket = "object-store-data"

Example config.toml for Docker:

[server]
host = "0.0.0.0"  # Accept connections from any interface
port = 8080

[backend]
type = "local"
root_path = "/app/data"  # Use container paths
physical_bucket = "object-store-data"

Environment Variables

All config options can be set via environment variables with the OBJECT_STORE__ prefix. Nested keys use double underscores:

export OBJECT_STORE__SERVER__HOST=0.0.0.0
export OBJECT_STORE__SERVER__PORT=8080
export OBJECT_STORE__BACKEND__TYPE=local
export OBJECT_STORE__BACKEND__ROOT_PATH=/app/data
export OBJECT_STORE__BACKEND__PHYSICAL_BUCKET=object-store-data

Environment variables override config file values, allowing you to use a base config file and override specific settings.

Backend Configuration

Local filesystem:

[backend]
type = "local"
root_path = "./data"
physical_bucket = "object-store-data"

S3:

[backend]
type = "s3"
region = "us-east-1"
physical_bucket = "my-bucket"
endpoint = "http://localhost:9000"  # Optional, for MinIO

Google Cloud Storage:

[backend]
type = "gcs"
physical_bucket = "my-gcs-bucket"

Azure Blob Storage:

[backend]
type = "azure"
account = "myaccount"
access_key = "myaccesskey"
physical_bucket = "mycontainer"

API Reference

Health Check

Returns service health status.

Buckets

Create a bucket:

POST /buckets
Content-Type: application/json

{
  "name": "my-bucket"
}

List buckets:

Delete a bucket:

Objects

Upload an object:

PUT /buckets/{bucket}/objects/{key}
Content-Type: application/octet-stream
x-object-meta-author: john
x-object-meta-version: 1.0

[binary data]

Custom metadata headers must be prefixed with x-object-meta-.

Download an object:

GET /buckets/{bucket}/objects/{key}

Get object metadata:

HEAD /buckets/{bucket}/objects/{key}

Delete an object:

DELETE /buckets/{bucket}/objects/{key}

List objects in a bucket:

GET /buckets/{bucket}/objects?prefix=folder/&max_keys=100

Query parameters:

prefix (optional): Filter objects by prefix
max_keys (optional): Limit number of results

Response Format

All JSON responses follow this structure:

Success (Bucket):

{
  "name": "my-bucket",
  "created_at": "2024-01-15T10:30:00Z"
}

Success (Object Metadata):

{
  "key": "path/to/file.txt",
  "size": 1024,
  "content_type": "text/plain",
  "etag": "abc123...",
  "last_modified": "2024-01-15T10:30:00Z",
  "metadata": {
    "author": "john",
    "version": "1.0"
  }
}

Error:

{
  "error": "Bucket not found: my-bucket"
}

Architecture

The service is organized into three main components:

object-store: HTTP API layer and service logic
object-store-backends: Storage backend implementations
clients: Client libraries for different languages

Client Libraries

Client libraries are available for multiple languages:

Go

go get github.com/metorial/object-storage/clients/go

import objectstorage "github.com/metorial/object-storage/clients/go"

client := objectstorage.NewClient("http://localhost:8080")
bucket, err := client.CreateBucket("my-bucket")

See clients/go/README.md for full documentation.

TypeScript/JavaScript

npm install @metorial/object-storage-client

import { ObjectStorageClient } from '@metorial/object-storage-client';

const client = new ObjectStorageClient('http://localhost:8080');
const bucket = await client.createBucket('my-bucket');

See clients/typescript/README.md for full documentation.

Rust

[dependencies]
object-storage-client = "0.1"

See clients/rust/README.md for full documentation.

License

Licensed under the Apache License, Version 2.0. See LICENSE file for details.