GitHub - notesofcliff/delve: A versatile and powerful data platform allowing interactive searches, dashboards, alerts, and more.

5 min read Original article ↗

Build Status GitHub Sponsors

Delve

Delve is a powerful, extensible platform for ingesting, transforming, and searching structured, unstructured, and semi-structured data. It is designed for easy local development, robust production deployments, and seamless integration with modern tools and containerization workflows.

Features

  • Ingest data from diverse sources (REST API, file tail, syslog, scheduled queries)
  • Transform and normalize data with custom pipelines
  • Perform powerful search and filtering with a pipeline syntax
  • Create interactive dashboards and visualizations
  • Set up alerts and notifications
  • Extend functionality with custom apps and commands

Project Structure

  • manage.py at the repository root for standard Django management
  • Core apps (e.g., events, users) and configuration in top-level folders
  • requirements.txt and pyproject.toml for Python dependencies
  • bootstrap.py for automated build, packaging, and asset management
  • frontend/ for JavaScript and SCSS assets
  • doc/ for user and admin documentation
  • utilities/cli/ for ingestion utilities such as tail-files.py and syslog-receiver.py

Quick Start

1. Clone the repository

git clone https://github.com/notesofcliff/delve
cd delve

2. Create and activate a virtual environment

python -m venv .venv
.venv\Scripts\activate  # On Windows
source .venv/bin/activate  # On Linux/macOS

3. Install dependencies

pip install -r requirements.txt

4. Run database migrations

5. Install frontend dependencies and build assets

npm install
npx webpack --config webpack.config.js

6. Collect static files

python manage.py collectstatic --no-input

7. Create a superuser

python manage.py createsuperuser

8. Start the development server

python manage.py runserver

9. (Optional) Start additional services

# Task scheduler
python manage.py qcluster

# Syslog server
python utilities/cli/syslog-receiver.py

# Tail log files
python utilities/cli/tail-files.py /var/log/*.log

Defaults: Delve ships with Whitenoise + CherryPy by default to keep air-gapped/offline use simple. Swap components as desired.

Dependency Management

All Python runtime dependencies are managed via a single, pinned requirements.txt at the repository root. Do not add runtime dependencies to pyproject.toml or use pip install . or pip install -e .. For local development, Docker, and ZIP/air-gapped workflows, always install with:

pip install -r requirements.txt

If you need to update dependencies, edit requirements.txt directly.

Database Creation in Docker Compose

When you set DELVE_DATABASE_NAME, DELVE_DATABASE_USER, and DELVE_DATABASE_PASSWORD in your .env, the Postgres container automatically creates the database and user with those credentials on first startup. No manual setup is required.

Using Delve with Docker Compose

Delve ships with a docker-compose.yaml for easy setup. Make sure to copy .env.example to .env and fill in required values (see comments in the file).

Build and Start All Services

docker-compose up --build

This will build the images and start the web server, worker, and Postgres database.

Run Database Migrations (required after first start)

docker-compose exec web python manage.py migrate

Create a Superuser (for admin access)

docker-compose exec web python manage.py createsuperuser

View Logs for All Services

Restart All Services

Stop and Remove All Containers

Visit http://127.0.0.1:8000/ in your browser to access the web UI.

Docker Troubleshooting & Cleanup

  • If a service fails to start, check logs with docker-compose logs <service>
  • If environment variables are missing, Compose will error out with a message (for required secrets and DB credentials)
  • To rebuild images after changing the Dockerfile, use docker-compose build

Inspect Docker State

  • List all containers (running and stopped):
  • List all images:
  • List all volumes:
  • Show disk usage (images, containers, volumes, build cache):

Clean Docker Environment (remove all containers, images, volumes, caches)

  • Remove stopped containers: 
    docker container prune -f
  • Remove unused images:
  • Remove unused volumes:
  • Remove everything (containers, images, volumes, networks, build cache): 
    docker system prune -a -f

Advanced: Automated Build & Packaging For Air-Gapped Systems

You can use bootstrap.py to automate building, packaging, and asset management for deployment to air-gapped systems. While containerization is also supported, this utility enables deployment to air-gapped environments without requiring dependencies on the target system.

After running the following commands, you will have a zip file under ./dist/ containing everything needed to deploy Delve to an air-gapped system, including source code, Python interpreter, frontend and backend dependencies, and more:

  • Clean build artefacts: 
    python bootstrap.py clean --all
  • Download and extract Python: 
    python bootstrap.py download_python
python bootstrap.py extract_python
  • Install Python dependencies: 
    python bootstrap.py run_pip_install
  • Install frontend dependencies and build assets: 
    python bootstrap.py run_npm_install
python bootstrap.py build_frontend
  • Collect static files: 
    python bootstrap.py collectstatic
  • Package everything: 
    python bootstrap.py package

Or run all steps in sequence:

See doc/admin/Bootstrap_Guide.md for full details and extensibility options.

Documentation

  • User Guide: doc/user/Getting_Started.md
  • Admin Guide: doc/admin/Installation_and_Setup.md, doc/admin/Bootstrap_Guide.md
  • API Reference: Browse the REST API via the web UI after starting the server

Key Concepts

  • Events: The core data unit, with indexed and extracted fields
  • Queries: Pipeline-based data retrieval and transformation
  • Ingestion: Multiple methods, including REST, file tail, and syslog
  • Field Extraction: Index-time and search-time extraction
  • Custom Apps: Extend Delve with new commands, dashboards, and APIs
  • Alerts: Search-based and processor-based alerting

Contributing

Contributions are welcome! Please see the documentation and open an issue or pull request.

Support

Delve is an open-source project maintained in my spare time.

If you find it useful, please consider sponsoring me on GitHub

License

Delve is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See doc/LICENSES.txt for details.