GitHub - josheppinette/python-logfmter: Add logfmt structured logging using the stdlib logging module and without changing a single log call.

Add logfmt structured logging using the stdlib logging module and without changing a single log call.

> logging.warn("user created", extra=user)

at=WARNING msg="user created" first_name=John last_name=Doe age=25

Why

enables both human and computer readable logs, recommended as a "best practice" by Splunk
formats all first and third party logs, you never have to worry about a library using a different logging format
simple to integrate into any existing application, requires no changes to existing log statements i.e. structlog

Install

Usage

This package exposes a single Logfmter class that can be integrated into the standard library logging system like any logging.Formatter.

Integration

Simply use the standard logger's basicConfig or dictConfig initialization systems to get started. Examples are provided below.

basicConfig

import logging
from logfmter import Logfmter

handler = logging.StreamHandler()
handler.setFormatter(Logfmter())

logging.basicConfig(handlers=[handler])

logging.error("hello", extra={"alpha": 1}) # at=ERROR msg=hello alpha=1
logging.error({"token": "Hello, World!"}) # at=ERROR token="Hello, World!"

dictConfig

If you are using dictConfig, you need to consider your setting of disable_existing_loggers. It is enabled by default, and causes any third party module loggers to be disabled.

import logging.config

logging.config.dictConfig(
    {
        "version": 1,
        "formatters": {
            "logfmt": {
                "()": "logfmter.Logfmter",
            }
        },
        "handlers": {
            "console": {"class": "logging.StreamHandler", "formatter": "logfmt"}
        },
        "loggers": {"": {"handlers": ["console"], "level": "INFO"}},
    }
)

logging.info("hello", extra={"alpha": 1}) # at=INFO msg=hello alpha=1

Notice, you can configure the Logfmter by providing keyword arguments as dictionary items after "()":

...

    "logfmt": {
        "()": "logfmter.Logfmter",
        "keys": [...],
        "mapping": {...}
    }

...

fileConfig

Using logfmter via fileConfig is not supported, because fileConfig does not support custom formatter initialization. There may be some hacks to make this work in the future. Let me know if you have ideas or really need this.

Configuration

There is no additional configuration necessary to get started using Logfmter. However, if desired, you can modify the functionality using the following initialization parameters.

keys

By default, the at=<levelname> key/value will be included in all log messages. These default keys can be overridden using the keys parameter. If the key you want to include in your output is represented by a different attribute on the log record, then you can use the mapping parameter to provide that key/attribute mapping.

Reference the Python logging.LogRecord Documentation for a list of available attributes.

import logging
from logfmter import Logfmter

formatter = Logfmter(keys=["at", "processName"])

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR processName=MainProceess msg=hello

mapping

By default, a mapping of {"at": "levelname"} is used to allow the at key to reference the log record's levelname attribute. You can override this parameter to provide your own mappings.

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "process"],
    mapping={"at": "levelname", "process": "processName"}
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR process=MainProceess msg=hello

datefmt

If you request the asctime attribute (directly or through a mapping), then the date format can be overridden through the datefmt parameter.

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "when"],
    mapping={"at": "levelname", "when": "asctime"},
    datefmt="%Y-%m-%d"
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR when=2022-04-20 msg=hello

defaults

Instead of providing key/value pairs at each log call, you can provide defaults:

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "when", "trace_id"],
    mapping={"at": "levelname", "when": "asctime"},
    datefmt="%Y-%m-%d",
    defaults={"trace_id": "123"},
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR when=2022-04-20 trace_id=123 msg=hello

This will cause all logs to have the trace_id=123 pair regardless of including trace_id in keys or manually adding trace_id to the extra parameter or the msg object.

Note, the defaults object uses format strings as values. This allows for variables templating. See "Aliases" guide for more information.

ignored_keys

Sometimes log records include fields that you don't want in your output. This often happens when other libraries or frameworks add extra keys to the LogRecord that are not relevant to your log format.

You can explicitly exclude unwanted keys by using the ignored_keys parameter.

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at"],
    mapping={"at": "levelname"},
    datefmt="%Y-%m-%d",
    ignored_keys=["color_message"],
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.info("Started server process [%s]", 97819, extra={"color_message": "Started server process [%d]"})
# at=INFO msg="Started server process [97819]"

Extension

You can subclass the formatter to change its behavior.

import logging
from logfmter import Logfmter


class CustomLogfmter(Logfmter):
    """
    Provide a custom logfmt formatter which formats
    booleans as "yes" or "no" strings.
    """

    @classmethod
    def format_value(cls, value):
        if isinstance(value, bool):
            return "yes" if value else "no"

	return super().format_value(value)

handler = logging.StreamHandler()
handler.setFormatter(CustomLogfmter())

logging.basicConfig(handlers=[handler])

logging.error({"example": True}) # at=ERROR example=yes

Guides

Aliases

Providing a format string as a default's key/value allows the realization of aliases:

import logging
from logfmter import Logfmter

formatter = Logfmter(
    keys=["at", "when", "func"],
    mapping={"at": "levelname", "when": "asctime"},
    datefmt="%Y-%m-%d",
    defaults={"func": "{module}.{funcName}:{lineno}"},
)

handler = logging.StreamHandler()
handler.setFormatter(formatter)

logging.basicConfig(handlers=[handler])

logging.error("hello") # at=ERROR when=2022-04-20 func="mymodule.__main__:12" msg=hello

Gotchas

Reserved Keys

The standard library logging system restricts the ability to pass internal log record attributes via the log call's extra parameter.

> logging.error("invalid", extra={"filename": "alpha.txt"})
Traceback (most recent call last):
  ...

This can be circumvented by utilizing logfmter's ability to pass extras via the log call's msg argument.

> logging.error({"msg": "valid", "filename": "alpha.txt"})
at=ERROR msg=valid filename=alpha.txt

Development

Required Software

If you are using nix & direnv, then your dev environment will be managed automatically. Otherwise, you will need to manually install the following software:

Additionally, if you aren't using nix, then you will need to manually build the "external" tools found in external. These are used during testing to verify compatibility with libraries from different ecosystems. Alternatively, you can exclude those tests with pytest -m "not external", but this is not recommended.

Getting Started

Setup

If you are using pyenv, you will need to install the correct versions of python using <runtimes.txt xargs -n 1 pyenv install -s.

$ direnv allow
$ pip install -r requirements/dev.txt
$ pre-commit install
$ pip install -e .

Tests

Run the test suite against the active python environment.

Run the test suite against the active python environment and watch the codebase for any changes.

Run the test suite against all supported python versions.

Contributing

Create an issue with all necessary details.
Create a branch off from main.
Make changes.
Verify tests pass in all supported python versions: tox.
Verify code conventions are maintained: git add --all && pre-commit run -a.
Create your commit following the conventionalcommits.
Create a pull request with all necessary details: description, testing notes, resolved issues.

Publishing

Create

Update the version number in logfmter/__init__.py.
Add an entry in HISTORY.md.

Commit the changes, tag the commit, and push the tags:

$ git commit -am "v<major>.<minor>.<patch>"
$ git tag v<major>.<minor>.<patch>
$ git push origin main --tags

Convert the tag to a release in GitHub with the history entry as the description.

Build

Upload

Table of Contents

Why