Understanding Ordinary Events

One major advantage of Ordinary’s architecture is that it makes it possible to derive insights about behavior and performance without a ton of additional time spent configuring and tuning instrumentation yourself.

When you use Ordinary, the facilities for request/response logging, as well as inter-service communication are wired up immediately for templates, actions, proxies, integrations and storage, with additional facilities for hooking into the core events system in cases where you BYO code (i.e. actions or client JavaScript).

In this post we’re going to begin walking through how this is accomplished and what it enables for folks who want to build on Ordinary.

Outline#

configuration
- stored | stdio | journald
- fine-tuning
  - level | sizes | headers | ip-addresses
system-architecture
custom-events
- actions
- client
  - events | px
event-structure
querying-events
- read | search
conclusion | future

Configuration#

As a host running ordinaryd you have a few exporters/formatters at your disposal:

--stored-logs (the primary focus of this post)
--stdio-logs (--stdio-logs-fmt: json (default), concise, pretty)
--journald-logs (queryable with journalctl: only available for Linux/systemd)

Stored#

If you’re looking for Ordinary to be your one-stop-shop for hosting (either for yourself or people you’re serving) and don’t want to run separate systems for ingesting and managing events, --stored-logs is gonna be your go-to option. With --stored-logs each domain gets its own logs directory where ordinaryd runs and allows for any tenant to access their logs via the ordinary CLI with the same credentials they use to publish/deploy their app.

Stdio#

Stdio logs can be useful if you already have an existing log management solution¹ that can process either JSON formatted logs or the default format from tracing. There is a pretty format mode as well, but it is primarily useful for debugging and not recommended to run in production.

# json
{"ts":"2026-06-24T19:22:59.732547Z","lvl":"INFO","spn":{"app":{"_":0,"domain":"ordinary.blog","host":"localhost:4433","path":"/","rid":"9f6c42a4-d4f6-45b4-a655-9db2e7dbc07d"}},"fld":{"headers":{"accept-encoding":"gzip, deflate"},"msg":"req","method":"GET","version":"HTTP/1.1"}}

# concise
2026-06-24T19:19:01.697827Z  INFO app{domain=ordinary.blog host=localhost:4433 rid=57166ed7-80c6-4aa7-82f2-1c83bc77151b path=/}: req version=HTTP/1.1 method=GET headers={"accept-encoding": "gzip, deflate"}

# pretty
2026-06-24T19:32:35.834964Z  INFO ordinary_utils::middleware: req, version: HTTP/1.1, method: GET, headers: {"accept-encoding": "gzip, deflate"}
   at core/shared/utils/src/middleware.rs:201
   in ordinary_utils::middleware::app with domain: ordinary.blog, host: localhost:4433, rid: 59a42593-a3b4-429b-8b9f-7f15bec775fb, path: /

Journald#

Journald logging is primarily useful if you’re already heavily integrated with the systemd init system and don’t want to introduce an additional dimension for log management.

Fine Tuning#

Different use cases require different verbosity levels, and properties included/ignored, so across all the logging modes there are some shared configuration options:

--log-level: trace | debug | info (default) | warn | error
--log-sizes: true | false (default)
--log-headers: true | false (default)
--log-ips: true | false (default)

Level#

The --log-level flag tells ordinaryd at what minimum verbosity level should events be considered. If you’re only interested in seeing warnings and errors, you can pass --log-level warn but if you’d like to see request/response logs at the info level you can omit the flag and accept the default.

Sizes#

Primarily geared toward functions of the storage service, the --log-sizes flag controls whether things like item/asset/content sizes are recorded on insert/delete.

Off by default, --log-headers decides whether request and response headers are logged. This flag affects the req/res headers for app, proxy and middleware endpoints.

Sensitive headers² are swapped for the string redacted by default; a companion flag --redacted-header-hash allows hosts to select a hashing algorithm to hash and truncate (6 bytes, base64 encoded) the sensitive values. The current hashing algorithm options are blake2 and blake3³.

IP Addresses#

Same as --log-headers, --log-ips is off by default. Depending on your deployment scenario you may or may not want to log IP addresses – if ordinaryd is a host’s primary entrypoint and an important part of keeping the service online includes monitoring for problematic behavior, associating an IP address or group of addresses with bad traffic is a valid use case.

System Architecture#

Looking primarily at the --stored-logs scenario – as ordinaryd doesn’t own much after export in the --stdio-logs and --journald-logs cases – the goal is to stay as simple as reasonably possible within ordinaryd’s domain to limit the possibility of data loss/disclosure.

On the CLI side we have a little more flexibility to be creative and try out some other tools.

logging architecture, explained in detail in the following paragraphs

On the left in the largest box titled “ordinaryd” (this is our ordinaryd api instance), we have some subcategories of responsibility. In green, we have the “apps” with 3 boxes that represent Ordinary Apps⁴: first.ordinary.blog, second.ordinary.blog, third.ordinary.blog.

Apps#

Each of these “apps” fields requests from the outside world (i.e. the yellow box titled “browser” with a “request” box inside). Each time a request comes through⁵ at least two events will be created: one for the req when the request comes in, and one for the res when the response goes back out. If that request hits a template or an action or an asset or a reverse proxy or any other service type, it will generate additional events letting you know which services were used to serve the particular req/res.

Logfiles#

When events are created, the tracing subscriber picks them up and processes them. For ordinaryd we’ve written a custom subscriber that is capable of sorting events into app-specific files. When the subscriber receives a new event it delegates it to a dedicated thread where the event can be routed to the correct logfile based its root span’s domain field. This is why in the pink box titled “files” we have the same first.ordinary.blog, second.ordinary.blog, third.ordinary.blog directories for each of the domains.

In each app’s logfile directory, we have the current open file that is actively being appended to (for some fun: tail -f logs/some.example.com/*Z | jq '.' on the host server) AND⁶ a larger number of compressed files.

TTL & Rotation#

The custom implementation of the logfile writer allows for custom configuration of file rolling, but also compresses files on rotate. It will also automatically clean up files older than the configured date.

If you’re running with --stored-logs the following configuration flags are also made available in order to configure TTL and file rotation.

--log-ttl-hours: 72 (default)
--log-rotation-mins: 60 (default)
--log-rotation-file-size: 10000000 (default, in bytes)

TTL hours sets a maximum time that a log can sit on the server. Rotation minutes sets the rate at which an active logfile will be compressed, stored, and a new file created, if not exceeding the size limit first. The rotation file size limit sets the max size of an open logfile before it must be compressed and rotated.

`ordinaryd` API#

Within our red box titled “API” we have a couple of arrow-looking boxes with the routes /v1/app/logs/metadata and /v1/app/logs/files/{file}. These routes allow the Ordinary CLI to check/sync the local and remote state.

When you run ordinary app logs sync all the Ordinary CLI first requests the /v1/app/logs/metadata so that it can check it against its local set of log files. If there are any differences between its local state and the remove ordinaryd state, it will make requests to /v1/app/logs/files/{file} to pick up any missing files or partial files it doesn’t have.

NOTE: requests to the ordinaryd logs endpoints will also record events for the app, so admins have a full access history for logs as well (including whether the host⁷ accessed your logs through the API⁸).

`ordinary` CLI#

Now, inside the blue box titled “ordinary” we have the “CLI”, “search index” and “files.” The CLI does state matching based solely on what’s in their app’s log files directory, but the search index still plays a major role in log management, locally.

The “search index” is a tantivy store that ingests JSON formatted events and makes them easily queryable via ordinary app logs search. The log files themselves can also be read via ordinary app logs read.

Custom Events#

The majority of events are generated automatically based on the configuration of your apps and services. The two scenarios where an application maintainer is given the opportunity to generate custom events are within actions – where you have the option to write custom code to wire up storage, integrations and other actions – and client side with JavaScript.

Actions#

In the actions case, if you’re running a JavaScript based action, simply using console.log() will create an event under the request and action’s spans. When writing a Rust action it’s a little more involved but basically looks like:

// INFO log (TRACE = 0, DEBUG = 1, INFO = 2, WARN = 3, ERROR = 4)
ordinary::trace(2, "Hello, Ordinary!")?;

In the future this will be better abstracted and is mostly unused at the moment.

Client#

On the client we expose some baked-in connection points to the events system via the events object exposed at /.ordinary/v1/js. The methods on this object buffer events into a browser-native IndexedDB store, and sync groups of compressed JSON events back to the /.ordinary/v1/events endpoint via the beacon API, on a configurable delay.

`events`#

<!-- in the <head> tag -->
<script type="module">
  import { events } from "/.ordinary/v1/js";
  
  // additional methods with same params: .trace(), .debug(), .warn(), .error()
  events.info("Hello World!", { some: "additional property" });
</script>

This will record an event that looks something like this:

2026-06-25T18:01:04.821719Z  INFO app{domain=ordinary.blog host=localhost:4433 rid=b71df1f3-f291-44a4-8fe4-8f230ec39949 ip=::1 path=/.ordinary/v1/events}:client: Hello World! evt={"cid": "dd8e34e2-ea67-4f50-82b9-b15cf5cbd84f", "mid": "83cfd7cb-5099-48d8-a9e2-3b2ac71ae5b7", "fld": {"some":"additional property"}, "p": "/", "ts": "2026-06-25T18:00:58.809000Z", "tz": "America/Los_Angeles", "v": "0.1.0"}

Nested under the /.ordinary/v1/events request span, we have the client span, with a message of Hello World!, a correlation ID⁹ of dd8e34e2-ea67-4f50-82b9-b15cf5cbd84f, a memory ID¹⁰ of 83cfd7cb-5099-48d8-a9e2-3b2ac71ae5b7, our custom field with the key of "some" and a value of "additional property", a path of "/", a timestamp of "2026-06-25T18:00:58.809000Z", a timezone of "America/Los_Angeles" and the app version of "0.1.0".

`px`#

In addition to the events methods, we have a built-in mechanism for registering page load that is a thin wrapper around an events.info() call, with the additional fields of {ua: navigator.userAgent, ref: document.referrer}. px() function can be useful for distinguishing between genuine and automated traffic.

As a noscript fallback for cases where determining whether the page was loaded in a browser, an app can instead load the /.ordinary/v1/img/px image¹¹.

<!-- in the <head> tag -->
<script type="module" src="/.ordinary/v1/js/px"></script>

<!-- somewhere early in the <body> tag -->
<noscript>
    <img loading="lazy" src="/.ordinary/v1/img/px" height="0" alt="browser loaded check" />
</noscript>

Event Structure#

To understand the anatomy of our events, we’ll start with a request event on the base route for https://ordinary.blog (this request was made when running locally but the production version you’re reading right now generated a very similar event when you visited).

{
  "ts":"2026-06-24T19:22:59.732547Z", // timestamp for the event
  "lvl":"INFO", // log level of the event (TRACE | DEBUG | INFO | WARN | ERROR)
  "spn": { // a map of spans where key is span name and the value is a map of span fields
    "app": { // the "app" span
      "_": 0, // the position of the app span in the span hirearchy
      "domain": "ordinary.blog", // the domain that matches the one defined in an `ordinary.json`
      "host": "localhost:4433", // the actual host for the sent request (if you have a custom domain/CNAME this is where that will be)
      "ip": "::1", // the IP address of the requester
      "path": "/", // the path requested (the "query" field will also be shown if query-string params are passed)
      "rid": "9f6c42a4-d4f6-45b4-a655-9db2e7dbc07d" // the request id pulled from the 'x-request-id' header or generated on request start
    }
  },
  "fld": { // event fields
    "headers": { // request headers
      "accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","connection":"keep-alive","host":"localhost:4433","priority":"u=0, i","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.5 Safari/605.1.15","x-request-id":"9f6c42a4-d4f6-45b4-a655-9db2e7dbc07d"
    },
    "msg": "req", // event message
    "method": "GET", // request method
    "version": "HTTP/1.1" // http version
  }
}

In this particular example we have just one span, but subsequent events nested under other services within the request span will also be appended.

The most important take-away from this section is that all of these fields become searchable once they’ve been indexed locally.

Querying Events#

In order to derive useful insights from your accumulated events, you need a mechanism by which to interrogate them.

Read#

In the simplest mode we can sync our files (ordinary app logs sync all) and then do a read operation – this will rip over all the files and send them to stdout. This allows us to pipe into other facilities for filtering and displaying.

A fairly unsophisticated example:

ordinary app logs read | grep '"msg":"req"' | jq '.'

The above command will grep for the request events and then format them all in stdout with jq.

If you want to read specific files you can use ordinary app logs list to get an ordered list of files and query only the latest synced file.

Search#

Slightly more powerful and interesting for complex queries is the search function. Using the QUICKWIT¹² query language reference we’re able to do a little more than basic pattern matching.

A search count example:

ordinary app logs search count \
    'fld.msg:req AND spn.app.path:"/posts/understanding-ordinary-events/" AND fld.headers.referer:"https://ordinary.blog/"'

This query will give us a count of how many times the https://ordinary.blog/posts/understanding-ordinary-events/ post has been opened from the https://ordinary.blog/ homepage.

A search all example:

ordinary app logs search all \
    'fld.msg:req AND spn.app.path:"/posts/understanding-ordinary-events/" AND fld.headers.referer:"https://"*' \
    | jq -s 'group_by(.fld[0].headers.referer)[] | { (.[0].fld[0].headers.referer): [.[] | .spn[0].app.ts] | length }' \
    | grep '"'

This query will give us the count of all requests with a referer header, grouped by referer header.

Conclusion#

There is a lot more that can be done here with just the features that are currently available, but the focus here is on scaffolding out the basic scope of what’s possible, and creating an opportunity for participants to imagine what else we can do.

Future#

To get the events component of the system to the point where it is “mature” a couple of important items remain:

SQL querying
built-in Prometheus metrics
OpenTelemetry exporter

SQL querying would be the biggest win from our perspective and allow for even greater refinement of insight as well as plugging into existing ecosystems and decades of tooling built up around SQL. DuckDB seems to be a reasonable candidate as its stated purpose is for these kinds of workloads, but it’s hard to compete with the reliability and ubiquity of SQLite. The intention would be for the SQL store to mirror the existing indexing paradigm with the search index, and for them to operate in parallel (i.e. files, search, SQL).

While not discussed explicitly yet, v0.9.0 adds support for authenticated proxying of metrics endpoints that ordinaryd can reach over a network; we’d like to expand this experience to app owners as well, and provide some default instrumentation out of the box (i.e. request counts, service invocations, data transfer, etc.).

And finally we’d like for people to be able to stick with what’s worked for them already. We don’t believe you should have to adopt all of our tooling in order to be successful using the parts of the ecosystem that do add value. If you’ve already got a robust dashboarding paradigm established for fleets of existing services, you should be able to say “here’s my OTel supported endpoint” and handle Ordinary Events where you’re most comfortable.