This is the third post in a series on building Metis, a trading engine I’m developing to answer one question: can understanding how energy flows let us see what moves natural gas prices before the market does?
The first two posts covered a SIMD performance investigation; a silent compiler fallback and a Windows power management cap that together produced a 92% performance gap on identical hardware. This post is different; it’s about data, and specifically about a problem that almost nobody in quant circles writes about directly.
Read any data thread on r/quant and you will hit one of two dead ends. Someone complains about a $50k vendor feed that arrived full of NaNs, or someone drops a "just use Databento" and walks away. Both assume the hard part is the plumbing. Get the data in cleanly and cheaply, and the rest follows.
I don’t think that’s true, data access is largely a solved problem. The hard problem is temporal semantics; what does this data actually measure, when does it measure it, and how does its measurement frequency relate to the phenomenon you’re trying to model? Get that wrong and clean data produces wrong answers very confidently.
Building Metis forced me to think about this explicitly, because the signals I’m tracking don’t share a clock.
Metis ingests eleven data sources. Here’s what their temporal structures actually look like:
LMP electricity prices from CAISO arrive sub-hourly — real-time grid stress, priced by the minute. EIA natural gas storage updates weekly, every Thursday. The US Drought Monitor publishes once a week, always on Tuesday, measuring categorical severity across the country. AIS maritime vessel data streams continuously but gets aggregated to daily port snapshots. Congressional bills have multiple economically meaningful dates per event — introduction, committee action, floor vote, presidential signature — and the market continuously reprices the probability of enactment at every step, meaning a single 'policy date' timestamp is a fiction. FRED macroeconomic series vary by series: some monthly, some quarterly, some revised backward.
The standard playbook is to resample everything to a shared daily frequency, align the timestamps, and feed it into an ML pipeline. It produces beautiful, clean tensors. It also creates a massive point-in-time (PIT) correctness problem. If you are not strictly respecting the exact publication timestamp, especially with backward-revised data like FRED, your daily resample is quietly injecting look-ahead bias directly into your model. Clean data will confidently give you a backtest that is entirely fictional.
Furthermore, it encodes a hidden assumption: that a weekly drought reading on Tuesday contains the same information content as a Tuesday LMP reading. They are not comparable. Forcing them into a shared timestamp doesn’t fix the alignment problem, it just hides it.
That changes the question to ask from “how do I align these” to “what does each source actually measure, and at what granularity does that measurement become meaningful?”
Congressional bills are where this gets most interesting, and most underappreciated.
The obvious approach is to timestamp a bill by its signing date and call it a signal. But a bill’s signing date is just one of several economically meaningful moments in its lifecycle, and often not the most important one. Introduction tells you something about legislative intent and political pressure. Committee action tells you something about viability. Floor passage tells you the policy is real and coming. Presidential signature makes it law. And then there’s a gap; sometimes months, sometimes years between law on paper and economic reality.
Trump’s tariff announcements are the clearest example of why this matters. The signing dates were the least interesting timestamps. The market started repricing supply chains before the ink dried, and the more consequential signal — countries remaking trade agreements without the US — played out over months after the legal fact. China and the EU didn’t wait for the next round of negotiations; they started building alternative relationships that will persist regardless of what US policy does next.
For LNG specifically this creates a structural question: when a US export policy changes, which timestamp do you use? The announcement, the legal enactment, the first cargo diversion, or the point where European terminals had locked in alternative supply contracts? These are not the same event. They’re four different signals with four different lags, and treating any one of them as the canonical “policy date” loses the others.
The Metis congress ingester captures multiple action dates per bill — introduction, latest action, and update date — specifically because collapsing them to a single timestamp would mean choosing which lag to discard. I would rather have the data and decide later than make that decision in the ingestion layer where it’s invisible.
Most pipelines answer this question with: as much as possible. More data is better. Go back to the beginning of available records.
That’s wrong, or at least imprecise. The better question is: how much history reflects the same data-generating process you’re trying to model?
For LNG specifically, the market structure from fifty years ago does not reflect current dynamics. The post-war energy expansion, the rise of the personal automobile, the first OPEC shock — these were not just historical events, they were structural breaks that changed the underlying system. Data from before those breaks describes a different market. Including it doesn’t provide more information, but rather noise from a system that no longer exists.
This is why the Metis ingestion config does not use uniform historical depth across sources. fred_building_permits has a target of ten years. cme_futures maintains a rolling two-year sliding window. lmp is incremental, only new data since last run, with a seven-day lookback for corrections. The depth is per-source, calibrated to when each source’s data became structurally representative of the current system.
The other implication is architectural: historical ingestion and live ingestion are separated in the codebase. Backfill is a one-time event. It runs progressively until it reaches the target depth, then automatically switches to incremental mode. This was a deliberate choice; I do the historical backfill work once, and then maintain the present. Treating them as the same operation conflates two different things.
Drought data is where the temporal frequency question gets most concrete. The US Drought Monitor publishes weekly. The naive pipeline just maps that onto the same weekly grid as EIA storage.
But drought conditions don’t actually change weekly. A region in severe drought on Tuesday was in severe drought last Tuesday. The weekly publication is just a measurement interval; accumulating daily interpolations would not give you more signal, it would just hallucinate resolution.
More importantly, the data is categorical. The Monitor classifies severity from D0 (abnormally dry) to D4 (exceptional drought). Most models just map this to a 0-5 continuous scale and call it a day. That encodes a massive assumption.
In a previous role, I worked alongside civil engineers in assessing roadway damage via EV sensors. One of the first things I had to do was read a field guide on road damage. To an outsider, the fact that they have hundreds of distinct names for “a crack in the road” looks absurdly verbose. But a surface fracture caused by thermal expansion means something entirely different than an alligator crack caused by water destroying the sub-base. The category isn’t just a label; it’s a diagnostic of the underlying physical failure. It tells you exactly what it will take to fix it.
Drought categories are the same. D0 and D4 are not just points on a line. They represent entirely different physical realities—surface dryness versus fundamentally depleted aquifers. D4 conditions affecting 10% of a major producing basin do not have a tidy, linear “5x” impact compared to D0 conditions. The physical relationship is non-linear, which means the economic impact on natural gas demand is non-linear.
Treating categories as continuous variables is an easy modeling choice, but it actively destroys the specific, physical reality the category was designed to capture.
The AIS maritime ingestion was the design decision that took the longest to get right, and the one I feel most confident about now.
The naive approach to tracking LNG supply via vessel data is to track vessels. Follow the LNG carriers. If a ship is moving, it’s delivering. If ships stop moving, something is wrong. This sounds reasonable until you think about what vessel-level tracking actually captures.
Individual vessel routes are noisy. A carrier rerouting around weather, making an unscheduled stop, or transiting a different channel looks the same in the data as a supply disruption. You’d need to know the vessel’s contract, its usual route, its expected port of call, and its charter arrangement to distinguish routine deviation from a meaningful signal. That information isn’t in AIS data. So vessel-level tracking gives you a lot of movement to explain and limited ability to explain it.
Ports change more slowly than vessels. The same way airports change more slowly than airplanes, LNG export terminals are structural. Sabine Pass doesn’t change its capacity configuration week to week. If the composition of vessels present at Sabine Pass shifts — fewer LNG carriers, more support vessels, unusual idle time — that’s a structural signal, not a routing coincidence.
The Metis AIS ingester does not track individual vessels. It tracks terminal composition: how many LNG carriers, general tankers, and support vessels are present at selected major US export terminals on a given day. A drop in LNG carrier presence at Sabine Pass is a structural signal; a vessel rerouting in the Atlantic is often just noise.
I don’t currently ingest Middle Eastern port data into Metis, but watching the Iran Strait situation unfold globally validated this architectural choice in real time. When the strait became contested, the relevant market signal wasn’t which specific vessels changed their routes. It was that LNG port activity across the region dropped broadly and simultaneously. Knowing the coordinates of individual vessels in the strait would have been a high-frequency distraction. Knowing that multiple terminals suddenly went quiet was the actual signal.
Everything described so far is operational. The nightband data is still a hypothesis.
The story goes like this: several years ago it became known that certain firms were using satellite imagery and computer vision to measure shadow lengths on oil storage tanks. The physics is straightforward — taller shadows mean more oil in the tank, because the floating roof rises with the contents and casts a longer shadow at a fixed sun angle. Physical observables as a proxy for inventory levels that companies don’t publish in real time.
While this a old and well known alt-data example, however the underlying principle still holds: if you can see something physical, and that physical thing has a known relationship to an economic variable, you have a leading indicator by definition. The economic variable gets reported later. The physical thing is visible now.
Nightband satellite imagery from NASA’s Black Marble dataset captures surface light emissions at night. Economic activity uses energy. More activity means more light. The hypothesis is that nightband readings over major natural gas consuming regions could act as a leading indicator of demand — you’d see the lights change before the demand figures were reported.
Whether that holds up in practice is genuinely unknown. The confounders are real and significant: weather affects visibility, political factors affect who gets power and when, seasonal variation in daylight affects emissions. Understanding that multiple variables influence what you see in one map is a prerequisite for using it, not a reason to avoid it. The experiment is whether the demand signal is strong enough to isolate.
It’s worth being honest that this is still at the hypothesis stage. I’m including it here because it illustrates the principle that runs through the whole stack: find physical observables that lead economic variables, and understand the lag. The shadow on the oil tank. The vessel composition at the terminal. The drought severity in the basin. The nightband over the city. All of them are downstream of physical reality. All of them show up in the data before they show up in headlines.
Databento is a great product, and vendor feeds absolutely have their place. But the decision to buy a clean packaged dataset is also a decision to accept someone else’s choices about temporal structure, historical depth, and what the data means.
The true cost lies in the methodical and deliberate effort needed to understand what each source actually measures and build the temporal model yourself. That’s harder than buying a clean feed; you’re past the world of tutorials and known examples, into the space where any choice can be reasoned as a good one. It’s also where the edge lives, if there is one — not in having data others don’t have, but in understanding it differently.
Next in the series: the ingestion architecture.
Accepting data sources that live in different time domains brings a massive engineering problem; how do you build an ingestion pipeline that respects those frequencies?