Z1T: Sparse Transformer-Like Models for Probabilistic Hardware

Our goal at Extropic is to build the ultimate physical substrate for intelligence. As all generative AI algorithms are ultimately probabilistic algorithms, and as silicon naturally becomes probabilistic at ultra low power, our mission in practice involves the extreme co-design of probabilistic algorithms for probabilistic hardware. We are a few years into this co-design journey now, having primarily focused on traditional probabilistic graphical models and diffusion-like models in our public works.^{[1][2][3][4][5]}

However, it goes without saying that the most dominant family of algorithms today is the Transformer^[6]. Training and inference for Transformer-based models account for an enormous share of the workloads being executed on modern digital hardware accelerators. The market's seemingly insatiable demand for running more instances of this architecture has led to one of the largest industrial buildouts in recent history, commanding extreme financial costs^[7] and causing a sharp spike in energy demand.^[8][9]

While our long-term thesis remains that our fundamentally new kind of hardware will, as a byproduct of long-timescale hardware-software co-evolution, necessarily give birth to whole new breed of models at the intersection of stochastic thermodynamics and generative AI that are native to this new paradigm^[10] , there is currently a very pressing need to address the energy-efficiency of inference of transformers.

To address this urgent issue, we investigated how our recently announced probabilistic sub-threshold CMOS chip, Z1, could potentially be used to improve the energy efficiency of transformer-like models.

DISAGGREGATED INFERENCE  We explore disaggregated model inference pipelining neural computations between TSU’s like Z1 and XPUs / FPGAs

THE Z1 GRAPH  Each pbit has 16 tunable couplings.

As there is currently a large installed base of GPUs / XPUs that are specialized for dense matrix multiplications (matmuls), we investigate how to run pipeline parallel workloads across heterogeneous compute, combining our thermodynamic accelerators like Z1 with traditional digital matmul accelerators in order to improve performance per watt at a rack and datacenter level.

Porting Transformers to Z1 hardware

In order to encode deep neural network computations into probabilistic hardware, we draw inspiration from the Boltzmann-machine lineage of neural computation^[11], and use averages of statistics from our thermodynamic computers to output classical vectors of numbers rather than sampling from the chip. We go into greater depth on our encoding strategy in the hardware encoding section, including how to have low-precision integer-like encodings via bundles of pbits on Z1.

Similar to networks of neurons in biology, Z1 has sparse physical connectivity between its pbits. That sparsity is key to its power efficiency and is a hard constraint of the hardware. In order to have an efficient embedding of operations into this substrate, we built our Z1T model architecture primarily with sparse neural-network operations in mind.

One could view this as a first step of algorithmic mutation away from mainstream transformers, traditionally optimized for GPU-like dense matrix multiply accelerators, towards optimization for next-gen sparse in-memory computing hardware like Z1. To further explore the ramifications of this design choice, we trained a number of Z1T sparse models at varying sizes, amounts of training compute, and levels of sparsity to demonstrate an empirical scaling law, featured in the scaling section.

Disaggregating Inference Across Heterogeneous Substrates

Disaggregation of inference across processing stages has been a topic that is top of mind with all hardware platforms. The Transformer’s compute-bound prefill and memory-bound decode steps are getting split onto separate hardware substrates. Similarly, we have seen attention-feedforward disaggregation, where the compute graph of a single decode pass is split in a pipeline across heterogeneous hardware.

Later in this blog, we outline our own custom decoding disaggregation strategy unique to Z1T. As our first generation systems come pre-packaged with FPGAs on the same board as Z1 chips, we explore how to leverage the FPGAs in conjunction with a scaled up network of Z1 chips in order to run Z1T in a model parallel fashion across Z1s and pipeline parallel across the heterogeneous network of FPGAs and Z1s.

The energy and latency estimates presented in this blog are for FPGA co-processors. It is worth noting that the companion processor could in principle be GPUs or other accelerators (XPUs), and something we may explore in future works.

Here we report our preliminary investigation into this direction for heterogeneous decode disaggregation. We outline below the components of the transformer-like architecture we constructed, how we can break down layers and divvy up the computations between Z1 chips and FPGAs, and how much energy this decoding pass requires relative to an H100 GPU.

This is an initial study in sparse neural-network and hardware co-design. While the design of Z1 predates these considerations, we are using them to inform future chip designs that will improve the performance of Z1T-style models on our hardware.

As this is the beginning of co-design between transformer-like models and probabilistic hardware, we are open sourcing our training recipes for the sparse transformers used to produce our scaling laws, as well as open sourcing the weights for one of the larger training runs for Z1T. We strongly believe in the power of open source for AI progress, and it is our delight to finally be able to open source our first model.

The current era of deep learning was kicked off with fairly predictable scaling laws^[12][13] given the training dataset size, training compute, and loss. This scaling law has held for 15 orders of magnitude so far and has been the foundation to the success of the whole AI ecosystem in the 2020s.

In order to kickstart an era of scaling inference on thermodynamic hardware, we created our own scaling experiments for models that fit on sparse, locally-connected hardware, and observe a scaling law that parallels that of traditional transformers, but now with the added sweepable variable of connectivity.

Sparse hardware like Z1 has a parent graph has fixed connectivity baked into the silicon, as such any graph that is a direct subgraph of the hardware will inherit this sparsity. Understanding the scaling behavior of model performance versus degree of graph connectivity is thus essential for hardware-algorithmic co-design. To this end, we explore a number of open questions:

LOCAL CONNECTIVITY VS SPARSITY  A chunk of the Z1 chip, with toroidal boundary conditions added for illustrative purposes, alongside the adjacency matrix of its coupling graph. Left visualizes physical connectivity, while the right visualizes the sparsity of any coupling matrix that fits within the hardware.

• How does loss change with sparsity and connectivity at fixed compute and data?
• How do sparse scaling trends compare to dense model scaling?
• Which design levers matter most for our next chips?

We investigate these questions in two directions: First, we build our Z1T architecture with a modified variant of attention called gated convolutional attention (GCA)^[14], whose sparse and localized tanh-linear operations map naturally to Z1. Second, we vary a shared connectivity parameter in a standard GPT-2^[15] style decoder transformer to study sparse projections and shortened attention context away from the Z1T-specific operations, to gain more insight into the general role of sparsity and fixed connectivity in the scaling behavior of transformer models.

A sparser transformer built for Z1

Here we sweep model size for an architecture matched to Z1's constraints: 4-bit weights and four incoming edges per output node (see the architecture section for details). We then use the FPGA for non-Z1-compatible operations, see the disaggregation section for details. We chose the currently featured split as it was our best-performing configuration for Z1 from our fairly broad agentically-driven explorations. Below we plot how the test loss scales against training FLOPs and parameter count, for Z1Ts trained on OpenWebText with the GPT-2 byte-pair encoding (BPE) tokenizer.

Our extrapolation shows that we need about an order of magnitude more FLOPs with our sparser Z1T model to achieve the same loss as a GPT-2 model. As we will see in later sections, the operations done on Z1 have about three orders of magnitude energetic efficiency gain compared to GPUs, and as such, Z1T ends up with a two orders of magnitude energy efficiency gain. See the efficiency section for a more in-depth breakdown and performance details.

Exploring the scaling of sparse Transformers

Before we dive into Z1T, let us focus first on the general effect of sparsity in transformer models. We want to know how important sparsity is when added as an axis in the standard scaling law formula to help inform future decisions around connectivity, weighing model performance versus energy efficiency. This model is a standard GPT-2 style decoder, with dot-product masked self attention, RMS normalization^[16], and GELU^[17] based feed forward blocks. We then train it on byte-tokenized OpenWebText.

In these experiments, we have a single additional knob: connectivity c. This controls two parts of the models, both the matrix multiplications (e.g. the W_Q, W_K, W_V projections, and the feed forward blocks), and the attention scores (via windowed attention). For c = N, each output node of the network is connected to N input nodes (e.g. the row of W in W @ x has N non-zero values), the attention scores are also windowed to the last N values.

Importantly, this is a fixed connectivity pattern, not a percentage sparsity. This differs from the usual sparse neural models which operate under fixed, e.g. 95%, sparsity considerations. As width increases, our percentage sparsity asymptotically becomes 100%. This is due to the fact that we fix the degree of the graph connectivity in silicon to be constant. In our experiments, this percentage varies from 5% to 99.8%.

Unlike some previous work^[18], we train with our fixed connectivity from the beginning and design our analysis around isoFLOP budgets.

We benchmark c ∈ {4, 16, 32, 64, 128} on sequence length 256, in addition to a fully dense baseline, from 3×10¹⁴ FLOPs to 10¹⁸.

First we plot the loss vs. the training FLOPs for a variety of sparsities and parameters. All parameters in these plots include both body parameters and the final matrix, the latter of which is always dense.

These results do provide some indication that dense FLOPs are more efficient than sparser FLOPs given the same parameter count. However, FLOPs do not pay the same price across substrates. We care about the energetic cost per token, and a sparse multiply and accumulate on Z1 is substantially more energy efficient than the equivalent operation on a GPU (in no small part due to the in-memory nature of Z1). Thus, a sparse model on Z1 can still reach the same performance at a fraction of the power, as we show later below.

Our chip, Z1, is built as a graphical model of probabilistic bits (pbits). These are binary stochastic CMOS circuits whose probability is controlled by a bias induced by neighboring pbits.^[19] The chip samples from a programmable Ising model^[3] and updates the states via Gibbs sampling^[20] at a 50 MHz internal update clock. Each pbit draws a new value conditioned on its neighbors, executing a chromatic Gibbs sampling algorithm in-memory. When two pbits have a connection, in physics one calls this a “coupling” between spins in the Ising model.

Z1 has a programmable parent graph that is fixed in the silicon. If a coupling exists in this parent graph \(\mathcal{G} = \{\mathcal{V},\mathcal{E}\}\), its value can be tuned in the corresponding Ising energy function:

\[ E_{\mathcal{G}}(z) \;=\; \sum_{j\in\mathcal{V}} h_j\, z_j \;+\; \sum_{\{j,k\}\in\mathcal{E}} J_{jk}\, z_j\, z_k \;=\; \mathbf{h}^{\top}\mathbf{z} + \mathbf{z}^{\top}\mathbf{J}\,\mathbf{z}, \qquad z_j \in \{-1,+1\} \]

where \(\mathbf{J}\) is the coupling matrix. As elements of this matrix \(J_{jk}\) can only be nonzero if there exists a corresponding \(\{j,k\}\in\mathcal{E}\) in the hardware graph, having a hardware graph of fixed degree induces sparsity in the potential coupling matrix.

For Z1 specifically, this graph of couplings has each node being of degree 16, for a total of 269,568 pbits and 2,135,904 coupling edges on a single chip. If we want to use Z1 couplings to embed certain operations directly in hardware, they will have to inherently be sparse, due to the fixed degree of the interaction graph. In the fabric figure above, we feature how a patch of the Z1 coupling graph can be viewed as a matrix, and show visually what we mean by connectivity versus sparsity. As one can see, the hardware connectivity matrix is very sparse, meaning that its matrix elements are largely zero.

This is unlike GPUs which achieve all-to-all connectivity between cores by having a cache memory hierarchy which all units have access to, which allows GPUs to perform matrix multiplications for matrices that are “dense”, i.e. have mostly non-zero matrix elements. Modern GPUs are optimized for large, regular, dense matrix multiplications.^[10] Sparse operations on a GPU can create irregular memory access patterns, while simultaneously not providing any significant speedups proportional to their sparsity levels.^[21] A matrix multiplication on a GPU between two 99% sparsity matrices is not 100× faster.

THE Z1 DIE  Eight cores, 269,568 pbits, 215,904 coupling parameters, >50 MHz sampling rate, <1 W.

Z1T adapts dense layers to fit nicely onto the Z1 chip’s fixed sparse topology. Not only do we embed what is effectively a sparse matrix-vector product, but we also compute activations in-place, as well as some variant of attention, as we will outline below.

Encoding continuous numbers into pbits

Neural-network activations and weights are typically floating-point values, while Z1 operates on pbits (which can also be represented as spins over {−1, 1}). To operate on continuous values with Z1, we need a way to map continuous numbers to pbit representations. Fundamentally this involves quantizing the continuous value into some number of pbits.

While there are a variety of schemes that could be considered, for example:

\[ x \;\approx\; \sum_{i=1}^{4} 2^{-i-1}\, s_i \;\equiv\; \mathbf{d}^{\top}\mathbf{s}, \qquad \mathbf{s} \in \{-1,+1\}^{4}, \quad d_j = 2^{-j-1} \]

We refer to this as the dy4p quantization of x: dyadic, because every scale is a power of two, with 4 bits of precision. This dy4p could be seen as the spin analogue of a 4-pbit probabilistic integer (p-Int) encoding, pInt4.

A p-Int is technically a distribution over quantized numbers \(p(x) = \sum_{\mathbf{s}} \delta(\mathbf{d}^{\top}\mathbf{s} - x)\, p(\mathbf{s})\) induced by the distribution over bit values of its several constituent pbits. In order to get a real number from it, in practice we take an empirical average from the stream of pbits:

\[ y \;=\; \mathbb{E}_{x \sim p(x)}[x] \;=\; \mathbb{E}_{\mathbf{s} \sim p(\mathbf{s})}\big[\mathbf{d}^{\top}\mathbf{s}\big] \;\approx\; \tfrac{1}{N} \sum_{k=1}^{N} \mathbf{d}^{\top}\mathbf{s}^{(k)} \;=\; \tfrac{1}{N} \sum_{k=1}^{N} \sum_{i=1}^{4} 2^{-i-1} s^{(k)}_i \]

where \(s^{(k)}_i\) is the \(k\)th sample of the \(i\)th pbit from a parallel stream of \(N\) empirical samples from each pbit. As this empirical average is approximate, we can view the uncertainty of the empirical mean as related to the bits of precision of an equivalent fixed-point number. More explicitly, given an empirical variance \(\sigma^2 = \mathbb{V}(x)\), we want an effective number of \(k\) bits of precision such that the standard deviation fits in a bin of the equivalent digital representation:

\[ \frac{\sigma}{\sqrt{N}} \;\leq\; \frac{1}{2^{k}} \;\Rightarrow\; k \;\leq\; \tfrac{1}{2}\log_2 N \;-\; \log_2 \sigma \]

This is shown on the right of the dy4p figure below, where for a given number of samples \(N\) coming from the pbit stream forming the dy4p, we show the effective equivalent bits of precision for a digital encoding. The interesting thing is that by using a probabilistic computer, we can have effectively higher bit precision than our physical original encoding of 4 bits, including what is effectively fractional bits of precision, and we can decide to tune this precision at runtime by simply scaling the number of samples. This is particularly significant as modern deep learning operates in the low-precision regime, as such probabilistic computers are naturally suited to represent activations, latents, and parameters.

tanh-linear units

One key building block of our components for Z1T is the tanh-linear unit, constructed from averaging one probabilistic cell. For one tanh-linear element of Z1, the energy of a visible node and its 16 neighbors is:

\[ E(v, h) \;=\; -\Big(\, b_v\, v \;+\; \sum_{j=1}^{16} J_j\, v\, h_j \;+\; \sum_{j=1}^{16} b_j\, h_j \Big), \qquad v,\,h_j \in \{-1, +1\} \]

Conditioning on the hidden nodes, the visible spin is a Bernoulli distribution over {−1, 1} which when averaged becomes a tanh of the local field.

\[ \mathbb{E}\!\left[\, v \mid h \,\right] \;=\; \tanh\!\Big( b_v + \sum_{j=1}^{16} J_j\, h_j \Big) \]

This tanh-linear unit can be applied in parallel for each output node.

A weighted activation can therefore be written as the expectation of a pbit with:

\[ \tanh(w x + b) \;\approx\; \tanh\!\big(w\,a^{\top} s + b\big), \qquad a_i = 2^{-i-1}, \quad J = w\,a^{\top} \]

This also extends to vectors. For an input x ∈ ℝ^D and spins s ∈ {−1,+1}^D×4 with 4 spins per value, the weight matrix can be directly encoded into the interactions over the flattened spin vector via:

\[ \tanh(W x + b) \;\approx\; \tanh\!\big( (W \otimes a)\,s + b \big) \]

The hardware naturally samples from a distribution with the expectation given by a sparse matrix-vector-product fused with a tanh. We can leverage these units to do a tanh-linear operations that are similar to a component of a neural network. In this work we assume that one has sufficient Z1 chips running side by side to place all sampled tanh-linears in a model-parallel fashion. While these encodings are necessary for practical Z1 execution, the results of Figure 1 do not include quantization of the activations.

A traditional Transformer block^[6] typically is constructed from a Self Attention block, a RMSNorm and a FeedForward Network. These primitives rely on dense matrix operations, and thus are typically deployed on GPUs. In this investigation, we try to adapt these primitives to the sparse tanh-linear layers we can run on our Z1 chip.

The RMSNorm layer is implemented as:

\[ \operatorname{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{D}\sum_{i=1}^{D} x_i^2 + \epsilon}} \odot \gamma + \beta \]

Empirically it has been shown that each component of RMSNorm resembles a tanh activation, and can be replaced with a scaled Dynamic Tanh (DyT).^[22] We can therefore substitute RMSNorm components with the DyT:

\[ \operatorname{DyT}(x) = \gamma \tanh(\alpha x) + \beta \]

This is naturally implementable on Z1 by fusing the shift and scale with subsequent layers.

Classic softmax-based attention is computed using dense queries, keys and values matrices. Given an input sequence X ∈ ℝ^T×D and projection matrices W_Q, W_K, W_V ∈ ℝ^D×D, we compute Q = XW_Q, K = XW_K, V = XW_V. Self attention is then

\[ \operatorname{Attention}(X) = \operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{D}}\right) V \]

This requires O(T²D) sequence work and dense matrix multiplication to encode the query-key scores. We instead adapt a gated convolutional attention architecture to sparse hardware.^[14] Given our token stream X, we instead use 4-sparse projection matrices to compute the gates Q, K, V. At a token in position t, head i:

\[ Y_t^{\,i} = \tanh\!\big(Q_t^{\,i}\big)\odot\frac{N_t^{\,i}}{D_t^{\,i}} \] \[ N_t^{\,i} = \operatorname{conv1d}\!\big(\exp(K^{i})\odot V^{i},\, \exp(w^{i})-1\big) + \sum_{j=1}^{t}\exp(K_j^{i})\odot V_j^{i} \] \[ D_t^{\,i} = \operatorname{conv1d}\!\big(\exp(K^{i}),\, \exp(w^{i})-1\big) + \sum_{j=1}^{t}\exp(K_j^{i}) \]

We can offload certain arithmetic and transcendental functions to an XPU. Our systems use an FPGA as a coprocessor, which allows for flexible operations and energy efficiency.

A simple feed-forward network can be mapped to our hardware by composing tanh sampling programs across the chip fabric. An MLP layer is given by

\[ \tanh(W x + b), \qquad x \in \mathbb{R}^{d_{\mathrm{in}}},\; W \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}},\; b \in \mathbb{R}^{d_{\mathrm{out}}} \]

For a sparse W we can compile^[3] this to hardware by laying out d_out many tanh-linear units on Z1 and stream the samples between the cores.

Visualizing gated convolutional attention

Pipelining tokens across heterogeneous hardware

Here we follow the journey of a token as it crosses between being processed on the FPGA and Z1.

The FPGA adds each block's input back around the attention and MLP transforms. It also runs the vocabulary readout that produces the logits, and Z1 samples the next token.

In this investigation, we've used an FPGA to get concrete estimates quickly. However, the dense vocabulary matmul and much of the remaining digital work could run on other accelerators, such as GPUs or dedicated accelerators from Cerebras, Etched, etc.

These early results suggest dense models are more efficient per FLOP. However, a sparse FLOP on Z1 costs far less energy than the equivalent operation on a GPU. Moreover, a sparse model can reach the same performance at a fraction of the power, as we show below.

Energy efficiency gains

Note that the following projections are based on theoretical chip energy consumption of Z1 based on our best estimates, which are anchored to reality from our experiments with similar pbits in X0. We exclude the dense logit final readout layer of computation in these estimates.

The first column is the model FLOPs utilization (MFU), the fraction of the theoretical peak H100 FLOPs reached by the measured throughput. In LLMs such as Llama 3, MFU is around 40%.^[23] For very sparse models, which H100s are less optimized for, one often sees much lower MFU.

The H100 / Z1T is the energy ratio between inference (excluding the final dense logit computation) on the H100 and Z1 system. This system includes all previous computations done on both the Z1 fabric and the connected FPGA (but does not include data movement between the two). The last column compares the layers that just run on Z1 (i.e. sparse tanh layers).

Model details

• Z1 topology: 8 cores × 33,696 pbits = 269,568 pbits, 2,135,904 hardwired edges
• Z1 sampling energy: 1.3e-14 J per sample
• Card assumption: samples and channels are placed across parallel Z1 chips; the required chip count is not modeled
• FPGA estimate: 0.2 pJ/matrix-multiply op, 3.0 pJ/scalar op, 1.5 W assumed static power
• Model dimensions: L=4, D=512, T=1024, GCA H=4, kernel=4, sparse k=4
• H100 reference: 32-bit floating-point peak energy 0.177 pJ per floating-point operation, the same next-token step run densely, with no sparsity exploited, model FLOPs utilization varied above

The Z1T estimate is 294.52 nJ per token: 8.74 nJ for Z1 sampling and 285.78 nJ for the included FPGA work (including the final logit readout on a FPGA yields approximately 136.4 µJ per token).

Assuming a single serial stream and no advanced pipelining methods, we estimate the throughput as follows.

Sampling Z1 accounts for ~16 µs of the larger estimate. Each sample is drawn in parallel on the fabric and averaged to get an estimation of the scalar tanh-linear output.

For the H100 comparisons, we evaluated the speed with batch-1 sequential decoding of the model. In both the H100 and Z1 cases, we exclude the final vocab logit computation. Since this model is quite small relative to the scale of a H100, the kernel launch times play a non-negligible impact on speed (this model achieves 0.006% MFU). If we were to batch on the GPU, H100s would be substantially more efficient. As such, we advise using our Z1 + XPU setup for decode rather than prefill, which is better done on GPU for now.

Latency model details

• Steps: 25 sequential sampling layers, 8 FPGA-to-Z1 data writing steps, 5 Z1-to-FPGA data reading, FPGA computes all the rest such as pooling, positional-encoding, and residual work
• Estimated time breakdown: 38.0 µs for FPGA orchestration (38 serial ops × 1 µs), 16.0 µs Z1 sampling (25 × 32 samples at 50 MHz), 3.2 µs data reading, 1.3 µs data writing, 0.13 µs FPGA sample averaging, 0.16 µs FPGA pool/positional/residual = 58.8 µs total
• Z1 I/O: 25.6 Gbit/s data transfer speed
• These estimates are conservative as we do not count parallel samples. Note that parallelizing sampling would consume the same amount of energy, but parallelized samples would reduce the time to achieve certain desired precision due to the higher rate of sampling and reduction of empirical variance. Please refer to our section on probabilistic encoding of continuous numbers for the equivalence between samples and digital precision.
• H100 baseline: NVIDIA H100 80GB HBM3, torch 2.7.0, dense-equivalent model, D=512, L=4, 11.55M body parameters in fp16, measured 2026-08-12

In this Z1T experiment, we evaluated how to put a transformer-like set of operations onto a joint probabilistic/FPGA fabric that was not originally designed for this set of operations. Even in this unoptimized regime, we found compelling energy efficiency gains at a time when energy consumption of ML models is a pressing issue.

Our energy estimates indicate that the FPGA consumes the vast majority (>95%) of the energy, which means that there are substantial improvements that could be made by a chip designed with transformer-like neural models like Z1T in mind. By shifting more of the operations onto an ultra-energy-efficient sub-threshold CMOS substrate, one could reap even larger energy savings, potentially reaching up to 1000× greater energy efficiency than GPUs.

Removing the FPGA bottleneck would take this approach to that level, in line with the Z1-only layer gains reported above. We leave this for future algorithmic and hardware explorations. This work was only the beginning of our co-design journey coupling our chips to the world of transformers. Many more iterations to come.

Guillaume Verdon Alexander Neagoe Owen Lockwood Seth Morton Extropic San Francisco

1. Lockwood, O., Béjanin, J., Bus, J., Chamberland, C., Huembeli, P., Schäfer, F., & Verdon, G. (2026). A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing. arXiv preprint arXiv:2607.16183. https://arxiv.org/abs/2607.16183↩
2. Jelinčič, A., Lockwood, O., Garlapati, A., Schillinger, P., Chuang, I. L., Verdon, G., & McCourt, T. (2026). An Efficient Probabilistic Hardware Architecture for Diffusion-Like Models. npj Unconventional Computing, 3, Article 30. https://doi.org/10.1038/s44335-026-00075-3↩
3. Extropic. (2026). From One to One Billion: Torx, Thermalizers, and Z1. https://extropic.ai/writing/from-one-to-one-billion↩
4. Verdon, G., Tyrpak, L., Lockwood, O., Morton, S., Neagoe, A., Sugolov, A., MacCormack, I., & Amico, M. (2026). A Framework for Stochastic Differentiable Programming. arXiv preprint arXiv:2608.01612. https://arxiv.org/abs/2608.01612↩
5. Amico, M., Jelinčič, A., Nancarrow, C. O., Tyrpak, L., Roberts, D., Morton, S., Sakthivadivel, D., Gopal, A., & Verdon, G. (2026). Thermalizing Stochastic Programs. arXiv preprint arXiv:2608.01615. https://arxiv.org/abs/2608.01615↩
6. Vaswani, A., et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html↩
7. Whitwam, R. (2026, July 23). Google just had its first negative cash flow quarter due to massive AI spending. Ars Technica. https://arstechnica.com/google/2026/07/google-just-had-its-first-negative-cash-flow-quarter-ever-due-to-massive-ai-spending/↩
8. International Energy Agency. (2025). Energy and AI. IEA. https://www.iea.org/reports/energy-and-ai/energy-supply-for-ai↩
9. Congressional Research Service. (2026). Data centers and their energy consumption: Frequently asked questions (CRS Report No. R48646). https://www.congress.gov/crs-product/R48646↩
10. Hooker, S. (2021). The Hardware Lottery. Communications of the ACM, 64(12), 58–65. https://doi.org/10.1145/3467017↩
11. Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), 147–169. https://doi.org/10.1207/s15516709cog0901_7↩
12. Hestness, J., et al. (2017). Deep Learning Scaling Is Predictable, Empirically. arXiv preprint arXiv:1712.00409. https://arxiv.org/abs/1712.00409↩
13. Hoffmann, J., et al. (2022). An Empirical Analysis of Compute-Optimal Large Language Model Training. Advances in Neural Information Processing Systems, 35. https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html↩
14. Zhai, S., et al. (2021). An Attention Free Transformer. arXiv preprint arXiv:2105.14103. https://arxiv.org/abs/2105.14103↩1↩2
15. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf↩
16. Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems, 32. https://proceedings.neurips.cc/paper/2019/hash/1e8a19426224ca89e83cef47f1e7f53b-Abstract.html↩
17. Hendrycks, D., & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415. https://arxiv.org/abs/1606.08415↩
18. Frantar, E., Riquelme, C., Houlsby, N., Alistarh, D., & Evci, U. (2023). Scaling Laws for Sparsely-Connected Foundation Models. arXiv preprint arXiv:2309.08520. https://arxiv.org/abs/2309.08520↩
19. Freitas, N., Delvenne, J.-C., & Esposito, M. (2021). Stochastic Thermodynamics of Nonlinear Electronic Circuits: A Realistic Framework for Computing Around kT. Physical Review X, 11(3), Article 031064. https://doi.org/10.1103/PhysRevX.11.031064↩
20. Gibbs sampling. (2026). In Wikipedia. https://en.wikipedia.org/wiki/Gibbs_sampling↩
21. Gale, T., Zaharia, M., Young, C., & Elsen, E. (2020). Sparse GPU Kernels for Deep Learning. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. https://doi.org/10.1109/SC41405.2020.00021↩
22. Zhu, J., Chen, X., He, K., LeCun, Y., & Liu, Z. (2025). Transformers Without Normalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR52734.2025.01388↩
23. Grattafiori, A., et al. (2024). The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783↩