Startup NeuReality wants to replace the host CPU in data center AI inference systems with dedicated silicon that can cut total cost of ownership and power consumption. The Israeli startup developed a class of chip it calls the network addressable processing unit (NAPU), which includes hardware implementations for typical CPU functions like the hypervisor. NeuReality’s aim is to increase AI accelerator utilization by removing bottlenecks caused by today’s host CPUs.
NeuReality CEO Moshe Tanach told EE Times its NAPU enables 100% utilization of AI accelerators.
“In the cloud, or during our on-prem tests here, we see that different use cases will use the [AI accelerators] differently,” he said. “Some will not go above 25-30% of the utilization of the GPU or the ASIC, and some will just leave the CPU idle because you’re running a big LLM, so you’re heavily bounded by the GPU and the memory interface, and the CPU is just sitting there not doing much. So the economics of the server today, when you’re using inference-specific accelerators, is quite ridiculous.”
Today’s AI servers might have two CPUs with a network interface controller (NIC), or sometimes a data processing unit (DPU) or smartNIC alongside every AI accelerator. This server would serve multiple virtual machines, with the CPU handling tasks like network termination, quality of service between clients, and data preparation before sending data to the AI accelerator.
By Hynetek Semiconductor Co., Ltd. 08.24.2026
By Juan Rey, Senior Vice President, General Manager and CTO of the Calibre segment, Siemens EDA 08.24.2026
By Arduino Team 08.19.2026
The problem with this setup is low accelerator utilization due to bottlenecks in these tasks caused by the host CPU.
“As AI accelerators become more powerful, the underutilization problem will get worse, because the CPU is still the limiting factor,” Tanach said. “Despite their power, CPUs are general-purpose. They were never designed for AI and hinder the efficient processing of AI queries—no matter how good the [AI accelerator].”
NeuReality wants to solve the utilization problem by separating AI pipeline processing from the CPU. The company has hardened CPU tasks like network termination and quality of service onto a heterogeneous compute chip specifically built for AI inference workloads at production scale. Tanach stresses that the NAPU is not an “AI CPU.” Rather, it is dedicated silicon for data center AI inference servers, designed to handle the volume and variety of queries of modern AI inference at scale. It is network-attached, meaning AI queries are directed from Ethernet straight to the NAPU.
The company’s performance figures for its first-gen NAPU, the NR1, show that an AI accelerator ASIC (in this case, the IBM AIU) can boost performance per Watt by approximately a factor of eight by replacing its host CPU with the NR1. While the NR1 was designed around the IBM AIU, it is general-purpose and can work with any AI accelerator after onboarding.
“We partnered with IBM Research and licensed some of their technology to develop the NR1-M AI Inference Module to provide the highest system efficiency with their [AI accelerator],” Tanach said. “We are in discussions with IBM about where the product would best be deployed to help enterprise customers gain better performance at a fraction of the cost.”
The NR1’s power envelope is 75 W, but this should be considered versus the envelope for the CPU plus NIC, he added.
Hardware acceleration
Neureality’s AI-Hypervisor is a key ingredient in the NAPU’s secret sauce. It handles interface-heavy tasks, including queue management and scheduling. In hardware, the AI-Hypervisor block is 64 small CPUs handling the programming model and a dispatching cluster.
“Instead of controlling all the compute engines from software, our compilers decide what to run on each compute engine, and they also generate artifacts for the hypervisor to run the sequence,” Tanach said. “This is where we fix the diminishing return of using many CPU threads to run many sequences in parallel. We offload that piece to hardware so we don’t need expensive CPUs to run the sequence.”
Dataflow between compute engines is decided by the compiler in advance, but is managed by the AI-Hypervisor. A descriptor is built with pointers to the relevant data, and sent to the compute engine. This technique relies on looking at different tables in memory that represent the descriptor and the pointers; 96 CPU threads doing the same thing will need to access the same tables, which can result in coherency problems, Tanach said. While in typical systems this is all done on the host CPU, NeuReality’s NR1 uses its hardware AI-Hypervisor.
“In software we have to use mutex [mutual exclusion] and all kinds of schemes that prevent us from breaking the coherency,” Tanach said. “In hardware, I can do all this in parallel in a much more efficient way, and I don’t need to use a single-thread machine that’s running at 2.5 or 3 GHz.”
Also on chip is hardware acceleration for common AI tasks, including video and audio codecs and general-purpose (digital signal processors) DSPs (Cadence Tensilica IP, with kernels to support Numpy, OpenCV, Python 2.0, etc). There are also some Arm CPUs that act as a fallback for any parts of the workload that cannot be efficiently performed elsewhere in the system, perhaps because the AI accelerator or DSP does not have the appropriate optimized kernels.
When data arrives from the network as a network request, NeuReality’s embedded NIC—the AI over Fabric (AIoF) engine—sends it directly to the AI-Hypervisor where it is added to a queue representing which client it came from. The CPU cores in the hypervisor read descriptors in the data and send it to the relevant compute engine (DSP, Codec, Arm CPU) or off the chip to the AI accelerator.
For example, images might be sent first to the JPEG decoder then back to a queue in the AI-Hypervisor where it is directed to whichever compute engine it is going to next—perhaps for resizing and quantization in the DSP. Then it goes to the hypervisor, then to an AI accelerator to run a face recognition CNN. There may be more processing steps, but when everything is done, the AI-Hypervisor sends the result to the network engine that sends a response back to the client.
Embedded NIC
Tanach said NeuReality’s embedded NIC cannot be compared to full-featured NIC chips on the market as it is more specialized, optimizing the networking overhead for AI. NeuReality developed a protocol, AIoF, which sits above Ethernet (TCP or RoCE). While there are some similarities between AIoF and NVMe over Fabric, there are some differences, too—AIoF supports Kubernetes-based orchestration and provisioning, with quality-of-service offloaded to hardware. The AIoF layer can be accessed via an API.
Splitting workloads between multiple servers is done by middleware—the AI-Hypervisor can load requests to any compute engine on any chip on the network. In this way, multiple AI accelerators can appear as one engine to run very large models. This capability was originally built for applications like Amazon Echo, where voice recognition, natural language processing (NLP), recommendation and speech synthesis would be done on four different servers, Tanach said, but it is also ideal for today’s large language model (LLM) workloads where models are huge. AI accelerators with direct accelerator-to-accelerator connectivity capabilities can take advantage of this to spread big models over multiple accelerators using only the PCI switch on NeuReality’s board (not via the NR1). AI accelerators without direct accelerator-to-accelerator connectivity must use the NR1.
NeuReality demonstrated its LLM setup at SC’23 with NR1s connected to Qualcomm AI100 devices, with one NR1 to one AI100. However, Tanach said the company is working on a setup with one NR1 hosting four AI100s using its NIC capabilities.
“If you have a lot of back and forth between the NR1 and the four accelerators, this might be the bottleneck, but what we’re seeing with this specific use case is that it’s not,” he said.
Software stack
Multiple layers of software simplify access to this heterogeneous compute subsystem. At the model level, NeuReality has full TensorFlow and Pytorch model support.
“We want to be a complementary solution to [AI accelerators]—if they don’t yet support a specific layer, we will complement them,” he said.
Above that is the AI pipeline layer, including pre- and post-processing. While recently introduced Pytorch 2.0 has features to simplify this pipeline, prior to that, pipelines were developed in C++, Python or even Java, Tanach said. So, NeuReality developed a Python and TVM toolchain to translate pipelines into compute graphs with compute nodes and control nodes that run on the NR1’s heterogeneous compute engines.
Part of the TVM toolchain is a compiler, which decides which parts of the workload will run on which type of compute engines; everything is converted to ONNX before handing relevant parts off to the AI accelerator supplier’s toolchain, or to the backends of on-chip engines. The compiler also generates instruction-level code for the AI-Hypervisor, which describes the compute graph.
Different AI accelerator programming models are supported by adjusting the AI-Hypervisor’s firmware. NeuReality currently supports AMD/Xilinx Alveo V70, IBM AIU and Qualcomm Cloud AI100; creating new firmware for other AI accelerators takes around four weeks, Tanach said. The platform will support AI accelerators from 400 TOPS to 2 POPS.
Above the pipeline layer is a service layer that connects to MLOps/Devops environments, including resource allocation, scheduling and provisioning. The provisioner, part of NRServer running on an on-chip management CPU—which is not part of the datapath—handles runtime assignment of compute engines based on the compiler-generated compute graph, and loads the compute graph descriptor into the AI-Hypervisor.
After that, all client requests coming over the network are to specific preloaded graphs, so the client sends requests over the network and NeuReality’s AIoF network engine terminates requests and loads them to queues in the AI-Hypervisor. When processing is complete, the response is sent back via the AIoF engine.
Appliance or module
NeuReality’s NAPU comes as an NR1-S appliance for CPU-free servers, or an NR1-M module that plugs into CPU server racks to offload CPU tasks.
The company is targeting applications like automatic speech recognition (ASR), NLP, fraud detection, secure telehealth, patient AI search queries, and computer vision, but the biggest opportunity may come with the scale of generative AI inference, Tanach said.
“Affordability is crucial to fuel broader genAI adoption in essential industries,” he said. “We are committed to making conventional AI applications more economically sustainable, paving the way for genAI growth.”