TileRT: The Runtime Turning AI Speed Into A Product Moat

TL;DR: Xiaomi claims its 1-trillion-parameter MiMo-V2.5-Pro-UltraSpeed mode exceeded 1000 decode tokens/s on one standard 8-GPU node through FP4 quantization, DFlash speculative decoding, and TileRT co-design^[1]. Z.ai separately documents a TileRT co-developed GLM-5.1-HighSpeed service at a vendor-reported 400 output tokens/s, with a 200K context window and 128K maximum output for selected enterprise customers^[4]. The figures are not comparable. The commercial signal is: Xiaomi put a premium on speed, Z.ai packaged it for enterprise use, and the runtime became part of the product.

Xiaomi did something more strategically revealing than publish a 1000 tokens/s demonstration. For a two-week June trial, it charged 3x the standard MiMo-V2.5-Pro API price while claiming roughly 10x the generation speed^[1]. Xiaomi put a price on waiting.

Z.ai made the quieter move. It placed a TileRT co-developed 400 output tokens/s service behind selective enterprise access and kept the features buyers expect from a flagship endpoint: long context, streaming, tool calling, structured output, and MCP support^[4]. One deployment tested a premium speed tier. The other tested whether low latency could survive inside a managed product.

A frontier model that arrives late is not a frontier product. It is a waiting room with weights.

That is the hook inside TileRT. The project looks like systems plumbing for people who enjoy kernel timelines and GPU profilers. The commercial signals say otherwise. While competitors keep selling intelligence as a benchmark table, TileRT is selling time: time to the first useful answer, time for a coding agent to iterate, and time for a voice system to respond before the conversation feels dead.

The New Premium SKU: Speed

Most inference stories begin with hardware utilization or cost per million tokens. TileRT's story begins with the wall-clock budget of the product. The real story isn't merely that TileRT makes models faster. It is that TileRT turns speed into a design primitive.

Agents do not only need answers. They need loops. A coding agent needs to propose, test, patch, and re-evaluate. A voice agent needs to respond before the conversation feels dead. In those settings, a model's value is shaped by what it can do inside a fixed latency budget.

TileRT's homepage pushes that argument to its promotional limit. It frames the market as moving from model quality, to token throughput, to speed itself as demand^[2]. The slogan is self-serving. The product logic is not.

What's often overlooked is that speed is not a vanity metric once models become workers. A faster model can attempt more rollouts, recover from mistakes sooner, and complete more tool loops inside the same user-visible delay.

The commercial thesis is not that every model needs 1000 tokens a second. It is that response time can be priced, packaged, and sold separately from model capability. Xiaomi and Z.ai supply different parts of that proof.

Two References, Two Different Proofs: Xiaomi And Z.ai

Xiaomi is the pricing and co-design signal. Z.ai is the managed-service signal. Reading their raw TPS figures as a race would miss the point, because the two deployments answer different commercial questions.

Xiaomi: The Public Co-Design And Pricing Proof

Xiaomi's June 8 announcement is the cleanest public TileRT reference because it ties the runtime to a high-profile product claim. Xiaomi presents MiMo-V2.5-Pro-UltraSpeed as a high-speed mode for its 1T-parameter MiMo-V2.5-Pro model and says the UltraSpeed release was developed with TileRT^[1].

The headline number is blunt: Xiaomi claims more than 1000 decode tokens/s and displays real-time generation peaking near 1200 tokens/s^[1]. The configuration matters more than the peak. Xiaomi says the result uses a single standard 8-GPU commodity node, not custom silicon^[1].

Here's the genius: TileRT is not positioned as a standalone magic layer. Xiaomi describes the result as extreme model-system co-design. FP4 quantization attacks memory bandwidth by selectively quantizing the MoE experts. DFlash speculative decoding reduces sequential depth by filling masked blocks in parallel^[12]. TileRT then adapts the system side to those algorithmic choices with a compilation engine and kernels built around the dynamic behavior of the pipeline^[1].

The chart is a commercial signal, not a unit-economics calculation. Xiaomi limited the offer to June 9 through June 23, 2026 and did not publish enough workload detail to normalize the ratio^[1]. Still, the experiment established the proposition: charge less than the claimed speed multiplier and sell the difference as responsiveness.

Xiaomi supplied the public pricing signal. Z.ai supplied the operational packaging.

Z.ai: The Managed-Service Proof

Z.ai's documentation for GLM-5.1-HighSpeed describes a high-speed version of the flagship GLM-5.1, optimized across inference engine, scheduling system, and infrastructure. Z.ai reports 400 output tokens/s and limits access to selected enterprise customers on the BigModel platform^[4]. A limited Toolin hands-on report observed 300 to 350 tokens/s in its comparison table, below the provider's headline, but did not disclose enough protocol detail to certify either number^[10].

The details are more revealing than the headline. GLM-5.1-HighSpeed keeps a 200K context window, supports a 128K max output length, and lists thinking mode, streaming, function calling, context cache, structured output, and MCP support^[4]. In other words, this is not framed as a toy fast path. It is positioned for coding agents, real-time interaction, business decision support, and real-time voice^[4].

TileRT's engineering post offers a plausible explanation for the category, not a disclosed bill of materials for Z.ai's endpoint. TileRT estimates that an 8x H200 server has nearly 38 TB/s of aggregate memory bandwidth and that GLM-5.1 activates about 42 GB per decode. Dividing those vendor-supplied figures yields an ideal bandwidth-only ceiling of roughly 905 tokens/s without MTP. That is back-of-the-envelope arithmetic, not an end-to-end benchmark. Communication, synchronization, cache behavior, compute, request scheduling, and sampling all reduce real output speed^[3][13].

That gap is the business. The product claims explain why speed matters. The execution model explains why it is difficult to reproduce.

These are not leaderboard scores. Xiaomi reports a vendor demonstration for a co-designed 1T model on one 8-GPU node. Z.ai documents a managed-service claim for a different model without matching hardware or test conditions. Neither source provides the same prompt length, output length, sampling settings, speculative-decoding acceptance rate, concurrency, time-to-first-token, tail latency, or end-to-end harness. The figures show separate product positions, not a direct speed ranking.

Let's be clear: neither reference proves TileRT will become the universal LLM runtime. The supported path is narrow, the hardware assumptions are serious, and the model-specific work is deep. Together, the deployments establish something smaller but still strategic: two vendors are trying to package latency as a distinct product property. To understand why that property may become defensible, the story has to move below the API.

The Mechanism: Closing The Execution Gap

The product story ends at the endpoint. The engineering story begins in the idle space between kernels. TileRT's core complaint is that traditional inference systems are built around the wrong unit of work. They launch operators. TileRT wants to schedule tiles.

In conventional frameworks, each operator becomes an execution boundary: host launch, synchronization, memory movement, compute, store, and then another operator. That model works tolerably well when each kernel is large enough to hide overhead. It becomes hostile when decode is squeezed into microsecond-scale steps.

TileRT's answer is persistent execution. The system compiles the model ahead of time into a long-running engine kernel. The host launches once. Execution stays resident on the GPU. Work is decomposed into tile-level tasks. Warp groups specialize into data movement, tensor compute, and communication roles. Intermediate results stay closer to registers, shared memory, and L2 cache rather than constantly round-tripping through global memory^[3][7].

The key phrase is "execution gap." TileRT uses that language for the idle space created by kernel launches, barriers, communication waits, and memory round trips^[3]. In older throughput-first systems, those gaps could be amortized. In ultra-low-latency systems, they become the product constraint.

This is an architectural thesis, not a published cross-runtime benchmark. The sources explain how TileRT says it closes execution gaps; they do not establish a uniform lead over every serving stack, model, batch size, or hardware generation. For rival inference stacks, the strategic implication is not a proven performance crown. It is that TileRT is betting on a different abstraction boundary.

That boundary leads directly to the moat, and to the constraint. The same specialization that makes TileRT interesting also makes it difficult to generalize.

The Moat And The Constraint: A Narrow Fast Path

This is where mechanism becomes strategy. TileRT is potentially valuable because it is not generic, and risky for the same reason. It sits inside a tile-ai ecosystem that includes TileLang, TileOPs, and TileScale^[2].

TileLang is the most established open-source signal in that ecosystem. Its repository describes a Pythonic domain-specific language for high-performance GPU, CPU, and accelerator kernels, with examples including GEMM, dequant GEMM, FlashAttention, and linear attention^[8]. TileRT's long-term story is not simply "install a wheel and go faster." It is a compiler culture that tries to align kernels, operators, distributed execution, and model-serving paths around tile-level thinking.

Compiler ecosystems become moats slowly, then suddenly. TileRT is far earlier and narrower than CUDA or Triton, and it has not earned their position. The strategic ambition is still recognizable: make the fast path programmable enough that model teams start designing around it.

The caveat is serious. TileRT's public release is still specialized, opinionated, and hardware-bound.

The June 2 v0.1.4 release focuses on faster decoding for DeepSeek-V3.2 and GLM-5^[6]. Its README says the wheel is a pre-built binary linked against exact ABI assumptions, with a pinned environment: 8x NVIDIA B200, CUDA 13.2 runtime support, Linux x86_64 with glibc >= 2.28, Python 3.12, PyTorch 2.11.0+cu130, transformers 4.46.3, and tokenizers 0.20.3^[5]. It also says v0.1.4 ships separate backend libraries for DeepSeek-V3.2 and GLM-5, with one backend loaded per Python process^[5].

That is not a casual developer experience. It is an infrastructure product in open-source clothing.

The trade-off is the story. An abstraction narrow enough to exploit model and hardware behavior can be commercially powerful, but it is not yet a general-purpose serving layer. The valuable systems will know enough about the model to delete waste, enough about the hardware to keep it fed, and enough about the product to optimize the delay users actually feel.

The question is no longer whether a narrow fast path can produce an impressive result. Xiaomi and Z.ai say it can. The harder question is whether TileRT can widen its model and hardware support without giving up the specialization that created the advantage. Its short release history is the first test.

The Seven-Month Test: What Must Happen Next

TileRT has moved fast enough that its version history is part of the strategy. The v0.1.0 alpha release began with an experimental DeepSeek-V3.2-Exp path in November 2025^[11]. By February, v0.1.3 had added GLM-5, Multi-Token Prediction, Top-P sampling, thinking mode, and longer context support^[9].

In roughly seven months of visible public history, TileRT moved from an experimental DeepSeek path to GLM support, a selective enterprise endpoint, and Xiaomi's headline deployment. The next test is breadth: more supported model architectures, reproducible comparisons on disclosed configurations, and speed tiers that persist after promotional windows.

The Bottom Line: Runtime Is Becoming The Product

TileRT is not the default way the world serves LLMs. It may never be. It does not need to become universal to matter. The constraints are real, the supported path is narrow, and the performance depends on model-specific engineering.

The evidence is narrower than a market verdict. Xiaomi's 1000+ figure and Z.ai's 400 figure remain vendor-reported, non-comparable claims. What they establish is the direction of travel: two vendors are attempting to sell low latency as a distinct product property, not merely an internal serving metric.

The uncomfortable truth is that speed becomes commercial before benchmarks become clean. TileRT has not proved that one runtime will rule LLM serving. It has shown how model teams can turn a deeply co-designed fast path into a premium experience. Once latency becomes a SKU, the runtime is no longer backstage machinery. It is product strategy.

That connects TileRT directly to the same infrastructure shift behind DeepSpec's inference economics and TwELL's sparse-kernel bet: the cost of intelligence is becoming inseparable from the time it takes to arrive.

Last updated: July 18, 2026