The Long-Term Bottleneck in AI Inference Costs May Shift from Compute to Caching

Disclaimer: This article represents only my personal reflections and review of public industry materials and business logic. It does not constitute any investment advice, nor does it involve any individual stock recommendations or position hints. Markets carry risks; decisions must be made independently.

After reading the official documentation from OpenAI, Anthropic, Google, and DeepSeek, along with the technical docs for vLLM, TensorRT-LLM, and LMCache, my judgment is that once AI serving scales up, competition will expand from "who has more GPUs" to "who can turn repeated computation into reusable cache." On the training side, HBM, optical interconnects, and power sit at the front of the line. On the inference side, caching widens the scope of observation to KV cache, prompt cache, and semantic cache, along with the enterprise SSDs, DRAM, and storage systems behind them.

Part of this logic already holds up: mainstream model providers and serving frameworks have written caching into their documentation as a formal mechanism for cutting costs and latency. OpenAI's official documentation states that Prompt Caching can reduce latency by up to 80% and input token costs by up to 90%, and that a hit occurs only when the prompt prefix is an exact match. In Anthropic's documentation, a cache read costs roughly 10% of the base input price, with two TTL tiers of 5 minutes and 1 hour. Google Gemini's documentation separates implicit caching from explicit caching, using the latter to guarantee cost savings. On DeepSeek's official pricing table, the V4 Flash cache hit input price is $0.0028 / 1M tokens, while a cache miss is $0.14 / 1M tokens, a 50x difference.[1][2][3][4]

Still, the deduction has to stop right here: even if the economics of caching hold up, you cannot extrapolate that "commodity NAND prices have 10x upside." Between those two points sit cache types, hardware tiers, concurrent read/write paths, expiration policies, model version changes, cloud provider bargaining power, and storage-cycle capacity expansions.

Hardware Tiers Across Four Types of Cache

Cache Type	Evidence of Real-World Existence	Primary Benefit	Hardware Likely to Be Used	Key Limitations
Prompt / prefix cache	OpenAI, Anthropic, and Google all have official prompt/context caching docs; vLLM has Automatic Prefix Caching	Reduces prefill cost and first-token latency for repeated prefixes	First, KV state in GPU/HBM; extending lifecycle uses CPU DRAM, or even external KV storage	Requires identical or similar prefixes; hits are affected by TTL, routing, concurrency, and request structure
KV cache reuse	vLLM, NVIDIA TensorRT-LLM, and LMCache all explicitly support KV cache reuse	Skips prefill computation for shared prompts, reducing TTFT and GPU cycles	HBM is the first tier; CPU DRAM can serve as host offload; LMCache supports file/block, S3, Redis/Valkey, distributed storage, and other secondary backends	Primarily improves prefill, does not improve long-answer decode; KV blocks get evicted via LRU
Semantic / RAG cache	Exists in engineering practice, though in public materials what I can directly match are mainly LMCache and serving docs describing reusable text or KV; semantic caching relies more on application-layer implementation	Hits on similar questions, document chunks, and RAG retrieval results	Vector databases, CPU DRAM, NVMe SSD, and object storage, depending on latency requirements	High risks around similarity thresholds and answer quality; cannot be reused via exact prefix matching like prefix cache
Long-document / codebase cache	Google explicitly lists "repeated queries over large document sets, code repository analysis, and long video analysis" as explicit caching scenarios; Anthropic lists long documents, coding assistants, and agentic tool use as applicable scenarios	Lowers the cost of repeatedly passing in long contexts	GPU/HBM + CPU DRAM + enterprise SSD, depending on cache retention time and concurrency scale	Multi-user isolation, permissions, data updates, and cache invalidation significantly increase system complexity

I do think caching will change the cost structure of AI serving, but drawing a direct equals sign to say "caching = NAND" is too crude. Broken down by hardware tier, a more accurate structure has four layers:

1. HBM / GPU memory: The first tier of KV cache, with the lowest latency and the highest cost.
2. CPU DRAM / host memory: NVIDIA TensorRT-LLM documentation mentions that using host memory offload can extend the reuse lifetime of a KV block, though data must be copied between the CPU and GPU.[6]
3. Enterprise NVMe SSD / local disk: Suited for a larger, slower, and more persistent L2 cache, as well as storing RAG data, model offload, and hot datasets.
4. Object storage / distributed KV storage: Suited for cross-instance sharing and low-frequency reuse, though latency, throughput, and consistency are harder to manage.

So the hardware demand driven by AI caching looks more like a tiered caching stack running from "HBM → DRAM → NVMe SSD → object storage," with a relatively low probability of growth concentrated in one tier.[5][6][7]

How Much Unit Cost Can Drop Depends on Four Multipliers

The public evidence available so far proves the direction of cost reduction, but it cannot yet be used directly to prove the profit margins of individual companies.

First, consider the facts stated in the documentation:

• OpenAI's official documentation states that Prompt Caching can reduce latency by up to 80% and input token costs by up to 90%. A cache hit only takes effect on an exact-match prefix, and requests are routed by prefix hash to a server that recently processed the same prompt.[1]
• Anthropic's documentation shows that cache writes are priced higher than regular input tokens, while cache reads cost about 10% of the base input price. Applicable scenarios include long system prompts, many-shot examples, long documents, multi-turn conversations, and agentic tool use.[2]
• DeepSeek's price spread is even more aggressive: for V4 Flash, the hit price is 2% of the miss price, or 98% lower; for V4 Pro, the hit price is roughly 0.833% of the miss price, or 99.17% lower.[4]
• vLLM explicitly states that Automatic Prefix Caching mainly saves on prefill and does not reduce decode when generating new tokens. When answers are very long, or when requests do not share a prefix, the benefit drops.[5]

To extrapolate from there, I break the change in unit cost into four multipliers:

单位成本下降幅度 ≈ cache hit rate × 可缓存输入占比 × prefill 在总成本中的占比 × hit/miss 价格差

Taken together, these four terms show quite concretely how different scenarios and vendors diverge:

• For long system prompts, long documents, codebases, RAG, and multi-turn agent tasks, the caching benefits may be substantial.
• For short questions, long outputs, heavy personalization, real-time tool calls, and low-repetition queries, the benefits shrink noticeably.
• For model providers with a large user base, the cache pool can more easily cover high-frequency queries, making hit rates more likely to improve.
• Smaller model providers may lack sufficient hit rates and the scale to amortize infrastructure, which widens the cost gap.

Even so, public materials only prove that caching lowers token billing and serving latency; they do not prove the true unit gross margins of OpenAI, Anthropic, Google, or DeepSeek. Model architecture, inference cluster utilization, batching, quantization, routing, product subsidies, enterprise discounts, and the share of free users all feed into actual profits in the end.

Examining Who Might Benefit Across Three Layers

I look at who might benefit across three separate layers: cloud and model providers, the software stack, and the hardware supply chain. The strength of evidence is not in the same tier across all three. For service providers, I buy the business logic that caching improves gross margins, though I leave aside exaggerated metaphors like "money-printing machine." For the hardware chain, I only buy a tier-by-tier transmission mechanism, and do not buy narratives where a single layer takes off in isolation.

Cloud Vendors and Model Providers

Once a cache hit occurs, a service provider can charge customers at a product value close to that of regular tokens while fulfilling the request at a lower compute cost. Some commentary in the materials calls this a money-printing machine, a metaphor that clearly goes too far; yet the underlying business logic holds, and gross margins, latency, and user experience could all improve.

The Serving Software Stack

At this layer, vLLM, TensorRT-LLM, and LMCache become much more important. LMCache's documentation states that it stores the KV cache of reusable text so that any serving instance can reuse it, lowering TTFT and saving GPU cycles and memory. Combined with vLLM in scenarios such as multi-turn QA and RAG, the project team claims 3–10x delay savings and GPU cycle reduction.[7]

Judging from these mechanisms, caching capabilities may evolve into an infrastructure differentiator among model clouds, doing more than minor application-layer optimizations. As for the 3–10x benefit figures, they are currently self-reported by the project team, so we will need independent benchmarks and production cases later to verify them.

Looking at the Hardware Chain by Tier

Hardware / Company Type	Rationale for Benefiting	Evidence Strength	Key Risks
HBM / High-end DRAM	First tier of KV cache and core resource for inference throughput; Micron's HBM page states HBM targets AI and HPC, supporting AI training and inference	Strong	Already fully priced in by the market; supply expansion and customer concentration affect profits
CPU DRAM / high-capacity memory	Host offload, long context, multi-instance caching, and inference server memory capacity	Moderately strong	Weaker price elasticity than HBM; cloud vendors have strong bargaining power
Enterprise NVMe SSD	Model offload, RAG hot data, secondary KV cache, and AI training/inference data paths; Micron 9550 official materials state it targets AI workloads, improving training time and lowering SSD/system power consumption in BaM/GIDS scenarios; DeepSpeed ZeRO-Inference can use SSDs for intelligent offload	Moderately strong	Whether this translates into NAND manufacturer profits depends on order volume, contract prices, competition, and capacity expansion
NAND Manufacturers	If hot datasets, cache pools, and secondary KV storage grow at scale, NAND bit demand benefits	Moderate	Commodity NAND is cyclical; supply expands quickly, and prices are easily suppressed by cloud vendors
SSD Controllers / Storage Systems / Network Storage	If the bottleneck lies in IOPS, latency, controllers, firmware, and distributed caching rather than raw NAND bits, value-add may shift here	Moderate	Requires identifying specific pure-play companies and financial statement revenue lines

Faced with this table, I won't boil it down to a simple verdict on "who benefits the most." A sounder approach is this: once inference shifts into a competition over operational efficiency, watch the links in the caching stack that are hardest to replicate quickly and most likely to show up in orders and gross margins. My verification sequence is to check first whether cloud vendors explicitly increase procurement of enterprise SSDs, DRAM, and storage systems; next, whether storage vendors' earnings reports cite AI inference, RAG, and cache workloads as demand drivers; and only last whether the value-add lands in enterprise SSDs, controllers, storage networking, caching software, or high-end DRAM. Commodity NAND manufacturers may benefit, but they still carry strong cyclicality.

Where This Thesis Would Weaken

The points below are the conditions that would make me lower my expectations. If any one of them holds true, I will dial back my expectations for "caching rewriting hardware demand" by a notch.

• If model architectures move away from Transformer KV cache toward paths that require less caching, the benefit to KV cache will decline.
• If cache hits happen primarily inside service providers without requiring large amounts of persistent storage, commodity NAND demand will be weaker than deduced above.
• If caching stays mainly in HBM and DRAM while SSDs serve only as a cold tier, NAND's price elasticity will not be that large.
• If cloud vendors lock in storage prices through long-term contracts, supplier revenue growth will not necessarily turn into gross margin expansion.
• If service providers cut costs via smaller models, quantization, batching, and speculative decoding, caching is only one of several cost-reduction paths.
• If the industry expands capacity too quickly, NAND could still follow traditional cyclical commodity logic.

Conversely, if storage prices rise high enough to rewrite whole-system BOM and capital expenditure budgets, customers will also look more aggressively for caching, offload, and architectural workarounds. I previously wrote about how this kind of overpricing backfires in Overpricing Bottlenecks Hurts AI CapEx.

Public Evidence Still Missing

I do not yet treat the five points below as established facts. For my earlier judgment to keep holding up, I will need to watch for this evidence in public materials going forward:

• Whether OpenAI, Google, Anthropic, and DeepSeek continue to disclose cache hit / miss price spreads, cache hit tokens, and latency improvements.
• Whether vLLM, TensorRT-LLM, and LMCache move from optional features to default deployment components in inference clusters.
• Whether enterprise SSD vendors list AI inference, RAG, KV cache, and model offload as new demand drivers in their earnings reports.
• Whether NAND contract prices, enterprise SSD ASP, inventory days, and capex guidance improve in tandem.
• Whether the share of memory, storage, and networking outside of GPUs rises within cloud vendors' AI capex.

References

1. OpenAI Prompt Caching Documentation, official product documentation, high credibility; the limitation is that it only explains the mechanisms of the OpenAI API and the upper bounds of price and latency benefits, without disclosing true gross margins. https://platform.openai.com/docs/guides/prompt-caching
2. Anthropic Prompt Caching Documentation, official product documentation, high credibility; the limitation is that pricing and TTL reflect Anthropic's current product strategy and do not equal industry-wide baseline costs. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
3. Google Gemini Context Caching Documentation, official product documentation, high credibility; the limitation is that the cost savings of explicit/implicit caching are tied to Google's models and billing methods. https://ai.google.dev/gemini-api/docs/caching
4. DeepSeek Models & Pricing, official pricing page, high credibility; the limitation is that prices can be adjusted, and price spreads do not equal true underlying cost differences. https://api-docs.deepseek.com/quick_start/pricing
5. vLLM Automatic Prefix Caching Documentation, open-source inference framework documentation, relatively high credibility; the limitation is that it describes framework capabilities and does not mean all production deployments use the exact same configuration. https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html
6. NVIDIA TensorRT-LLM KV Cache Reuse Documentation, official technical documentation, high credibility; the limitation is that it focuses on the deployment path of TensorRT-LLM. https://nvidia.github.io/TensorRT-LLM/advanced/kv-cache-reuse.html
7. LMCache Documentation, open-source project documentation, medium-to-high credibility; the limitation is that performance gains are self-reported by the project team and need to be verified against independent benchmarks and production cases. https://docs.lmcache.ai/
8. Micron 9550 SSD Product Page & AI Workload Blog, official vendor materials, suitable for illustrating product positioning and vendor perspectives; the limitation is that they are marketing materials, so performance figures need to be verified alongside third-party benchmarks and actual procurement. https://www.micron.com/products/storage/ssd/data-center-ssd/9550-ssd