15 posts tagged with "Edge AI"

手机 NPU 的峰值算力九年间从 0.6 TOPS 涨到 80 TOPS，但一个 3B 模型在端侧的流畅运行，靠的早已不是 TOPS 数字。当云端推理进入万亿参数时代，端与云的分工边界在哪里，正在被重新谈判。

移动 NPU 的进化与瓶颈​

从 2017 年麒麟 970 与苹果 A11 首次引入移动 NPU，到 2026 年的旗舰平台，端侧 AI 算力完成了百倍跨越：

但 TOPS 从未是真实体验的保证。第三方评测显示，骁龙 8 Elite Gen 5 在运行 8B 量化模型时会因 LPDDR5X 总线带宽（约 85GB/s）饱和而跌到每秒 12 个 tokens 附近——内存带宽而非算力，才是端侧推理的第一瓶颈。峰值 80 TOPS 跑不快一个 8B 模型，这就是移动 AI 的现实：算力年年翻倍，带宽却受制于手机内存的功耗与体积约束，增速远远掉队。

端云分工：三层负载的清晰分层​

2026 年旗舰机型的实践已形成相对稳定的分工范式：

• 端侧必留：唤醒词、拍照语义处理、实时转写、通话翻译等常驻功能——低延迟、离线可用、隐私敏感，且可在微瓦级功耗下待机运行；
• 端侧可选：3B 到 8B 级量化小模型承担的摘要、改写、本地图像生成，牺牲部分质量换取零网络依赖；
• 云端包办：前沿推理模型、视频生成、复杂智能体编排——端侧算力与显存天花板决定了这些负载短期内无可下沉。

三星宣称到 2026 年底将 Galaxy AI 覆盖 8 亿台设备，其策略正是典型的混合路由：个人数据本地处理，复杂推理按需上云。谷歌 Tensor G5 的思路则提供了另一个注脚——它不在原始算力上争第一，而是专为 Gemini Nano 做软硬协同优化，用更低的 TOPS 跑出更好的实际效果。

边界正在向哪里移动​

三个变量将决定端云边界的走向。其一是内存技术：移动内存带宽的持续爬升与更大容量封装的普及，可能把端侧可流畅运行的模型推到 10B 以上；其二是模型效率：蒸馏与量化每前进一代，端侧可承载的能力就上移一档；其三是隐私监管：数据出境与个人信息保护要求趋严，会持续把更多推理"推"回设备内部。综合业界普遍预期，端侧将稳态承接约七成的推理调用次数，但按算力消耗总量计，云端仍将占据绝对大头——调用次数的多，不等于算力的大。

结语：端云不是对手，而是流水线的两站​

把端云关系理解为竞争，是对这场变革最大的误读。手机 NPU 的角色是"常驻的感官与反射"，云端大模型是"按需调用的大脑"，两者通过意图路由与模型分级协同工作。对芯片产业而言，这意味着两端将长期并行增长：云端拼每瓦 tokens，端侧拼每毫安时 tokens——两个优化方向，同样残酷，也同样机会巨大。真正的赢家，是那些把"什么时候在端、什么时候上云"做成无感体验的厂商。

RTX Spark 笔记本的首发潮终于来了。微软与联想相继公布旗舰级 AI PC：一款把矛头直接对准 MacBook Pro，一款则用128GB 统一内存和1 PFLOP FP4 算力重新定义 Windows 阵营的端侧 AI 上限。这批产品背后，是一段长达三年的等待史。

在 AI 芯片的舆论场里，云端训练卡占据九成话题，但真正以百万级数量铺进街道、园区、工厂和交通路口的，是另一类低功耗边缘芯片。算能科技的 BM1684X 就是其中的隐形冠军：它很少出现在发布会头条，却是安防集成商选型清单上的常客。

规格：把"够用"做到极致​

BM1684X 属于算能第四代自研 TPU（张量处理器），基于 Bernoulli 架构，采用台积电 12nm 工艺。它的高度集成是最大特色——一颗 SoC 同时集成 8 核 ARM Cortex-A53 处理器、TPU 计算单元与视频编解码引擎：

为什么集成商爱它​

边缘 AI 的交付逻辑与云端完全不同：客户要的不是峰值算力，而是每瓦性能、每路视频成本和开发周期。BM1684X 在这三点上都有硬通货。

首先是视频能力。32 路 1080P 硬解码加上 AI 推理再编码回流的全流程能力，让一台巴掌大的盒子就能完成一个路口的人脸识别、车牌识别与视频结构化，无需额外服务器。其次是功耗与散热：满载不到 20 瓦、被动无风扇设计、支持宽温运行，意味着它可以装在户外机箱里常年免维护。第三是软件成熟度：SOPHON SDK 提供从模型编译到部署的一站式工具链，支持 PyTorch、TensorFlow、ONNX、PaddlePaddle 等主流框架一键转换，Docker 容器化部署让算法上线如同拷贝文件。

值得一提的是，随着开源小模型成熟，BM1684X 还解锁了新玩法：据公开报道，它已可本地运行 LLaMA2、Qwen、ChatGLM、Whisper 等轻量化模型，把边缘盒子从"视频分析器"升级为"本地小脑"。

形态与生态位​

BM1684X 的落地形态极为丰富：既可以是独立运行的边缘计算盒子（SoC 模式），也可以作为 PCIe 加速卡插入边缘服务器（PCIe 模式）。英码、天启等大量方案商基于它推出标准品与行业定制机，覆盖智能安防、智慧交通、智慧工地、智慧零售、工业质检等场景。在其之上，算能还有更低功耗的 BM1688（16 TOPS）承接成本敏感型需求，形成高配与低配的完整梯度。

从公司背景看，算能由比特大陆的 AI 芯片业务发展而来，继承了多年 TPUNB 自研架构积累；据公开报道，其边缘 AI 产品在国内安防市场占有重要份额，前代 BM1684 的累计出货已达相当规模。

小结：边缘市场没有新闻，只有交付​

BM1684X 的故事没有惊心动魄的转折，只有对真实需求的持续打磨：20 瓦功耗、32 路解码、成熟的 SDK、透明的价格。当行业目光都聚焦在云端大芯片的军备竞赛时，正是这类产品在支撑 AI 的最后一公里。对于需要大规模部署视频分析与边缘推理的团队，BM1684X 仍是当下验证最充分、生态最完善的国产选择之一。

2026 年秋天，"统一内存"成了 AI 硬件圈最卷的词。AMD 和 NVIDIA 在 AI PC 上贴身肉搏，桌面端堆出 748GB 的"超算"，数据中心则用 HBM4 把带宽拉到 22TB/s。容量数字一路上涨，但真正决定体验的，是那个藏在参数表角落里的带宽。

统一内存要解决什么​

传统 PC 架构里，CPU 和 GPU 各管一摊内存：模型文件躺在系统内存里，推理前要先通过 PCIe"搬运"到显存，模型一大，加载时间以分钟计，显存不够还会溢出到磁盘。统一内存的思路是把这堵墙拆掉——CPU 和 GPU 共享同一块物理内存，模型加载变成一次指针映射，权重不用搬，KV Cache 也能被两个处理器同时访问。

AMD 用 VGM（可变显存滑块）实现这一点：Ryzen AI Max+ 395 的 128GB 内存里，最多 96GB 可划给 GPU 当显存；新的 Pro 495 则允许从 192GB 里划出 160GB。NVIDIA 的路线更彻底，RTX Spark 的 GB10 超级芯片通过 NVLink-C2C 让 Arm CPU 与 Blackwell GPU 直接共享 128GB LPDDR5X，地址空间完全一致。

四档产品对比表​

2026 年 10 月的市场上，统一内存产品已经拉开四个档次：

值得注意的是两家新品发布的贴身节奏：AMD 在 10 月 5 日公布 Pro 495，两天后 NVIDIA 就办了 RTX Spark 新品活动。AMD 还准备了后续牌——代号 Gorgon Halo 的 Ryzen AI Max 400 同样最高 192GB，Pro 495 加速到 5.2GHz，并宣称是首款能在本地跑 300B 级模型的 x86 客户端处理器。整机侧，联想 YOGA Pro 15 搭载 RTX Spark，统一内存规格为 9400MT/s、256-bit 位宽。

带宽才是隐藏变量​

把四档产品的带宽排成一列，差距比容量悬殊得多：256GB/s（395）、约 272GB/s（Pro 495）、273GB/s（DGX Spark 64GB）、300GB/s（RTX Spark 128GB），再到 7.1TB/s 与 22TB/s——AI PC 与数据中心之间差了整整两个数量级以上。

这解释了一个反直觉的现象：容量相同的两台机器，token 生成速度可能差出数倍。大模型推理的 decode 阶段是典型带宽受限负载，每生成一个 token 都要把权重和 KV Cache 扫一遍，带宽直接决定每秒吐出多少字。RTX Spark 靠 300GB/s 在四款 AI PC 里领先；而 DGX Station 的 748GB 之所以值钱，不只因为容量能装下万亿参数，更因为 252GB HBM3e 提供的 7.1TB/s 带宽让模型真的跑得动，而不是"装得下、跑得慢"。

还有一档容易被忽略：DGX Spark 双机通过内置 ConnectX-7 的 QSFP 直连，可把内存池化到 128GB，模型上限提升至 2000 亿参数级，Qwen3.8 27B 实测峰值较单台提升最高 70%——用网络扩展统一内存，是介于单机和工作站之间的第三条路。

VGM 与 CUDA 统一内存的分配策略差异​

同样叫统一内存，AMD 和 NVIDIA 的分配哲学并不相同。AMD 的 VGM 是显式的静态划分：用户拖滑块决定划多少 GB 给 GPU，划出去的部分 CPU 不能再用。优点是行为可预测、显存语义清晰，对 llama.cpp、Stable Diffusion 这类工具友好；代价是弹性差，划多了浪费、划少了要重划。

NVIDIA 的 CUDA 统一内存是动态的：CPU 与 GPU 共享统一地址空间，按需在两者间迁移页面。开发者不用预判"该分多少"，跑什么模型都是同一套逻辑；但页面迁移的粒度和时机由运行时控制，遇到突发访问模式时可能引入抖动。对追求开箱即用的消费用户，CUDA 路线体验更顺滑；对需要稳定显存预算的 Agentic 工作流，VGM 的确定性反而更吃香。

选型建议：按模型规模分档​

小结​

统一内存之战没有单一最优解：128GB 与 192GB 争夺 AI PC 入场券，748GB 的 DGX Station 把数据中心内存架构搬上桌面，288GB HBM4 则继续守卫机架级吞吐。选型的第一变量不是容量，而是带宽——256GB/s 与 22TB/s 之间隔着三个数量级的体验鸿沟；第二变量是分配策略——VGM 的确定性划分与 CUDA 的动态迁移各有适配场景。先想清楚要跑多大的模型、多少并发，再看参数表，才不会被大数字带偏。

In 2026, the AI hardware market is undergoing a fundamental shift from the "training race" to "deployment as king." As large models move from technology demos to large-scale commercial deployment, hardware form factors, technology roadmaps, and the competitive landscape are undergoing systematic change.

Publisher: CSHIA Research (中智盟咨询) Author: Zhou Jun

Five Core Trends​

Trend 1: Shift in compute demand structure — inference becomes the main engine of growth​

The biggest change in the 2026 AI hardware market is the shift in the center of gravity of compute demand from training to inference.

According to market data:

• In 2026, global AI inference compute demand is expected to grow over 60% year-over-year
• Inference compute will exceed training compute for the first time, becoming the dominant workload of AI infrastructure

This shift stems from AI applications moving from "model development" into the "large-scale deployment" stage — enterprises no longer train large models frequently, but instead transform AI capability into real business value through high-frequency inference calls.

Key manifestations​

1. Inference chip market explosion: Shipments of dedicated inference chips (ASICs) are expected to grow 129%, with their share of AI servers rising from under 20% in 2025 to 27.8%.

2. Cost structure optimization: NVIDIA's Rubin platform reduces inference token cost to 1/10 of the previous generation, pushing inference applications from "luxury" to "commodity."

3. Workload characteristics change: Inference tasks show "high-frequency, long-pipeline, low-latency" characteristics, demanding higher real-time responsiveness from hardware.

Latest GTC 2026 developments (June 1, Taipei)​

NVIDIA CEO Jensen Huang announced several major inference compute advances at GTC 2026 Taipei:

• Vera Rubin platform enters full production: The NVL72 rack system delivers agentic throughput 10× that of the previous-generation Grace Blackwell, designed for Agentic AI
• Vera CPU officially launched: 88-core Armv9.2 custom Olympus architecture, highest single-thread IPC in the world, 1.5TB LPDDR5X memory, 1.2 TB/s bandwidth, native FP8 support
• RTX Spark AI PC chip: Co-developed with MediaTek (codename N1X), Blackwell-architecture GPU with 1 PFLOP AI compute, 128GB unified memory, TSMC 3nm, reshaping the Windows PC ecosystem
• AI Factory platform DSX: Four components — DSX Sim (digital-twin simulation), DSX OS (resource orchestration), DSX MaxLPS (power optimization), DSX Flex (grid coordination)

This trend means the competitive focus for hardware vendors is no longer "peak single-card compute" but "inference energy efficiency" and "system-level optimization capability."

Trend 2: Edge and on-device AI — the scaled deployment of compute moving downstream​

2026 is the pivotal year for edge AI hardware moving from proof-of-concept to scaled deployment.

As cloud inference cost pressure rises and privacy compliance requirements tighten, compute is accelerating its migration toward data sources, spawning explosive growth in hardware form factors such as edge servers, AI terminals, and smart devices.

Three deployment scenarios​

Technology breakthroughs​

1. On-device model compression: Through quantization, distillation and other techniques, models with tens of billions of parameters are compressed to run on-device.

2. Heterogeneous compute architecture: CPU+NPU+GPU coordination maximizes performance under power constraints.

3. Memory bandwidth optimization: Application of HBM technology in edge chips alleviates the "memory wall" problem.

The edge AI explosion means hardware design must balance "performance density" with "power efficiency," and traditional general-purpose chips face specialization challenges.

Trend 3: Dedicated chips and heterogeneous computing — breaking the monopoly of a single architecture​

In 2026 the AI chip market will show a "one superpower, many strong players, a hundred flowers blooming" competitive landscape.

Although NVIDIA maintains its advantage in training, in segmented markets such as inference, edge, and specific scenarios, dedicated chips (ASICs) and heterogeneous computing solutions are rising rapidly.

Major technology roadmap comparison​

Market landscape changes​

1. Domestic substitution accelerates: China's AI chip vendors raise their share in inference, edge and other scenarios to over 30%.

2. Open-source ecosystem rises: Open-source frameworks such as ROCm and OpenML lower the barrier to dedicated-chip development.

3. Chiplet technology popularizes: Integrating chips of different process nodes through advanced packaging achieves a balance of performance and cost.

4. GTC 2026 new products accelerate deployment (June 1, Taipei):

   • Vera Rubin platform: NVL72 rack system, agentic throughput 10× Grace Blackwell
   • Vera CPU: 88-core Olympus custom architecture, designed for Agentic AI low latency
   • RTX Spark: In partnership with MediaTek and Microsoft, reshaping the Windows PC ecosystem, 1 PFLOP AI compute
   • Nemotron 3 Ultra: SSM+MoE hybrid architecture, 5× faster inference, 30% lower cost

The core logic of this trend is: no single chip can dominate all AI scenarios; scenario fragmentation spawns technology-roadmap diversification.

Trend 4: Energy efficiency and thermal management — from technical challenge to business bottleneck​

As AI chip power consumption breaks the kilowatt level (NVIDIA Rubin GPU reaches 2300W), energy efficiency and thermal management have been upgraded from "supporting technology" to "core bottleneck."

In 2026, single-cabinet power density will exceed 240kW, traditional air cooling completely fails, and liquid cooling changes from "optional" to "mandatory."

Key data​

• Power cost share: The share of power cost in AI data center operating cost rises from 15% to 35%
• Thermal value increases: A single GB300 server's liquid-cooling components are worth about $50,000, 15-20% of hardware cost
• PUE optimization: Liquid-cooled data centers can bring PUE down to under 1.1, but upfront investment rises 30%

Technology evolution directions​

1. Tiered liquid cooling: Cold-plate (mainstream), immersion (high density), two-phase cooling (frontier)

2. Power architecture upgrade: From 12V to 48V/800V high-voltage DC, reducing conversion losses

3. Intelligent thermal management: AI predictive cooling, dynamically adjusting cooling strategy based on load

This trend means a hardware vendor's competitiveness depends not only on chip performance but more on "system-level energy efficiency optimization capability"; the importance of supporting technologies such as thermal management, power delivery, and cabinet design rises substantially.

Trend 5: AI-native hardware ecosystem — from "compatibility" to "reconstruction"​

In 2026, AI hardware is undergoing a paradigm shift from "adapting to AI" to "built for AI."

Traditional general-purpose hardware architectures struggle to meet the unique demands of AI workloads, spurring the rise of AI-native hardware design philosophy.

Three reconstruction directions​

1. Compute architecture reconstruction​

• Memory hierarchy optimization: HBM4 memory bandwidth breaks 3TB/s, compute-in-memory architecture reduces data movement
• Interconnect upgrade: NVLink 6.0 reaches 1.8TB/s bandwidth, supporting direct GPU-to-GPU communication
• Heterogeneous integration: Through advanced packaging, CPU, GPU and memory are stacked to boost bandwidth and reduce latency

2. Software-defined hardware​

• Reconfigurable logic: FPGA and DPU support dynamic algorithm loading, adapting to different AI models
• Compiler optimization: AI compilers (e.g., MLIR) automatically optimize hardware resource allocation
• Hardware abstraction layer: Unified programming interfaces shield underlying hardware differences

3. Ecosystem co-evolution​

• Model-hardware co-design: Large-model architectures account for hardware constraints (e.g., sparsification, quantization)
• Open-source hardware design: Application of RISC-V in AI chips lowers the development barrier
• Vertical integration: Cloud vendors' self-developed chips (e.g., AWS Graviton, Google TPU), software-hardware co-optimization

The essence of this trend is: the characteristics of AI workloads (matrix operations, high parallelism, memory sensitivity) are redefining hardware design principles, and the universality advantage of traditional x86 architecture is weakened in AI scenarios.

Key Conclusions and Outlook​

1. The five trends are interconnected and mutually reinforcing​

The inference demand explosion drives edge deployment, edge scenarios spawn dedicated chips, high power consumption forces an energy-efficiency revolution, and all changes ultimately point to the reconstruction of the AI-native hardware ecosystem.

The core driver of this round of change is AI moving from "technology demo" to "commercial deployment"; hardware must satisfy the industry requirements of "scale, low cost, high reliability."

2. Opportunity windows for industry participants​

For industry participants, the opportunities in 2026 lie in:

• ✅ Capture the inference dividend: Deploy inference-specific chips and system optimization
• ✅ Deepen vertical scenarios: Customize hardware solutions for specific industries/applications
• ✅ Break the energy-efficiency bottleneck: Liquid cooling, high-voltage DC, AI thermal management and other technologies
• ✅ Build an open ecosystem: Open-source frameworks, open standards, cross-industry collaboration

Vendors that can provide "end-to-end solutions" rather than "single-point chips" will gain an advantageous position in this reshuffle.

3. Dynamic adjustment and continuous evolution​

The above analysis is based on early-2026 market data and industry forecasts; actual development may adjust dynamically due to factors such as technology breakthroughs, policy adjustments, and market demand changes.

Industry Implications​

2026 is a watershed year for the AI hardware industry:

• From "compute race" to "deployment as king"
• From "single-point breakthroughs" to "system optimization"
• From "general-purpose architecture" to "dedicated customization"
• From "performance first" to "energy efficiency balance"

Vendors that can keenly capture trends, rapidly adjust strategy, and sustain technological innovation will seize the initiative in the AI hardware "era of deployment."

References:

• CSHIA Research, "2026 AI Hardware: Five Transformations and the Rules for Survival"
• "AI Hardware Enters the 'Era of Deployment'," Sohu Tech, February 10, 2026