MLCommons has published results of the industry-standard MLPerf v4.0 benchmark for inference.
The Intel Gaudi 2 AI accelerator remains the only benchmarked alternative to Nvidia H100 for generative AI (GenAI) performance and provides strong performance-per-dollar.
Intel is also the only server CPU vendor to submit MLPerf results. Intel’s 5th Gen Xeon results improved by an average of 1.42x compared with 4th Gen Intel Xeon processors’ results in MLPerf Inference v3.1.
“We continue to improve AI performance on industry-standard benchmarks across our portfolio of accelerators and CPUs. Today’s results demonstrate that we are delivering AI solutions that deliver to our customers’ dynamic and wide-ranging AI requirements,” says Zane Ball, Intel corporate vice-president and GM: DCAI Product Management.
“Both Intel Gaudi and Xeon products provide our customers with options that are ready to deploy and offer strong price-to-performance advantages.”
The MLPerf results provide customers with industry-standard benchmarks to evaluate AI performance.
Intel Gaudi 2 results
The Intel Gaudi software suite continues to increase model coverage of popular large language models (LLMs) and multimodal models. For MLPerf Inference v4.0, Intel submitted Gaudi 2 accelerator results for state-of-the-art models Stable Diffusion XL and Llama v2-70B.
Due to strong customer demand for Hugging Face Text Generation Inference (TGI), Gaudi’s Llama results used the TGI toolkit, which supports continuous batching and tensor parallelism, enhancing the efficiency of real-world LLM scaling.
For Llama v2-70B, Gaudi 2 delivered 8035.0 and 6287.5 for offline and server tokens-per-second, respectively.
On Stable Diffusion XL, Gaudi 2 delivered 6.26 and 6.25 for offline samples-per-second and server queries-per-second, respectively.
With these results, Intel Gaudi 2 continues to offer compelling price/performance, an important consideration when looking at the total cost of ownership (TCO).
Intel 5th Gen Xeon results
Following hardware and software improvements, Intel’s 5th Gen Xeon results improved by a geomean of 1.42x compared with 4th Gen Intel Xeon processors’ results in MLPerf Inference v3.1.
As an example, for GPT-J with software optimizations including continuous batching, the 5th Gen Xeon submission showed about 1.8x performance gains compared with the v3.1 submission. Similarly, DLRMv2 showed about 1.8x performance gains and 99.9 accuracy due to MergedEmbeddingBag and other optimizations utilizing Intel AMX.
In collaboration with Intel OEM partners Cisco, Dell, Quanta, Supermicro and WiWynn have delivered their own MLPerf submissions. Additionally, Intel has submitted MLPerf results for four generations of Xeon products, starting in 2020, and Xeon is the host CPU for many accelerator submissions.
AI Solutions on Intel Developer Cloud
5th Gen Xeon processors and Intel Gaudi 2 accelerators are available for evaluation in the Intel Developer Cloud, where users can run both small- and large-scale training (LLM or GenAI) and inference production workloads at scale, manage AI compute resources and more.