MangoBoost Delivers Impressive MLPerf Inference v5.0 Results Using AMD Instinct MI300X

MangoBoost Sets a New Standard in AI Efficiency

MangoBoost is stirring up the tech world with its latest achievement in AI data center efficiency. The company has just raised the bar with its submission for MLPerf Inference v5.0, showcasing its Mango LLMBoost AI Enterprise MLOps software. This powerful software has delivered jaw-dropping results on AMD Instinct MI300X GPUs, hitting the highest recorded performance for Llama2-70B in the offline inference category. What’s even more impressive? This marks the first-ever multi-node MLPerf inference result on the MI300X GPUs. By harnessing the power of 32 MI300X GPUs spread across four server nodes, Mango LLMBoost has outperformed all previous benchmarks, even those set by competitors using nVidia H100 GPUs.

Unmatched Performance and Cost Efficiency

When it comes to performance, MangoBoost is leaving the competition in the dust. Their MLPerf submission boasts a stunning 24% performance advantage over the best published result from Juniper Networks, which utilized 32 NVIDIA H100 GPUs. With Mango LLMBoost, users can expect to achieve 103,182 tokens per second (TPS) in offline scenarios and 93,039 TPS in server scenarios on AMD MI300X GPUs. This is a significant leap from the previous best of 82,749 TPS on NVIDIA’s H100.

But it’s not just about performance; it’s also about cost. The AMD MI300X GPUs are priced between $15,000 and $17,000, a stark contrast to the hefty $32,000-$40,000 price tag of NVIDIA H100 GPUs. This means that Mango LLMBoost can deliver up to 62% cost savings while still offering top-notch inference throughput. In fact, when you break it down, the Mango LLMBoost + MI300X system delivers approximately 2.8 times more inference throughput per $1,000 spent compared to the H100-based system. For those keeping an eye on the budget, this is a game-changer.

Mango LLMBoost: A Scalable and Flexible MLOps Solution

What sets Mango LLMBoost apart is its enterprise-grade AI inference capabilities, which offer seamless scalability and cross-platform compatibility. The software supports over 50 open models, including Llama, Qwen, and DeepSeek, making deployment a breeze with just one line of code via Docker. Plus, it’s cloud-ready, available on platforms like AWS Marketplace, Microsoft Azure Marketplace, and Google Cloud Platform, while also offering on-premises deployment for enterprises that prioritize control and security.

Some standout features of Mango LLMBoost include:

–

Auto Parallelization

: Efficiently distributes large models across GPUs and nodes.
–

Auto Config Tuning

: Optimizes runtime parameters based on workload characteristics.
–

Auto Context Scaling

: Dynamically adapts memory usage to maximize GPU utilization.
–

Auto Disaggregated Deployment

: Ensures flexible deployment across multiple inference stages.

Collaboration with AMD: Unlocking the Full Potential of MI300X GPUs

MangoBoost’s remarkable results are a product of a close partnership with AMD. By leveraging the ROCm software stack, they’ve been able to maximize the performance of MI300X GPUs, resulting in a scalable and efficient AI inference solution that can be deployed effortlessly across single-node or multi-node clusters.

Extending Performance Leadership to AWS and Beyond

But the accolades don’t stop there. Beyond the MLPerf results, Mango LLMBoost has been rigorously tested across various cloud and on-premises configurations. For instance, on an 8×NVIDIA A100 GPU setup from AWS, Mango LLMBoost achieved an astonishing 138 times faster inference than Ollama. It also significantly outperformed HuggingFace TGI and vLLM across multiple model sizes, including LLaMA3.1-70B, DeepSeek-R1-Distill-Qwen-32B, and LLaMA3.1-8B. Cost-wise, Mango LLMBoost stands out with the lowest GPU cost per million tokens, slashing inference costs by over 99% compared to Ollama and more than 30% compared to vLLM on high-throughput workloads.

Expanding AI Infrastructure Solutions

In addition to the Mango LLMBoost software, MangoBoost is not stopping there. They’re also offering hardware acceleration solutions based on Data Processing Units (DPUs) to enhance AI and cloud infrastructure. Their products include:

–

Mango GPUBoost

: RDMA acceleration for multi-node inference and training via RoCEv2.
–

Mango NetworkBoost

: TCP/IP stack offloading for improved CPU efficiency.
–

Mango StorageBoost

: High-performance NVMe/TCP initiator and target solutions for scalable AI storage.

With these innovative solutions, MangoBoost is not just setting the standard; they’re redefining what’s possible in AI infrastructure. So, whether you’re a tech enthusiast or a business leader, keep an eye on MangoBoost—it’s clear they’re just getting started!

Background Information

---

About AMD:

AMD, a large player in the semiconductor industry is known for its powerful processors and graphic solutions, AMD has consistently pushed the boundaries of performance, efficiency, and user experience. With a customer-centric approach, the company has cultivated a reputation for delivering high-performance solutions that cater to the needs of gamers, professionals, and general users. AMD's Ryzen series of processors have redefined the landscape of desktop and laptop computing, offering impressive multi-core performance and competitive pricing that has challenged the dominance of its competitors. Complementing its processor expertise, AMD's Radeon graphics cards have also earned accolades for their efficiency and exceptional graphical capabilities, making them a favored choice among gamers and content creators. The company's commitment to innovation and technology continues to shape the client computing landscape, providing users with powerful tools to fuel their digital endeavors.

  

Latest Articles about AMD

---

About Google:

Google, founded by Larry Page and Sergey Brin in 1998, is a multinational technology company known for its internet-related services and products. Initially for its search engine, Google has since expanded into various domains including online advertising, cloud computing, software development, and hardware devices. With its innovative approach, Google has introduced influential products such as Google Search, Android OS, Google Maps, and Google Drive. The company's commitment to research and development has led to advancements in artificial intelligence and machine learning.

  

Latest Articles about Google

---

About Microsoft:

Microsoft, founded by Bill Gates and Paul Allen in 1975 in Redmond, Washington, USA, is a technology giant known for its wide range of software products, including the Windows operating system, Office productivity suite, and cloud services like Azure. Microsoft also manufactures hardware, such as the Surface line of laptops and tablets, Xbox gaming consoles, and accessories.

  

Latest Articles about Microsoft

---

About nVidia:

NVIDIA has firmly established itself as a leader in the realm of client computing, continuously pushing the boundaries of innovation in graphics and AI technologies. With a deep commitment to enhancing user experiences, NVIDIA's client computing business focuses on delivering solutions that power everything from gaming and creative workloads to enterprise applications. for its GeForce graphics cards, the company has redefined high-performance gaming, setting industry standards for realistic visuals, fluid frame rates, and immersive experiences. Complementing its gaming expertise, NVIDIA's Quadro and NVIDIA RTX graphics cards cater to professionals in design, content creation, and scientific fields, enabling real-time ray tracing and AI-driven workflows that elevate productivity and creativity to unprecedented heights. By seamlessly integrating graphics, AI, and software, NVIDIA continues to shape the landscape of client computing, fostering innovation and immersive interactions in a rapidly evolving digital world.

  

Latest Articles about nVidia

Technology Explained

---

AWS: Amazon Web Services (AWS) is a cloud platform powered by Amazon that enables users to access cloud computing services, such as storage, data analytics, and distributed computing. It offers users the ability to utilize both on-demand and pay-as-you-go computing services, making it a great option for the computer industry. It offers a wide range of services with great flexibility for a variety of uses. It can help companies build powerful web and mobile applications, run large-scale analytics, quickly provision servers and other services, design sophisticated architectures for data storage, and more. AWS provides access to a wide range of services such as virtualization, storage, database, monitoring, analytics, and other services that can help organizations increase agility, manage complexity, and remain on the cutting edge of technology. Many big and famous organizations use AWS services to give them a competitive edge, and more and more companies are turning to this service for their computer needs.

Latest Articles about AWS

---

CPU: The Central Processing Unit (CPU) is the brain of a computer, responsible for executing instructions and performing calculations. It is the most important component of a computer system, as it is responsible for controlling all other components. CPUs are used in a wide range of applications, from desktop computers to mobile devices, gaming consoles, and even supercomputers. CPUs are used to process data, execute instructions, and control the flow of information within a computer system. They are also used to control the input and output of data, as well as to store and retrieve data from memory. CPUs are essential for the functioning of any computer system, and their applications in the computer industry are vast.

Latest Articles about CPU

---

GPU: GPU stands for Graphics Processing Unit and is a specialized type of processor designed to handle graphics-intensive tasks. It is used in the computer industry to render images, videos, and 3D graphics. GPUs are used in gaming consoles, PCs, and mobile devices to provide a smooth and immersive gaming experience. They are also used in the medical field to create 3D models of organs and tissues, and in the automotive industry to create virtual prototypes of cars. GPUs are also used in the field of artificial intelligence to process large amounts of data and create complex models. GPUs are becoming increasingly important in the computer industry as they are able to process large amounts of data quickly and efficiently.

Latest Articles about GPU

---

NVMe: Non-Volatile Memory Express (NVMe) is a newly developed technology that has been gaining traction in the computer industry. This technology is a standard interface which allows for high-speed storage and retrieval of data from solid state drives (SSDs). NVMe is designed to increase the speed of data transfers in storage systems by enabling a direct connection to PCI Express (PCIe) bus, resulting in significantly faster access times compared to traditional interface protocols such SSDs. NVMe is particularly useful for applications that require lightning-fast access to large amounts of high-value data. NVMe-based SSDs are being widely adopted in the computer industry and are being employed to power data centers, high-end workstations, and gaming machines to support lightning-fast data processing and retrieval, which unlocks possibilities for machine learning, real-time analytics, edge computing, and other cutting-edge applications. NVMe is proving to be an invaluable tool in the field of computing, offering immense

Latest Articles about NVMe