Here is Why the Qualcomm Cloud AI 100 is Not Everywhere
We compared it to common LPDDR options from AMD, NVIDIA, and Apple
Qualcomm has been making waves for its new AI 200 series and its push (again) into the data center CPU business. We wanted to take a quick look at what the current generation of AI accelerators looks like, the Cloud AI 100. To be fair, these have been out for some time.
Here we have two Qualcomm Cloud AI 100 accelerators. Depending on the marketing material, these can be called the AIC100, Cloud AI100, or Cloud AI 100.
These are the smaller accelerators with 32GB of LPDDR4X memory and are connected to each other and to the host system using a Microchip PCIe switch. Also, just to be clear, these are accelerators that are not the larger, higher memory versions. This dual accelerator setup is instead something small enough that it can be packaged and placed in a notebook form factor. There was a thought that this could be an option for those who did not want to use a unified memory solution and wanted more memory than the NVIDIA RTX Pro 5000 Blackwell mobile edition at 24GB. Indeed, you can still pay a premium and order notebooks with these today.
We tried both the NVIDIA and the Qualcomm versions of this notebook, and the NVIDIA RTX Pro 5000 Blackwell Mobile edition version will have a review on the STH main site, and the Qualcomm version you will only find here for reasons that will become apparent.
Before we get too far in this, you are probably wondering whether our setup varies greatly from a reference setup. Did we see the numbers we saw because of some major setup issue? Frankly, when we saw the numbers, we were interested in this as well, so we took our results and wanted to see how close to the reference figures we were:
This type of benchmarking always has a variance, which is just part of the exercise. Still, being within 1.4% of reference numbers is close enough that most in the industry would say this is within a normal benchmark variance. In other words, our setup was performing as expected.
With that, we had a number of systems that we wanted to compare this against:
1. The NVIDIA DGX Spark (or other GB10 instances)
2. The AMD Ryzen AI Max+ 395
3. Apple Mx Max chips (M1 Max, M4 Max, M5 Max)
We decided to use the NVIDIA DGX Spark because it was the unit that is not in our 8x GB10 cluster. The AMD Ryzen AI Max+ 395 we are using the GMKtec EVO-X2 128GB. AMD has other options for that chip, but we had the GMKtec running Ubuntu, so it made it very easy to fold into our suite. For the MacBook Pro, we are using the top GPU core configuration option for each generation. The M1 Max topped out at 64GB, our M4 Max was also a 64GB configuration, and the M5 Max is 128GB. The M1 Max was relatively close in terms of release timing to the Cloud AI 100 which is why we went back to that one, plus we had the machine available.
Also, we are not doing optimization on any of these other platforms. These are just up and running, an easy path. There are so many variables doing this kind of work, so assume that for the AMD, NVIDIA, and Apple numbers, there are ways to increase performance. Also, we limited our set a bit because we needed to use models that run on Qualcomm, or that could compile somewhat easily. We are also using smaller models because we knew on the Qualcomm side that we would be a bit more limited on quantization levels, and the fact that 64GB is the least common denominator for memory capacity.
With that, let us get to the performance.