- I don’t think picking a handful of SPEC benchmarks that approximate today’s most common agentic workloads (compiling code, interpreting Python) and then calling them “agentic benchmarks” is misleading at all.
That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
- The four benchmarks selected are cppcheck, llvm, cpython, and gcc [1]. These are all essentially compiler benchmarks... and all of the compiler benchmarks in SPEC cpu2026! This makes the benchmark selection somewhat suspicious to me, since it's not particularly representative of a diverse set of workloads.
I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows.
[1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
- Let’s not forget the fact that despite their new market cap and being the beneficiary of having a near monopoly on making incredibly complex pickaxes during a gold rush… Nvidia is still the company with a long and consistent history of misleading their customers via marketing. Their greatest hits include:
- The digital equivalent of the VW emissions scandal where drivers detected when they were being benchmarked and altering rendering for better results
- Selling GPUs as having 4gb vram when it was only 3.5gb usable, the remaining 0.5 being absurdly slower and causing performance loss when used
- Using intentionally misleading naming schemes to obfuscate things like memory bus width being drastically different between what superficially appeared to be similarly spec’d cards
A dozen other less egregious but similarly disingenuous decisions
But to be clear, I’m a huge fan and continue to run Nvidia because their products are generally incredible regardless
- I'm glad to see the competition; nothing provokes meaningful change without it. Not surprised at Nvidia's blatant fabrications though; more of the same we've seen time and again (Superchip anyone, with 2+ year old designs).
- This is neat. I'm only halfway through but some really cool discussion about the internals of the chips and what they offer.
