The Truth About Benchmarks: SPC-1 Matters

Benchmarks: they’re the flag flown by every storage vendor. They’re the numbers behind the big claims, the verification of the promises made about systems.

But not all benchmarks are created equal. A couple of flash storage providers recently announced  benchmark results — from homegrown tests. While that doesn’t mean that the resulting numbers aren’t necessarily valid, it does remove the objectivity that a third party can provide.

Specifically, SPC-1 benchmarks. In my opinion, the SPC-1 benchmark results are the most important, and telling source of storage performance comparisons out there. It’s a top tier independent auditing facility that provides tremendous insights for companies in the pre-qualification phase of analyzing potential storage solutions. Objective, relevant, and verifiable: these are the kinds of promises enterprises should be looking for when considering various published storage performance results. Let’s see…

Objective:  SPC-1 stakes their claim clearly on their homepage:

SPC-1 benchmark results provide a source of comparative storage performance information that is objective, relevant, and verifiable. That information will provide value throughout the storage product lifecycle, which includes development of product requirements, product implementation, performance tuning, capacity planning, market positioning, and purchase evaluations. The SPC-1 Benchmark is designed to be vendor/platform independent and are applicable across a broad range of storage configuration and topologies.

Relevant:  

SPC-1, specifically, runs a single workload that demonstrates storage system performance under typical critical applications such as OLTP and database operations.

Verifiable:  

The submission process guarantees results that are valid and, furthermore, match existing products that customer can actually buy. Vendors cannot run a special configuration that is specifically constructed to test better. In addition to properly submitting a product, and completing a SPC audit, results are reviewed for at least 60 days by an SPC-1 approved peer group who are given the opportunity to raise any possible compliance issues.

SPC-1 is a multistep process, meaning any published results have been verified by multiple parties not associated with a particular vendor. For enterprise companies looking to make a major storage investment, this kind of validation, and information, is key.

Learn more about SPC-1 and their benchmark results here.

New Call-to-action