Zilliz Adds Cost-Aware Benchmarking to VDBBench, the Open-Source Vector Database Benchmark

Skip to Content News Archives Economy Energy Oil & Gas Renewables Electric Vehicles Mining Commodities Agriculture Real Estate Mortgages Mortgage Rates Finance Banking Insurance Fintech Cryptocurrency Work Wealth Smart Money Wealth Management Investor Personal Finance Family Finance Retirement Taxes High Net Worth FP Comment Executive Women Puzzmo Newsletters Financial Times Business Essentials More Innovation Information Technology FP500 Podcasts Small Business Lives Told Tails Told Shopping Financial Post Store Obituaries Place a Notice Advertising Advertising With Us Advertising Solutions Postmedia Ad Manager Sponsorship Requests Classifieds Place a Classifieds ad Working Profile Settings My Subscriptions Saved Articles My Offers Newsletters Customer Service FAQ News Economy Energy Mining Real Estate Finance Work Wealth Investor FP Comment Executive Women Puzzmo Newsletters Financial Times Business Essentials This advertisement has not loaded yet, but your article continues below.HomeBusiness Wire News ReleasesPMN Press ReleasesThis section is The content in this section is supplied by Business Wire for the purposes of distributing press releases on behalf of its clients. Postmedia has not reviewed the content. by Business Wire Zilliz Adds Cost-Aware Benchmarking to VDBBench, the Open-Source Vector Database BenchmarkAuthor of the article: You can save this article by registering for free here. Or sign-in if you have an account.THIS CONTENT IS RESERVED FOR SUBSCRIBERS ONLYSubscribe now to read the latest news in your city and across Canada.Exclusive articles from Barbara Shecter, Joe O'Connor, Gabriel Friedman, and others.Daily content from Financial Times, the world's leading global business publication.Unlimited online access to read articles from Financial Post, National Post and 15 news sites across Canada with one account.National Post ePaper, an electronic replica of the print edition to view on any device, share and comment on.Daily puzzles, including the New York Times Crossword.SUBSCRIBE TO UNLOCK MORE ARTICLESSubscribe now to read the latest news in your city and across Canada.Exclusive articles from Barbara Shecter, Joe O'Connor, Gabriel Friedman and others.Daily content from Financial Times, the world's leading global business publication.Unlimited online access to read articles from Financial Post, National Post and 15 news sites across Canada with one account.National Post ePaper, an electronic replica of the print edition to view on any device, share and comment on.Daily puzzles, including the New York Times Crossword.REGISTER / SIGN IN TO UNLOCK MORE ARTICLESCreate an account or sign in to continue with your reading experience.Access articles from across Canada with one account.Share your thoughts and join the conversation in the comments.Enjoy additional articles per month.Get email updates from your favourite authors.THIS ARTICLE IS FREE TO READ REGISTER TO UNLOCK.Create an account or sign in to continue with your reading experience.Access articles from across Canada with one accountShare your thoughts and join the conversation in the commentsEnjoy additional articles per monthGet email updates from your favourite authorsSign In or Create an AccountVDBBench already tests vector databases under real production workloads; The new release lets any team measure what a vector database actually costs to run in production — across data freshness, filtering, multitenancy, and cold starts — not just peak queries per second.REDWOOD SHORES, Calif. — Zilliz, a leading AI data infrastructure company and the creator of Milvus, recently announced a major update to VectorDBBench (VDBBench), an open-source, vendor-neutral benchmark for vector databases, adding cost as a first-class dimension alongside performance under production conditions — not just how fast a system performs in an idealized test.Most benchmarks optimize for a single headline number: peak queries per second (QPS) on static, fully indexed data. But teams rarely choose a vector database on speed alone. They need to know what it costs to hit a target QPS, when newly written data actually becomes searchable, how filters and payload size change the query surface, how a system holds up across thousands of tenants, and how it responds on the first query after sitting idle. VDBBench now measures those dimensions directly.Get the latest headlines, breaking news and columns.By signing up you consent to receive the above newsletter from Postmedia Network Inc.A welcome email is on its way. If you don't see it, please check your junk folder.The next issue of Top Stories will soon be in your inbox.We encountered an issue signing you up. Please try again“Most benchmarks answer one question — how fast can it go on ideal data,” said James Luan, VP of Engineering at Zilliz. “But teams buy on total cost and real production behavior. We built VDBBench as an open, reproducible benchmark so any team can see what a vector database will actually cost and how it will behave on their own workload — and decide for themselves.”Building on VDBBench 1.0, which moved benchmarking closer to production by testing under streaming ingestion, filtering, recall, and concurrency, the latest release adds cost as a first-class dimension through four new cloud-oriented test cases:Insert readiness and write cost — separating when a write is accepted, when it becomes searchable, and when it is fully indexed, alongside the cost of loading data.Payload-aware search — how response shape (IDs, metadata, or full vectors) and filters change QPS, latency, and recall.Multitenant search — sustained throughput across many tenants and namespaces, as in SaaS workloads.Cold-start latency — the first query against a collection that has gone idle, versus the warmed path.The results feed a new Cost Leaderboard with a cost-performance (Pareto) view that models operating cost at target QPS levels, showing where usage-metered serverless pricing is most efficient at low, spiky volume and where provisioned, flat-rate capacity becomes more cost-effective for the sustained traffic typical of production workloads.Built to Benchmark Any Vector Databaseis open-source and supports more than 30 vector databases and search systems. To show the new cases in action, the Cost Leaderboard launches with a sample evaluation of several widely used managed vector databases — Pinecone, Turbopuffer, and Zilliz Cloud — chosen to illustrate how differently products can behave on freshness, filtering, multitenancy, and cold starts. The comparison is a demonstration, not a ranking: because the benchmark is open, teams can reproduce every case, swap in any other candidate, and run it against their own production-like data.In the sample run, for instance, Zilliz Cloud returned newly inserted data as searchable immediately at a single-digit-dollar bulk-load cost, held a near-flat cold-start latency profile, and grew more cost-effective as sustained query volume rose — an illustration of the performance-and-cost balance the new cases are built to reveal. A product tuned for a different workload could show different strengths, which is exactly why every case is open to reproduce.“This isn’t about naming a winner,” Luan added. “Different databases are built for different workloads, and the right choice depends on freshness, filtering, tenancy, budget, and many other factors. The results aren’t ours to declare — they’re yours to reproduce.”The VDBBench Cost Leaderboard is available now at zilliz.com/vdbbench-leaderboard-v2. VectorDBBench is open source on GitHub, where teams can reproduce the cases or benchmark their own candidates, and share results via GitHub or the Milvus Discord community.For more details, read the VDBBench blog.Zilliz is a leading AI data infrastructure company and the creator of Milvus, the world’s most widely adopted open-source vector database, with 45,000+ GitHub stars and over 100 million Docker pulls. Zilliz helps enterprises and AI startups make their unstructured data searchable, analyzable, and governable — turning text, images, audio, video, and more into a strategic asset for production AI.Zilliz’s technology centers on Milvus and Zilliz Cloud. Milvus is an open-source, lake-native vector database purpose-built for 100-billion-scale vector search. Zilliz Cloud extends that foundation into a fully managed Vector Lakebase platform, combining the high-throughput, low-latency serving capabilities of vector databases with the openness, scalability, and economics of multimodal data lakes. Zilliz powers more than 10,000 enterprises and AI-native startups worldwide, including MiniMax, OpenEvidence, Filevine, Exa, Salesforce, and Read AI.Headquartered in Redwood Shores, California, Zilliz is backed by leading investors, including Aramco’s Prosperity 7 Ventures, Temasek’s Pavilion Capital, Hillhouse Capital, 5Y Capital, Yunqi Partners, and Trustbridge Partners. Learn more at Zilliz.com.View source version on businesswire.com: This advertisement has not loaded yet.Notice for the Postmedia NetworkThis website uses cookies to personalize your content (including ads), and allows us to analyze our traffic. Read more about cookies here. By continuing to use our site, you agree to our Terms of Use and Privacy Policy.