The Rise of Open-Source AI Evaluations: How Gorgias is Changing B2B Software Procurement

The modern software landscape is witnessing a fundamental shift in how business-to-business (B2B) buyers evaluate artificial intelligence. For decades, vendor procurement relied heavily on static marketing collateral, analyst magic quadrants, idealized feature checklists, and carefully curated product demonstrations. However, the advent of generative AI and autonomous software agents has rendered traditional evaluation methods obsolete. Traditional software behaves deterministically; a legacy customer relationship management tool executes the exact same database query every time a user clicks a button. In contrast, artificial intelligence agents are probabilistic systems. An AI customer support agent may deliver an exemplary, accurate response on a Monday, yet generate a hallucinated or incorrect answer by Thursday. Furthermore, silent background model updates deployed by vendors can drastically alter system performance overnight without traditional software release notes capturing the shift.

Recognizing this critical visibility gap, $100M ARR e-commerce customer experience leader Gorgias has pioneered a transparent paradigm shift by publishing a comprehensive, open-source competitive evaluation of its proprietary AI agent against 17 competing market vendors. Utilizing a rigorous testing harness comprising 8,356 live customer support conversations, the benchmark represents an unprecedented level of radical transparency in the enterprise software sector. The initiative was strongly encouraged by SaaStrFund, which led Gorgias’s original seed financing round and has long advocated for radical honesty in B2B software marketing. The resulting benchmark report, accompanied by a fully open-sourced testing harness on GitHub, provides a definitive blueprint for how modern AI-driven enterprises must validate performance in an era where trust is the ultimate competitive currency.

The Anatomy of the Gorgias Benchmark: Methodology and Scope

To understand the weight of the Gorgias benchmark, one must examine the exhaustive engineering effort required to execute it. Traditional software benchmarking relies on surveys, self-reported vendor specifications, and controlled lab environments. Gorgias took a fundamentally different approach by testing live products against named competitors in real-world scenarios.

The evaluation framework was built upon several core pillars designed to eliminate subjective bias while maintaining absolute accountability:

• Fully Open-Sourced Harness: The entire testing infrastructure, evaluation code, and scoring scripts have been made publicly available via GitHub, allowing any engineer or technical buyer to independently audit, replicate, or modify the test parameters.
• Scale and Scope: The evaluation subjected 18 distinct vendor agents to a standardized battery of 8,356 live customer service interactions specifically tailored to e-commerce workflows.
• Comprehensive Grading Rubric: Every agent response was systematically captured and graded against a versioned, publicly documented rubric that measured metrics such as resolution rate, hallucination frequency, latency (such as Envive’s 7.9-second benchmark), and overall conversational accuracy.
• Inclusion of Deficits: Crucially, the published report does not frame Gorgias as an invincible market monopoly. While Gorgias captured the top overall ranking in general support capabilities, the benchmark transparently highlights categories where competitors—most notably rival e-commerce AI firm Yuma—outperformed Gorgias in specific automation metrics.

By publishing these results warts and all, Gorgias has fundamentally redefined what constitutes a credible competitive analysis in the enterprise software marketplace.

Why Transparent Evals Matter: The Changing Dynamics of B2B Procurement

The decision by Gorgias to publish unvarnished evaluation data addresses several profound transformations currently reshaping software procurement channels. Industry analysts and venture capitalists point to four primary drivers making open-source evals an imperative for modern tech companies.

First, B2B buyers have grown deeply cynical regarding traditional marketing narratives. Modern customer support leaders have been inundated with vendor-generated comparison charts that invariably crown the publisher as the undisputed market leader. When an enterprise vendor demonstrates the intellectual honesty to highlight competitor strengths, psychological validation occurs. Buyers immediately grant credibility to the categories where the publisher claims victory precisely because the company was transparent about the domains where it lost. Gorgias claiming the number one spot in overall support carries exponentially more weight when the same report openly acknowledges Yuma’s lead in pure automation efficiency.

Second, publishing detailed evals essentially performs the buyer’s arduous technical diligence for them. In the past, evaluating 18 different vendors required months of resource-intensive pilot programs, countless sales calls, and expensive consulting engagements. By deploying a standardized, rigorous framework across thousands of real conversations, Gorgias has done the heavy lifting for the entire e-commerce sector. While vendors retain a structural advantage by defining the initial scoring weights and testing rubrics, the radical visibility of the underlying data ensures that buyers can verify every claim made in the report.

Third, the emergence of AI-driven procurement agents is rapidly altering how software shortlists are generated. Increasingly, enterprise software evaluations do not begin with a human browsing a static vendor website or downloading a gated PDF whitepaper. Instead, procurement teams rely on frontier large language models like Claude or ChatGPT to research, analyze, and synthesize software recommendations. These autonomous research agents cannot parse gated marketing PDFs, but they can easily read, verify, and cite versioned GitHub repositories, open scoring rubrics, and structured benchmark data. Companies that embrace open evaluations will find their products naturally favored and cited by autonomous procurement agents, while closed, opaque competitors will be systematically filtered out of consideration.

Finally, internal product development velocity is radically accelerated when competitive telemetry is made public. Within Gorgias, metrics concerning competitor speed, resolution accuracy, and latency are no longer buried as confidential intelligence within internal executive slide decks. They are public-facing data points, updated continuously, and tied directly to public version-control histories. Because the grading rubrics are versioned transparently in public repositories, internal engineering teams cannot artificially manipulate metrics or quietly alter scoring criteria to obfuscate product shortcomings. This creates an unyielding internal feedback loop that forces continuous product improvement.

A New Template for the AI Industry

As artificial intelligence continues its rapid integration into every vertical of enterprise software, the old playbook of opaque product marketing is rapidly losing efficacy. Gorgias has established a new gold standard for transparency, proving that acknowledging competitive weaknesses does not weaken a market leader—it solidifies its authority.

For software vendors navigating the crowded AI landscape, the strategic directive moving forward is clear: publish the evaluations. Write the rubrics, open-source the testing harnesses, test against named competitors, and make the data checkable. In an era where buyers and autonomous agents alike demand verifiable proof over polished marketing promises, transparency is no longer merely a philanthropic gesture of goodwill; it is the most formidable competitive moat a technology company can build.