Companies are becoming increasingly interested in AI and how it can be used to (potentially) boost productivity. They are, however, wary of the risks. According to a recent Workday survey, the top barriers to AI implementation are the timeliness and reliability of the underlying data, potential bias, and security and privacy.
Scott Clark, who previously co-founded the AI training and experimentation platform SigOpt (which was acquired by Intel in 2020), saw a business opportunity and set out to build “software that makes AI safe, reliable, and secure.” Clark founded Distributional to develop the first version of this software, with the goal of scaling and standardizing tests to different AI use cases.
“Distributional is building the modern enterprise platform for AI testing and evaluation,” Clark told TechCrunch in an email interview. “As the power of AI applications grows, so does the risk of harm. Our platform is built for AI product teams to proactively and continuously identify, understand and address AI risk before it harms their customers in production.”
Clark was inspired to start Distributional after experiencing tech-related AI challenges at Intel following the acquisition of SigOpt. As Intel’s VP and GM of AI and high-performance compute, he found it nearly impossible to ensure that high-quality AI testing occurred on a regular basis.
“The lessons I drew from my convergence of experiences pointed to the need for AI testing and evaluation,” Clark continued. “Whether from hallucinations, instability, inaccuracy, integration or dozens of other potential challenges, teams often struggle to identify, understand and address AI risk through testing. Proper AI testing requires depth and distributional understanding, which is a hard problem to solve.”
The core product of Distributional aims to detect and diagnose AI “harm” from large language models (similar to OpenAI’s ChatGPT) and other types of AI models, attempting to semi-automatically determine what, how, and where to test models. Clark claims that the software provides organizations with a “complete” view of AI risk in a sandbox-like pre-production environment.
“Most teams choose to assume model behavior risk, and accept that models will have issues.” Clark said. “Some may try ad-hoc manual testing to find these issues, which is resource-intensive, disorganized, and inherently incomplete. Others may try to passively catch these issues with passive monitoring tools after AI is in production … [That’s why] our platform includes an extensible testing framework to continuously test and analyze stability and robustness, a configurable testing dashboard to visualize and understand test results, and an intelligent test suite to design, prioritize and generate the right combination of tests.”
Clark was vague on the specifics of how this all works — as well as the broad outlines of Distributional’s platform. In his defense, he stated that the company is still in the process of co-designing the product with enterprise partners.
So, given that Distributional is pre-revenue, pre-launch, and has no paying customers, how can it compete with the AI testing and evaluation platforms that are already on the market? After all, there are many, including Kolena, Prolific, Giskard, and Patronus, many of which are well-funded. As if the competition wasn’t fierce enough, tech behemoths such as Google Cloud, AWS, and Azure provide model evaluation tools as well.
Clark believes that Distributional is distinct in its enterprise-oriented software. “From day one, we’re building software capable of meeting the data privacy, scalability and complexity requirements of large enterprises in both unregulated and highly regulated industries,” the CEO said. “The types of enterprises with whom we are designing our product have requirements that extend beyond existing offerings available in the market, which tend to be individual developer focused tools.”
If everything goes as planned, Distributional will begin generating revenue sometime next year when its platform becomes generally available and a few of its design partners convert to paid customers. Meanwhile, the startup is raising capital from venture capitalists; Distributional announced today that it has closed a $11 million seed round led by Andreessen Horowitz’s Martin Casado, with participation from Operator Stack, Point72 Ventures, SV Angel, Two Sigma, and angel investors.
“We hope to usher in a virtuous cycle for our customers,” Clark said. “With better testing, teams will have more confidence deploying AI in their applications. As they deploy more AI, they will see its impact grow exponentially. And as they see this impact scale, they will apply it to more complex and meaningful problems, which in turn will need even more testing to ensure it is safe, reliable, and secure.”
