The recent controversy over Sam Altman’s sudden departure from OpenAI and subsequent hiring by Microsoft reminds us that the LLM space is tumultuous. The field is in its infancy. Such growing pains are to be expected. Paradigm-shifting events will likely continue for several years, as must happen with novel and powerful technologies.
While forward-thinking companies are working hard to integrate LLMs into their offerings, this event has forced us to remember that model providers cannot be simply trusted. A robust LLM stack, necessarily relying on infrastructure and events out of its control, needs to intelligently integrate with multiple public LLM providers as well as support private LLMs if it is to be competitive.
An LLM gateway sits in front of LLMs, public or private, abstracting away the backend inference, tokenization, security measures, and other operations. It allows one to take advantage of powerful LLMs without having to dedicate resources to maintaining and configuring them.
By multi-LLM gateway, we mean an LLM gateway that integrates with several public and private LLMs. But why is it necessary to be a mutli-LLM gateway? The short answer is abstraction: End users should not be affected be the performance or availability of a single model.
Integration with multiple public and private LLMs is now an absolutely necessity. What if tomorrow OpenAI changed their privacy policy in such a way that now conflicts with your internal compliance demands? Enterprises who have built on a foundational multi-LLM gateway would endure such a change without issue.
Above, you see a schematic drawing of how our SecureGPT™ achieves robustness in the face of unreliable model providers. Enterprise apps are built on top of SecureGPT™, which integrates with several model providers, including OpenAI’s ChatGPT, Anthropic’s Claude, and Google’s PaLM. Depending on the use case, the user may be able to choose which model he uses or to use a privately trained, enterprise-specific LLM.
Such integration is not as simple as maintaining keys to multiple models and toggling a switch. Industry leading LLM gateways perform sophisticated backend logic with layered dependence on LLMs. To give a brief example: Say ChatGPT is down. With a multi-LLM gateway, the user may simply select another model to use. However, if that gateway also calls ChatGPT to verify user intent to implement its guardrails, then the gateway’s integration with other LLMs is useless. The gateway may integrate with several public LLMs for inference or chat completion, but if it relies on a single model to perform a crucial intermediate function, then it is not truly multi-LLM—it depends fundamentally on one model’s availability.
Such a gateway is really just a “wrapper” around ChatGPT—providing little if any value beyond ChatGPT itself—despite its claims. And this example is not contrived. LLM applications rely on LLMs for more than just text generation. They may be called on to fact-check (e.g., using Claude to fact check ChatGPT’s output) or to implement security measures (e.g., asking ChatGPT to analyze a user’s request for relevance before responding).
Our SecureGPT™ was designed with such problems in mind. Secure by design, it does not depend on the proper functioning of any one model or any one company. OpenAI could dissolve tomorrow and applications built on our secure LLM gateway would function exactly the same.
Sam Altman’s situation is troubling. OpenAI’s corporate turmoil is troubling. The only thing that forward-thinking enterprises can do in the face of the uncertainty inherent in this space is to build on a secure foundation.
About Quantum Gears
Forum Systems and its subsidiary, Quantum Gears, are leading the Enterprise GenAI revolution. Patent-pending products—like SecureGPT™, ContractsGPT™, BenefitsGPT™, and Forum Sentry—mitigate the unpredictable nature of LLMs through integration with corporate APIs, ensuring LLM output is truthful and accurate. Used by some of the largest global companies for building intelligent business workflows, Forum’s suite of products provides unique, industry-leading solutions that allow enterprises to reinvent themselves with GenAI.