Persona Development:
LLM Guardrails
LLMs promise to augment the productivity of many personas within an enterprise. But unfettered use of LLMs poses a two-fold problem: employees may not use the LLMs for work-related tasks and LLMs output may not align with your corporate values. For this reason, QS SecureGPT ships with guardrails to align conversations and moderate content.
But once guardrails are in place, the question arises: How to define the personas and use cases that users are allowed to talk about? In practice, how are specific roles and tasks aligned with the LLM guardrails that restrict conversations?
QS SecureGPT catalyzes this process by defining a route to move from initial GenAI pilots to full-fledged deployment at scale.
The first step in persona development is creating a list of potential use cases. While LLMs are extremely powerful, exactly how they can assist personas within your organization needs to be carefully considered.
QS SecureGPT ships with a cookbook of pre-built use cases to help in defining the most impactful use cases for your organization, like writing and reviewing code, analyzing data, developing contracts, parsing legal regulations and so on.
Once the initial use cases are defined and the first models are deployed, we recommend allowing a small set of users access to the tool without any guardrails or topic alignment in place. These first users are instructed to use the tool as they normally would, but they will be allowed to discuss any topics. This gives QS SecureGPT the opportunity to collect actual user data.
When learning mode is enabled, the platform learns from the intent of each prompt. This data is then used to refine the guardrails. The LLM learns from the real-life usage and can therefore better perform its topic alignment function. For example, if there is a guardrail that allows discussion of data & analytics, and your organization primarily uses Microsoft, the guardrail can be refined to ensure questions about PowerBI, DAX and the like always are allowed.
Once sufficient data is collected (usually about two weeks of usage) learning mode can be turned off the the guardrails can be activated. At this stage, the deployment can be opened up to more users since there are guardrails in place.
At this stage, users should be encouraged to send feedback if they feel the guardrails are too restrictive. If their questions are being blocked incorrectly, QS SecureGPT allows users to downvote the blocked response and send feedback.
This feedback is then reviewed by admins and staff and used to further refine the guardrail behavior.
Once the personas have been defined and the system has learned from the behavior of real end users, the deployment can be opened up to all users within an organization.
At this point, the guardrails should have become intelligent enough to block more than 99% of legitimately off-topic conversations and to allow the rest.
QS SecureGPT’s observability capabilities allow admins and staff to continue monitor usage and collect feedback on the guardrail behavior. They can, for example, set alerts if a particular user is often having off-topic conversations.