Controlling Model Behavior (Workbench)

The Workbench is OWL's main interface for LLM control. It's where you land when you enter the OWL app, and is always accessible via the WORKBENCH link in the top menu.

The Workbench has two sides:

• the left side allows you to specify controls / modifications for an LLM
• the right side allows you to prompt the LLM.

Workbench Left Side – Controls

You can specify the following on the left side of the Workbench:

• Book Portfolio – A book portfolio includes content that the LLM accesses to inform its answers. See Managing Books and Portfolios for details on building and managing book portfolios. You can choose a book portfolio either by:
  • using the drop-down menu, and then choosing a portfolio from the list (typing characters into the box searches the list of portfolios; "No Portfolio" at the bottom of the list avoids any control by a book portfolio); or
  • clicking Explore Books
    • this takes you to Public Portfolios, where you can view all public portfolios, or click My Portfolios to view your private portfolios and public portfolios that you have shared
    • from either Public Portfolios or My Portfolios, choose + Workbench on the portfolio that you want to use.
• Constitution – A constitution specifies how an LLM should respond in terms of (a) an identity statement, (b) core values, (c) prohibited behaviors and (d) output style. See Managing Constitutions for details on building and managing constitutions. You can choose a constitution either by:
  • using the drop-down menu, and then choosing a constitution from the list (click the "eye" icon to view the current constitution; choosing "No Constitution" avoids any control by constitution); or
  • clicking Manage Constitutions, and choosing Set Active on the "three dots" menu on the tile for the constitution that you want to use.
• Model Configuration
  • AI Model allows you to choose an LLM – For the OWL MVP, this is either ChatGPT, Gemini or Claude (versions are specified). We will soon add other LLMs.
  • The other Model Configuration settings relate to how OWL processes books in your chosen portfolio using retrieval-augmented generation (RAG).
    • Knowledge Base allows you to choose the vector embedding model used to process snippets of books – this is currently either Google Vertex AI or Pinecone.
    • Max Sources – This specifies the maximum number of RAG snippets that OWL will retrieve when responding to a prompt. Choose a low number to access only the most relevant content from the portfolio, and a higher number to access a broader range of content. You can set a default for this value under "Maximum RAG Snippets" in Settings.
    • Edit Meta-Prompt allows you to change the instructions for how the LLM responds using RAG snippets (click the "eye" icon to view the current meta-prompt).

Workbench Right Side – Prompting

This is where you enter prompts as you would for any LLM, with the added dimension of access to a wide variety of tests for evaluating LLM performance.

[write once multi-turn interface is available]

See Managing Tests for an explanation of how to create and manage your own tests.