Controlling Model Behavior (Workbench)

The Workbench is OWL's main interface for LLM control. It's where you land when you enter the OWL app, and is always accessible via the WORKBENCH link in the top menu.

The Workbench has two sides:

• the left side allows you to specify controls / modifications for an LLM
• the right side allows you to prompt the LLM.

Workbench Left Side – Controls

You can specify the following on the left side of the Workbench:

• Book Portfolio – A book portfolio includes content that the LLM accesses to inform its answers. See Managing Books and Portfolios for details on building and managing book portfolios. You can choose a book portfolio either by:
  • using the drop-down menu, and then choosing a portfolio from the list (typing characters into the box searches the list of portfolios; "No Portfolio" at the bottom of the list avoids any control by a book portfolio); or
  • clicking Explore Books
    • this takes you to Public Portfolios, where you can view all public portfolios, or click My Portfolios to view your private portfolios and public portfolios that you have shared
    • from either Public Portfolios or My Portfolios, choose + Workbench on the portfolio that you want to use.
• Constitution – A constitution specifies how an LLM should respond in terms of (a) an identity statement, (b) core values, (c) prohibited behaviors and (d) output style. You can choose a constitution using the drop-down menu (choosing "No Constitution" avoids any control by constitution). Click the "eye" icon to view the current constitution. See Managing Constitutions for details on building and managing constitutions.
• Model Configuration
  • AI Model allows you to choose an LLM – For the OWL MVP, this is either ChatGPT, Gemini or Claude (versions are specified). We will soon add other LLMs.
  • The other Model Configuration settings relate to how OWL processes books in your chosen portfolio using retrieval-augmented generation (RAG).
    • Knowledge Base allows you to choose the vector embedding model used to process snippets of books – this is currently either Google Vertex AI or Pinecone.
    • Max Sources – This specifies the maximum number of RAG snippets (up to 20) that OWL will retrieve when responding to a prompt. Choose a low number to access only the most relevant content from the portfolio, and a higher number to access a broader range of content. You can set a default for this value under "Maximum RAG Snippets" in Settings.
    • Meta-Prompt options allow you to Edit or View the instructions for how the LLM responds using RAG snippets.

Workbench Right Side – Prompting

This is where you enter prompts as you would for any LLM, with the added dimension of access to a wide variety of tests for evaluating LLM performance.

A prompt can be any text that you choose to enter in place of "Enter prompt here ..." in the window below Test / Prompt. If a test is selected (see below), you must choose "Freeform Input" in the drop-down menu to be able to enter text of your choice.

To use a pre-defined test, just select it on the drop-down menu. You can type text in the search box of the drop-down to search for tests before selecting (the search is of both test name and test content). See Managing Tests for an explanation of how to create and manage your own tests.

Once you have chosen a prompt or test (and selected), click Seek Wisdom at the bottom to execute the prompt. OWL Beta then does so, using the controls set on the left side of the screen.

After receiving an initial response from OWL, you can continue the conversation as with any LLM.