Ordinary Wisdom Beta Documentation
This documentation is for the beta version of the Ordinary Wisdom Limited (OWL) application for controlling and evaluating the behavior of large language models (LLMs).
OWL Beta uses three main control techniques:
- Book portfolios – You can specify a portfolio of "books" (a book can be any document), requiring the LLM to answer based on the content of the books, with control of how the books are used via a "meta-prompt".
- Constitutions – You can specify how an LLM should respond in terms of (a) an identity statement, (b) core values, (c) prohibited behaviors and (d) output style.
- Evaluation – You can evaluate the performance of an LLM (either the native version or as modified by Athena using a book portfolio and/or constitution) against selected values, and use the evaluation to adjust the LLM's behavior.
We are also working actively on a fourth control technique, which will be released soon:
- Semantic harness (output control) – We will use various methods to modify LLM output, including (a) measuring and avoiding "strain" against specified human values, (b) measuring factual strain (i.e. avoiding hallucinations), (c) controlling / restricting domain / topic (e.g. finance or healthcare), and other methods.
Sections
- Controlling Model Behavior (Workbench) – The OWL Workbench is the basic interface for controlling LLM behavior, using book portfolios, constitutions and other settings.
- Managing Books and Portfolios – You can select books available from Project Gutenberg (~76,000 books), access other books made available by OWL or other users, or upload your own books.
- Managing Constitutions – OWL Beta allows you to build your own constitutions. We will soon allow you to choose from shared constitutions.
- Managing Tests – We allow you to define "tests" for evaluating model behavior, and share them publicly.
- Evaluating Model Behavior – OWL Beta allows you to evaluate model behavior against values that you select, using tests that you select.
- Application Programming Interface (API) – If you subscribe to our Enterprise plan, you can integrate OWL output into your own website or application.