Ordinary Wisdom Documentation

This documentation covers the main functions of the Ordinary Wisdom Limited (OWL) application for controlling and evaluating the output of large language models (LLMs).

OWL’s platform allows for:

1. Granular control of LLM output through:
   1. Portfolios of documents (e.g. policies, regulatory) that inform AI output
   2. Constitutions of rulesets and values that control AI output
   3. Advanced ML and post-generation control methods
2. Evaluation of AI outputs against specified values, including comparison and scoring methods, and LLM-as-a-judge
3. API allowing aligned and controlled LLMs to be used in any application

We are also working on:

1. Semantic harness (output control) – which (a) measures and avoids "strain" against specified human values, (b) measures factual strain (i.e. avoiding hallucinations), (c) controls/restricts domain/topic, and other methods.

Sections

• Workbench – Control and Evaluation of Model Behavior
  • Score
  • Compare
  • Panel Evaluation
• Portfolios – Sets of documents or books that inform the LLM.
• Constitutions – Rulesets for your LLM.
• Tests – Tests for evaluating model output and behavior.
• API – Integrate OWL controls into your technology stack, or application.

Next: Workbench