Evaluating Model Behavior

Evaluating LLM behavior is a core OWL function, and one that requires more technical set-up than other functions. Each evaluation runs tests against specified values for (1) a selected books portfolio, using (2) selected LLM(s) and (3) a selected constitution. The key steps of an evaluation are:

• (optional) Setting the evaluation parameters
• Choosing book portfolio, constitution, LLM(s) and tests
• Running the evaluation.

Setting the Evaluation Parameters

You can optionally control two sets of parameters for evaluations via your Settings, which are available via the drop-down menu from your account name at the right side of the top menu.

• LLM Evaluation Instructions
  • These explain to LLM(s) acting as a "judge" how to evaluate the LLM you are evaluating. The default instructions are intended to be robust under a variety of circumstances.
  • We recommend care in changing these instructions (including a basic understanding of JSON), as misconfigured instructions will affect the reliability of evaluation.
• LLM Evaluation Principles
  • These are the principles that the "judge" LLM(s) use for evaluation. The default principles are based on the core values of Principlism and Stoicism.
  • You may change the principles, delete principles, or add new principles, using the same JSON format as the default.

Choosing Book Portfolio, Constitution, LLM(s) and Tests

Evaluations are tied to book portfolios, so the first step in setting up an evaluation is to choose a portfolio from Public Portfolios or My Portfolios. After opening the portfolio by clicking View, choose Start Evaluation.

To set up the evaluation details, choose the relevant values in the fields for:

• Constitution – any constitution available to you
• Prompt LLM Provider – the LLM to be evaluated, as modified by OWL
• Evaluator LLM Providers – one or more LLMs used to judge responses by the Prompt LLM Provider
• Tests – up to 20 tests used for evaluation (leaving blank causes OWL to choose 10 tests randomly).

Running the Evaluation

To run an evaluation, click Start Evaluation at the bottom of the Start Portfolio Evaluation screen.

The evaluation results appear on screen, and can be accessed later from the portfolio screen by clicking Evaluations. The evaluation results include a substantial amount of detail about the evaluation process, including explanations of scores.