Scorers
A scorer defines what a result means: its name, value type, and passing rule. A reported scorer records scores that your evaluation code calculates. A composite scorer calculates a score in PostHog from other scores on the same item. For the basics, see how offline evals work.
Add more scores
Use a separate scorer for each thing you want to measure, such as correctness, relevance, or whether the agent selected the right tool.
| Scorer type | Result value | Example |
|---|---|---|
| Boolean | true or false | Did the agent complete the task correctly? |
| Numeric | A number | How relevant is the answer, from zero to five? |
| Categorical | A list of category keys | Which failure types occurred, such as ["wrong_tool"]? |
Create scorers with the SDK or in Scorers. Each result references a fixed scorer version, so configuration changes don't reinterpret earlier results.
For each additional reported score, call upload_result() / uploadResult() with the same EvaluationItem and the new scorer's version ID. Include all reported and composite results in your expected result count.
Use reasoning to explain a score. Set application_trace_id / applicationTraceId on the item to link it to an AI Observability trace.
Failed checks and evaluator errors
Use value=False / value: false for a failed check. The default status, ok, means the evaluator produced a score, not that the item passed.
If the evaluator can't produce a score, upload status="error" / status: "error", omit value, and include an error_message / errorMessage. Use skipped for a check you didn't run, or not_applicable when a scorer doesn't apply to an item.
Combine scores with a composite scorer
A composite scorer combines reported scores for one item into a boolean, numeric, or categorical result.
For the support-agent example, Support task passed requires both Correct tool and Correct arguments to pass. Three of the six cases pass both checks, giving a 50% pass rate.
Create the composite
Use the latest SDK. Save this as create_composite.py or create_composite.mts and run it once with the quickstart's environment variables and existing scorers.
Save the printed scorer ID:
You can also create this scorer in Scorers. Choose Composite formula and Boolean, add the two input scorers, and set their Formula name fields to tool_correct and arguments_correct. Use tool_correct and arguments_correct as the formula and True means pass as the passing value.
Each input uses a fixed scorer version. New input versions don't update the composite automatically. Changing its formula or inputs creates a new composite version; existing results stay unchanged.
Request a calculation for each item
Upload the reported inputs and a calculation request together for each item. Creating the scorer alone doesn't calculate results.
In support_eval.py or support_eval.mts, add EvaluationResult to the SDK imports. Keep the agent, evaluator, and cases, then replace the client block below them with this code. It uses the input versions saved in the composite.
For a composite result, send only the item, scorer version, and calculate=True / calculate: true. PostHog calculates the value. Check each returned status and error_code / errorCode: an upload can succeed while a calculation fails.
This run expects 18 results: twelve reported and six composite. Accepted errors and skipped results count toward the total. Start a new experiment to test another composite version; completed runs can't accept new results.
Sending the inputs and calculation together works with the quickstart's permissions, including on retries. To reuse previously uploaded inputs, also grant evaluation:read and llm_analytics:read. For larger batches, keep each item's results together so the SDK can send them in one request.
Open the experiment to see Support task passed alongside the two reported scorers. Items without a calculation request show Not requested and don't affect its pass rate.
Write formulas
These formulas use the two boolean inputs from the support-agent example:
| Output | Formula | Meaning |
|---|---|---|
| Boolean | tool_correct and arguments_correct | Pass only when both checks are true. |
| Numeric | (number(tool_correct) + number(arguments_correct)) / 2 | Return 0, 0.5, or 1 for each item. |
| Categorical | "pass" if tool_correct and arguments_correct else "fail" | Return a category key configured on the composite. |
Formulas read raw input values. number(True) is 1 and number(False) is 0, even if an input scorer treats true as a failure. For categorical inputs, test category keys with expressions such as "helpful" in relevance.
In the UI, the formula editor suggests a formula from the output type and inputs' passing rules. Manual edits stay unchanged when you edit inputs. Click Use default formula to restore the suggestion.
You can use arithmetic (+, -, *, /), comparisons, and, or, not, conditional expressions, and the number, min, max, and abs functions. Each formula can reference up to 16 reported scorer versions. Every selected input must appear in the formula. Composite scorers can't use other composites as inputs or combine results across different items.
Handle missing inputs and calculation errors
If a pinned input result is missing, PostHog rejects the entire upload request. Include that input for the same item and version when you retry.
If an input has status error, skipped, or not_applicable, the composite inherits that status without a value. If several statuses apply, the order is error, then skipped, then not_applicable. Input errors use the code composite_input_error.
A calculation error, such as division by zero or a value outside the composite's configured bounds, produces an error result with code composite_formula_error. Valid reported inputs in the same request are still saved. These errors don't enter the composite's mean or pass rate.





