Braintrust SDK

Braintrust is a platform for evaluating and shipping AI products. To learn more about Braintrust or sign up for free, visit the website or check out the docs.

The SDKs include utilities to:

• Log experiments and datasets to Braintrust
• Run evaluations (via the Eval framework)

This template shows you how to use the Braintrust SDK. This starter templated was ported from this one on GitHub.

• Click Fork on this val
• Get your Braintrust API key at https://www.braintrust.dev/app/settings?subroute=api-keys
• Add it to your project Environment Variables (on the left side bar of this project) as BRAINTRUST_API_KEY
• Click Run on the tutorial val

First, install the Braintrust SDK:

npm install braintrust autoevals

or

yarn add braintrust autoevals

Then, create a file named tutorial.eval.ts with the following code:

import { Eval } from "braintrust";
import { LevenshteinScorer } from "autoevals";

Eval("Say Hi Bot", {
  data: () => {
    return [
      {
        input: "Foo",
        expected: "Hi Foo",
      },
      {
        input: "Bar",
        expected: "Hello Bar",
      },
    ]; // Replace with your eval dataset
  },
  task: (input) => {
    return "Hi " + input; // Replace with your LLM call
  },
  scores: [LevenshteinScorer],
});

Then, run the following command:

BRAINTRUST_API_KEY=<YOUR_API_KEY> \
    npx braintrust eval tutorial.eval.ts

For more information, check out the docs:

• TypeScript
• Python