Braintrust SDK

Braintrust is a platform for evaluating and shipping AI products. To learn more about Braintrust or sign up for free, visit our website or check out the docs.

This repository contains the Javascript SDKs for Braintrust. The SDKs include utilities to:

• Log experiments and datasets to Braintrust
• Run evaluations (via the Eval framework)

This template shows you how to use the Braintrust SDK. This starter templated was ported from this one on GitHub.

• Click Fork on this val
• Get your Braintrust API key at https://www.braintrust.dev/app/settings?subroute=api-keys
• Add it to your project Environment Variables (on the left side bar of this project) as BRAINTRUST_API_KEY
• Click Run on the tutorial val

First, install the Braintrust SDK:

npm install braintrust autoevals

or

yarn add braintrust autoevals

Then, create a file named tutorial.eval.ts with the following code:

import { Eval } from "braintrust";
import { LevenshteinScorer } from "autoevals";

Eval("Say Hi Bot", {
  data: () => {
    return [
      {
        input: "Foo",
        expected: "Hi Foo",
      },
      {
        input: "Bar",
        expected: "Hello Bar",
      },
    ]; // Replace with your eval dataset
  },
  task: (input) => {
    return "Hi " + input; // Replace with your LLM call
  },
  scores: [LevenshteinScorer],
});

Then, run the following command:

BRAINTRUST_API_KEY=<YOUR_API_KEY> \
    npx braintrust eval tutorial.eval.ts

For more information, check out the docs:

• TypeScript
• Python