semanticSearchTurso

@janpaul123

Public

Script

Fork

0 Likes

v22

May 29, 2024

Readme

Part of Val Town Semantic Search.

Uses Turso to search embeddings of all vals, using the sqlite-vss extension.

Call OpenAI to generate an embedding for the search query.
Query the vss_vals_embeddings table in Turso using vss_search.
- The vss_vals_embeddings table has been generated by janpaul123/indexValsTurso. It is not run automatically.
- This table is incomplete due to a bug in Turso.

semanticSearchTurso

import { decode as base64Decode, encode as base64Encode } from "https://deno.land/std@0.166.0/encoding/base64.ts";

import { createClient } from "https://esm.sh/@libsql/client@0.6.0/web";

import { sqlToJSON } from "https://esm.town/v/nbbaier/sqliteExportHelpers?v=22";

import { db as allValsDb } from "https://esm.town/v/sqlite/db?v=9";

import OpenAI from "npm:openai";

export default async function semanticSearchPublicVals(query) {

const sqlite = createClient({

url: "libsql://valsembeddings-jpvaltown.turso.io",

authToken: Deno.env.get("TURSO_AUTH_TOKEN_VALSEMBEDDINGS"),

});

const openai = new OpenAI();

const embedding = await openai.embeddings.create({

model: "text-embedding-3-small",

input: query,

encoding_format: "base64",

dimensions: 256,

});

const embeddingBinary = base64Decode(embedding.data[0].embedding as any);

const res = await sqlToJSON(

await sqlite.execute({

sql:

"WITH matches AS (SELECT rowid, distance FROM vss_vals_embeddings WHERE vss_search(embedding, :embeddingBinary) LIMIT 50) SELECT id, distance FROM matches JOIN vals_embeddings ON matches.rowid = vals_embeddings.rowid",

args: { embeddingBinary },

}),

) as any;

const results = [];

for (const row of res) {