semanticSearchNeon

@janpaul123

Public

Script

Forked from janpaul123/semanticSearchBlobs

Fork

0 Likes

v10

June 17, 2024

Readme

Part of Val Town Semantic Search.

Uses Neon to search embeddings of all vals, using the pg_vector extension.

Call OpenAI to generate an embedding for the search query.
Query the vals_embeddings table in Neon using the cosine similarity operator.
- The vals_embeddings table gets refreshed every 10 minutes by janpaul123/indexValsNeon.

semanticSearchNeon

import { decode as base64Decode, encode as base64Encode } from "https://deno.land/std@0.166.0/encoding/base64.ts";

import { Client } from "https://deno.land/x/postgres/mod.ts";

import { db as allValsDb } from "https://esm.town/v/sqlite/db?v=9";

import { blob } from "https://esm.town/v/std/blob";

import OpenAI from "npm:openai";

const dimensions = 1536;

export default async function semanticSearchPublicVals(query) {

const client = new Client(Deno.env.get("NEON_URL_VALSEMBEDDINGS"));

await client.connect();

const openai = new OpenAI();

const queryEmbedding = (await openai.embeddings.create({

model: "text-embedding-3-small",

input: query,

dimensions,

})).data[0].embedding;

const embeddedBinaryString = `[${queryEmbedding.join(",")}]`;

const result = await client

.queryObject`SELECT id, embedding <=> ${embeddedBinaryString} AS distance FROM vals_embeddings ORDER BY embedding <=> ${embeddedBinaryString} LIMIT 50`;

return result.rows.map((row) => {

const [author_username, name, version] = row.id.split("!!");

return { author_username, name, version, similarity: 1 - parseFloat(row.distance) };

});

}

const exampleQuery = "check dynamicland website for changes and email me";

console.log(await semanticSearchPublicVals(exampleQuery));

Val Town is a social website to write and deploy JavaScript.

Build APIs and schedule functions from your browser.

Comments

Nobody has commented on this val yet: be the first!

v10

June 17, 2024