Public
HTTP
Forked from pomdtr/valTownSearch
janpaul123-valtownsemanticsearch.web.val.run
v39
June 17, 2024
Readme
This val hosts an HTTP server that lets you search all vals based on vibes. If you search for "discord bot" it shows all vals that have "discord bot" vibes.
It does this by comparing embeddings from OpenAI generated for the code of all public vals, to an embedding of your search query.
This is an experiment to see if and how we want to incorporate semantic search in the actual Val Town search page.
I implemented three backends, which you can switch between in the search UI. Check out these vals for details on their implementation.
- Neon: storing and searching embeddings using the pg_vector extension in Neon's Postgres database.
- Searching: janpaul123/semanticSearchNeon
- Indexing: janpaul123/indexValsNeon
- Blobs: storing embeddings in Val Town's standard blob storage, and iterating through all of them to compute distance. Slow and terrible, but it works!
- Searching: janpaul123/semanticSearchBlobs
- Indexing: janpaul123/indexValsBlobs
- Turso: storing and searching using the sqlite-vss extension. Abandoned because of a bug in Turso's implementation.
- Searching: janpaul123/semanticSearchTurso
- Indexing: janpaul123/indexValsTurso
All implementations use the database of public vals, made by Achille Lacoin, which is refreshed every hour. The Neon implementation updates every 10 minutes, and the other ones are not updated. I also forked Achille's search UI for this val.
Please share any feedback and suggestions, and feel free to fork our vals to improve them. This is a playground for semantic search before we implement it in the product for real!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
/** @jsxImportSource npm:preact **/
import codeOnValTown from "https://esm.town/v/andreterron/codeOnValTown?v=50";
import semanticSearchBlobs from "https://esm.town/v/janpaul123/semanticSearchBlobs";
import semanticSearchNeon from "https://esm.town/v/janpaul123/semanticSearchNeon";
import semanticSearchTurso from "https://esm.town/v/janpaul123/semanticSearchTurso";
import { render } from "npm:preact-render-to-string";
const githubQuery = (query: string) => encodeURIComponent(`${query} repo:pomdtr/val-town-mirror path:vals/`);
async function handler(req: Request) {
const url = new URL(req.url);
if (url.pathname == "/opensearch.xml") {
return new Response(
`<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>VAL VIBES: Val Town Semantic Search</ShortName>
<Description>Semantic search for vals using the Github API</Description>
<Url type="text/html" method="get" template="https://janpaul123-valtownsemanticsearch.web.val.run/search?q={searchTerms}"/>
<Url type="application/opensearchdescription+xml" template="https://janpaul123-valtownsemanticsearch.web.val.run/opensearch.xml"/>
<Image height="16" width="16" type="image/png">https://pomdtr-favicons.web.val.run/val-town</Image>
<moz:SearchForm>https://janpaul123-valtownsemanticsearch.web.val.run/search</moz:SearchForm>
</OpenSearchDescription>`,
{
headers: {
"Content-Type": "application/opensearchdescription+xml",
},
},
);
}
function form(query, type) {
return (
<form method="GET" action="/search">
<fieldset>
<div role="search">
<input
type="search"
name="q"
value={query}
placeholder="Search for vals..."
autocomplete="off"
aria-label="Search"
/>
<input type="submit" value="Search" />
</div>
<div style={{ display: "flex" }}>
<label>
<input name="type" value="neon" type="radio" checked={type == "neon"}></input>Neon
</label>
<label>
<input name="type" value="blobs" type="radio" checked={type == "blobs"}></input>
Blobs (not updated)
</label>
<label>
<input name="type" value="turso" type="radio" checked={type == "turso"}></input>
Turso (not updated; small subset of vals)
</label>
</div>
</fieldset>
</form>
);
}
if (url.pathname == "/search") {
const query = url.searchParams.get("q");
const type = url.searchParams.get("type") || "neon";
const items = await ({
neon: semanticSearchNeon,
blobs: semanticSearchBlobs,
turso: semanticSearchTurso,
}[type])(query);
const accept = req.headers.get("Accept");
if (accept == "text/json" || accept == "application/json") {
return Response.json(items);
}
const seenItems = new Set();
return new Response(
render(
<html>
<head>
<title>VAL VIBES: Val Town Semantic Search</title>
<link rel="icon" href="https://pomdtr-favicons.web.val.run/val-town" />
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/@picocss/pico@2/css/pico.min.css"
/>
<link href="https://cdn.jsdelivr.net/npm/prismjs@v1.29.0/themes/prism.css" rel="stylesheet" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
</head>
<body>
<main class="container">
<h1>
<a href="/" class="contrast">VAL VIBES: Val Town Semantic Search</a>
</h1>
<section>
{form(query, type)}
</section>
<hr />
Val Town is a social website to write and deploy JavaScript.
Build APIs and schedule functions from your browser.
Comments
Nobody has commented on this val yet: be the first!
v39
June 17, 2024