pomdtr
I mainly enjoy building dev tools:
- VS Code integration: https://github.com/pomdtr/valtown-vscode
- CLI: https://github.com/pomdtr/vt
Public vals
317
pomdtr
code_search_is_easy
HTTP
Code Search is Easy Earlier this week, Tom MacWright posted Code Search is Hard . He describes the research he his doing to improve the code search experience of Val Town . It was a great read, and you might have seen it trending on Hacker News . As Val Town's most active user (behind Steve Krouse, one of the founders of Val Town), I for sure agree with Tom that the search feature needs improvements. But while reading his post, I immediately thought of a different approach to the problem. And a few hours later, Val Town Search was born. Do things that don't scale How does this new shiny search engine work? Well, it's quite simple. I wrote a Deno script that fetches all vals from the Val Town API. #!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
path.join(import.meta.dirname!, "blocklist.txt")
)
.split("\n")
.map((line) => line.trim())
.filter((line) => line.length > 0);
let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
console.log("fetching", url);
const resp = await fetch(url);
if (!resp.ok) {
console.error(resp.statusText);
Deno.exit(1);
}
const { data, links } = await resp.json();
vals.push(...data);
if (!links.next) {
break;
}
url = links.next;
}
Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
const slug = `${val.author.username}/${val.name}`;
if (blocklist.includes(slug)) {
console.log("skipping", slug);
continue;
}
const userDir = path.join(dir, val.author.username);
Deno.mkdirSync(userDir, { recursive: true });
Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
} I pushed the data to a Github Repository (now private) I added a Github Action that runs the script every hour to refresh the data. #!/usr/bin/env -S deno run -A
import * as path from "https://deno.land/std/path/mod.ts";
const dir = path.join(import.meta.dirname!, "..", "vals");
const blocklist = Deno.readTextFileSync(
path.join(import.meta.dirname!, "blocklist.txt")
)
.split("\n")
.map((line) => line.trim())
.filter((line) => line.length > 0);
let url = `https://api.val.town/v1/search/vals?limit=100&query=+`;
const vals = [];
while (true) {
console.log("fetching", url);
const resp = await fetch(url);
if (!resp.ok) {
console.error(resp.statusText);
Deno.exit(1);
}
const { data, links } = await resp.json();
vals.push(...data);
if (!links.next) {
break;
}
url = links.next;
}
Deno.removeSync(dir, { recursive: true });
Deno.mkdirSync(dir, { recursive: true });
for (const val of vals) {
const slug = `${val.author.username}/${val.name}`;
if (blocklist.includes(slug)) {
console.log("skipping", slug);
continue;
}
const userDir = path.join(dir, val.author.username);
Deno.mkdirSync(userDir, { recursive: true });
Deno.writeTextFileSync(path.join(userDir, `${val.name}.tsx`), val.code);
} I created a simple frontend on top of the Github Search API that allows you to search the data. It's hosted on Val Town (obviously). That was it. I didn't have to build a complex search engine, I just used the tools that were available to me. Is this a scalable solution for Val Town? Probably not. Am I abusing the Github API? Maybe. Does it work better than the current search feature of Val Town? Absolutely! I hope that the val.town engineers will come up with a search feature that will put my little project to shame. But for now, you won't find a better way to search for vals than Val Town Search . PS: This post was written / is served from Val Town
1
pomdtr
valTownSearch
HTTP
Val Town Search Search for vals using the Github API. Either use the provided UI, or the query param: https://val-town-search.pomdtr.me/search?q=fetchJSON How does it work ? I've wrote about it! Todos [x] Embed the results in the UI [x] Refresh the vals on a cron using a github action [ ] Improve layout on small screens [ ] Support json Accept header [ ] Add pagination params [ ] Allow to filter by authors
1