Log inSign up
stevekrouse avatar
Public
Script
Fork
0 Likes
Readme

Auto-Upgrade for HTTP Preview

This val is experimentally testing if we can use an LLM to determine if an old-style HTTP val needs to be upgraded for the new HTTP runtime, currently in preview.

You can read more about the breaking change and upgrade proccess here: https://blog.val.town/blog/http-preview/#breaking-changes

In some light testing, it seems like ChatGPT 3.5 and 4o both are bad at this task, so I'm pausing experimenting with this for now.

Example output from 4o:

[
  {
    "name": "harlequinChickadee",
    "probabilityUpgradeNeeded": true,
    "reason": "The current code structure has several functions and program logic outside the main handler, including word selection, game state management, and SVG generation. These parts would not re-run with the new runtime, potentially affecting functionality. They need to be moved inside the handler to ensure consistent behavior across requests."
  },
  {
    "name": "redElephant",
    "probabilityUpgradeNeeded": "100%",
    "reason": "The initialization of `fs` and `vscode` objects should occur \n    inside the handler in the new runtime in order to ensure that they are \n    freshly created for each request. This is critical since the new runtime \n    does not rerun code outside of the handler for each request.."
  },
  {
    "name": "untitled_indigoNightingale",
    "probabilityUpgradeNeeded": false,
    "reason": "The code initializes and configures the Hono app outside of the handler, but it does not appear to have any stateful logic that would need to be re-calculated with each request. Each request will call the handler provided by Hono and should behave the same in the new runtime."
  },
  {
    "name": "untitled_pinkRoundworm",
    "probabilityUpgradeNeeded": true,
    "reason": "The functions `addComment` and `getComments` as well as the initialization \nof the KEY variable perform actions that are intended to be run per request. These need to be moved \ninside the relevant HTTP handler to ensure the behavior remains consistent in the new runtime."
  },
  {
    "name": "untitled_harlequinIguana",
    "probabilityUpgradeNeeded": false,
    "reason": "The provided code defines the handler directly without any side effects or additional code outside the handler. The behavior should remain the same in the new runtime."
  },
  {
    "name": "untitled_moccasinHeron",
    "probabilityUpgradeNeeded": false,
    "reason": "The code outside the handler is just a constant string declaration, which does not change behavior between requests. The handler itself handles requests correctly and independently."
  },
  {
    "name": "untitled_maroonSwallow",
    "probabilityUpgradeNeeded": false,
    "reason": "All the code, including the check for authentication,\n        is inside the handler function. This means the behavior will stay \n        the same with the new runtime."
  },
  {
    "name": "wikiOG",
    "probabilityUpgradeNeeded": true,
    "reason": "The function `getWikipediaInfo` defined outside of the handler makes network requests and processes data for each request. In the new runtime, this function would only be executed once and cached. To ensure the same behavior in the new runtime, this function should be moved into the handler."
  },
  {
    "name": "parsePostBodyExample",
    "probabilityUpgradeNeeded": false,
    "reason": "All the code is inside the handler, so the behavior will remain consistent in the new runtime."
  },
  {
    "name": "discordEventReceiver",
    "probabilityUpgradeNeeded": false,
    "reason": "All the relevant code for handling requests and logging input is inside the handler.\n        No code needs to be moved for the new runtime to function correctly."
  }
]

Feel free to fork this and try it yourself! If you could get it working, it'd be a big help for us as we upgrade the thousands of HTTP vals.

Future ideas:

  • Better LLM (Claude 3.5)
  • Better prompt
  • More examples
  • JSON mode
  • having it reply with upgraded code
  • send pull requests to users with public vals that probably need upgrading
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
import { zip } from "https://esm.town/v/pomdtr/sql";
import { db } from "https://esm.town/v/sqlite/db";
import OpenAI from "npm:openai";
async function getVals(username, type, limit) {
const res = await db.execute({
sql: `
SELECT * FROM vals
where
author_username = ?
AND type = ?
LIMIT ?
`,
args: [
username,
type,
limit,
],
});
return zip(res);
}
async function checkHTTPPreviewUpgrade(code) {
const openai = new OpenAI();
const completion = await openai.chat.completions.create({
messages: [
{
role: "system",
content: `
We built a new runtime for HTTP vals that is much faster at scale.
Your job is to input code from the old runtime and ensure the behavior is the same
in the new runtime.
Up until now, every time a val it starts a new Deno process and installs all dependencies from scratch.
With the new runtime, we keep Deno processes around and forward multiple requests to it.
There is one breaking change with this new feature: it only re-runs the val handler,
not any of the other code in val. This means that if any code outside of handler needs
to run on every request, to upgrade to the new runtime, move it into your handler.
Reply ONLY in JSON in the format: {
probabilityUpgradeNeeded: boolean,
reason: string,
}
No code fences.
`,
},
{
role: "user",
content: `const number = Math.random();
export default function(req: Request) {
return Response.json(number);
}`,
},
{
role: "assistant",
content: JSON.stringify({
probabilityUpgradeNeeded: "90%",
reason: `In the old runtime, every request to this handler would return
a newly generated random number. However in the new runtime, the
the random number would be cached sometimes. To ensure the
same behavior in the new runtime, we need to move the random generation
inside the handler.
`,
}),
},
{
role: "user",
content: `import { BrowserWebSocketClientAdapter } from "npm:@automerge/automerge-repo-network-websocket";
import { isValidAutomergeUrl, Repo } from "npm:@automerge/automerge-repo/slim";
/* set up Automerge's internal wasm guts manually */
import { automergeWasmBase64 } from "npm:@automerge/automerge/automerge.wasm.base64.js";
import * as automerge from "npm:@automerge/automerge/slim";
await automerge.next.initializeBase64Wasm(automergeWasmBase64);
/* This example will return the contents of a documentID passed in as the path as JSON. */
export default async function(req: Request): Promise<Response> {
const docId = new URL(req.url).pathname.substring(1);
if (!isValidAutomergeUrl("automerge:" + docId)) {
return Response.error();
}
const repo = new Repo({ network: [new BrowserWebSocketClientAdapter("wss://sync.automerge.org")] });
const handle = repo.find(docId);
const contents = await handle.doc();
return Response.json(contents);
}
`,
},
{
role: "assistant",
content: JSON.stringify({
probabilityUpgradeNeeded: "20%",
reason: `The only code outside of the handler is initializing a wasm module,
Val Town is a social website to write and deploy JavaScript.
Build APIs and schedule functions from your browser.
Sign up with GitHub
Comments
Nobody has commented on this val yet: be the first!
July 18, 2024