Log inSign Up
Avatar

karfau

24 public vals
Joined August 11, 2023

A helper to get the raw data of a val, using the very nice implementation from @pomdtr.raw.

Usage: https://www.val.town/v/karfau.test_getRaw

Also look at @karfau.rawUrl to just get the raw url of another val inside a val.

Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
import { rawUrl } from "https://esm.town/v/karfau/rawUrl";
import { raw } from "https://esm.town/v/pomdtr/raw?v=90";
export const getRaw = async (ref: Ref, extension: Ext = "json") => {
const rawExt = ["ts", "md", "json"].includes(extension) ? extension : "json";
const res = await raw(
new Request(rawUrl(ref, rawExt)),
);
if (!res.ok || rawExt !== "json")
return res.text();
const json = await res.json();
return extension !== "json" && extension in json ? json[extension] : json;
};
type Ref = [
user: string,
valName: string,
];
type ValData = {
"id": string;
"author": {
"id": string;
"username": string;
};
"name": string;
"code": string;
"privacy": "public" | "unlisted";
"public": boolean;
"version": number;
"runStartAt": string;
"runEndAt": string;
"logs": string[];
"output": {
"type": string;
"value": string;
};
"error": string | null;
"readme": string | null;
"likeCount": number;
"referenceCount": number;
};
type Ext = "ts" | "json" | "md" | keyof ValData;
2
3

Test runner

to be able to run a number of tests (e.g. on a different val). check the references for seeing how it is used. It is extracted into a val to avoid having all that clutter in the same val as your tests.

Each test is a named function (which can be async), the name is used as the name for the test.

  • the input passed as the first argument is passed to each test, great for importing assertion methods, stubs, fixed values, ... everything that you do not mutate during a test
  • if a function is async (it returns a promise) there is a timeout of 2 seconds before the test is marked as failed.
  • all tests are called in the declared order, but async tests run in parallel afterwards, so don't assume any order
  • if a test starts with skip it is not executed
  • if a test fails it throws the output, so it appears in the read box below the val and the evaluation/run is marked red
  • if all tests pass it returns the output, so it appears in the grey box and the evaluation/run is marked green.

Note: If you are using the test runner to store the result in that val, as described above, it is considered a "JSON val" and has a run button, but it also means that another of your vals could update the val with just any other (JSON) state. Alternatively you can define a function val that calls the test runner and have a separete val to keep the curretn test results, but it means after updating the tests you need to fest save that val and then reevaluate to val storing the test state.

Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
import { getRaw } from "https://esm.town/v/karfau/getRaw";
import { sleep } from "https://esm.town/v/stevekrouse/sleep?v=1";
export async function testRunner<
Input extends {
val?: Ref;
},
>(input: Input, ...tests: Test<Input>[]) {
let reportHeader = "";
try {
const [userHandle, valName] = input.val || [];
if (userHandle && valName) {
const version = await getRaw([
userHandle,
valName,
], "version");
if (typeof version === "number") {
reportHeader = `Testing @${userHandle}.${valName} v${version} ${new Date().toISOString()}:\n`;
}
}
}
catch {
}
let failedCount = 0;
let skipCount = 0;
let start = Date.now();
const result = (await Promise.all(tests.map(async (t, index) => {
const name = t.name.replaceAll(/_/g, " ");
if (name.startsWith("skip")) {
skipCount++;
return `${index + 1} [${name}]`;
}
const tstart = Date.now();
try {
const p = t(input);
if (p && p instanceof Promise) {
await Promise.race([
p,
sleep(2000).then(() => {
throw "async test did not complete in time";
}),
]);
}
return `${index + 1} ${name}: passed in ${Date.now() - tstart}ms`;
}
catch (error) {
failedCount++;
return `${index + 1} ${name}: failed after ${Date.now() - tstart}ms\n ${
/^assert/i.test(error.toString()) ? error : error.stack
}`;
}
}))).join("\n").replaceAll(/\033\[[\d;]+m/g, "");
if (failedCount) {
throw `${reportHeader}${failedCount} of ${tests.length} tests failed ${
skipCount ? `, ${skipCount} skipped` : ""
} (${Date.now() - start}ms):\n${result}`;
}
return `${reportHeader}${tests.length - skipCount} tests passed ${skipCount ? `, ${skipCount} skipped` : ""} (${
Date.now() - start
}ms):\n${result}`;
}
type Ref = [
userHandle: string,
valName: string,
];
type Test<
Input extends {
val?: Ref;
},
> = ((input: Input) => void) | ((input: Input) => Promise<void>);
83
3

This is a great template if you want to solve the https://adventofcode.com puzzles in a TDD style:

  1. dump your puzzle input into the input function
  2. copy/paste sample input and output from the page
  3. once all assertions pass, the solution is calculated

There is one function for each star that can be achieved.

Some helpers for common tasks are defined at the end of code: debug, toInt, exctractNumbers, sum, ... ?

PS: Did you know advent of code goes as far back as 2015 and you can still solve those puzzles?

Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import { expect } from "https://esm.town/v/karfau/chai";
expect(firstStar(`sample input`), "*1 sample 1").to.equal("expected output");
// expect(firstStar(``), "*1 sample 2").to.equal("?");
function firstStar(input: string) {
return "expected output"; // implement :)
}
// as as soon as the assertions pass, the solution is calculated
console.log("solution *1:", firstStar(input()));
expect(secondStar(``), "*2 sample 1").to.equal("?");
// expect(secondStar(``), "*2 sample 2").to.equal("?");
function secondStar(input: string) {
return "?";
}
console.log("solution *2:", secondStar(input()));
function input() {
return ``;
}
function toInt(s: string) {
return parseInt(s);
}
function extractNumbers(line: string) {
return line.match(/\d+/g).map(toInt);
}
// [0,1,2,3].reduce(sum, 0) => 6
function sum(sum: number, current: number) {
return sum + current;
}
// wrap a value to print and return it
function debug<T>(value: T, msg = "debug"): T {
console.log(msg, value);
return value;
}
6
2

This val has been created to avoid certain shortcomings of @vtdocs.verifyGithubWebhookSignature. So it was created as a mix/evolution of two sources:

This code is covered by tests which you can copy to run them, see @karfau.test_SignatureCheck

This val does not contain any val.town specific code (@-imports, console.email...), so it should be possible to run in Deno as is, potentially even in modern browsers (that support crypto and TextEncoder and modern ES syntax).

Usage

const myGithubWebhook = (req: Request) => {
  const {verify} = @karfau.SignatureCheck(); // you have to call it to get the verify function!
  const body = await req.text();
  const signature = req.headers.get("X-Hub-Signature-256");
  const verified = await verify(
    {payload:body, signature},
    @me.secrets.myGithubWebhookSecret,
    // optionally provide fallback secrets (as many as needed)
    // @me.secrets.myGithubWebhookSecretFallback
  );
  if (!verified) {
    return new Response(`Not verified`, 401);
  }
  const payload = JSON.parse(body);
  // actually do things in your webhook
};

By default the reason for failing verification is logged to console.error, but you can pass it a different handler:

const {verify} = @karfau.SignatureCheck((reason) => { throw new Error(reason); });

(be aware that it will silently fail if you don't try catch it in an endpoint and the return code will be 502)

Why

@vtdocs.verifyGithubWebhookSignature has the following issues:

  • it relies on the verify method of the outdated @octokit/webhooks-methods@3.0.2 which has (at least) two bugs that can make a difference when used in a webhook
    • it can throws errors instead of just returning false, which can be triggered by sending an invalid signature
    • it can be lured into checking a SHA1 signature if the signature header starts with sha1=
  • you need to pass the secret and payload as argument to the val, which makes them appear in the evaluation logs you produce (they are only visible for the author of the val if you run them as an API, but it still feels odd to see the secret in the evaluation logs.)
  • parameters are all of type string and the order can be confused
  • you can not use fallback secrets for rotating
Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
export function SignatureCheck(
onInvalid: (reason: string) => void = console.error,
) {
const algorithm = { name: "HMAC", hash: { name: "SHA-256" } };
const extractable = false;
const encoder = new TextEncoder();
const hexToUInt8Array = (string: string) =>
new Uint8Array(
(string.match(/[\da-fA-F]{2}/g) || []).map((s) => parseInt(s, 16)),
);
const verify = async (input: {
payload: string;
signature: string;
}, ...secrets: string[]): Promise<boolean> => {
const [secret, ...fallbackSecrets] = secrets;
try {
if (!input || !input.payload || !input.signature || !secret) {
throw "received at least one falsy argument";
}
const { payload, signature } = input;
const sigHex = signature.replace(/^sha\d+=/, "");
if (!/^([\da-fA-F]{2})+$/.test(sigHex)) {
throw "signature not matching expected pattern";
}
const keyBytes = encoder.encode(secret);
const key = await crypto.subtle.importKey(
"raw",
keyBytes,
algorithm,
extractable,
["sign", "verify"],
);
const sigBytes = hexToUInt8Array(sigHex);
const dataBytes = encoder.encode(payload);
const equal = await crypto.subtle.verify(
algorithm.name,
key,
sigBytes,
dataBytes,
);
if (!equal) {
throw "crypto.subtle.verify resolved false";
}
return true;
}
catch (error) {
if (fallbackSecrets.length > 0) {
return verify(input, ...fallbackSecrets);
}
onInvalid("[SignatureCheck.verify] " + error);
return false;
}
};
return { verify };
}
4
2

Finds the top level named val that triggered the execution of this val. (Ignores any untitled vals, since those are sometimes created as an execution context.)

By passing a function as the first argument you receive all references as arguments, and can pick or transform them, e.g.

Possible Limitation: https://discord.com/channels/1020432421243592714/1143386413748994143

Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
export function mainReference<T = ValRef>(transform?: Transform<T>): T {
const stack = new Error().stack;
let refs: ValRef[] = stack.match(/@@({.+?})@@/g).reverse().map((e) =>
JSON.parse(e.slice(2, -2))
);
const splitStack = stack.split("\n ");
//console.log("mainReference:stack", splitStack.length, splitStack);
//console.log("mainReference:all", refs);
refs = refs.filter(({ userHandle, valName, callNumber }) => {
if (
typeof userHandle !== "string" || typeof valName !== "string" ||
typeof callNumber !== "number"
)
return false;
// sometimes a dynamic val is created on the top level
if (valName.startsWith("untitled_"))
return false;
return true;
});
// console.log('mainReference:filtered', refs);
if (transform)
return transform(...refs);
return refs[0] as T;
}
type ValRef = {
userHandle: string;
valName: string;
callNumber?: number;
};
type Transform<T> = (...ref: ValRef[]) => T;
5
2

A type guard for vals that expect a WebApi Request as the first parameter.

Since a val can be invoked by run, web, express or even email, the correct type for a web endpoint should be req?: Request | unknown.

By using this type guard and handling the false case, all later code can safely access the request:

export const myVal = (req?: Request | unknown) => {
  if (!@karfau.isRequest(req)) {
    return `This val has to be used with the web endpoint, see ${@neverstew.thisValUrl()}`;
  }
  if (req.method === 'GET') {
    // ...
  }
}
Fork
1
2
export const isRequest = (req?: Request | unknown): req is Request =>
!!req && req instanceof Request;
5
2

Usage

Create valimport {githubEmojiUrl} from "https://esm.town/v/karfau/githubEmoji";
console.log(githubEmojiUrl('+1'))
//"https://github.githubassets.com/images/icons/emoji/unicode/1f44d.png?v8"

or in a browser

<img src="https://karfau-githubEmoji.web.val.run/+1"></img>

Looks like in the preview.

curl https://karfau-githubEmoji.web.val.run/+1

(prints "https://github.githubassets.com/images/icons/emoji/unicode/1f44d.png?v8")

If the name you pass (as argument or in the request path) is not in the list, it returns octocat

The list of names can be accessed using githubEmojiNames or by calling https://karfau-githubemoji.web.val.run/names

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import { GITHUB_EMOJIS } from "https://esm.town/v/karfau/GITHUB_EMOJIS";
import { fetch } from "https://esm.town/v/std/fetch?v=4";
export const isEmoji = (key: string) => Object.hasOwn(GITHUB_EMOJIS, key);
export const githubEmojiUrl = (name: string, fallback = "octocat"): string =>
GITHUB_EMOJIS[isEmoji(name) ? name : fallback];
export const githubEmojiNames = (): string[] => Object.keys(GITHUB_EMOJIS);
export default async function githubEmoji(req: Request): Promise<Response> {
const contentType = req.headers.get("Content-Type");
const nameFromURL = new URL(req.url).pathname.split("/").filter(Boolean)[0];
if (nameFromURL === "names") {
return Response.json(githubEmojiNames());
}
const url = githubEmojiUrl(nameFromURL);
if (contentType || contentType === "application/json") {
return Response.json(url);
}
try {
return fetch(url);
} catch (error) {
console.error(req.url, nameFromURL, error);
return new Response(
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"64\" height=\"64\" viewBox=\"218.329 117.251 67.385 67.385\" xml:space=\"preserve\"><path style=\"stroke:#e9d535;stroke-width:0;stroke-dasharray:none;stroke-linecap:butt;stroke-dashoffset:0;stroke-li
{ headers: { "Content-Type": "image/svg" }, status: 307, statusText: "Currently not available" },
);
}
}
1
1
Fork
1
2
3
4
5
6
7
8
9
10
11
12
import { rawUrl as rawUrl2 } from "https://esm.town/v/karfau/rawUrl";
import { getRaw } from "https://esm.town/v/karfau/getRaw";
export const test_getRaw = {
readme: await getRaw(["karfau", "getRaw"], "md"),
code: await getRaw(["karfau", "getRaw"], "ts"),
// without passing an extension we get a json with all data
json: await getRaw(["karfau", "getRaw"]),
// use the extension to access any existing key in the json
version: await getRaw(["karfau", "getRaw"], "version"),
rawUrl: rawUrl2(["karfau", "getRaw"]),
};
1

A helper for creating the URL to request the raw data of a val, defaulting to use the very nice implementation from @pomdtr.raw. Also look at @karfau.getRaw to get the data of a val inside a val.

Fork
1
2
3
4
5
6
7
8
9
10
export const rawUrl = (
[user, valName]: Ref,
extension: Ext = "json",
[rawUser, rawVal]: Ref = ["pomdtr", "raw"],
) => `https://${rawUser}-${rawVal}.web.val.run/${user}/${valName}.${extension}`;
type Ref = [
user: string,
valName: string,
];
type Ext = "ts" | "json" | "md";
2
1

A webhook to approve dependency PRs created by bots

The webhook can be configured on the repo or on the org level

  • it needs to have the Payload URL set to the "web endpoint" of the val ( ... -> Endpoints -> Copy web endpoint)
  • it needs to POST a json payload
  • it needs to receive the workflow_runs events
  • it needs to have the webhookSecret configured to the same value as in val town secrets (line 7)

(else response is 401: Not verified)

It will only approve if all of the following conditions are met:

  • the event action is completed, the workflow_run.conclusion has to be success, and the event is related to exactly one pull request
    (else response is 202: Ignored (event))
  • the PR is authored authored by one of the users listed in allowedAuthors (line 5)
    (else response is 202: Ignored (pr author))
  • the githubApiToken configured in line 9 needs to have repo access to the related repository
    (else response is 50x: some error message)
  • a branch protection has to be configured that requires at least one review approval and at least one status check
    (else response is 202: Ignored (branch protection))
  • the PR has auto-merge enabled
    (else response is 202: Ignored (pr status))
  • the PR has any failing status checks (even if not required)
    (else response is 202: Ignored (pr checks))
  • the current value for dryRun is false (line 3)
    (else response is 200: Would have been approved (dryRun))

If it approves the PR, it leaves a comment pointing to the website of this val.

Fork
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
import { SignatureCheck } from "https://esm.town/v/karfau/SignatureCheck";
import { response } from "https://esm.town/v/karfau/response";
import { isRequest } from "https://esm.town/v/karfau/isRequest";
import { thisValUrl } from "https://esm.town/v/neverstew/thisValUrl?v=1";
import process from "node:process";
export const githubWebhookApproveDependencyPRs = async (req?: Request | unknown) => {
// if true: do all the checks but don't approve PRs (status code 200 instead of 201)
const dryRun = false;
// only PRs created by these authors will be considered
const allowedAuthors = ["renovate[bot]"];
// the secret shared with the webhook
const webhookSecret = process.env.githubWebhookApproveDependencyPRs;
// the token to make github requests (needs `repo` permissions)
const githubApiToken = process.env.githubApproveDependencyPRsToken;
const valHttpUrl = thisValUrl();
const approvalMessage = `Automatically approved by ${valHttpUrl}`;
if (!isRequest(req)) {
return `Use the web endpoint for this val, see ${valHttpUrl}`;
}
else if (req.method === "GET") {
return new Response(
`Redirecting to <a href="${valHttpUrl}">${valHttpUrl}</a>`,
{ status: 303, headers: { Location: valHttpUrl } },
);
}
else if (req.method !== "POST") {
return response(
405,
`Method ${req.method} not allowed, see ${valHttpUrl}`,
);
}
const body = await req.text();
const signature = req.headers.get("X-Hub-Signature-256");
const { verify } = SignatureCheck();
const verified = await verify({ payload: body, signature }, webhookSecret);
if (!verified) {
return response(401, `Not verified, see ${valHttpUrl}`);
}
const payload: Payload = JSON.parse(body);
const { action, workflow_run, sender } = payload;
if (
action !== "completed" || workflow_run.conclusion !== "success" ||
workflow_run.pull_requests.length !== 1
) {
return response(202, "Ignored (event)");
}
const { Octokit } = await import("npm:@octokit/core");
const octokit = new Octokit({
auth: githubApiToken,
});
const { repository, organization } = payload;
const [pull_request] = workflow_run.pull_requests;
const pr =
(await octokit.request(`GET /repos/{owner}/{repo}/pulls/{pull_number}`, {
owner: organization.login,
repo: repository.name,
pull_number: pull_request.number,
})).data;
if (!allowedAuthors.includes(pr.user.login)) {
return response(202, "Ignored (pr author)");
}
if (!pr.auto_merge) {
return response(202, "Ignored (pr status)");
}
const checks = (await octokit.request(pr.statuses_url, {
headers: {
"X-GitHub-Api-Version": "2022-11-28",
},
})).data;
if (checks.find((check) => check.state !== "success")) {
return response(202, "Ignored (pr checks)");
}
const { required_pull_request_reviews, required_status_checks } =
(await octokit.request(
"GET /repos/{owner}/{repo}/branches/{branch}/protection",
{
owner: organization.login,
repo: repository.name,
branch: pull_request.base.ref,
headers: {
"X-GitHub-Api-Version": "2022-11-28",
},
},
)).data;
if (
required_pull_request_reviews?.required_approving_review_count !== 1 ||
required_status_checks?.contexts.length === 0
) {
return response(202, "Ignored (branch protection)");
}
if (dryRun) {
return response(200, "Would have been approved (dryRun)");
}
await octokit.request(
"POST /repos/{owner}/{repo}/pulls/{pull_number}/reviews",
{
owner: organization.login,
repo: repository.name,
pull_number: pull_request.number,
1