Log inSign up
yawnxyz avatar
Public
HTTP
Fork
1 Like
yawnxyz-sidebartoarchive.web.val.run
Readme

Sidebar.io Archiver

Sidebar's been one of my biggest design resources for the last decade. Since sidebar.io's break announcement this morning (https://sidebar.io/break/) I've set out to mess around with using Val.town to archive sidebar. The Google Sheet can be found here: https://docs.google.com/spreadsheets/d/1RghvMfPTR5xvHMAk1pKYuH2pFYYKOJ6bH0LLkHNZiuc/edit?usp=sharing

Initially I was just piping data to Google Sheets but that's slow, wasteful and kind of dumb... instead I just wrapped Hono around the code to provide a download the CSV to browser.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
import { Hono } from 'npm:hono';
import sheet from 'npm:@yawnxyz/sheetlog@0.0.15';
import { fetchJSON } from "https://esm.town/v/stevekrouse/fetchJSON?v=41";
import { Sema } from 'npm:async-sema';
import { modelProvider } from 'https://esm.town/v/yawnxyz/ai';
// Public Sheet:
// https://docs.google.com/spreadsheets/d/1RghvMfPTR5xvHMAk1pKYuH2pFYYKOJ6bH0LLkHNZiuc/edit?gid=0#gid=0
sheet.setup({
sheet: "links",
sheetUrl: "https://script.google.com/macros/s/AKfycbzavst1u2rbRAw-bkIrhoQT07507yvSHRFSkMKkDRQvsHAAe9lQ78s8c1TIfJJb7F50/exec"
})
// Create a semaphore to limit concurrency
const sema = new Sema(5); // Adjust the number based on your desired concurrency
export async function addLinksToSheet(links) {
let counter = 0;
for (const link of links) {
await sema.acquire();
(async () => {
try {
// let embeddings = await modelProvider.gen({
// embed: true,
// value: JSON.stringify(link)
// });
// console.log(`Adding link ${counter}:`, link.title, link.url, embeddings);
console.log(`Adding link ${counter}:`, link.title, link.url);
counter++;
await sheet.log({
title: link.title,
url: link.url,
postedAt: new Date(link.postedAt).toLocaleString('en-US', { timeZone: 'UTC' }),
data: JSON.stringify(link),
// embeddings: embeddings.embedding.join(','),
});
} finally {
sema.release();
}
})();
}
}
export async function getSidebar(limit = 1, start = 0) {
const gql = {
"operationName": "posts",
"variables": {
"input": {
"limit": limit,
"offset": start,
"enableTotal": true,
"enableCache": true,
"filter": {
"status": {
"_eq": 2
}
},
"sort": {
"postedAt": "desc"
}
}
},
"query": `query posts($input: MultiPostInput) {
posts(input: $input) {
results {
...PostFragment
__typename
}
totalCount
__typename
}
}
fragment PostFragment on Post {
_id
title
shortTitle
url
credit
twitterName
postedAt
createdAt
pageUrl
body
userId
categories {
...CategoryFragment
}
}
fragment CategoryFragment on Category {
_id
name
slug
pagePath
}`
};
// console.log('gql:', JSON.stringify(gql));
let results = await fetchJSON(`https://archive.sidebar.io/graphql`, {
Val Town is a social website to write and deploy JavaScript.
Build APIs and schedule functions from your browser.
Sign up with GitHub
Comments
Nobody has commented on this val yet: be the first!
June 12, 2024