Log inSign up
gwoods22 avatar
Public
Script
Fork
0 Likes
Readme

Website Scraper Template

A basic website scraper template that can be ran on an interval to check for changes. It uses blob storage to save whatever data you'd like to compare the website to.

Uses my sendNotification val to alert yourself of the change.

Steps to use

  1. Add the scrapeURL you would like to scrape and pick a name for your blob storage blobKey.

  2. Change the "selector" value to fit your needs. cheerio is used to find the part of the page you'd like to check.

  3. Adjust the if statement to detect changes and update your blob

  4. Craft a message to be sent with sendNotification() function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import sendNotification from "https://esm.town/v/gwoods22/sendNotification";
import { blob } from "https://esm.town/v/std/blob";
import * as cheerio from "npm:cheerio@1.0.0";
const scrapeURL = "";
const blobKey = "";
export default async () => {
const variable: string = await blob.getJSON(blobKey) ?? "";
const response = await fetch(scrapeURL);
const body = await response.text();
const $ = cheerio.load(body);
const newVariable = $("selector").attr("id");
const arr = $("multi selector").toArray().map((x) => $(x).text().trim());
if (newVariable !== variable) {
await blob.setJSON(blobKey, newVariable);
const message = `New variable is ${newVariable}`;
await sendNotification("🔊 Update! 🔊", message, scrapeURL);
console.log(message);
} else {
console.log("Variable is still", variable);
}
};
Val Town is a social website to write and deploy JavaScript.
Build APIs and schedule functions from your browser.
Sign up with GitHub
Comments
Nobody has commented on this val yet: be the first!
September 6, 2024