Log inSign up
harryhood avatar
Public
Script
Fork
0 Likes
Readme

Summary

This function allows you to run a rate limited async pool to make sure no more than poolLimit items at a time are run for a given waitTime.

Example Usage

async function fetchURL(url: string): Promise<string[]> {
  const response = await fetch(url);
  const html = await response.text();
  const urls = extractUrlsFromResponse(html);
  return urls;
}

const allUrls = (await rateLimitedAsyncPool(
  ["url1", "url2", "url3"], 2, fetchURL, 500
)).flat();
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import { asyncPool } from "https://esm.town/v/harryhood/asyncPool";
// A generic utility function that handles concurrency and rate-limiting
export async function rateLimitedAsyncPool<T, V>(
items: T[],
poolLimit: number,
fn: (item: T) => Promise<V>,
waitTime: number,
): Promise<V[]> {
return asyncPool(items, poolLimit, async (item) => {
const result = await fn(item);
await new Promise((resolve) => setTimeout(resolve, waitTime));
return result;
});
}
Val Town is a social website to write and deploy JavaScript.
Build APIs and schedule functions from your browser.
Sign up with GitHub
Comments
Nobody has commented on this val yet: be the first!
November 18, 2023