Public vals
120
processTextInChunks
@yawnxyz
Script
Text Chunk Processor This JavaScript function processes long text inputs by breaking them into smaller chunks, applying a custom processing function to each chunk, and optionally handling overlaps between chunks. Function Signature processTextInChunks(inputText, processFn, config) Parameters inputText (string): The long input text to be processed. processFn (function, optional): A function to process each chunk. Defaults to logging the chunk. config (object, optional): Configuration options. Config Options chunkLength (number): Length of each chunk. Default: 1000. overlapLength (number): Length of overlap between chunks. Default: 0. Usage const longText = "Your very long text here...";
const result = processTextInChunks(longText,
(mainChunk, overlapChunk, info) => {
// Process the chunk here
console.log(`Processing chunk from ${info.startIndex} to ${info.endIndex}`);
console.log(`Main chunk: ${mainChunk}`);
console.log(`Overlap: ${overlapChunk}`);
return mainChunk.length; // Example return value
},
{
chunkLength: 500,
overlapLength: 50
}
);
console.log(result); Notes If processFn is not provided, the function will log each chunk and return the original text. If processFn returns a value, these values will be collected in an array and returned. If processFn doesn't return anything, the original input text is returned. The processFn receives three arguments: mainChunk : The current chunk of text being processed. overlapChunk : The overlapping text from the previous chunk (empty for the first chunk). An info object containing: startIndex : Start index of the current chunk in the original text. endIndex : End index of the current chunk in the original text. isLastChunk : Boolean indicating if this is the last chunk. Use Cases Processing large texts in smaller, manageable pieces. Applying transformations or analysis to text while maintaining context through overlaps. Tokenization or parsing of large documents.
blobby
@yawnxyz
Script
Blobby Blobby is a simple wrapper around blob w/ more helpers for scoping, uploading/downloading, writing and reading strings, and so on. Todo Support lunr / semantic search, and embeddings Collections that support pointing to multiple blobs, like {description, embeddings, fileblob, ...} with a shared index / lookup
SidebarToArchive
@yawnxyz
HTTP (deprecated)
Sidebar.io Archiver Sidebar's been one of my biggest design resources for the last decade. Since sidebar.io's break announcement this morning (https://sidebar.io/break/) I've set out to mess around with using Val.town to archive sidebar.
The Google Sheet can be found here: https://docs.google.com/spreadsheets/d/1RghvMfPTR5xvHMAk1pKYuH2pFYYKOJ6bH0LLkHNZiuc/edit?usp=sharing Initially I was just piping data to Google Sheets but that's slow, wasteful and kind of dumb... instead I just wrapped Hono around the code to provide a download the CSV to browser.
getContentFromUrl
@yawnxyz
HTTP
getContentFromUrl Use this for summarizers.
Combines https://r.jina.ai/URL and markdown.download's Youtube transcription getter to do its best to retrieve content from URLs. https://arstechnica.com/space/2024/06/nasa-indefinitely-delays-return-of-starliner-to-review-propulsion-data
https://journals.asm.org/doi/10.1128/iai.00065-23 Usage: https://yawnxyz-getcontentfromurl.web.val.run/https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10187409/ https://yawnxyz-getcontentfromurl.web.val.run/https://www.youtube.com/watch?v=gzsczZnS84Y&ab_channel=PhageDirectory