Surprising embeddings

\n Embeddings. They're one of the parts of the LLM/AI wave that I sort of like.\n

\n An embedding model takes a bit of text and returns a big array of floating-point numbers that represent its\n position in a sort of _hyper-dimensional embedding space_. You interpret this array as sort of a big coordinate,\n like longitude & latitude & elevation & time and a thousand other dimensions, and you get sort of the word's place\n in the computer's… mind-brain, or whatever.\n

\n Embeddings are pretty cool when they work, because they sort of capture the idea of{\" \"}\n 'vibes', which makes them useful for search applications.\n Someone can search for 'lettuce' and get results for 'spinach' too because they're similar. It's amazing when it\n works. And pretty weird when it doesn't!\n

\n This project is based on this{\" \"}\n \n blog post I read last year\n {\" \"}\n about 'how RAG fails', which was unfortunately written by someone{\" \"}\n with a RAG alternative to sell.\n

\n But maybe maybe RAG is doomed, in some ways? Half the time it does something magical, and half the time it\n produces something deeply weird. It thinks that, for example, \"the 1900s\" are more similar to \"the 1700s\" than to\n \"20th century.\" Most of these, I think, are related to how the RAG is a little literal and focused on the style of\n a phrase as well as the content of it. On the other hand, all other search solutions, like fuzzy matching and most\n ranking algorithms, are also some sort of good-enough heuristic, so maybe RAG is totally fine.\n

\n Play around. You can enter up to 8 words or phrases. The thickness and opacity of the connections indicates how\n connected the model thinks the things are.\n

\n \n)\n\nconst examples =

\n Examples:{\" \"}\n {[\n {\n title: \"Apples and bananas are really different\",\n q: \"/?word=apples&word=oranges&word=bananas&word=grapefruit&word=blueberry&word=blackberry&word=raspberry&word=\",\n },\n {\n title: \"Lexus is way different from Tesla\",\n q: \"/?word=porsche&word=ferrari&word=tesla&word=kia&word=lexus&word=toyota&word=hyundai&word=\",\n },\n {\n title: \"Cats and dogs are more similar to each other than to canine and feline\",\n q: \"/?word=dog&word=cat&word=feline&word=canine&word=&word=&word=&word=\",\n },\n {\n title: \"Chess pieces wtf\",\n q: \"/?word=pawn&word=king&word=queen&word=rook&word=castle&word=&word=&word=\",\n },\n ].map((ex, i, arr) => {\n return (\n <>\n {ex.title}\n {i === arr.length - 1 ? \"\" : \", \"}\n \n )\n })}\n

\n\n// Based on https://github.com/Shawns2759/vector-similarity MIT\nfunction dotProduct(a: number[], b: number[]): number {\n return a.reduce((sum, value, index) => sum + value * b[index], 0)\n}\n\nfunction vectorMagnitude(a: number[]): number {\n return Math.sqrt(a.reduce((sum, value) => sum + value * value, 0))\n}\n\nfunction cosineSimilarity(a: number[], b: number[]): number {\n const dotProd = dotProduct(a, b)\n const magnitudeA = vectorMagnitude(a)\n const magnitudeB = vectorMagnitude(b)\n return dotProd / (magnitudeA * magnitudeB)\n}\n\nconst app = new Hono()\napp.get(\"/\", async (c) => {\n const r = blockAi(c.req.headers.get(\"User-Agent\"))\n if (r) return r\n\n if (working) {\n return c.text(\"Sorry, another embedding is being fetched, wait a moment and try again\")\n }\n\n working = true\n const inputQ = c.req.queries(\"word\")\n const items = (inputQ ?? []).filter(Boolean)\n const inputResult = v.safeParse(inputSchema, items)\n\n if (inputResult.success) {\n const input = inputResult.output\n const embeddings = await client.embed({\n input,\n model: \"voyage-3-lite\",\n })\n\n const zipped = input.map((word, i) => {\n return {\n word,\n embedding: embeddings.data?.at(i)?.embedding,\n }\n })\n\n const data = {\n nodes: zipped.map(z => {\n return {\n id: z.word,\n }\n }),\n links: zipped.flatMap((z, i) => {\n return zipped.slice(i + 1).map(z2 => {\n return {\n source: z.word,\n target: z2.word,\n value: cosineSimilarity(z.embedding, z2.embedding),\n }\n })\n }),\n }\n\n let nodePool = data.nodes.slice()\n let ordered = []\n\n const linkOrder = data.links.sort((l1, l2) => {\n return l2.value - l1.value\n })\n\n console.log(linkOrder)\n for (let link of linkOrder) {\n const s = nodePool.find(n => {\n return n.id === link.source\n })\n const t = nodePool.find(n => n.id === link.target)\n nodePool = nodePool.filter(n => n !== s && n !== t)\n console.log(s, t)\n if (s) ordered.push(s)\n if (t) ordered.push(t)\n }\n\n data.nodes = ordered.concat(nodePool)\n\n setTimeout(() => {\n working = false\n }, THROTTLE)\n\n return c.html(\n \n \n Surprising embeddings\n \n\n \n \n \n

\n Surprising embeddings\n \n Using Voyage AI voyage-3-lite model\n \n

\n \n \n \n

\n\n \n\n {examples}\n\n

{intro}

\n\n \n \n ,\n )\n }\n\n return c.html(\n \n \n Surprising embeddings\n \n\n \n \n \n

\n Surprising embeddings\n \n Using Voyage AI voyage-3-lite model\n \n

\n\n {examples}\n\n \n\n

{intro}

\n \n \n ,\n )\n})\n\nexport default app.fetch",[],[179],{"_103":188,"_95":189,"_140":190,"_142":-5,"_143":131,"_144":191,"_146":147,"_105":192,"_100":193,"_150":151,"_152":194,"_154":26,"_155":195,"_98":196},"caf606ce-da98-11ef-a3ba-e6cdfca9ef9f","visualization","83d70fb2-da85-11ef-a00c-e6cdfca9ef9f",306,["D",1737753524665],156,"import * as d3 from \"https://cdn.jsdelivr.net/npm/d3@7.9.0/+esm\"\n\nconst width = document.querySelector(\"main\").offsetWidth\nconst dataText = document.querySelector(\"#data\").innerHTML\nconsole.log(dataText)\n\nconst data = JSON.parse(\n dataText,\n)\n// Specify the dimensions of the chart.\nconst height = 700\n\n// Specify the color scale.\nconst color = d3.scaleOrdinal(d3.schemeCategory10)\n\n// The force simulation mutates links and nodes, so create a copy\n// so that re-evaluating this cell produces the same result.\nconst links = data.links.map((d) => ({ ...d }))\nconst nodes = data.nodes.map((d) => ({ ...d }))\n\nconst valueRange = d3.extent(data.links, d => d.value)\nconst scale = d3.scaleLinear().domain(valueRange).range([1, 40])\nconst opacityScale = d3.scaleLinear().domain(valueRange).range([0.5, 1])\nconst distanceScale = d3.scaleLinear().domain(valueRange).range([width, width / 10])\n\nconst positions = Array.from({ length: nodes.length }, (_, i) => {\n return {\n x: (Math.cos((i / nodes.length) * Math.PI * 2) * width / 2.5) + width / 2,\n y: (Math.sin((i / nodes.length) * Math.PI * 2) * width / 2.5) + width / 2,\n }\n})\n\n// Create the SVG container.\nconst svg = d3\n .create(\"svg\")\n .attr(\"width\", \"100%\")\n .attr(\"viewBox\", [0, 0, width, height])\n .attr(\"style\", \"max-width: 100%; aspect-ratio: 1/1;\")\n\n// Add a line for each link, and a circle for each node.\nconst link = svg\n .append(\"g\")\n .selectAll()\n .data(links)\n .join(\"line\")\n .attr(\"stroke-opacity\", d => opacityScale(d.value))\n .attr(\"stroke\", d => {\n return d3.interpolateLab(color(d.source), color(d.target))(0.5)\n })\n .attr(\"stroke-width\", (d) => scale(d.value))\n .attr(\"x1\", d => {\n const idx = nodes.findIndex(n => n.id === d.target)\n return positions[idx].x\n })\n .attr(\"y1\", d => {\n const idx = nodes.findIndex(n => n.id === d.target)\n return positions[idx].y\n })\n .attr(\"x2\", d => {\n const idx = nodes.findIndex(n => n.id === d.source)\n return positions[idx].x\n })\n .attr(\"y2\", d => {\n const idx = nodes.findIndex(n => n.id === d.source)\n return positions[idx].y\n })\n\nconst node = svg\n .append(\"g\")\n .selectAll()\n .data(nodes)\n .join(\"g\")\n .attr(\"transform\", (d, i) => `translate(${positions[i].x}, ${positions[i].y})`)\n\nnode\n .append(\"circle\")\n .attr(\"r\", 50)\n .attr(\"stroke\", \"#fff\")\n .attr(\"stroke-width\", 1.5)\n .attr(\"fill\", (d) => color(d.id))\n\nnode\n .append(\"text\")\n .attr(\"x\", 0)\n .attr(\"text-anchor\", \"middle\")\n .attr(\"font-size\", 14)\n .attr(\"font-weight\", \"bold\")\n .attr(\"fill\", \"white\")\n .attr(\"y\", \"0.31em\")\n .text(d => d.id)\n .clone(true).lower()\n .attr(\"fill\", \"none\")\n .attr(\"stroke\", \"black\")\n .attr(\"stroke-width\", 3)\n\nnode.append(\"title\").text((d) => d.id)\n\nnode.on(\"mouseenter\", e => {\n link.attr(\"opacity\", l => {\n const d = d3.select(e.target).data()[0]\n return (l.source === d.id || l.target === d.id) ? 1 : 0.1\n })\n})\n\nnode.on(\"mouseleave\", d => {\n link.attr(\"opacity\", d => opacityScale(d.value))\n})\n\ndocument.querySelector(\"#root\").appendChild(svg.node())\n\nconsole.log(data)",[],[189],"counts",{"_199":31,"_200":149,"_201":31,"_202":149},"pulls","branches","forks","envVars","contents",[],"file",{"_103":159,"_95":99,"_140":160,"_142":-5,"_143":131,"_144":130,"_146":147,"_105":207,"_100":162,"_150":151,"_152":163,"_154":26,"_98":208},["D",1738862357875],[99],"upstream","routes/_app.x.$handle.$name.branch.$branch.history_.$",{"_93":94,"_95":96,"_97":41,"_98":99,"_101":212,"_205":216,"_219":220,"_279":280},{"_103":104,"_95":96,"_105":213,"_107":-5,"_108":109,"_110":111,"_112":-5,"_11":113,"_114":-5,"_115":26,"_116":117,"_118":117,"_119":120,"_121":120,"_122":120,"_123":117,"_124":26,"_20":214,"_128":215},["D",1737736218273],{"_93":94,"_126":127},{"_100":130,"_95":41,"_103":131,"_132":104,"_133":-5,"_134":-5},{"_103":159,"_95":99,"_140":160,"_142":-5,"_143":131,"_144":130,"_146":147,"_105":217,"_100":162,"_150":151,"_152":163,"_154":26,"_98":218},["D",1738862357875],[99],"history",[221,223,229,234,239,244,249,254,259,265,270,275],{"_103":159,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":162,"_144":130,"_154":26,"_105":222,"_146":147},["D",1738862357875],{"_103":224,"_140":160,"_95":99,"_150":151,"_152":225,"_142":-5,"_143":131,"_100":226,"_144":227,"_154":26,"_105":228,"_146":26},"7e208732-e4ae-11ef-85f6-e6cdfca9ef9f","export default Object.keys({\n \"AI2Bot\": {\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n \"function\": \"Content is used to train open language models.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Explores 'certain domains' to find web content.\",\n },\n\n \"Ai2Bot-Dolma\": {\n \"description\": \"Explores 'certain domains' to find web content.\",\n \"frequency\": \"No information provided.\",\n \"function\": \"Content is used to train open language models.\",\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n },\n \"Amazonbot\": {\n \"operator\": \"Amazon\",\n \"respect\": \"Yes\",\n \"function\": \"Service improvement and enabling answers for Alexa users.\",\n \"frequency\": \"No information provided.\",\n \"description\":\n \"Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.\",\n },\n \"anthropic-ai\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"Applebot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Search Crawlers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain references to the website. More info can be found at https://darkvisitors.com/agents/agents/applebot\",\n },\n \"Applebot-Extended\": {\n \"operator\": \"[Apple](https://support.apple.com/en-us/119829#datausage)\",\n \"respect\": \"Yes\",\n \"function\": \"Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.\",\n },\n \"Bytespider\": {\n \"operator\": \"ByteDance\",\n \"respect\": \"No\",\n \"function\": \"LLM training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Downloads data to train LLMS, including ChatGPT competitors.\",\n },\n \"CCBot\": {\n \"operator\": \"[Common Crawl Foundation](https://commoncrawl.org)\",\n \"respect\": \"[Yes](https://commoncrawl.org/ccbot)\",\n \"function\": \"Provides open crawl dataset, used for many purposes, including Machine Learning/AI.\",\n \"frequency\": \"Monthly at present.\",\n \"description\":\n \"Web archive going back to 2008. [Cited in thousands of research papers per year](https://commoncrawl.org/research-papers).\",\n },\n \"ChatGPT-User\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Takes action based on user prompts.\",\n \"frequency\": \"Only when prompted by a user.\",\n \"description\": \"Used by plugins in ChatGPT to answer queries based on user input.\",\n },\n \"Claude-Web\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"ClaudeBot\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\":\n \"[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"cohere-ai\": {\n \"operator\": \"[Cohere](https://cohere.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Retrieves data to provide responses to user-initiated prompts.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Retrieves data based on user prompts.\",\n },\n \"cohere-training-data-crawler\": {\n \"operator\":\n \"Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"cohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products. More info can be found at https://darkvisitors.com/agents/agents/cohere-training-data-crawler\",\n },\n \"Crawlspace\": {\n \"operator\": \"[Crawlspace](https://crawlspace.dev)\",\n \"respect\": \"[Yes](https://news.ycombinator.com/item?id=42756654)\",\n \"function\": \"Scrapes data\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Provides crawling services for any purpose, probably including AI model training.\",\n },\n \"Diffbot\": {\n \"operator\": \"[Diffbot](https://www.diffbot.com/)\",\n \"respect\": \"At the discretion of Diffbot users.\",\n \"function\": \"Aggregates structured web data for monitoring and AI model training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.\",\n },\n \"DuckAssistBot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"DuckAssistBot is used by DuckDuckGo's DuckAssist feature to fetch content and generate realtime AI answers to user searches. More info can be found at https://darkvisitors.com/agents/agents/duckassistbot\",\n },\n \"FacebookBot\": {\n \"operator\": \"Meta/Facebook\",\n \"respect\": \"[Yes](https://developers.facebook.com/docs/sharing/bot/)\",\n \"function\": \"Training language models\",\n \"frequency\": \"Up to 1 page per second\",\n \"description\":\n \"Officially used for training Meta \\\"speech recognition technology,\\\" unknown if used to train Meta AI specifically.\",\n },\n \"FriendlyCrawler\": {\n \"description\": \"Unclear who the operator is; but data is used for training/machine learning.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"We are using the data from the crawler to build datasets for machine learning experiments.\",\n \"operator\": \"Unknown\",\n \"respect\": \"[Yes](https://imho.alex-kunz.com/2024/01/25/an-update-on-friendly-crawler)\",\n },\n \"Google-Extended\": {\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n \"function\": \"LLM training.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.\",\n },\n \"GoogleOther\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Image\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Video\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GPTBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Scrapes data to train OpenAI's products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.\",\n },\n \"iaskspider/2.0\": {\n \"description\": \"Used to provide answers to user queries.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"Crawls sites to provide answers to user queries.\",\n \"operator\": \"iAsk\",\n \"respect\": \"No\",\n },\n \"ICC-Crawler\": {\n \"description\":\n \"Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train and support AI technologies.\",\n \"operator\": \"[NICT](https://nict.go.jp)\",\n \"respect\": \"Yes\",\n },\n \"ImagesiftBot\": {\n \"description\":\n \"Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.\",\n \"frequency\": \"No information.\",\n \"function\":\n \"ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products\",\n \"operator\": \"[ImageSift](https://imagesift.com)\",\n \"respect\": \"[Yes](https://imagesift.com/about)\",\n },\n \"img2dataset\": {\n \"description\": \"Downloads large sets of images into datasets for LLM training or other purposes.\",\n \"frequency\": \"At the discretion of img2dataset users.\",\n \"function\": \"Scrapes images for use in LLMs.\",\n \"operator\": \"[img2dataset](https://github.com/rom1504/img2dataset)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"ISSCyberRiskCrawler\": {\n \"description\": \"Used to train machine learning based models to quantify cyber risk.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train machine learning models.\",\n \"operator\": \"[ISS-Corporate](https://iss-cyber.com)\",\n \"respect\": \"No\",\n },\n \"Kangaroo Bot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Kangaroo Bot is used by the company Kangaroo LLM to download data to train AI models tailored to Australian language and culture. More info can be found at https://darkvisitors.com/agents/agents/kangaroo-bot\",\n },\n \"Meta-ExternalAgent\": {\n \"operator\": \"[Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)\",\n \"respect\": \"Yes.\",\n \"function\": \"Used to train models and improve products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"\\\"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.\\\"\",\n },\n \"Meta-ExternalFetcher\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher\",\n },\n \"OAI-SearchBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"[Yes](https://platform.openai.com/docs/bots)\",\n \"function\": \"Search result generation.\",\n \"frequency\": \"No information.\",\n \"description\": \"Crawls sites to surface as results in SearchGPT.\",\n },\n \"omgili\": {\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/)\",\n \"function\": \"Data is sold.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.\",\n },\n \"omgilibot\": {\n \"description\":\n \"Legacy user agent initially used for Omgili search engine. Unknown if still used, `omgili` agent still used by Webz.io.\",\n \"frequency\": \"No information.\",\n \"function\": \"Data is sold.\",\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)\",\n },\n \"PanguBot\": {\n \"operator\": \"the Chinese company Huawei\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu. More info can be found at https://darkvisitors.com/agents/agents/pangubot\",\n },\n \"PerplexityBot\": {\n \"operator\": \"[Perplexity](https://www.perplexity.ai/)\",\n \"respect\":\n \"[No](https://www.macstories.net/stories/wired-confirms-perplexity-is-bypassing-efforts-by-websites-to-block-its-web-crawler/)\",\n \"function\": \"Used to answer queries at the request of users.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Operated by Perplexity to obtain results in response to user queries.\",\n },\n \"PetalBot\": {\n \"description\": \"Operated by Huawei to provide search and AI assistant services.\",\n \"frequency\": \"No explicit frequency provided.\",\n \"function\": \"Used to provide recommendations in Hauwei assistant and AI search services.\",\n \"operator\": \"[Huawei](https://huawei.com/)\",\n \"respect\": \"Yes\",\n },\n \"Scrapy\": {\n \"description\":\n \"\\\"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for a variety of uses including training AI.\",\n \"operator\": \"[Zyte](https://www.zyte.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"SemrushBot\": {\n \"operator\": \"[Semrush](https://www.semrush.com/)\",\n \"respect\": \"[Yes](https://www.semrush.com/bot/)\",\n \"function\": \"Scrapes data for use in LLM article-writing tool.\",\n \"frequency\": \"Roughly once every 10 seconds.\",\n \"description\":\n \"SemrushBot is a bot which, among other functions, scrapes data for use in ContentShake AI tool reports.\",\n },\n \"Sidetrade indexer bot\": {\n \"description\": \"AI product training.\",\n \"frequency\": \"No information.\",\n \"function\": \"Extracts data for a variety of uses including training AI.\",\n \"operator\": \"[Sidetrade](https://www.sidetrade.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"Timpibot\": {\n \"operator\": \"[Timpi](https://timpi.io)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data for use in training LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Makes data available for training AI models.\",\n },\n \"VelenPublicWebCrawler\": {\n \"description\":\n \"\\\"Our goal with this crawler is to build business datasets and machine learning models to better understand the web.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for business data sets and machine learning models.\",\n \"operator\": \"[Velen Crawler](https://velen.io)\",\n \"respect\": \"[Yes](https://velen.io)\",\n },\n \"Webzio-Extended\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Webzio-Extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models. More info can be found at https://darkvisitors.com/agents/agents/webzio-extended\",\n },\n \"YouBot\": {\n \"operator\": \"[You](https://about.you.com/youchat/)\",\n \"respect\": \"[Yes](https://about.you.com/youbot/)\",\n \"function\": \"Scrapes data for search engine and LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Retrieves data used for You.com web search engine and LLMs.\",\n },\n})",10,334,["D",1738862356323],{"_103":230,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":231,"_144":232,"_154":26,"_105":233,"_146":26},"7c956b08-e4ae-11ef-8c1d-e6cdfca9ef9f",9,333,["D",1738862353732],{"_103":235,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":236,"_144":237,"_154":26,"_105":238,"_146":26},"7c81834a-e4ae-11ef-8c1d-e6cdfca9ef9f",8,332,["D",1738862353603],{"_103":240,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":241,"_144":242,"_154":26,"_105":243,"_146":26},"7c5450a0-e4ae-11ef-8c1d-e6cdfca9ef9f",7,331,["D",1738862353307],{"_103":245,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":246,"_144":247,"_154":26,"_105":248,"_146":26},"7c308b0c-e4ae-11ef-8c1d-e6cdfca9ef9f",6,330,["D",1738862353072],{"_103":250,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":251,"_144":252,"_154":26,"_105":253,"_146":26},"7b9919de-e4ae-11ef-b131-e6cdfca9ef9f",5,329,["D",1738862352079],{"_103":255,"_140":160,"_95":99,"_150":151,"_152":163,"_142":-5,"_143":131,"_100":256,"_144":257,"_154":26,"_105":258,"_146":26},"3bea6eda-da9c-11ef-8603-e6cdfca9ef9f",4,315,["D",1737755002666],{"_103":260,"_140":160,"_95":99,"_150":151,"_152":261,"_142":-5,"_143":131,"_100":262,"_144":263,"_154":26,"_105":264,"_146":26},"3032d780-da9c-11ef-8603-e6cdfca9ef9f","bots = Object.keys({\n \"AI2Bot\": {\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n \"function\": \"Content is used to train open language models.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Explores 'certain domains' to find web content.\",\n },\n \"Ai2Bot-Dolma\": {\n \"description\": \"Explores 'certain domains' to find web content.\",\n \"frequency\": \"No information provided.\",\n \"function\": \"Content is used to train open language models.\",\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n },\n \"Amazonbot\": {\n \"operator\": \"Amazon\",\n \"respect\": \"Yes\",\n \"function\": \"Service improvement and enabling answers for Alexa users.\",\n \"frequency\": \"No information provided.\",\n \"description\":\n \"Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.\",\n },\n \"anthropic-ai\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"Applebot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Search Crawlers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain references to the website. More info can be found at https://darkvisitors.com/agents/agents/applebot\",\n },\n \"Applebot-Extended\": {\n \"operator\": \"[Apple](https://support.apple.com/en-us/119829#datausage)\",\n \"respect\": \"Yes\",\n \"function\": \"Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.\",\n },\n \"Bytespider\": {\n \"operator\": \"ByteDance\",\n \"respect\": \"No\",\n \"function\": \"LLM training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Downloads data to train LLMS, including ChatGPT competitors.\",\n },\n \"CCBot\": {\n \"operator\": \"[Common Crawl Foundation](https://commoncrawl.org)\",\n \"respect\": \"[Yes](https://commoncrawl.org/ccbot)\",\n \"function\": \"Provides open crawl dataset, used for many purposes, including Machine Learning/AI.\",\n \"frequency\": \"Monthly at present.\",\n \"description\":\n \"Web archive going back to 2008. [Cited in thousands of research papers per year](https://commoncrawl.org/research-papers).\",\n },\n \"ChatGPT-User\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Takes action based on user prompts.\",\n \"frequency\": \"Only when prompted by a user.\",\n \"description\": \"Used by plugins in ChatGPT to answer queries based on user input.\",\n },\n \"Claude-Web\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"ClaudeBot\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\":\n \"[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"cohere-ai\": {\n \"operator\": \"[Cohere](https://cohere.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Retrieves data to provide responses to user-initiated prompts.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Retrieves data based on user prompts.\",\n },\n \"cohere-training-data-crawler\": {\n \"operator\":\n \"Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"cohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products. More info can be found at https://darkvisitors.com/agents/agents/cohere-training-data-crawler\",\n },\n \"Crawlspace\": {\n \"operator\": \"[Crawlspace](https://crawlspace.dev)\",\n \"respect\": \"[Yes](https://news.ycombinator.com/item?id=42756654)\",\n \"function\": \"Scrapes data\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Provides crawling services for any purpose, probably including AI model training.\",\n },\n \"Diffbot\": {\n \"operator\": \"[Diffbot](https://www.diffbot.com/)\",\n \"respect\": \"At the discretion of Diffbot users.\",\n \"function\": \"Aggregates structured web data for monitoring and AI model training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.\",\n },\n \"DuckAssistBot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"DuckAssistBot is used by DuckDuckGo's DuckAssist feature to fetch content and generate realtime AI answers to user searches. More info can be found at https://darkvisitors.com/agents/agents/duckassistbot\",\n },\n \"FacebookBot\": {\n \"operator\": \"Meta/Facebook\",\n \"respect\": \"[Yes](https://developers.facebook.com/docs/sharing/bot/)\",\n \"function\": \"Training language models\",\n \"frequency\": \"Up to 1 page per second\",\n \"description\":\n \"Officially used for training Meta \\\"speech recognition technology,\\\" unknown if used to train Meta AI specifically.\",\n },\n \"FriendlyCrawler\": {\n \"description\": \"Unclear who the operator is; but data is used for training/machine learning.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"We are using the data from the crawler to build datasets for machine learning experiments.\",\n \"operator\": \"Unknown\",\n \"respect\": \"[Yes](https://imho.alex-kunz.com/2024/01/25/an-update-on-friendly-crawler)\",\n },\n \"Google-Extended\": {\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n \"function\": \"LLM training.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.\",\n },\n \"GoogleOther\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Image\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Video\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GPTBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Scrapes data to train OpenAI's products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.\",\n },\n \"iaskspider/2.0\": {\n \"description\": \"Used to provide answers to user queries.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"Crawls sites to provide answers to user queries.\",\n \"operator\": \"iAsk\",\n \"respect\": \"No\",\n },\n \"ICC-Crawler\": {\n \"description\":\n \"Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train and support AI technologies.\",\n \"operator\": \"[NICT](https://nict.go.jp)\",\n \"respect\": \"Yes\",\n },\n \"ImagesiftBot\": {\n \"description\":\n \"Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.\",\n \"frequency\": \"No information.\",\n \"function\":\n \"ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products\",\n \"operator\": \"[ImageSift](https://imagesift.com)\",\n \"respect\": \"[Yes](https://imagesift.com/about)\",\n },\n \"img2dataset\": {\n \"description\": \"Downloads large sets of images into datasets for LLM training or other purposes.\",\n \"frequency\": \"At the discretion of img2dataset users.\",\n \"function\": \"Scrapes images for use in LLMs.\",\n \"operator\": \"[img2dataset](https://github.com/rom1504/img2dataset)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"ISSCyberRiskCrawler\": {\n \"description\": \"Used to train machine learning based models to quantify cyber risk.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train machine learning models.\",\n \"operator\": \"[ISS-Corporate](https://iss-cyber.com)\",\n \"respect\": \"No\",\n },\n \"Kangaroo Bot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Kangaroo Bot is used by the company Kangaroo LLM to download data to train AI models tailored to Australian language and culture. More info can be found at https://darkvisitors.com/agents/agents/kangaroo-bot\",\n },\n \"Meta-ExternalAgent\": {\n \"operator\": \"[Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)\",\n \"respect\": \"Yes.\",\n \"function\": \"Used to train models and improve products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"\\\"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.\\\"\",\n },\n \"Meta-ExternalFetcher\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher\",\n },\n \"OAI-SearchBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"[Yes](https://platform.openai.com/docs/bots)\",\n \"function\": \"Search result generation.\",\n \"frequency\": \"No information.\",\n \"description\": \"Crawls sites to surface as results in SearchGPT.\",\n },\n \"omgili\": {\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/)\",\n \"function\": \"Data is sold.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.\",\n },\n \"omgilibot\": {\n \"description\":\n \"Legacy user agent initially used for Omgili search engine. Unknown if still used, `omgili` agent still used by Webz.io.\",\n \"frequency\": \"No information.\",\n \"function\": \"Data is sold.\",\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)\",\n },\n \"PanguBot\": {\n \"operator\": \"the Chinese company Huawei\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu. More info can be found at https://darkvisitors.com/agents/agents/pangubot\",\n },\n \"PerplexityBot\": {\n \"operator\": \"[Perplexity](https://www.perplexity.ai/)\",\n \"respect\":\n \"[No](https://www.macstories.net/stories/wired-confirms-perplexity-is-bypassing-efforts-by-websites-to-block-its-web-crawler/)\",\n \"function\": \"Used to answer queries at the request of users.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Operated by Perplexity to obtain results in response to user queries.\",\n },\n \"PetalBot\": {\n \"description\": \"Operated by Huawei to provide search and AI assistant services.\",\n \"frequency\": \"No explicit frequency provided.\",\n \"function\": \"Used to provide recommendations in Hauwei assistant and AI search services.\",\n \"operator\": \"[Huawei](https://huawei.com/)\",\n \"respect\": \"Yes\",\n },\n \"Scrapy\": {\n \"description\":\n \"\\\"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for a variety of uses including training AI.\",\n \"operator\": \"[Zyte](https://www.zyte.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"SemrushBot\": {\n \"operator\": \"[Semrush](https://www.semrush.com/)\",\n \"respect\": \"[Yes](https://www.semrush.com/bot/)\",\n \"function\": \"Scrapes data for use in LLM article-writing tool.\",\n \"frequency\": \"Roughly once every 10 seconds.\",\n \"description\":\n \"SemrushBot is a bot which, among other functions, scrapes data for use in ContentShake AI tool reports.\",\n },\n \"Sidetrade indexer bot\": {\n \"description\": \"AI product training.\",\n \"frequency\": \"No information.\",\n \"function\": \"Extracts data for a variety of uses including training AI.\",\n \"operator\": \"[Sidetrade](https://www.sidetrade.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"Timpibot\": {\n \"operator\": \"[Timpi](https://timpi.io)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data for use in training LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Makes data available for training AI models.\",\n },\n \"VelenPublicWebCrawler\": {\n \"description\":\n \"\\\"Our goal with this crawler is to build business datasets and machine learning models to better understand the web.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for business data sets and machine learning models.\",\n \"operator\": \"[Velen Crawler](https://velen.io)\",\n \"respect\": \"[Yes](https://velen.io)\",\n },\n \"Webzio-Extended\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Webzio-Extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models. More info can be found at https://darkvisitors.com/agents/agents/webzio-extended\",\n },\n \"YouBot\": {\n \"operator\": \"[You](https://about.you.com/youchat/)\",\n \"respect\": \"[Yes](https://about.you.com/youbot/)\",\n \"function\": \"Scrapes data for search engine and LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Retrieves data used for You.com web search engine and LLMs.\",\n },\n})",3,313,["D",1737754983007],{"_103":266,"_140":160,"_95":267,"_150":151,"_152":261,"_142":-5,"_143":131,"_100":90,"_144":268,"_154":26,"_105":269,"_146":26},"24202dc6-da9c-11ef-aa2d-e6cdfca9ef9f","ingeniousApricotRattlesnake",312,["D",1737754962753],{"_103":271,"_140":160,"_95":267,"_150":151,"_152":272,"_142":-5,"_143":131,"_100":149,"_144":273,"_154":26,"_105":274,"_146":26},"19dd7e9a-da9c-11ef-aa2d-e6cdfca9ef9f","bots = {\n \"AI2Bot\": {\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n \"function\": \"Content is used to train open language models.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Explores 'certain domains' to find web content.\",\n },\n \"Ai2Bot-Dolma\": {\n \"description\": \"Explores 'certain domains' to find web content.\",\n \"frequency\": \"No information provided.\",\n \"function\": \"Content is used to train open language models.\",\n \"operator\": \"[Ai2](https://allenai.org/crawler)\",\n \"respect\": \"Yes\",\n },\n \"Amazonbot\": {\n \"operator\": \"Amazon\",\n \"respect\": \"Yes\",\n \"function\": \"Service improvement and enabling answers for Alexa users.\",\n \"frequency\": \"No information provided.\",\n \"description\":\n \"Includes references to crawled website when surfacing answers via Alexa; does not clearly outline other uses.\",\n },\n \"anthropic-ai\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"Applebot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Search Crawlers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Applebot is a web crawler used by Apple to index search results that allow the Siri AI Assistant to answer user questions. Siri's answers normally contain references to the website. More info can be found at https://darkvisitors.com/agents/agents/applebot\",\n },\n \"Applebot-Extended\": {\n \"operator\": \"[Apple](https://support.apple.com/en-us/119829#datausage)\",\n \"respect\": \"Yes\",\n \"function\": \"Powers features in Siri, Spotlight, Safari, Apple Intelligence, and others.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Apple has a secondary user agent, Applebot-Extended ... [that is] used to train Apple's foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.\",\n },\n \"Bytespider\": {\n \"operator\": \"ByteDance\",\n \"respect\": \"No\",\n \"function\": \"LLM training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Downloads data to train LLMS, including ChatGPT competitors.\",\n },\n \"CCBot\": {\n \"operator\": \"[Common Crawl Foundation](https://commoncrawl.org)\",\n \"respect\": \"[Yes](https://commoncrawl.org/ccbot)\",\n \"function\": \"Provides open crawl dataset, used for many purposes, including Machine Learning/AI.\",\n \"frequency\": \"Monthly at present.\",\n \"description\":\n \"Web archive going back to 2008. [Cited in thousands of research papers per year](https://commoncrawl.org/research-papers).\",\n },\n \"ChatGPT-User\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Takes action based on user prompts.\",\n \"frequency\": \"Only when prompted by a user.\",\n \"description\": \"Used by plugins in ChatGPT to answer queries based on user input.\",\n },\n \"Claude-Web\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"ClaudeBot\": {\n \"operator\": \"[Anthropic](https://www.anthropic.com)\",\n \"respect\":\n \"[Yes](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)\",\n \"function\": \"Scrapes data to train Anthropic's AI products.\",\n \"frequency\": \"No information provided.\",\n \"description\": \"Scrapes data to train LLMs and AI products offered by Anthropic.\",\n },\n \"cohere-ai\": {\n \"operator\": \"[Cohere](https://cohere.com)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Retrieves data to provide responses to user-initiated prompts.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Retrieves data based on user prompts.\",\n },\n \"cohere-training-data-crawler\": {\n \"operator\":\n \"Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"cohere-training-data-crawler is a web crawler operated by Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products. More info can be found at https://darkvisitors.com/agents/agents/cohere-training-data-crawler\",\n },\n \"Crawlspace\": {\n \"operator\": \"[Crawlspace](https://crawlspace.dev)\",\n \"respect\": \"[Yes](https://news.ycombinator.com/item?id=42756654)\",\n \"function\": \"Scrapes data\",\n \"frequency\": \"Unclear at this time.\",\n \"description\": \"Provides crawling services for any purpose, probably including AI model training.\",\n },\n \"Diffbot\": {\n \"operator\": \"[Diffbot](https://www.diffbot.com/)\",\n \"respect\": \"At the discretion of Diffbot users.\",\n \"function\": \"Aggregates structured web data for monitoring and AI model training.\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Diffbot is an application used to parse web pages into structured data; this data is used for monitoring or AI model training.\",\n },\n \"DuckAssistBot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"DuckAssistBot is used by DuckDuckGo's DuckAssist feature to fetch content and generate realtime AI answers to user searches. More info can be found at https://darkvisitors.com/agents/agents/duckassistbot\",\n },\n \"FacebookBot\": {\n \"operator\": \"Meta/Facebook\",\n \"respect\": \"[Yes](https://developers.facebook.com/docs/sharing/bot/)\",\n \"function\": \"Training language models\",\n \"frequency\": \"Up to 1 page per second\",\n \"description\":\n \"Officially used for training Meta \\\"speech recognition technology,\\\" unknown if used to train Meta AI specifically.\",\n },\n \"FriendlyCrawler\": {\n \"description\": \"Unclear who the operator is; but data is used for training/machine learning.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"We are using the data from the crawler to build datasets for machine learning experiments.\",\n \"operator\": \"Unknown\",\n \"respect\": \"[Yes](https://imho.alex-kunz.com/2024/01/25/an-update-on-friendly-crawler)\",\n },\n \"Google-Extended\": {\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n \"function\": \"LLM training.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Used to train Gemini and Vertex AI generative APIs. Does not impact a site's inclusion or ranking in Google Search.\",\n },\n \"GoogleOther\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Image\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GoogleOther-Video\": {\n \"description\":\n \"\\\"Used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data.\",\n \"operator\": \"Google\",\n \"respect\": \"[Yes](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)\",\n },\n \"GPTBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"Yes\",\n \"function\": \"Scrapes data to train OpenAI's products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Data is used to train current and future models, removed paywalled data, PII and data that violates the company's policies.\",\n },\n \"iaskspider/2.0\": {\n \"description\": \"Used to provide answers to user queries.\",\n \"frequency\": \"Unclear at this time.\",\n \"function\": \"Crawls sites to provide answers to user queries.\",\n \"operator\": \"iAsk\",\n \"respect\": \"No\",\n },\n \"ICC-Crawler\": {\n \"description\":\n \"Use the collected data for artificial intelligence technologies; provide data to third parties, including commercial companies; those companies can use the data for their own business.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train and support AI technologies.\",\n \"operator\": \"[NICT](https://nict.go.jp)\",\n \"respect\": \"Yes\",\n },\n \"ImagesiftBot\": {\n \"description\":\n \"Once images and text are downloaded from a webpage, ImageSift analyzes this data from the page and stores the information in an index. Our web intelligence products use this index to enable search and retrieval of similar images.\",\n \"frequency\": \"No information.\",\n \"function\":\n \"ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products\",\n \"operator\": \"[ImageSift](https://imagesift.com)\",\n \"respect\": \"[Yes](https://imagesift.com/about)\",\n },\n \"img2dataset\": {\n \"description\": \"Downloads large sets of images into datasets for LLM training or other purposes.\",\n \"frequency\": \"At the discretion of img2dataset users.\",\n \"function\": \"Scrapes images for use in LLMs.\",\n \"operator\": \"[img2dataset](https://github.com/rom1504/img2dataset)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"ISSCyberRiskCrawler\": {\n \"description\": \"Used to train machine learning based models to quantify cyber risk.\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data to train machine learning models.\",\n \"operator\": \"[ISS-Corporate](https://iss-cyber.com)\",\n \"respect\": \"No\",\n },\n \"Kangaroo Bot\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Kangaroo Bot is used by the company Kangaroo LLM to download data to train AI models tailored to Australian language and culture. More info can be found at https://darkvisitors.com/agents/agents/kangaroo-bot\",\n },\n \"Meta-ExternalAgent\": {\n \"operator\": \"[Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)\",\n \"respect\": \"Yes.\",\n \"function\": \"Used to train models and improve products.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"\\\"The Meta-ExternalAgent crawler crawls the web for use cases such as training AI models or improving products by indexing content directly.\\\"\",\n },\n \"Meta-ExternalFetcher\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Assistants\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Meta-ExternalFetcher is dispatched by Meta AI products in response to user prompts, when they need to fetch an individual links. More info can be found at https://darkvisitors.com/agents/agents/meta-externalfetcher\",\n },\n \"OAI-SearchBot\": {\n \"operator\": \"[OpenAI](https://openai.com)\",\n \"respect\": \"[Yes](https://platform.openai.com/docs/bots)\",\n \"function\": \"Search result generation.\",\n \"frequency\": \"No information.\",\n \"description\": \"Crawls sites to surface as results in SearchGPT.\",\n },\n \"omgili\": {\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/)\",\n \"function\": \"Data is sold.\",\n \"frequency\": \"No information.\",\n \"description\":\n \"Crawls sites for APIs used by Hootsuite, Sprinklr, NetBase, and other companies. Data also sold for research purposes or LLM training.\",\n },\n \"omgilibot\": {\n \"description\":\n \"Legacy user agent initially used for Omgili search engine. Unknown if still used, `omgili` agent still used by Webz.io.\",\n \"frequency\": \"No information.\",\n \"function\": \"Data is sold.\",\n \"operator\": \"[Webz.io](https://webz.io/)\",\n \"respect\": \"[Yes](https://web.archive.org/web/20170704003301/http://omgili.com/Crawler.html)\",\n },\n \"PanguBot\": {\n \"operator\": \"the Chinese company Huawei\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"PanguBot is a web crawler operated by the Chinese company Huawei. It's used to download training data for its multimodal LLM (Large Language Model) called PanGu. More info can be found at https://darkvisitors.com/agents/agents/pangubot\",\n },\n \"PerplexityBot\": {\n \"operator\": \"[Perplexity](https://www.perplexity.ai/)\",\n \"respect\":\n \"[No](https://www.macstories.net/stories/wired-confirms-perplexity-is-bypassing-efforts-by-websites-to-block-its-web-crawler/)\",\n \"function\": \"Used to answer queries at the request of users.\",\n \"frequency\": \"Takes action based on user prompts.\",\n \"description\": \"Operated by Perplexity to obtain results in response to user queries.\",\n },\n \"PetalBot\": {\n \"description\": \"Operated by Huawei to provide search and AI assistant services.\",\n \"frequency\": \"No explicit frequency provided.\",\n \"function\": \"Used to provide recommendations in Hauwei assistant and AI search services.\",\n \"operator\": \"[Huawei](https://huawei.com/)\",\n \"respect\": \"Yes\",\n },\n \"Scrapy\": {\n \"description\":\n \"\\\"AI and machine learning applications often need large amounts of quality data, and web data extraction is a fast, efficient way to build structured data sets.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for a variety of uses including training AI.\",\n \"operator\": \"[Zyte](https://www.zyte.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"SemrushBot\": {\n \"operator\": \"[Semrush](https://www.semrush.com/)\",\n \"respect\": \"[Yes](https://www.semrush.com/bot/)\",\n \"function\": \"Scrapes data for use in LLM article-writing tool.\",\n \"frequency\": \"Roughly once every 10 seconds.\",\n \"description\":\n \"SemrushBot is a bot which, among other functions, scrapes data for use in ContentShake AI tool reports.\",\n },\n \"Sidetrade indexer bot\": {\n \"description\": \"AI product training.\",\n \"frequency\": \"No information.\",\n \"function\": \"Extracts data for a variety of uses including training AI.\",\n \"operator\": \"[Sidetrade](https://www.sidetrade.com)\",\n \"respect\": \"Unclear at this time.\",\n },\n \"Timpibot\": {\n \"operator\": \"[Timpi](https://timpi.io)\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"Scrapes data for use in training LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Makes data available for training AI models.\",\n },\n \"VelenPublicWebCrawler\": {\n \"description\":\n \"\\\"Our goal with this crawler is to build business datasets and machine learning models to better understand the web.\\\"\",\n \"frequency\": \"No information.\",\n \"function\": \"Scrapes data for business data sets and machine learning models.\",\n \"operator\": \"[Velen Crawler](https://velen.io)\",\n \"respect\": \"[Yes](https://velen.io)\",\n },\n \"Webzio-Extended\": {\n \"operator\": \"Unclear at this time.\",\n \"respect\": \"Unclear at this time.\",\n \"function\": \"AI Data Scrapers\",\n \"frequency\": \"Unclear at this time.\",\n \"description\":\n \"Webzio-Extended is a web crawler used by Webz.io to maintain a repository of web crawl data that it sells to other companies, including those using it to train AI models. More info can be found at https://darkvisitors.com/agents/agents/webzio-extended\",\n },\n \"YouBot\": {\n \"operator\": \"[You](https://about.you.com/youchat/)\",\n \"respect\": \"[Yes](https://about.you.com/youbot/)\",\n \"function\": \"Scrapes data for search engine and LLMs.\",\n \"frequency\": \"No information.\",\n \"description\": \"Retrieves data used for You.com web search engine and LLMs.\",\n },\n}",311,["D",1737754945539],{"_103":276,"_140":160,"_95":267,"_150":151,"_152":66,"_142":-5,"_143":131,"_100":31,"_144":277,"_154":26,"_105":278,"_146":26},"13ce0f60-da9c-11ef-8394-e6cdfca9ef9f",310,["D",1737754935375],"versions",[281,283,284,285,286,287,288,289,290,291,292,293],{"_205":221,"_282":223},"previous",{"_205":223,"_282":229},{"_205":229,"_282":234},{"_205":234,"_282":239},{"_205":239,"_282":244},{"_205":244,"_282":249},{"_205":249,"_282":254},{"_205":254,"_282":259},{"_205":259,"_282":265},{"_205":265,"_282":270},{"_205":270,"_282":275},{"_205":275,"_282":-5},"routes/_app","actionData","errors"]

History for bots