/** * Gemini AI Object Detection and Bounding Box Visualizer * * This application uses Google's Gemini 1.5 Pro AI model to perform object detection * on user-uploaded images and visualize the results by drawing bounding boxes. * * Key features: * - Integrates with Google's Generative AI API * - Processes user-uploaded images for object detection * - Visualizes detected objects with bounding boxes * - Allows customization of detection parameters (grid, contrast) * - Stores API key securely in browser's local storage * * The app is built using Hono.js and can be deployed on Val Town or run locally with Deno. * It's an adaptation of Simon Willison's original concept, expanded for web use. * * Source: https://www.val.town/v/yawnxyz/geminiBbox/ * Original inspiration: https://github.com/simonw/tools/blob/main/gemini-bbox.html */ import { Hono } from "https://deno.land/x/hono@v3.11.7/mod.ts"; import { cors } from "https://deno.land/x/hono@v3.11.7/middleware.ts"; import { marked } from "npm:marked"; const intro = `

Gemini AI Bounding Box Demo

This application visualizes object detection results by drawing bounding boxes on images using the Google's Gemini 1.5 Pro AI model.

API keys are only stored in your browser's local storage.

Images are only sent to Google's Gemini API servers for processing.

Try the following images (click the links to add them):

`; const markdownJsonRegex = /```json\n([\s\S]*?)```/; export const module = ` `; export const html = ` Gemini AI Bounding Box Demo ${module}

${intro}

Upload an image

Or enter an image URL:

Get an image description

Sometimes getting a description of the image helps you understand what the AI sees. This can help you better set up the prompt and settings.

Detect Bounding Boxes

Use this to detect bounding boxes of objects in an image. Use the image description to help with setting up the prompt. Rows and columns determine how many tiles the image will be divided into — for simple images, 1x1 is best.

For the free API you might get the "Resource has been exhausted" error if you make too many requests too quickly.

Apply High Contrast Filter

Rows:

Columns:

Delay between requests (ms):

Grid Increment:

Credits

This experiment uses a lot of ideas and code from Simon Willison's original post which you can read here: https://simonwillison.net/2024/Aug/26/gemini-bounding-box-visualization/

Source: https://www.val.town/v/yawnxyz/geminiBbox/
Original: https://github.com/simonw/tools/blob/main/gemini-bbox.html

A labspace.ai experiment

`; const app = new Hono(); app.get("/", (c) => c.html(html)); // Add the CORS proxy endpoint app.use('/proxy/*', cors()); app.get('/proxy/*', async (c) => { console.log("[proxy request]", c.req.url) const url = decodeURIComponent(c.req.url.split('/proxy/')[1]); if (!url) { return c.text('No URL provided', 400); } try { const response = await fetch(url, { headers: { 'Accept': 'image/*', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' } }); if (!response.ok) { console.error(`Proxy error for ${url}:`, { status: response.status, statusText: response.statusText, headers: Object.fromEntries(response.headers.entries()) }); return c.text(`Failed to fetch: ${response.statusText}`, response.status); } const contentType = response.headers.get('content-type'); const blob = await response.blob(); return new Response(blob, { headers: { 'content-type': contentType || 'application/octet-stream', 'access-control-allow-origin': '*' } }); } catch (error) { return c.text(`Proxy error: ${error.message}`, 500); } }); export default (typeof Deno !== "undefined" && Deno.env.get("valtown")) ? app.fetch : app;

geminiBbox

Versions

v0