Building Mneme: A Local Visual Memory Server for Photos and Videos
When we talk about "AI Memory," we usually mean text. We index Chat histories, PDFs, and codebase repositories using vector embeddings. But what about visual memory? If an agent is tasked with editing a video or finding a specific photo, text embeddings are useless.
To solve this, I built Mneme (named after the Greek muse of memory)—a local visual memory server that runs on my own hardware.
The Architecture of Visual Memory
Mneme is essentially an automated ingestion pipeline combined with a multimodal embedding database. When I drop a folder of RAW photos or MP4 videos into a watched directory, Mneme goes to work:
- Extraction: For videos, it uses FFmpeg to extract one frame every second.
- Captioning (Optional): It uses a small, fast local Vision Language Model (VLM) like LLaVA to generate a brief text description of the frame.
- Multimodal Embedding: It uses a model like CLIP to generate a vector embedding directly from the image pixels.
- Storage: It stores the metadata, the text embedding, and the image embedding in a local LanceDB instance.
Why Local?
You could theoretically build this using the OpenAI API. But if you have 50,000 personal photos and 2 terabytes of video footage, uploading all of that to a cloud provider is a non-starter for both privacy and cost reasons.
By running Mneme locally, the data never leaves the machine.
Querying the Server
Because Mneme uses multimodal embeddings (CLIP), you can search the database using either text or an image.
If I query: "Find pictures of my dog running in the snow," the system embeds that text string into the same vector space as the images. It finds the closest pixel-embeddings and returns the file paths.
Even more powerfully, if I provide an image of a specific BMX trick, I can query: "Find me video clips where this exact pose occurs."
Mneme exposes this entire database to my agentic workflows via an MCP server, allowing my AI assistants to 'see' my entire digital archive on command.