Local Scene-Aware Video Processing for LLMs

claude-real-video is a local CLI tool that prepares videos for LLM analysis by extracting scene-based frames and transcribing audio. It optimizes token usage through intelligent pixel-diff deduplication and scene-change detection to ensure only unique, relevant content is processed. By keeping all operations local, it provides a private and flexible alternative to cloud-based video AI services.

Key Points

• Uses scene-aware extraction and density floors instead of fixed-interval sampling to capture meaningful visual changes.
• Employs sliding-window pixel-difference deduplication to ensure repeated shots or static scenes do not waste LLM context.
• Processes all data locally on the user's machine, supporting both local files and various web URLs via yt-dlp.
• Provides comprehensive audio handling by utilizing existing subtitles or generating new ones with Whisper.
• Generates a structured manifest and frame set that can be easily uploaded to any multimodal LLM.

Sentiment

The overall sentiment is mixed but constructive. HN generally agrees that scene-aware preprocessing is a clever and useful way to make video more manageable for multimodal model workflows, but it pushes back on broad claims that this lets an LLM truly watch video. The community is most skeptical about motion inference, token cost, hosted-model privacy, and Claude-specific positioning, while still acknowledging the project as a useful tool in the current gap between ordinary chat interfaces and video-native models.

In Agreement

• Scene-change extraction is a practical improvement over transcript-only or fixed-interval frame sampling because it targets visually meaningful moments.
• The tool fits real workflows where people want to analyze recorded processes, measurements, demonstrations, or generated video without manually selecting frames.
• Producing a manifest with frames and transcript makes the output convenient to feed into different models or local workflows.
• Several commenters have built or used similar pipelines, suggesting the problem is real and the approach is technically relevant.
• Deduplication and contact-sheet-style representations can reduce redundant visual input and help models reason over selected moments.

Opposed

• Keyframes are still images, so the approach can miss motion, timing, object permanence, and subtle animation details that define actual video understanding.
• Claude may be an inefficient or expensive target for this workflow compared with Gemini or local vision-language models designed for multimodal input.
• The privacy framing is too strong if extracted frames are sent to a hosted provider rather than processed entirely locally.
• For structured tasks such as reading gauges or measurements, deterministic computer vision may be simpler and more reliable than sending video-derived images to an LLM.
• The Claude-specific name may mislead users because the tool is broader than one provider and may be useful outside LLM workflows.