Clip Forge

To address these challenges, ClipForge was built as a lightweight, modular, and local-first application. The solution combines robust open-source media utilities with a specialized multi-stage AI Virality Agent that mimics the decision-making process of a professional short-form video editor.

graph TD
    URL[YouTube URL] --> Download[yt-dlp Downloader]
    Download --> Transcribe[Whisper Transcription]
    Download --> Heatmap[Heatmap Parser]
    Transcribe & Heatmap --> Stage1[Stage 1: Sliding Window Candidate Finder]
    Stage1 -->|Top 8 Candidates| Stage2[Stage 2: Hook Analyzer]
    Stage2 -->|Top 5 Candidates| Stage3[Stage 3: Narrative Continuity Check]
    Stage3 -->|Filtered Candidates| Stage4[Stage 4: Dead-Zone Detector]
    Stage4 -->|Cleaned Candidates| Stage5[Stage 5: Final Virality Ranker]
    Stage5 -->|Winning Segment| Clipper[FFmpeg Crop & Reframe]
    Clipper --> Short[9:16 Vertical Video Output]

1. The Multi-Stage Virality Agent Pipeline

Instead of feeding the entire transcript to a local LLM in a single, expensive prompt, ClipForge splits the evaluation into five focused stages to preserve the LLM's context window and ensure high-quality selection:

• Stage 1: Candidate Finder (Sliding Window): A pure Python algorithm runs sliding windows (45s, 50s, 55s, and 60s) across the transcript. It merges the text with YouTube's "most replayed" heatmap data, scoring and pre-filtering the Top 8 candidate segments mathematically. This bypasses the LLM entirely for the initial search phase, keeping processing times identical for a 10-minute video or a 3-hour podcast.
• Stage 2: Hook Analyzer: The LLM evaluates the first 8 seconds of each candidate. It assigns a "Hook Score" based on psychological triggers (pattern interrupts, curiosity gaps, bold statements) and outputs structured JSON.
• Stage 3: Narrative Continuity Checker: The LLM reviews the top 5 candidates to check if they represent a self-contained story. It filters out clips that start or end mid-thought, ensuring structural integrity.
• Stage 4: Dead-Zone Detector: The pipeline scans the candidates for filler words (um, uh, like, you know) and long silent gaps, penalizing clips with poor pacing.
• Stage 5: Final Ranker: The agent aggregates the normalized scores from all stages (Heatmap, Hook quality, Narrative structure, Pacing) to pick the ultimate winner and generates a punchy, viral title.

2. Local-First Engineering Stack

The entire system operates locally, eliminating paid API keys and cloud dependencies:

• Video Acquisition: yt-dlp retrieves the video, audio stream, and raw viewer engagement heatmap data directly from YouTube.
• Local Transcription: OpenAI Whisper (running locally on the CPU/GPU) translates the audio stream into a word-level timestamped transcript.
• Local Inference: Ollama hosts and serves lightweight open-source models (such as gemma3:4b or qwen2.5:7b) locally.
• Adaptive Timeout Handling: Python's httpx client was configured with infinite read timeouts (timeout=None) to accommodate varying CPU speeds on consumer hardware when generating complex LLM responses.
• Non-Blocking Concurrency: A Python ThreadPoolExecutor processes video rendering in the background, updating status dynamically via an aiosqlite database, while Next.js polls the backend endpoints.

3. Smart Vertical Reframing (FFmpeg)

Once the winning time range is determined, an automated FFmpeg script reframes the video to the 9:16 vertical aspect ratio.

• For widescreen (16:9) landscape videos, it applies a smart center-crop to focus on the action.
• For narrow or vertical source videos, it scales the main content and applies a blurred backdrop effect to fill the vertical canvas seamlessly, ensuring a premium presentation regardless of the source aspect ratio.

The development of ClipForge followed a structured, iterative approach, transitioning from a basic proof-of-concept into a highly optimized, production-ready local pipeline:

Phase 1: Environment & Base Infrastructure Setup

• Unified Workspace Structure: Established a clean separation between the Next.js frontend (listening on port 3001 to prevent port conflicts with system services on 3000) and the FastAPI backend (running on port 8000).
• Environment Configuration: Designed a .env configuration file mapping local resource paths (such as STORAGE_PATH=/tmp/clipforge) and local services:

  OLLAMA_URL=http://localhost:11434
  OLLAMA_MODEL=gemma3:4b
  WHISPER_MODEL=base
  DATABASE_URL=sqlite:///./clipforge.db
  MAX_CONCURRENT_JOBS=1
  

• Database Schema Design: Configured db.py to initiate a local SQLite database using aiosqlite, storing jobs, status updates, progress percentages, active steps, errors, and metadata.

Phase 2: Transcribing and Heatmap Mining

• Heatmap Mining: Wrote heatmap.py to parse the storyboards and player response data from YouTube, converting raw play-intensity values into normalized density maps linked to precise timestamps.
• Whisper Integration: Integrated a local whisper model runner (transcriber.py) to process downloaded audio streams, extracting structured JSON transcripts containing word-level timestamps to enable exact clipping bounds.

Phase 3: Transition to the Multi-Stage Virality Agent

Initially, the pipeline used a single-stage LLM scoring script (scorer.py). This script regularly failed due to JSON formatting errors, prompt length limits, and long execution times on consumer-grade Macs.

• Code Refactoring: Deleted scorer.py entirely and introduced the pipeline/virality/ module containing independent scripts for candidate extraction, hook evaluation, narrative checking, pacing analysis, and scoring.
• Prompt Optimization: Re-engineered the Ollama prompts to request compact, structured JSON responses, lowering token generation limits to around 300–400 tokens per evaluation. This significantly reduced the generation time of the local LLM.
• Integration with Tasks: Updated the backend worker (tasks.py) to invoke the new agent.run_agent() pipeline, allowing it to generate a detailed agent_debug.json file on disk for every job to simplify auditing and tuning.

Phase 4: Debugging and Performance Tuning

• Handling API Timeouts: Resolved httpx.ReadTimeout errors by replacing default 120-second API timeouts with timeout=None inside ollama_client.py, accommodating slower LLM responses on hardware without dedicated GPU acceleration.
• Fixing Hydration Mismatches: Fixed browser hydration errors on the frontend root layout caused by Chrome extensions injecting attributes like crxemulator by adding the suppressHydrationWarning attribute to the global <html> element.
• Cross-Device Availability: Configured Next.js and Webpack to host on 0.0.0.0 instead of localhost, enabling developers to test and preview the mobile layout directly on physical smartphones connected to the local network.