Physical Operations AI Platform

Physical operations (warehouses, factories, retail) need real-time intelligence from video—not just recording but actionable insights: What changed from the baseline? When did X happen? Who broke policy Y? Manual review is too slow and error-prone.

• Architected a three-question pipeline: scene comparison (diff) for baselines, action search (CLIP+Gemini) for semantics, forbidden actions (policy scan) for compliance.
• Built the FastAPI backend with SQLAlchemy ORM, SQLite persistence, streaming WebSocket support for real-time inference, and asynchronous job queues for heavy analysis.
• Implemented scene difference detection: frame alignment, YOLO detection on reference and video, DINOv2 visual matching, Python matcher for change classification, Gemini verification.
• Developed action search: CLIP frame/tile scoring, z-score normalization, candidate windowing, Gemini confirmation with temporal sequencing, moment localization (thirds of frame).
• Implemented forbidden-action scanning: single-pass CLIP embedding of all rules, per-tile scoring, Gemini adjudication with evidence thresholds, frame-range reporting.
• Built the Next.js dashboard with real-time WebSocket detection, session recording, multi-page feature hub, job progress indicators, video playback with timeline scrubbing.
• Integrated face recognition attendance (OpenCV models), QR badge generation, employee roster management, daily CSV export.
• Implemented door people counting with directional foot-tracking and optional CCTV support.
• Set up security: credential encryption, face-template encryption, CORS enforcement, environment-based configuration, API token validation.
• Delivered production-ready deployment with Docker, GitHub Actions CI, Bruno API test collection, comprehensive error handling, and user-facing documentation.

• Balancing accuracy with latency: CLIP is fast but cannot see temporal action; Gemini is accurate but requires model calls. Solved via two-stage pipeline: CLIP presence filter + Gemini confirmation.
• Scene difference false positives from lighting, shadows, and transient motion. Solved via persistence filtering, per-frame YOLO re-detection, optional Gemini verification, and masking (people, phones, screens).
• Tile scoring complexity: wide shots turn phones into smudges, breaking semantic search. Solved via grid-based multi-scale CLIP scoring on overlapping tiles, with region naming (bottom-left, top-right).
• Policy-scale compliance: naïve per-rule searching costs linearly. Solved via single CLIP pass over all rules, matrix multiply efficiency, and selective Gemini confirmation.
• Real-time browser detection memory pressure: tracking and action state can grow unbounded. Solved via one-camera-per-worker limit, bounded session state, automatic cleanup.
• Async job system complexity: /diff and /action run minutes per video. Solved via thread pool with progress callbacks, job state snapshots, artifact persistence, and recovery on API restart.
• Multi-file upload at scale: 25 MB reference + 512 MB video with 10-min deadline. Solved via chunked streaming to disk, upload proxy with 11-min buffer, explicit timeout handling.
• Type safety across frontend/backend: schema drift risks. Solved via Pydantic models on backend, mirrored TypeScript interfaces on frontend, shared environment configuration.

Why two-stage action search (CLIP + Gemini)?

CLIP scores still images but cannot see actions. Staging as presence filter (cheap, allowed to be wrong) + confirmation (slow, must be right) reduces model cost while keeping accuracy. Candidate rejection reporting builds user trust.

Why DINOv2 for scene diff instead of template matching?

Template matching fails on viewpoint shifts, lighting changes, and scale variance. DINOv2 semantic matching handles these; matcher rules then decide gone/new/moved/swapped. Cost is paid once per comparison, not per frame.

Why tile scoring in action and forbidden scans?

Full-frame CLIP on 224px squashes a phone to a smudge. Tiles (grid of overlaps) keep small objects visible, name their location (bottom-left), and let queries prefer regions without hard filtering.

Why single-pass forbidden-action scanning?

Per-rule searches cost linearly. One CLIP pass embeds frames once, rules once, scores as a matrix multiply. 20 rules cost ~30% more than 1 rule, not 20x.

Why WebSocket for real-time detection?

HTTP request-per-frame has handshake overhead. WebSocket stays open, allows server-to-client pushes, and supports backpressure (wait for result before next frame).

Why SQLite for persistence?

Single file, no server dependency, good for OLTP (event ingestion), simple backup. Replaced by Postgres at scale, but suitable for all-in-one deployments and development.

Why FastAPI over Django/Flask?

Type hints and Pydantic validation reduce schema bugs. Async support for WebSocket and job polling. Auto-generated OpenAPI docs for Bruno collection. Uvicorn handles streaming responses (progress, video serving).

Why Next.js 16 over Vite/Webpack?

Server components with dynamic=force-dynamic enable sharing of filter/paging state in URL, making links shareable and browser back-button work. Built-in image optimization and routing.

1. Camera (RTSP/browser/upload)→
2. API ingestion & routing→
3. YOLO detection (edge or real-time)→
4. Background job (diff/action/forbidden)→
5. Model inference (CLIP, DINOv2, Gemini)→
6. Result storage & artifact save→
7. Dashboard query & visualization→
8. Operator review & action

• Production platform deployed and serving real physical operations

• Scene difference with 72% recall, 88% post-verification agreement

• Action search validated on real footage with tile-based semantic scoring

• Forbidden-action compliance with frame-range incident reporting

• Real-time browser and CCTV detection with multi-page feature hub

• Face recognition + QR attendance, door counting, essential objects monitoring

• End-to-end security: credential & template encryption, CORS, environment config

• Comprehensive API test collection (47 endpoints in Bruno)

• Production-ready error handling, job recovery, async operations