2026/AI/Completed
RealTag
Tag any object in a live camera feed — and find it again later.
Hybrid ML app: React Native Vision Camera client + FastAPI server. YOLO11 detects, SAM2 segments (auto or tap-to-mask), DINOv2 embeds crops so the same physical object can be re-identified across frames, angles, and lighting — with custom tags stored on-device.
- Detect → segment → embed pipeline
- Tap-to-segment + auto masks
- Embedding re-ID across frames
The Problem
Detecting an object once in a camera frame is a solved problem. Recognizing that same physical object again — across frames, angles, and lighting — and attaching your own name and notes to it is a harder re-identification problem. It needed to feel like a live camera app, not a research notebook.
What I Built
RealTag pairs a React Native Vision Camera client with a LAN FastAPI inference server:
- Detect — YOLO11 finds objects in periodic JPEG snapshots from the live feed.
- Segment — SAM2 produces masks automatically or via tap-to-segment at a point.
- Tag — name, notes, and color on the selected object; crop embeddings from DINOv2.
- Re-identify — later frames match embeddings (similarity threshold) so the same object keeps its tag.
The client keeps overlays stable between polls with IoU tracking, draws with Skia, and persists tags locally with MMKV. Case study visuals are captures from the live hybrid app + server — auto detection with masks, manual SAM2, tagging, and saved-tag state.
Major Features
- Full-screen live camera — continuous detect loop against a local ML server.
- Auto + tap-to-segment — YOLO proposals and SAM2 point prompts for precise masks.
- Custom tags — not a product catalog; you name what matters in your scene.
- Embedding re-ID — DINOv2 crop vectors match the same instance later.
- Stable overlays — client-side IoU tracking between server polls so boxes don’t flicker.
Key Challenges
Chaining three models — detection, segmentation, embedding — fast enough for a live feed meant cropping to detected regions before the heavier SAM2 and DINOv2 stages, rather than running them on full frames. Keeping the FastAPI server responsive under a continuous phone stream was the other core constraint. Earlier on-device / Plant Tracker experiments informed the eventual hybrid LAN design.
Results
RealTag runs as an end-to-end hybrid ML pipeline: real-time detection and segmentation on device-streamed frames, custom tagging, and embedding-based re-identification across sessions.
Stack Decisions
YOLO11, SAM2, and DINOv2 each solve a distinct part of the problem — fast boxes, precise boundaries, stable embeddings — and no single off-the-shelf model handles all three well. FastAPI keeps PyTorch serving overhead low on a LAN; Vision Camera + Skia keep the mobile UX at full-screen camera fidelity.
Screens
