02 · Interactive Experiment
Hand-Tracking AR Experience
Real-time computer vision meets 3D graphics, directly in the browser.
Overview
A browser-based augmented reality experience enabling touchless interaction through real-time hand gesture recognition, rendered without native app dependencies.
Challenge
The problem
Most AR experiences require a native app, external hardware, or heavy SDKs. The goal was to prove that expressive, gesture-driven 3D interaction could run directly inside a browser tab on a modern laptop.
Approach
How it was built
MediaPipe's hand-landmark model was paired with a Three.js scene, webcam frames streamed through WebRTC, 21 hand keypoints mapped onto a WebGL overlay, and the render loop tuned so pose updates did not block painting.
Process & features
- 01Real-time hand landmark detection
- 0221 keypoints mapped to a WebGL overlay
- 03Webcam frames streamed via WebRTC
- 04Render loop tuned so pose updates do not block painting
Technology
Outcome
Gesture-driven 3D interaction running in a browser tab, no install, no native app.
60 FPS
Render loop
0
Native installs
21
Hand keypoints
Next project