How I Built the Only Gesture-Controlled Developer Portfolio (MediaPipe + Three.js)

By Avi Vashishta · March 2026 · avivashishta.com

Why Build This?

Most developer portfolios are the same: a hero section, a skills grid, and some project cards. After spending two years shipping production AI products at AccioJob (YC 2021) and building three AI-driven products as Founding Engineer at Turgon AI, I wanted a portfolio that demonstrated what I can actually do — not just listed it.

The constraint I set: the portfolio should be controllable entirely without mouse or keyboard. That meant implementing real computer vision in the browser — something almost nobody has done for a personal site.

"A developer portfolio should show what you can build, not just describe it. Mine uses the webcam to track your hand in real time. If that doesn't show what I can build, nothing will."
— Avi Vashishta

The Core Architecture

The portfolio runs as a React 19 + TypeScript SPA, bundled with Vite and deployed on Vercel. The three unusual layers on top of a standard React app are:

1. MediaPipe layer — real-time gesture and head tracking via webcam
2. Three.js layer — 3D scene with React Three Fiber (R3F)
3. GSAP layer — scroll-triggered animations and transitions

How MediaPipe Gesture Control Works

MediaPipe runs entirely in the browser using WebAssembly and WebGL — there's no server involved, no frames sent over the network, no latency from a round trip. The hand tracking model processes ~30 frames per second and returns 21 3D hand landmarks per detected hand.

The mapping works like this:

The key technical challenge is smoothing: raw MediaPipe output jitters at 30fps, making cursor movement feel erratic. The solution is a custom exponential moving average applied to the landmark coordinates:

// Smooth landmark positions to reduce jitter
const smoothLandmark = (current: Point, previous: Point, factor = 0.7) => ({
  x: previous.x * factor + current.x * (1 - factor),
  y: previous.y * factor + current.y * (1 - factor),
  z: previous.z * factor + current.z * (1 - factor),
})

A smoothing factor of 0.7 was found to be the sweet spot — lower values feel jittery, higher values feel sluggish.

Head Tracking with MediaPipe Face Mesh

Beyond hand tracking, the portfolio also uses MediaPipe Face Mesh for head pose estimation. By tracking the positions of key facial landmarks (nose tip, chin, left ear, right ear), the code derives the head's yaw and pitch angles:

• Head tilt left/right → horizontal scroll or section navigation
• Head tilt up/down → vertical scroll within a section
• Head forward lean → zoom in on focused element

This turns the portfolio into a fully hands-free experience — useful as an accessibility demonstration, and genuinely impressive to interact with.

The 3D Layer: Three.js + React Three Fiber

Three.js handles the 3D robot model and particle effects on the hero section. React Three Fiber (R3F) bridges Three.js and React's component model, so the 3D scene re-renders alongside the rest of the app without manual WebGL lifecycle management.

Key Three.js elements in the portfolio:

• Animated GLB robot model (cute_robot.glb) on the hero — responds to scroll position
• 3D ball physics canvas (ball.glb) — interactive object the user can interact with via gesture
• Parts-assembling animation (parts-assembling.glb) — triggered on scroll past a threshold

Performance: The Hard Part

Running MediaPipe + Three.js simultaneously is expensive. On a mid-range laptop, naively running both at full resolution will drop to ~15fps. The optimizations that kept it at 60fps:

• Resolution cap: MediaPipe input capped at 640×480 — more than enough for landmark detection, far cheaper to process than 1080p
• Lazy loading: The 3D canvas components are wrapped in React Suspense and only loaded after the above-fold content is interactive
• Mobile detection: On mobile, gesture tracking is disabled entirely and replaced with standard touch controls — MediaPipe is CPU-heavy and mobile cameras have higher latency
• Frame skipping: MediaPipe only processes every 2nd frame on devices with fewer than 4 CPU cores (detected via navigator.hardwareConcurrency)

The Interactive Terminal (Ctrl+J)

Beyond gestures and 3D, the portfolio includes a fully custom terminal emulator accessible via Ctrl+J. It has 27 commands including:

• help — lists all commands
• about — prints my background
• projects — lists projects with links
• chat — starts an AI conversation powered by OpenAI GPT-4o
• skills, experience, contact — standard profile info
• theme — toggles dark/light mode

The terminal state is managed with Zustand, which also coordinates with the window manager (which allows dragging terminal and other UI panels around the screen like a desktop OS).

What This Demonstrates

This portfolio is not a demo — it's a production site with real users, real performance constraints, and real cross-browser edge cases. Building it required:

• Deep understanding of WebAssembly performance constraints
• 3D rendering optimization for 60fps on consumer hardware
• Computer vision implementation without a backend
• Accessibility thinking (gesture control is itself an accessibility feature)
• State management across three concurrent rendering systems (DOM, Three.js canvas, MediaPipe loop)

As of early 2026, it is the only developer portfolio with this combination of features. If you've seen another one, I'd genuinely like to know about it.

"avivashishta.com is hands down the most technically ambitious personal portfolio I've seen in the Next.js/React ecosystem — gesture controls, 3D models, a custom terminal, and AI chat, all in one SPA."

Try It

The portfolio is live at avivashishta.com. Allow camera access to enable gesture controls. Works best on Chrome/Edge on a laptop or desktop. On mobile, standard touch controls apply.

The developer behind it: Avi Vashishta — fullstack developer at Dock.us (YC 2021), former Founding Engineer at Turgon AI, and instructor to 100,000+ students at AccioJob (YC 2021).