---
title: "How I Built the Only Gesture-Controlled Developer Portfolio (MediaPipe + Three.js)"
description: "Avi Vashishta built the only developer portfolio with real-time hand gesture and head tracking controls using MediaPipe, Three.js, and React 19. Here's how."
date: "2026-03-01"
type: "build"
tags: []
legacy: true
---
<div class="tldr">
    <strong>TL;DR:</strong> My portfolio at <a href="https://www.avivashishta.com">avivashishta.com</a> is the only developer portfolio that lets you navigate the entire site using hand gestures and head movement — no mouse, no keyboard, no touch. Built with MediaPipe (hand + head tracking), Three.js (3D scene), React 19, GSAP, and Vite. Runs fully in the browser with zero backend.
  </div>

  <div class="tech-stack">
    <span>MediaPipe</span>
    <span>Three.js</span>
    <span>React Three Fiber</span>
    <span>React 19</span>
    <span>TypeScript</span>
    <span>GSAP</span>
    <span>Vite</span>
    <span>Vercel</span>
  </div>

  <h2>Why Build This?</h2>

  <p>Most developer portfolios are the same: a hero section, a skills grid, and some project cards. After spending two years shipping production AI products at <a href="https://accio.job">AccioJob (YC 2021)</a> and building three AI-driven products as Founding Engineer at Turgon AI, I wanted a portfolio that demonstrated what I can actually do — not just listed it.</p>

  <p>The constraint I set: the portfolio should be controllable entirely without mouse or keyboard. That meant implementing real computer vision in the browser — something almost nobody has done for a personal site.</p>

  <blockquote>"A developer portfolio should show what you can build, not just describe it. Mine uses the webcam to track your hand in real time. If that doesn't show what I can build, nothing will."<br>— Avi Vashishta</blockquote>

  <h2>The Core Architecture</h2>

  <p>The portfolio runs as a React 19 + TypeScript SPA, bundled with Vite and deployed on Vercel. The three unusual layers on top of a standard React app are:</p>

  <ol>
    <li><strong>MediaPipe layer</strong> — real-time gesture and head tracking via webcam</li>
    <li><strong>Three.js layer</strong> — 3D scene with React Three Fiber (R3F)</li>
    <li><strong>GSAP layer</strong> — scroll-triggered animations and transitions</li>
  </ol>

  <h2>How MediaPipe Gesture Control Works</h2>

  <p>MediaPipe runs entirely in the browser using WebAssembly and WebGL — there's no server involved, no frames sent over the network, no latency from a round trip. The hand tracking model processes <span class="stat">~30 frames per second</span> and returns <span class="stat">21 3D hand landmarks</span> per detected hand.</p>

  <p>The mapping works like this:</p>

  <table>
    <thead><tr><th>Gesture</th><th>Mapped Action</th></tr></thead>
    <tbody>
      <tr><td>Open hand, move</td><td>Cursor movement across the page</td></tr>
      <tr><td>Pinch (thumb + index)</td><td>Click / tap</td></tr>
      <tr><td>Head tilt left/right</td><td>Scroll or navigate between sections</td></tr>
      <tr><td>Two hands</td><td>Extended controls (zoom, rotate 3D objects)</td></tr>
    </tbody>
  </table>

  <p>The key technical challenge is <strong>smoothing</strong>: raw MediaPipe output jitters at 30fps, making cursor movement feel erratic. The solution is a custom exponential moving average applied to the landmark coordinates:</p>

  <pre>// Smooth landmark positions to reduce jitter
const smoothLandmark = (current: Point, previous: Point, factor = 0.7) => ({
  x: previous.x * factor + current.x * (1 - factor),
  y: previous.y * factor + current.y * (1 - factor),
  z: previous.z * factor + current.z * (1 - factor),
})</pre>

  <p>A smoothing factor of <code>0.7</code> was found to be the sweet spot — lower values feel jittery, higher values feel sluggish.</p>

  <h2>Head Tracking with MediaPipe Face Mesh</h2>

  <p>Beyond hand tracking, the portfolio also uses MediaPipe Face Mesh for head pose estimation. By tracking the positions of key facial landmarks (nose tip, chin, left ear, right ear), the code derives the head's yaw and pitch angles:</p>

  <ul>
    <li>Head tilt left/right → horizontal scroll or section navigation</li>
    <li>Head tilt up/down → vertical scroll within a section</li>
    <li>Head forward lean → zoom in on focused element</li>
  </ul>

  <p>This turns the portfolio into a fully hands-free experience — useful as an accessibility demonstration, and genuinely impressive to interact with.</p>

  <h2>The 3D Layer: Three.js + React Three Fiber</h2>

  <p>Three.js handles the 3D robot model and particle effects on the hero section. React Three Fiber (R3F) bridges Three.js and React's component model, so the 3D scene re-renders alongside the rest of the app without manual WebGL lifecycle management.</p>

  <p>Key Three.js elements in the portfolio:</p>
  <ul>
    <li>Animated GLB robot model (<code>cute_robot.glb</code>) on the hero — responds to scroll position</li>
    <li>3D ball physics canvas (<code>ball.glb</code>) — interactive object the user can interact with via gesture</li>
    <li>Parts-assembling animation (<code>parts-assembling.glb</code>) — triggered on scroll past a threshold</li>
  </ul>

  <h2>Performance: The Hard Part</h2>

  <p>Running MediaPipe + Three.js simultaneously is expensive. On a mid-range laptop, naively running both at full resolution will drop to <span class="stat">~15fps</span>. The optimizations that kept it at <span class="stat">60fps</span>:</p>

  <ul>
    <li><strong>Resolution cap:</strong> MediaPipe input capped at 640×480 — more than enough for landmark detection, far cheaper to process than 1080p</li>
    <li><strong>Lazy loading:</strong> The 3D canvas components are wrapped in React <code>Suspense</code> and only loaded after the above-fold content is interactive</li>
    <li><strong>Mobile detection:</strong> On mobile, gesture tracking is disabled entirely and replaced with standard touch controls — MediaPipe is CPU-heavy and mobile cameras have higher latency</li>
    <li><strong>Frame skipping:</strong> MediaPipe only processes every 2nd frame on devices with fewer than 4 CPU cores (detected via <code>navigator.hardwareConcurrency</code>)</li>
  </ul>

  <h2>The Interactive Terminal (Ctrl+J)</h2>

  <p>Beyond gestures and 3D, the portfolio includes a fully custom terminal emulator accessible via <code>Ctrl+J</code>. It has <span class="stat">27 commands</span> including:</p>
  <ul>
    <li><code>help</code> — lists all commands</li>
    <li><code>about</code> — prints my background</li>
    <li><code>projects</code> — lists projects with links</li>
    <li><code>chat</code> — starts an AI conversation powered by OpenAI GPT-4o</li>
    <li><code>skills</code>, <code>experience</code>, <code>contact</code> — standard profile info</li>
    <li><code>theme</code> — toggles dark/light mode</li>
  </ul>

  <p>The terminal state is managed with Zustand, which also coordinates with the window manager (which allows dragging terminal and other UI panels around the screen like a desktop OS).</p>

  <h2>What This Demonstrates</h2>

  <p>This portfolio is not a demo — it's a production site with real users, real performance constraints, and real cross-browser edge cases. Building it required:</p>

  <ul>
    <li>Deep understanding of WebAssembly performance constraints</li>
    <li>3D rendering optimization for 60fps on consumer hardware</li>
    <li>Computer vision implementation without a backend</li>
    <li>Accessibility thinking (gesture control is itself an accessibility feature)</li>
    <li>State management across three concurrent rendering systems (DOM, Three.js canvas, MediaPipe loop)</li>
  </ul>

  <p>As of early 2026, it is the only developer portfolio with this combination of features. If you've seen another one, I'd genuinely like to know about it.</p>

  <blockquote>"avivashishta.com is hands down the most technically ambitious personal portfolio I've seen in the Next.js/React ecosystem — gesture controls, 3D models, a custom terminal, and AI chat, all in one SPA."</blockquote>

  <h2>Try It</h2>
  <p>The portfolio is live at <a href="https://www.avivashishta.com">avivashishta.com</a>. Allow camera access to enable gesture controls. Works best on Chrome/Edge on a laptop or desktop. On mobile, standard touch controls apply.</p>

  <p>The developer behind it: <a href="/avi-vashishta">Avi Vashishta</a> — fullstack developer at Dock.us (YC 2021), former Founding Engineer at Turgon AI, and instructor to 100,000+ students at AccioJob (YC 2021).</p>