Turning Yourself Into Spider-Man in React (MediaPipe Pose → a Mixamo Rig in React Three Fiber)

Published August 2026 · avivashishta.com

Overview

Step out of frame and press a key. Three-second countdown, and the app keeps a still photo of your empty room. Now walk back in — and you are not there. Spider-Man is standing exactly where you were, at your height, turning when you turn, tilting his head when you tilt yours, curling his fingers when you curl yours. Move fast and the suit moves fast. Throw a punch at the lens and the arm foreshortens at you.

No green screen. No depth camera. No suit with markers on it. One webcam, two MediaPipe models, and a rigged GLB, all running in a React 19 tab at 60 fps.

The first wall: you cannot animate a Sketchfab embed

The character came from a Sketchfab model, and my first instinct was to drive the embed directly. That does not work, and it is worth knowing why before you waste an afternoon on it.

The Sketchfab Viewer API gives you a scene graph, node transforms (translate, rotate, setMatrix) and playback control over animation tracks that were baked into the file. It gives you no access whatsoever to skeleton joints, and the docs are explicit that only MatrixTransform nodes can be moved. You can play an animation somebody else authored. You cannot pose a rig.

So the model has to come down as a GLB and get loaded yourself. Which introduces the second wall.

368 MB of Spider-Man

The good version of the model was 368 MB. Not geometry — 632,794 triangles is about 40 MB of buffers. The other 328 MB was thirty PNG textures at 4096×4096, several of them over 20 MB each.

Two passes fixed it.

Textures. Decode every image out of the GLB's binary chunk, resize to 1024, re-encode as JPEG at quality 88 (WebP only where an alpha channel is actually in use), then rebuild the buffer with recomputed byteOffsets. 328 MB of PNG becomes about 5 MB. For a character that occupies maybe 600 px of screen height, 4K albedo maps are pure waste.

Geometry. glTF-Transform does the rest:

368 MB → 14 MB, same rig, same triangle count.

One thing to watch: meshopt re-partitioned the single shared skin into one skin per mesh (1 → 16). That is harmless — every skin still references the same bone nodes, so posing a bone moves all of them — but it looks alarming in an inspector. Loading it back needs the decoder wired up:

Reading the body: two landmarkers, one video element

@mediapipe/tasks-vision is already a dependency of this portfolio for its gesture controls, so the setup is familiar. Both models run in VIDEO mode against the same <video> element, on the same monotonically increasing timestamp:

Always wrap that in a try and retry with delegate: 'CPU'. The GPU delegate fails on more machines than you would like, and the failure is a rejected promise, not a degraded result.

Pose gives you two arrays that matter. landmarks are normalised image coordinates — where the joint is on screen. Pose also provides worldLandmarks: metres, origin between the hips, axes parallel to the image. Those two get used for completely different jobs, and mixing them up is the root of a lot of bad mocap:

• worldLandmarks drive the rig. Rotations need undistorted 3-D directions.
• landmarks drive the layout. Where on screen he stands and how big he is are 2-D screen questions.

MediaPipe's axes are x-right, y-down, z-negative-toward-camera, so everything gets mapped once on the way in and never thought about again:

The retargeting core: aim, don't copy

The temptation is to place bones at landmark positions. Do not. Your arm and Spider-Man's arm are different lengths, and a skinned mesh with joints yanked to arbitrary positions tears. Bone rotations are the only thing you should ever write.

So the primitive is: rotate this bone so the direction toward its child points along the direction I measured on the human. Everything below is one function, applied down the hierarchy, parents before children.

Two details in there earn their keep. The weight lets a limb fade back to its rest pose when tracking confidence drops, instead of snapping or flailing. And after applying, you must record the bone's actual resulting world rotation — not the target you asked for — because smoothing means the two differ, and the children solve against where the parent really ended up.

The whole body is then a table:

Turning around needs frames, not directions

A direction has no twist. Aim the spine bones from hips to shoulders and the torso will bend correctly and never turn — the whole upper body just inherits its facing from the pelvis. Which is why the first version stared straight at the camera no matter which way I sat.

The torso needs full three-axis frames. Build one from a lateral axis and an up axis, compare it against the same frame in the rig's rest pose, and you have an absolute orientation:

Then spread the twist between them along the spine instead of snapping it at one joint. Because both are corrections applied to a rest orientation, you can interpolate the deltas directly:

The part nobody warns you about: if you sit at a desk, your hips are behind it. MediaPipe still reports hip landmarks, but they're guesses, and a pelvis frame built on guesses does not rotate. So the pelvis borrows the shoulder line as its lateral axis in proportion to how little the hips can be trusted:

The head gets the same treatment, with its lateral axis from your ears and its up axis from the cross product of ear-line and nose direction. As you turn far enough that your face leaves view, the head's frame fades back to simply following the chest — otherwise it snaps the moment the face landmarks go.

Tested against a synthetic body rotated about its axis: at 30°, 60° and −45°, pelvis, chest, head and arms all follow within 0.1°. Repeated with hip confidence dropped to 0.1 to simulate the desk — still a full 60° turn.

Matching your size: least squares, not height

Getting him to be your size is the part that looks trivial and is not. My first attempt measured your on-screen height head-to-heel, scaled the model to match, and pinned his hips to your hips.

It was visibly wrong, and it jumped.

Wrong, because the proportions of a heroically-built character are not yours — match the total height and everything in between lands slightly off. Jumpy, because when your legs leave the frame MediaPipe keeps reporting hips, knees and ankles as extrapolated guesses somewhere below the bottom edge, and the fit stretches him to reach them. As those confidences flicker, the size pops.

The right tool is a weighted least-squares similarity fit: given the already-posed rig, find the single uniform scale s and 2-D offset t that best land his joints on yours. Closed form, no solver:

Four things make it behave:

1. Reject off-frame joints outright. A landmark outside the viewport is a guess. It contributes to neither the fit nor the pose.
2. Weight by trust, not equally. Shoulders and hips are rigid and well tracked (1.0), elbows and knees less (0.6), wrists and ankles least (0.35).
3. One robust pass. Compute residuals, drop anything beyond ~2.5× the median, re-solve. One mis-tracked limb should not be allowed to resize a whole character.
4. Rate-limit the scale, not the position. Scale is clamped to 3% change per frame with a slow filter on top; translation stays fully responsive. Size should drift, position should snap.

Ship a lock button too. Auto-fit is the right default, but the moment it looks right the honest feature is a key that freezes the scale and lets position keep tracking.

Fingers: solve in the hand's own frame

This is the bug I would most like to save you. Fingers were tracking, and every hand came out as the same claw regardless of what my hands did.

The cause: I assumed HandLandmarker's worldLandmarks live in the same camera-aligned space as the pose model's, and aimed finger bones straight at those vectors. They do not. The hand model reports its 21 points in its own frame, so every finger was being aimed at a fixed direction that had nothing to do with my hand.

The fix is to stop caring what frame the tracker used. Build a palm frame from the tracked hand itself, express each finger direction in that frame, then push it back out through the wrist's actual posed orientation. Only the shape of the hand survives the round trip:

The wrist itself is oriented from the body model's wrist, index and pinky landmarks, which are camera-aligned and trustworthy. Body model for the wrist, hand model for the knuckles.

This is testable, and worth testing, because "the fingers look a bit off" is not a signal you can iterate on. Re-express the hand landmarks in a completely different frame — 63° about an arbitrary axis — and re-run:

Frame-independent, shape-sensitive. Before the fix, the frame rotation moved them as much as the shape did.

Punches: perspective and speed-adaptive smoothing

Two things stop a punch from landing. The first is projection: under an orthographic camera an arm thrown at the lens does not get bigger, it just slides. Switching to perspective, matched so the character's plane spans the same range the ortho frustum did, keeps all the placement maths identical while limbs aimed at the camera foreshorten properly:

The second is latency. A fixed smoothing factor is a bad trade: enough to kill jitter when you hold still is far too much when you move fast. So smooth adaptively — a cheap One-Euro-style filter that opens up as a joint accelerates:

Jitter is a low-speed problem and lag is a high-speed problem, so tie the filter to speed and both go away.

Compositing: the plate is what erases you

Rendering the character is only half of it. If you draw him over the live feed, you get Spider-Man standing in front of a man in a t-shirt.

The trick is a background plate: a still frame of the empty room, captured before you walk in. Three layers on a 2-D canvas, every frame:

That single setTransform is load-bearing. Mirroring a 3-D scene is not a rotation, so do not try to mirror the character in world space — you will flip its handedness and spend an hour wondering why one hand is inside out. Do all the maths in raw camera coordinates and apply one reflection to the finished composite.

In React Three Fiber, the WebGL canvas needs gl: { alpha: true } and a transparent clear colour so it can be drawn over the plate, and you read gl.domElement straight into the 2-D context.

The five bugs, each of which looked like something else

And a sixth that was purely my own fault: in the headless test rig I built to check the maths, the synthetic face pointed away from the camera — I had the sign of the nose's z wrong. That flipped the head bone 150° in every test run while real webcam data was perfectly fine. Half an hour of debugging code that was already correct. If you build a mock, mock the coordinate conventions too.

Testing computer vision without a camera

Worth its own note, because it is what made the rest of this tractable. I never once debugged the retargeting maths by standing in front of a webcam and squinting.

Instead: a synthetic landmark generator behind a ?mock flag that produces a parametric body — arms raising, fingers curling, the whole thing rotatable by a ?yaw=60 parameter, with a ?frz=1.7 flag that freezes the clock so two runs are comparable. Then headless Chromium reads bone quaternions straight out of the scene and asserts on them.

That turns vague into numeric:

• "does he turn?" → pelvis, chest, head and arms all within 0.1° of the commanded yaw
• "are the fingers right?" → 0.10° under a frame rotation, 27° under a shape change
• "does the arm go up?" → left hand's world y above the head's world y

Overlaying the tracked skeleton on the posed rig in a screenshot is the other half. If the green dots sit on the joints, the fit is right; if they float, it isn't. You can read that in a second, and you cannot read it from a number.

Performance

Two MediaPipe models plus a 632k-triangle skinned mesh sounds like it should crawl. On an M-series MacBook it ran at 60–90 fps. What matters:

• GPU delegate for both landmarkers, with a CPU fallback. This is the single biggest lever.
• 720p input. Landmark detection does not get meaningfully better at 1080p and costs a lot more.
• One matrix update per frame. The size fit needs world-space joint positions, so update the rig's world matrices once and read from that — do not call getWorldPosition in a loop that triggers recomputation.
• Offer the lite pose model behind a flag for weaker machines. Same API, swap the .task URL.
• frustumCulled = false on skinned meshes. Their bounding volumes do not follow the bones, and they will vanish at the worst moment.

What it still can't do

Being honest about the edges, because they are structural:

• The plate is a photo. Anything that moves behind you — a person walking past, a curtain — gets frozen. The upgrade is live person segmentation, compositing him against the current room instead of a still.
• No occlusion. He is drawn on top of everything. Put a desk between you and the camera and he stands in front of it.
• One depth plane. Walking toward the camera is handled as scale, which is close but not the same as translation in depth.
• No forearm twist. Direction-only aiming cannot recover roll along a bone's own axis. The wrist frame hides most of it; rotate your palm slowly with a straight arm and you will see it.

Why bother

Because it is the same stack as the gesture controls on this portfolio, pushed one step further: from "read a hand and move a cursor" to "read a whole body and drive a character". Everything hard about it — retargeting onto a skeleton you did not author, sizing one body against another, rejecting data a model reports confidently but cannot actually see — shows up again in anything that maps the physical world onto a rig.

And it takes one webcam, one browser tab, and no server. That was not true three years ago.

If you want more builds in this shape, there is the gesture-controlled portfolio itself, a drivable 3D car game in React Three Fiber, and a punch-through ice wall with Voronoi shatter.