Making an interactive digital frame with head-tracking using Three.js and TensorFlow.js

Over the past few weeks, I've been working on a new side project to replicate a visual effect called "head-coupled perspective". This technique is not new but I was interested in figuring out how to make it work using Three.js so I could make some interactive art with head-tracking.

Here's the end result:

As the user moves, the perspective changes to give the impression of being able to look inside the frame, even though this is a 2D display.

The graphics are made using Three.js, the plant is a 3D model downloaded from Sketchfab and the head-tracking is done using the MoveNet model in TensorFlow.js.

While doing some research around implementing the perspective effect, I learnt that it had to do with changing the projection matrix of the camera and stumbled upon a pull request to the Three.js repo, that seemed to be close to what I was looking for.

The PR had been merged and a new util called frameCorners() had been added to the library. According to the docs, this util "sets a PerspectiveCamera's projectionMatrix and quaternion to exactly frame the corners of an arbitrary rectangle". This sounded exactly like what I needed! If you look closely at the demo above, you can notice that, as the perspective changes, the external corners of the box do not change position.

Updating the camera's projection matrix

The way to use this util is to pass it the camera, and 3 vectors representing the coordinates of the points that will represent your arbitrary rectangle.

CameraUtils.frameCorners(
  camera,
  bottomLeftCorner,
  bottomRightCorner,
  topLeftCorner,
  false // This boolean is for the argument `estimateViewFrustum` but to be honest I don't quite understand what it means.
);

In my scene, I have a plane geometry used to create 5 meshes that make up my "box". This geometry is about 100x100, and each mesh using it is has a different position and rotation depending on which side of the box it is used for.

Here's some code sample to illustrate what I'm talking about

// Top part of the box
planeTop.position.y = 100;
planeTop.rotateX(Math.PI / 2);

// bottom part of the box
planeBottom.rotateX(-Math.PI / 2);

// Back of the box
planeBack.position.z = -50;
planeBack.position.y = 50;

// Right side of the box
planeRight.position.x = 50;
planeRight.position.y = 50;
planeRight.rotateY(-Math.PI / 2);

// Left side of the box
planeLeft.position.x = -50;
planeLeft.position.y = 50;
planeLeft.rotateY(Math.PI / 2);

Considering these positions, we can create vectors to represent the points we want to use for our camera:

let bottomLeftCorner = new THREE.Vector3();
let bottomRightCorner = new THREE.Vector3();
let topLeftCorner = new THREE.Vector3();

bottomLeftCorner.set(-50.0, 0.0, -20.0);
bottomRightCorner.set(50.0, 0.0, -20.0);
topLeftCorner.set(-50.0, 100.0, -20.0);

The bottomLeftCorner vector has a x position of -50 to match the x coordinate of planeLeft, a y position is 0 to match the y position of planeBottom which default value is 0, and a z position of -20 to have a bit of depth but not too much.

It took me some time to understand how to choose the coordinates of my vectors to get the effect I wanted but this GIF helped a lot:

As you change the coordinates of the vectors, the camera changes position and orientation to frame these corners.

This was only one part of the solution though, the second part kinda happened accidentally. 😂

OrbitControls

Once I managed to get the correct coordinates for my vectors and use the frameCorners() util, the position of the camera was fitting the right rectangle but when trying to change the perspective with face tracking, something weird was happening.

I wish I had recorded it at the time so I could show you what I mean but I'm going to try to explain it anyway.

In the demo at the beginning of this post, you can see that no matter how the perspective changes, the back plane is always parallel to me. What happened when I only used frameCorners() is that this plane was rotating, so the vector's z position was changing, which didn't give a realistic effect at all.

A bit like the GIF below but imagine it happen only on one side:

TIL this is called the "Dolly zoom"!

To try to debug it, I thought maybe using OrbitControls would help, to let me rotate around my scene and maybe use a camera helper to see what was going on, but instead, it just fixed my problem!

By only adding let cameraControls = new OrbitControls(camera, renderer.domElement);, I was now able to change the perspective of the scene without the back plane rotating, which made it look much more realistic!

What happened next is pure laziness... I could have looked deeper into how OrbitControls work to find out exactly which part I needed but instead, to save some time (this is only a side project after all), I made some updates directly into the OrbitControls.js file.

I located where the function handleMouseMoveRotate was, duplicated it and called the new one handleFaceMoveRotate to handle face movements. I modified it a little to receive the face coordinates instead of mouse coordinates, and TADAAA!! It worked! 🎉

Next steps

I'd like to create a few more scenes and I have an idea to push this project a little bit further but I feel like I need a break from it right now.

When I spend too much time trying to debug a side project, it sometimes removes the fun of it. I need to leave it on the side for some time and get back to it when I feel excited to work on it again. 😊

In the meantime, feel free to check out the code