Motion-controlled Street Fighter with the Bangle.js watch and WebAI

08/12/2025

Back in 2019, I worked on building a prototype of Air Street Fighter using 3 different inputs: a Daydream controller, an Arduino MKR1000 and a phone with the Web Sensor API, making a custom machine learning model using TensorFlow.js to play using gestures.

Over the years I also experimented with making it thought controlled using the Neurosity Crown and updating it to use the Arduino Nano 33 BLE Sense with Tiny Motion Trainer.

A few weeks ago, I was at the WebAI summit where I met the creator of Bangle.js, a hackable, open source smart watch that can run JavaScript. All attendees at the conference got lucky enough to get one and I thought it would be a great new input for the prototype.

Demo

If you have a Bangle.js watch, try the live demo!

You can check out the repository to learn more.

Gathering samples

First of all, you need to train a model with custom gesture data so you can then detect live gestures and apply them to the game. For this prototype, I only enabled 3 gestures: "punch", "hadoken", "shoryuken".

To get accelerometer data, here's the JavaScript code that handles sending the right command to the watch via Bluetooth:

export const getAccelerometerData = () => {
  UART.write(`\x10reset()\n`) // Clear out everything currently running.
    .then(() => new Promise((resolve) => setTimeout(resolve, 500))) // Wait for a bit just to make sure.
    .then(() =>
      UART.write(
        `\x10Bangle.setPollInterval(50); Bangle.on("accel",e=>Bluetooth.println(E.toJS({t:"acc", x:e.x, y:e.y, z:e.z})));Bluetooth.println();\n`
      )
    ) // 50ms = 20Hz.
    .then(function() {
      const connection = UART.getConnection();
      connection.removeListener("line", dataLineReceived); // Remove any existing listener so we don't get duplicates.
      connection.on("line", dataLineReceived);
    })
    .catch((e) => {
      console.log("Connection Failed:" + e);
      stopData();
    });
};

The dataLineReceived function then handles aggregating the data.

const dataLineReceived = (line: unknown) => {
  const json = UART.parseRJSON(line);

  if (json) {
    const MAGNITUDE = Math.sqrt(
      json.x * json.x + json.y * json.y + json.z * json.z
    );
    const SAMPLE: number[] = [json.x, json.y, json.z, MAGNITUDE];

    dataBuffer.push(SAMPLE);

    if (dataBuffer.length > WINDOW_SIZE) {
      dataBuffer.shift();
      DATA.push({ label: gesture?.index, data: dataBuffer });
      dataBuffer = [];
    }
  }
  if (!json) {
    return stopData();
  }
};

Once some samples have been recorded, it's time to train the model.

Training the model

Below is the code I wrote to build my model. There are a lot of different ways to build machine learning models so this is just what got me a better accuracy.

export const createModel = async (data) => {
  const { xs, ys, model, dataMeanTensor, dataStdTensor } = tf.tidy(() => {
    const NUM_CLASSES = CLASSES.length; // this is ["punch", "hadoken", "shoryuken"]

    const DATA_FLAT = data.flatMap((d) => d.data.flat());
    const xDims = data[0].data.flat().length;

    const labels = data.map((d) => d.label);
    const xs = tf.tensor2d(DATA_FLAT, [data.length, xDims]);
    const ys = tf.oneHot(tf.tensor1d(labels).toInt(), NUM_CLASSES);

    const { mean, variance } = tf.moments(xs, 0);
    const dataStd = tf.sqrt(variance);

    const model = tf.sequential();
    model.add(
      tf.layers.dense({
        units: 25,
        activation: "relu",
        inputShape: [xDims],
      })
    );
    model.add(
      tf.layers.dense({
        units: 10,
        activation: "relu",
      })
    );

    model.add(tf.layers.dense({ units: NUM_CLASSES, activation: "softmax" }));

    const optimizer = tf.train.adam(0.001);
    model.compile({
      optimizer: optimizer,
      loss: "categoricalCrossentropy",
      metrics: ["accuracy"],
    });

    return {
      xs,
      ys,
      model,
      dataMeanTensor: mean,
      dataStdTensor: dataStd,
    };
  });

  const savedMean = await dataMeanTensor.array();
  const savedStd = await dataStdTensor.array();

  dataMeanTensor.dispose();
  dataStdTensor.dispose();

  const expectedDims = savedMean.length;

  const modelSpecs = { savedMean, savedStd, expectedDims };

  // I'm storing the model specs in local storage here so I can fetch them on the live game page
  localStorage.setItem("model_specs", JSON.stringify(modelSpecs));

  const xs_normalized = tf.tidy(() => normalizeTensor(xs, savedMean, savedStd));

  console.log("\n--- Starting Model Training ---");
  await model.fit(xs_normalized, ys, { epochs: 500, verbose: 0 });
  console.log("--- Training Complete ---\n");

  // Dispose of training tensors (xs, ys) now that they are no longer needed
  xs.dispose();
  ys.dispose();
  xs_normalized.dispose();

  // I'm saving the model in local storage here so I can fetch it on the live game page
  await model.save("localstorage://gestures-model");

  return model;
};

Once the model is trained and saved, it can be used for live predictions.

Running predictions

Loading the model

The model is stored in local storage so it can be loaded with:

const model = await tf.loadLayersModel("localstorage://gestures-model");

Loading the model settings

Same thing with the settings that were saved in the step before:

if (localStorage.getItem("model_specs")) {
  const { savedMean: sm, savedStd: sStd, expectedDims: eDims } = JSON.parse(
    localStorage.getItem("model_specs")
  );
  setSavedMean(sm);
  setSavedStd(sStd);
  setExpectedDims(eDims);
}

Getting predictions

For this step, the code to connect to the watch, get the accelerometer data and call the dataLineReceived function is the same as before, except that now, the callback function needs to run predictions, so here is the predict function:

export const predict = async (
  dataBuffer,
  model,
  savedMean,
  savedStd,
  expectedDims
) => {
  // Initialize tensors for cleanup in the finally block
  let inferenceXs = null;
  let inferenceXs_normalized = null;
  let predictOut = null;
  let finalProbabilitiesTensor = null;
  let predictionIndexTensor = null;

  try {
    inferenceXs = tf.tensor2d(dataBuffer.flat(), [1, dataBuffer.flat().length]);

    if (inferenceXs.shape[1] !== expectedDims) {
      throw new Error(
        `Inference data mismatch! Expected ${expectedDims} features, but got ${inferenceXs.shape[1]}.`
      );
    }

    inferenceXs_normalized = normalizeTensor(inferenceXs, savedMean, savedStd);

    predictOut = model.predict(inferenceXs_normalized);

    finalProbabilitiesTensor = tf.softmax(predictOut);

    predictionIndexTensor = finalProbabilitiesTensor.argMax(-1);
    const predictionIndex = (await predictionIndexTensor.data())[0];
    const winner = CLASSES[predictionIndex];

    const probabilitiesArray = await finalProbabilitiesTensor.array();
    const scores = probabilitiesArray[0];
    const confidenceScore = scores[predictionIndex];

    return { winner, confidenceScore, scores };
  } catch (e) {
    console.error("Prediction Error:", e);
    throw e;
  } finally {
    // Dispose of ALL Tensors created in the try block
    if (inferenceXs) inferenceXs.dispose();
    if (inferenceXs_normalized) inferenceXs_normalized.dispose();
    if (predictOut) predictOut.dispose();
    if (finalProbabilitiesTensor) finalProbabilitiesTensor.dispose();
    if (predictionIndexTensor) predictionIndexTensor.dispose();
  }
};

From there, the winner gesture label returned can be used to trigger the animation and sound for a punch, hadoken or shoryuken in the game.

If you're curious to learn more about the game logic specifically, you can check the source code in the repository.

Conclusion

With the previous sensors I worked with, I had access to both accelerometer and gyroscope data and I think it made it more accurate. The Bangle.js does not have a gyroscope so the gesture recognition is using the x, y and z value of the accelerometer with a single value for the magnetometer. However, the form factor of it being a smartwatch is really nice because it can serve different purposes and it is something people can naturally wear on a day to day basis.

Throughout the various versions of this project, I only ever implemented the gameplay for a single character but it would be really fun to have someone connecting with another watch and be able to play the second player!