Turning your phone into a virtual-joystick
Update: I kept working on this and I have released it as a package for Windows, Linux and macOS. Check it out:
https://github.com/zenineasa/joystick/releases/tag/v1.0.0
--
During the days when I was pursying my master's programme, my friends and I used to occasionally go to a classroom in the university, turn on a projector, connect devices like Nintento Switch or Gaming computers and loads of joysticks, and play different simple multiplayer games; MarioKart was my favourite. From time-to-time, when I get together with people, I ponder that it would be a good idea if I bought such devices.
Indeed, I do have a laptop, which could easily run such games; SuperTuxKart is similar enough to MarioKart and it can run on Linux, Windows and Mac. However, I do not have joysticks with me at the moment. Therefore, I think it would be a good idea if I simply worked on a project that would enable using our phones as joysticks.
From a high-level, the plan is to host APIs on a NodeJS server that would use RobotJS to perform keyboard inputs; and the phone could interact with these APIs using a web app. With this, people in the same home network can use their phones as joysticks to play different games.
Step 0: Install NodeJS
Follow the instructions on NodeJS's official website: https://nodejs.org/en/download/package-manager
Step 1: Create a sample ExpressJS project
Run the following commands to initialize a NodeJS project, and fill in all the details as prompted.
mkdir joystick && cd joysticknpm init
Now, run the following command to install ExpressJS into the project.
npm install express
Now, create a file named 'index.mjs' and add the following content:
import express from 'express';
const app = express();
const PORT = 3000;
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.listen(PORT, (error) => {
if(!error) {
console.log("Server running...");
} else {
console.log("Error: ", error);
}
});
Run the following command to start the express server:
node index.mjs
At this point, you can open the following URL and observe the response to the get request: http://localhost:3000/
To view this page from your phone or another device in the same local network, simply replace 'localhost' with your IP address.
Step 2: Install and test RobotJS
The following command can be used to install RobotJS in our project.
npm install robotjs
Let us modify 'index.mjs' to create a request that would trigger a keypress when it is requested. The following content may be added to 'index.mjs', perhaps before 'app.listen' is invoked.
import robot from 'robotjs';
app.get('/testKeystroke', (req, res) => {
setTimeout(() => {
robot.keyTap('a');
robot.keyTap('s');
robot.keyTap('w');
robot.keyTap('d');
robot.keyTap('enter');
robot.keyTap('t');
robot.keyTap('y');
robot.keyTap('u');
robot.keyTap('g');
}, 3000);
res.send(true);
});
Now, if you re-run the server, open http://localhost:3000/testKeystroke on your browser and immediately switch to a text editor (within 3 seconds), you should be able to see some text being typed into the editor.
Step 3: Create the user interface for the joystick
Inside the project, create a new folder named 'view'. Make three files inside it named 'index.hml', 'script.js' and 'styles.css'. The contents of the files are as follows:
index.html:
<!DOCTYPE html>
<!-- Copyright (c) 2024 Zenin Easa Panthakkalakath -->
<html>
<head>
<title>Joystick</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no">
<link rel="stylesheet" href="styles.css">
</head>
<body>
<div class="direction">
<div class="button" id="NW"> </div>
<div class="button" id="N">▲</div>
<div class="button" id="NE"> </div>
<div class="button" id="W">◀</div>
<div class="button" id="C">⊙</div>
<div class="button" id="E">▶</div>
<div class="button" id="SW"> </div>
<div class="button" id="S">▼</div>
<div class="button" id="SE"> </div>
</div>
<div class="action">
<div class="button" id="A">A</div>
<div class="button" id="B">B</div>
<div class="button" id="C">C</div>
<div class="button" id="D">D</div>
<div class="button" id="X">X</div>
<div class="button" id="Y">Y</div>
</div>
<script src="script.js"></script>
</body>
</html>
styles.css:
/* Copyright (c) 2024 Zenin Easa Panthakkalakath */
html,
body {
overflow: hidden;
height: 100%;
}
body {
touch-action: manipulation;
-ms-touch-action: manipulation;
-webkit-user-select: none;
-ms-user-select: none;
user-select: none;
position: absolute;
width: 100%;
height: 100%;
display: flex;
justify-content: center;
align-items: center;
}
.direction {
background-color: #000;
width: 10em;
height: 10em;
border-radius: 50%;
align-items: center;
display: grid;
grid-template-columns: auto auto auto;
margin-right: 1em;
}
.direction .button {
color: #fff;
text-align: center;
}
.direction .active{
color: #00faff;
}
.action {
align-items: center;
display: grid;
grid-template-columns: auto auto auto;
margin-left: 1em;
}
.action .button {
background-color: #000;
border-radius: 50%;
color: #fff;
padding: 1.5em;
margin: 0.5em
}
.action .active{
color: #00faff;
}
script.js:
/* Copyright (c) 2024 Zenin Easa Panthakkalakath */
// Store touch event ID and corresponding button ID
const touchEventIDVsButtonID = {};
function processKeyEvent(touchID, buttonID, startMoveOrEnd) {
if (startMoveOrEnd === 'end') {
sendKeyEvent(touchEventIDVsButtonID[touchID], 'end');
delete(touchEventIDVsButtonID[touchID]);
} else if (startMoveOrEnd === 'start') {
sendKeyEvent(buttonID, 'start');
touchEventIDVsButtonID[touchID] = buttonID;
window.navigator.vibrate(100); // Haptic feedback
} else { // 'move' event
if (touchEventIDVsButtonID[touchID] !== buttonID) {
sendKeyEvent(touchEventIDVsButtonID[touchID], 'end');
sendKeyEvent(buttonID, 'start');
touchEventIDVsButtonID[touchID] = buttonID;
window.navigator.vibrate(100); // Haptic feedback
}
}
}
// Function to send joystick key press events to the backend
function sendKeyEvent(buttonID, startOrEnd) {
const data = {
buttonID: buttonID,
startOrEnd: startOrEnd
};
fetch('/keyEvent', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(data),
})
.then(response => response.json())
.then(data => {
console.log('Success:', data);
})
.catch((error) => {
console.error('Error:', error);
});
}
// To prevent accidentally invoking refresh
document.addEventListener('touchmove', function (event) {
if (event.touches.length > 1 || event.scale && event.scale !== 1) {
event.preventDefault();
}
}, { passive: false });
document.addEventListener('touchstart', function (event) {
if (event.touches.length > 1) {
event.preventDefault();
}
}, { passive: false });
document.addEventListener('touchstart', function (event) {
event.preventDefault();
for (let touch of event.touches) {
const target = document.elementFromPoint(touch.clientX, touch.clientY);
if (target.classList.contains('button')) {
target.classList.add('active');
processKeyEvent(touch.identifier, target.id, 'start');
}
}
});
document.addEventListener('touchmove', function (event) {
event.preventDefault();
for (let touch of event.touches) {
const target = document.elementFromPoint(touch.clientX, touch.clientY);
if (target.classList.contains('button')) {
target.classList.add('active');
processKeyEvent(touch.identifier, target.id, 'move');
}
}
});
document.addEventListener('touchend', function (event) {
for (let touch of event.changedTouches) {
const target = document.elementFromPoint(touch.clientX, touch.clientY);
if (target.classList.contains('button')) {
target.classList.remove('active');
//processKeyEvent(touch.identifier, target.id, 'end');
}
processKeyEvent(touch.identifier, '', 'end');
}
});
The HTML and CSS files render a neat looking joystick on the screen. The JavaScript file detects the different kinds of touch happening on the screen and invokes a POST request to let the backend know which keys are pressed. You should be able to see that I have included some haptic feedback to different touch actions to make the feel while using the virtual-joystick better.
Step 4: Taking actions at the backend
After I manually checked which all keys can be triggered using RobotJS, and which keys generally work with the game that I would like to play after implementing this, I arrived at pool of keys. Whenever a new device is connected, a set of keys get assigned to the device from the available pool.
The original 'index.mjs' file has been modified to have an API that receives touch information from the UI and translate that to keyboard strokes. The entire file content is shown below:
/* Copyright (c) 2024 Zenin Easa Panthakkalakath */
import os from 'os';
import express from 'express';
import robot from 'robotjs';
import { dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = dirname(fileURLToPath(import.meta.url));
let numButtonsPerPlayer = 0;
const buttonIDVsOrder = {
'N': numButtonsPerPlayer++,
'E': numButtonsPerPlayer++,
'W': numButtonsPerPlayer++,
'S': numButtonsPerPlayer++,
'A': numButtonsPerPlayer++,
'B': numButtonsPerPlayer++,
'C': numButtonsPerPlayer++,
'D': numButtonsPerPlayer++,
'X': numButtonsPerPlayer++,
'Y': numButtonsPerPlayer++,
};
let lastPlayerID = -1;
const playerIPVsID = {};
const availableKeys = [
//'-', '=', '[', ']', ';', '\'', ',', '.', '/', '\\', '`', ' '
];
for(let i = 'a'.charCodeAt(0); i < 'a'.charCodeAt(0)+26; ++i) {
availableKeys.push(String.fromCharCode(i));
}
for (let i = 0; i < 9; ++i) {
availableKeys.push(i.toString());
}
function getPlayerID(playerIP) {
if (playerIPVsID.hasOwnProperty(playerIP)) {
return playerIPVsID[playerIP];
}
const playerID = ++lastPlayerID;
playerIPVsID[playerIP] = playerID;
return playerID;
}
function getKeyboardButton(playerIP, buttonNumber) {
const playerID = getPlayerID(playerIP);
return availableKeys[playerID * numButtonsPerPlayer + buttonNumber];
}
function getLocalIPAddress() {
const interfaces = os.networkInterfaces();
for (const interfaceName in interfaces) {
const iface = interfaces[interfaceName];
for (let i = 0; i < iface.length; i++) {
const alias = iface[i];
if (alias.family === 'IPv4' && !alias.internal) {
return alias.address;
}
}
}
return 'localhost';
}
const app = express();
const PORT = 3000;
app.use(express.json());
app.use(express.static(__dirname + '/view'));
app.get('/', (req, res) => {
res.sendFile(__dirname + '/view/index.html');
});
app.post('/keyEvent', (req, res) => {
const data = req.body;
const playerIP = req.headers['x-forwarded-for'] || req.connection.remoteAddress;
for (let i = 0; i < data.buttonID.length; ++i) {
robot.keyToggle(
getKeyboardButton(playerIP, buttonIDVsOrder[data.buttonID[i]]),
data.startOrEnd === 'start' ? 'down' : 'up'
);
}
res.send(true);
});
app.listen(PORT, (error) => {
if(!error) {
console.log(`Server running at http://${getLocalIPAddress()}:${PORT}`);
} else {
console.log("Error: ", error);
}
});
Step 5: Open it on the phone
As I run the program, the server's address is now displayed in the terminal, which can be opened on a phone.
Step 6: Play SuperTux (or any other game you choose) with it
Open SuperTux, edit the controller settings to match with the virtual-joystick. You can do this by simply double clicking on the each action setting, and then pressing the button on the joystick. Once you are all set, just go ahead and play!
... TODO: Add a video of me playing the game! ...
While I can play the game better with a keyboard at the moment, playing with the joystick is a lot of fun. Perhaps with more practice, I might be able to play even better using the joystick.
Comments
Post a Comment