Beyond the Mouse — Akshay R Nair

ABOUT

This project was developed as part of an Interaction Design course at IDC, IIT Bombay. Guided by Prof. Jayesh Pillai & Prof. Jayti Bandopadhyay.

This is an exploratory project to recognise everyday objects for digital navigations, and the leveraging the affordances they bring to the table.

TEAM

Akshay R Nair

Riddhi Ashokkumar

Rohit Waghmare

THE INTENT

What if everyday things could help us navigate the digital world?

Everyday objects come with their own familiar interaction affordances. A glass invites holding, tilting, and pouring; a knob invites turning. These physical qualities can sometimes offer a more intuitive mode of interaction than a mouse on a 2D plane.

This exploration looked at using objects people already have as inputs for digital navigation. For example, in a simulation where you need to pour water from a jar, tilting a real glass can feel far more natural than dragging a cursor.

The challenge was to sense these movements reliably and distinguish intentional interactions from everything else happening around the object.

FIRST TRY / UNDERNEATH CAMERA

Tracking the object from below

Our first prototype used an opaque surface with a camera underneath.

A Python script tracked contrasting blobs rather than identifying specific objects, which made the system considerably accurate and allowed almost any opaque object to be used as an input.

The limitation was that the object needed to remain close to the surface (or in contact) and also have a good sharp lighting over it.

A palm resting on it could also be detected as a blob, making it difficult to distinguish the intended object from the hand manipulating it.

Underneath camera and Python blob tracking.

SECOND TRY / KINECT

Using depth to isolate movement and tilt

Kinect gave us much more accurate motion and tilt tracking.

We first scanned the environment without the object, then introduced the object into the scene. Then by subtracting the background scan from the new depth reading, we could isolate and track the object reliably.

Relying on the depth sensors and outdated software, we managed to roughly detect movement and tilt of object.

The main limitation was physical setup. Kinect needed to be placed roughly 1.5 metres away, making it difficult to integrate into a practical desk-based interaction. We needed the same tracking accuracy in a much more compact setup.

Occulusion still remains an issue when using any camera setup.

Exploring how object can be distinguished from enviornment using depth in kinect.

Final setup, and establishing a horizontal scanning.

The Kinect prototype tracking object movement and tilt.

THIRD TRY / OVERHEAD CAMERA

Reading movement from above

We moved the camera above the interaction surface.

Initially, we considered using contrasting stickers as markers, but instead made the object itself the marker by capturing its appearance and tracking it directly.

Occlusion remained a limitation, so we turned it into an interaction. Intentionally covering the object, such as waving a hand above it, could trigger actions like a right-click.

Depth was estimated through scale: as the object moved closer to the camera, it appeared larger in the frame. This allowed us to detect lifting, although tilt could not be tracked reliably.

This became the most practical setup, combining object tracking, depth cues, and intentional occlusion without requiring additional markers.

Object move and tap over a contrasting background, captured by the camera above.

The object replaces the cursor.

Object interaction and the digital response.

EXHIBITION

Bringing the exploration into people's hands

For the exhibition, we turned the system into a space-shooter game where visitors could choose any everyday object and use it to control the spacecraft.

Moving the object across the surface controlled the ship in 2D, lifting it triggered a speed boost, and tapping it on the surface fired a shot.

Instead of designing a new controller and asking people to learn it, the interaction could emerge from the familiar ways they already handle everyday objects.

The exploration showed how movement, lifting, tapping, and even occlusion could become meaningful digital inputs, allowing ordinary objects to temporarily take on entirely new interactive roles.

Beyond the Mouse at the exhibition: an everyday object becomes a tangible input.