Gesture recognition systems have been around for quite some time now, but we still can’t say we have discovered neither all the possibilities they have to offer nor the approaches we can take.
The emergence of integrated sensors on the market, including infrared transmitters and receivers, and precise timing systems operating with great accuracy make the use of this technology very encouraging.
There are countless types of sensors that can be used for gesture recognition. Given a chance to try ourselves in this field, we’ve decided to create our new product – a gesture recognition system using Time of Flight (ToF) camera. Even the best gesture recognition system has no point without practical use cases, so an application allowing to play Tetris using gestures was developed.
Time of Flight
ToF — is a measure of the time it takes for an object, particle, or wave (e.g. acoustic, electromagnetic, etc.) to travel a distance through a medium. This information can then be used to establish a time standard, as a means of measuring velocity or path length allowing the camera to detect objects in three dimensions.
The technology works by sending multiple pulses of light up to five meters away. Bright pulses return to the 3D ToF camera when they hit an object, and the time it takes to return to the camera is used to calculate the object’s distance or depth. Just think of it as sonar or echolocation, albeit in the light rather than in sound.
Unlike regular cameras, ToF doesn’t need an external light source to perform correctly, so it is the perfect solution for places that are not always well lit, for example, car interiors. Due to their small dimensions, ToF sensors can be used in a wide range of applications.
The most popular among them are:
- proximity sensors for robots
- toilet paper and soap dispensers
- flushing cisterns in toilets
- sink mixers
- object sensors in robotic vacuum cleaners
- cheap user presence detectors in laptops and monitors
- inventory management systems in vending machines
- vending machine coin counters
- ground proximity detectors for drones
- ceiling proximity detectors for indoor drones
- presence and gesture sensors in retail outlets
- or playing Tetris!
The next step after getting the image was to answer the question ‘What’s in the image we’ve just received from the ToF camera?’. There isn’t any information on the internet about developing gesture recognition on the ToF sensor. After playing with deterministic algorithms for some time, we have decided to bring out the big guns. We created a special algorithm for image preprocessing first and then prepared a convolutional neural network model, collected a dataset containing snapshots of gestures we wanted our neural network to recognize. After few iterations of optimizing the model, we achieved a recognition accuracy of 98%.
What’s more, we managed to run our recognition system in real-time on Raspberry Pi 4!