In anticipation of the post-Covid «touchless everything» trend, DataArt team developed two PoCs for using gesture recognition technologies in public spaces. It is reasonable and fairly cost-effective to equip all municipal devices with a similar technology after the pandemic.
Natural User Interfaces (NUI), which allow direct user interaction with a device via touch, gestures, or voice, have been the talk of the technology world for a few decades now. However, while the popularity of touchless technologies has been gradually increasing, designers remained reluctant to invest much time and effort in promoting new types of interaction — why rack user’s brain if high-fidelity, touch-based interactions are already used so widely?
Touchless — the Imperative of the “New Normal”
Coronavirus pandemic is taking NUI to a new level. As Microsoft CEO Satya Nadella, famously said, “We’ve seen two years’ worth of digital transformation in two months.” In the post-Covid, or rather with-Covid, world, NUI, and touchless interfaces in particular, are becoming a new imperative. We tend to avoid touching screens and buttons in public places — cafes, airports, hotels, shopping malls, museums — which had been previously handled by thousands of fingers. These publicly used surfaces cannot be cleaned frequently enough as a virus mitigation measure. Touchless interfaces, however, can help reduce the spread of germs. For tech companies, it is now the right time to explore motion and gesture recognition technologies and utilize them for all public-facing devices, like customer input and information terminals.
DataArt Releases PoCs for Touchless Interfaces
A team at DataArt’s creative tech incubator, TalentLab recently developed two proofs of concept for using gesture recognition technologies in any public space. The PoCs prove how easily public touch interfaces can be transformed into touchless ones. Though the technology itself is not new, it has not yet gained wide implications. We anticipate it should — as touchless experience is already a part of the “new normal” user experience.
DataArt’s TalentLab showed that there is only one step between the development and implementation of world-changing technology.
The PoCs are variations of a perceptual user interface based on a computer vision technology that allows machines to recognize human movements and interpret them as commands. For instance, when one navigates through the menu of a public terminal, the software recognizes the skeletal structure of a human hand via an Intel SR300 camera. Based on this information, the system estimates the movement pattern of each joint and interprets the gestures.
To prove the concept of touchless technology in a food and drink venue, we initially used the PoC for a typical food ordering interface, where a person navigates through the menu without touching the actual display.
DataArt Orchestra
One of the PoC versions, titled DataArt Orchestra, uses a 3D AI-based full-body skeleton tracking SDK, Cubemos, and Intel ®RealSense™ technologies. The software, developed on .NET, C#, Windows Presentation Foundation, recognizes and tracks hand movements of a person that stands in front of the camera. He/she places an order by performing either of the four easiest dynamic gestures — up, down, right, and left. Cubemos software recognizes certain joint points along with the length of some joint segments and the relation between them.
DataArt Touchless Five
Another version of the PoC, DataArt Touchless Five, was developed based on the Project Prague library by Microsoft, a cutting-edge hand gesture tracking SDK, and Intel ®RealSense™ SR300 camera. The software allows the person who stands in front of the camera to control a virtual palm on the screen and navigate through the interface with it, not touching the actual screen, as if he/she moves a computer mouse in mid-air.
Understanding that users get easily frustrated if their interaction with a device is not immediately successful, we tried to keep the set of input movements simple and provided visual helpers to make them more intuitive. Human hand movements are not precise, especially in mid-air. For a machine, it is difficult to interpret imprecise or fast movements, and accurately identify gestures in loose clothing. In further iterations, the PoCs may be improved to recognize more elaborate gestures and movements of both hands. It can also be extended to navigate menus in various multi-purpose terminals. The cost of the solution compares favorably to installing a public sensor display for customer input.
The Scope of Application for Touchless Technologies
Touch technologies, used almost everywhere now, can and should be replaced with touchless ones where possible: public transport ticket kiosks, vending, and food-ordering machines, ATMs, elevators, digital whiteboards, information and wayfinding terminals in large buildings, like shopping malls, hotels, and airports. Touchless exhibits for zoos, exhibitions, science centers, and children’s museums will gain traction as public recreation spaces start to reopen after quarantines. Another benefit of touchless interfaces is the fact they could be utilized in settings and situations where touching is impractical or impossible, like kitchens or operating rooms.
Though humanity is yet to experience more dramatic transformations after the pandemic, it is already clear that Covid-19 has re-shaped the current digital landscape and catalyzed the adoption of new technologies. Software advancements that used to take years are now ready within months, if not weeks. It turned out that future technologies are already here. At DataArt, we move new types of user-software interaction forward by developing DataArt Orchestra and Touchless Five, PoCs for using a touchless technology in a public interface.
