SXSW: New user interface turns voice into 3D text

Mitsubishi Electric Corporation has announced a new user interface (UI) called SwipeTalk Air, which it believes to be the world’s first UI to use augmented reality (AR) technology to render spoken words instantly as three-dimensional text during live video recordings. Text positioning is implemented by swiping a finger over the tablet or smartphone screen. 

The company also announced its development of a SwipeTalk Air app on iOS that integrates the UI, video recording and other functions for extra-expressive videos that are expected to enliven social media.

A prototype of SwipeTalk Air will be displayed at the SXSW (South by Southwest) digital technology exhibition in Austin, Texas, from March 8 to 17. Operability and functions will be further upgraded prior to launching the UI and application in commercial products and services.

Main Features

1. UI renders spoken words as text along a swiped path on the screen

The UI immediately renders spoken words as 3D text in a video as the cameraperson swipes a finger across the device screen. The device is pointed at the subject, a few words are spoken, a line is swiped on the screen and then the text instantly appears in the swiped area.

Video recording and editing are seamlessly integrated, so the experience of capturing the moment live is not interrupted and users can enjoy inputting text intuitively.

2. App incorporates various functions to create highly expressive videos

It helps users achieve a new video expression by placing letters in anywhere of virtual midair in the screen. As the arranged text will look fixed in place, even when changing the camera angle, taking videos that combine such text and recorded objects enables new ways of expressing oneself and new uses based on users’ ideas. Due to swift text positioning, the user can easily produce videos full of creativity.

3. Text display using AR markers

When the subject wears an AR marker (camera-recognizable images) on the subject’s finger, it will make text follow the path of the subject’s moving finger.

4. AR picture drawing

Lines can be drawn in the video by tracing them on-screen to supplement text with illustrations, such as arrows, pattern or cartoon-like decoration, to enhance the effect and clarity of the message.

5. Multilingual translation

The positioned text can be translated into a foreign language instantly. When the displayed text is changed to another language, it is read out loud in that language to enable communication and collaboration with people who speak other languages.