The future of user interface design

When he was working as a scientific consultant for Steven Spielberg, John Underkoffler captured the imagination of many digital designers with his futuristic, gesture controlled UI in the film Minority Report. Spielberg said to Underkoffler: “Please tell me that in 50 years we still won't be operating a UI with a mouse and a keyboard!”

Niels van Hamersveld

Today, around 15 years later, this is generally still the case, but not for long. Underkoffler spent 15 years working at the MIT Media Lab and became the founder of Oblong Industries. Underkoffler has since transformed the vision he presented in the film into a practical application, which he will initially use as a professional superpower in work environments.


The driver behind this will be g-speak, a platform developed by Oblong. It will be used to develop multi-user, multi-screen and multi-device applications that are controlled with gestures. G-speak is going to unleash a UI revolution. In work environments to start off with, but eventually on every screen in the world. Because as Underkoffler says: “Our work won't be finished until every single computer works this way”. These are some of the unique features of this gesture-controlled UI.

1. Hand gestures

The most significant feature is the use of gestures for control. A set of intuitive hand gestures can be used to control an interface, and we no longer have to rely on a mouse and keyboard. As Underkoffler says:

“Give people their hands back!”

With the UI as the exoskeleton, the operation of computers will become more human and more physical than ever before, and thus embody the ultimate extension of the human will.

2. Multiple users and multiple screens

All the screens in a room are linked to each other, and elements can be dragged from one screen to the other by pointing and swiping. Before long, with a simple swipe of the hand you will be able to move videos from your smartphone to your neighbour’s tablet, who can then throw them up on his smart TV. This means you can interact with multiple people using the UI because for the first time the computer will understand the concept of interacting with different people and because “pixels belong to everyone”.

3. UI for large spaces

Underkoffler emphasises the importance of embracing the advantages of scale. If you want to look at or design something big, you also have to do it in a big space. We can understand complex data much easier when we can experience it visually in 3D, and see how all the data is interrelated. You will soon be able to swim through a sea of data, so to speak, which you can actually understand and use for designs.

It sounds abstract, but we believe this will be the most significant breakthrough for big data applications. Most data is still too abstract and overwhelming for marketers and developers to handle in a practical way.


G-speak + Viv

Personally, I want to find out how a gestural interface like g-speak can be combined with a conversational AI like Viv. Viv allows you to talk to a computer like a normal person, and it is able to understand very complex voice commands. A combination of G-speak and Viv would be a tremendous step forward in making verbal and non-verbal communication with digital systems more human-like.

In any case, G-speak will be a great tool for designers when it comes to spatial designing. But it won't end with practical work applications because Underkoffler has made it clear that Oblong won't stop until every computer in the world uses its system.