Senior Project: Virtual Cafe (2/2016)

This project starts when our lab (Intelligent Systems Research Laboratory or ISL) received a Google Tango device to play with. Big thanks to VR1, our sponsor who obtained the device for us. The idea of this work is to explore what kinds of APIs Google Tango supports and what we can do with this technology.

It later became a topic for the senior project called “Virtual Cafe”. This research aims to study and to implement an application of Tango Project by Google with Lenovo Phab 2 Pro device. We focus on creating 3D Object Recognition into the device by creating an augmented reality application that can recognize the indoor environment and objects in the real world and render the area and recognized objects in the virtual world.

The objectives of the study include:

  1. Learning surrounding environments and virtualizing them.
  2. Training the dataset to let them recognize a particular object.
  3. Visualization of real object into the virtual world
  4. Allowing user to interact with objects in the virtual world.

The tools used in this study are:

  1. Tango Device
  2. Point Cloud Library (3D point cloud processing)
  3. Unity (Game engine)

The result of the study was as follows:

  1. The application is able to map the area in the real world and transforms it into the virtual world, allowing some manipulations to such an area such as changing the texture of the room.
  2. The application can recognize a 3D object from trained datasets and display the virtual object in the virtual room.
  3. Users can interact with the objects in the virtual world.

The picture below shows corresponding point clouds (left) of the real environment (right).

Examples of point clouds of the same object captured from different distances.

Here is a video clip demonstrating our work.

 

Challenges

  1. Access to the data captured by the Tango device is quite limited. We can’t access the raw data. We can only access the data provided by their APIs.
  2. Collecting datasets for training and testing is quite troublesome.

Deliverables

 

screen-shot-2017-07-02-at-11-45-42-pm

Leave a Reply

Your email address will not be published. Required fields are marked *