ART & TECH | HYE SOO YANG

Variable   Reality  II 

MS Thesis, MIT Media Lab

        The basic concept of Variable Reality is essentially based on the idea of enabling users to read a digital book anywhere while maintaining the physical quality of a real book. The Variable Reality system uses a lightweight and compact head- mounted display (HMD) device that is see-through. Two types of books designed for the system are the Augmented Book and the Virtual Book. 

Variable Reality Virtual Bookshelf

Variable Reality Demonstration Video

Variable Reality Setup —a physical book (left) and the HMD (right)

Interaction Techniques

(a) Index Fingertip —the user’s index

     fingertip is used for interaction just 

     like a mouse cursor on screens  

(b) Quick Swipe —a quick opening and

      closing of the hand motion 

(c) Stay Click — staying (or hovering) at a

      place for 1.8 seconds using the index

      fingertip 

(d) Open-Close Hand —a quick motion of

      moving the hand from right to left while

      keeping the pointing gesture 

Augmented Book

Virtual Book

        Based on the stable book- and hand-tracking algorithms along with the recognition of various gestures designed for user interaction, I built several Variable Reality applications for both the Augmented Book and the Virtual Book. These applications import convenient navigation techniques available on digital books and yet maintain the physicality of a paper book. The goal of each application is to create an intuitive and effective reading experience for the users. Incorporating natural hand gestures associated with reading activity adds to the positive user experience in reading. 

        In the Augmented Book scenario, the user can take any book from the real environment and turn it into a digital book, which can be seen through the HMD (as shown in the image above). The book becomes an augmentable surface for overlaying paper pages with digital information without the need of a visual marker for tracking. As a result, it enables the user to quickly switch between reading a paper book and reading a digital book with a single physical book. The title of a book is detected using a feature detector and the page number can be detected with the help of the OCR technology. This enables the user to store and retrieve digital data on a specific physical page. The Augmented Book also integrates natural hand gestures associated with reading activity in order to provide an intuitive reading experience. A fingertip is predominantly used as a mouse cursor but is used as a pen tip for annotations. 

        Ideally, the Virtual Book makes it possible for users to hold a 3-D virtual book in the hand and read it anywhere. It is designed to bring extended mobility and accessibility to the digital library designed for on-the-go reading experience. The user does not need anything other than the hand to bring up a virtual book on the palm. Similar to the Augmented Book, it integrates the hand gesture recognition system, which enables users to efficiently manipulate the book. 

        Variable Reality further incorporates a three-dimensional digital bookshelf that closely resembles the look of a real bookshelf. Using a simple gesture, the user can display it, zoom into a portion of it and pick out a book to read. This book can be in the form of either the Augmented Book or the Virtual Book. 

(a) Open/Close Hand (Face Up) —a quick

      opening and closing the hand with the

      palm looking up

(b) Right Hand Swipe —a swift, right-to-left

      hand movement using the right hand

(c) Right Hand Twist —a quick rotating

      motion of the right hand around the  

      middle finger using the wrist

(d) Right Hand Pinch —a hand gesture similar

      to how one would pinch someone

(e) Two-Hand Swipe Open/Close — a quick

      two hand movement separating apart

      from one another

(f) Finger Fold —folding one of the five

     fingers

(g) One-Hand Swipe —left hand movement

      from left to right with the palm facing up

(h) One-Hand Quick Bounce —a sudden

      down and up hand movement

(i) Quick Open-Close (Face Down) —a quick

      closing and reopening of the left hand

Applications

CONCEPT

INTRODUCTION

      Variable Reality is a novel Augmented Reality (AR) based interactive system focused on bringing intuitive on-the-go reading experiences to the users by enabling always-available access to and presentation of any digital book contents in a complete spatial form of a book. Variable Reality introduces two types of books: the Augmented Book and the Virtual Book.

      The Augmented Book incorporates computer vision based corner detection and tracking systems to detect the corners of a physical book, information of which is used to overlay paper book pages with digital pages using a Head-Mounted Display (HMD) device. As a result, any real book can instantly turn into a different book while still retaining the physical properties of the book, conforming to the legacy of the paper book metaphor. The Virtual Book is an entirely virtual 3-D book that extends the accessibility and mobility of reading materials. When in absence of a physical book, the user can retrieve this virtual book on the palm of a hand –a well-established pose of how one holds a book when reading. The weightless and portable nature of the Virtual Book makes it possible for users to comfortably hold larger physical books such as unabridged dictionaries or the World Book Encyclopedia.

      In both augmented and virtual book scenarios, natural hand gestures inherently associated with the reading activity are defined and used in various Variable Reality applications. 

Webpage designed by Hye Soo Yang