Modeling Social Interactions

Published in CoRL-2021, NeurIPS-2021 Cooperative AI Workshop

Robots must interact socially with humans. We present a computational framework that models social interactions as recursive Markov Decision Process (MDP). This formulation allows machines to understand what it means to help or hinder one another. Enabling robots to exhibit social skills could lead to smoother and more positive human-robot interactions. For instance, a more caring assistive robot. The model may also enable scientists to measure social interactions quantitatively, which could help scientists to study social impairements.

[Social MDP Website | MIT News]

Robots that Understand Linear Temporal Logic

Published in IROS-2020, CoRL-2020, Frontiers Robotics & AI

Robots must execute commands that are extended in time while being responsive to changes in their environments. A popular representation for such commands is linear temporal logic, LTL. We demonstrate a reinforcement learning (RL) agent which uses a compositional recurrent neural network that takes as input an LTL formula and determines satisfying actions. This RL agent also helps us learn programs that capture what sentences mean.

Robots that Understand Language

Published in ICRA-2020, Findings of EMNLP 2021

Executing natural language instructions from raw observations requires an integrated solution for language, perception, and planning. We demonstrate how an embodied agent uses reasoning about the future to follow natural language commands with an end-to-end compositional model trained on little data while generalizing to new environments.

[Tech Xplore | MIT News]

Multiagent Reasoning System for Commonsense Knowledge Integration

2010/08 - 2012/01, Published in ACM TiiS-2012

While commonsense knowledge bases have been used in many intelligent systems, the quality of output results is limited by the coverage of single knowledge base. This multiagent reasoning system features a planning-based approach to integrating reasoning methods from multiple common sense knowledge bases to answer queries. The reasoning results of one reasoning method are passed to other reasoning methods to form a reasoning chain to the target context of a query. It also provides methods for developers to access the integrated reasoning results in real time.

[Application Video in Video Navigation | Presentation in IUI-2013]

Language Explorer: Adaptive Language Learning Using Commonsense Knowledge

2011/10 - 2011/12, MIT Common Sense Reasoning for Interactive Applications Final Project

Collaboration: Stephanie Leung (Harvard)

Language Explorer is a mobile application that adapts to a learner’s context and capability. The Language Explorer leverages the crowdsourced commonsense knowledge base ConceptNet, the location based service Foursquare, and dialogues in online courses to automatically arrange materials and generate dialogues that fit the learner’s current situation.

[Final Paper]

Chinese ConceptNet & AnalogySpace

2009/09 - 2011/12, Published in IJCAI-2011

Collaboration: Rob Speer (MIT) on ConceptNet 5 Integration

Using the data collected from social games, this research creates the largest Chinese commonsense knowledge base (over one million sentences) in the world. Collaborated with MIT Media Lab, this knowledge base is integrated into ConceptNet and demonstrates its reasoning capability with AnalogySpace. In addition to reasoning with one language, this research also features algorithms to make analogical reasoning across languages and generate new questions for crowdsourcing by comparing inference results in different knowledge bases.

[Demo Video | Data available in ConceptNet 5]

PTT Virtual Pets: Commonsense Knowledge Collection Game

2008/06 - 2009/08, Published in HCOMP-2009

Collaboration: Bani Chan, Kai-yang Chiang, Rax Wang

Virtual Pets is a community-based question answering game built on PTT, the largest bulletin board system in Taiwan. The game creates mechanisms to utilize the rich interactions in the community to collect commonsense knowledge. The questions and answers in the game are exchanged between players to get more answers and verifications, which are shown as pets' homework. Using pets as virtual agents for community users, this game has successfully collected commonsense sentences and verifications since November 2008.

[Demo Video | Game available on telnet://ptt.cc ->(P)lay ->(C)hicken]

MovISee

2014/09 - Present

Collaboration: Yen-Ting Cho, Yen-Ting Yeh

MovISee is a digital software and platform for people to play and create individual games and videos with their own existing digital files. We use depth camera to create mixed reality for people to explore the selected information and ultimately transform understanding of people’s body movements as means to composite customized visual outputs.

[Video Gallery]

Cubic Film

2013/01 - 2014/09

Collaboration: Yen-Ting Cho, Yen-Ting Yeh

Cubic film is a system to layer film frames into pixel cubes and provide different scanning mechanisms for users to experience moving images in new ways/dimensions. Each new scanning line produces a row of picture elements/pixels and disrupts the recorded time and spatial relationship.

Sample cubic films: [1] [2]

Hand Painted Film Plus

2012/01 - 2012/09

Collaboration: Yen-Ting Cho, Yen-Ting Yeh

Hand Painted Film Plus is a digital interface to create film strips, allowing participants to paint with camera, re-configuring the created sequence to explore new patterns and experiencing the transformation between time and space. Participants are especially surprised to see how static images moved.

ScienceVR: Science Experiments in VR

2016/10 - Present, Finalist of 2016 AT&T AR/VR Challenge

Collaboration: Jackie Lee, Yero Yeh, Chih-Shiang Chou

ScienceVR is a VR experience for science education. We created an environment that allows users to do science experiments in VR. With real-time physics simulation, users can interact with virtual magnets and coils, watch them attracting and repelling each other, generate induced currents, and react to electrically-generated magnetic fields – all while seeing the magnetic fields through Michael Faraday's force lines. We believe learning science is like learning a new language. This immersive environment would be very helpful to learn sciences.

[Website | Faraday's Lab Demo Video | Introduction on AT&T Website]

4DSpaceTime: Navigating Memories in VR

2015/10 - 2016/06

Collaboration: Jackie Lee, Yero Yeh

4DSpaceTime is a Virtual Reality experience inspired by Interstellar's tesseract scene. We re-construct this VR space from photos/videos - one's memories. Time becomes a physical dimension for you to travel forward/backward in time. You focus on one of your friends and families' faces and lean forward - like riding a Segway - to travel forward in time. The space itself was constructed based on relationships of photos and videos.

[Demo Video]

TaiwanUXD.org

2014/10 - 2015/03

Collaboration: Jackie Lee

TaiwanUXD.org is a crowdsourcing platform for project owners to ask questions to community members. Through the user answers and profiles, the platform identifies secrets for project owners to learn user's behavior to refine their products. It is also a website to list the projects and members of Bay Area Taiwan User Experience and Design Group.

Flora: Mobile Flower Image Recognition Service

2010/04 - 2011/07, Gold Award of 2010 Hinet Telsoft Competition and 8th YuShow Cup Creativity Award

Flora is an iPhone application for ordinary people to identify what a flower is. A user just takes a picture of the unknown flower, the app will automatically send the picture to the cloud server, process its images, and then send all information about this flower back to the user. Flora also has a built-in flower card collection game to promote interactions with flowers. Whenever a user finds a flower, the user is awarded with a flower card to share with friends.

[Demo Video]

Automatic Panoramic Image Stitching using CUDA

2010 Spring, Parallel Programming Final Project

This is an application for finding panorama automatically. We implemented Scale-Invariant Feature Transform (SIFT) feature extraction/match, gain compensation, and multi-band blending in Compute Unified Device Architecture (CUDA). Using CUDA, we improve the performance of Panoramic Image Stitching nearly 10 times faster. Our system can build the panorama in one minute with 6 1000x800 pixel input images, whereas the sequential version takes about 12 minutes to compute the same panorama.

[Report | Slide]

SDIO Card with World Real-Time Clock/Alarm

2007 Fall, Digital System Design Final Project

This project simulates a Secure Digital Input Output (SDIO) card interface to control and display a world real-time clock on LCD. It supports three functionalities: real-time clock, alarm, and change time zone. The card can operate with battery only; there is no need to connect it to a SDIO host.

[Website]