Projects‎ > ‎


CartooNect: Sensory motor playing system using cartoon actions

 FUJIMURA Wataru1, IWDATE Shoto1, SHIRAI Akihiko1

1Kanagawa Institute of Techonology, Atsugi Japan

cartoonect at

Abstract—CartooNect is a VR game system that used KINECT by the motion of whole body based on child's imagination by manga actions. In our plan of experiment, children or adult draw a picture by hand drawing on paper that background and properties like the world in manga and/or picture book. When a player stand in front of the system, the player's whole body is displayed to player in a real time with the drawn picture as a background and they can move free to play as an actor in virtual world. The outline of the system is that the effect happens on the screen according to the operation recognized because KINECT connected with the computer does the operation decided beforehand while controlled by OpenNI. When player make a pose, the visual and the sound effect can be reproduced to decorate their imagination. This project tells a message to children to show and share their kiddy imagination using whole body expressions. Through the embodiment of their own hero and the heroine imaginations using a latest VR techniques, visitors can enjoy to create the world interactively, collaboratively, freely, then they may understand new function of virtual reality motivations is not only in the computer but also human imaginations.

Keywords-KINECT; Natural Input, OpenNI, Manga, Interaction, drawing, sensory-motor play


I.           Introduction

This project is focusing to find a new VR game system and its contents using current emerging video game technologies. Microsoft KINECT [1] (formerly Natal) is a most advanced device in this moment to detect player dynamic motions by a depth camera and its software.

As we did before at Nintendo Wii [2] and WiiRemote, a lot of hackers had tried to reveal and release the technology. However, it was rapidly shared via OpenNI[3] framework by the key technology suppliers, PrimeSence, Willow Garage and SideKick by free.

Then this project had thought again about what is a state of innovation then it took an orientation to realize “sensory motor play” in children imagination.

II.         Sensory motor play in children immagination

“Sensory motor play” is the first stage of Piaget's cognitive development [4]. It can be observed at 0 to 24 months period basically, but it constructs most important function of intellectual development through their playing naturally. It can be explained as obtaining the external world to make interact with something from the brain. Then this stage is not limited in a baby, childhoods to teenagers are also studying slowly when they face to a new behavior in the external world.

Piaget also defined “Symbolic Play” and “Rule Play” stages after the sensory motor stage in development. They are very important to develop human linguistic skill and logical thinking and relations but they are also important to work out a game play in current video games design.

In a classic video games design, it always based on rule plays to define winners or losers. Logical human interfaces like buttons and sticks were functioned as logical input to measure players' rapidness, preciseness and toughness in a virtual with some scenario, logics and renditions. It may affect to children’s sensory motors if their parents do not limit the playing time in their development stage. Because of obtaining the sensory motor, it can give an amusing sense breezy when children just run in a field. It means the gaming system should focus to move players than showing complex things.  This hypothesis can be applied not only for children but also adults in Virtual Reality entertainment system because they can be just a beginner like a baby in a virtual world in development.

III.        Playing using cartoon actions

Cartoons like Manga, Bande-Dessinee (B.D.) and picture books give a great part of imaginations to children. Now we have a various contents from these cartoons to media like cinema, video games and TV animations. However, these media are mainly based on consuming models and most media are non-interactive exclude video games. It sounds a normal sense but it might limit human imagination and innovation into a consuming model. Because children can draw their original worlds and personas by themselves thanks emulating some creations of cartoons.

This project focuses to apply cartoon actions in virtual world to enhance such a creativity and motivation. The aim is not to develop a character merchandising game. It should be shared and be motivated the human creations which include drawing, motion and reactions in a virtual world.

IV.        Technology

“CartooNect” is a code name of supposed system. It is a coined word of Cartoon and KINECT. “KINECT” is also a coined word of kinematics and connect. Then it might be a best word to explain the technical characteristic “Connecting into cartoon world using Kinect” for the public.

A.    Installation


Figure 1.   Concept sketch of project

CartooNect is very smart system. It is configured by KINECT, PC and a projector in 3x3m space. Through our experiments, it should keep 2.5m distance from KINECT to a player and each player should keep about 1m diameter or more. In our testing space, we had used an ultra-short range projector (NEC NP-U310WJD) with an aspherical mirror. It can be replaced by rear projection in an exhibition.

B.    Software and Algorythms

OpenNI is an open and emerging framework to create “Natural Input”. It contains communication drivers to KINECT OS “XIRON” via USB connection for several platforms. It also has useful APIs to handle human kinematics like skeleton structure, transformation functions from depth map to real coordinates and/or projection coordinates with actual sample code. However it glows day by day then it should be refer the latest information at the website ( to obtain the technical details.

In the alpha development, we had tried to develop our own algorithms to recognize human motions to detect cartoon hero style actions. It had designed as Figure 3.


Figure 2.   Example of  a rule based motion detections

Figure 3 describes a series of motion detection to realize homage of a manga “DragonBall fighting game. Through the development and testing, we had found it limits the player’s actions which just replay “pre-recorded actions” in a rule play using matching of motion. It is normal sense if we will apply well prepared model in a character merchandizing product but it is completely opposite with our concept. Additionally, the matching was also not worked well because it needs very precise motion in 3D and some player who has different shape could not control it well.


Figure 3.   Example of a figure caption. (figure caption)

Figure 4 describes our final implementation, “part kinematics based detection” in a same context. In the detection of “Guard” and “Crouch”, it compares a part of series of bones with a target motion in 2D coordinates using simple dot products. An evaluation function for a target motion multiples Cosines of angles between target bone matrices and current player’s bone matrices.  If each bone has very close value of angles, the output becomes to 1.0. This model can detect combination motion like “Crouching + Guard” by multistep values.


Figure 4.   Natrual condition in prototype application


Figure 5.   “Crouching” motion detected (face and background)

Figure 6.   “Guard” motion detected. (a picture overrayed)



Figure 7.   A combination detection and effect of Guard and Crouching.

Figure 4 7 are screenshots from our prototype application. To describe, they shows observing kinematics as lines on depth shadow of player. When it detects “Crouching” motion by legs, it changes the background and picture on the face. And when it detects “Guard” by arms, it shows a large picture on the player. Figure 7 is a result to play to make a combination effect of Guard and Crouching.

V.         Contents and Experience

We are improving the system to create a valuable experience through “CartooNect”, a code name of proposing content. It is a coined word of Cartoon and KINECT. (“KINECT” is also a coined word of kinematics and connect.) Then it might be a best word to describe the technical characteristic “Connecting into cartoon world using Kinect” for the public.

Figure 8.   Realized experience of “CartooNect”

Figure 9.   Top view of floor plan in Laval Virtual ReVolution 2011

Figure 10.    Scanning process to generate the scene.

Figure 11.    A scene of CartooNect (in development)

In the experience, the visitors can choose a world or draw it by themselves. Background, props and cartoon calligraphies can be put by hand drawing and scanning. This design has a role to make immerse players by commitment to generated world.

All target motions can be saved as effects with drawn textures, calligraphies with particle effects. This is very simple implementation but it allows creating new motion interaction between players.

Figure 12.    Players had invented their own harmonized action

Detected kinematics could be applied to a realistic 3D model but it looked not funny like a combat figure then we will develop a special character dynamic animation engine to improve CartooNect.

VI.        Conclusion

CartooNect had been developed in a very short time (approximately 2 weeks in training period). It will be tested in more actual playful conditions to add new functions and experiences. Through this project, we are touching a new generation which is sensory-motor oriented computer entertainment. This point of view is more important than just developing an application of KINECT content. We would like to continue to share our knowledge and methodologies into actual virtual reality industry and next generation of video games.


The project thanks to Taisuke YAMASHITA, Tomohiro MIYAGAWA, Yukua KOIDE, Haruka OGUMA and Motofumi HATTORI for their creation, management and encouragement.


[1]     Microsoft KINECTTM,

[2]     Nintendo Wii®,

[3]     OpenNI,

[4]     Tracy S. Kendler, Levels of cognitive development. Routledge, 1995 

[5]     TORIYAMA Akira, Bird Studio, “Dragon Ball”, Shueisha.

[6]     ARAKI Hirohiko, LUCKY LAND COMMUNICATIONS,  “JoJo’s Bizzare adventure”, SHUEISHA, 1987-2005. 













<Japanese description and updates below> 日本語解説がページの右側になります
If you need academic papers, here is link of PDFs. 関連論文は以下のダウンロードページにあります. 


"CartooNect" for Laval Virtual ReVolution 2011





「GAMIC: Game Action Motion Interaction Controller」ゲームアクション・モーションインタラクション・コントローラー.


白井研・Laval Virtualでの新作「PARAOKE:聴衆指向のカラオケエンタテイメントシステムのためのパラレル拡張現実」動画.
現在,Laval Virtual ReVolution2012での発表にむけ,開発中進行中です.