by Joseph Herman
Motion capture, or MoCap, is the technique of capturing a human actor’s physical performance and applying the captured data to a 3D computer-generated character so that it moves in the same convincing, life-like manner. Unlike time-consuming 3D computer animation, in which an animator must manually manipulate the poses of a digital character’s skeletal rig (its underlying virtual armature), motion capture allows you to capture the movement of a character in real time. Computer animation, unlike motion capture, can take days or even weeks to accomplish.
So if motion capture is so fast, why not use it all the time? Why bother animating digital characters by hand at all? The answer to those questions lies in the kind of work you are doing. If you are looking for stylized, exaggerated or cartoonish motion for your characters — the kind of motion you would expect to see in a Looney Tunes animation — you’ll get the best results by manually animating them. Just be prepared to spend lots of time doing so. That is what the art of animation is all about.
However, if you want truly life-like movement, or want to integrate 3D characters into live-action shots alongside human actors (think motion pictures like 2009’s Avatar or 2001’s Planet of the Apes), then motion capture is your best bet. James Cameron was, in fact, an early pioneer of using motion capture in feature films when he used it as early as Titanic (1997). Today, it’s used extensively in the motion picture industry for visual effects.
One problem with traditional motion capture, however, is the fact that motion capture systems are often expensive and involved. Most of the time, they require actors to wear complex sensor-laden suits.
Let’s Go Markerless
Several years ago, a company called iPi Soft, based in Moscow, developed a motion capture system that was totally markerless. In other words, to the great interest of many who work in animation and visual effects, it was offering a new technology that didn’t require an actor to wear any suit at all — or even special clothing for that matter. Instead, the system analyzed the motion of the body and automatically extracted the MoCap data from the motion. It also had an option to work with depth sensors like the Microsoft Kinect or small, inexpensive video cameras such as the Sony PlayStation3 Eye or the Logitech C922 (see Figure 1).
I wanted to see how good iPi Soft Motion Capture worked so I got my hands on the depth sensor configuration, consisting of two Kinects, and downloaded iPi Soft’s software from its website: ipisoft.com. After jumping around and doing a little dancing, I fed the depth videos into iPi Soft to track my movements and applied the results to a 3D character.
iPi’s markerless motion capture worked as promised, without the hassle or expense of a MoCap suit. The era of “Motion Capture for the Masses” had begun.
iPi Soft Gets Real (Time)
Recently, iPi Soft released version 4.1 of its markerless MoCap system, and with it an important new feature for which users have been waiting a long time, something that has been available only in more costly systems: real-time tracking for multiple depth sensors (see Figure 2).
Previously, when using depth sensors, motion-capturing with iPi Soft was a two-step process. First, record your performance. Then track it.
While this method is effective, some users may prefer to see a real-time preview of their characters before and during the performance. The ability to track a 3D character in real time is important because you can immediately see what’s working and what’s not, and hone the actor’s performance accordingly. You can also check to see if the mesh is intersecting or colliding with itself, or practice your moves before recording them.
With version 4.1, that is now possible and it’s “an important development milestone that users have been anticipating” according to Michael Nikonov, iPi Soft founder and chief technology architect (whom I had the pleasure of meeting not long ago in New York). “We’re excited about real-time tracking with multiple depth sensors to obtain quality MoCap data easily and quickly without the need for multiple recording, tracking and motion transfer iterations,” Nikonov says.
Depth Sensors or Video Cameras
iPi Soft Motion Capture allows you to use multiple depth sensors or video cameras (or both). For depth sensors, iPi Soft supports such products as the Microsoft Kinect, Orbbec3D and Intel RealSense. It will also support the new Project Kinect for Azure (also known as Kinect 4) when it is released. For video cameras, iPi Soft Motion Capture supports web cameras such as the Sony PlayStation3 Eye or action cameras such as GoPro.
Which one should you choose when motion capturing with iPi Soft — depth sensors, web cameras or action cameras? The truth is, depending on your needs, each setup has its advantages and disadvantages. There is no “one size fits all” answer.
The benefit to using depth sensors is that they are easier to set up and calibrate and require less space due to their wider angle of view. The minimum required space is 7×4 feet. That means you can set up a few depth sensors in a confined space such as a small office, or even your living room for that matter. Perhaps one of the greatest benefits to using depth sensors is that the new real-time tracking for live preview in version 4.1 works with them (see Figure 3). It also doesn’t matter what color your clothes are.
While they can work in small spaces, a limitation to using depth sensors is that their capture area is limited to 7×7 feet. Thus, they are not your best choice if your MoCap performance requires sweeping movements over large areas. Depth sensors are also limited to a frame rate of 30 fps. While that’s fine in many cases, video cameras allow for faster frame rates, which can be useful when tracking extremely fast motions. Also, they cannot be used outdoors.
In the case of web cameras, such as the Sony PlayStation3 Eye and the Logitech C922, iPi Soft Motion Capture supports three to 16 cameras (see Figures 4 and 5). Compared to depth sensors, they allow for somewhat better tracking and their ability to record at 60 fps may avoid errors during very fast motions. Web cameras also allow for a much larger capture area, up to 20×20 feet (perfect for martial arts maneuvers or acrobatic dancers) and, importantly, allow you to track three actors at the same time.
In addition, motion tracking with web cameras can have less problems with bodily occlusion than depth sensors, especially if you use more than six cameras. They also allow you to do your motion capturing outdoors (but you should try to avoid direct sunlight). Web cameras are also inexpensive. A single Sony PlayStation3 Eye is a scant few bucks, so buying anywhere from three to 16 of them won’t cost much.
Setup and calibration, however, does take more time than with depth sensors, and you must have a computer with enough USB ports available — although you’ll only need half the number of ports as cameras (for example, three ports for six cameras and four for eight). You’ll also need a fast hard drive, such as an SSD. In addition, while web cameras offer more space for motion capture, they also need it. They require a minimum of 13×13 feet.
Like web cameras, you can use between three and 16 action cameras such as GoPro (see Figure 6). The benefit to using action cameras is that you don’t need external power when recording — only the camera’s built-in batteries. That means you can capture your motion in places where an electric outlet is nowhere to be found. In fact, you don’t even need a computer. Action cameras also have higher frame rates, such as 90 and even 120 fps, which is useful when doing very fast motion.
For action cameras, in version 4.1 there is a new feature that lets you automatically synchronize videos recorded based on flash detection. The new algorithm detects flashes between separate videos and synchronizes them automatically, greatly speeding things up. Previously, you had to manually sync the videos from the various cameras. The downside to action cameras, however, is that they require more effort to set up. But once you do, you’re good to go.
In summary, an iPi dual depth sensor configuration is practical, easy to set up and tracks many kinds of motions in a small space. It also takes advantage of the new real-time preview. For those who want to capture multiple actors, or need lots of space and complex motions, web cameras would be a better choice, though they require more learning in the beginning. Configuring multiple action cameras requires more setup, so they should be used if mobility is your foremost requirement or if you plan to capture extremely fast motion.
It should be noted that some users choose to use both Kinect sensors and Sony PS3 Eye cameras at the same time; the depth sensors are convenient for fast prototyping as to check creative ideas.
Conclusion
iPi Soft’s markerless motion capture solution is an easy and affordable way to create great looking and accurate motion capture without the expense and complexity found in other, suit-based products. The software also includes advanced algorithms for Jitter Removal and Trajectory Filtering, both of which will improve the quality of the motion. For those who wish to track props and hands, iPi Soft Motion Capture supports the use of motion controllers, such as the PlayStation Move.
iPi Soft Motion Capture is available in an Express, Basic or Pro edition. The Basic edition supports two depth sensors as well as three to six RGB web or action cameras, while the Pro edition supports up to four depth sensors and 16 RGB cameras. Both versions support real-time tracking for live feedback when using depth sensors.
The Pro edition adds support for multiple persons tracking, multiple video card support and batch processing. The Basic version costs $695 for a perpetual license, or you can rent it for $345 per year. Pro costs $1,995 with a yearly rental of $995. The Express edition, which only supports one depth sensor (probably too limited for professional use) costs $195 with no yearly rental option.
With wide applications in feature films, television productions and shorts, iPi Motion Capture is a great way to record motion capture performances. Whether you are creating a dramatic sword fight between armor-clad knights in an imaginative fantasy realm or a humorous droid-like character for a science fiction TV series, iPi Soft makes motion capture easy and affordable. More information can be found on iPi Soft’s website (www.ipisoft.com).
