Project Neverland

Team TheatAR is a student pitch project at Carnegie Mellon's Entertainment Technology Center (Fall 2018) that is creating Project Neverland. It aims to create a theatrical experience in which a live actor and an animated character can share the same stage, in realtime, using augmented reality.

I’ll be focused on my contribution and process here, see our team website for the more complete process.

After a lot of iteration and experimentation with different kinds of hardware, we chose a story that would complement the strengths and weaknesses of wearable augmented reality as well as make a good case for the use of the technology in a theatrical setting. To meet those ends, we chose to recreate the nursery scene from Peter Pan. Peter and Wendy will be live actors, and Tinkerbell will be an animated augmented reality character. We are aiming to have Tinkerbell act in a believable way with both the physical set, and the actors.

After considering several possible Tinker Bell models, we considered character design and the inclusion of a game engine-ready rig for animation. We purchased a fairy model and outsourced biped and facial rigging to the amazing Sahar Kausar, so that I was free to collect reference videos and hammer out the animation pipeline between creating acting sequences in Maya, while moving and plotting the character route in Unity.

Gold Spike - Technical Pipeline Iteration

Animation Cueing System

Tinker Bell’s animation pipeline posed a particular challenge. Faced with the task of creating close to 2.5min of animation in 3.5 weeks, we made a list of what the pipeline needed.

• Flexibility - Animation implementation needed to accomodate rehearsal notes on a day-to-day basis

• Variety - A number of movement variants needed to be produced in a short period of time

• Human input - Cues needed to be manually delivered in order to

For her performance, she could not be completely animated in either Maya or Unity if we were to retain iterative flexibility within our scope and asset management, and her performance also needed to be relatively dynamic in order allow our actors to act freely. An unusual pipeline was developed in collaboration with our programmers and creative director, specifically for this project. Animation production resulted in an interesting role reversal of software; her flight path in Unity was treated like a cinematic animation done in a continuous sequence, and her body was animated in Maya in short gameplay-like cycles. These cycles would be layered on top of the flight path in Unity, and they could be played at the same time.

Maya Animation

In order to create animation as efficiently as possible, only two Maya files were maintained. One was made for locomotion, and the other was made for acting. There was a baseline idle made in each one, and then necessary animations were created in layers. In this way, elements like wing flutter and y-axis movement could be preserved.

These clips were exported into Unity, and incorporated into Unity’s Timeline feature. Maya clips were all integrated into their own track, and the flight path was animated in Unity itself, on a separate track.

Unity Animation

As the earlier prototypes were completed in Unity to nail down a ‘gold spike’ in the implementation process into Unity’s system, we built on that to further develop Tinkerbell’s performance animation. In order to have Tinker Bell feel as though she was constantly moving, even while she was waiting for a cue, we made use of Unity’s Timeline feature to orchestrate this process. Cues were scripts that were placed on activation tracks in Unity’s Timeline, which froze the Timeline and moved Tinker Bell’s animation to a state machine with specified idle moments for Tinker Bell’s mood at that point in the script.

There were a few goals that we were not able to hit for the way we handled animation in Unity; we realized that the cue markers needed to be bookended by idle states, and entering the state machine which handled Tinker Bell’s animation while waiting for the next cue. This was necessary because we did not have time to figure out a way to blend Tinker Bell’s animation in and out of cues smoothly while also using the Timeline in an older version of Unity (2017.4), so this was a solution which produced the smallest noticeable snapping between animation clips because she was in relatively the same position.

Halves Presentation - Week 9

After some debate, we decided to present a live demo at our Halves Presentation, which was closer to the 2/3 mark of the way through the semester.

Following some feedback and critique, we made some changes. We initially adjusted Tinker Bell’s dress textures to the color of autumn leaves while keeping the story (the original JM Barrie script) in mind, but we ended up pushing the color of her dress to be a saturated pink because we had issues in which either her body wasn’t appearing clearly in the hologram produced by the HoloLens, or the golden color was appearing too close to her skin tone and giving the impression she was unclothed. (Before and after color adjustments are shown below.)

Cueing Errors - Hololens Clicker Reliability

The biggest hurdle to our cueing system was that the HoloLens clicker was not completely reliable. We could expect that its failure to respond would occur at least twice per performance; however, we had no way of predicting on which cue this would happen. Our actors were incredible, and as we ran through rehearsals, they ended up adapting small improv moments for every cue to make sure that the energy of the performance never game to a standstill. To make sure that the light and sound cueing system for the actors never went ahead of the animation, we saw in the back of the theater. When cueing animation, I relied on seeing Unity’s timeline progress in a text box integrated by our programmers into the master build, not whether I had pressed the clicker.