Post by veu on Jun 18, 2023 15:21:12 GMT -5
Source: beforesandafters.com/2023/06/14/behind-the-bluescreen-shoot-that-brought-the-little-mermaids-underwater-scenes-to-life/
ARTICLES
Behind the bluescreen shoot that brought ‘The Little Mermaid’s’ underwater scenes to life
By Ian FailesPosted onJune 14, 2023
A step-by-step guide from production visual effects supervisor Tim Burke on the making of the underwater scenes in the film.
Rob Marshall’s The Little Mermaid features around an hour of underwater footage, including extensive song sequences involving Ariel (Halle Bailey) and a host of other mermaids and mermen, and sea creatures.
Scenes involving Ariel would largely be filmed with Bailey in a variety of rigs against bluescreen, with the idea that only her live-action head and arms would be retained in the final shots.
Everything else about the character was computer-generated, and of course sometimes she was a completely synthetic digital double.
To get to the bluescreen filming portions of the photography, production visual effects supervisor Tim Burke was part of months-long methodical pre-planning and choreography process, which he describes below for befores & afters.
Filming it like any other film
Tim Burke: Rob Marshall’s background is in musical theatre, he loves working with the actors on sets, but he knew the approach to this film had to be totally different to anything he’d done before. So I was very keen not to turn the whole process into an overwhelming technical exercise, instead I wanted him to be able to work with the actors freely, with as few limitations as possible, and as close to how he would normally work as possible.
Establishing mermaid movement
The first thing I wanted to establish was how Rob imagined a mermaid would move. What was his taste in terms of how fast she would swim, how she would perform different underwater actions, rolls, dives, tumbles etc. So very early on in pre-production we commissioned a set of animation tests where we built a simple greyscale mermaid without any designs to base it on. We weren’t aware that Halle was cast at that point.
Framestore animation supervisor Pablo Grillo and MPC animation supervisor Ferran Domenech were set the task of doing some animation studies, which they both came at from different perspectives. I worked with them to create something that I felt was of the right taste and feel for the film and then showed those to Rob and he responded incredibly well, he really loved them. He would say, ‘That’s too fast,’ or, ‘That’s beautiful,’ or he would give very specific notes. He’s a director who knows what he likes and being a choreographer himself, was able to communicate about movement very clearly.
Decisions about shooting methodology
Having established how we wanted the mermaids to move, I was able to break down how we would achieve the actual filming and technical aspects of the shoot. The animation studies really illustrated that when a mermaid swims her whole body is involved in creating the movement. She’s propelling herself through beats of her tail, which then translates through the body into the abdomen, chest, shoulders and head, which showed us that most of the character would have to be animated because of the physical restrictions of the actor being held in a rig.
Behind the shoot.
However, we were keen to capture the actual live-action aspects of the faces because I wanted Rob to be able to direct their performances live on set. There’s an hours’ worth of underwater material with the mermaids and Triton and different merfolk, and it felt like we could achieve quite a lot by filming the actors themselves, especially when you consider the amount of dialogue scenes involving close-ups and medium close-ups, so it didn’t make sense to go down the path of having to do everything completely digitally. Once the methodology was agreed upon I then asked special effects supervisor Steve Warner to design the various rigs that would allow us maximum freedom to move the actors on stage, but all being completely manually operated.
How rehearsals set-up the bluescreen shoot
Rob’s process involves rehearsing with a small group of dancers and his two main choreographers. He would block out scenes, especially the musical numbers, on the stage floor. He has a very basic process where he’ll represent set spaces by using cardboard boxes or taping out lines of a set.
He rehearsed a lot of sequences like this, which allowed him to block timing and get a feel for a scene. Dion Beebe, his DOP, would shoot these scenes using a Sony a7, which allowed them to have something that editor Wyatt Smith could then cut with.
The process was really hands-on from the very beginning, and they produced little edits of basic scene blocking for each of the musical numbers and some of the drama scenes. For other scenes, Rob storyboarded in a more conventional way and those boards were cut and put together with dialogue and music as well.
We ingested the cuts into previs and using animators from the facilities, expanded the design of the sequences by translating the movement from set into swimming actions, changing directions, having characters travel vertically, diagonally, roll or tumble, all of the things the actors couldn’t do on the set blocking. We developed a lot of the animation rigs for all of the characters through this process, discovering the kind of movement they’d need to do. We ended up with very sophisticated previs cuts for each of the drama and musical numbers. Then we broke the shots down into what could or could not be filmed as live-action.
Re-thinking rigs
In past experiences of using robotic arms or computer programmed rigs, I’ve felt that you see the actor being moved around physically by the rig, but you don’t get a sense that they are leading the movement itself. I was keen that we should allow Halle to drive the performance, so her upper body movement and head really directed where she was travelling, initiating the movement which the rig motion then followed.
(L-R): Scuttle (voiced by Awkwafina), Flounder (voiced by Jacob Tremblay), and Halle Bailey as Ariel in Disney’s live-action THE LITTLE MERMAID. Photo courtesy of Disney. © 2023 Disney Enterprises, Inc. All Rights Reserved.
So we designed different rigs that would allow her to have several different axes of rotation. Some of the tuning fork rigs had slip rings in them, so she could actually do 360 degree rolls. But the real key was that the rigs were being manually pushed around on stage. There was a base which would allow forwards and lateral movements on which there was a pivoted, adjustable arm. A small crew moved the base and arm around, then the actor, who was on the end of the arm, was moved by a team of puppeteers, who could assist with the physical movements.
The actor was generally picked from the hips and it took a lot of core strength for Halle to hold herself in position. But this allowed her to create a true feeling of underwater movement in the upper body, leading actions with her head and arms, and then the puppeteers were able to help her legs follow. This gave her a lovely flow of movement through her actual body.
We marked everything out on the floor with grids which allowed everyone to know where they needed to be at any one time. There were several weeks of rig rehearsals with all the actors doing their different scenes. If a rig didn’t allow an actor to move quite in the way they wanted to do, they could make adjustments, or we could decide to do that shot as a digital character.
When it came to the shoot we were so well prepped Rob could just focus on performance, he wasn’t having to think, ‘Well, should she be moving that way? Or is she going too fast?’ We trusted the system and that allowed him to direct the emotional performance of the actors. We had a six-week bluescreen dry for wet shoot blocked, and the process was so efficient we actually got ahead of schedule.
How much of the body to keep
There was no way we could use the body because the hips were fixed and weren’t moving in the way that a mermaid would. So we actually ended up using only the actors’ faces and hands for everything that was filmed live action for the underwater sequences. Each shot was body tracked and then animated over the top. Everything from the clavicle down became CG, and of course digital hair was added to every actor as well as the animated body.
There are a lot of very long shots in the film, many 30 seconds-plus, so often we’d use multiple live-action takes for one shot. For example, we might use a live-action element of Halle taking off, then transitioning to a fully CG character, then we’d pick it up again with another take, sometimes shot on a different rig for a different style of movement. We’d transition out of that again into another CG digi-double and then back into another live-action take. Some of these shots have three or four different live-action elements all stitched together with digital faces to cover the transitional periods in between.
The art of underwater hair
The other thing, which was fundamental to the dry for wet underwater approach, was to give the actors CG hair so that the movement could be controlled and styled in post.
Early tests showed that if we just let underwater physics move the hair as it should when the mermaid swam, or turned, or stopped, that it actually looked kind of ugly, often not what we wanted. It might cover the actor’s faces or it may flatten when moving quickly, which then made for ugly profiles. So we developed different hair simulations and different types of movement based on Rob’s taste and direction.
If Ariel was moving quickly, the hair would flow beautifully, but never straighten out, it would always have a little bit of soft movement within it. If she was delivering a dialogue scene, it would gently, in a non-distracting way, float with the subtle underwater currents around her. Rob never wanted anything to be distracting if you were in the midst of the dialogue scene between actors. If she was moving in a musical number we allowed the hair to really flow in a more expressive way, to capture the feeling and mood of the song.
Facial capture
The previs was broken down into three main categories, live action shot faces, live action to digital takeovers and fully digital characters. We used the ILM Anyma system on one of the stages at Pinewood for facial capture. We’d shoot two or three live action scenes on the bluescreen stage, and then we’d then take Halle, or Mellissa, over to the Anyma capture booth, and then repeat the scenes to capture the actors facial performance for the fully digital shots. We also reconfigured the Anyma rig with an extra camera to give us a wider capture volume for more expressive head turns.
We did this so we could capture and cover all of the performances again with Anyma while they were still fresh in the actor’s mind and while they were still ‘living’ those particular scenes. Because if you leave it all to the end, they’ve then got to get back into the moment of the scene which can be difficult for the actor and director.
The challenge of lighting
We had different lighting looks for different depths of water, using various caustic lighting rigs to help give interaction on the actors’ faces, which we would then match into and translate across their bodies. The caustic light was played stronger the closer we were to the surface and the lighting was obviously more diffuse the deeper we got. We captured all the details and data about the lighting rigs on set then recreated them as closely as possible to light the CG environments.
Halle Bailey as Ariel in Disney’s live-action THE LITTLE MERMAID. Photo courtesy of Disney. © 2023 Disney Enterprises, Inc. All Rights Reserved.
Dion and Rob wanted the world to be photorealistic and we referenced the Blue Planet series alot, along with hours of material our research team found. From our early lighting tests, we realised that we couldn’t be totally faithful to how light would really behave underwater because we would lose so much colour, depth and clarity. Color gets absorbed the deeper you go and red is the first colour to get absorbed at around 15 feet of depth, a problem for a mermaid with red hair!
So, we decided that we’d allow a lot more light to penetrate the depths than would happen in the real world and Dion lit with that in mind. He used a lot of bounce light, as well, especially in Triton’s throne room to give the idea of sunlight bouncing back off the sand surfaces, as well as the caustic light rigs.
One of the many tricky things to overcome was that the actors were wearing tracking skullcaps and that meant there wasn’t the light occlusion or shadows cast from their hair. It was a challenge to blend CG hair onto the faces and get the shadows to work because practical light would penetrate into areas of the side of the face, temples and cheeks where it shouldn’t. That meant there was a lot of clean-up work, blending digital patches onto the real faces to make them all appear as if they were lit in the same environment.
Creatures under the sea
We started doing animation studies based on the designs for each of the characters as they had been concepted, and soon realised that they were too much like the characters in the animated film, the eyes and lips were all a little bit too cartoony. Rob wanted everything to look photorealistic, they were supposed to be real animals who could speak. So we changed the designs, referencing a ghost crab for Sebastian, a sergeant major fish for Flounder and the northern gannet for Scuttle.
Then we got into the details of how they would speak, how would Sebastian talk? Crabs have got a very unpleasant looking mouth that would have made him very unappealing, so we simplified it. But we didn’t want to give him human lips, we didn’t want to anthropomorphize any of the characters, so he had to have a very simple basic mouth that would just open and close enough to deliver dialogue and emote a little. Scuttle and Flounder were less of an issue. Quite early in pre-production we did the sound recording for the three actors playing the characters., which was great because it gave us performances to work off whilst we developed the characters.
Awkwafina would give lovely little head and eye movements that you could translate into expressive moments for Scuttle. Daveed, as Sebastian, might raise a hand to make a point when he was saying something, and then we’d use that as a gesture to maybe raise a claw. There was not much Jacob could really do as a fish, unfortunately, but there was enough there from which Pablo Grillo and the team could work with and start to get character and performance.
We created quite sophisticated animation of all the characters in previs, so in the same way as we used the previs to drive the rig performance and movements for Ariel, we used the previs to drive the onset puppet performance of our creatures. Robin Guiver and his team of puppeteers used 3D printed versions of each of the characters–although Scuttle was actually a little soft puppet because she had to move in different ways. They were all on different rod mounts designed by Robin, the team clad in blue throughout the whole production.
The puppeteers learned the scenes, and everyone got together in the final stage of the rehearsals so that Rob could see everything coming together. It really helped Halle to know each of the puppeteers playing Flounder, Scuttle and Sebastian, so that she could build up a relationship and a rapport with them as if they were characters. It was lovely seeing them all working together on the set.
ARTICLES
Behind the bluescreen shoot that brought ‘The Little Mermaid’s’ underwater scenes to life
By Ian FailesPosted onJune 14, 2023
A step-by-step guide from production visual effects supervisor Tim Burke on the making of the underwater scenes in the film.
Rob Marshall’s The Little Mermaid features around an hour of underwater footage, including extensive song sequences involving Ariel (Halle Bailey) and a host of other mermaids and mermen, and sea creatures.
Scenes involving Ariel would largely be filmed with Bailey in a variety of rigs against bluescreen, with the idea that only her live-action head and arms would be retained in the final shots.
Everything else about the character was computer-generated, and of course sometimes she was a completely synthetic digital double.
To get to the bluescreen filming portions of the photography, production visual effects supervisor Tim Burke was part of months-long methodical pre-planning and choreography process, which he describes below for befores & afters.
Filming it like any other film
Tim Burke: Rob Marshall’s background is in musical theatre, he loves working with the actors on sets, but he knew the approach to this film had to be totally different to anything he’d done before. So I was very keen not to turn the whole process into an overwhelming technical exercise, instead I wanted him to be able to work with the actors freely, with as few limitations as possible, and as close to how he would normally work as possible.
Establishing mermaid movement
The first thing I wanted to establish was how Rob imagined a mermaid would move. What was his taste in terms of how fast she would swim, how she would perform different underwater actions, rolls, dives, tumbles etc. So very early on in pre-production we commissioned a set of animation tests where we built a simple greyscale mermaid without any designs to base it on. We weren’t aware that Halle was cast at that point.
Framestore animation supervisor Pablo Grillo and MPC animation supervisor Ferran Domenech were set the task of doing some animation studies, which they both came at from different perspectives. I worked with them to create something that I felt was of the right taste and feel for the film and then showed those to Rob and he responded incredibly well, he really loved them. He would say, ‘That’s too fast,’ or, ‘That’s beautiful,’ or he would give very specific notes. He’s a director who knows what he likes and being a choreographer himself, was able to communicate about movement very clearly.
Decisions about shooting methodology
Having established how we wanted the mermaids to move, I was able to break down how we would achieve the actual filming and technical aspects of the shoot. The animation studies really illustrated that when a mermaid swims her whole body is involved in creating the movement. She’s propelling herself through beats of her tail, which then translates through the body into the abdomen, chest, shoulders and head, which showed us that most of the character would have to be animated because of the physical restrictions of the actor being held in a rig.
Behind the shoot.
However, we were keen to capture the actual live-action aspects of the faces because I wanted Rob to be able to direct their performances live on set. There’s an hours’ worth of underwater material with the mermaids and Triton and different merfolk, and it felt like we could achieve quite a lot by filming the actors themselves, especially when you consider the amount of dialogue scenes involving close-ups and medium close-ups, so it didn’t make sense to go down the path of having to do everything completely digitally. Once the methodology was agreed upon I then asked special effects supervisor Steve Warner to design the various rigs that would allow us maximum freedom to move the actors on stage, but all being completely manually operated.
How rehearsals set-up the bluescreen shoot
Rob’s process involves rehearsing with a small group of dancers and his two main choreographers. He would block out scenes, especially the musical numbers, on the stage floor. He has a very basic process where he’ll represent set spaces by using cardboard boxes or taping out lines of a set.
He rehearsed a lot of sequences like this, which allowed him to block timing and get a feel for a scene. Dion Beebe, his DOP, would shoot these scenes using a Sony a7, which allowed them to have something that editor Wyatt Smith could then cut with.
The process was really hands-on from the very beginning, and they produced little edits of basic scene blocking for each of the musical numbers and some of the drama scenes. For other scenes, Rob storyboarded in a more conventional way and those boards were cut and put together with dialogue and music as well.
We ingested the cuts into previs and using animators from the facilities, expanded the design of the sequences by translating the movement from set into swimming actions, changing directions, having characters travel vertically, diagonally, roll or tumble, all of the things the actors couldn’t do on the set blocking. We developed a lot of the animation rigs for all of the characters through this process, discovering the kind of movement they’d need to do. We ended up with very sophisticated previs cuts for each of the drama and musical numbers. Then we broke the shots down into what could or could not be filmed as live-action.
Re-thinking rigs
In past experiences of using robotic arms or computer programmed rigs, I’ve felt that you see the actor being moved around physically by the rig, but you don’t get a sense that they are leading the movement itself. I was keen that we should allow Halle to drive the performance, so her upper body movement and head really directed where she was travelling, initiating the movement which the rig motion then followed.
(L-R): Scuttle (voiced by Awkwafina), Flounder (voiced by Jacob Tremblay), and Halle Bailey as Ariel in Disney’s live-action THE LITTLE MERMAID. Photo courtesy of Disney. © 2023 Disney Enterprises, Inc. All Rights Reserved.
So we designed different rigs that would allow her to have several different axes of rotation. Some of the tuning fork rigs had slip rings in them, so she could actually do 360 degree rolls. But the real key was that the rigs were being manually pushed around on stage. There was a base which would allow forwards and lateral movements on which there was a pivoted, adjustable arm. A small crew moved the base and arm around, then the actor, who was on the end of the arm, was moved by a team of puppeteers, who could assist with the physical movements.
The actor was generally picked from the hips and it took a lot of core strength for Halle to hold herself in position. But this allowed her to create a true feeling of underwater movement in the upper body, leading actions with her head and arms, and then the puppeteers were able to help her legs follow. This gave her a lovely flow of movement through her actual body.
We marked everything out on the floor with grids which allowed everyone to know where they needed to be at any one time. There were several weeks of rig rehearsals with all the actors doing their different scenes. If a rig didn’t allow an actor to move quite in the way they wanted to do, they could make adjustments, or we could decide to do that shot as a digital character.
When it came to the shoot we were so well prepped Rob could just focus on performance, he wasn’t having to think, ‘Well, should she be moving that way? Or is she going too fast?’ We trusted the system and that allowed him to direct the emotional performance of the actors. We had a six-week bluescreen dry for wet shoot blocked, and the process was so efficient we actually got ahead of schedule.
How much of the body to keep
There was no way we could use the body because the hips were fixed and weren’t moving in the way that a mermaid would. So we actually ended up using only the actors’ faces and hands for everything that was filmed live action for the underwater sequences. Each shot was body tracked and then animated over the top. Everything from the clavicle down became CG, and of course digital hair was added to every actor as well as the animated body.
There are a lot of very long shots in the film, many 30 seconds-plus, so often we’d use multiple live-action takes for one shot. For example, we might use a live-action element of Halle taking off, then transitioning to a fully CG character, then we’d pick it up again with another take, sometimes shot on a different rig for a different style of movement. We’d transition out of that again into another CG digi-double and then back into another live-action take. Some of these shots have three or four different live-action elements all stitched together with digital faces to cover the transitional periods in between.
The art of underwater hair
The other thing, which was fundamental to the dry for wet underwater approach, was to give the actors CG hair so that the movement could be controlled and styled in post.
Early tests showed that if we just let underwater physics move the hair as it should when the mermaid swam, or turned, or stopped, that it actually looked kind of ugly, often not what we wanted. It might cover the actor’s faces or it may flatten when moving quickly, which then made for ugly profiles. So we developed different hair simulations and different types of movement based on Rob’s taste and direction.
If Ariel was moving quickly, the hair would flow beautifully, but never straighten out, it would always have a little bit of soft movement within it. If she was delivering a dialogue scene, it would gently, in a non-distracting way, float with the subtle underwater currents around her. Rob never wanted anything to be distracting if you were in the midst of the dialogue scene between actors. If she was moving in a musical number we allowed the hair to really flow in a more expressive way, to capture the feeling and mood of the song.
Facial capture
The previs was broken down into three main categories, live action shot faces, live action to digital takeovers and fully digital characters. We used the ILM Anyma system on one of the stages at Pinewood for facial capture. We’d shoot two or three live action scenes on the bluescreen stage, and then we’d then take Halle, or Mellissa, over to the Anyma capture booth, and then repeat the scenes to capture the actors facial performance for the fully digital shots. We also reconfigured the Anyma rig with an extra camera to give us a wider capture volume for more expressive head turns.
We did this so we could capture and cover all of the performances again with Anyma while they were still fresh in the actor’s mind and while they were still ‘living’ those particular scenes. Because if you leave it all to the end, they’ve then got to get back into the moment of the scene which can be difficult for the actor and director.
The challenge of lighting
We had different lighting looks for different depths of water, using various caustic lighting rigs to help give interaction on the actors’ faces, which we would then match into and translate across their bodies. The caustic light was played stronger the closer we were to the surface and the lighting was obviously more diffuse the deeper we got. We captured all the details and data about the lighting rigs on set then recreated them as closely as possible to light the CG environments.
Halle Bailey as Ariel in Disney’s live-action THE LITTLE MERMAID. Photo courtesy of Disney. © 2023 Disney Enterprises, Inc. All Rights Reserved.
Dion and Rob wanted the world to be photorealistic and we referenced the Blue Planet series alot, along with hours of material our research team found. From our early lighting tests, we realised that we couldn’t be totally faithful to how light would really behave underwater because we would lose so much colour, depth and clarity. Color gets absorbed the deeper you go and red is the first colour to get absorbed at around 15 feet of depth, a problem for a mermaid with red hair!
So, we decided that we’d allow a lot more light to penetrate the depths than would happen in the real world and Dion lit with that in mind. He used a lot of bounce light, as well, especially in Triton’s throne room to give the idea of sunlight bouncing back off the sand surfaces, as well as the caustic light rigs.
One of the many tricky things to overcome was that the actors were wearing tracking skullcaps and that meant there wasn’t the light occlusion or shadows cast from their hair. It was a challenge to blend CG hair onto the faces and get the shadows to work because practical light would penetrate into areas of the side of the face, temples and cheeks where it shouldn’t. That meant there was a lot of clean-up work, blending digital patches onto the real faces to make them all appear as if they were lit in the same environment.
Creatures under the sea
We started doing animation studies based on the designs for each of the characters as they had been concepted, and soon realised that they were too much like the characters in the animated film, the eyes and lips were all a little bit too cartoony. Rob wanted everything to look photorealistic, they were supposed to be real animals who could speak. So we changed the designs, referencing a ghost crab for Sebastian, a sergeant major fish for Flounder and the northern gannet for Scuttle.
Then we got into the details of how they would speak, how would Sebastian talk? Crabs have got a very unpleasant looking mouth that would have made him very unappealing, so we simplified it. But we didn’t want to give him human lips, we didn’t want to anthropomorphize any of the characters, so he had to have a very simple basic mouth that would just open and close enough to deliver dialogue and emote a little. Scuttle and Flounder were less of an issue. Quite early in pre-production we did the sound recording for the three actors playing the characters., which was great because it gave us performances to work off whilst we developed the characters.
Awkwafina would give lovely little head and eye movements that you could translate into expressive moments for Scuttle. Daveed, as Sebastian, might raise a hand to make a point when he was saying something, and then we’d use that as a gesture to maybe raise a claw. There was not much Jacob could really do as a fish, unfortunately, but there was enough there from which Pablo Grillo and the team could work with and start to get character and performance.
We created quite sophisticated animation of all the characters in previs, so in the same way as we used the previs to drive the rig performance and movements for Ariel, we used the previs to drive the onset puppet performance of our creatures. Robin Guiver and his team of puppeteers used 3D printed versions of each of the characters–although Scuttle was actually a little soft puppet because she had to move in different ways. They were all on different rod mounts designed by Robin, the team clad in blue throughout the whole production.
The puppeteers learned the scenes, and everyone got together in the final stage of the rehearsals so that Rob could see everything coming together. It really helped Halle to know each of the puppeteers playing Flounder, Scuttle and Sebastian, so that she could build up a relationship and a rapport with them as if they were characters. It was lovely seeing them all working together on the set.
