Editfest : Avater Editors
Coverage by Nobuhiro Hosoki
From the left, Editor John Refoua, Editor Stephen E. Rivkin
(Q): How did you get to be a motion picture editor?
Stephen Rivkin: Well I actually started, I went to film school in Minneapolis College of Art and Design and worked as a commercial industrial film editor for a number of years until I met a young film director who was trying to break into features. I cut a product line film for snowmobiles for him and he asked me to edit his little dramatic feature. So I juggled some commercial clients for a year and a half and made a little film that Roger Corman picked up, a little romantic film that released in 1980, called “Brooklyn.” And then the rest is a Hollywood kind of back and forth of working with this director every time he got a job and then I eventually moved out to Los Angeles and started working in the industry full-time.
John Refoua: I didn’t really know what I wanted to do until I was about 26, 27 years old. I’d be going around doing all sorts of odd jobs. Liberal Arts; that’s what that education does. And at some point, I know my wife never believes this, but it came down to I wanted to be a chef or a film editor. The main difference is if you put too much of something in with film editing, you take it out. I’d been working in television one-hour dramas; I did “Dark Angel. One of the directors directed the last episode, then he went on to do “Ghost of the Abyss” and he asked me if I would be the editor on it. My first movie was actually “Reno 911!” it was my summer job, I would go and do that in the summer, work on “Reno 911!” Those guys said “We’re going to do a movie,” and I went “Yeah, right.” And they said “No we are; we want you to cut it.” And that’s how it happened.’’
Stephen Rivkin: What’s well known is that James Cameron is a great editor, and he has an editing place.
John Refoua: Well we should actually say we are not the only editors of “Avatar.” James Cameron was the third person in our trio of editors and army of assistants, and Jim was very involved in the process and he likes to get his hands in there. He polishes scenes that we give him; there were scenes that he cut from the beginning. This was very unconventional in its approach. The film was cut at least three times; first in performance, a second in virtual camera, and third the traditional here’s the film, we have to do all the things we have to do to make a movie work. But we’ll take you through these steps today, which I think is very enlightening in terms of how this film was made, the methodology involved, because it is very unique.
(Q): Where did he? Did he teach himself how to edit?
John Refoua: He learned on “Strange Days.”
Stephen Rivkin: I think as a director, directors have to wear a lot of caps, and they learn everybody’s job. They seldom are good at everything but here’s a guy who is quite brilliant at wearing many hats. One of the actors said “About the only thing he can’t do is act.”
JR: Or sing.
(Q): How did people something “Strange Days”?
Stephen Rivkin: He basically learned how to do Avid. And then in “Titanic” he wanted to cut the whole thing himself and in “Avatar” he wanted to cut it also. He actually started editing before we came on and he turned over some sequences he had cut already.
(Q): Kubrick and Cameron.
Stephen Rivkin: He said he met with him. He actually had a sit-down meeting with him. I can’t remember at what point it was, but Jim knows a lot of people and are obviously able to get to people the normal person wouldn’t be able to get to. But just to back up one step, I think in terms of Jim learning everything, he’s worked with editors over the years and he’s credited learning from people like Mark Goldblatt, who cut a lot of his early films. The situations that you encounter in the editing room as a director are extremely beneficial to the learning process of how films are made.
And as editors we tend to know this stuff and other people don’t, but it is a crucial step in the way of making films. And the mistakes that are made in production are often realized at this point; “Oh my god, what did I do? Where did I go from here?” And many editors have saved scenes that were in trouble, and directors learn from this and they take it back to the next picture and they grow and they learn. The difference with Jim is that he likes to go to the next level. He likes to understand things to the point where he can do it, and as John said, at the beginning he thought he was going to edit “Avatar” all by himself.
It didn’t take long for him to realize, and he admits it, that this was just way beyond anything he had ever expected because of the process. John came on a couple of months before me, and then it became Jim had the scenes and he shot the virtual cameras and we were constantly busy preparing performance capture and all of these things. We could see a little bit of what we had to prepare when we get into it, but we wanted to show an introductory piece to get an overview, and then we can talk more and get into some specific things.
John Refoua: Jim’s a genius; he really is. He’s a creative genius, he’s a scientist. Even though he hasn’t had the formal education, he knows, he consults, he actually does. He can go in and take it one to one or one to 50 with UCLA scientist and then he can come in and make creative choices and creative decisions. He’s a unique individual. I worked on the movie two years and eight months, not that I was counting.
(Q): I just saw James Cameron on “Larry King.” He was on “Larry King” because he feels he has a possible solution to the oil spill.
John Refoua: Well he dealt with experts in underwater diving and photography and he designed all of the motor operated vehicles that the navy uses.
Stephen Rivkin: He went down to the “Titanic.”
(Q): On some level you’re happy but you’re also scared?
Stephen Rivkin: It was not so much fear, but there was a stress level I think. Because you have to understand; these shots that ended up in the film took as much as a year to complete. So we were in a constant state of deadline to turn over so many minutes every month that we were on the show, and that meant it had to go through the process of choosing performances, preparing them for camera, virtual camera, editing those sequences, getting Jim to sign off on them, and sending them to WETA.
And there were a whole bunch of other steps in between that are pretty complicated. We had our own 3D people analyzing the 3D, the virtual camera, adjusting, lighting, props, costumes, everything. And these scenes had to be delivered and I don’t know, we had like 10 to 15 minutes a month we had to deliver.
John Refoua: There was never enough time. We weren’t sure we were going to make the 2009 December deadline. It was very stressful as far as that. Sometimes that leads into fear, but that kind of goes together I guess.
Stephen Rivkin: You learn to work at a constant stress level.
(Q): What kind of equipment was invented for this?
Stephen Rivkin: It was mostly software that was done, so all the actually pieces of technology pretty much existed. The boom that they designed to go around the actors for motion capture of facial performance, that was new, but the actual technology of it existed. They wrote tons of software and they actually wrote the software to take that image, there was a little camera built into that boom, add a microphone and LED lights that lit up the face of the actor, and they had to write software for their capture process and then take that image and then turn it into a language that the computer could understand and could then project it onto the animated LED. So that process was sort of like an algorithm that they wrote, which they would then just plug in and you would see the face move. And then they had to tweak it a little bit because the structure of a Na’avi face is different than the structure of a human’s face.
(Q): You’re saying you worked really long and hard. How many hours a day? Like six days a week, seven days a week, 10 hours a day several days in a row?
Stephen Rivkin: It was six to seven days a week.
John Refoua: 12 hours at least.
Stephen Rivkin: That’s a short day.
John Refoua: 14 hours usually. He likes to work 14 hour days.
Stephen Rivkin: And I have to say, editorial traditionally has been nine to eight or whatever. Sometimes people work their own hours; they put in their 10, 12 hours. Because of the way the virtual cameras were shot, either John or myself would be on the stage when Jim was playing back actor’s performances and shooting with the cameras. And those days could start anywhere from seven, eight o’clock in the morning. It was very unusual for editors to be on the set during this type of production but it proved extremely useful because we had a lot of interaction with Jim.
We were there initially to interpret what these specific loads or parts of scenes were built for, because we had limitations on the system of how much we could play at one time. Depending on the number of characters or what was involved. It evolved into we started cutting these things as he was shooting them, streaming into the Avid, and he would review cuts in progress and say “Oh well what do you think? Maybe we need a close-up here, maybe we need that there?” It was very interesting.
John Refoua: The thing that we should try to explain to make it as clear as possible is there was a lot of action session in the movie, which is done in live-action. Now we’ve gone sort of half-way between the time when we started motion capture, performance capture, and then he went of to New Zealand for about four months and came back. The live-action 3D, it’s complicated enough as it is; there’s a lot going on there, you have to keep track of all the background place that will be there, and then the left eye and the right eye. We carried the left eye and right eye in live-action on our timeline.
So that was complicated, but that was basically not as complicated as all the other stuff that we did. The other stuff that we did had to do with performance capture. The first thing we have to do is we have to capture the performance of the actors; nothing happens until that. So we weren’t sitting on the stage for that process; that was Jim. Jim and his actors and all the computer guys basically doing whatever they’re doing. Doing two takes, 18 takes, however many takes it takes until Jim said “That’s the performance I want.” Now, he didn’t have to worry about, like Steve was saying, anything else; just the performance. Once he captured a performance, then it could come to us in editorial. Then we would go through and we would cut using the reference cameras that you saw.
They were all just basically guys, it’s not the angle that he wants, it’s just some guy trying to make sure he’s got Zoe’s face. And if Zoe movies he goes “Oh my god, I’ve got to go over here now.” It all changes, so it’s not representative of what’s going to be there, but that’s all we have. So we would cut the scene using all of this. The reason you saw the claws is that they didn’t want to overburden the system with too much storage. So it’s easy just to digitize it in as a clog and then we would blow up the section that we wanted. Then we cut that at first.
Then at that point in the movie you’re cutting it like any scene; we put sound effects in, we put music in, you put everything in. Now it’s just guys standing there on a green stage, but you pretend that they’re standing on Pandora and that they’re Na’avi. That’s how we had to do that in order to pick which of the 18 takes of each set-up that he wanted. Then we would turn that over to our in-house lab, they would take two, three months, they would put in all the digital stuff – the animals, the birds, the feathers and all that stuff – and then he would want to shoot what he called a virtual camera. So at that point we would sit on the stage because we had to explain why did we do it this way?
(Q): How long was it that you were actually editing those performances?
Stephen Rivkin: Ongoing.
John Refoua: It was ongoing because we would do performance capture on a particular scene or maybe five scenes, then he would stop down and we would edit for a while. And then he would go “Okay I want to do virtual camera for a while,” then he would do that. Then he’d say “Okay it’s time to do performance capture on this stuff now,” so it was an ongoing process. They were designing the sets, they were designing the environments, they were designing the trees. So there were a lot of things going on.
Stephen Rivkin: We had to, by necessity, feed Weta finished scenes. There was no way to go through all the capture, shooting all the virtual editing, and then turn it over; there would be no way to finish it. It had to be piecemeal.
(Q): This work load is ridiculously dimensional. Did you guys step into this work flow or were you consulted for it?
Stephen Rivkin: No, it was being kind of innovated as we went along. We had to figure out if the demand was Jim needs six scenes prepared for camera, or maybe it was more like 20 scenes. By the time he came back from New Zealand we had to have any number of scenes ready for him to shoot because he could look at a set and say this isn’t ready, this isn’t ready, and kick it back, and we always had to have scenes on deck. So it was a constant rotation of scenes that were in various phases of development.
(Q): What were you responsible for inventing in terms of process? Were you applying skills that you had developed elsewhere and you had all those skills, or were you actually re-formulating your skills?
John Refoua: I think that will be clearer as we go along, because it was in a process of how do we put all this together so that we can shoot a virtual camera on it?
Stephen Rivkin: That was new. I don’t think any of us, including Jim, had ever done anything like that before; worked with pure performances and select performances to be included in a scene. The beauty of this is that when it comes to shock creation that’s all he’s concerned about.
John Refoua: The hardest thing for me as an editor, and I think Steve shares in this, is that as an editor you come in, you sit down, you look at your dailies, you go okay this is a great close-up, this is a great wide shot, and that’s how you pick your performances. But you couldn’t do that on this movie, and you couldn’t tell what angle you’re going to be in. So when we’re doing our first pass doing a performance cut we just basically focused on what is the best performance. We couldn’t react in that manner that an editor does and says oh this is a great wide shot, I want to use this.
(Q): Could you give us a brief idea of what the editing suites were like? What equipment, how many systems you had?
John Refoua: Four walls, a window, and an Avid basically.
Stephen Rivkin: Jim had his Avid, we each had Avid, and we had a first assistant who ran the room, and probably three of four other assistants and a visual effects editor and two assistants, because there was a workflow of scenes going out to Weta and coming back. If you think in terms of once the performances, now keep in mind all those performances are stacked underneath the virtual cameras after they’re cut, and there was a very elaborate process of syncing dailies and making sure that everything that was contained in the virtual shot was stacked underneath. So at any moment Jim could click down and see the reference of an element. It may be a crowd, it may be a secondary character. Sometimes we combined one actor from take three and one from take four in the same take, and each of them would have their own track. It gets pretty crazy.
(Q): In terms of memory did you guys break a new record for how much memory you needed?
Stephen Rivkin: Well I think in terms of storage, we had a couple of cutting rooms that had to be kept in sync because we had Jim’s facility in Malibu, we had the stage in Playa Vista. At one time we had three because in New Zealand we had that going while they were down there because we were passing stuff back and forth. We ultimately moved on of them to Fox, where we spent the last few months.
John Refoua: We couldn’t have an assistant chasing audio at first, so I want to give credit to those who kicked ass on that movie and kept everything going.
Stephen Rivkin: We used an Isis, which I think we had up to 50 or so…
John Refoua: 42 terabytes at one point.
Stephen Rivkin: Yeah, it got bigger.
John Refoua: And then we actually separated each of our projects because each of our projects, the one project got bigger than five gigs and so we had to separate it. We had like six or seven of those going.
(Q): Was the low-res, pre-res in stereo or was it through the animation?
Stephen Rivkin: The initial camera that he shot and was streamed onto the stage was a center eye, which we called it. It was basically a one eyed version of that. And after the cut was approved we’d get stereo renders of it and we would look at it through stereo.
(Q): And what did you use to preview the stereo?
Stephen Rivkin: It was encoded onto a machine called an Aquity and we streamed it in RealD .
(Q): How easy will it be to re-implement this? Millions and millions of dollars have gone into inventing and investing in the technology so how soon can we expect other projects with this technology to come along?
Stephen Rivkin: They’re already happening. There are a number of films being done using this type of performance capture and facial capture. “Tin-Tin,” which is a film that’s in production now, there are a couple others.
John Refoua: “Real Steel.”
Stephen Rivkin: “Real Steel.” Films don’t necessarily have to be wall-to-wall; there could be a specific sequence that requires this type of technology.
(Q): The 3D aspect of the film was created in CG, not on the stage?
John Refoua: It’s easy to confuse those things. The camera is able to shoot in 3D, but because it’s CG you can basically say at this point I want to look at this eye. I center eye doesn’t really exist; there’s no center eye, but the computer can extrapolate it instantaneously right then and there. It’s live-action if you shoot it in 2D and then convert it to 3D, it’s a whole different concept than if you’re shooting a digital camera, a CG creation that’s a virtual camera. It’s a stereo capture; it’s all there.
Stephen Rivkin: Because the actors are in a three-dimensional space, the cameras above the volume are capturing their every motion in a 3D space. So basically, wherever that virtual camera is, it has a stereo viewpoint based on what your eyes would see. But we’re just monitoring one of those eyes when we cut initially because it’s just too much to stream.
John Refoua: But all the stereo decisions were made before anything was turned over to Weta to create a high-res version. All the changes as far as where the focal length is going to be, what the convergence of the two cameras are going to be, all those decisions are made before we turned it over to Weta. So they had all that information and that’s what they went on.
(Q): Did that change your methodology?
John Refoua: That’s one thing. I think on a live-action issues we had to be aware of convergence laws and keep account of that, because in live-action you couldn’t affect that too much. There are instances where you blow it up a little bit and change the convergence of the shot coming in and going out, but in CG we had the ability to change that after the cut. So we would do our kind of a pass on it, then we would make the convergence match our cut. So it was very advantageous that way.
(Q): Does everybody know how 3D works? Can you give them a quick?
(Q): That can get a little confusing.
John Refoua: Basically you have two eyes and there are a couple of different issues. As human beings we’re using to seeing things where you focus is where both your eyes converge. That’s an innate ability that we don’t even think about. But cameras have the ability to focus on one thing but the convergence is actually over there or over here, so you have to control that. By convergence I mean if you take your glasses off in a 3D movie you’ll see there are things that are doubled up and there is usually a part of the image that’s solid and looks like a 2D image to you; you can focus on it really nice and cleanly.
That is called convergence. That is where the two cameras basically are pointing, and it doesn’t necessarily mean that’s where the shot is focused. In “Avatar” we made sure that wherever the convergence was, that’s where the focus was also. In live-action we have to cheat a little bit; sometimes the focus was someone else but because the cut would work better if the convergence was here instead of there we can blow it up and move the shot basically. The left and the right eye you can adjust them to a certain extent. So that’s what I mean when I’m talking about convergence. That’s not to be confused with conversions; that’s a whole different thing.
Stephen Rivkin: Basically, you have these two lines and the glasses separate them to simulate your human vision in 3D so that there’s a constant fluctuation of left-right eye that is separated and you’re able to see things in stereo.
John Refoua: If you look at a shot in 3D and you’re focusing on something here, maybe in the foreground or the background, but the conversion of the shot is the opposite side, your eyes will feel uncomfortable. Your eyes will be able to tell that hey, my eyes would be comfortable looking over there but that’s not in focus, this is the part that’s in focus.
John Refoua: We’ll work very long hours and I joke this was not a job; this was an enlistment. It really was like enlisting into this process, and people did what they had to do. Unfortunately loved ones and families suffered and had to stand by and it was ultimately a very rewarding experience and it was very difficult, I’m not going to lie about it. So now you saw a little bit about how a simple scene with a couple of characters was captured.