Thursday 15 October 2009

Moviestorm Light - and Dark

A friend of mine who teaches film-making in Bristol, Phil South, tells his first-year students every year that making movies is about painting with light. When you strip it down to its fundamentals, what film-makers do is capture reflected light with a camera, and project that onto a screen. The first exercise he gives them is simply to take a stationary person on an empty stage, a single camera, and a single light, and see what happens on a monitor when you move that light and camera around. By lighting the person differently and picking different camera angles, you can create powerful images on the screen which have an innate story in the viewer's mind.

Of course, another way to look at this is to say that you're painting with shadows. It's as important to decide what doesn't get lit as to decide what you show. Moviestorm's had shadows in since the very early days, but it's still a long way from the sort of shadows you see in real life, and we know it.

Moviestorm pioneer Phil "Overman" Rice sent us an impassioned email a week or so ago on this subject. With his permission, here's the correspondence between him and Dave. You may find it illuminating.



Gentlemen,

We're starting to get to the point where we can make top of the line cinema with Moviestorm when we pay attention and edit properly. Screw pre-vis, I say. This thing officially has strong legs of its own, as films like [NAME REMOVED] prove.

There's really only one thing about this film that screams a jarring reminder that this is low budget animation: those goddamn flickering shadows. I've found the only reliable way to avoid them is to just turn directional lighting off, or at least so far down that shadows are barely visible. But look at what these guys did with shadows in this piece! Can you imagine this film being nearly as good without shadows?

No, we've got to have them, they are a crucial device.

What is the outlook on getting this bug fixed? Honestly, when it first came to notice, there were so many other bugs in Moviestorm that at the time it didn't particularly stick out (though I confess it's always bugged me). But now, esp. with 1.1.7, you guys are continuing to obliterate the imperfections of this movie-making tool, such that this 1.0 remnant shadow problem is really
really standing out. And it's a bug that puts off regular viewers as much as (if not more so) than machinimator viewers, who tend to be much more forgiving of these kind of things. If we want to reach an audience beyond the pool of our fellow creators, we're going to have to stop implicitly asking them to lower their expecations when they sit down to watch our movies. The audience doesn't want to read "sorry about..." in the description before they watch, and the directors don't want to feel obliged to say it.

This film is, in every other regard, a hands-down jury candidate for this year's Machinima Expo. But we're really stuck in contemplation of whether to bother sending it up, solely because that godawful recurring visual glitch is almost certainly going to nix its chances with viewers who are just watching films and aren't educated on the day-to-day handicaps of this or that machinima platform. We aren't sending up anything else with that kind of visual bug, it almost doesn't seem fair to put it up there and hope a jury who we've asked to pick out the best of the best will just ignore such an obvious flaw.

We've got to stop doing this to Moviestorm films. What is causing the blocky and flickering shadows? What needs to happen to fix it?

I know you've got ten thousand things on your plate, things you want to do with the program. But this very old issue really has to become a renewed priority, if it isn't already. It's the kind of flaw that cannot be edited around, and turning off the feature that causes it has potentially dire consequences to the light and shadow vocabulary of film language. And the number of otherwise really stupendously great films it infects and spoils is only growing.

Please help us with this. Help us make the kinds of films that are going to have storytellers - even those with an image quality fetish - running to your software in droves. We are SO CLOSE to being able to snag the attention of beauty buffs, so very very close. This wart-on-the-nose-of-Angelina-Jolie is really the only thing standing in our way.

Your concerned user, friend, and advocate,
Phil Rice


Hi Phil,

I could not agree more with your sentiments. For all of its flaws I'm much more interested in machinima as the final product than as pre-vis. Right from the start I've wanted movies made in Moviestorm to be watchable for their own sake carried by the quality of the storytelling and without the visual quality spoiling the experience. Now I know we've got a long way to go but machinima has Moore's law in its favour: the hardware keeps getting more powerful and the software is slowly catching up. I've just played through Batman Arkham Asylum so I know we're behind the bar right now. I have to plead poverty for the moment but we have the virtuous circle on our side with good movies attracting new movie makers to Moviestorm who then go on to produce further good movies and slowly our community (and revenues) grows allowing us to build a better tool for making better looking movies.

We've made many improvements to the visual quality of movies but you're right about the quality of our shadows letting the show down. They've been bugging me also. Moviestorm's renderer has supported shadows from the start precisely because they are so essential to painting a good scene. I also want a whole lot more of course: HDR, global lighting, ambient occlusion, et al, but those are still for the future.

So, what can we do about them?

Firstly we use a shadow mapping technique to create our shadows. Effectively we take a shot of the scene from the point of view of the light This is pretty standard in state of the art game engines and our shadow quality actually compares quite well with most such engines. The difference though is that most games keep the cameras mostly in long shot but the good cinematic shots are close ups. The limited resolution of the shadow map is what causes the pixellated shadows. The shadow maps are anchored to the scene and so we get flicker as characters move through them (and even tiny movements are enough to move by a shadow pixel or two).

The other main shadow technique we could have used is to build geometric shadow volumes and use a stencil mask. However these have some major disadvantages for our use: they don't self shadow and the art has to be carefully constructed to make correct silhouettes. Generally shadow volumes have been rejected by games engines in favour of shadow maps for some time now.

Fortunately there are a lot of new tricks to be applied to shadow maps that could deliver most of the visual quality that we want for our movies - though in all honesty they'll still fall short of what you can get from an offline renderer.

Currently we use one shadow map for the whole scene. We take a lot of care to match the shadow map to the intersection of the camera and light view volumes and bias the pixel density to be higher closer to the camera but this is never enough for large scenes particularly when the light direction and the camera direction is opposed (shadows towards the camera tend to be the worst). This tracking happens frame by frame which is why you often see the shadow quality pop.


However a better tactic is to use multiple shadow maps - cascaded shadow maps are best current practice. Each shadow map might be relatively small but a smaller shadow map confined close to the camera can give a far better result than a large shadow map covering the whole scene. For example if we had a shadow map for each character close to the camera and a shadow map for the near scene and a shadow map for the far scene we should get much better results. This is quite a significant change to our renderer but one I'm actively pursueing in the rare moments I get to do some R&D.

In the shorter term there might be some ways to make our single shadow map more effective. One quick fix is to use a higher density shadow map (a decent card will support up to 4096x4096 but we've held back to 1024x1024 so far). This will help some (and I've tested it) but nothing like enough.

One trick used in game engines with static levels is to place the shadow casters manually to cover the area as effectively as possible. I don't really like this approach as it would require some complex tracking movements to get the best effects for close ups and this conflicts with the ease of use requirements for Moviestorm.

Another quick fix (but with its own drawbacks) is to soften the shadows further with a larger sampling mask. Effectively this puts a depth based blur on the shadows. This can look quite good on large scenes, the blurring hides much of the flicker, but it also loses any crispness in the shadows just where you might want them. Again it is a technique for higher end cards as it needs a lot of shader instructions to implement. But it could provide a worthwhile improvement in the right places.

So, let me end with a question for you as a movie maker: if you have to compromise on shadows, what are the most important characteristics you're looking for? Are soft shadows better/worse than hard shadows? Is the flicker the most annoying aspect or the coarse pixellation? Finally, is it enough to have high quality shadows in the final render even if the real time shadows are pretty rubbish?

Thanks for taking the time to raise this - sorry it's taken a little while to get back to you. I can't promise when you'll see improvements to the shadows, but know that I'll be doing what I can.

Cheers,
Dave


By way of explanation, I added illustrations to Dave's reply. As you can see, with a tiny set, the shadow on the wall looks just fine. As soon as we add in a large building, though, the shadows on the wall and on the character's neck become pixelated.


Thank you so much, Dave, for your detailed reply. This is a challenging issue, and one that I've seen games with multi-million dollar budgets and gigantic dev teams grapple with. Mass Effect has flickering shadows on some cards; GTA IV is plagued by the pixellated shadow edges even on a monster rig. So I can appreciate this is no trivial fix. And I am really grateful to learn that it bothers you guys too, that it's not an issue you've given up on. I can't tell you what a relief that is, just knowing that.

To answer your questions:
"So, let me end with a question for you as a movie maker: if you have to compromise on shadows, what are the most important characteristics you're looking for? Are soft shadows better/worse than hard shadows? Is the flicker the most annoying aspect or the coarse pixellation? Finally, is it enough to have high quality shadows in the final render even if the real time shadows are pretty rubbish?"

Soft vs. hard is a tough decision to make. It sounds like softening could be a way of reducing artifacts, but when the hard shadows are done right, boy oh boy, are they effective. Ideal world? I'd wish for an advanced user setting to switch between the two based on the situation. Forced to choose, hard shadows are better... but soft shadows without pixellated edges trump that for me. I'd wager no one but 3d graphics buffs are going to be distracted by a soft shadow when the scene's lighting might call for hard... but a jagged edge shadow can potentially distract everyone.

Flicker vs. Pixellation - Hands down, the flicker is the most distracting element... perhaps because it's so "foreground" in nature. The jagged edges can often be cropped out with clever framing... but the flicker often affects faces, which more often than not are essential to the shot.

Final Render vs. Real Time - One of Moviestorm's strengths right now is that affinity of the image at real time with what gets rendered. No surprises. HOWEVER, if the changes were in a positive direction such as with regard to shadow quality, yeah I'd be crazy not to love that. Would that mean a significant increase in final render time? (Honestly, that part doesn't matter much to me, I'm completely comfortable with a bit more wait for a tastier "bake".) In short, yes, I'd be fine with that and think most users would be as well.

Cheers, and thanks again,

Phil

What do you think? What do you want to see? How important is lighting to you, and how would it change your film-making if we gave you better shadows? Would you make different films? The same films but in a different style?

13 comments:

twak said...

don't let dave sleep until it's this good

LivingRock said...

First of all, thanks for the new update! Although I have ran into a few bugs, the workflow speed increase is so nice.

In regards to the shadow issues - I think that the best option is to allow the end-user to select the level of soft v. hard edges. The same end-user options should be available between "real-time" rendering (as it is now) and a more intense (multi-layer) rendering process that would take more time but provide a higher qulaity end result.

I have a question about the jagged shadow edges. Can anything be done with antialinsing to reduce the shadow jags?

I am so excited about the future of MovieStorm. Thank you for all your hard work.

Satscape said...

I've mentioned it before, but I'll say it again now. It would be nice to have a 'final render' mode that takes 3 days/months to render, but looks photo-realistic and bloody fantastic, which may be getting away from the speedy near-live production of machinima, but would be nice all the same. This would include proper, realistic shadows and ray tracing, but that's maybe some way in the future, so a step towards that would be very nice. Having said that, the rendering quality in Moviestorm has improved over the last year.

IceAxe said...

I've tinkered with the shadows, and yes, I would have liked shadows without the jaggy edges. I'm not really so aware of flickering shadows though - hadn't noticed that.
The idea of "draft mode" shadows in real time with proper shadows in render would be acceptable to me.
I also think it would be cool if light didn't pass through solid walls. Yes, we learn to work around it... but it would make for better lightscapes. (Is that a word?)

sisch said...

Both the jagged edges and the flickering of the shadows make me often decide against using shadows at all - or doing a scene from a far off viewpoint with shadows, and a close-up without them (I did that in the darker scenes of Saving Grace). The flickering shadows on a face spoil everything for me, so I put them off (even if my heart bleeds for the loss of visual impact).

I can say that if the shadows were better looking, I would use them far, far more often, because shadows add so much to the atmosphere of a film!

I agree with Phil - I wouldn't mind so much if the shadows looked shitty in preview when they wou look great and crisp in the render!

AngriBuddhist said...

Since I've come to Moviestorm, the product has improved dramatically. It's appearance, ease of use, visual resolution, and functionality are all leaps and bounds above where it was 2 years ago.

All that being said, light and shadow (two things MS has always done), hasn't really improved much in that time.

From Overman's 1-2 year old pioneer post asking about directional lighting to Hugh Hancock's 1-2 year old suggestions about the same, on through every post about buggy shadows, I've read every post on the subject.

I agree with Phil, and since Dave is in agreement, Dave as well. Given a choice, soft shadows are preferred, flashing is the most annoying, give us the option for advanced shadow rendering, and ForgeUK hit it on the head, extra time for an 'extended final render mode' for crisp beautiful everything wouldn't bother me in the slightest. I'd actually love to wait longer if it solved this issue, yes please!

Chat Noir Studios said...

I'll add another vote to ForgeUK's suggestion of an advanced/enhanced rendering mode. That would be stupendous.

Ricky Lee Grove said...

Thank you for sharing these emails with the Moviestorm public. I think it's a sign of the commitment to the community (and to Moviestorm development) that you'd push the lighting/shadows issue to the fore instead of avoiding the problem (like many other software developers do).

Phil and I have just finished watching over 150 machinima films, many of whom were made in Moviestorm and the single technical problem with many of these films is the flickering shadows. And even though Moviestorm can produce some of the best looking scenes in machinima, they often lose believability when shadows are popping on the side of a face or on part of a characters costume.

I agree that soft shadows would be much preferable to flickering ones. Having the ability to choose between the two would be ideal. Also, the short term fix you suggest of using a higher density shadow map (or perhaps the ability to change this manually) could help with the problem until you come up with a longer term solution.

Good luck!

rampa said...

I would find the high quality shadows at render to be the most desirable. What we have is good enough for real-time preview.

I think that in light of the coffers not be bottomless at MS Towers, it is best to concentrate resources on the solution that will effect the most people the most strongly. Visual quality moves us all. Go ahead and add a few more goodies in at render time asd well. I know there is a lot of work going on towards filters that apply in post, so I say go ahead and make the shadows super awesome, and let a movie render in a longer time. The final render has never been in real-time anyway. Thats why our movies always will play at the proper frame rate. Remember how game based Machinima can slow down and speed up because its being recorded in real-time. SF chose very wisely to have the final render not real-time.

DavidB said...

how about plugging the Moviestorm output (geometry and lighting)into a nice ray tracing engine.

I can wait for it to take 30 seconds a frame on a G200

http://www.overclock.net/hardware-news/587936-youtube-nvidia-interactive-raytracing-engine-demo.html

or, as is 'coming soon', I would love to see what we could do uploading the scenes to the AMD Fusion Rendering Cloud

http://developer.amd.com/gpu_assets/ATIGPGPUComputingFusionRenderCloudGDC09CompatibilityMode.pdf

http://vgchartz.com/forum/thread.php?id=86846

Hugh said...

A word of caution - I've seen Machinima and Machinima producers abandon real-time rendering a whole bunch of times. It's rarely worked out well. Being forced to work in real-time is a significant part of the style - as soon as there's the possibility of a higher-quality render if you just wait a day, it becomes very hard not to use that as your default view, and use the real-time view just as a rough preview.

Having said that, things may have changed since the last time we went through this, or there might be a clever workflow workaround. I'm not saying "non-realtime is DOOOM!" necessarily, but it's tended to bog down any production that got involved with it.

Joel Lovell said...

I'm all for any tricks that make the final render as pristine as possible.

Because I go back and forth from my MacBook Pro 17" running OSX and a monster Dell PC running Vista, I have to use it at basic graphics level on the mac, then when i copy over the project on the pc I can crank it. Price of mobility.

Of course, my biggest distraction is the lack of content. I. Need. More.

Joel

Overman said...

@Hugh wrote: "A word of caution - I've seen Machinima and Machinima producers abandon real-time rendering a whole bunch of times. It's rarely worked out well."

Got any examples? I wasn't aware this was being attempted all that often.