Moviestorm Light - and Dark

A friend of mine who teaches film-making in Bristol, Phil South, tells his first-year students every year that making movies is about painting with light. When you strip it down to its fundamentals, what film-makers do is capture reflected light with a camera, and project that onto a screen. The first exercise he gives them is simply to take a stationary person on an empty stage, a single camera, and a single light, and see what happens on a monitor when you move that light and camera around. By lighting the person differently and picking different camera angles, you can create powerful images on the screen which have an innate story in the viewer's mind.

Of course, another way to look at this is to say that you're painting with shadows. It's as important to decide what doesn't get lit as to decide what you show. Moviestorm's had shadows in since the very early days, but it's still a long way from the sort of shadows you see in real life, and we know it.

Moviestorm pioneer Phil "Overman" Rice sent us an impassioned email a week or so ago on this subject. With his permission, here's the correspondence between him and Dave. You may find it illuminating.

Gentlemen,

We're starting to get to the point where we can make top of the line cinema with Moviestorm when we pay attention and edit properly. Screw pre-vis, I say. This thing officially has strong legs of its own, as films like [NAME REMOVED] prove.

There's really only one thing about this film that screams a jarring reminder that this is low budget animation: those goddamn flickering shadows. I've found the only reliable way to avoid them is to just turn directional lighting off, or at least so far down that shadows are barely visible. But look at what these guys did with shadows in this piece! Can you imagine this film being nearly as good without shadows?

No, we've got to have them, they are a crucial device.

What is the outlook on getting this bug fixed? Honestly, when it first came to notice, there were so many other bugs in Moviestorm that at the time it didn't particularly stick out (though I confess it's always bugged me). But now, esp. with 1.1.7, you guys are continuing to obliterate the imperfections of this movie-making tool, such that this 1.0 remnant shadow problem is really really standing out. And it's a bug that puts off regular viewers as much as (if not more so) than machinimator viewers, who tend to be much more forgiving of these kind of things. If we want to reach an audience beyond the pool of our fellow creators, we're going to have to stop implicitly asking them to lower their expecations when they sit down to watch our movies. The audience doesn't want to read "sorry about..." in the description before they watch, and the directors don't want to feel obliged to say it.

This film is, in every other regard, a hands-down jury candidate for this year's Machinima Expo. But we're really stuck in contemplation of whether to bother sending it up, solely because that godawful recurring visual glitch is almost certainly going to nix its chances with viewers who are just watching films and aren't educated on the day-to-day handicaps of this or that machinima platform. We aren't sending up anything else with that kind of visual bug, it almost doesn't seem fair to put it up there and hope a jury who we've asked to pick out the best of the best will just ignore such an obvious flaw.

We've got to stop doing this to Moviestorm films. What is causing the blocky and flickering shadows? What needs to happen to fix it?

I know you've got ten thousand things on your plate, things you want to do with the program. But this very old issue really has to become a renewed priority, if it isn't already. It's the kind of flaw that cannot be edited around, and turning off the feature that causes it has potentially dire consequences to the light and shadow vocabulary of film language. And the number of otherwise really stupendously great films it infects and spoils is only growing.

Please help us with this. Help us make the kinds of films that are going to have storytellers - even those with an image quality fetish - running to your software in droves. We are SO CLOSE to being able to snag the attention of beauty buffs, so very very close. This wart-on-the-nose-of-Angelina-Jolie is really the only thing standing in our way.

Your concerned user, friend, and advocate,
Phil Rice

Hi Phil,

I could not agree more with your sentiments. For all of its flaws I'm much more interested in machinima as the final product than as pre-vis. Right from the start I've wanted movies made in Moviestorm to be watchable for their own sake carried by the quality of the storytelling and without the visual quality spoiling the experience. Now I know we've got a long way to go but machinima has Moore's law in its favour: the hardware keeps getting more powerful and the software is slowly catching up. I've just played through Batman Arkham Asylum so I know we're behind the bar right now. I have to plead poverty for the moment but we have the virtuous circle on our side with good movies attracting new movie makers to Moviestorm who then go on to produce further good movies and slowly our community (and revenues) grows allowing us to build a better tool for making better looking movies.

We've made many improvements to the visual quality of movies but you're right about the quality of our shadows letting the show down. They've been bugging me also. Moviestorm's renderer has supported shadows from the start precisely because they are so essential to painting a good scene. I also want a whole lot more of course: HDR, global lighting, ambient occlusion, et al, but those are still for the future.

So, what can we do about them?

Firstly we use a shadow mapping technique to create our shadows. Effectively we take a shot of the scene from the point of view of the light This is pretty standard in state of the art game engines and our shadow quality actually compares quite well with most such engines. The difference though is that most games keep the cameras mostly in long shot but the good cinematic shots are close ups. The limited resolution of the shadow map is what causes the pixellated shadows. The shadow maps are anchored to the scene and so we get flicker as characters move through them (and even tiny movements are enough to move by a shadow pixel or two).

The other main shadow technique we could have used is to build geometric shadow volumes and use a stencil mask. However these have some major disadvantages for our use: they don't self shadow and the art has to be carefully constructed to make correct silhouettes. Generally shadow volumes have been rejected by games engines in favour of shadow maps for some time now.

Fortunately there are a lot of new tricks to be applied to shadow maps that could deliver most of the visual quality that we want for our movies - though in all honesty they'll still fall short of what you can get from an offline renderer.

Currently we use one shadow map for the whole scene. We take a lot of care to match the shadow map to the intersection of the camera and light view volumes and bias the pixel density to be higher closer to the camera but this is never enough for large scenes particularly when the light direction and the camera direction is opposed (shadows towards the camera tend to be the worst). This tracking happens frame by frame which is why you often see the shadow quality pop.

However a better tactic is to use multiple shadow maps - cascaded shadow maps are best current practice. Each shadow map might be relatively small but a smaller shadow map confined close to the camera can give a far better result than a large shadow map covering the whole scene. For example if we had a shadow map for each character close to the camera and a shadow map for the near scene and a shadow map for the far scene we should get much better results. This is quite a significant change to our renderer but one I'm actively pursueing in the rare moments I get to do some R&D.

In the shorter term there might be some ways to make our single shadow map more effective. One quick fix is to use a higher density shadow map (a decent card will support up to 4096x4096 but we've held back to 1024x1024 so far). This will help some (and I've tested it) but nothing like enough.

One trick used in game engines with static levels is to place the shadow casters manually to cover the area as effectively as possible. I don't really like this approach as it would require some complex tracking movements to get the best effects for close ups and this conflicts with the ease of use requirements for Moviestorm.

Another quick fix (but with its own drawbacks) is to soften the shadows further with a larger sampling mask. Effectively this puts a depth based blur on the shadows. This can look quite good on large scenes, the blurring hides much of the flicker, but it also loses any crispness in the shadows just where you might want them. Again it is a technique for higher end cards as it needs a lot of shader instructions to implement. But it could provide a worthwhile improvement in the right places.

So, let me end with a question for you as a movie maker: if you have to compromise on shadows, what are the most important characteristics you're looking for? Are soft shadows better/worse than hard shadows? Is the flicker the most annoying aspect or the coarse pixellation? Finally, is it enough to have high quality shadows in the final render even if the real time shadows are pretty rubbish?

Thanks for taking the time to raise this - sorry it's taken a little while to get back to you. I can't promise when you'll see improvements to the shadows, but know that I'll be doing what I can.

Cheers,
Dave

By way of explanation, I added illustrations to Dave's reply. As you can see, with a tiny set, the shadow on the wall looks just fine. As soon as we add in a large building, though, the shadows on the wall and on the character's neck become pixelated.

Thank you so much, Dave, for your detailed reply. This is a challenging issue, and one that I've seen games with multi-million dollar budgets and gigantic dev teams grapple with. Mass Effect has flickering shadows on some cards; GTA IV is plagued by the pixellated shadow edges even on a monster rig. So I can appreciate this is no trivial fix. And I am really grateful to learn that it bothers you guys too, that it's not an issue you've given up on. I can't tell you what a relief that is, just knowing that.

To answer your questions:
"So, let me end with a question for you as a movie maker: if you have to compromise on shadows, what are the most important characteristics you're looking for? Are soft shadows better/worse than hard shadows? Is the flicker the most annoying aspect or the coarse pixellation? Finally, is it enough to have high quality shadows in the final render even if the real time shadows are pretty rubbish?"

Soft vs. hard is a tough decision to make. It sounds like softening could be a way of reducing artifacts, but when the hard shadows are done right, boy oh boy, are they effective. Ideal world? I'd wish for an advanced user setting to switch between the two based on the situation. Forced to choose, hard shadows are better... but soft shadows without pixellated edges trump that for me. I'd wager no one but 3d graphics buffs are going to be distracted by a soft shadow when the scene's lighting might call for hard... but a jagged edge shadow can potentially distract everyone.

Flicker vs. Pixellation - Hands down, the flicker is the most distracting element... perhaps because it's so "foreground" in nature. The jagged edges can often be cropped out with clever framing... but the flicker often affects faces, which more often than not are essential to the shot.

Final Render vs. Real Time - One of Moviestorm's strengths right now is that affinity of the image at real time with what gets rendered. No surprises. HOWEVER, if the changes were in a positive direction such as with regard to shadow quality, yeah I'd be crazy not to love that. Would that mean a significant increase in final render time? (Honestly, that part doesn't matter much to me, I'm completely comfortable with a bit more wait for a tastier "bake".) In short, yes, I'd be fine with that and think most users would be as well.

Cheers, and thanks again,

Phil

What do you think? What do you want to see? How important is lighting to you, and how would it change your film-making if we gave you better shadows? Would you make different films? The same films but in a different style?