Light the photograph never saw

Written while this is being built. None of the relighting below is on by default yet, and the first pieces of it are in testing.

NVIDIA shipped DLSS 5 this month. It takes the frame a game has already drawn and repaints it to look photographed: the light on skin, the glow through leaves, the shadow where a wall meets the ground. It runs on the newest RTX cards, and it has moved the bar for what a game can look like.

We are working on the same picture from the other end. A game starts from shapes and textures and spends everything it has getting to something that looks real. Every place here starts as a photograph of a real street, so it looks real before we have done anything. What it cannot do is answer to light it never saw.

That is the relighting work, and it is the most ambitious thing we have tried. The goal is a photographed street that behaves like a lit game world. Shine a lamp into it and the real brick lights up and throws a real shadow. Park a car in it and the car sits in the same light as the wall beside it. Move the sun, and the shadows of the real buildings move with it.

Parts of that work today, in the workshop rather than in the game. Lamps shone into a captured town light the photographed surfaces themselves, brick by brick, with shadows. The sun that was shining on the day of a capture can be found in it, taken out, and put back somewhere else. The first pieces to reach the game are about the car: lit like the street around it, dimmer under a tree, in the shade of the buildings in Olsztyn. Those are in testing now.

So the bar is DLSS 5, and the aim is to go past it from this end: a real photograph that takes new light, rather than a drawn frame taught to pass for one. And it has to run where the rest of this runs, in a browser tab, on the laptop or the phone you already own, with nothing to install. As far as we can find, nobody has shown that. We are trying to be the first.

There is one rule on the way. Where the photograph is right, nothing is allowed to paint over it. Every step is checked against the capture it sits in, and anything that makes a real photo look less real does not ship.

The other half is time. Every place here is one instant: still air, light that never moves, and anybody who walked through the shot smeared into a wall or gone. 4D Gaussian splatting is the same kind of capture with time in it, a place as it was over a stretch of time rather than in a single moment.

That is where the rest of a living street has to come from: leaves in the wind, water that moves, cloud shadows crossing a square, a sun that travels across an afternoon, people who were really there. Drawn on top of a photograph those would be effects. Captured, they are the place.

Relighting and time turn out to be one project. A sun that moves needs a street that can take new light, and a street that moves needs light that follows it. The first thing both need is a photographed splat that can change on every frame inside a browser, at a frame rate worth having. Changing its light every frame works now. Moving it is the next measurement, and it decides how quickly the rest arrives.

We are keeping the details close for now, which is not how this log usually reads. What we will publish are pictures and numbers, side by side with the photograph, when they hold up.