Written while this is in the working tree and behind switches. None of it is on by default.
The reason these places look the way they do is not that we light them well. It is that we do not light them at all. A captured street carries the actual light that was falling on it, bounced off real brick, through real leaves, at whatever time of day somebody stood there with a camera. A game spends its entire lighting budget trying to arrive at that, and we start there.
The advantage has an exact shape, and so does its cost. Everything we add to the photograph arrives with no light on it at all. The car does not cast a shadow onto the street. The street does not throw shade back across the car. The car does not reflect the road it is on, and the road does not know it is there. Each of those is small, and together they are the difference between a car in a place and a car on top of a picture of a place.
So the first thing is a shadow. A captured surface can now receive one from the sun, which sounds obvious and was not: a capture is millions of soft overlapping blobs rather than a surface, and asking each of them where it stands relative to a shadow is not a question the renderer is built to answer. It works. A box standing on a captured floor now measures 34 against 181 in brightness between the shade it casts and the ground beside it, where before the two were the same number. That is the first time anything we put into one of these places has changed the light in it.
The second thing is a confession. The renderer keeps a record of how far away each pixel is, so that effects can ask. The captured world was not in it. Not partly. At all. Every effect reading that record has been reading empty space where the entire town is, which means the ambient shading under the car was computed against a world that was not there, and the anti-aliasing was treating the whole street as though it were at infinity. I wrote a fix. Then I wrote a probe to check the fix, and the probe reported that my fix had also been doing nothing, for a different reason, on every frame since I wrote it.
It works now. I mention it because the useful output of that day was not the fix, it was the probe, and I would not have written the probe if I had trusted the first one.
The third thing is pixels. Modern games render fewer pixels than they show you and rebuild the rest from the last several frames, which is what the upscaling everyone argues about actually is. It suits us unusually well, because a captured town holds still and the camera is the only thing moving, which is the easy case. Ours has no neural network in it yet, and rendering at half resolution already measures 34.3 decibels against the full picture where simply stretching the small one gets 32.0, and it cuts the frame-to-frame shimmer on fine detail from 3.4 to 2.1. Whether it buys frames on a phone is the one number this machine cannot produce, and until a real device says so it stays a claim.
All three of those are about making a frozen moment behave. What they cannot fix is that it is frozen.
Every place you can drive here is one instant. The air is still, the light never moves, and anybody who walked through the shot is either smeared into a wall or gone. You can add weather to that, and we could, and it would be a drawn effect sitting on top of a photograph, which is the exact seam this whole project exists to avoid.
Four-dimensional Gaussian splatting is the same capture technique with time in it. Instead of a place as it was at one instant, a place as it was over a span of them: the leaves moving, the light shifting, the street doing what the street was doing. It is the difference between a photograph of a market and a market. Traffic and passers-by stop being models we add and become part of what was measured. Weather stops being a shader. A sun that moves stops requiring us to invent the light a moving sun would have made, because at some point during the capture it actually made it.
That is the version of this I want to build, and I want to be careful about how excited I sound. It is early. It is being tried separately from the demo, on its own captures, and the honest summary is that the file sizes are frightening and nobody has shown one streaming into a browser tab at a frame rate. The reason it is worth saying out loud anyway is that everything above, the shadows and the reflections and the reconstructed pixels, is work to make a still picture behave as though it were not one. This is the one that would mean it was not one.
It is being tried. It is not a date.