Moving the sun in a photographed street is the hardest thing on this project's list, and the reason is simple enough to say in one line: the sun that was shining on the day of the capture is not lighting the scene, it is painted into it. Every shadow is a colour, not a calculation. Add a new sun at runtime and you get a street with two suns in it, the new one you can move and the old one you cannot.
So the first job is not adding light. It is finding the light that is already there.
The tool searches the sky for the direction whose shadows best explain the darkening on the ground, holding sky visibility fixed so that a wall in a corner does not get mistaken for a wall in shade. It never solves for the colour of the surface underneath, which sounds like a shortcut and is actually the whole trick: if you only ever ask for the ratio between the old light and the new one, whatever you got wrong about the paint cancels out. Exactly, when the sun stays put. Nearly, when it moves a little. Less and less, the further it goes.
Then I pointed it at the ten places in the game, and the result was not the one I had designed for.
Eight of them have no sun at all. They were captured under overcast sky, which is a light with no direction in it, and there is nothing to find because there is nothing there. One more, La Sarraz, has a weak hazy one. Exactly one place, Olsztyn, has a real sun: up 48 degrees, with the sun about three and a half times brighter than the sky, and when I overlay the shadows it predicts onto the capture they land on the real ones.
I had been building a subtraction tool. For eight of ten places there is nothing to subtract, and that changes what this work is rather than cancelling it. On an overcast capture every shadow you add is honest physics on a surface with no competing shadow already painted on it. The hard case, the one where you must remove before you can add, turns out to be the rare one. Olsztyn is the test: take its crisp baked shadow edge out and see whether a ghost of it remains. It does not.
Three things nearly fooled me on the way, and all three produced confident wrong answers rather than errors.
The first: my own fit called a street with no sun in it merely weak, because correlation alone cannot tell faint directional light from none. It needed a second test, asking whether brightness actually steps across the edges the predicted shadow draws. The second: two places answered that their sun was at 89 degrees, straight overhead. That is not a sun, that is the tool measuring how much sky each point can see, and any answer above about 75 degrees has to be thrown out. The third is my favourite, because it was a one-line definition: I was calling anything facing upward ground, which quietly swept in every roof, and with roofs in the set 60 per cent of the wall normals came out backwards.
None of this is in the game. It is offline, it runs on a processor rather than a graphics card, and what it produces is a copy of a place with the light moved. Whether that becomes times of day you can pick, or a sidecar that lets the sun move while you watch, is not decided.
The part I would not have guessed at the start is that most of this work is not relighting. It is finding out, honestly, whether there is anything in a given photograph to relight at all, and being willing to publish that the answer was mostly no.