The engine hit 9.91% median error this morning, down from 10.98% a month ago, and I was asleep for all of it.
This is an experiment, not a product, and nobody has ever been handed a number that came out of it.
What runs every night is what people call a Karpathy loop now. Andrej Karpathy wrote the Software 2.0 essay years ago saying most software would stop being code somebody typed and start being parameters somebody tuned, and tuning is a search problem, and machines search better than we do. He is also the guy who gave us vibe coding and the autonomy slider. Mine is set pretty far over.
So at 5am a cron job wakes up, reads what the last 90 sessions learned, runs a benchmark against 10,000 closed sales, finds the worst misses, changes one thing, and measures again. Keeps it or throws it out. Writes down why. Then it does that nine more times and goes back to sleep.
Here is the part worth your time.
For most of the summer it was reporting 3.86% error. I was thrilled. I told people. I bragged a little, which should have been the tell, because when it’s too good to be true it’s too good to be true.
It was peeking. Five places in the code where the engine could see things that only exist after a house lists, which is the one thing it can never see, since the whole point is guessing before the sign goes in the yard. The worst one picked which comps to look at based on the actual sold price. It was choosing its evidence from the answer key.
Not going to lie, that stung.
And I found four of the five by hand. A dumb grep script found the last one, sitting right where I had already looked twice.
Taleb put it best in Fooled by Randomness, we mistake luck for skill constantly and we don’t find out until the bill comes. Mine came as a benchmark log instead of a margin call. Honest number after the fix was 10.98%, a seven point swing the wrong way, and still the most useful number this thing has ever given me.
Now if the leak check fails the experiment gets reverted no matter how good it looked. Especially then.
A listing agent with the price sheet in front of them lands around 6%. We are chasing that blind, four points back, which is fair and a little humbling.
The reality is I do not know if this works yet. Ninety nights, a full point, every failure written down next to every win. It could flatten out at 9.9 and sit there forever and that is an answer too.
Either way I will write it down.
Be good to people.