Why does the recipient hand the sender a fresh public key shortly before the payment is signed?

So the sender can't secretly pre-build a longer chain in advance — the payment's details aren't known until the last moment, forcing any attack to start from behind.

A dishonest sender's double-spend works by mining a secret parallel chain that pays the coin back to himself, then releasing it once it overtakes the honest chain. If he knew the transaction ahead of time, he could work on that hidden chain continuously and strike when lucky enough to be far ahead.

Giving a new public key just before signing means the transaction can't be prepared early — the attacker can only begin his secret chain after the payment is made, already trailing the honest chain he must overtake.