How does challenge-response with a digital signature work for login?

Bob sends a random challenge in the clear. Alice signs it with her private key. Bob verifies the signature using her public key.

* Alice signs the challenge with her private key; Bob verifies with her public key — the modern default. *

Setup: Alice has sk_Alice; Bob has pk_Alice.

Login:

1. Alice → Bob: "I'm Alice".
2. Bob → Alice: random challenge (cleartext nonce).
3. Alice → Bob: signature = SIGN(sk_Alice, challenge).
4. Bob: VERIFY(pk_Alice, challenge, signature) → OK / NOK.

Why this is the modern preferred variant:

• Server doesn't have to do any encryption — just signature verification (cheap).
• The challenge is in the clear, so any eavesdropper sees it but can't produce a valid signature.
• Same as the encryption variant: server compromise doesn't leak Alice's secret.

This is the heart of:

• SSH public-key auth — server sends challenge, client signs with private key.
• WebAuthn / FIDO2 / passkeys — browser holds the key, signs server's challenge, and binds the signature to the origin so phishing fails.
• TLS client certificates — client signs the handshake transcript.

Tip: The fundamental phishing-resistance of WebAuthn comes from this scheme plus origin-binding — the signature includes the origin's domain, so a signature made for paypaI.com won't be accepted by paypal.com.

Go deeper:

• Digital signature — Wikipedia — signing with the private key, verifying with the public key, and why it proves origin.