What happens if a button input has no pull-up or pull-down resistor at all?

While the button is open, the pin is floating: connected to nothing, so it picks up noise and reads random 0s and 1s. The LED flickers, or turns on when you only touch the wire.

A digital input draws almost no current. That is what makes it good at sensing a voltage, and also what makes it unreliable when nothing drives it. A pin connected to nothing (high impedance, floating) has no defined voltage. Stray capacitance, nearby signals and even a hand near the wire move it across the threshold between 0 and 1.

The resistor fixes this by making one level the default. It is weak enough that the closed switch easily overrides it, and strong enough that noise does not. This is why every button input needs one, either:

• the internal pull-up or pull-down, enabled in software (the usual case on the ESP32), or
• an external resistor, for pins without internal pulls (GPIO34-39) or when a stronger pull is wanted, for example on long cables.

A floating input is easy to miss on the bench. The circuit may seem to work and then misbehave on another day or another board. If an input reads random values, check first whether anything sets its idle level.

Go deeper:

• Wikipedia — High impedance — what "floating" means electrically and why such inputs pick up noise.