Why is it insufficient to tell good from bad actors and actions only by their network address?

Because it ignores insiders, network addresses are easy to fake (spoofing), and permitted protocols can be misused.

1. Insiders: the model assumes everyone inside is good. An employee with bad intent, or malware on an internal machine, has an "inside" address and is trusted by default.
2. Spoofing: a source address is just a field in a packet. An attacker can forge it to look like a trusted network.
3. Protocol misuse: the firewall allows, say, HTTPS or DNS because the business needs them. Data theft or command-and-control traffic can travel inside exactly those allowed protocols.

In terms of the Basic Cyber Security Model (only good actors doing good actions in a desired environment), the perimeter gives a cleanly defined system boundary, but the address is a weak proxy for "good actor" and the port a weak proxy for "good action".