What are the drawbacks of isolated identities from the relying party's point of view?

The RP has to store the user's identity proof and other identity information and manage it securely, risking reputational damage on a breach, and it must implement expensive modern authentication such as two-factor methods itself.

For the provider, being its own IdP is a burden rather than a benefit:

• Custody of sensitive data: to check identities the RP must keep an identity proof (at least the credentials) together with other personal data. A leak of that store damages the provider's reputation, regardless of whether the shop's core business had anything to do with identity.
• Cost of state-of-the-art authentication: offering contemporary, more secure login (for instance SMS codes to the user's mobile number, or hardware-backed second factors) is expensive to build and run, and every isolated RP has to do it separately.

Delegating identity to a specialised provider removes both burdens, which is the economic argument for central identities.