Actually a project manager can delegate everything but accountability. It is the project manager's job to monitor that his delegated tasks are done properly and on time. The person to whom you delegated the tasks to is accountable to you, but never to the stakeholders. You, the project manager, is always accountable to the stakeholders even for the delegated tasks.
Yes you can delegate authority (if you have it), but not accountability, and yes, life isn't fair.
BUT, think about it this way, your stakeholders (to whom you are accountable) are accountable, for example, to the shareholders of the company. Are you? Life suddenly IS fair.