As the project manager, you are the one ultimately accountable to the project sponsor for the delivery of the project. If you are working on a particularly large or complex project, you may have to delegate since there will be many tasks where others have the necessary skills. So in spite of disadvantages, you often do not have a choice.
When you delegate, you are entrusting someone to carry out the work in a manner consistent with the standards established for the project. You are giving up control, so you need to be sure the right resource is assigned and that the right standards are in place. For software projects, this may mean establishing local coding standards and an adherence to the methodology. In construction, there are local laws and regulations which must be met.
Delegation can also be abused (by the PM) if you throw too many tasks on one person. Likewise the people who receive the delegated tasks may abuse their power.
There is a possiblity of misuse of company assets, both tangible and intangible.
Where you are unsure about delegation, you may choose a more collaborative or perhaps more monitored approach.