First I don't know where you're getting your statistics from, 90%?
The last Standish report, which is a very pessimistic report according to many, but also respected by most organizations, says that in 2009 we had:
- 32% Success
- 44% Challenged
- 24% Failed
Which means that even if you consider a challenged project a failed project, the total failure rate is 68%. But a challenged project is not a failed project...
The difference between a challenged and a failed project is that the first project is done, and although it is over-budget/behind schedule/with reduced requirements, it is still done, and still serves the company and adds ROI. A failed project, on the other hand, is either canceled somewhere in the middle of the road, or is done, but does not add any ROI (or done but the resulting product or service is never used).
Now after this long introduction, let me answer your question on why 24% of IT projects fail. Here are the reasons:
- Software project management is maturing, but still not mature enough. While construction project management is a very mature process, software project management is still considered immature. There are many issues with a vague/non-existing process: How to handle requirements? How to manage uneducated clients (with respect to the relevant industry, for example, an investor wanting a website)? How to CLEARLY define the scope of the project? How to accurately estimate tasks (programming is not straight forward as constructing; there are some programming tasks that are very hard to do, often because of the inexperience of the programmer or the limitation of the programming language)?
- Software project managers do not have the same experience as construction project managers. While construction project managers are formally educated and carefully selected to manage construction projects, software project managers are often promoted from within the company (many of them were programmers before) because of their "outstanding" leadership, management, and communication skills. Sometimes they are given training, sometimes they're left on their own. The quality is just not the same.
- Several methodologies to choose from: Software project management has both the blessing and the curse of having more than one project management methodologies to choose from. Agile, Waterfall, XP, Scrum, etc... And none of these methodologies has yet proved itself as the ultimate solution.
- Crazy project managers: Yes, this is a reason. You have a lot of crazy project managers out there, and they're mostly in the software industry. Either they're fanatical to the process/methodology they like (regardless of whether the project adapts to it or not), or into company politics, or, last but not least, into playing (demotivating) tricks on their teams.
- Constant bickering between functional mangers and project managers: Because of the nature of most organizations, IT/Software project managers do not own their resources, functional managers do. This often leads to conflicts between the two (with the functional manager ultimately proving his power).
- Delicate resources: Programmers, unlike construction workers, are delicate creatures, they are sensitive, they have a huge ego, and they don't like being told what to do by someone they perceive as "ignorant" when it comes to knowing the things they do. Software project managers spend extra time and effort pampering their programmers. This leads to delays in the project, and of course, cost increases.
I can think of many other reasons, including scope creep, scope inflation, student syndrome, etc... but they all fall, in one way or the other, under one of the points I mentioned above.