The Chesire Cat in Alice in Wonderland said that it doesn't matter what road you are taking if you don't know where you are going. So it would be important to follow some methodology, even if informal. Or perhaps good practices instead of best practices.
Project still fail, imho, because we are human and do not always learn from our mistakes. As humans, we self-sabotage our work and our companies by not following sound best practices and finding ways to resist or get out of following a step in a methodology. We take short cuts because we are not offered infinite time. And in our fast paced world, time is at a premium. Best practices are adopting to this new reality, however they are still not perfectly followed. In a perfect world, if we faithfully followed the methods (rather than try to avoid them), we'd be doing much better.