There is a W3C presentation that helps frame the historical reasons for HTML5. The slides are available in plain text  and Technicolor. At the risk of confusing the facts,* I’ll try to explain why this is important and what is important about it.
For over a decade, XHTML1/1.1, a successor to HTML4, has been the most current version of HTML. XHTML1/1.1 leveraged the strengths of XML to create well-formed Web pages. These pages could be validated against a schema to test for compliance to a standard. Perhaps most importantly, it helped fix the issue of cross-browser incompatibility.
XHTML2 was going to be the successor to XHTML1/1.1; as the name suggests. However, this ended up not being the case. HTML5 is. Here’s why: XHTML2 was actually a different language; a new abstract approach to HTML. In at least the immediate future, making Web pages would’ve been made more difficult. It was a departure from the trajectory of many HTML traditions:
- IMG elements were being phased out in favor of OBJECT elements.
- The anchors, A elments, were being phased out because, “all elements may now play the role of a hyperlink.“ 
The objections to adopting XHTML2 were compounded by the fact that XHTML2 was not reverse compatible, by design. This meant that browsers that could already read HTML4 and XHTML1/1.1 could not read XHTML2. It kinda’ seemed like an effort by a consortium of smarty-pants engineers to force an idealized hypertext markup language onto the World Wide Web; with disregard for the immediate needs – and sights set on the long run.
Fortunately, there was outrage about all of this  and the W3C took a different tact. The next generation of Web pages would be made using HTML5, rather than a new markup language. HTML5 would incrementally change HTML, instead of completely overhauling it. It would be a forgiving syntax, one that anticipates that there will be deviation from standards. Instead of forcing compliance – it makes recommendations for how Web browsers should adapt. It also adds some new features.
*If nothing else, read this: http://coding.smashingmagazine.com/2009/07/29/misunderstanding-markup-xhtml-2-comic-strip/ Misunderstanding Markup: XHTML2/HTML5 Comic Strip by Brad Colbow.
- HTML5, XHTML2 – Learning from history how to drive the future of the Web: http://www.w3.org/2009/Talks/05-20-smith-html5-xhtml2/
- HTML5, XHTML2 – Learning from history how to drive the future of the Web: http://www.slideshare.net/sideshowbarker/html5-and-xhtml2
- XHTML™ 2.0 XHTML Image Module: http://www.w3.org/TR/xhtml2/mod-image.html#sec_20.1.
- XHTML™ 2.0 XHTML Hypertext Module: http://www.w3.org/TR/2004/WD-xhtml2-20040722/mod-hypertext.html#sec_10.1.
- Jeffrey Zeldman Present: The Daily Report – XHTML 2 and all that (The Sky is Falling): http://www.zeldman.com/daily/0103b.shtml#skyfall
- HTML5 Differences from HTML4: http://www.w3.org/TR/html5-diff/