How can I tell it failed? 4 things: Look at the rise of OSR during the 4th edition era. I see that as a sign of Old Schoolers who may have rediscovered the game in the 3.+ era, but then went further back when 4th went forward.
Second the rise to dominance of Paizo during 4th edition era as gamers who liked the 3.5 rules, were happy to keep playing them.
3rd, the demise of 4th edition products: nothing says failure like people not buying your game, right Radakai?
Last: Distributors, they confirm Pathfinder out sells D&D in their channels.
Net result: people still wanted to play D&D, just not the official game as currently presented by Hasbro.
So let's put you in the shoes of Joe Hasbro (my these are expensive shoes): You have an RPG property that:
1. Is known world wide
2. Has a mixed public perception
3. Has high costs to support: art, publishing, warehousing, staff, support teams
4. Does not have a solid repeat purchase model
5. Has lost market share
6. Is in a declining/decaying market category
7. limited mass market placement (book stores)...which is on very shaky footing.
So if I am Joe Hasbro, why would I want to make 5.0? Why wouldn't I license a company to produce the RPG for me....or maybe I just sell the RPG portion of the property outright.
The boardgames do well enough for me, and I could make those in my sleep. For me they are cheap to produce, and I can charge a premium for them. I have tons of miniature molds so I could dump out minis like there is no tomorrow into my games as modules. A couple maps, some cards, a few scenarios and I'll make that random monster in red, blue and green for different power levels or what ever the kids call them. I have enough clout to get these games into Target, Wal-Mart, and TRU. That's where the sales are! So I'll make the board games I am good at and get them placed where I know they will sell. We did D&D clue, so why not Greyhawk Risk...and Forgotten Realms Risk. LotR Risk worked quite well... Why not a D&D version of Trouble? or Dungeonland instead of Candy Land? Chutes and Ladders-how about Traps and Tunnels? What if Sorry was re-themed with magic spell cards? Can I keep giving away million dollar ideas for free? Anyway, as Joe Hasrbo, I'll let some other schmo can relieve his dreams of the 80's (checkered bandana not included.)
GW did it with Warhammer RPG, so let's assume it happens, and Hasbro licenses, or maybe sells, the RPG business to someone: Who would it go to, and would (could?) they cancel the OGL as part of the agreement to take it?