In essence, it's pretty easy to tell if a coder improved an FTP client. It runs faster, it has more useful features, it crashes less often (if at all).
The same can't be said about table top RPGs. In the end, the results were something like you'd expect if Lucas somehow open sourced Star Wars: 5,000 permutations and modifications of the foundational material, none of which achieved wide acceptance compared to the original but all of which were embraced by someone.
The PDF Market: PDFs benefited immensely from the OGL, as they gave publishers a big market to tap into. In addition, the design of 3e facilitated short, cheap, but eminently useful designs, exactly the sort of stuff that helped PDFs establish a foothold in the market.
Sharing: Even if designers didn't improve each other's design, as I talk about below, they did swap stuff back and forth. Admittedly, this sells short the true potential of open source (iterative improvements driven by end users), but it was at least a start. It's possible that sharing is the best that open gaming can offer designers.
Training: This is likely the most underrated aspect of the OGL: it allowed freelancers to better migrate skills from one company to the next. Good freelance RPG writers and designers are in critically short supply. Anyone telling you otherwise has low standards. The OGL made it more likely for writers to build and sustain a skill set useful to a number of companies. By extension, gamers saw better designed stuff come from designers who could spend a few years working on the same game.
Iterative, User Driven *What* Now? We never saw a sustained effort to improve the fundamental rules of D&D, and it's debatable that any such improvement would be embraced as such by enough end users. That's the key, driving component of open source in software: the people using the software improve it. All those thousands of people, combined, see and fix problems faster and more accurately than an isolated team. Improvements propagate quickly throughout the user community.
In essence, open source finds problems faster, fixes problems faster, and spreads those fixes faster. A "problem" could be anything from a bug (your FTP client crashes when you try to upload a file) to spotting a gap in the features offered by a program (your FTP client can't upload multiple files at once, and your users would love to add that feature).
The crippling problem for open gaming is that no one can agree on what problems need to be fixed, no one can agree how to fix the problems that have been agreed on, and publishers want to profit from offering those changes.
In essence, gaming ran counter to three of the biggest benefits offered by using open source.
There was a time when I pictured an active community of designers, all grinding away on D&D to make it better. I think that happened, but only in a fragmentary manner. Some people wanted levels gone, others wanted hit points fixed (with "fixed" defined differently for each group).
At the end of the day, most people wanted books of monsters, character options, and adventures. Products either stuck with the baseline or created a new baseline for a fragment of the original audience to then stick to.
I don't think it's fair to say that open gaming was a failure, it just took a different path in gaming when compared to software. The important thing is that it got people to think like open source developers and act like them on an individual scale, even if we didn't see the same network of successive improvements, bug fixing, and distribution.
I think that, in the future, we'll look back at this decade as the time that a broad community of RPG players formally took on the mantle of designers. Open gaming, the indie movement, and PDF sales have made it more possible now than ever for a good GM with a knack for writing to put together a book and get it out there for others to see.
The one advantage of open source that we did leverage was in recruiting a far, far larger pool of talent. We might never have agreed on what needed improvement, and we never did put that OGC wiki together, but there are more people today designing and publishing RPG material than ever before.
That alone makes it a success. Tabletop RPGs continue to survive (dare I say thrive without kicking off a 4e flamewar?) precisely because of their DIY nature. Open gaming made more people into designers and publishers, and that's a good thing for this hobby, because that's the key, defining trait of what makes RPGs what they are.
So, that's my take on it.