Breaking the Magic Quadrant Spell

When it comes to industry analyst firms, you should ask yourself this: Do you want a firm that tells you what you want to hear or one that tells what you need to hear?

With the former, you are essentially paying a so-called objective party to validate what you tell them to say. Not very constructive input, but hey, it sure is nice to hear. The latter is not always so comfortable, but that hard truth can help you improve your business. And that’s what industry analysts should be all about.

Over the July 4th weekend, I was thinking about a recent conversation I had with a CMO in the Silicon Valley. While he sees the benefit of ROI case studies and research that demonstrates value, he objects to the more nebulous reports that rank vendors based on ‘magic’ criteria. No one seems to understand what exactly Magic Quadrants measure, although many vendors have learned how to game them.

He shared with me how his team was able to push an older brand into the Leaders Quadrant for the first time by flooding Gartner with customer-win tweets. I’m guessing a healthy increase in the budget played a role too.

“Gartner is frustrating,” he shared. “It is essentially an industry tax.” He’s not the first vendor to tell me this. And yet so many vendors continue to highlight their position in the Magic Quadrant as a selling point. A metric that is unquantifiable, at least to the outside world. Secret at best. Aptly named Magic. Some sort of formula that weighs brand recognition, market share and future product roadmaps – and quite possibly the size of the budget spent with Gartner.

The real magic behind Gartner’s quadrants is of course fear. Most vendors are not willing to risk being excluded from the report, or pushed down to the dreaded lower left Niche Player quadrant. Many vendors complain, but begrudgingly pay the ‘tax’ to be included. We’ve yet to see that brave vendor step to the plate and walk away.

It reminds me of the SATs for college admission. There was a time when this test absolutely determined which students got accepted to top schools. A debate on the test’s fairness and ability to determine aptitude ensued for more than a decade before several brave universities started to discount the test. They looked at the scores, but it no longer determined their short list. Others followed suit and in fact some colleges don’t look at the scores at all now. This has allowed colleges to consider other, more important factors for admission. The SAT magic was broken.

There is another example closer to our industry – COMDEX. This was once the most important trade conference for technology and vendors would spend enormous amounts of money on their show presence – booths, signage, sponsorships, and activities. In the 90s, we saw a number of large vendors complain that they didn’t see a great deal of value, but felt obligated to be at COMDEX. They paid the ‘tax.’ It was a defensive move: Not being there was too risky.

In 2000 IBM, Apple and Compaq (now part of HP) decided to pull out of the show and reinvest that money into more effective marketing efforts. The show declined rapidly for several years before ending in 2003. Just imagine the ROI from the dollars saved by companies not attending COMDEX.

And shouldn’t every marketing dollar spent be about value? Customers are constantly evaluating vendors to ensure they get the most out of their investments. Shouldn’t vendors do the same with analysts? Are they getting the truth, or just what they want to hear?