IT Research Firms Get the Smackdown from the Wall Street Journal

Today, Lee Gomes' Portal column in the Wall Street Journal today entitled, "Vendors Still Paying For IT Research That Flatters Them" (subscription required) delivered a serious bodyblow to technology consulting and research outfit, The Aberdeen Group.  Gomes' essential point was that companies like Aberdeen (and I'd say all other big IT and Finance research firms) are far from unbiased and actually quite conflicted in the work they do. 

Let's first understand Gomes' arguments.  He writes about the old Aberdeen "which was for the most part a "pay-for-praise" operation.  If you saw an Aberdeen report saying that Acme MicroMacro sold world-class solutions, you could be sure that Acme had written Aberdeen a world-class check."

He then goes on to discuss how Aberdeen has repositioned itself by ending this practice and now doing "sponsored research."  He writes, "The new Aberdeen is better than the old one.  How much better, though, is the question, because the new reports also seem conspicuously flattering."

Basically, sponsored research lets firms pay ~$30,000 to Aberdeen who then conduct surveys with tech users.  The sponsors Gomes talked to said they like to sponsor a report because it provides a "chance to rise above the noise of the marketplace by being associated with something customers consider "research"."  Even sponsors know the research is of dubious value, but it gives them an air of credibility.

He goes onto describe the conflicts which are clear.  Aberdeen did 212 reports last year with 4-5 sponsors each.  He writes, "But if much of your top line is dependent on getting tech companies to sponsor your research reports, you've got quite an incentive to design questionnaires that will yield the kind of reports tech vendors will want to sponsor." 

"In that regard," Gomes writes, "Aberdeen delivers.  The reports seem to invariably discover that "best in class" companies use, or are thinking about using, or somehow embody, whatever technology the report happens to be discussing."

I hope more people will take heed of what Gomes is writing.  I've written about it before in my blog as there are too many inherent conflicts that such organizations have and so their advice is far from fair and balanced.  But when someone brings it up in the Wall Street Journal, it will get attention which it deserves.  I'm sure the PR machines of the big research firms are going into overdrive to demonstrate how unbiased they are, but there is no way to get around these massive conflicts.  Beyond research sponsorship, other conflicts include:

• Conferences are hosted with technology vendors sponsoring booths, dinners, etc.  In many instances, it is these vendors who speak at these conferences.  Why pay to goto a conference to hear a salesguy drone on when you could call him and he'll come to you? 
• There are entire magazines which claim to provide objective advice and "best practices" to readers but whose article authors are consultants and vendors pushing concepts that they "happen" to have expertise in.

As I have argued previously in my blog, (see one posting) you have to remain skeptical because objective research on the myriad self-anointed "best practices" is often not objective and the practices they're advocating are often far from best.  So before you use a report to help make the case to buy a piece of software or undertake an initiative to mirror some "best practice" within your organization, be mindful of the source.

Comments

[Industry Analyst]

Mr. Sanwal,

I just read your blog on the Aberdeen Group. I fear that the problems go deeper than just those firms that do surveys and research for vendors. I am an industry analyst and have seen some incredibly disturbing practices even in firms that are "independent." (luckily my present firm is quite strict on keeping us analysts honest and well separated from the segment of the business that does do vendor sponsored surveys).

In a prior analyst firm the official public position was that the analysts did NOT receive any compensation from vendors and did not work directly for them. Internally analysts were told to do what was needed to get vendor support but that it was to be kept secret. Being new to the analyst world I didn't accept any compensation (and little was ever offered!) from vendors that I wrote about. I was totally upfront with my clients in that I didn't have any vested interest in any of the firms that I wrote about.

I was introduced to the real world of analysts not too long after going into the area. In one situation I had written a detailed report and had praised the technology of one innovative company (now defunct). In that report I also mentioned several other small companies that were looking very promising at the time. After my note was published one client called me and grilled me on what my relationship was with those firms. I could honestly tell him that I had no financial interest in them at all and had received no compensation from any of them. I did tell him that I was inviting those companies to participate in a forum on their industry, but they were paying their own way to the conference. He then asked if any of the other analysts had any financial ties with these firms. I answered that no, our company didn't allow us to be financially associated with the firms that we reported on. Boy was I in for a rude awakening!

Shortly after this call with the client one of the firms mentioned in the report was purchased by a large vendor. At the next group analyst meeting I mentioned this as a point of interest. The research director stated that yes, he was aware of this and he'd made some money on the deal. He then stated that he had been on the advisory council of the firm that had been purchased and had been given stock in the company. I was furious as I had put my reputation on the line with a client when I told them that no one had any vested interest in these companies. I went to my manager who then explained that the company policy allowed such things, but they were NEVER disclosed to the public or our clients. I was clearly naive in this area.

I later discovered that the this senior member of the firm was in bed with numerous different companies as an "advisor" and tended to write nice and glowing reports on those firms with which he was associated. Over the next several years I found out that I was one of the few analysts that didn't take gifts or advisory positions with the firms that we reported on. I've not been with that firm for several years now, but I am still in contact with some of the analysts and, from what I'm told (second hand, clearly) this practice of lying to the clients is still in place and widely practiced -- especially by some of the research directors and more senior analysts.

The most that I've ever gotten from a firm is a dinner along with other analysts (well, about 5 years ago I got a nice home wireless router worth about $80, but I wasn't covering that technology at the time).

So, I guess the point of this rambling note is that even those firms that do claim to be vendor independent may simply not be providing accurate information.

Sincerely,
Industry Analyst

[alan pelz-sharpe]

I co-run a small analyst firm that works hard to take a strong ethical line. We do not write papers for vendors, nor do we consult to them, in fact we pay our own travel and accomodation costs at vendor!
We do this as our research business is designed to deliver indepth technical evaluations of vendor software to buyers. We believe our selves to be near unique in the analyst business by taking this route.
Sadly, many buyers hear the words 'Independent' and tar us all with the same brush. It is time that the industry really took this issue on seriously and starts to clean up its act.
We may be a small firm, but we are growing quickly and are profitable, proving that you don't need to depend on vendor hand outs to do well.
Best
Alan

Alan Pelz-Sharpe
Principal
CMS Watch