How much confidence should you have in your analyst?

How much confidence should you have in your analyst?

Gathering data is a common initial step in commercial decision-making. Words are the most common type of data collection (also called qualitative data, or unstructured data). To help product managers and sales personnel determine the greatest product design and the most successful message to send to clients, marketing researchers conduct focus groups, conduct in-depth interviews, or utilize open-ended questions in surveys. Additionally, HR managers conduct interviews with potential employees to help the organization choose the best applicant for a given position. The specialists that collected the data undertake an analysis of these terms after the data collection is complete.

According to recent research, experts' ratings of the reliability of health information on the Internet were examined (Craigie M, Loader B, Burrows R, Muncer S. 2002 Jan-Mar; 4(1): e2) to assess the consistency of experts' qualitative data analysis while analyzing qualitative data. An online message board was used to collect data from 18 threads (a series of linked posts) written by people with a chronic illness. One or more questions were posted at the beginning of each thread. Then, various people responded to those questions. 

Five clinicians who worked together in the same specialty unit and had at least five years of expertise treating the specified condition were involved in the data processing. The physicians established the following two scales in order to analyze the data. An A-to-F 6-point scale was used to classify the opening statement or question. A is great; B is good but lacking in specifics; C is poor but lacking in information; D is ambiguous; E is misleading or irrelevant; and F is unintelligible. D represents evidence-based, good; B represents accepted knowledge; C represents personal opinion; D represents misleading, irrelevant; E represents untrue; and F represents potentially harmful.

The Kappa, gamma, and Kendall's W statistical tests were used to compare the codes given by each of the five experts. The findings demonstrated a lack of agreement between the codes of all five experts when it came to the first inquiry and the subsequent answers. The codes given by two of the five experts were statistically significantly different, and alternative pairings of experts exhibited discrepancies in the codes allocated to the answers. One doctor labeled an answer "A = evidence-based, outstanding," whereas another doctor labeled the identical response "E = false," or even "F = potentially harmful." In basic words,

Here are some things to keep in mind:

This research was conducted by clinicians who have been treating the particular chronic condition for at least five years. Compared to even the most experienced market researchers studying qualitative consumer data or the most experienced human resource managers assessing candidate data, these analysts are much more knowledgeable about the research topic. Is there any hope that less-trained professionals will be able to consistently analyze their data if highly-trained specialists have shown inconsistent processing of qualitative data?

Answers were evaluated according to whether or not they were "evidence-based" (see code A). This is a measurable standard. The vast majority of business qualitative research, in contrast to this one, focuses on subjective factors like personal preferences, values, and morality. How can the less-trained professionals be expected to consistently use a huge number of subjective criteria when reviewing qualitative data if the physicians fail to consistently apply a single objective criterion?

It's understandable that you'd be concerned about having your focus groups studied by a market research firm. About 12,000 words are typically held in a focus group. There were a total of 18 threads analyzed in this investigation. Postings in a typical thread are roughly 120 words long. There were 10,800 words in this research, which means it was less than a single focus group's worth of information. In comparison, a typical market research study consists of four to eight focus groups, or four to eight times the amount of text required. It's very uncommon for a market researcher to collect vast amounts of data, but how likely is it that they'll be able to establish consistency if they only have access to a small portion of that data?

When a human resources manager is screening prospects, how concerned should you be? About 6,000 words are included in an interview transcript that lasts an hour (when hiring middle and top managers, the interviews might take a whole day with an order of magnitude more words). Interviewing a few people may generate 30,000 or more words of data (for five candidates). This research found that just two interviews had consistent results, so how likely is it that an HR manager would be consistent with a much bigger dataset?

As a client of an investment analyst, how much should you be concerned? Thousands of words may be found in an annual report. There are more than 65,000 words and 100 pages in IBM's 2004 annual report. When it comes to an investment analyst, how likely is it that they will exhibit consistency when they analyze a much bigger dataset (such as annual reports, financial statements, and press releases from a few firms) if the specialists in this research were unable to do so?

When two physicians were asked the identical question, they each gave a different code. Some answers were marked "A" for evidence-based and outstanding; others were rated "E" for false or "F" for potentially harmful. What's the truth? This is medicine, after all, and neither position can be correct. Do you know who to trust? You're in charge of making the final choice, so what should you do? As long as you feel the first doctor is correct, you should see the answer as excellent advice and follow its instructions. Run for your life if you feel that the second doctor is correct. In light of this, how can we rely on individuals who claim to be experts in their field if they are unable to interpret a tiny dataset consistently?