When Della Creelman received the invitation from Television Preview to a screening, she thought it sounded like a fun evening. Creelman figured she'd get to see some pilot episodes for potential TV shows and give her opinions.
She got to see TV pilots all right.
Bad ones, with grainy video quality, that were filmed a decade ago. The thick cell phones with extendable antennas and an outdated laugh track were just some of the clues that she wasn't watching the next "Heroes."
The object wasn't for the audience of more than 150 people to pick the next big TV hits. It was to get their reactions to the ads embedded in the shows for such companies as Wal-Mart, Iams pet food and Puffs tissue.
"It was dishonest. It was totally bogus," said Creelman, who had taken her 16-year-old grandson to the screening as a treat. "Everything was so cheesy. I would have never come if I knew it was to watch TV commercials."
Creelman and the others who had queued up early to get into the Television Preview screening are not alone in spending 2 1/2 hours watching outdated shows and answering survey questions about dandruff shampoo and dog food.
For years, Television Preview has been luring would-be viewers with promises of prizes and the chance, as the invitation reads, to "directly influence what you see on television in the future." But a Web search reveals numerous annoyed Television Preview attendees who have written about their experiences, which closely match those of Creelman.
Television Preview, despite the Hollywood address on its letterhead, is run by ARS Group, an Evansville, Ind., company. Competitors and industry insiders say that ARS, which has been in business for almost 40 years, is a top advertising research company.
A RATINGS BOMB - In its invitation to prospective viewers, television Preview implies it will show pilot episodes of potential TV shows. In fact, most of the shows were filmed a decade ago. The real objective is to gauge reaction to commercials. CNS Photo.
The company declined to comment for this story but on its Web site lists a variety of research services it uses to measure the effectiveness of advertising in print, radio, television and the Internet. In past interviews and in white papers, ARS said it has worked with companies or agencies of big-names companies such as Procter & Gamble, Pfizer and General Motors.
Despite the company's reputation, some industry experts say that Television Preview's test experience may be unethical and also ineffective.
Bob Barocci, president and chief executive of the Advertising Research Foundation, a trade group for the industry, said he was surprised that a company of ARS's stature would run an event like Television Preview's screening.
Barocci said the way people are lured to the event with unclear language - which, for instance, promises participants they will be evaluating "not-yet released television material being considered for nationwide broadcast" - goes against the industry's best practices. The foundation's guidelines state that questionnaires should not mislead possible respondents about their purpose. While ARF says it is OK "to disguise certain specifics about the study," it should be done sparingly and should not deceive respondents deliberately.
"The whole process sounds really awful and whether it will produce good results sounds highly unlikely," Barocci said. He said the use of old shows and the uncomfortable viewing conditions can skew the results.
"It certainly isn't very high quality," he said. "The context of the messaging is extremely important."
The Television Preview process consists of sitting in a hotel conference room with four TVs in the center of the room, surrounded by standard conference room chairs. There's water, Jolly Rancher candy and a few bite-sized Snickers bars for snacking. Attendees are given a booklet listing various brand names and asked to circle which brands they prefer. The host explains that winners of the drawings will get the type of products they use. It turns out the $250 in prizes that the company promises aren't awarded at the event. Instead, the six chosen winners must wait two to four weeks to get a "grab bag" of the products they selected worth about $40. Many of the products listed are the ones featured in the television commercials.
The host also asks attendees to rate the actors and their chemistry with one another as well as the plot and concept.
Rob Petersen of San Diego, who attended a recent screening was confused by the TV shows he saw.
"I was trying to figure out if they were just badly made or if they were from years ago," he said.
After the host asked basic questions about the shows, he moved on to ask participants about how often they purchase dandruff shampoo, whether attendees trust the recommendations of Consumer Reports and how familiar they are with certain prescription drugs.
Some degree of subterfuge is needed when evaluating commercials, said David Stewart, a professor of marketing at University of Southern California. Often, if audience members know they are supposed to watch for the commercials, it will taint their responses, he said. "If you alert people and tell them to be critics, they will be critics, but it doesn't measure effectiveness," he said.
Still, others in the industry say participants should be told - at least by the end of the test - the aim of the study. At the Television Preview screening, participants were never informed of the real purpose of the evening.
Lee Weinblatt, chief executive of The Pretesting Co., another well-known advertising research firm, said anyone who evaluates ads with his firm eventually learns the purpose of the exercise. Misleading the public only makes it harder to recruit for future tests, he added. "That's what gets people to not want to participate in the future because they have been part of something that just does not deliver," he said.
Weinblatt said he also makes sure that test subjects are compensated - $10 to $100, depending on the type of advertising - and that they get to watch shows they like. It makes the experience more enjoyable for the subject and gets better data, he said. "If I force you to watch what you don't want to watch, then you are going to get fidgety in your seat," he said.
Nile Rowan, a senior vice president of client services for Millward Brown, one of the top advertising research firms, said his company is upfront with its test subjects and makes the evaluation process as short as possible - 10 to 15 minutes. If it takes too long, "people get bored," which can compromise the quality of the research, he said. And there are questions about whether Television Preview's testing methods are relevant as television viewing habits have changed with the advent of remote controls and digital video recorders.
Weinblatt said he suspects that ARS stays with the Television Preview format because its historical data are based on that method. "There are many companies who have been doing this for so many years that they can't walk away from it because their norms are based on it," he said.
© Copley News Service