The broadcast brass hunkered down in their respective headquarters reviewing pilots for the upcoming season are getting their annual reminder of where true power lies this time of year.
It's not with parent-company power brokers like News Corp. chairman Rupert Murdoch, who are known to partake in the occasional pilot screening. Nor is it the lowly 20-something assistants just lucky to be in the room--though their opinion often matters more than the chairman's because they're in the target demographic.
In many cases, what really matters in the screening rooms is research. Scores gleaned from audience samples, whether from outsourced test facilities or cable-system narrowcasts, become valuable currency as broadcasters decide which shows merit series orders for the schedules, set to be announced next week.
Beverly Bolotin, president of testing company ASI Entertainment, believes that data grading viewer feedback on everything from specific characters to their overall impression of a show help evaluate a program's prospects.
"Those seeking to make informed decisions will continue to use research that increases the likelihood of putting the strongest shows on the air, particularly as the competitive landscape intensifies over time," she said.
And yet pilot-season veterans say the process and presentation of audience research is flawed, not to mention ineffectual, when it comes to picking pilots.
"With the limited time researchers have to present and the lack of sophistication of people in those rooms have with that information, it plays a disproportionately important role," said Jordan Levin, former CEO of WB Network and now a partner in Generate, a new multiplatform production company.
Executives from ABC, CBS, NBC, Fox, and the nascent CW network declined comment.
Although audience testing has been part of the pilot-picking process for decades, Levin dispensed with the ritual in the 2004-05 season after tiring of its decidedly mixed track record over the years at WB.
No skeptic of pilot testing is without more than a few horror stories of research gone wrong. For Levin, one lesson came while observing a focus group in 1996, when a series he already had committed to got savaged by a pair of teenage girls. The rest of the group, who had appreciated the show in their own test scores, apparently were intimidated into silence.
The show in question: 7th Heaven, a cornerstone of WB's schedule for a decade.
"The test could have come back, 'Teenage girls are turned off by this,'" Levin said. "You never know unless you're there and paying attention."
But Levin and others are less distrustful of the research itself--which even skeptics believe has value if analyzed with due deliberation--and blame its role in the ever-secretive pilot screenings. Each network executes this process differently: Most break down their ranks into smaller groups who report their findings separately, though this year, ABC convened everyone into two theaters.
Ted Harbert, CEO of E! Networks and a veteran of many screening sessions in past stints as chairman of ABC Entertainment and head of NBC Studios, has been through every configuration and said there's no right way to do it.
"Invariably what happens is 10 room captains will report this to you: Some people liked it, and some people didn't," Harbert said. "I would walk out of the room and my head would explode."
Harbert's horror story: "[ABC series] thirtysomething didn't test well at all because there was so much honesty it was painful," Harbert recalled. "Our head of ad sales said we will not put this on the air, but we fought like hell for it. It went on to get the highest unit price for female viewers in ABC history."
Pilot screenings are said to start out as kind of a utopian vision imagined by human resources, where employees from opposite ends of the corporate ladder get to vote and debate a pilot's attributes. But after days of nonstop viewing of dozens of pilots, the human body maxes out on appropriate levels of Starbucks and Visine. The hopeless subjectivity of dialogue and consensus-building gives way to the comforting tangibility of numeric data.
"You could get a vote where 75 percent of the people in the room like it," Harbert said. "Then the research comes in, and it's not good--that show is dead. It's like it never even existed."
Pilot screenings can be a period of incredible frustration for creative execs who have spent six months working with producers on a given pilot, only to find that the opinions of different division heads from the likes of marketing and public relations are taken into consideration--and that's when the numbers really start flying.
"All of the sudden, there's this influx of other voices, and those voices are very strong in those rooms," Levin said. "Sometimes what happens is pilots that test well get picked up because the noncreative people feel comfortable with a quantitative test."
Complicating matters, the studios that produce the pilots often furnish their own research and--surprise, surprise--it's not rare that their findings contradict the research commissioned by the network. Often enough, you can tell a lot about a network's health by its level of adherence to any audience testing.
"I think if you're a network doing well, you believe the research is telling you something you already know," said Glenn Gordon Caron, executive producer of NBC's Medium. "If you're doing poorly, you grasp whatever divining rod you can get your hands on."
Caron recalls a drama pilot he once worked on where he brought in an actor he wanted to cast in the lead role, but the network didn't share his passion. So they shot a screen test of the actor and showed it to 300 people at the Preview House, an old Hollywood testing facility. The results were not encouraging, but Caron managed to prevail anyway.
The show: Moonlighting. The actor ABC execs doubted: Bruce Willis.
"I'm dubious of testing," Caron said. "I don't think it's revelatory of anything other than how it does in the context in which you play it."
Ultimately, research ends up as a talking point for network presidents hyping select series that rack up high scores. Nary an "upfront" presentation to advertisers goes by without at least one network highlighting one particular pilot with a three-word designation that sends media buyers' eyes rolling: "highest-testing pilot."
"I'm always hesitant to accept research from the people who are touting it," said Shari Anne Brill, vp and director of programming at Carat USA. "It's like the National Dairy Board saying, 'Milk is good.'"
"Highest-testing pilot" has become, with some exceptions, a jinx on par with gracing the cover of Sports Illustrated. Among the long-forgotten series ushered in to such fanfare include CBS's Center of the Universe, which featured Roseanne star John Goodman, and NBC's Emeril, with Food Network chef Emeril Lagasse. Also among the dead that earned that distinction: CBS's 1996 Rhea Perlman comedy Pearl, UPN's 1998 Western drama Legacy, and WB family comedy Greetings From Tucson in 2002.
Last year was something of an aberration in that the "highest-testing pilot" appellation turned out to be dead on instead of dead on arrival. NBC Entertainment president Kevin Reilly touted the title character of a quirky pilot called My Name Is Earl as highest-testing pilot in 15 years.
"I'm sure Earl was no easy sell within NBC, but the fact that the character tested well helped Kevin," Levin said.
There are plenty of classic shows whose greatness was apparent in the research phase, including The Cosby Show and Golden Girls. But there also have been plenty of classic series famous for performing poorly in audience testing, including Seinfeld, Everybody Loves Raymond, and Roseanne. A breakout comedy, in particular, can be tough to predict because the best of the genre often tend to be polarizing or puzzling in their bid to break the mold. "Comedy testing I find to be particularly unreliable," Harbert said.
Tim Brooks, senior vice president of research at Lifetime and a pilot-season veteran from his days at NBC, believes research is not going away anytime soon. He is particularly high on testing the viability of characters, which he feels hold the key to a series' sustainability over the long run. "If you don't have appealing characters, it doesn't matter what else is appealing," he said.
Brooks recalls the 1981 pilot for Steven Bochco's classic NBC drama Hill Street Blues, in which two characters, beat cops Andy Renko and Bobby Hill, are shot and killed in a drug den. When testing on the pilot was reviewed, NBC researchers broke the bad news to Bochco: He had eliminated two of the most beloved characters on the show.
Although the pilot was guaranteed a pickup, Brooks said Bochco reconsidered the pilot's story, tacking on a scene at the end explaining that the cops were not dead but in intensive care, allowing them to return. Said Brooks, "Research saved their lives."