This is what I hate about attempts to evaluate advertising. Ace Metrix recently graded ads from several brands that featured their CEOs against ones that didn’t, concluding that the CEO ads were better, in some cases.

But here’s what Ace graded them on: Persuasion, watchability, desire, relevance, change, attention, information and likeability. Persuasion is certainly important as is relevance, but the spots have not been evaluated in terms of whether they steal market share.

What Ace has done is the equivalent of judging contestants on American Idol. That is, they are evaluated by everything other than meaning. The evaluation didn’t uncover what they mean and whether those meanings were important.

Consider this. The spots featuring CEOs that worked best in Ace’s opinion were Papa John’s (John Schnatter) and Samuel Adams (Jim Koch). Yet, what did those ads mean?

Papa John’s is about price and better ingredients – the exact same messages as its competitors. Not a share-stealing category. (In fact, Papa John’s and its competitors are battling so much over price they threaten to price themselves right out of the market.)

Samuel Adams is about the quality of the beer and how much care it takes brewing the beer. (And also building on heritage that Boston Beer has been brewing Sam Adams for decades.) What’s so different about that? To get someone to switch to Sam Adams, with these messages, you’d have to believe the current beer you’re drinking is of bad quality. Who drinks a beer they believe sucks?

All the characteristics Ace tested have a place, certainly, because you do want your ads to be those things. But none of them are the most important.

You can be the best at all those things and, without a different and more meaningful message than your competitors, they don’t make an ounce of difference.