2. Pros and Cons of Measuring Effectiveness
Objections of creativesObjections of creatives
Evaluate alternative
strategies
Evaluate alternative
strategies
Avoid costly mistakesAvoid costly mistakes
Increase efficiency in
general
Increase efficiency in
general
Disagreement on
what to test
Disagreement on
what to test
Research problemsResearch problems
Cost of measurementCost of measurement
Determine if objectives are
achieved
Determine if objectives are
achieved
TimeTime
Advantages Disadvantages
4. Measuring Effectiveness: Marketers’ Views
Issue Agree
MPM is a key priority for today’s
technology companies.
90%
I am dissatisfied with our ability to
demonstrate marketing programs’
business impact and value.
80%
It is important to define, measure, and
take concrete steps in the area of
advertising accountability.
61.5%
I can forecast the impact on sales of a
10% cut in marketing spending.
37%
5. How to test
• Testing guidelines
• Appropriate tests
How to test
• Testing guidelines
• Appropriate tests
Measuring Advertising Effectiveness
When to test
• Pretesting
• Posttesting
When to test
• Pretesting
• Posttesting
Where to test
• Laboratory tests
• Field tests
Where to test
• Laboratory tests
• Field tests
What to test
• Source factors
• Message variables
• Media strategies
• Budget decisions
What to test
• Source factors
• Message variables
• Media strategies
• Budget decisions
6. Pretesting Methods
On-air Tests
Dummy Ad VehiclesConsumer Juries
Portfolio Tests
Physiological Measures
Theater Tests
Rough Tests
Concept Tests
Readability Tests
Comprehension and
Reaction Tests
Laboratory Field
7. Posttesting Methods
Recall TestsRecall Tests
Inquiry TestsInquiry Tests
Association
Measures
Association
Measures
Single-
Source
Systems
Single-
Source
Systems
Tracking
Studies
Tracking
Studies
Recognition
Tests
Recognition
Tests
MethodsMethods
9. Test Your Knowledge
Laboratory methods of testing for advertisement
effectiveness generally offer high control but low
_____.
A) realism
B) external validity
C) costs
D) generalization
E) value
11. Positioning Advertising Copy Test (PACT)
1. Provide measurements relevant to objectives of advertising
2. Require agreement on how results will be used
3. Provide multiple measures
4. Be based on a model of human response to communications
5. Consider multiple versus single exposure to the stimulus
6. Require alternative executions to have same degree of finish
7. Provide controls to avoid biasing effects of exposure context
8. Take into account basic considerations of sample definition
9. Demonstrate reliability and validity
12. Test Points
3.Finished art or
commercial pretesting
3.Finished art or
commercial pretesting
1.Concept Testing1.Concept Testing
2.Rough Testing2.Rough Testing
4.Market testing
(posttesting)
4.Market testing
(posttesting)
Occurs at
Various Stages
Occurs at
Various Stages
13. Concept Testing
Explores consumers’ responses to ad
concepts expressed in words, pictures, or
symbols
ObjectiveObjective
Alternatives are exposed to consumers who
match the target audience
Reactions & evaluations sought through focus
groups, direct questioning, surveys, etc.
Sample sizes depend on the number of
concepts and the consensus of responses
MethodMethod
Qualitative and/or quantitative data
evaluating and comparing alternative
concepts
OutputOutput
14. Rough Art, Copy, and Commercial Testing
Number of ads that can be
evaluated is limited
Preference for ad types
may overshadow objectivity
Consumer may become a
self-appointed expert
A halo effect is possible
Cost effectiveness
Endorsements by
independent third parties
Achievement of credibility
Comprehension and Reaction Tests
Consumer Juries
Control
Advantages Disadvantages
16. An Animatic Example from Airwalk
*Click outside of the video screen to advance to the next slide
17. Spot Resulting from Animatic Rough
*Click outside of the video screen to advance to the next slide
18. Pretesting Finished Print Ads
Based on syllables per 100 words
Other factors also considered
Readability
Tests
Readability
Tests
A laboratory method
Includes test and control ads
Portfolio test have problems
Portfolio
Tests
Portfolio
Tests
Distributed to random sample homes
Product interest may still bias results
Dummy
Advertising
Vehicles
Dummy
Advertising
Vehicles
19. Test Your Knowledge
_____ is a method of testing ads by placing them
randomly in certain copies of regularly distributed
magazines.
A) Vehicle source testing
B) Burke's reflection test
C) A Flesch test
D) Dummy advertising vehicle testing
E) A contextual test
20. Pretesting Finished Broadcast Ads
Theater Tests
•Measures changes in
product preferences
•May also measure . . .
•Interest in and reaction
to the commercial
•Reaction from an
adjective checklist
•Recall of various
aspects included
•Interest in the brand
presented
•Continuous reactions
On-Air Tests
•Insertion in TV programs
in specific markets
•Limitations are imposed
by “day-after recall”
•Physiological Measures
23. Starch-Scored Sports Illustrated Ad
Noted score– the
percentage of readers who
remember seeing the ad
Seen-associated score—
the percentage who
remember seeing or reading
any part of the ad identifying
the product or brand
Read most score—the
percentage of readers who
report reading at least half of
the copy portion of the ad.
24. Posttests of Broadcast Commercials
Test marketing
Day after recall
tests
Persuasive
measures
Diagnostics
Comprehensive
measures
Single-source
tracking
Tracking
studies
Testing
27. Essentials of Effective Testing
Use a consumer
response model
Use a consumer
response model
Use pretests and
posttests
Use pretests and
posttests
Use multiple
measures
Use multiple
measures
Understand and
implement
proper research
Understand and
implement
proper research
Establish
communications
objectives
Establish
communications
objectives TestingTesting
28. Test Your Knowledge
Good tests of advertising effectiveness must address
the nine principles established by PACT. One of the
easiest ways to do this is to follow a decision sequence
model. The first step in the model is to:
A) Understand the appropriate research
B) Create a model that uses multiple
measures
C) Establish communication objectives
D) Decide whether to use posttests or
pretests
E) Develop a consumer response model
29. Measuring Effectiveness of Other Programs
Shopping cart signageShopping cart signage
Ski resort-based mediaSki resort-based media
In-store radio and televisionIn-store radio and television
Other mediaOther media
Non-
traditional
media
Non-
traditional
media
Sales
promotions
Sales
promotions
Sponsor-
ships
Sponsor-
ships
Exposure methodsExposure methods
Tracking measuresTracking measures
Relation to text
This slide relates to the material on pp. 615-619 of the text.
Summary Overview
This slide shows some of the reasons why companies should measure effectiveness of their communications programs, as well as why they don’t. Reasons why such measures should be taken include:
To avoid costly mistakes
To evaluate strategies implemented
To increase the efficiency of advertising in general
To determine if objectives are achieved
Many managers choose not to measure, citing the following reasons:
The costs associated with measuring effectiveness
Problems with research methods
Disagreement as to what to test
Objections of creative
Time
Use of this slide
This slide can be used to discuss the reasons why managers should conduct research to measure the effectiveness of advertising and other components of the IMC program, as well as some of the reasons why they may resist doing so.
Relation to textThis slide relates to page 615 and Exhibit 19-1 of the text.
Summary OverviewAdvertising can be an expensive proposition. Testing ad effectiveness helped one retailer discover that promoting Tide detergent generated more cross-selling opportunities than did promotions of non-premium detergent brands. This meant getting more bang for the same advertising buck.
Use of this slideUse this slide to give a real-world benefit that came from ad effectiveness testing.
Relation to textThis slide relates to page 617 and Figure 19-1 of the text.
Summary OverviewThis slide presents marketers’ views toward measuring effectiveness, the result of two recent studies. Although advertisers know it’s important to measure effectiveness, with as many as 90 percent considering it a priority, many do not do so. Or, if they do, they are not confident with the results.
Use of this slide
This slide can be used as an introduction to the reasons companies give for not measuring the effectiveness of advertising and promotion strategies.
Relation to text
This slide relates to material on pp. 619-623 of the text.
Summary Overview
This slide presents some of the issues that have to be considered in the measurement of advertising effectiveness. These include:
What to test
Source factors
Message variables
Media strategies
Budgeting decisions
When to test
Pretesting
Posttesting
Where to test
Laboratory tests
Field tests
How to test
Testing guidelines
Appropriate tests
Use of this slide
This slide presents various factors that must be taken into consideration when measuring the effectiveness of advertising and other IMC elements. It can be used to introduce the material on the following slides, which address many of these issues.
Relation to text
This slide relates to material on pp. 620-622 and Figure 19-2 of the text.
Summary Overview
This slide discusses some of the various pre-testing methods available, further classifying them as to where they are to be conducted - laboratory or field. Pre-tests are those measures taken prior to the implementation of the campaign. Laboratory tests are those in which participants are brought to a specific location for testing, while field tests are those conducted in more natural viewing situations.
Use of this slide
This slide can be used to introduce and provide an overview of pre-testing measures as well as laboratory and field testing.
Relation to text
This slide relates to material on pp. 620-622 and Figure 19-2 of the text.
Summary Overview
This slide provides examples of some of the various types of posttests available. Posttests occur after the ad or commercial has been in the field. The field measures presented on this slide are used to determine the effectiveness of the ads once the campaign has been implemented.
Use of this slide
This slide can be used to discuss the types of post-testing methods that are available to marketers to measure the effectiveness of their advertising programs.
Relation to text
This slide relates to material on pp. 621-622 and Exhibit 19-3 of the text.
Summary Overview
This slide shows a NASCAR race car that was sponsored by Lowe’s, the nation’s second largest home improvement center. Lowe’s recently conducted extensive field experiments to determine the effectiveness of its various forms of advertising, including television spots, newspaper ads, circular catalogs, and even its sponsorship of NASCAR auto racing.
Use of this slide
This slide can be used as part of a discussion of how companies such as Lowe’s use field testing to determine the impact of their IMC programs. You might ask students how a company could go about determining the value of a NASCAR sponsorship.
Ans: A
Relation to textThis slide relates to page 622 of the text.
Summary OverviewEffectiveness testing can take place either in the laboratory and in the field. The major advantage of the lab setting is the control it affords the researcher. The major disadvantage is the lack of realism.
Field tests occur under natural viewing situations, complete with the realism of noise, distractions, and comforts of home. The major disadvantage of field testing is the lack of control. It also takes more time in money. So, realism is gained at the expense of other important factors.
It is up to the researcher to determine which trade-offs to make.
Use of this slideUse this slide when discussing where testing should take place.
Relation to text
This slide relates to the material on pp. 622-623 and Figure 19-3 of the text.
Summary Overview
Because measuring effectiveness is not an easy task, twenty-one of the largest U.S. advertising agencies have endorsed a set of principles aimed at “improving the research used in preparing and testing ads, providing a better creative product for clients, and controlling the cost of TV commercials”. This set of nine principles, called Positioning Advertising Copy Testing (PACT), is designed to establish guidelines for good copy testing research. The nine PACT principles are shown here.
Use of this slide
This slide can be used to present the nine PACT principles that were developed to guide advertising copy testing. You might want to discuss each of these, as they are essential to the development of good copy testing methods.
Relation to text
This slide relates to material on p. 623 of the text
Summary Overview
This slide provides an overview of the testing process that may occur at various stages throughout the development of an advertising campaign. The stages where testing may occur and types of testing that might be done include:
Concept generation research
Rough, prefinished art, copy, and/or commercial testing
Finished art or commercial pretesting
Market testing of ads or commercials (posttesting).
Use of this slide
This slide provides an overview of the testing process and can be used to provide an overview of various forms of testing that are used. The subsequent slides provide specific information about each of these.
Relation to text
This slide relates to material on pp. 623-624 and Figure 19-4 of the text.
Summary Overview
This slide summarizes what is involved in concept testing, which is conducted very early on in the research process. It shows the objective, methods and outputs associated with this form of testing. One of the more common methods of concept testing is through the use of focus groups, though field testing is often employed, and the Internet has resulted in an increase of concept testing online.
Use of this slide
This slide can be used to further explain concept testing which is one of the research methods that is used very early on in the campaign development process.
Relation to text
This slide relates to material on pp. 6248-627 of the text.
Summary Overview
This slide shows the methods of rough art, copy and commercial testing available to the marketer and the advantages and disadvantages associated with these. Because of the high costs associated with commercial development and production, many marketers attempt to measure the potential success of the commercial prior to completing the finished product.
Use of slide
This slide can be used to explain the use of the rough art, copy and commercial testing. It should be noted that for messages that do not involve high emotional content, these rough forms correlate highly with the finished product
Relation to text
This slide relates to material on p. 626 of the text and Figure 19-7.
Summary Overview
There are a number of options available for rough testing purposes. The choice of which to use will be based on the type of commercial to be tested.
This slide shows the three broad categories into which rough testing can be classified including:
Animatic rough - includes a succession of drawings/cartoons, rendered artwork, still frames and simulated movement
Photomatic rough -successions of photographs often showing real people/scenery, etc. with still frames and simulated movement
Live-action rough - employs live motion, stand-in involvement often with nonunion crews, limited props and opticals, and location settings.
Use of this slide
This slide can be used to show the various forms of rough copy testing.
Relation to text
This material relates to material on pp. 624-626 and Figure 19-7 of the text.
Summary Overview
This slide shows an example of an animatic which is one of the forms of rough testing. The animatic is from the “Action Sport Heroes” campaign that was used for Airwalk shoes in the late ‘90s. Animatics are used to test commercials prior to complete development. This saves on production costs as money does not have to be spent to actually produce the commercial in final form. Animatics can also be used to present advertising concepts to clients to assess their reactions before money is spent producing a commercial.
Use of this slide
This slide can be used to provide students with an example of an animatic that was actually used and then developed into a finished commercial spot. The actual commercial is shown on the next slide.
Relation to text
This slide relates to material on pp. 624-626 and Figure 19-7 of the text.
Summary Overview
This slide shows the finished Airwalk commercial that resulted from the animatic shown in the previous slide. As noted, animatics are used to provide a rough copy test providing the marketer with insight into whether to proceed with the finished commercial.
Use of this slide
This slide can be used to demonstrate the effective use of animatics. The finished commercial shown here was tested in animatic form and performed well which resulted in the decision to actually produce the spot.
Relation to text
This slide relates to material on pp. 627-628 of the text that discusses pretesting.
Summary Overview
This slide shows a number of methods for pretesting finished print advertisements. The purpose of pretesting print ads is to attempt to determine how the finished product will perform, make corrections if needed, and fine-tune the message. As the slide shows, some of the options available for this type of testing include:
Portfolio tests—a laboratory methodology designed to expose respondents to a portfolio consisting of both control and test ads.
Readability tests—measures the communications efficiency of the copy in a print ad
Dummy advertising vehicles—“dummy” magazines are created including articles, and test and non-test ads. The magazines are randomly distributed to homes in a specified area. Readers are interviewed to assess their reactions to both the editorial and the ads.
The determination as to which method will be used will be based on the needs and objectives of the advertiser.
Use of this slide
This slide can be used to demonstrate some of the various research methods available to marketers for the pretesting of print advertisements.
Ans: D
Relation to text
This slide relates to material on pp. 628-632 of the text.
Summary Overview
A variety of methodologies are available for pretesting broadcast commercials. A few of the most commonly used are presented in this slide including:
Theater test - participants are invited to view pilots of new TV programs. Embedded in the programs (or following immediately after) are the test commercials. Theater tests can be used to take a variety of measures.
On-air tests - testing of finished or rough commercials through actual TV programs in specified test markets. On-air tests offer some of the same advantages and disadvantages of theater tests but in a more realistic setting.
Physiological measures - a laboratory method for measuring involuntary responses to commercials such as heartbeat, pupil dilation, eye movement, brain waves and galvanic skin response
Theater tests and on-air testing are commonly employed, physiological measures are used less often.
Use of this slide
This slide can be used to discuss the various methods for testing reactions to finished broadcast commercials.
Relation to text
This slide relates to material on pp. 630-632 of the text.
Summary Overview
This slide shows the various physiological measures, a less common method of pretesting finished commercials. These measures include:
Pupil dilation
Galvanic skin response
Eye tracking
Brain waves
Use of this slide
These measures indicate the receiver’s involuntary response to the ad, theoretically eliminating biases associated with voluntary measures. This slide can be used to discuss how physiological responses are measured.
Relation to text
This slide relates to material on pp. 632-635 and Exhibits 19-4 and 19-5 of the text.
Summary Overview
This slide shows the various methods that are used to determine the effectiveness of print advertising once the ads have been completed and put into the market. A number of methods are available including:
Inquiry tests
Recognition tests
Recall
Tracking studies
These methods are often used by advertisers to determine how well print ads are doing in terms of getting attention, being read, and being remembered.
Use of this slide
This slide can be used to discuss the various options available to marketers for the testing of how well print ads are performing in the market.
Relation to text
This slide relates to material on pp. 634-635 and Figure 19-14 of the text.
Summary Overview
This slide shown an example of a Starch scored ad. The Starch method is one of the more commonly employed posttest measures of print ads. It uses three measures including:
Noted score– the percentage of readers who remember seeing the ad
Seen-associated score—the percentage who remember seeing or reading any part of the ad identifying the product or brand
Read most score—the percentage of readers who report reading at least half of the copy portion of the ad.
Starch measures have been used in the magazine industry for many years, and have become one of most commonly employed measures of advertising effectiveness.
Use of this slide
This slide can be used to provide an example of a Starch scored advertisement.
Relation to text
This slide relates to material on pp. 636-639, Exhibit 19-5 and 19-6, and Figure 19-16 of the text.
Summary Overview
This slide shows the variety of measures available for the market testing of finished broadcast commercials. These include:
Day after recall test - the most commonly performed measure.
Persuasive measures – attempting to measure persuasive impact of the commercial on brand selection.
Diagnostic - measures to determine viewers’ evaluations of the commercial, including comprehension and clarity.
Comprehensive measures - a combination of each of the three previous measures.
Test marketing - testing of finished commercials in specified test markets.
Single source tracking - tracking of the impact of the commercial from TV set to the retail store through the use of scanner data
Tracking studies - measuring the effects on various factors (recall, awareness, and attitudes) over time.
Use of this slide
This slide can be used to demonstrate the various options available for the marketer interested in measuring the effectiveness of a finished commercial.
Relation to text
This slide relates to material on pp. 636-637 and Exhibit 19-5 of the text.
Summary Overview
This slide shows an example of a provider of research service, Ipsos-ASI. The company provides a variety of research services for print and broadcast measurement, including diagnostics and comprehensive measures.
Use of this slide
This slide can be used to provide an example of one of the many research companies whose services are available for commercial testing.
Relation to text
This slide relates to material on pp. 623 and 640, and Figure 19-3 of the text.
Summary Overview
This slide shows the principles important to good copy testing. It is clear that some of these principles are easily accomplished, whereas others require substantially more effort.
Principles 1 and 2: should require minimum effort providing measurements relative to the objectives sought and determining a priori how the results will be used.
Principle 3: slightly more difficult, but largely in control of the researcher, providing multiple measurements
Principle 4: one of the most difficult factors to control. Most current methods do little more than provide recall scores.
Principle 5: slightly difficult, can be accomplished with proper research design.
Principle 6: should require minimum effort, providing equivalent test ads.
Principle 7: one of the most difficult factors to control. Lab measures are artificial and vulnerable. Field measures often lose control.
Principle 8: slightly difficult, requires little more than sound research methodology.
Principle 9: one of the most difficult factors to control due to concerns for reliability and validity.
Use of this slide
This slide can be used to discuss problems with current research methods when compared to the criteria established by PACT.
Relation to text
This slide relates to material on pp. 641 of the text.
Summary Overview
This slide shows the essentials of effective advertising testing. To be truly effective, the testing method should:
Establish communications objectives
Use a consumer response model
Use both pretests and posttests
Use multiple measures
Understand and implement proper research
Solid research should meet these criteria. Research methodologies that meet these criteria will be the most effective for providing insight into the performance of advertising.
Use of this slide
This slide can be used to provide an overview of the criteria for effective measurement.
Ans: C
Relation to text
This slide relates to material on pp. 641-644 of the text.
Summary Overview
This slide shows a number of methods for measuring the effectiveness of other program elements not previously discussed. These measures include:
Sales promotions—a number of organizations measure sales promotions, including MarketSource, Schnucks, Smitty’s Super Valu, Vons and Shopper Trak.
Nontraditional media—examples include shopping cart signage, ski resort-based media, in-store radio and television, and other media.
Sponsorships—essentially, measures of sponsorship can be categorized as exposure-based methods or tracking measures.
Use of this slide
This slide can be used to discuss the increased use of integrated marketing communications programs, which has led to more interest in determining the synergistic effects of all program elements.
Relation to text
This slide relates to material on pp. 644-647 and Figure 19-23 of the text.
Summary Overview
This slide illustrates the importance of measuring effectiveness and efficiency. Based on the belief that integrated marketing communications improves both the efficiency and the effectiveness of a campaign, the research company Integration contends that most traditional measurement techniques focus only on the former of these.
Use of this slide
This slide can be used to discuss the increased demand for marketing managers to prioritize the media vehicles used to promote their brands, and to measure both the efficiency and the effectiveness of media used to establish contacts with consumers.
