The Importance of Rank
•Download as PPTX, PDF•
4 likes•4,840 views
This presentation looks at the importance of RANK in everyday decision making (we use it as a fast heuristic for decision making - evolution baked this into us) and in the context of business. To begin with, we briefly review why markets tend towards unequal distributions of market share over time, with market leaders taking more than their fair share. We discuss how these inequalities emerge due to the multitude of individual consumer decisions based on incomplete market information and our brains’ ability to make the best, most resource-efficient decisions with the least amount of information. To understand mental limits, we show how many brands people can keep in mind within a category at one time and that this limit is a function of the human brain, not the number of brands in the market. Finally, we look at popular awareness and usage metrics to evaluate which ones link back to actual consumer behaviour in order to recommend which ones are worth measuring and which add little value and should thus be removed from surveys.
Recommended
Recommended
More Related Content
More from Socialphysicist
More from Socialphysicist (9)
Recently uploaded
Recently uploaded (20)
The Importance of Rank
- 1. The Importance of Rank Or How Our Brains Constrain Survey Responses Or The Enormous Power of Winning
- 3. The enormous power of winning This presentation is about how these inequalities emerge 1 2 3 4 5 1 2 3 4 5
- 4. “One is wonderful. Two is terrific. Three is threatened. Four is fatal ”~ Larry Light
- 5. Retailers (UK)Retailers (China) Auto (UK)Auto (Germany) Sources: Retailers = KWP, Auto = Mintel
- 6. Source: Kohli, R and Sah, R. 2003. “Market Shares: Some Power Law Results and Observations”, Harris School Working Paper, Series 04.1 “Market share versus rank across 506 food brands and 665 sporting-goods brands “ ” Power laws!
- 7. Man can be thought of as perfectly rational John Maynard Keynes
- 8.
- 9. The „Chicago School‟ of market research
- 12. We make good enough choices from a „stacked deck‟ Sources: http://www.pepsico.com/Annual-Reports/2008/performance/n-america-csd.html
- 14. 65.1 38.9 28.2 25.2 24.0 17.0 2.5 1.2 0.9 0.7 0.7 0.5 Newspaper 1 Newspaper 2 Newspaper 3 Newspaper 4 Newspaper 5 Newspaper 6 Usage of brand Frequency of purchase Source: UK newspaper market
- 15. We do the best we can within our cognitive limits
- 16. Get the important bits right and the rest will follow
- 17. The brain is impressive…
- 18. The brain is impressive…but it does have limits
- 22. 3.9 Mean Standard deviation = 3.7 Data: 1,267 studies, 88 countries, 205 categories, 1,058,290 respondents 3.0 2.0 Median Mode So, just how many brands do we realistically hold in mind at a time?
- 23. Standard deviation = 3.7 Data: 1,267 studies, 88 countries, 205 categories, 1,058,290 respondents 17.9 23.1 15.6 11.4 8.0 6.0 4.3 3.9 1.4 0.9 3.9 1 2 3 4 5 6 7 8 9 10 The rest %ofobservations Evoked set size
- 24. 2.3 Auto 3.3 Finance 5.0 Consumer 3.1 Energy 3.3 Business Services 3.4 Technology 2.3 Polling & Social 3.1 Healthcare Standard deviation = 3.7 Data: 1,267 studies, 88 countries, 205 categories, 1,058,290 respondents
- 26. 0 10 20 30 40 50 60 70 80 90 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #brandsinbrandlist Average evoked set size 0.36 0.13 Correlation Linear R2 1,267 studies Standard deviation = 3.7 Data: 1,267 studies, 88 countries, 205 categories, 1,058,290 respondents
- 27. Evolutionmade us care about
- 31. Source: Louw & Hofmeyr (2012) Reality Check: The Relationship Between What We Ask and What People Actually Do
- 32. 69.5 23.6 11.3 4.1 R² = 0.97 1 2 3 4 Shareofwallet(paneldata) Share of wallet rank Power law! Rank 1 brands get most of the share n=984 UK laundry detergent | Actual panel data supplied by KWP
- 33. UK laundry detergent | Actual panel data supplied by KWP 010%20%30%40%50%60%70%80%90% %ofobservations Share of wallet Rank 1 Rank 2 Rank 3 Rank 4
- 34. x y logx logy
- 35. Power law! Source: Krumme, C, et al (2013) The predictability of consumer visitation patterns Caption: “Probability of visiting a merchant, as a function of merchant visit rank, aggregated across all individuals. Dashed line correspond to power law fits P(r) , r2a to the initial part of the probability distribution with a 5 1.13 for the European and a 5 0.80 for the North American database” Retailer visits by rank Power law!
- 36. 60.2 32.5 18.9 17.7 13.3 10.7 9.0 7.0 5.6 5.4 4.8 3.1 3.9 2.1 3.6 1.4 2.2 2.0 4.1 2.6 2.3 R² = 0.93 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 10.5 11.0 Shareofwallet(paneldata) Performance rank Power law! Top rated (i.e. rank 1) brands get most of the share UK laundry detergent | Actual panel data supplied by KWP
- 37. too much information It is possible to measure
- 38. First mention i.e. rank 1 0.57 0.28 0.17 All unaided aware i.e. rank 2, 3… Aided aware i.e. All ranks Awareness metric Countries: UK, China | Categories: Laundry, Retail Stores | Number of datasets: 5 | Actual panel data supplied by KWP Source: Hofmeyr, J & Louw, A. 2012. Reality Check : The Relationship Between What We Ask and What People Actually Do. ESOMAR 3D Conference 2012, Amsterdam Respondent-level correlation with spend
- 39. Respondent-level correlation with spendUsage metric Most often i.e. rank 1 0.71 0.58Past 3 months 0.37Ever bought i.e. all ranks Countries: UK, China | Categories: Laundry, Retail Stores | Number of datasets: 5 | Actual panel data supplied by KWP Source: Hofmeyr, J & Louw, A. 2012. Reality Check : The Relationship Between What We Ask and What People Actually Do. ESOMAR 3D Conference 2012, Amsterdam 0.62 0.56 Regularly i.e. rank 2, 3… Past 1 month
- 40. 78 36 36 36 Cell 1 Cell 2 Cell 3 Cell 4 # attributes in survey Less measurement; richer data
- 41. 78.0 17.0 18.0 28.0 Cell 1 Cell 2 Cell 3 Cell 4 # attributes selected
- 42. 2.2 3.0 3.2 4.8 Cell 1 Cell 2 Cell 3 Cell 4 # ticks per respondent
- 43. 13.0 3.0 3.5 5.5 Cell 1 Cell 2 Cell 3 Cell 4 Total survey time
- 45. 3.9
- 46. 1 2 3 4
- 49. Thank youAcknowledgements: Anna Retief Elanie de Beer Jannie Hofmeyr Ken Bell Bruno Gonçalves Constantin Michael Kyle Findlay Senior R&D Executive Kyle.Findlay@tnsglobal.com The TNS Global Brand Equity Centre
- 50. First mention i.e. rank 1 0.57 0.28 0.17 All unaided aware i.e. rank 2, 3… Aided aware i.e. All ranks Awareness metric Countries: UK, China | Categories: Laundry, Retail Stores | Number of datasets: 5 | Actual panel data supplied by KWP Source: Hofmeyr, J & Louw, A. 2012. Reality Check : The Relationship Between What We Ask and What People Actually Do. ESOMAR 3D Conference 2012, Amsterdam Respondent-level correlation with spend 0.92 0.81 0.68 Aggregate-level correlation with spend
- 51. Respondent-level correlation with spendUsage metric Most often i.e. rank 1 0.71 0.58Past 3 months 0.37Ever bought i.e. all ranks Countries: UK, China | Categories: Laundry, Retail Stores | Number of datasets: 5 | Actual panel data supplied by KWP Source: Hofmeyr, J & Louw, A. 2012. Reality Check : The Relationship Between What We Ask and What People Actually Do. ESOMAR 3D Conference 2012, Amsterdam 0.62 0.56 Regularly i.e. rank 2, 3… Past 1 month 0.96 0.96 0.96 0.98 0.93 Aggregate-level correlation with spend
Editor's Notes
- NOTE: The data for this presentation were updated since the original paper was written with double the number of respondents (500k to 1million)Abstract:The traditional market research paradigm believes that the more data you measure, the more potential for insight the data holds. However, this paper takes the counter-intuitive standpoint that 'less is more'. Drawing on the author's familiarity with neuroscience, five years worth of actual panel behavioural data in three categories, and data from 624 studies across 547,945 respondents, the paper argues that, just as it is important to ask the right questions in a survey, it is similarly important to measure 'just enough' but not too much information about brands, since over-shooting the mark can actually be detrimental to the ultimate quality of your data and insights. To begin with, we briefly review why markets tend towards unequal distributions of market share over time, with market leaders taking more than their fair share. We discuss how these inequalities emerge due to the multitude of individual consumer decisions based on incomplete market information and our brains’ ability to make the best, most resource-efficient decisions with the least amount of information. To understand mental limits, we show how many brands people can keep in mind within a category at one time and that this limit is a function of the human brain, not the number of brands in the market. Finally, we look at popular awareness and usage metrics to evaluate which ones link back to actual consumer behaviour in order to recommend which ones are worth measuring and which add little value and should thus be removed from surveys.
- As we shall see, consumers are resource- and information-poor, energy-optimising decision makers. We use simple heuristics and holistic judgments to make decisions that take into account the relative performance of brands by ranking them based on an arbitrary criterion (or set of criteria) that result in the emergence of characteristic inequalities in market share over time (Figure 1 visually summarises this). Therefore, in order to accurately model our clients’ business environments, market researchers need to tailor their questions to match the way the way that the human mind works.The rest of this paper will be devoted to providing evidence for this line of logic, working backwards from our understanding of how market structure develops over time. Readers should take away an appreciation for the importance of measuring the specific question responses that rank highly in respondents’ minds.(The term ‘arbitrary’ is used in this paper to refer to metrics that have been chosen based on individual judgement or collective consensus. It does not necessarily imply that the metric is unreasonable or unsupported by evidence. For example, cash money is an arbitrary signal and store of wealth that society has agreed upon.)
- The human brain is a marvellous thing. Within each of our heads sits an organ that can out-compute the world’s most powerful machines when it comes to making judgments that are important to our survival such as identifying novel patterns and making quick decisions. Our brains attempt to do this with the least amount of energy and information possible as both are incredibly scarce resources. What is interesting about how the brain and human behaviour work in a business context is that through the energy-optimising, effort-reducing actions of numerous individuals, we see the emergence of consistent patterns of market share. Branded markets are complex systems whose characteristics, such as the market share that each brand accumulates over time, emerge due to the interactions of individual agents with limited cognitive resources and information about the overall environment. Individuals navigate their social environment by employing heuristic decision making ‘mechanisms’ such as taking what other people do as a signal of the value of that activity; rating entities that are more familiar as more important or relevant; and so on (many of these mechanics are covered in the field of behavioural economics). These mechanisms are employed, often unconsciously, because no person has perfect information about an environment and the mechanisms have been shown to (or are believed to) improve one’s survival chances. Consistent patterns of market share inequality have been described and modelled by many scholars in one form or another as far back as the nineteenth century with general observations of human systems by researchers like Vilfredo Pareto (1896, most famously with his Pareto Principle or the 80-20 Rule), George Kingsley Zipf (1949) and Herbert Simon (1955); and, the patterns still represent a rich vein of enquiry today.Nature’s attempt to conserve energy has been well documented amongst, for example, ants (e.g. Oettler, 2013) and our human drive to do the most with the least was famously recognised by George Kingsley Zipf (who we will hear more about later) as far back as 1949 in his book, Human Behavior and the Principle of Least Effort.(When this mechanism goes wrong, it can lead to the ‘bandwagon effect’ and herd behaviour as evidenced in fads and market bubbles)
- In the past few decades, increasingly more empirical descriptions have been articulated of the market share patterns that emerge over time in branded markets. Boston Consulting’s Bruce Henderson (1976; see also Sheth & Sisodia, 2002) captured the insight that market share inequalities tend to form over time with his ‘rule of three” which broadly states that any category will contain three competitors that dominate the market – a market leader and two challengers. A variation of the same insight is captured by an aphorism attributed to Larry Light when talking about a brand’s rank in the market in terms of its size: “One is wonderful. Two is terrific. Three is threatened. Four is fatal”. Buzell (1981) also postulated the existence of natural market structures as far back as 1981. A few years later, Goodhardt, Ehrenberg and Chatfield (1984) suggested the Dirichlet distribution as a tool for modelling market shares. Ehrenberg and co’s work went on to identify regularities in market research data that relate back to consistent inequalities between the top ranked brands in terms of share size and the rest of the brands in the market, including the elucidation of the Double Jeopardy rule (e.g. Goodhardt, Ehrenberg & Barwise, 1990). Researchers such as Byron Sharp have further built on these foundations (see for example Sharp, 2010).(CEO of brand consultancy, Arcature and Chief Brands Officer of the InterContinental Hotels Group)
- A more recent body of literature bridges the gap between market research-specific models and models observed across a broad variety of complex systems in many other domains, pointing to the apparent universal nature of these patterns. Specifically, researchers have shown that the skewed market share figures observed in most branded markets can be described, along with many other systems, by the general class of distributions known as ‘power laws’ which have been identified in natural and social complex systems as diverse as human heart rates (e.g. Huikuri, 2008), city sizes (e.g. Batty, 2005), social network characteristics (e.g. Barabási & Bonabeau, 2003), sports statistics (Deng, et al, 2011) and mammalian metabolic rates (e.g. West 1998; Dodds, Rotham & Weitz, 2001), amongst many others.
- Within the context of brands, Li (2003) has pointed towards evidence of power law distributions in firm sizes and firm debts. Kohli and Sah (2003) have gone further to show that the market share distribution across hundreds of branded markets follows a power law (see Figure 2) while Hofmeyr, et al (2010, 2012) has shown that brand market shares can be modelled with greater accuracy than any other survey-based modelling technique in the public domain by using the Zipf equation (a power law).
- Classical economic models rely on the assumption that people are perfectly rational – that they have perfect information about a market and make the best decision in every situation. However, advances in psychology, neuroscience and, most recently, behavioural economics have shown that these idealised models fall far short of the reality of how people behave and make decisions. Human behaviour is far messier and noisier than classical models acknowledge and this has profound effects on our understanding of how social and economic processes work.
- This paper argues that, similar to classical economics, the market research industry labours under false assumptions - consumers are not perfectly rational actors. Rather, they have bounded rationality and employ decision making aids and shortcuts to optimise their choices. These decision making mechanisms lead to systematic biases in the formation of market share, and if market researchers want to accurately model these processes, we need to focus on measuring the responses to popular metrics that most closely link to how the human mind works. One of the most important mechanisms in this regard is a focus on rank as a vital component of decision making by beings with bounded rationality.
- Why do consistent patterns of inequality form in market share distributions? The existing literature does not have much to say about how individuals make their decisions such that heavy-tailed, power law distributions tend to form over time. However, the patterns appear to emerge due to feedback loops inherent in the interactions between brands and individuals with limited cognitive resources and information about the environment. Findlay (2009) posited that such patterns come about due to a preferential attachment mechanism whereby the probability of a brand being bought is proportional to its existing market presence with the view that such a mechanism, when applied recursively over successive purchasing cycles across the market as a whole, leads to a power law distribution of market share (see also Barabási and Albert, 1999, for a more general discussion of this mechanism). While no single mechanism has been shown to underlie all buying behaviour, what is clear is that biases in buying behaviour emerge because individuals have bounded rationality – we are not able to collect or evaluate every possible piece of information relating to a ‘perfect’ decision – which leads to systematic biases in aggregate behaviour.
- A side effect of focusing on the most important pieces of information in a respondent’s mind is that we only need to measure a subset of responses (e.g. brand mentions, attribute associations or similar) to a question since not all pieces of survey information are equally valuable. Indeed, as we will see, including too much information in your analyses can actually be detrimental to overall data quality and, thus, to the quality of insights gleaned from the data. However, before we can get to the practicalities of what is important for survey measurement, we need to lay the groundwork in terms of what our industry, and science in general, knows about market structure,a topic that market researchers and their clients deal with on a daily basis…(This paper places a strong emphasis on survey measurement in terms of U&A (usage and awareness), brand equity and measures which link back to actual in-market behaviour, which is, after all, generally the thing that our clients really care about)
- Part of our research concerns itself with identifying the boundaries of consumer ‘rationality’. Specifically, we have looked at just how many brands people hold in mind when making judgements about brands by looking at the ‘evoked set’ sizes across a large number of our datasets for a variety of categories and countries.
- An evoked set represents the number of brands that an individual can hold in mind when evaluating brands and making purchase decisions. It is constructed by asking a respondent two questions: what brands have they used, and what brands would they consider using. This gives us a rough idea of the universe of brands that is relevant to a respondent. Looking across all 547,945 respondents in our dataset, we see that the average evoked set size is 3.4 (median 3). NOTE: The above values have been updated based on our sample of 1million+ respondents
- Unsurprisingly though, we do see some sector variation in the average evoked set size as summarised by Table 2. The average evoked set size roughly aligns with other areas of human mental life that are constrained by cognitive limitations. For example, in human relationships, despite the explosion of social media, most people only maintain two to three close friends (Marsden, 1987; McPherson, Smith-Lovin & Brashears, 2006) and most people only have regular SMS conversations with one other person (Wu, et. al., 2010). In addition, humans can only keep about four items in short term memory at one time (Miller, 1956; Cowan, 2001) and most people only think one or two steps ahead in game theory tasks that require them to predict what others will do (Bosch-Domènech, et al, 2002).(A Portuguese saying captures our limited ability to keep multiple close friends and the logic of asymmetric use of scales that we will discuss shortly: “You have five friends, and the rest is landscape” (Williams, 2011).)A smaller evoked set might be less surprising in low-use categories such as Auto where few people own more than one or two cars and many people often repurchase the same brand. However, even in a more promiscuous sector such as Consumer which has the highest average evoked set in our data, the average evoked set size is still relatively low overall at 4.0 (median 3). This seems to point to innate evoked set size limitations that are not just a function of the sector which might contradict the expectations of some readers who assume that the evoked set size depends on the number of brands in the market. For example, some might assume that if there are 100 cereal brands to choose from, then the average evoked set in the cereals category will be significantly higher than in another category where there are only 20 brands for people to choose from. However, our data shows that this is not the case.
- Interestingly, the beer category produces some of the highest average evoked set sizes e.g. UK (11.7); Germany (10.2); Poland (9.6); France (9.8); Spain (9.6); although these are counter-balanced by Asian countries such as India (2.8), Thailand (2.9) and Cambodia (3.7) NOTE: The above values have been updated based on our sample of 1million+ respondents
- To explore the relationship between number of brands in the market and evoked set size, we looked at the correlation between brand list length and average evoked set size for each of the 624 studies in our dataset. We found the correlation between these two variables to be 0.36. In other words, there is a weak, if any, relationship between the number of potential brands that a consumer could choose between and the number of brands that they hold in their mind as relevant to their purchase decisions. The scatterplot in Figure 4 shows that there is no clear relationship between evoked set size and brand list length.(Similarly weak relationships emerge when we look within specific categories, although there is some category variation )
- The Importance of RankWe have established that humans are not able to hold every brand in mind at the same time and we have seen that most people only keep about three or four brands in mind when making decisions within a category. Let us now have a look at the importance of ranking in order to understand how the human mind mentally manages and manipulates the brands that it thinks are relevant...in knowing which products receive a 5, 4.5 or 4 star review, but beyond this, one could use a simple heuristic to group the remaining ratingsHumans spend a lot of time ranking objects based on arbitrary criteria. We love using rankings as a signal of fitness where a rank of ‘1’ usually denotes the most desirable position. We use words like “best” and “worst” which, by their very nature, incorporate a relative comparison between entities in order to evaluate whether something outperforms or underperforms something else.
- We rank individuals and objects based on a variety of criteria that relate back to evolutionary fitness and resource scarcity, such as physical prowess (e.g. top ranked sportsmen), wealth (e.g. the Forbes wealthiest people list), convenience (e.g. fastest smartphone) and efficiency (e.g. lowest energy consuming appliances). These rankings tell us something about the ability of the entity being ranked to do more with scarce resources since, at the most basic level, an entity’s rank gives us a simple heuristic for working out its probability of survival and reproduction (or its ability to help us survive and reproduce).
- The logic of ranking assigns value to the top ranked entities but quickly loses interest in the lower ranks based on the recognition that there is rapidly diminishing value in being familiar with the lower ranks. For example, competitors in Olympic events are ranked on first place (gold), second place (silver), third place (bronze) and ‘the rest’. The XKCD cartoon in Figure 5 humorously captures the same logic in the context of online product ratings. Past the top three ranks, the rest of the competitors are not ranked, even though they collectively make up a larger group of people than the three top ranks
- Even within the market research industry, we recognise the asymmetric importance of rank. This logic is built into popular metrics such as Reicheld’s (2003) Net Promoter Score, which assigns the most importance to the top end of its 10-point rating scale (see Figure 6). Similarly, Hofmeyr et. al.’s (2008) Attitudinal Equity metric gives asymmetric importance to its input scales.
- Hofmeyr and Rice (2001) have described a model of brand equity based on how the human mind works, with some of the key insights of the model being that humans make decisions based on the relative, rather than absolute, performance of brands in relation to each other i.e. their rank. This is a resource-efficient way for the brain to make quick summary judgements without the need for comprehensive cost-benefit analyses that a) are too resource- and time-intensive to work as a decision strategy conducive to long-term survival, and b) rely on information that a person does not realistically have.They suggest that people make relative judgements wherein the absolute level of prestige a brand gains (e.g. score, rating, etc.) is not as important as its relative position versus other brands in terms of rank (e.g. is it the best, second best, etc.). They use a measure that asks respondents to rate brands based on general, holistic evaluations of brand performance using a 10-point scale. While there are good reasons for Hofmeyr and Rice to propose such a scale, Fortunato et al. (2006) have shown that any arbitrary measure of performance or prestige can be used and that decisions made on the basis of these rankings can lead to a power law distribution over time.Fortunato, et. al. (2006) put the intuition that not all data is created equal into a formal context by explaining how rank can be important. Using network theory, they have shown that a power law distribution can emerge in complex systems when edges (i.e. links, connections, etc.) are created between nodes (i.e. entities such as consumers and brands) based on the relative perception of prestige compared to its surrounding neighbours (i.e. its rank in terms of prestige) rather than their neighbours’ absolute levels of prestige. They have shown that the accumulation of edges in this way leads to a power law distribution in the degree distribution of the nodes. To use a market research analogy, they have described how brands can accumulate market share through individual purchases based on how the brands are ranked relative to other relevant competitors, and how this process of buying based on rank can lead to the emergence of a characteristic power law distribution in market share over time. They have also shown that the prestige criteria can be based on ‘endogenous fitness attributes’ derived from the topology of the network (e.g. number of existing edges a node has, etc.) or ‘exogenous fitness attributes’ that relate to an arbitrary attribute of the node itself. Market research primarily concerns itself with the measurement of exogenous fitness attributes relating to brands such as satisfaction, awareness, brand equity, etc.
- The Value of RankAs we have discussed, ranking is a very important tool for evaluating brands within cognitive constraints. Let us now look at the prominence of various rank positions in respondents’ minds. We know that being ranked first is preferable to being ranked second and we have seen how the market share differences between brands within a specific market scale non-linearly (see Universally Skewed Distributions section). Our panel data shows that the value of rank also scales non-linearly in other areas. If we look at rank in terms of share of wallet, we see that it follows a power law in the cases that we have looked at. For example, in the UK laundry detergent market (see Figure 7), Rank 1 brands in terms of spend receive 69.5% share of wallet on average, Rank 2 brands receive 23.6%, etc. Spending on lower ranked brands rapidly diminishes, and thus, so does the value of measuring these brands.
- Most surveys do not have the luxury of linking back to actual panel behaviour data as shown in Figure 7 though. However, our research has shown that we can still get a good idea of the non-linear manner in which share of wallet is divided up between brands by looking at the share of wallet that a brand receives based on its rank on our general 10-point brand performance question. The specific wording of the question is: “Using the scale shown, when you take into account everything you look for in a [category], how do you rate each [brand]?”This question is useful for understanding respondents’ general rating of a brand as it is holistic in nature (it does not ask about specific aspects of performance) and it relies on relative, rather than absolute, judgments insofar as it asks respondents to implicitly rate brands in the context of other brands in the market. In this way, it mirrors how the mind works and satisfies Fortunato, et. al.’s criterion of an exogenous fitness attribute. When we look at the average share of wallet that a brand receives from a person (based on actual panel purchase behaviour) versus the brands performance rank, we see that Rank 1 brands receive the most, with rapidly diminishing shares as rank decreases. Figure 8 shows our specific results.These results tell us that by measuring a respondent’s Rank 1 brand, we capture the majority of their spend, and by widening the net to capture their remaining top ranked brands, we capture most of their remaining spend. However, past a certain point, we gain little additional information.
- Casting the Net Too WideMany market researchers believe that questions should capture as much information as possible. However, as Figures 7, 8 and much of our other research shows, there is diminishing value in asking questions that are too broad. As we have already discussed, the human mind does not take into account all information when making decisions and so neither should market researchers.Generally speaking, the information that is of most relevance when a person makes a purchase decision is the top-ranked brands in their evoked set (or attributes or similar, depending on the metric being measured) as these brands and their associations come most easily to mind and are the most likely to relate to their future behaviour. These brands have built up strong associations over time through effective marketing, exposure and usage. As we have seen, the size of this group of brands (the evoked set) is quite stable across markets and countries. It is the brands that people rank highly that they are biased towards purchasing and evaluating other brands against, and their decisions are often made quickly, unconsciously and in the moment; or, with what Kahneman (2011) refers to as ‘System 1 thinking”. It is our hypothesis that the important brands to measure in most purchase situations are the brands that automatically come to mind by virtue of their high rank in terms of the criterion in question, and that, by expanding the measurement of a question beyond this, we destroy the clarity of the measurement through the introduction of noise.
- Table 3 shows how the link to actual behaviour is destroyed by asking too much information. Specifically, we correlate several popular awareness (first mention, unaided, aided) and usage (most often, regularly use, past 1 month, past 3 months, ever used) metrics against what people actually did according to twelve months worth of panel purchases. We see that as we start measuring metrics that include increasing numbers of lower ranked brands in terms of awareness and usage, our correlation with actual behavior start to decrease, both at an aggregate level (e.g. “30% of the sample mention Brand X”) and at a respondent level (e.g. “Respondent Y mentions Brand X”). It is also worth noting that some metrics (especially usage metrics) do a bad job of linking to what people actually do at a respondent level, even though their poor performances are hidden at an aggregate level where they show high correlations with what people actually do collectively. This counter-intuitive phenomenon is known as “mutually compensating” error (see Findlay, Michael & Hofmeyr, 2012 for more on this phenomenon). For the purposes of this paper, it is sufficient to state that not all metrics are created equal and that we should only focus on those that link back to actual behaviour while jettisoning the rest in order to reduce survey length, save money and reduce the demands of the respondent task. In the process, we improve data quality by cultivating more responsive and engaged respondents that are more likely to return.
- Finally, as an example of how limiting the respondent task to only a subset of highly ranked responses can actually improve data quality, we investigated multiple ways of measuring attribute assocations. We first measured attribute associations in the standard way, allowing respondents to associate brands with all the attributes in the attribute list (78 in total). This lead to an average of 2.2 associations or ‘ticks’ per respondent as shown in Table 4.
- We then reduced the list of 78 attributes by focusing on those that were actionable and related to actual behaviour. We presented three more respondent cells with a reduced, more relevant list of 36 attributes and further asked them to select which attributes from the list were important to them (i.e. the highly ranked attributes) in three different ways. All three cells only associated brands with the subset of highly ranked attributes that they selected.
- When respondents were able to only answer for a subset of highly ranked attributes chosen from a more relevant list, the average number of ticks went up per respondent giving us richer data to work with.
- Thus, by focusing on the highly ranked attributes, we reduced the respondent task and the time taken to complete the question while actually ending up with richer responses.
- This paper has shown that not all survey information is equally valuable. By including too much information in our analyses, we can actually dilute our understanding of what people do.We have shown that top-ranked brands and responses link best to what people actually do. This is because the human mind, while powerful, has limitations. We all have, as Herbert Simon put it, ‘bounded rationality’. Our research shows that one of these cognitive limits is that people work with an evoked set of about three brands on average. In other words, humans only hold about three brands in mind, on average, when evaluating brands. This finding holds across the 547,945 people that we looked at (although there is obviously individual and sector variation).
- We have further shown how the human mind uses a brand’s relative rank to make decisions, and we have gone some way towards quantifying the value of the top rank positions by showing that the distribution of real-world share of wallet that a brand receives dimishes sharply as its rank decreases.Finally, we have discussed how the mechanism of relative ranking within a constrained evoked set can lead to the characteristically skewed market share distributions that we see in most markets.
- The implications of this paper are potentially far-reaching for market researchers. Our research shows that there is a sweet spot in terms of measuring many popular market research metrics such as usage (most often is best) and awareness (first mention unaided is best). Hitting these sweet spots improves the link between stated survey behaviour and real-world behaviour.
- The benefits of incorporating rank thinking into one’s survey design are many: data quality improves; survey lengths are reduced as fewer questions and responses need to be measured; and, the respondent task is reduced, improving the chances that the respondent will give better quality responses and return to answer surveys in the future.
- On the basis of the findings outlined in this paper, we recommend that all market researchers consider the role that rank plays in the measurement of the metrics that they work with on a daily basis.
- Table 3 shows how the link to actual behaviour is destroyed by asking too much information. Specifically, we correlate several popular awareness (first mention, unaided, aided) and usage (most often, regularly use, past 1 month, past 3 months, ever used) metrics against what people actually did according to twelve months worth of panel purchases. We see that as we start measuring metrics that include increasing numbers of lower ranked brands in terms of awareness and usage, our correlation with actual behavior start to decrease, both at an aggregate level (e.g. “30% of the sample mention Brand X”) and at a respondent level (e.g. “Respondent Y mentions Brand X”). It is also worth noting that some metrics (especially usage metrics) do a bad job of linking to what people actually do at a respondent level, even though their poor performances are hidden at an aggregate level where they show high correlations with what people actually do collectively. This counter-intuitive phenomenon is known as “mutually compensating” error (see Findlay, Michael & Hofmeyr, 2012 for more on this phenomenon). For the purposes of this paper, it is sufficient to state that not all metrics are created equal and that we should only focus on those that link back to actual behaviour while jettisoning the rest in order to reduce survey length, save money and reduce the demands of the respondent task. In the process, we improve data quality by cultivating more responsive and engaged respondents that are more likely to return.