Bing is using Deep Learning to improve search relevance and counter threats

Frédéric Dubut | @copernix
Christi Olson | @christijolson
#TechSEOBoost
How Bing is using Deep Learning (and modern Machine
Learning) to improve relevance and counter new threats
Search and Spam Fighting in
the Age of Deep Learning
–

#TechSEOBoost
The shifting landscape of Search

#TechSEOBoost
Bing is the largest application of AI at Microsoft

#TechSEOBoost
AI is powering NLP and NLG for Voice Engagements
What should I eat?
Have you tried staring into the
fridge, closing it, then returning
as if there might be something
new in there? Give it a few tries,
then order pizza.

#TechSEOBoost
Ways AI is used in Search Today
Ranking
Recommendations
Query Mapping
Sequence Classification
Computer Vision
…
and many more!

#TechSEOBoost
What is the difference
between AI, Machine
Learning and
Deep Learning?

#TechSEOBoost
Human (cognitive) intelligence exhibited by
technology.
The practice of using algorithms to
parse data, learn from it, then perform
a certain type of task.
A subfield of machine learning. It’s a
technique for training neural networks with
many layers and decision making nodes.
AI, ML, DL build upon each other

#TechSEOBoost
1957 – Term Frequency
1972 – Inverse Document Frequency
𝑇𝐹𝐼𝐷𝐹 𝑞, 𝑑, 𝐷 = ෍
𝑡∈𝑞
𝑇𝐹 𝑡, 𝑑 ∙ 𝐼𝐷𝐹(𝑡, 𝐷)
𝑇𝐹 𝑡, 𝑑 =
𝑓 𝑡,𝑑
𝑑
𝐼𝐷𝐹 𝑡, 𝐷 = log(
𝑁 𝐷−𝑑𝑓𝑡+0.5
𝑑𝑓𝑡+0.5
)
TF – Hans Peter Luhn (IBM)
IDF – Karen Spärck Jones (Univ. Cambridge)
Once Upon A Time

#TechSEOBoost
1994 – BM25 (“Best Match”)
𝐵𝑀25 𝑞, 𝑑, 𝐷 = ෍
𝑡∈𝑞
𝑇𝐹′ 𝑡, 𝑑 ∙ 𝐼𝐷𝐹(𝑡, 𝐷)
𝑇𝐹′
𝑡, 𝑑 =
𝑓𝑡,𝑑 ∙ 𝑘 + 1
𝑓𝑡,𝑑 + 𝑘 ∙ 1 − 𝑏 + 𝑏 ∙
𝑑
avgdl
2004 – BM25F (multiple weighted fields)
෪𝑓𝑡,𝑑 = σ 𝑠 𝑣𝑠 𝑓𝑡,𝑠
෪𝑑 = σ 𝑠 𝑣𝑠 𝑠 avg෩dl
One Step Further

#TechSEOBoost
A Spammer’s Dream
𝑇𝐹′ 𝑡,𝑑 =
𝑓𝑡,𝑑 ∙ 𝑘 + 1
𝑓𝑡,𝑑 + 𝑘 ∙ 1 − 𝑏 + 𝑏 ∙
𝑑
avgdl
෪𝑓𝑡,𝑑 = σ 𝑠 𝑣𝑠 𝑓𝑡,𝑠
෪𝑑 = σ 𝑠 𝑣𝑠 𝑠 avg෩dl
0
0.5
1
1.5
2
2.5
3
0 5 10 15 20
Score
Term Frequency
BM25 (k=1.6, b=0.75, avg doc size)
Keyword Stuffing
(same term repeated many times)
URL & Title Keyword Stuffing
(same term repeated many times in URL & title)
Machine Generated Content
(many different terms in the document)
Thin Content
(short, very targeted documents)
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
0 500 1000 1500 2000
Score
Document Size
BM25 (f=5, k=1.6, b=0.75, avgdl=1000)

#TechSEOBoost
1998 – Link structure
𝑅′ 𝑢 = 𝑐 ෍
𝑣∈𝐵 𝑢
𝑅′
(𝑣)
𝑁𝑣
+ 𝑐𝐸(𝑢)
2006 – User behavior
𝑆 𝑀 𝑑, 𝐼 𝑑, 𝑂 𝑑, 𝑤𝐼 =
𝑤𝐼
1
𝐼 𝑑 + 1
+
1
𝑂 𝑑 + 1
1
𝑂 𝑑 + 1
Adding More Signals

#TechSEOBoost
RankingSignals ?ML
Blending It Together
Machine-Learned Ranking
2005 – RankNet

#TechSEOBoost
A Neural Network… in 2005!
Input layer: 569 features
Hidden layer: 10 hidden nodes
Output layer: document “score”
Cost function: ~number of pairwise errors
RankNet

#TechSEOBoost
Bing was the first search
engine to be powered by
neural networks.

#TechSEOBoost
RankingSignals ?ML
Blending It Together
Machine-Learned Ranking
2005 – RankNet
2006 – LambdaRank
2010 – LambdaMART

#TechSEOBoost
Neural Networks work best with…
• A lot of training data.
• A lot of computing power.
Boosted tree ensembles as an alternative.
Fast forward…
• We have a lot of data!
• We have a lot of computing power!
• Lots of research around NNs.
LambdaMART BM25
<= 2 >2
#BackLinks
<= 2 > 2
Age
<= 3 > 3
1000s of other trees

#TechSEOBoost
2013 – Deep Structured Semantic Models
𝑥 → (𝑦 1 , 𝑦 2 , … , 𝑦 128 )
𝑅(𝑞, 𝑑) = cosine(𝑦𝑞, 𝑦 𝑑)
2014 – Convolutional Latent Semantic Model
(captures rich contextual structures better)
Beyond Keywords

#TechSEOBoost
The Vector Space
“Things to do with children in the Big Apple”

#TechSEOBoost
Duplicate/Copied Content – Find & Replace
REPLACING KEYWORDS WITH SYNONYMS
Bringing the children to NYC, but not sure what the best NYC childs
attractions are? Don’t sweat it – there are dozens of exciting things to do
with your children in NYC! New York City features some of the best toy
shops in the world, and the bright lights and massive skyscrapers make
NYC one of the best places to bring your child! You’ll be dazzled by all of
the fun things to do with children in New York City!
With so many popular childrens attractions and cheap events in NYC, it
can be hard to decide where to look first. Luckily, NYCTourist has you
covered. Scroll below for the top things to do with children in NYC.
New York City is the city everyone deserves to experience at least once. There’s something for
everyone in the Big Apple so don’t let your child miss out on their slice. Travelling with children
may be difficult and finding appropriate attractions may be even harder but it’s possible to do.
From tall city buildings to vast green parks New York City is always exciting, so make sure
your children see that side too. Here are some activities that will satisfy any children Big Apple
appetite and keep you adults interested too.
1. Broadway shows
Let the performers leave you and your children in awe with a magnificent show earning
Broadway status. See classics this summer such as The Lion King, Annie, or Cinderella or for a
little more action there is Newsies and Spider-man. Children will talk about the performances
long after it’s over and with shows playing both on Broadway and off, there will always be
something for the entire family to enjoy.
Top 10 Things for Children This Summer
ORIGINAL
Bringing the kids to NYC, but not sure what the best NYC kids attractions
are? Don’t sweat it – there are dozens of exciting things to do with your
kids in NYC! New York City features some of the best toy shops in the
world, and the bright lights and massive skyscrapers make NYC one of
the best places to bring your kids! You’ll be dazzled by all of the fun
things to do with kids in New York City!
With so many popular kids attractions and cheap events in NYC, it can be
hard to decide where to look first. Luckily, NYCTourist has you covered.
Scroll below for the top things to do with kids in NYC.
New York City is the city everyone deserves to experience at least once. There’s something for
everyone in the Big Apple so don’t let your kids miss out on their slice. Travelling with kids may
be difficult and finding appropriate attractions may be even harder but it’s possible to do. From
tall city buildings to vast green parks New York City is always exciting, so make sure your kids
see that side too. Here are some activities that will satisfy any kids Big Apple appetite and keep
you adults interested too.
1. Broadway shows
Let the performers leave you and your kids in awe with a magnificent show earning Broadway
status. See classics this summer such as The Lion King, Annie, or Cinderella or for a little more
action there is Newsies and Spider-man. Kids will talk about the performances long after it’s
over and with shows playing both on Broadway and off, there will always be something for the
entire family to enjoy.
Top 10 Things for Kids This Summer

#TechSEOBoost
Duplicate/Copied Content – Reordering
NYC Tour Tips
Tip #1: Check the age requirements for tours and tickets. For some
tours you need to be age 6 or more. Broadway shows are typically age
4 or more. And the 9’11 museum is probably best for those 10 and
older.
Tip #2:Do a Tour! They’re an easy and convenient way to see the city.
Guides are great at making it fun and interesting for kids (and adults.)
Believe me, you won’t regret doing a tour.
Tip #3: Book Early! Tours are super popular and sell out months in
advanced. When you see something you like, reserve your dates right
away.
CHANGING THE SEQUENCE OF CONTENTORIGINAL
NYC Tour Tips
Tip #1: Do a Tour! They’re an easy and convenient way to see the city.
away.
older.

#TechSEOBoost
Still Duplicate/Copied Content?
NYC Tour Tips
away.
older.
NYC Tour Tips
away. It’s never to early to start planning.
Tip #3: Check the age requirements for tours and tickets in NYC. For
some tours you need to be age 6 or more. Broadway shows are
typically age 4 or more. And the 9’11 museum is probably best for those
10 and older.
Tours are one of the best ways to get to see a city, especially if you are on a
time crunch, or if the city is huge and you aren’t sure where to start. Here are
three tips to help you with your planning:
Are you planning your trip to New York City and wondering what activities and
things to do to keep your entire family entertained? NYC is great for kids and
families alike. Check out our Tips and Guides for your 2018 visit to NYC with kids.
DIFFERENT
CONTENT
ADDING NEW CONTENTORIGINAL

#TechSEOBoost
Image Source: https://legallogik.com/wp-content/uploads/2017/05/TicketAide-Ticket-Stop-Sign.jpg

#TechSEOBoost
Collect known encoding failures:
{chevy monte carlo}
{chevrolet monte carlo}
different
{where do horses race}
{what are race horses}
same
Being More Resilient
Training
Set

#TechSEOBoost
Key Takeaways
• Deep Learning improves search relevance
by focusing on intent instead of keywords.
• These advanced models will increase the
complexity of technical SEO.
• Quality content will become easier to
match with the appropriate user queries.
• Deep Learning helps detect some of the
more advanced spam techniques…
• …but will create new attack vectors that
we are working hard to anticipate.

#TechSEOBoost
Thank You –
Stay connected. Say Hi.
Ask Questions and
Provide your input!
fdubut@microsoft.com
chriol@microsoft.com

Bing is using Deep Learning to improve search relevance and counter threats

Recommended

Recommended

More Related Content

Similar to Bing is using Deep Learning to improve search relevance and counter threats

Similar to Bing is using Deep Learning to improve search relevance and counter threats (20)

More from Catalyst

More from Catalyst (20)

Recently uploaded

Recently uploaded (20)

Bing is using Deep Learning to improve search relevance and counter threats