This document summarizes a real-world machine learning problem of selecting mobile advertisements to display to users. It describes the problem of choosing from a large pool of ads while dealing with imbalanced click-through data and temporal trends. The solution involved segmenting customers and ads, handling imbalanced data, and using time-series features to improve a ranking algorithm. The results were a 10% increase in clicks and conversions for the mobile operator. Future work may explore more multi-criteria decision making algorithms.

1. A S E T O F R E A L - L I F E M A C H I N E L E A R N I N G
P R O B L E M S A N D S O L U T I O N S F O R M O B I L E
A D V E R T I S E M E N T
S A D I E V R E N S E K E R
S L I D E S A R E A V A I L A B L E A T :
W W W . S A D I E V R E N S E K E R . C O M
Real Life Machine Learning Case
on Mobile Advertisement

2. Outline
 Problem Definition
 Details of Data
 Methodology and Solution
 Results Achieved
 Conclusion and Future Directions

3. Use Case
Mobile Users
GSM Operator
Content
Ads
Mobile Services

4. Mobile Advertisement and Problems
Mobile Marketing in Turkey,
3 Operators
73.2 Million active subscribers
Market Size: 1.4 Billion USD
Question:
Which ad from the
ad giver will be displayed
on the Content?

5. Problems and Data
 Pool of Advertisements
 Customer Profiling (missing info)
 Click streams
 Demography
 Operator / Device info
 Prediction in Real time
 Data Splitting and Selection (Seasonality, Splitting
data (Train / Test))
 Imbalanced Data

6. Temporal Data
In Real Time, No split point for train/test
In experiments you can split

7. Which Data to Use?
Statistics
until now
now
time
morning

8. Which Data to Use?
Statistics
until now
now
time
morning
Statistics
from
Yesterday

9. Which Data to Use?
Statistics
until now
now
time
morning
Statistics
from
Yesterday
Same
time Slot
from
Yesterday

10. Which Data to Use?
Statistics
until now
now
time
morning
Statistics
from
Yesterday
Same
time Slot
from
Yesterday
Statistics
from last
week
Same
time Slot
from last
week

11. Which Data to Use?
Statistics
until now
now
time
morning
Statistics
from
Yesterday
Same
time Slot
from
Yesterday
Statistics
from last
week
Same
time Slot
from last
week
Or Last Month?
Or Last Year?

12. Temporal Feature Selection
 Hour of day
 Day of week
 Special days and events (football games, holidays)
 Last n minutes (what is the optimal period of time in
Time Series analysis?)

13. Imbalanced Data
 Purchase / Non-Purchase
 Less than 1%
 Error rate calculation

14. Methodology
 Feature Extraction
 Customer Segmentation
 Click Streams
 User Agent
 Geographical Information
 Product/Advertisement Segmentation
 Advertisement Network
 Advertisement Look and Feel
 Time Series Analysis
 Time Based Training Data Decision
 Algorithm Selection
 Algorithm Optimization

15. Customer Segmentation

16. Solution: Imbalanced Data Sets
 Synthetic Data generation (SMOTE)
 Anomaly detection / Outlier Detection
 Resampling (Random Undersampling)
 Penalizing the model
Purchase Not purchase
Actual
classPredicted
class
C1 ¬ C1
C1 True
Positives
(TP)
False
Negatives
(FN)
¬ C1 False
Positives
(FP)
True
Negatives
(TN)

17. Advertisement Segmentation
 Predefined Segments and advertisements are
prepared for the given segment by experts
 Matching Algorithms
Customer
Segment
Advertisement
Segment
Match

18. Advertisement Segmentation
 Predefined Segments and advertisements are
prepared for the given segment by experts
 Matching Algorithms
Customer
Segment
Advertisement
Segment
Match
Time
Click
Stream

19. Advertisement Segmentation
 Predefined Segments and advertisements are
prepared for the given segment by experts
 Matching Algorithms
Customer
Segment
Match
Time
Click
Stream
w1
w2 w3
Advertisement
Segment
Ad
click stream factor (γ), content relativeness of web page history item i (η),
time spent on web page (t), publisher relativeness (π), ads previously displayed (α).

20. Implementation and Environment
 Rapid Miner for experiments
 Weka + Java in production
 Some Python, MSSQL Stored procedures and C#
modules for speed.

21. Results
 Previously a ranking algorithm was implemented.
 At the start of week they put 50 new advertisements and they
rank the algorithms with their success in daily basis.
 About 10% increase in clicks and subscriptions (Click
rates: originally 5.2/1000 (reported quarterly), now
6.1/1000), (Subscription rates: originally 38.2% ,
now 45.2%)

22. Future Work
 MCDM Algorithms
 ANP [30], VIKOR [31,32], TOPSIS [33], SAW [34], AHP
[35,36], Decision-Making Trial and Evaluation Laboratory
(DEMATEL) [37], Preference Ranking Organisation Method
for Enrichment Evaluations (PROMETHEE) [38], Data
Envelopment Analysis (DEA) [39,40], ELECTRE [41–44].
Additionally, some new MCDM techniques developed in
recent years, these techniques are; generalized regression with
intensities of preference (GRIP) [45], Complex Proportional
Assessment Method (COPRAS) [46–48], ARAS [48–50],
MOORA [51], and MOORA plus the full multiplicative form
(MULTIMOORA) [52], Step-Wise Weight Assessment Ratio
Analysis (SWARA) [53], Weighted Aggregated Sum Product
Assessment (WASPAS) [54]

23. References
 Teng-Kai Fan, Chia-Hui Chang , "Sentiment-oriented contextual advertising" Knowledge and Information Systems, June 2010, Volume 23, Issue 3, pp 321–344
 Peng-Ting Chen, Hsin-Pei Hsieh , “Personalized mobile advertising: Its key attributes, trends, and social impact “,Technological Forecasting & Social Change,
79 (2012) 543–557
 I.S. Chang, Y. Tsujimura, M. Gen, T. Tozawa, An efficient approach for large scale project planning based on fuzzy Delphi method, Fuzzy Sets. Syst. 76 (3)
(1995) 277–288.
 Seker, S. E., “Computerized Argument Delphi Technique”, IEEE Access, 2015, v. 3, pp. 368 - 380
 . David Jingjun Xu, Stephen Shaoyi Liao, Qiudan Lid, “Combining empirical experimentation and modeling techniques: A design research approach for
personalized mobile advertising applications ”, Decision Support Systems 44 (2008) 710–724
 H. Wold, Introduction to the second generation of multivariate analysis, in: H. Wold (Ed.), Theoretical Empiricism, Paragon House, New York, 1989.
 Abdi. H., & Williams, L.J. (2010). "Principal component analysis". Wiley Interdisciplinary Reviews: Computational Statistics. 2 (4): 433–459
 Kai Li , Timon C. Du , “Building a targeted mobile advertising system for location-based services“, Decision Support Systems, v. 54, 2012, pp. 1-8
 Sandra Soroa-Koury, Kenneth C.C. Yang , “Factors affecting consumers’ responses to mobile advertising from a social norm theoretical perspective“,Telematics
and Informatics, 27 (2010) 103–113
 Chia-Ling ‘Eunice’ Liu, Rudolf R. Sinkovics,, Noemi Pezderka, Parissa Haghirian , “Determinants of Consumer Perceptions toward Mobile Advertising — A
Comparison between Japan and Austria “,Journal of Interactive Marketing 26 (2012) 21–32
 Sevtap Ünal, Aysel Erci, Ercan Keser, “Attitudes towards Mobile Advertising – A Research to Determine the Differences between the Attitudes of Youth and
Adults “,Procedia Social and Behavioral Sciences 24 (2011) 361–377
 Toshihiko Yamakami, “A Long Interval Method to Identify Regular Monthly Mobile Internet Users“,Advanced Information Networking and Applications -
Workshops, 2008. AINAW 2008. 22nd International Conference n2008.
 Seker, S. E. " Temporal logic extension for self-referring, nonexistence, multiple recurrence, and anterior past events", Turkish Journal of Electrical
Engineering & Computer Sciences, v.23, is. 1, pp. 212-230
 Chawla, N., Bowyer, K., Hall, L., & Kegelmeyer, W. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 2002, 16:
341-378.
 Hodge, Victoria J., Austin, Jim, “A Survey of Outlier Detection Methodologies”, Artificial Intelligence Review, v.22, is. 2, 2004
 Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3, Article 15 (July 2009), 58 pages.

24. References
 Saaty, T.L. Decision Making with Dependence and Feedback: The Analytic Network Process; RWS Publisher:
 Pittsburgh, PA, USA, 1996.
 Opricovic, S. Multicriteria optimization of civil engineering systems. Fac. Civ. Eng. Belgrade 1998, 2, 5–21.
 Opricovic, S.; Tzeng, G.H. Multicriteria planning of post-earthquake sustainable reconstruction.
 Comput. Aided Civ. Infrastruct. Eng. 2002, 17, 211–220. [CrossRef]
 Hwang, C.; Yoon, K. Multiple Attribute Decision Making: Methods and Applications, A State of the Art
Survey;
 Sprinnger-Verlag: New York, NY, USA, 1981.
 MacCrimmon, K.R. Decisionmaking among Multiple-Attribute Alternatives: A Survey and Consolidated
Approach;
 DTIC Document; DTIC: Fairfax, VA, USA, 1968.
 Saaty, T.L. On polynomials and crossing numbers of complete graphs. J. Comb. Theory A 1971, 10, 183–184.
 [CrossRef]
 Saaty, T.L. What is the Analytic Hierarchy Process?; Springer: Berlin, Germany, 1988.
 Fontela, E.; Gabus, A. The DEMATEL Observer; DEMATEL 1976 Report; Battelle Geneva Research Center:
 Geneva, Switzerland, 1976.
 Mareschal, B.; Brans, J.P.; Vincke, P. PROMETHEE: A New Family of Outranking Methods in
 Multicriteria Analysis; ULB-Universite Libre de Bruxelles: Brussels, Belgium, 1984.
 Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res.
 1978, 2, 429–444. [CrossRef]

25. References
 Charnes, A. Data envelopment Analysis: Theory, Methodology and Applications; Springer: Berlin, Germany, 1994.
 Roy, B. Classement et choix en présence de points de vue multiples. RAIRO Oper. Res. Rech. Opér. 1968, 2,
 57–75.
 Roy, B. Problems and methods with multiple objective functions. Math. Program. 1971, 1, 239–266. [CrossRef]
 Roy, B.; Bertier, P. La méthode ELECTRE II/une application au media planning. In Proceedings of the 6th
 International Conference on Operation Research, Dublin, Ireland, 21–25 August 1972.
 Roy, B. ELECTRE III: Un algorithme de classements fondé sur une représentation floue des préférences en
 présence de criteres multiples. Cah. CERO. 1978, 20, 3–24.
 Figueira,J.R.;Greco,S.;Słowin ́ski,R.Buildingasetofadditivevaluefunctionsrepresentingareference
 preorder and intensities of preference: GRIP method. Eur. J. Oper. Res. 2009, 195, 460–486. [CrossRef]
 Zavadskas, E.K.; Kaklauskas, A.; Sarka, V. The new method of multicriteria complex proportional assessment
 of projects. Technol. Econ. Dev. Econ. 1994, 1, 131–139.
 Zavadskas, E.K.; Antucheviciene, J. Multiple criteria evaluation of rural building’s regeneration alternatives.
 Build. Environ. 2007, 42, 436–451. [CrossRef]
 Zavadskas, E.K.; Kaklauskas, A.; Turskis, Z.; Tamoš aitiene, J. Selection of the effective dwelling house walls
 by applying attributes values determined at intervals. J. Civ. Eng. Manag. 2008, 14, 85–93. [CrossRef]
 Turskis, Z.; Zavadskas, E.K. A novel method for multiple criteria analysis: Grey additive ratio assessment
 (ARAS-G) method. Informatica 2010, 21, 597–610.
 Zavadskas, E.K.; Turskis, Z. A new additive ratio assessment (ARAS) method in multicriteria
 decision-making. Technol. Econ. Dev. Econ. 2010, 16, 159–172. [CrossRef]
 Brauers, W.K.M.; Zavadskas, E.K. The MOORA method and its application to privatization in a transition
 economy. Control Cybern. 2006, 35, 445–469.
 Brauers, W.K.M.; Zavadskas, E.K. Comparative analysis of MOORA, MULTIMOORA, VIKOR and TOPSIS
 for MOP. In Proceedings of the 9th International Conference on Multiple Objective Programming and Goal Programming
(MOPGP ’10): Book of Abstracts, Sfax, Tunisia, 24–26 May 2010; University of Sfax: Sfax, Tunisia, 2010; p. 51.
 Kerš uliene, V.; Zavadskas, E.K.; Turskis, Z. Selection of rational dispute resolution method by applying new step-wise weight
assessment ratio analysis (Swara). J. Bus. Econ. Manag. 2010, 11, 243–258. [CrossRef]
 Zavadskas, E.K.; Turskis, Z.; Antucheviciene, J.; Zakarevicius, A. Optimization of weighted aggregated sum
 product assessment. Elektron. Elektrotech. 2012, 122, 3–6. [CrossRef]

26. Real Life Machine Learning Case on Mobile
Advertisement
www.SadiEvrenSeker.com
Published in CSCI 2016,
Dec 15 – 17, soon will be
indexed in IEEEXPLORE