DMM ではレコメンドエンジンに Apache Spark を使用しています。どのような経緯で採用し、どのようなアーキテクチャで実現しているかの説明です。2016/03/29 ヒカラボ発表資料。

1. DMM.COM SPARK

2. 2015/4 - DMM labo
API

3. AGENDA
DMM
Apache Spark
DMM
Tips

4. DMM

5. DMM

8. SPARK
UC Berkekey Apache
Scala, Python, Java, SQL, R API

10. (2014/09)
Mahout
Spark
Java, Scala, Python
GraphLab

12. WHY SPARK
MLlib, GraphX
Hadoop

13. Hadoop

14. item to item
user to item
popular

15. 1. (Tracking API)
2. (Hive on Spark)
3. (Spark)
4. (Sqoop)
5. API(Play)

16. (TRACKING API)
Javascript
API
RDB Hadoop

17. (HIVE ON SPARK)
Spark

18. (SPARK - ITEM2ITEM)

19. val itemToItems = userProducts.join(userProducts).filter {
case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => item1
}.map {
case (user, ((item1, keyword1, score1), (item2, keyword2, score2))) => ((item
}.reduceByKey(_ + _).mapValues(math.sqrt(_)).map {
case ((item1, keyword1, item2), score) => ((item1, keyword1), (item2, score))
}.groupByKey().mapValues(_.toList.sortBy(_._2).reverse.take(config.numDisplayIt
case ((item1, keyword1), items) => items.size >= config.numDisplayItems
}.cache()

20. (SPARK -
USER2ITEM)
MLlib ALS( )

21. val model = ALS.train(ratings.map(_._1), config.alsRank,
config.alsNumIterations, config.alsLambda)
val predictions = model.predict(candidates).groupBy(_.user).map {
case (user, ratings) =>
(user, ratings.toList.sortBy(_.rating)
.reverse.take(config.numDisplayItems))
}.cache()

22. (SPARK)
RDB Hadoop

23. Sqoop MariaDB

24. API
item2item(id: ItemId): List[ItemId]
user2item(id: UserId): List[ItemId]
popular : List[ItemId]

25. DEPLOY AND EXECUTE
Jenkins + Build Pipeline + BuildFlow

26. (2015/09)
Jenkins + Build Pipeline + BuildFlow
Job Script + Git
Hive
Spark
Sqoop
Recommend API(Node.js)
MariaDB(Galera Cluster)

27. Jenkins + Build Pipeline + BuildFlow
Job Script + Management API
Hive on Spark
Spark
Sqoop
Recommend API(Play)
MariaDB(Galera Cluster)

28. Management API
File
Hive on Spark
Hive 3
Play
Spark, Hive UDF Util

30. AB PDCA

31. [ ]

32. 701

33. 75 % ↑
97% ↑

34. TIPS
use dataframes or datasets
hive

35. executor
memoryOverhead
cheat sheet
Top 5 Mistakes to Avoid When Writing Apache Spark
Applications

37. HIVE
Spark
HiveContext
Hive on Spark

38. DATAFRAMES DATASETS
(1.3 - ) Dataframes
(1.6 - ) Datasets
Project Tungsten(1.5 - )

40. Realtime Recommend
Dataframes & Datasets
Graphframes