introduction of the factorization machine and neural factorization machine,

1. FACTORIZATION MACHINE
Presenter: Shuai Zhang, CSE, UNSW

2. Content
• Factorization Machine (Rendle, 2010, ICDM)
• Neural Factorization Machine (Xiangnan He, 2017, WWW)

3. Factorization Machine
A generic approach that allows to mimic most factorization
models by feature engineering.
Combines the generality of feature engineering with the
superiority of factorization models in estimating interactions
between categorical variables.
It works well on sparse features.

4. Preliminary
Matrix Factorization:
𝑅 ≈ 𝑃 𝑇
𝑄
Low-rank Matrices 𝑃 ∈ 𝑅 𝑈 ×𝐾
and 𝑄 ∈ 𝑅 𝐼 ×𝐾
Regularized Squared Loss
𝐿 = 𝑅 𝑢𝑖 − 𝑃𝑢
𝑇
𝑄𝑖
2
+ 𝜆(∥ 𝑃 ∥2+∥ 𝑄 ∥2)
Variants: SVD++, timeSVD++, etc.

5. Factorization Machine
Most recommendation problems assume that we have a
consumption/rating dataset formed by a collection of (user, item,
rating) tuples.
But we have plenty of item metadata such as categories, tags that
can be used to make better predictions.
Factorization machine is very suitable for feature-rich datasets. It
can learn both low and high order interactions between features.

6. Factorization Machine
Sparse real valued feature vectors x

7. Factorization Machine
Model equation (Degree = 2):
Here, n is the length of the feature vectors
is the dot product of two vectors of size k:
Linear Regression
Higher Order
Interactions

8. Factorization Machine
Following the definition, if we assume that each feature vector
is only made up of user and item one-hot representations.
We have,

9. Factorization Machine
By designing the input features, we can recover many classic
models:
• SVD++
• Pairwise interaction tensor factorization
• Factorized Personalized Markov Chains

10. Factorization Machine
FM can be applied to a variety of prediction tasks.
Regression: least squared error
Binary classification: hinge loss or logit loss
Ranking: pairwise loss

11. Factorization Machine
LibFM: Factorization Machine Library, http://www.libfm.org/
• It is a software implementation for factorization machines
that features stochastic gradient descent (SGD) and
alternating least squares (ALS) optimization as well as
Bayesian inference using Markov Chain Monte Carlo (MCMC)
fastLib, https://github.com/ibayer/fastFM
• It allows you to use Factorization Machines in Python (2.7 &
3.x) with the well known scikit-learn API.

12. Neural Factorization Machine
Expressiveness Limitation of Factorization Machine.
It is essentially a linear model, and it may suffer from
insufficient representation ability for modelling real data with
complex inherent structure and regularities

13. Neural Factorization Machine
NFM is formulated as follows:
Global bias
Linear regression
Core component of NFM. It
is actually a multi-layered
feedforward neural
network

14. Neural Factorization Machine
The core component f(x):

15. Neural Factorization Machine
Embedding layer:
=
Bi-Interaction Layer:

16. Neural Factorization Machine
Hidden layer:

17. Neural Factorization Machine
To summarize, we give the formulation of NFM predictive
model as:

18. Neural Factorization Machine
If we set L the to zero and fix h to a constant vector of (1,…,1),
we can recover the FM model exactly:

19. Neural Factorization Machine
NFM can also applied to a variety of prediction tasks, including
regression, classification and ranking.
Regularization tasks: Dropout

20. Comparison
Feature is consists of User ID, Item ID and Tags.

21. Comparison

22. Comparison

23. References
1. He, Xiangnan, and Tat-Seng Chua. "Neural factorization machines for sparse predictive analytics."
Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information
Retrieval. ACM, 2017.
2. Rendle, Steffen. "Factorization machines with libfm." ACM Transactions on Intelligent Systems and
Technology (TIST)3.3 (2012): 57.
3. https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf
4. https://getstream.io/blog/factorization-recommendation-systems/
5. https://arxiv.org/abs/1505.00641

24. Thanks!

25. Week Report
• Last week
• Reviewed a IJCAI paper.
• Prepared the reading group
• Wrote the Book Chapter: deep neural networks meet recommender
systems
• This week
• Write the Book Chapter
• Review a Journal Paper