Lecture for Neural Networks study group held on February 8, 2020. Reference book: http://hagan.okstate.edu/nnd.html Video: https://youtu.be/TyyoPU13ME0 Python demo codes: https://bit.ly/3893GHB Initiated by Taiwan AI Group (https://www.facebook.com/groups/Taiwan.AI.Group/permalink/2017771298545301/)

1. Chapter 17
Radial Basis Function Network
Jason Tsai (蔡志順)
February 8, 2020
Mozilla Community Space Taipei

2. *Copyright Notice:
Most materials from this presentation are taken
from the book “Neural Network Design 2nd edition”
authored by Martin T. Hagan, Howard B. Demuth,
Mark Hudson Beale and Orlando De Jesús. The other
quoted sources are mentioned in the respective
slides. This presentation itself adopts Creative
Commons license.

3. Radial Basis Function
*Figure adopted from https://bit.ly/381JXsV

4. Analogy to Biology

5. In Brief
• Radial basis functions (RBFs) were originally used for
interpolating unstructured data as an approximator in
high dimensions. It’s also used in kernel methods, e.g.
support vector machine (SVM).
• It’s local and suffers from the “curse of dimensionality”.
• A classical two-layer RBF network is commonly trained in
two stages. First the RBF centers (weights) and scaling
parameters (biases) in RBF layer are calculated, and then
the weights (and biases) of the output layer are adapted.
The RBF centers can be trained through unsupervised or
supervised learning. The regular output layer is adapted
by supervised learning.

6. Miscellaneous RBFs
*Figure adopted from https://bit.ly/37cPOfb

7. Multivariable Interpolation
*Figure adopted from https://bit.ly/2tm8WZi

8. RBF Network Architecture
*Figure adopted from https://bit.ly/2FJ9lYj

9. RBF Network
Gaussian Basis Function
When Gaussian function is used:
(Sometimes b is omitted.)

10. RBF Network (Cont’d)

11. Parameter Variations

12. Parameters Matter!
*Figure adopted from https://bit.ly/2QVlCiO

13. XOR Example

14. XOR Example (Cont’d)

15. XOR Example Decision Regions

16. Two categories Example
*Figure adopted from https://bit.ly/2FJ9lYj

17. Two categories Example (Cont’d)
*Figure adopted from https://bit.ly/2FJ9lYj

18. Training
RBF center:
• Subset of training set
• Use Kohonen competitive layer, k-means, …, etc. to
perform clustering
• Use subset selection techniques, such as orthogonal
least squares
• Set bias with degree of overlap between RBFs
Learning:
• Define cost function, such as linear least squares
• Train or fine-tune with gradient descent backpropagation

19. Linear Least Squares

20. Orthogonal Least Squares

21. OLS Algorithm

22. Backpropagation

23. Deep RBFN in History: LeNet-5
*Figure adopted from https://bit.ly/3706EgX

24. Further Reading
• Pourya Habib Zadeh, Reshad Hosseini, Suvrit Sra,
"Deep-RBF Networks Revisited: Robust Classification
with Rejection", arXiv preprint (2018),
https://arxiv.org/abs/1812.03190
• Andrew Hryniowski, Alexander Wong, "DeepLABNet:
End-to-end Learning of Deep Radial Basis Networks
with Fully Learnable Basis Functions", arXiv preprint
(2019), https://arxiv.org/abs/1911.09257