This document provides an overview of an upcoming course on inverse problems and regularization. The course will cover three topics: inverse problems, compressed sensing, and sparsity and L1 regularization. Inverse problems involve recovering an unknown signal x0 from noisy observations. Regularization is used to incorporate prior information and make the problem well-posed. Compressed sensing allows signals to be sampled below the Nyquist rate if they are sparse. The L1 norm is used as a convex relaxation of the sparsity prior, allowing sparse recovery problems to be solved as convex programs.

1. Low Complexity
Regularization of
Inverse Problems
Cours #1
Inverse Problems
Gabriel Peyré
www.numerical-tours.com

2. Overview of the Course
• Course #1: Inverse Problems
• Course #2: Recovery Guarantees
• Course #3: Proximal Splitting Methods

3. Overview
• Inverse Problems
• Compressed Sensing
• Sparsity and L1 Regularization

4. Inverse Problems
Recovering x0
RN from noisy observations
y = x 0 + w 2 RP

5. Inverse Problems
Recovering x0
RN from noisy observations
y = x 0 + w 2 RP
Examples: Inpainting, super-resolution, . . .
x0

6. Inverse Problems in Medical Imaging
x = (p✓k )16k6K

7. Inverse Problems in Medical Imaging
x = (p✓k )16k6K
Magnetic resonance imaging (MRI):
x
ˆ
ˆ
x = (f (!))!2⌦

8. Inverse Problems in Medical Imaging
x = (p✓k )16k6K
Magnetic resonance imaging (MRI):
x
ˆ
Other examples: MEG, EEG, . . .
ˆ
x = (f (!))!2⌦

9. Inverse Problem Regularization
Observations: y = x0 + w 2 RP .
Estimator: x(y) depends only on
observations y
parameter

10. Inverse Problem Regularization
Observations: y = x0 + w 2 RP .
Estimator: x(y) depends only on
observations y
parameter
Example: variational methods
1
x(y) 2 argmin ||y
x||2 + J(x)
x2RN 2
Data fidelity Regularity

11. Inverse Problem Regularization
Observations: y = x0 + w 2 RP .
Estimator: x(y) depends only on
observations y
parameter
Example: variational methods
1
x(y) 2 argmin ||y
x||2 + J(x)
x2RN 2
Data fidelity Regularity
Choice of : tradeo
Noise level
||w||
Regularity of x0
J(x0 )

12. Inverse Problem Regularization
Observations: y = x0 + w 2 RP .
Estimator: x(y) depends only on
observations y
parameter
Example: variational methods
1
x(y) 2 argmin ||y
x||2 + J(x)
x2RN 2
Data fidelity Regularity
Choice of : tradeo
No noise:
Noise level
||w||
0+ , minimize
Regularity of x0
J(x0 )
x? 2 argmin J(x)
x2RQ , x=y

13. Inverse Problem Regularization
Observations: y = x0 + w 2 RP .
Estimator: x(y) depends only on
observations y
parameter
Example: variational methods
1
x(y) 2 argmin ||y
x||2 + J(x)
x2RN 2
Data fidelity Regularity
Choice of : tradeo
No noise:
This course:
Noise level
||w||
0+ , minimize
Regularity of x0
J(x0 )
x? 2 argmin J(x)
x2RQ , x=y
Performance analysis.
Fast computational scheme.

14. Overview
• Inverse Problems
• Compressed Sensing
• Sparsity and L1 Regularization

15. Compressed Sensing
[Rice Univ.]
x0
˜

16. Compressed Sensing
[Rice Univ.]
x0
˜
y[i] = hx0 , 'i i
P measures
N micro-mirrors

17. Compressed Sensing
[Rice Univ.]
x0
˜
y[i] = hx0 , 'i i
P measures
P/N = 1
N micro-mirrors
P/N = 0.16
P/N = 0.02

18. CS Acquisition Model
˜
CS is about designing hardware: input signals f
L2 (R2 ).
Physical hardware resolution limit: target resolution f
2
x0 2 L
˜
array
resolution
CS hardware
x0 2 R
N
micro
mirrors
RN .
y 2 RP

19. CS Acquisition Model
˜
CS is about designing hardware: input signals f
L2 (R2 ).
Physical hardware resolution limit: target resolution f
x0 2 L
˜
array
resolution
x0 2 R
N
micro
mirrors
y 2 RP
CS hardware
,
,
...
2
RN .
,
Operator
x0

20. Overview
• Inverse Problems
• Compressed Sensing
• Sparsity and L1 Regularization

21. Redundant Dictionaries
Dictionary
=(
m )m
RQ
N
,N
Q.
Q
N

22. Redundant Dictionaries
Dictionary
Fourier:
=(
m
m )m
= ei
RQ
N
,N
Q.
·, m
frequency
Q
N

23. Redundant Dictionaries
Dictionary
Fourier:
m )m
=(
m
N
,N
Q.
=e
i ·, m
frequency
Wavelets:
m
RQ
= (2
j
R x
m = (j, , n)
scale
position
orientation
n)
=1
Q
N
=2

24. Redundant Dictionaries
Dictionary
Fourier:
m )m
=(
m
N
,N
Q.
=e
i ·, m
frequency
Wavelets:
m
RQ
= (2
j
R x
m = (j, , n)
scale
position
orientation
n)
DCT, Curvelets, bandlets, . . .
=1
Q
N
=2

25. Redundant Dictionaries
Dictionary
Fourier:
m )m
=(
m
N
,N
Q.
=e
i ·, m
frequency
Wavelets:
m
RQ
= (2
j
R x
m = (j, , n)
scale
position
orientation
n)
DCT, Curvelets, bandlets, . . .
Synthesis: f =
m
xm
m
=
=1
x.
=f
Q
x
N
Coe cients x
Image f =
x
=2

26. Sparse Priors
Ideal sparsity: for most m, xm = 0.
Coe cients x
J0 (x) = # {m xm = 0}
Image f0

27. Sparse Priors
Ideal sparsity: for most m, xm = 0.
Coe cients x
J0 (x) = # {m xm = 0}
Sparse approximation: f = x where
argmin ||f0
x||2 + T 2 J0 (x)
x2RN
Image f0

28. Sparse Priors
Coe cients x
Ideal sparsity: for most m, xm = 0.
J0 (x) = # {m xm = 0}
Sparse approximation: f = x where
argmin ||f0
x||2 + T 2 J0 (x)
x2RN
Orthogonal
:
=
= IdN
f0 , m if | f0 ,
xm =
0 otherwise.
f=
ST
(f0 )
m
| > T,
ST
Image f0

29. Sparse Priors
Coe cients x
Ideal sparsity: for most m, xm = 0.
J0 (x) = # {m xm = 0}
Sparse approximation: f = x where
argmin ||f0
x||2 + T 2 J0 (x)
x2RN
Orthogonal
:
=
= IdN
f0 , m if | f0 ,
xm =
0 otherwise.
f=
ST
(f0 )
Non-orthogonal :
NP-hard.
m
| > T,
ST
Image f0

30. Convex Relaxation: L1 Prior
J0 (x) = # {m xm = 0}
Image with 2 pixels:
x2
x1
q=0
J0 (x) = 0
J0 (x) = 1
J0 (x) = 2
null image.
sparse image.
non-sparse image.

31. Convex Relaxation: L1 Prior
J0 (x) = # {m xm = 0}
Image with 2 pixels:
x2
J0 (x) = 0
J0 (x) = 1
J0 (x) = 2
null image.
sparse image.
non-sparse image.
x1
q=0
q
priors:
q=1
q = 1/2
Jq (x) =
m
|xm |q
q = 3/2
q=2
(convex for q
1)

32. Convex Relaxation: L1 Prior
J0 (x) = # {m xm = 0}
J0 (x) = 0
J0 (x) = 1
J0 (x) = 2
Image with 2 pixels:
x2
null image.
sparse image.
non-sparse image.
x1
q=0
q
q=1
q = 1/2
Jq (x) =
priors:
m
Sparse
1
prior:
|xm |q
J1 (x) =
m
|xm |
q = 3/2
q=2
(convex for q
1)

33. L1 Regularization
x0 RN
coe cients

34. L1 Regularization
x0 RN
coe cients
f0 = x0 RQ
image

35. L1 Regularization
x0 RN
coe cients
f0 = x0 RQ
image
K
w
y = Kf0 + w RP
observations

36. L1 Regularization
x0 RN
coe cients
f0 = x0 RQ
image
K
w
= K ⇥ ⇥ RP
N
y = Kf0 + w RP
observations

37. L1 Regularization
x0 RN
coe cients
f0 = x0 RQ
image
K
y = Kf0 + w RP
observations
w
= K ⇥ ⇥ RP
Sparse recovery: f =
N
x where x solves
1
min
||y
x||2 + ||x||1
x RN 2
Fidelity Regularization

38. Noiseless Sparse Regularization
Noiseless measurements:
x
x=
x
argmin
x=y
m
y
|xm |
y = x0

39. Noiseless Sparse Regularization
Noiseless measurements:
y = x0
x
x=
x
argmin
x=y
m
x
x=
y
|xm |
x
argmin
x=y
m
y
|xm |2

40. Noiseless Sparse Regularization
Noiseless measurements:
y = x0
x
x=
x
argmin
x=y
m
x
x=
y
|xm |
x
argmin
x=y
m
y
|xm |2
Convex linear program.
Interior points, cf. [Chen, Donoho, Saunders] “basis pursuit”.
Douglas-Rachford splitting, see [Combettes, Pesquet].

41. Noisy Sparse Regularization
Noisy measurements:
x
y = x0 + w
1
argmin ||y
x||2 + ||x||1
x RQ 2
Data fidelity Regularization

42. Noisy Sparse Regularization
Noisy measurements:
y = x0 + w
x
1
argmin ||y
x||2 + ||x||1
x RQ 2
Data fidelity Regularization
x
argmin ||x||1
|| x y||
x
Equivalence
|
x=
y|

43. Noisy Sparse Regularization
Noisy measurements:
y = x0 + w
x
1
argmin ||y
x||2 + ||x||1
x RQ 2
Data fidelity Regularization
x
argmin ||x||1
|| x y||
Algorithms:
Iterative soft thresholding
Forward-backward splitting
see [Daubechies et al], [Pesquet et al], etc
Nesterov multi-steps schemes.
x
Equivalence
|
x=
y|

44. Inpainting Problem
K
Measures:
y = Kf0 + w
(Kf )i =
⇢
0 if i 2 ⌦,
fi if i 2 ⌦.
/