Interdisciplinary Mathematical Sciences -

Vol. 19

Kernel-based Approximation
Methods using MATLAB

Gregory Fasshauer
Illinois Institute of Technology, USA

Michael McCourt
University of Colorado Denver, USA

^ World Scientific
BEIJING SHANGHAI HONG KONG TAIPEI CHENNAI
LONDON SINGAPORE
• •
•
NEW JERSEY • • • •
 Contents

Preface vii

An Introduction to Kernel-Based Approximation Methods

and Their Stable Computation 1

1. Introduction 3

1.1 Positive Definite Kernels: Where Do They Fit in the Mathematical

Landscape? 3
1.2 A Historical Perspective 5
1.3 The Fundamental Application: Scattered Data Fitting 7

1.3.1 The Haar-Mairhuber-Curtis theorem: Why using kernels

is a "natural" approach 9
1.3.2 Variations of scattered data fitting 11
1.4 Other Applications 12
1.4.1 Statistical data
fitting 12
1.4.2 Machine learning 13

1.4.3 Numerical solution of PDEs 13

1.4.4 Computational finance 14

1.5 Topics We Do Not Cover 15

2. Positive Definite Kernels and Reproducing Kernel Hilbert Spaces 17

2.1 Positive Definite Kernels 17

2.2 Hilbert-Schmidt, Mercer and Karhunen-Loeve Series 20
2.2.1 Hilbert-Schmidt operators 20
2.2.2 The Hilbert Schmidt eigenvalue problem 22
2.2.3 Mercers theorem 24
2.2.4 Examples of Hilbert-Schmidt integral eigenvalue problems

and Mercer series 25

2.2.5 Iterated kernels 30

xi
 Contents
xii

31
2.2.6 Fourier and Karhunen-Loeve expansions
Kernel Hilbert 32
2.3 Reproducing Spaces
Feature 36
2.4 Maps

41
3. Examples of Kernels

41
3.1 Radial Kernels
kernels 41
3.1.1 Isotropic radial
radial kernels 44
3.1.2 Anisotropic
Translation Invariant Kernels 45
3.2
Series Kernels 46
3.3
Power series and series kernels 47
3.3.1 Taylor
3.3.2 Other series kernels 48

General Kernels 49
3.4 Anisotropic
kernels 49
3.4.1 Dot product
3.4.2 Zonal kernels 50

kernels 52
3.4.3 Tensor product
Radial Kernels 53
3.5 Compactly Supported
Multiscale Kernels 54
3.6
55
3.7 Space-Time Kernels
Learned Kernels 56
3.8
56
3.9 Designer Kernels
3.9.1 Periodic, kernels 57

58
3.9.2 Chebyshev kernels

Kernels in Matlab 61
4.

Radial Kernels in MATLAB 62

4.1
distance matrices in Matlab 63
4.1.1 Symmetric
4.1.2 General distance matrices in Matlab 64

in Matlab 66
4.1.3 Anisotropic distance matrices
Matlab 68
4.1.4 Evaluating radial kernels and interpolants in . .
.

Kernels in Matlab 72
4.2 Compactly Supported
Zonal Kernels in Matlab 76
4.3
4.4 Tensor Product Kernels in Matlab 77

4.5 Series Kernels in Matlab 79

The Connection to
89
5. Kriging

Random Fields and Random Variables 90

5.1
of Spaces 94
5.2 Duality
and Prediction via Kriging 96
5.3 Modeling
5.3.1 best linear unbiased predictor 96
Kriging as

99
5.3.2 Bayesian framework
101
5.3.3 Confidence intervals
 xiii
Contents

105
5.3.4 Semi-variograms
5.4 Karhunen-Loeve Expansions and Polynomial Chaos 106

107
5.5 Generalized Polynomial Chaos

The Connection to Green's Kernels 111

6.

6.1 Introduction Ill

6.2 Green's Kernels Defined 112

6.3 Differential Eigenvalue Problems 114

115
6.4 Computing Green's Kernels

An Computing the Brownian bridge kernel as

6.4.1 example:
115
Green's kernel
Generalizations of the Brownian bridge kernel 117
6.4.2
118
6.5 Classical Examples of Green's Kernels
120
6.6 Sturm-Liouville Theory
121
6.7 Eigenfunction Expansions
6.8 The Connection Between Hilbert-Schmidt and Sturm-Liouville
123
Eigenvalue Problems
6.9 Limitations 124

6.10 125
Summary

Iterated Brownian Bridge Kernels: A Green's Kernel Example 127

7.

Kernels 127
7.1 Derivation of Piecewise Polynomial Spline
127
7.1.1 Recall some special Green's kernels
7.1.2 A family of piecewise polynomial splines of arbitrary odd
129
degree
7.1.3 Benefits of using a kernel representation for piecewise
131
polynomial splines
7.2 Derivation of General Iterated Brownian Bridge Kernels 132

Kernels 134
7.3 Properties of Iterated Brownian Bridge
134
7.3.1 Truncation of the Mercer series
136
7.3.2 Effects of the boundary conditions
orders 139
7.3.3 Convergence
kernels bounded domains 139
7.3.4 Iterated Brownian bridge on . .

7.3.5 "Flat" limits 143

7.3.6 Summary for functions satisfying homogeneous boundary-

conditions 146

Generalized Sobolev 147

8. Spaces
How Native Were Viewed Until Recently 147
8.1 Spaces
8.2 Generalized Sobolev Spaces on the Full Space Rd 152

8.2.1 Two different kernels for H2(E) 155

156
8.2.2 Higher-dimensional examples
xiv Contents

8.2.3 Summary for full-space generalized Sobolev spaces 158

8.3 Generalized Sobolev Spaces on Bounded Domains 158

8.3.1 Modifications of the Brownian bridge kernel: A detailed

investigation 160

8.3.2 Summary for generalized Sobolev spaces on bounded

domains 167

8.3.3 An alternative framework for boundary value problems on

167
[a,b]
8.4 Conclusions 168

9. Accuracy and Optimality of Reproducing Kernel Hilbert

Space Methods 171

171
9.1 Optimality
Different of Error 172
9.2 Types
9.3 The "Standard" Error Bound 172

175
9.4 Error Bounds via Sampling Inequalities
175
9.4.1 How sampling inequalities lead to error bounds
9.4.2 Univariate sampling inequalities and error bounds 176

9.4.3 to iterated Brownian bridge kernels 181

Application
in dimensions 183
9.4.4 Sampling inequalities higher
9.5 Dimension-independent error bounds 184

9.5.1 Traditional dimension-dependent error bounds 185

9.5.2 Worst-case weighted L2 error bounds 185

10. "Flat" Limits 189

10.1 Introduction 189

191
10.2 Kernels with Infinite Smoothness
193
10.3 Kernels with Finite Smoothness
197
10.4 Summary and Outlook

An Unfortunate Misconception 199

11. The Uncertainty Principle -

199
11.1 Accuracy vs. Stability
11.2 and Stability 201
Accuracy

12. Alternate Bases 203

Basis Functions 204

12.1 Data-dependent
Standard basis functions 204
12.1.1

12.1.2 Cardinal basis functions 206

12.1.3 Alternate bases via matrix factorization 208

210
12.1.4 Newton-type basis functions
12.1.5 SVD and weighted SVD bases 215

and Numerical Eigenfunctions 217

12.2 Analytical
 xv
Contents

12.2.1 Eigenfunctions given analytically 218

221
12.2.2 Eigenfunctions obtained computationally
226
12.3 Approximation Using Eigenfunctions
12.4 Other Recent Preconditioning and Alternate Basis Techniques . .
230

Stable via the Hilbert-Schmidt SVD 231

13. Computation
13.1 A Formal Matrix Decomposition of K 232

13.2 Obtaining a Stable Alternate Basis via the Hilbert Schmidt SVD .
235

13.2.1 Summary: How to use the Hilbert-Schmidt SVD 241

13.3 Iterated Brownian Bridge Kernels via the Hilbert-Schmidt SVD .

243

13.4 Issues with the Hilbert-Schmidt SVD 248

13.4.1 Truncation of the Hilbert-Schmidt series 248

13.4.2 Invertibility of 4>i 250

of Alternate Bases for Gaussian Kernels 252

13.5 Comparison

255
14. Parameter Optimization
14.1 Modified Golomb-Weinberger Bound and Kriging Variance ....
256

14.1.1 How to avoid cancelation while computing the power

257
function (kriging variance)
14.1.2 How to stably compute the native space norm of the

interpolant (Mahalanobis distance) 258

14.2 Cross-Validation 260

14.3 Maximum Likelihood Estimation 263

14.3.1 MLE independent of process variance 264

14.3.2 MLE with process variance 265

14.3.3 A deterministic derivation of MLE 266

to the Selection of Good Kernel Parameters 267

14.4 Other Approaches . .

14.5 Goals for Parametrization Judgment Tool 269

a

Advanced Examples 273

15. Scattered Data Fitting 275

276
15.1 Approximation Using Smoothing Splines
15.2 Low-rank Approximate Interpolation 280

286
15.3 Interpolation on the Unit Sphere
Considerations for Scattered Data Fitting 290
15.4 Computational
alternate basis 291
15.4.1 The cost of computing/implementing an . .

structure in kernel computations 292

15.4.2 Exploiting

and Surrogate Modeling 295

16. Computer Experiments
295
16.1 Surrogate Modeling
xvi Contents

16.2 Experimental Design 297

16.3 Models for Standard Test Functions 298

Surrogate
16.3.1 Piston simulation function 298

16.3.2 Borehole function 304

306
16.4 Modeling From Data
16.5 Fitting Empirical Distribution Functions 307

17. Statistical Data via Gaussian Processes 315

Fitting
17.1 Geostatistics 315

17.2 Data Fitting 324

Anisotropic
17.3 Data Fitting Using Universal Kriging and Maximum Likelihood

Estimation 327

18. Machine 335

Learning
Networks 336
18.1 Regularization
18.2 Radial Basis Function Networks 337

18.2.1 Numerical experiments for regression with RBF networks .

339

Vector Machines 343

18.3 Support
18.3.1 Linear classification 344

18.3.2 Kernel classification 346

18.3.3 Numerical experiments for classification with kernel SVMs 350

18.3.4 Computational consideration for classification with kernel

SVMs 354

18.3.5 Linear support vector regression 358

18.3.6 Nonlinear support vector regression 359

19. Derivatives of and Hermite Interpolation 361

Interpolants
362
19.1 Differentiating Interpolants
19.1.1 Cardinal function representation of derivatives 362

19.1.2 Error bounds for simultaneous approximation 363

364
19.1.3 Global differentiation matrices

19.1.4 Local differentiation matrices 369

Hermite 377
19.2 Interpolation
19.2.1 kernel-based Hermite interpolation 378
Nonsymmetric
19.2.2 kernel-based Hermite interpolation 381
Symmetric
19.2.3 Generalized Hermite interpolation via the Hilbert -Schmidt

SVD 383

Gradient 384
19.2.4 An example: interpolation
386
19.2.5 Kriging interpretation
via Derivatives of Eigenfunctions 387
19.3 Doing Hermite Interpolation . . .

19.3.1 Differentiation of a low-rank eigenfunction approximate

388
interpolant
 xvii
Contents

An example: Derivatives of Gaussians eigenfunctions 389

19.3.2 . . .

19.4 Multiphysics Coupling 392

19.4.1 Meshfree coupling 395

An example: coupled 2D heat equation 396

19.4.2
considerations 401
19.4.3 Computational

20. Kernel-Based Methods for PDEs 403

403
20.1 Collocation for Linear Elliptic PDEs

collocation in the standard basis 404

20.1.1 Nonsyinmetric
407
20.1.2 Nonsymmetric collocation using the Hilbert-Schmidt SVD
20.2 Method of Lines 411

20.3 Method of Fundamental Solutions 416

20.4 Method of Particular Solutions 420

20.5 Kernel-based Finite Differences 423

Collocation 425
20.6 Space-Time

431
21. Finance

21.1 Brownian motion 431

21.1.1 Brownian motion and the Brownian motion kernel 432

21.1.2 Geometric Brownian motion 433

434
21.1.3 Pricing options and high-dimensional integration
21.1.4 A generic error formula for quasi-Monte Carlo integration
436
via reproducing kernels
of asset pricing through quasi-Monte Carlo 437
21.1.5 Example ....

21.2 Black-Scholes PDEs 440

Black-Scholes PDEs 441

21.2.1 Single-asset European option through
445
21.2.2 Pricing American options

Appendix A Collection of Positive Definite Kernels and Their

447
Known Mercer Series

A.l Piecewise Linear Kernels 447

kernel 447
A. 1.1 Brownian bridge
A.1.2 Brownian motion kernel 448

linear kernel 448

A. 1.3 Another piecewise
448
A.2 Exponential Kernel
449
A.2.1 Domain: [0,1]
449
A.2.2 Domain: [—L, L]
449
A.2.3 Domain: [0, oo)
A.3 Other Continuous Kernels 450

450
A.3.1 Tension spline kernel
A.3.2 Relaxation spline kernel 451

451
A.3.3 Legendre kernel
xviii Contents

A.4 Modified Exponential Kernel 451

A.5 Families of Iterated Kernels 452

A.5.1 Iterated Brownian bridge kernels 452

A.5.2 Periodic spline kernels 452

A.5.3 Periodic kernels 453

A.5.4 Chebyshev kernels 453

A.6 Kernel for the First Weighted Sobolev Space 454

A. 7 Gaussian Kernel 455

A.8 Sine Kernel 455

A. 9 Zonal Kernels 456

A.9.1 Spherical inverse multiquadric 456

A.9.2 Abel-Poisson kernel 456

Appendix B How To Choose the Data Sites 457

458
B. l Low Discrepancy Designs
in Statistics 460
B.2 Optimal Designs
461
B.3 Optimal Points in Approximation Theory

C A Few Facts from Analysis and Probability 463

Appendix

Appendix D The GaussQR Repository in Matlab 467

D.l Accessing GaussQR 467

D.2 Common functions in GaussQR 468

D.3 Full Hilbert-Schmidt SVD sample solver 469

473
Bibliography

Index 505