Analysis of Data - Curve Fitting and Spectral Analysis

Simulating physical systems on a computer is similar to

experimental work. This is a reasonable analogy because computer
simulations produce data in much the same way as laboratory
experiments. In both cases, one must analyze the output (data).
We now discuss two aspect of data analysis, namely, curve
fitting and spectral analysis, which will involve Fourier
transforms.

Global Warming Example

Carbon dioxide levels have been rising steadily since the

industrial revolution. The figure below show the increase in the
carbon dioxide concentration(parts per million(ppm)) during the
1980s, as measured in Mauna Loa, Hawaii. The measurements were
taken every 14 days starting in 1981. The estimated error is
0.16 ppm.

step=14/365;
>> t=1981+(0:229)*step;
>> plot(t,global1,'+k')
>> axis([1980 1990 335 360])
>> xlabel('Year')
>> ylabel('CO_{2} (ppm)')

Page 1
A question we might ask when analyzing this data set is

What is the estimated rate of increase


of CO2 concentration per year?

This question leads us to study curve fitting, which is the

simplest type of data analysis.

General Theory

Suppose that we have a data set of N points (xi,yi). We want to

fit this data to a function Y(x;{aj}), where {aj} is a set of M
adjustable parameters. Our objective is to find the values of
these parameters for which the function best "fits" the data.
Intuitively, we expect that if our curve fit is good, then the
graph of the data set (xi,yi) and
the function Y(x;{aj}) will show the curve passing "near" the
points as in the figure below.

"i
y • Y(x;{a j })
i
!i
•

• •

xi

We can quantify this statement by measuring the distance between

a data point and the curve

Δ i = Y (xi ;{a j }) − yi





Our curve fitting criterion will be that the sum of the square
of the errors be a minimum, that is, we need to finda set {aj}
that minimizes the function

Page 2
 N N
D({a j }) = ∑ Δ i2 = ∑ ⎡⎣Y (xi ;{a j }) − yi ⎤⎦
2

i =1 i =1

This process will give us the least squares fit. It is not the
only way to obtain a curve fit, but it is the most common.

Often, the data points have an estimated error bar (or

confidence interval), which we write as yi ± σ i . In this case we
must modify the fit criterion to give less weight to points with
the most error. Therefore we define
2
N ⎡Y (x ;{a }) − y ⎤
2
N
⎛ Δi ⎞
χ ({a j }) = ∑ ⎜ ⎟ = ∑ ⎣
2 i j i⎦

i =1 i =1 ⎝σ ⎠ σ 2




i i

which is called the chi-square function. The chi-square function

is the most commonly used fitting function because if the errors
are Gaussian distributed, we can make statistical statements
concerning the goodness of the fit.

Linear Regression

We first consider fitting the data set with a straight line,

Y (x;{a1 , a2 }) = a1 + a2 x
This type of curve fit is known as linear regression. We want to
determine a1 and a2 such that

χ ({a1 , a2 })∑
2
N
[a 1 + a2 xi − yi ]
2

i =1 σ i2
is minimized. The minimum is found by differentiating the chi-
square function and setting the derivatives to zero:
∂χ 2 N
= 2∑ 1
[ a + a2 xi − yi ] = 0
∂a1 i =1 σ i2
∂χ 2 N
= 2∑ 1
[ a + a2 xi − yi ] xi = 0




∂a2 i =1 σ i2
or
a1S + a2 ∑ x − ∑ y = 0
a1 ∑ x + a2 ∑ x 2 − ∑ xy = 0





where


N
1 N
xi N
yi N
xi2 N
xi yi
S=∑ , ∑x = ∑ , ∑y = ∑ , ∑ x2 = ∑ , ∑ xy =∑
i =1 σ i i =1 σ i i =1 σ i i =1 σ i σ i2
2 2 2 2
i =1

Page 3
Since the sums may be computed directly from the data, they are
known constants. We thus have a linear set of two simultaneous
equations in the unknowns a1 and a2. The solution of these
equations is

a1 =
∑ y∑ x 2 − ∑ x ∑ xy
, a =
S ∑ xy − ∑ y∑ x
S∑ x 2 − ∑ x ( ) S∑ x 2 − ∑ x ( )
2 2 2

Notice that if σ i is a constant, that is, if the error is the
same for all points, then the σ's cancel out of these results
and the parameters a1 and a2 are independent of the error bar. In
this case we just put σ i = 1 in the formulas.

Next we want to obtain an associated error bar, σ a j , for the

curve fit parameter aj. Using the standard law of propagation of
errors, we have
2
N
⎛ ∂a j ⎞ 2
σ aj = ∑ ⎜
2
⎟ σi
i =1 ⎝ ∂yi ⎠





using the results above for a1 and a2 and doing some algebra we
get

σ a1 =
∑x 2

, σ a2 =
S
S∑ x 2 − (∑ x ) S∑ x 2 − (∑ x )
2 2

Notice that σ a j is independent of yi .
If the error bars on the data are constant ( σ i = σ 0 ), the error in
the parameters is
σ x2 σ0 1
σ a1 = 0 , σ a2 =
− x − x
2 2 2 2
N x N x
1 1



x =
N
∑ x , x2 = ∑ x2
N
Finally, if the data set does not have an associated set of
error bars, we can estimate σ 0 from the data
1 N
σ ≈ (standard-deviation) =
2
0 ∑
N − 2 i =1
[
2
yi − (a1 + a2 xi )]
2

Note that this sample variance is normalized to N-2 since we
have already extracted tow parameters a1 and a2 from the data.

Many non-linear curve-fitting problems may be transformed into

linear problems by a simple change of variable. For example,

Page 4




Z(x; {α , β }) = α eβ x
may be written as a linear relation using the change of variable

Z(x; {α , β }) = α eβ x
ln Z = Y , ln α = a1 , β = a2




ln Z = Y = ln α + β x ln e = a1 + a2 x
Similarly, to fit a power law of the form

Z(t; {α , β }) = α t β
ln Z = Y , ln t = x , ln α = a1 , β = a2




ln Z = Y = ln α + β ln t = a1 + a2 x
These transformations should be familiar because you use them
whenever you plot data using semilog or log-log scales.

The same minimization procedure as above can be used for fitting

other functions with more parameters.

Goodness of Fit

We can easily fit every data point if the number of parameters,

M, equals the number of data points, N. In this case we have
either built a Rube Goldberg(cartoonist famous for creating
absurdly elaborate machines that performed trivial tasks) theory
or have not taken enough data. Because each data point has an
error, we do not expect the curve to pass exactly through the
data. However, we ask,

With the given error bars, how


likely is it that the curve


actually describes the data?

Of course, if we are not given any error bars, then there is

nothing that can be said concerning the goodness of the fit.

Common sense suggests that if the fit is good, then on the

average the difference between fitted-curve and data should be
approximately equal to the error bar, yi − Y (xi ) ≈ σ i .

Putting this into the definition of χ , we get χ ≈ N . Yet we

2 2

know that the more parameters we use, the easier it is to get

the curve to match the data - the fit can be perfect if M = N.

Page 5
This suggests that we take our rule of thumb for a good fit to
be




χ2 ≈ N − M

Of course, this is only a crude indicator, but it is better than

just "eye-balling" the curve.

If we find χ >> N − M , then we are not using an appropriate

2

function, Y (x) for the curve fit or the error bars, σ i , are too
small. On the other hand, if χ << N − M , then the fit is so
2

spectacularly good that we suspect that the error bars are

actually too large.

Example of linear regression:

First consider the m-file lsftest below. This program generates

a a sample data set using the equations



xi = i , σ i = α , yi = c1 + c2 xi + c3 xi2 + αℜ Gi
where ℜ Gi is a gaussian distributed random number with unit
variance. Note that all the data points have the same estimated
error α . It then uses the linear regression function linreg to
fit a curve to the data. Finally it plots all the results.

Note the full structure of the codes including explanatory

comment and formatted printing.

function [a_fit, sig_a, yy, chisqr] = linreg(x,y,sigma)

% Function to perform linear regression (fit a line)
% Inputs
% x

Independent variable
% y

Dependent variable
% sigma
Estimated error in y
% Outputs
% a_fit
Fit parameters: a(1) is intercept, a(2) is slope
% sig_a
Estimated error in the parameter a()
% yy

Curve fit to the data
% chisqr
Chi squared statistic

% Evaluate various sigma sums

sigmaTerm = sigma.^(-2);
s = sum(sigmaTerm);
sx = sum(x.*sigmaTerm);
sy = sum(y.*sigmaTerm);

Page 6
sxy = sum(x.*y.*sigmaTerm);
sxx = sum((x.^2).*sigmaTerm);
denom = s*sxx-sx^2;

% Compute intercept a_fit(1) and slope a_fit(2)

a_fit(1) = (sxx*sy-sx*sxy)/denom;
a_fit(2) = (s*sxy-sx*sy)/denom;

% compute error bars for intercept and slope

sig_a(1) = sqrt(sxx/denom);
sig_a(2) = sqrt(s/denom);

% Evaluate curve fit at each data point and compute Chi^2

yy = a_fit(1)+a_fit(2)*x; % curve fit to data
chisqr = sum(((y-yy)./sigma).^2); % Chi square
return;

% lsftest program for testing linreg

clear all;

% Initialize data to be fit.

% Data is linear or quadratic(depending input)
% plus random number.
fprintf('Curve fit data is created using the function\n');
fprintf(' y(x) = c(1) + c(2)*x + c(3)*x^2 \n');
c = input('Enter the coefficients as [c(1) c(2) c(3)]: ');
N = 50; % number of data points
x = 1:N;
% x = [1 2 3 ........ N]
randn('state',0); % initialize random number generator
alpha = input('Enter estimated error bar: ');
r = alpha*randn(1,N); % Gaussian distributed random vector
y = c(1) + c(2)*x + c(3)*x.^2 + r;
sigma = alpha*ones(1,N);

% constant error bar

% Fit the data to a straight line(even if quadratic)

M = 2; % number of fit parameters
% Linear regression (straight line) fit
[a_fit, sig_a, yy, chisqr] = linreg(x,y,sigma);

% print out fit parameters, including error bars

fprintf('Fit parameters:\n');
for i = 1:M
fprintf(' a(%g) = %g +/- %g \n',i,a_fit(i),sig_a(i));
end

% graph the data with error bars, and fitting function

Page 7
figure(1); clf; % bring figure 1 window forward
errorbar(x,y,sigma,'or'); % graph data with error bars
hold on;

% freeze plot to add line fit
plot(x,yy,'-k');

%Plot fit on same graph
xlabel('x_i'); ylabel('y_i and Y(x)');
title(['\chi^2 = ',num2str(chisqr),' N-M = ',num2str(N-M)]);

The first plot is a curve fit result with input values

c = [2.0 0.5 0.0] (underlying linear data), α=2.0.

>> lsftest
Curve fit data is created using the function
y(x) = c(1) + c(2)*x + c(3)*x^2
Enter the coefficients as [c(1) c(2) c(3)]: [2.0 0.5 0.0]
Enter estimated error bar: 2.0
Fit parameters:
a(1) = 2.46232 +/- 0.574279
a(2) = 0.484955 +/- 0.0195998

The fit gives the parameters a = [2.46 ± 0.57, 0.485 ± 0.020]

with χ 2 = 46.1 . The fit is good since χ 2 ≈ N − M = 50 − 2 = 48 .

Page 8
The second plot is a curve fit result with input values
c = [2.0 0.5 -0.02] (underlying quadratic data), α=2.0.

>> lsftest
Curve fit data is created using the function
y(x) = c(1) + c(2)*x + c(3)*x^2
Enter the coefficients as [c(1) c(2) c(3)]: [2.0 0.5 -0.02]
Enter estimated error bar: 2.0
Fit parameters:
a(1) = 11.3023 +/- 0.574279
a(2) = -0.535045 +/- 0.0195998

The fit gives the parameters a = [11.30 ± 0.57, -0.5355 ± 0.020]

with χ 2 = 262.8 . The fit is poor since χ 2 >> N − M = 50 − 2 = 48 .

Spectral Analysis

The carbon dioxide data shown earlier fro Mauna Loa, Hawaii has
a general upward trend but also a significant periodicity due to

Page 9
the annual seasonal cycle. If a data set exhibits periodic
oscillations (or if we suspect it contains periodic
oscillations), we want to fit it using trigonometric functions.
This class of problems moves us from the regime of curve fitting
to that of spectral analysis. In these notes we will only
introduce some basic aspects of this vast subject, including
discrete Fourier transform and the power spectrum.
Take a vector of N data points y = [ y1 y2 y3 ...........yN ] ; we call the data
set a time series because transform methods are most often used
in signal analysis. The data is evenly spaced in time, so t j +1 = jτ
where τ is the sampling interval, that is, the time increment
between data points, and j = 0,1,2,.....,N-1. We define the
vector Y, the discrete Fourier transform of y, as
N −1
Yk +1 = ∑ y j +1e−2 π ijk / N




j=0

where i = −1 and k = 0,1,......,N-1. The inverse transform is

2

1 N −1
y j +1 = ∑ Yk +1e2 π ijk / N




N k=0
Note that texts (and numerical libraries) will vary slightly in
how they define this transform, especially in how it is
normalized. You must always carefully check the definition in
any other programming language you might use.
Each point Yk +1 of the transform has an associated frequency,
k
fk +1 =




τN
The lowest (nonzero) frequency is
1 1
f2 = =




τN T
where T is tghe length of the time series. To measure very low
frequencies, we need to analyze long time series. The highest
frequency is
N −1 1
fN = ≈




τN τ
so to measure very high frequencies we need to use a short
sampling rate. For real time series the highest frequency is
actually 1 / (2τ ) , which is the Nyquist frequency. We will discuss
this shortly.

The program below creates a time series

Page 10
 y j +1 = sin(2π fs jτ + φ s )





The signal is a sine wave of frequency fs and phase φ s . The
program evaluates Yk +1 and plots both the signal and its
transform. Note, although y is real, Y is complex, so we
separately consider its real and imaginary parts.

% fttest - Discrete Fourier transform demos

clear all;
% Initialize the sine wave time series to be transformed
N = input('Enter number of points: ');
freq = input('Enter frequency of sine wave: ');
phase = input('Enter phase of the sine wave: ');
tau = 1; % time increment
t = (0:(N-1))*tau; % t = [tau, 2*tau, ......]
y = sin(2*pi*t*freq+phase); % Sine wave time series
f = (0:(N-1))/(N*tau); % f = [0, 1/(N*tau), ....]
% compute transform using direct summation method
% or using fast Fourier transform (FFT) algorithm
Method = menu('Compute transform by','Direct summation','FFT');
if (Method == 1) % Direct summation
twoPiN = -2*pi*sqrt(-1)/N;
for k = 0:N-1
expTerm = exp(twoPiN*(0:N-1)*k);
yt(k+1) = sum(y.*expTerm);
end
else % Fast Fourier transform
yt = fft(y);
end
% Graph of time series and its transform
figure(1); clf;
plot(t,y,'-k');
title('Original Time Series');
ylabel('Amplitude'); xlabel('Time');
figure(2); clf;
plot(f,real(yt),'-k',f,imag(yt),'--k');
legend('Real','Imaginary');
title('Fourier Transform');
ylabel('Transform'); xlabel('Frequency');
% compute and graph power spectrum of the time series
figure(3); clf;
powspec = abs(yt).^2;
semilogy(f,powspec,'-k');
title('Power Spectrum (unnormalized)');
ylabel('Power'); xlabel('Frequency');

Page 11
Let us discuss a few examples (sampling interval τ = 1 for all).
Case #1: N =50 data points, signal frequency fs = 0.2 and
phase = 0, we obtain the sine wave, transform and power spectrum
shown below.

>> fttest
Enter number of points: 50
Enter frequency of sine wave: 0.2
Enter phase of the sine wave: 0.0

Page 12
The discrete sampling of the sine wave is evident from the
jaggedness in the time series plot. Notice that the real part of
the transform is zero and the imaginary part has spikes at
frequencies f = 0.2 and 0.8 (k = 10 and 40). Notice also the
similar plots for the transform and power spectrum(because real
part = 0). The existence of the power spectrum peak at 0.2 makes
physical sense. The other peak will be discussed shortly.

Case #2: N =50 data points, signal frequency fs = 0.2 and

phase = pi/2, we obtain the cosine (because of the phase) wave,
transform and power spectrum shown below.

>> fttest
Enter number of points: 50
Enter frequency of sine wave: 0.2
Enter phase of the sine wave: pi/2

Page 13
Notice that the imaginary part of the transform is zero and the
real part has spikes at frequencies f = 0.2 and 0.8 (k = 10 and
40). Notice also the similar plots for the transform and power
spectrum(because imaginary part = 0).

Case #3: N =50 data points, signal frequency fs = 0.2123 (frequency

does not fall on a grid point in this case) and phase = pi/2, we
obtain the sine wave, transform and power spectrum shown below.

>> fttest
Enter number of points: 50
Enter frequency of sine wave: 0.2123
Enter phase of the sine wave: 0.0

Page 14
Page 15
Notice that we still have a peak around the frequency of the
sine wave, but the structure is more complicated. In this
example, because the frequency of the signal is not equal to a
multiple of 1 / τ N = 1 / 50 , our Fourier transform is not a simple
spike. For this reason the (unnormalized) power spectrum given
by
2




Pk +1 = Yk +1 = Yk +1Yk*+1
looks very different.

Aliasing and Nyquist Frequency

Now we choose f = 0.8, N = 50, and phase =0.

>> fttest
Enter number of points: 50
Enter frequency of sine wave: 0.8
Enter phase of the sine wave: 0

Page 16
Comparing this with the results for f = 0.2 (instead of 0.8) we
find that they are almost identical - the time series only
differ by a phase shift of π. But how is this possible since
these sine waves have completely different frequencies?

Consider the plot below.

>> y1=sin(2*pi*(0.2)*(0:0.1:10)+pi);
>> y2=sin(2*pi*(0.8)*(0:0.1:10));
>> plot((0:0.1:10),y1,'-k',(0:0.1:10),y2,'-k');
>> xlabel('Time'); ylabel('Amplitude')
>> hold on;
>> y11=sin(2*pi*(0.2)*(1:9)+pi);
>> plot((1:9),y11,'or');
>> hold off;

Page 17
The two sine waves have frequencies 0.2 and 0.8 - the former is
shifted by π. When the sampling interval is τ = 1, the two data
sets for these sine waves (the circles) are identical. This
phenomenon is known as aliasing.

Not surprisingly, because of aliasing there is a limit as to how

high a frequency we can resolve for a given sampling interval τ.
This upper bound, called the Nyquist frequency, is
1
fNy =




2τ
For the examples above, fNy = 1 / 2 , since τ = 1. Truncating the
Fourier transform at the upper bound means that we discard the
upper half of the vector Y.

Another way to understand the upper bound is to consider the

following "information" argument. The (real) time series y
contains N data points, but its Fourier transform contains N
complex data points. However, the information content of the
signal and its transform must be the same. Since the transform
contains twice as many data values (a real an an imaginary value
for each point), it must contain a duplicate of the signal. This
is why the Nyquist frequency cutoff truncates the transform
vector by half.

Now look at the results of the following code (read the

comments).

Page 18
% m-file extractfps
% extract frequency and power spectrum
clear;
% a common use of the FFT is to find the
% frequency components of a signal buried
% in a noisy time domain signal. Consider
% the data sample at 1000 Hz. Form a
% signal containing 50 Hz and 120 Hz and
% corrupt it with some zero-mean random noise
t=0:0.001:0.6;
x=sin(2*pi*50*t) + sin(2*pi*120*t);
y=x+2*randn(size(t));
figure('Position',[200,200,300,300]);
plot(y(1:150),'-k');
% it is difficult to identify frequency components
% from looking at the original signal.
% Converting to the frequency domain
% the DFT of the noisy signal y(t)
% is found by taking the 512 point FFT
Y=fft(y,512);
% the power spectral density, a measurement of the energy at
% various frequencies, is
Pyy=Y.*conj(Y)/512;
% the first 256 points
% (Nyquist frequency discussion says that
% same information is contained in discarded
% part of vector)
% can be graphed on a meaningful frequency axis with
f=1000*(0:255)/512;
figure('Position',[600,200,300,300])
plot(f,abs(Pyy(1:256)),''-k')

Original signal with noise:

Page 19
Power spectrum filtering out 50 Hz and 120 Hz signals

Page 20
Clearly this is a wonderful tool for extracting periodic signals
from noise.

The FFT and how it works

In the original "direct summation" definition of the discrete

Fourier transform, the number of operations required to compute
Yk+1 for a single value of k is O(N), where N is the number of
data points in the time series. To compute the entire vector Y
for all values of k (from 0 to N-1), thus requires O(N2)
operations. For many years spectral work was hampered because it
was computationally prohibitive to analyze large data sets.

In 1956, Cooley and Tukey introduced an algorithm that later

became known as the fast Fourier transform (or FFT). They show
that by cleverly rearranging the order in which the calculation
was performed, the number of operations could be reduced to O
(Nlog2N). Their original algorithm was limited to a case where
N=2n, that is, the number of points in the time series is a power
of two. Sophisticated implementations of the FFT such as
MATLAB's built-in version can handle any value of N but are
still most efficient when N is a power of two. If N is a prime
number, the number of operations returns to O(N2). The
computation time using the FFT is significantly shorter than the
direct summation method (both give the same answer).

Finally, let us take a quick look at Fourier series.

Fourier Synthesis

For the purposes of this lab, we define the Fourier cosine/sine

series as:
∞ ∞
f (t) = ∑ an cos(nω t) + ∑ bn sin(nω t)



n=0 n=0

where
⎛ an ⎞ 2 T ⎛ cos(nω t)⎞ 2π
⎜⎝ b ⎟⎠ = T ∫ ⎜⎝ sin(nω t) ⎟⎠ f (t)dt , ω=
T



n 0

There is a theorem of Dirichlet which says that as long as a

function f(t) is piecewise continuous, the Fourier series is
guaranteed to behave itself, and converge to the function. The
nth term in the series is supposed to get small at least as fast
as 1/n, and faster if there are no discontinuities in the
function you’re trying to synthesize.

Page 21
There are even and odd functions in the world, and these need
only cosine or sine series respectively. Every function can be
written as a sum of an even and an odd function.

More commonly in physics, we work with the complex form of

Fourier series:
∞
f (t) = ∑ce n
−inω t

n = −∞

where
∞

−∞
cn = ∫ f (t)einω t dt





You can use an identity of complex exponentials, namely,

eiθ = cosθ + i sin θ

to relate the coefficients an and bn to cn.

Example: f(t) is the square wave function shown below:

⎧+1 for 0 ≤ t ≤ π
f (t) = ⎨
⎩ −1 for − π ≤ t ≤ 0





As discussed in class we have all an = 0 and
⎧ 4 / nπ n=1,3,5,7,......
bn = ⎨




⎩0 n=2,4,6,8,......
which gives

Page 22
 4 ∞ sin nt
f (t) = ∑
π n =1 n
odd

% illustrate Fourier Series

% square wave pulse
% gives sequence of images building up to final curve
x=-pi:pi/100:pi;
val = zeros(1,length(x));
figure(1);
for m=1:2:100
f=4/(m*pi)*sin(m*x);
val=val+f;
m
clf;
plot(x,val,'-k');
axis([-pi pi -1.5 1.5]);
hold on;
pause(1);
end

The final image looks like:

Page 23