Autocorrelation and Crosscorrelation

We must now change our focus for a while away from the synthesis of random processes, to some mathematical measures that are essential if we are to develop practical uses of these ideas. Our goal now is to develop some mathematical measures and properties of random processes that will help us when the complete statistical nature of the process is not known. In a practical sense, we can estimate probability density functions from real measurements of a random process - a practice we have employed several times using synthesized data to generate probability density functions from histogram plots. We need to interpret these results cautiously though. They were typically developed from a single member function of a process and we need to understand if and when this might be a reasonable assumption.

What we have seen evidenced in our simulation studies for instance is that the mean and variance of certain random process can be estimated numerically from a single realization of a process. This was done "without knowledge" of the probability density function of the random variable. The algorithms we employed used time-series data to estimate these measures rather than numerous realizations of the random process. We had implicitly assumed that this was a valid approach and must now formalize some of these ideas. In the end we will define two concepts - wide-sense stationarity and ergodicity. These are often necessary assumptions as we translate our ideas about random signals to real applications.

Expected Values of a Random Process

The Mean Function

We introduced the idea of the expected or mean value of a random process earlier in our discussions and it is reasonable to revisit the idea here before moving on to the notion of autocorrelation. Recall that the expected value or mean function of a continuous-time random process is denoted as,

Thus, the expected value or mean function of a random process is, in a general sense, a function of time. In certain cases however, particularly for wide-sense stationary processes, this mean function will be a constant. Clearly to determine this mean function requires complete knowledge of the probability density function, p_X(x).One of our goals in the next few weeks is to understand what we can do when this is not known. Our connection will be through the time-domain analysis of a typical member function of x(t) where we will equate certain time-domain measures with the expected value of the process.

For now, let's review two familiar examples. Ex. 6.1 and 6.2 in the textbook calculate the expected values of two different sine wave random processes.

It is worth noting the essential differences in these two results. In the first case, the mean function is a sine wave whose magnitude is the mean value of the random variable A. In the second case, the mean function is a constant and is equal to zero. It is easy to see why this is the case by considering some typical member functions of each random process.

For any times t₁ and t₂ in example 6.1, the mean values of the ensemble will be the mean of a uniformly distributed random variable between 0 and 1 (a₀=1 in this simulation). If t₁ and t₂ are different then the mean values will be scaled relative to one another by the time function sin(w t).

For any time t₁ or t₂ in example 6.2, the mean value of the ensemble will be the mean value of a random variable that ranges between -1 and 1 (a=1 in this simulation). Although these may not be uniformly distributed in this case, the probability density function will have even symmetry over the interval [-1,1] and the mean value will then be equal to zero. [See example 4.28 on page 123 of textbook.]

For random processes with a finite number of member functions - n, the probability density function of the process is discrete and the mean function of the process can be shown to be,

Example 6.10.

A random process, X(t), consists of two member functions, namely x₁(t)=1 and x₂(t)=sin(t). Each member is equaly likely to occur with equal probability. Calculate the mean function and the autocorrelation function for the random process. Is the random process wide-sense stationary? Explain why the process in not strict-sense stationary in terms of it's n-th order probability density function.

The Autocorrelation Function

We will now define a new expectation measure of a random process X(t) that is known as the autocorrelation function. In its most general form it looks somewhat intimidating. Importantly though, for a particular class of random processes (those that are wide-sense stationary) it reduces to a simpler form that we can interpret more easily. This simpler form is what we typically use and we often assume that random processes that occur in real problems satisfy the conditions for this class of processes. This is, in fact, often a necessary assumption in order to develop tractable solutions.

The two forms of the expectation are given below.

An opportunity: As we have seen before, these integrals may be daunting for some of you but they are ultimately "just one more integral" that we have to deal with. Hopefully, this presents an opportunity for success not for failure [Sorry - that is a real cruddy statement!]. In some senses you may find it appropriate to compare it to the convolution integral. In both cases we have a product of two time functions that are integrated, and in both cases there is a variable t that represents a shift in the position of one function relative to the other. In the convolution case, we integrate with respect to t and the result is a function of time. In this case however, the variable t does not disappear after integration as it does in the case of the convolution integral. Rather, here we are integrating with respect to one or two random variables and the result after integration is a function of both time (t) and t . This is true in the most general of cases although in the case of wide-sense stationary processes, the autocorrelation function becomes a function of t only.

A cautionary note: The autocorrelation integral above shows two integrations - one with respect to the random variable for x₁(t) and one with respect to the random variable for x₂(t) (or in the simplified case - with respect to x(t) and x(t+t )). If these are independent, identically distributed random variables then the joint probability density function is simply the product of the respective pdfs. [Section 4.6, Ex. 4.20, Pg. 111] Importantly, this is true for processes that are strict-sense stationary [Ex. 6.6, pg.212]. From a practical perspective, when calculating the autocorrelation function under these conditions, one of the random variables "integrates out", and we need only integrate with respect to one of the random variables. If X(t) however is a function of more than one random variable (see example 6.9, pg. 215) and these random variables are independent, then we need to use the joint probability density function in the calculation of the autocorrelation integral. This will simply be the product of both independent pdfs, and integration with respect to each random variable will be necessary.

A useful property: If the process is also ergodic, then we can estimate the autocorrelation function R_X(t ) from a single member function of the process. In this case, we make use of the following integral relationship known as the time autocorrelation function. We can note two things: the integral does not require knowledge of the probability density function and the similarity with the convolution integral is even more apparent.

A special case: When the random process consists of a finite number of functions - n, the probability density function of the process is discrete and the autocorrelation function of the process can be shown to be,

(1) We must practice the calculation of this autocorrelation function based on ensemble averaging and using known probability density functions.

(2) We must learn to interpret these results in a constructive way. "What does it tell us about the random process?" is an obvious question. Is it related conceptually to the expected values of a random variable that you encountered earlier?

(3) We must answer the question - "Can we use the time autocorrelation function above, that calculates the autocorrelation function from a member function of a random process, and interpret these results within our existing probabilistic framework?"

Some sample calculations

Example 6.3.

Calculate the autocorrelation function for the sine wave random process X(t) = A sin(wt) for all t, where w is a constant and A is a uniform random variable distributed between zero and a₀.

Example 6.4.

Determine the autocorrelation function of the sine wave random process X(t) = Y(t) sin(wt) for all t, where w is a constant and Y(t) is a random process. 

Wide-sense stationarity

It is worthwhile that we clarify now what we mean by wide-sense stationary processes. Processes in this class have non-time-varying mean and autocorrelation functions. Thus,

Example 6.7.

Determine the autocorrelation function for the sine wave random process, X(t) = a sin(w t+q ) for all t, where a and w are constant, while q is a uniformly distributed random variable on the interval [0,2p ]. Verify that the process is wide-sense stationary.

Example 6.10.

A random process, X(t), consists of two member functions, namely x₁(t)=1 and x₂(t)=sin(t). Each member is equaly likely to occur with equal probability. Calculate the mean function and the autocorrelation function for the random process. Is the random process wide-sense stationary? Explain why the process in not strict-sense stationary in terms of it's n-th order probability density function.

The Crosscorrelation Function

Given two random processes, X(t) and Y(t) (commonly the input and output of some system), we define the expected value of the product of the random processes as the crosscorrelation function, provided it exists. Since we can form this cross-correlation in two different ways, there are two cross-correlation functions that are generally not equal. These are expressed as,

From a mathematical perspective, the calculation of this expectation is similar to that of the autocorrelation function and should not present too many new difficulties. Again, it is important to remember that the integration here will be with respect to the random variables that define the random processes and not with respect to time. It is easy to get confused here since you are very comfortable and familiar with similar integrals in time-domain analysis.

We can extend our interpretation of strict-sense and wide-sense stationarity to include both random processes. Essentially, if the joint nth-order probability density function is independent of t , the processes are jointly strict-sense stationary. Also, if both processes are wide-sense stationary and the cross-correlation function is independent of time, then the processes are jointly wide-sense stationary.

Random Sequences - Parallel Definitions

Our definition of the mean function, and the auto- and cross-correlation function extends directly to the random sequences that we have considered earlier in the course. In addition, the concepts of strict- and wide-sense stationarity are paralleled in a random sequence context. The following definitions are made.

Interpreting the Autocorrelation and the Crosscorrelation Functions

Now that we can calculate auto- and cross-correlation functions, albeit with "simple" random processes and assuming complete knowledge of the probability density functions, we can address the question of - "What do they mean?" In this regard, some of the properties of the function are helpful and some simulation studies will help develop an intuitive feel for what these functions look like. You should expect that the functions are not simply developed for the heck of it, although their practical application will not really be developed until the next course in this sequence.

Properties of the Autocorrelation Function

  The mean square value of the random process is equal to the value of the autocorrelation function at t = 0.

  The autocorrelation function is an even function of t .

  The largest value of the autocorrelation function always occurs at t = 0.

  If the random process has a dc component, or non-zero mean value, then the autocorrelation function will contain a constant component. If the process is ergodic, then the value of the autocorrelation function for large values of t is equal to the squared value of this component.

  If the random process contains a periodic component, then the autocorrelation function will contain a periodic component with the same frequency.

  If the random process is ergodic with zero mean, and has no periodic components, then

  The autocorrelation function cannot have an arbitrary shape. In particular, the autocorrelation function cannot contain flat tops, vertical sides, or any other type of amplitude discontinuity.

  The autocorrelation function is not unique. Different random processes can have the same autocorrelation function.

   The autocorrelation function does not provide sufficient information about a random process to determine its statistical characteristics.

A Detailed Example - Binary Processes and their Autocorrelation Functions

 This example will be based on the binary signal examples discussed in the course text and in the supplementary handout.

Numerical Estimation of the Autocorrelation and Crosscorrelation Functions

Numerical algorithms that estimate the time auto- and cross-correlation functions from time-series data are coded in MATLAB as a function in the Signal Processing Toolbox. Since the auto- and cross-correlation functions are essentially the same computation, the algorithm computes estimates of both functions. Recognizing the limits placed on us by our assumptions of stationarity and ergodicity, we can use the algorithm effectively to develop some understanding of the nature and properties of both functions.

Why crosscorrelation?

The crosscorrelation function, like the autocorrelation function, is a bit difficult to get a real handle on. It is not as intuitive yet as the convolution operation that you are so familiar with in deterministic signal and system analysis but has the same sort of relevance. You will get to this in the follow on course to this one. For now we need to accept that fact that the crosscorrelation function provides us with some measure of how one signal depends on another. When, these two signals are the input and output of some system, the crosscorrelation function provides a vehicle for signal and system analysis and design.

Ensemble and Time Crosscorrelation Functions

Like the autocorrelation function, the crosscorrelation function is an expectation function of the product function formed using models of two random signals, X(t) and Y(t). Since this product can be formed in two different ways, we have two crosscorrelation functions.

From a mathematical perspective, the calculation of this expectation is similar to that of the autocorrelation function and should not present too many new difficulties. Again, it is important to remember that the integration here will be with respect to the random variables that define the random processes and not with respect to time. It is easy to get confused here since you are very comfortable and familiar with similar integrals in time-domain analysis.

The calculation of these integrals requires complete knowledge of the random process, including knowledge of the probability density functions of the processes X(t) and Y(t). Since X(t) is usually some random process for which a finite number of ensemble functions have been observed, and since Y(t) is often the output process of some system, this is information is rarely available in most practical situations. In general, to effectively use these integral calculations, we need to make assumptions about the random processes based on the available experimental data.

A practical solution to the dilemma here lies in the use of the time crosscorrelation functions. As before, the use of these functions and/or their numerical approximations, requires assumptions about ergodicity. That is, we assume that for some processes, the time crosscorrelation function provides reliable estimates of the ensemble crosscorrelation functions.

Our notions of stationarity also extend to crosscorrelation functions. If the processes X(t) and Y(t) are individually stationary (either in the strict or wide sense), and the cross-correlation functions depend on t only, then the processes are jointly wide-sense stationary. Like wise, if the joint n-th order, probability density function depends on t only, then the processes are jointly stationary in the strict sense.

Some Properties

 The quantities R_XY(0) and R_YX(0) have no particular significance and do NOT represent mean square values of any random process.

 Crosscorrelation functions are NOT even functions of t . The do however satisfy the symmetry property,

This is useful since if we know one of the cross correlation functions, we can simply determine the other using this symmetry property.

 The crosscorrelation function does not always have its maximum value at the origin.

 If two random processes are statistically independent (one process does not depend on the other), then the following property is satisfied

 If the mean value of either random process then is zero, the crosscorrelation function is zero for all values of t .

An Example of the Utility of The Crosscorrelation Function

It is readily shown (See Example 6.21, Pg. 234) that the crosscorrelation of the input and output of a linear system excited by white noise is an estimate of the impulse response of a system. This provides an experimental mechanism for system identification where the time crosscorrelation function can be used to estimate input-output crosscorrelation based on experimental data. Mathematically this is developed as follows.

We can model this process using a SIMULINK linear system simulation and by processing the simulation data in MATLAB.

 Construct the SIMULINK model shown below. Set the maximum simulation step length to 1 mS. This will "force" the simulation to run at a fixed step length and avoids the need to sample the input/output data at a uniform sample time.

 Run the simulation. Three variables should appear in the MATLAB workspace: the input X, the output Y and the simulation time vector tout.

  Code the following file to estimate the input autocorrelation function (it should be impulsive), the linear system's impulse response, and the input-output crosscorrelation that should approximate the system's impulse response.

 Explore the utility of the identification method by changing the dynamics (poles and zeros) of the system model transfer function and by changing the parameters used to define the crosscorrelation estimate.