3.1.

Six-Aperture Simulations

A large number of data sets were simulated and images were reconstructed, for the purpose of speed, with the six-aperture synthetic aperture shown in Fig. 4. When showing these synthetic apertures here, the sums of the circular apertures are shown, but the actual synthetic aperture is averaged in the areas of overlap rather than summed.

Fig. 4

Six-aperture synthetic aperture.

For reference, an ideal incoherent image through a single one of the six apertures is shown in Fig. 5. In this rendition of the 1951 US Air Force bar target, the finest pair of three bars that can easily be distinguished is indicated by the arrow, which is group 1, element 6, or (1,6) for short. Each group is worth a factor of two in resolution, which is the same as a $\text{delta-}\mathrm{NIIRS}=1$, where NIIRS is the National Image Interpretability Rating Scale.⁹ Each element within a group is worth a factor of $2^{1/6}=1.12246$, or 12.2% additional resolution, equivalent to a delta-NIIRS of $1/6=0.167$. Each delta-NIIRS of 0.1 is worth a factor of $2^{0.1}=1.0718$ in resolution. A delta-NIIRS of 0.1 is considered to be just perceptible.¹⁰ Hence a delta-element, being worth 0.167 delta-NIIRS, would be significantly greater than (1.67 times) just discernible. Table 1 shows the relationship among delta-groups, delta-elements, delta-NIIRS, and ratios of effective resolution.

Fig. 5

Ideal incoherent image through a single aperture. In all images of bar targets, the arrow points to the finest pair of three bars that can easily be distinguished.

Table 1

Relationship between delta-groups, delta elements, delta-NIIRS (loss in NIIRS), and resolution ratio.

For comparison with the incoherent image, in which we can discern (1,6), Fig. 6 shows an ideal coherent, speckled image through the same single aperture, in which we can discern (0,3) or (0,4), or 1 group plus 2 or 3 elements worse resolution, equivalent (according to Table 1) to a loss of 1.3 to 1.5 in NIIRS and a factor of 2.5 to 2.8 in resolution. When viewing such poor images, one should “zoom out” or demagnify the image, putting the finest detail within a favorable part of the contrast sensitivity curve of the human visual system, allowing one to discern the greatest detail. Note that for much larger synthetic apertures, as will be shown later, one must zoom in or magnify the images to see the finest detail.

Fig. 6

Ideal coherent, speckled image from a single aperture.

From Figs. 5 and 6, we see that the effect of speckle in coherent imaging (of any type) of optically rough objects plays a major role in effective resolution (a factor of 2.5 to 3) and image interpretability. Speckle is much more dominant for optical and near-infrared than it is for microwave SAR; in the microwave wavelength regime, the world is much smoother and targets often contain multiple glints that can aid in target recognition.

By gathering multiple ($N_s$) images with different transmitter/receiver locations relative to the object, each image having a different realization of the speckle pattern, one can average the speckled intensities together to yield a reduced-speckle image, which approaches an incoherent image as the $N_s$ approaches infinity. The speckle contrast, initially unity, is reduced by the factor $1/\mathrm{sqrt}(N_s)$.¹¹ Figure 7 shows an ideal speckle-reduced image with $N_s=100$ from a single aperture. It has a noisier appearance than the ideal incoherent image shown in Fig. 5, but it has almost the same resolution. Figure 8 shows an ideal speckle-reduced image with $N_s=100$ from the six-aperture synthetic aperture. With (2,6) discernible, it has a full factor of two better resolution than the single-aperture image, on account of the synthetic aperture having approximately twice the effective width as the single aperture.

Fig. 7

Ideal single-aperture speckle reduced image, the average of $N_s=100$ image intensities.

Fig. 8

Ideal image from six-aperture synthetic aperture with $N_s=100$.

Figure 9 shows the result of simulating data with $N_{\mathrm{pps}}=100$ photons per speckle, performing the image reconstruction, including two-plane phase retrieval, correcting the relative piston, tips, and tilt (PTT) phase for each aperture, and averaging over $N_s=100$ speckled images. Its resolution and quality is comparable to that of the ideal (noise-free) image shown in Fig. 8, demonstrating that our two-plane pupil field reconstruction algorithm and aperture-phasing algorithm are working well, and that $N_{\mathrm{pps}}=100$ is more than enough signal for success. Figure 10 shows the same thing but for the low light level $N_{\mathrm{pps}}=4$, which still has almost the same resolution as for the higher SNR for this high-contrast target.

Fig. 9

Reconstructed image for six-aperture synthesis with $N_{\mathrm{pps}}=100$, $N_s=100$.

Fig. 10

Reconstructed image for six-aperture synthesis with $N_{\mathrm{pps}}=4$, $N_s=100$.

Table 2 shows results from a number of reconstructions for a variety of signal levels and speckle realizations averaged for this high-contrast target. It shows the importance of having at least several speckle frames over which to average. Averaging over 10 speckle frames improved the resolution roughly by a factor of 2. It also shows that resolution does not improve much above $N_{\mathrm{pps}}=2$ or 4 (very low light levels), for this high-contrast target; the reconstruction algorithms worked very well even for these very low signal levels.

Table 2

Results of image reconstruction for the six-aperture synthesis for varying number of photons per speckle (Npps) and number of speckle realizations averaged (Ns). Upper line: group, element just discernible; lower line: delta-NIIRS relative to ideal incoherent image.

In Ref. 12, we showed that the same algorithm worked for the large 72-aperture synthetic aperture shown in the bottom of Fig. 1, for the higher SNRs ($N_{\mathrm{pps}}=100$), but the pupil-phasing algorithm (after fields were successfully reconstructed over individual apertures) was not adequate for low SNRs. To work well for low SNRs and large synthetic apertures, a least-squares reconstruction algorithm is needed, but time did not permit it implementation. In addition, reconstruction was possible with annular apertures, despite missing areas in the synthesized aperture, although the images produced had faint halos surrounding the bright points of the image. Those halos were successfully removed from a speckle-averaged image ($N_s=10$) by a Wiener–Helstrom filter designed for incoherent images.¹² In what follows, we explore the combination of the more difficult annular apertures with the more difficult case of an object of realistic contrast.

3.2.

Nine-Annular-Aperture Simulations with Realistic Contrast

All the imagery shown so far was for a high-contrast US Air Force bar target. One would expect that a low-contrast scene would require a larger SNR than a high-contrast scene. In addition, it can be expected that the aperture-phasing algorithm will have decreasing accuracy with decreasing scene contrast, since it relies on a cross-correlation algorithm to match areas of the images from the different individual apertures. To illustrate that effect, a series of simulation experiments was performed on an image with a more realistic contrast. For this study, the nine-annular-aperture synthetic aperture shown in Fig. 11 was used.

Fig. 11

Nine-annular-aperture synthetic aperture.

For reference, Fig. 12 shows a simulated ideal incoherent image through the nine-annular-aperture synthetic aperture (without Wiener filtering). The original photograph was taken with a Nikon D-90 DSLR camera, from a roof-top of a four-story office building, of a parking lot on a sunny day and includes (left to right) a front loader, a man walking, a water truck, and a pick-up truck, with trees in the background. Figures 13Fig. 14Fig. 15–16 show ideal images for $N_s=1$, 10, 100, and 1000 speckle frames averaged, respectively. For $N_s=1$ (no speckle averaging), one can easily discern that there is an object in the location of the water truck, but not the two other vehicles. For $N_s=10$, one can detect all three vehicles and discern their sizes and shapes. For $N_s=100$, one can also see the walking man and easily see features on the vehicles such as tires. The resolution for $N_s=100$ is comparable to that of the ideal incoherent image, but it has a noisier appearance. For $N_s=1000$, one gets something approaching the ideal incoherent images. From these results we see that for more realistic, lower-contrast (than the bar targets) scenes, one needs a greater number than $N_s=10$ speckle realizations to be able to extract all the information from the images. Further simulations would be required to quantify that number.

Fig. 12

Ideal (noise-free) incoherent image for nine-annular aperture synthesis.

Fig. 13

Ideal (noise-free) coherent image for nine-annular aperture synthesis, $N_s=1$.

Fig. 14

Ideal (noise-free) coherent image for nine-annular aperture synthesis, $N_s=10$.

Fig. 15

Ideal (noise-free) coherent image for nine-annular aperture synthesis, $N_s=100$.

Fig. 16

Ideal (noise-free) coherent image for nine-annular aperture synthesis, $N_s=1000$.

The images of the realistic-contrast scenes aforementioned were for the noise-free ideal images with averaging over speckle realizations. Next, image reconstruction experiments were performed, varying the SNR. For one to appreciate the kind of data that the reconstruction algorithms work on, Fig. 17 shows the noise-free image from a single annular aperture (perhaps the water truck is detectable).

Fig. 17

Ideal (noise-free) coherent image for a single annular aperture, $N_{\mathrm{pps}}=\text{infinity}$, $N_s=1$.

Figure 18 shows a noisy image from a single annular aperture, with $N_{\mathrm{pps}}=4$ (shot noise only; no detector noise). This is the image frame of data going into the Gerchberg–Saxton-like algorithm for estimating the fields in the aperture plane. One can see the individual photon events. Because of the manner in which the simulations were performed, it is sampled at the final resolution of the synthetic aperture, so it is oversampled by about a factor of three in each dimension before the photon noise was applied.

Fig. 18

Noisy coherent image for a single annular aperture, $N_{\mathrm{pps}}=4$, $N_s=1$.

Figure 19 shows the image reconstructed for the nine-annular-aperture synthesis with $N_{\mathrm{pps}}=4$, $N_s=10$. While that amount of noise allowed the aperture phasing to be adequate for the high-contrast bar target, it was substantially degraded for this realistic-contrast target: the image is not just noisier, it is also blurred compared with the ideal noise-free image shown in Fig. 14, because of reduced-quality aperture phasing as well as the phase retrieval for individual apertures in the presence of this large amount of noise.

Fig. 19

Reconstructed image for nine-annular-aperture synthesis with $N_{\mathrm{pps}}=4$, $N_s=10$.

For the case of higher SNR, shown in Fig. 20, with $N_{\mathrm{pps}}=100$ ($\mathrm{SNR}=10$), the aperture-phasing algorithm worked very well, as can be seen by comparing this image with the ideal noise-free image shown in Fig. 14, where they match even at the level of individual speckles. Hence, one needs $N_{\mathrm{pps}}$ to be something greater than 4 but less than 100 for the case of a target with more realistic contrast. Further simulations will be required to narrow down the number of photons needed.

Fig. 20

Reconstructed image for nine-annular-aperture synthesis with $N_{\mathrm{pps}}=100$, $N_s=10$.

3.3.

Aperture Array Phasing Accuracy

The relative phasing of the different apertures within the synthetic aperture, including relative PTT phases where the tip and tilt phases are equivalent to telescope relative pointing errors, were corrected with a subpixel accuracy registration algorithm. The requirements on the accuracy can be thought of as follows. For images, the effects of diffraction due to the finite aperture, and that due to motion blur (similar to summing images suffering from misregistration), both can be described by multiplicative transfer functions in the Fourier domain and by convolutions in the image domain, similar to the result in [Ref. 13, Eq. (8.5-4)] for the case of diffraction and aberrations. Furthermore, if two Gaussians are convolved together (multiplied in the Fourier domain), the result is a Gaussian having a variance equal to the sum of the variances of the two original Gaussians. Similarly, we assume here that the variances of the convolution of some other spread functions approximately add, making the width of the convolution approximately the square root of the sum of the squares of the individual widths. Then for images, if one has a root-mean-squared (rms) pointing error of $s$ diffraction-limited resolution elements, then the net resolution element (here, the final resolution of the synthesized aperture) will have width $\sim \mathrm{sqrt}(1+s^2)$ times the diffraction-limited resolution. In that case, a misregistration by 0.25 resolution elements rms will degrade the resolution by a factor of 1.03, a misregistration by 0.5 resolution elements rms will degrade the resolution by a factor of 1.12 (one element in the bar target), and a misregistration by 1 resolution element rms will degrade the resolution by a factor of 1.41 (three elements in the bar target). Based on those numbers, one might require a residual telescope pointing error, after correction, to be around 0.5 resolution elements or less.

In the 72-aperture simulations done earlier¹² for the high-contrast bar targets for $N_{\mathrm{pps}}=100$, the actual pointing errors in the estimates from the noisy data were 0.12 resolution elements rms, which are negligible. For $N_{\mathrm{pps}}=4$, the actual pointing errors in the estimates from the noisy data were 0.77 resolution elements rms, slightly more than the goal of 0.5 resolution elements, yielding a nonnegligible error. This error would probably be lower than 0.5 if the data were fed into a least-squares reconstruction algorithm, which we did not have a chance to implement. This becomes a bigger problem with the realistic-contrast scene than with the bar targets, but the size of the effect has not yet been quantified. For that realistic-contrast target, it appeared that for $N_{\mathrm{pps}}=4$ the phasing algorithm yielded significant residual errors whereas for $N_{\mathrm{pps}}=100$ the phasing algorithm worked very well.

4.1.

PRF, Speckle Velocity, and Pulse Length

The PRF of the laser should be sufficient to allow no large gaps in the synthetic aperture. The PRF must be fast enough so that, for the example of four rows of apertures shown in the top of Fig. 1, the first row of telescopes moves forward by half its diameter between pulses (the speckles move across the aperture at twice the speed that the aperture moves forward, since both the apertures and the transmitter are moving forward at the same speed). In the bottom of that figure, the first telescope pupil in the synthetic aperture appears again one diameter from its previous position. (The manner in which we are speaking of these things is accurate for imaging in a direction perpendicular to the direction of motion of the synthetic aperture; various geometrical effects must be taken into account when pointing forward or to the side.) Accounting for the fact that the speckles in the pupil plane move backward at the same speed, $v_p$, as the forward motion of the array, a speckle speed relative to the aperture will be $v_s=2v_p$. For a telescope of diameter $D_1$, the synthetic aperture will move by one aperture width in time $D_1/(2v_p)$, which suggests a time between pulses of $D_1/(2v_p)$ and a $\mathrm{PRF}=2v_p/D_1$. For example, for three scenarios, a ground vehicle with $D_1=10\mathrm{cm}$ and $v_p=15\mathrm{m}/\mathrm{s}$, an unmanned aerial vehicle with $D_1=10\mathrm{cm}$ and $v_p=150\mathrm{m}/\mathrm{s}$, and a low-earth-orbiting satellite with $D_1=20\mathrm{cm}$ and $v_p=7.8\mathrm{km}/\mathrm{s}$, the PRFs would be 300 Hz, 3 kHz, and 78 kHz, respectively, requiring very fast detector arrays.

As mentioned in the image reconstruction section, in order to achieve adequate image quality, multiple synthetic-aperture images, each with an independent speckle pattern, are usually needed. Collecting multiple speckle patterns would seem to be less expensive than building a much larger array of telescopes to achieve the desired resolution.

The length of an individual pulse should be short enough to freeze the speckles at the detector, or one must have an optical compensation for translating speckles. The speckles will be moving at twice the speed of the imaging platform, assuming that the laser illuminator is near the receiver traveling at approximately the same velocity. The diameter of one of the speckles in the pupil plane will be

The speckles move their own width in time $d_s/(2v_p)=\lambda R/(2v_p{w}_o)$. To keep the speckle from moving by $1/4$ its diameter (less than that would be desirable), the laser pulse duration would have to be less than $1/4$ of that, or $\lambda R/(8v_p{w}_o)$. This is equivalent to a coherence length of $c\lambda R/(8v_p{w}_o)$, where $c$ is the speed of light. This coherence length allows only objects/scenes of depth half the coherence length (due to the round-trip distance of the reflected light), or of depth $c\lambda R/(16v_p{w}_o)$, to be imaged coherently. Note that reducing the width, $w_o$, of the illuminated object increases the speckle width proportionally and increases the allowable uncompensated speckle motion proportionally. For wide-field imaging (large $w_o$) with a fast-moving platform, the allowed object depth may be unacceptably small, in which case it would be necessary to have, as part of the receiver (or transmitter) optics that scans along with the speckle motion to freeze it over a longer pulse length. Since the pupil will be reimaged to a smaller scale, this can be done with a fast-steering mirror or with an acousto-optic modulator on the reduced-diameter beam.

Independent of the illumination diameter $w_o$, for a width $D$ of the synthesized aperture, the synthesized aperture subtends an angle of $D/R$, but the angular motion is half that, $D/(2R)$. If 10 such synthetic apertures were gathered sequentially without pause in-between, for 10 speckle realizations, the total angular subtense of the synthetic aperture would be $5D/R$. For most scenarios of interest, for long-range imaging, one can collect many synthetic apertures without the object appearing to be different due to different illumination and viewing angles.

4.2.

Doppler Shift and Timing

While heterodyne systems must compensate for the Doppler shift due to the radial velocity of the imaging platform relative to the object, in order for the return beam to properly interfere with the LO, this direct-detection approach is unbothered by Doppler shifts.

For heterodyne detection, great care must be taken so the return pulse arrives at the same time as an LO pulse at the detector array. In the case of our direct-detection system, it is only necessary that the shutter be open when the return pulse arrives and that the shutters of the two detector arrays per telescope are open at the same time.

4.3.

Link Budget

If the product of the two-way atmospheric transmittance, transmitter and receiver optical transmittance and quantum efficiency is $\eta$, and the mean object intensity reflectivity is ${\tau }_o$, then a laser pulse with energy $E_p$ joules (per pulse) at range $R$, reflected from the target area and falling onto a single collecting element of length and width $d_d$ results in a mean number of photons at a pupil-plane detector element of

For a given wavelength and pulse energy, we see that there is a direct trade-off between the number of photons per speckle and the area, $w_o^2$, of the illuminated object or scene. Solving for the illuminated width that gives a desired $N_{\mathrm{pps}}$ ($\sim 4$ for a high-contrast target, according to our image reconstruction simulations) we get

It is interesting to note that Eqs. (3) and (4) are independent of range (aside from absorption losses through the atmosphere) and independent of resolution! For a given resolution, if one doubles the range, then the aperture must double in diameter to preserve the resolution, which results in gathering the same total number of photons. For a given range, to make the resolution twice as fine, one must double the aperture diameter, gathering four times the number of photons, but those photons are spread over four times as many resolution elements, so the number of collected photons per resolution element (i.e., per speckle) stays the same.

Suppose that one has a laser with the high average power $E_L=200\mathrm{W}$ and that we need $N_{\mathrm{pps}}=4\text{photons}/\text{speckle}$ (adequate for a high-contrast target), and that ${\tau }_o=0.2$ and $\eta =0.5$ and $\lambda =1\mu \mathrm{m}$. Then, Eq. (4) predicts the following scenarios:

Note that cases 2 and 3 are independent of the speed of and distance to the sensor platform. The area coverage rate is proportional to $E_L$, which may be produced by either one or multiple lasers.

Case 3 is probably the scenario one would choose to collect, taking advantage of the 10 speckle realizations in order to achieve the desired image quality.

These calculations are for the case of a high-contrast target such as the USAF bar targets. For targets of more natural contrast, larger SNRs (larger values of $N_{\mathrm{pps}}$) and larger numbers of speckle realizations, $N_s$, are needed.

4.4.

Speckle Boiling

Not only do speckles translate as the illumination angle relative to the object changes but they also “boil,” ¹¹ a phenomenon sometimes referred to as the “memory effect.” One might wonder about how this affects image quality. Fortunately the effect is small for remote sensing scenarios. According to Ref. 9, an intensity correlation of $e^{-2}$ occurs for an angular change of

4.5.

Heterodyne Versus Direct Detection

Direct detection with phase retrieval was chosen over heterodyne sensing for this study in order to overcome the difficulties in heterodyne sensing, including the logistics of distributing the LO to multiple, possibly disjoint telescopes, subwavelength stability of the LO relative to the illumination beam, temporal delay needed to interfere the LO with the return beam, correcting the relative Doppler shift of the LO relative to the return beam, etc. Since the telescopes in the group are relatively close to one another, however, reducing some of the difficult logistics, it is worth comparing heterodyne and direct detection.

First, with heterodyne detection, nearly all the light reflected from the object and captured by the aperture of a telescope can go to the single heterodyne channel. This avoiding of splitting the light into two approximately equal channels would appear to imply a sqrt(2) increase in SNR, but that is not the case. For direct detection, still all the photons are collected, but by two detector arrays rather than just one. Furthermore, introduction of the LO, if performed with spatial heterodyne, requires more pixels in the detector array than for direct detection, but direct detection requires two detector arrays rather than just one. From this perspective, heterodyne sensing may or may not have an advantage over direct detection.

Heterodyne definitely benefits from the “heterodyne advantage,” namely, that read noise and dark current essentially go away if the LO is much brighter than the light from the object, which almost always will be the case. This benefit goes away if one employs photon-limited detectors with direct detection.

The computation of the field from heterodyne data can be a purely linear process, whereas phase retrieval reconstruction is inherently nonlinear. One would expect heterodyne to have an image SNR advantage for this reason.

Once the fields in the individual apertures are reconstructed for each telescope, then the remainder of the process—synthetic-aperture assembly, aperture phasing, pupil registration and high-resolution image formation, and averaging over speckle realizations—would be the same whether one performs heterodyne sensing or direct detection with phase retrieval.

For the purpose of comparison with direct detection employing phase retrieval, images were computed to have the same noise statistics as for heterodyne detection (since simulating the entire heterodyne process was beyond the scope of this effort). To compare with Fig. 19 (for direct detection with phase retrieval), zero-mean complex Gaussian random noise was added to an ideal nine-annular aperture synthetic aperture field (having no phase errors) to give it the noise that would be expected with eight photons per speckle, double the number per detector array as for the direct-detection approach. It was assumed that there were zero registration errors or aperture-phasing errors. Figure 21 shows the resulting image. It is sharper than the direct-detection image shown in Fig. 19, with a better definition of the rectangular shape of the water truck in the center of the image. This comparison is unfair in that the heterodyne image was not subjected to the aperture-phasing errors of the direct-detection result. The noise characteristics of the two images are different. With detection only in the pupil plane, the heterodyne image has noise spread over the entire computational window of the image, whereas the phase-retrieval image, constrained by the focal-plane noisy image, has most of its noise energy confined to the illuminated region of the object.

Fig. 21

Heterodyne image with noise appropriate to eight photons per speckle in the pupil plane, $N_s=10$. Perfect aperture phasing was assumed.

Figure 22 shows the heterodyne image with noise appropriate for 200 photons per speckle for comparison with the direct-detection result shown in Fig. 20. In this case, the results are virtually indistinguishable. This is probably because the residual speckle noise dominates over the photon noise. Hence, the noise advantage of heterodyne over direct detection is only a factor at low light levels.

Fig. 22

Heterodyne image with noise appropriate to 200 photons per speckle in the pupil plane, $N_s=10$. Perfect aperture phasing was assumed.

4.6.

Fourier Ptychography Versus 2-Plane Phase Retrieval

Another direct-detection alternative to the architecture analyzed in this paper is Fourier ptychography.^5,6 It is a method for synthesizing a coherent aperture from image-plane intensities, mostly used for microscopy. It involves coherently illuminating the sample from multiple different angles to synthesize a larger aperture. For the long-range imaging application of interest here, however, similar ideas and algorithms can be used to perform aperture synthesis with moving apertures. It is similar to the aperture synthesis described for the two-plane intensities already mentioned, but with the following differences. First, only measurements of the low-resolution focal-plane images are employed. Second, the overlaps of the individual aperture locations within the synthetic aperture are much denser than the overlaps shown in Fig. 1. Hence there is much more redundancy within the synthetic aperture, and the synthetic aperture will be smaller than, and the resolution will be poorer than, for the two-plane approach for a given number of telescopes and laser pulses. In microscopic imaging applications, Fourier ptychography has been shown to be robust with the dense sampling. Requiring only a single detector array in each telescope rather than a beamsplitter and two detector arrays makes the individual telescopes simpler than the two-plane approach. The image reconstruction is different, too. Rather than reconstructing a field across each pupil, synthesizing an aperture from those fields, and then phasing the pupils within the aperture, in Fourier ptychography the phase retrieval is performed directly on the synthesized pupil-plane array. A single synthesized complex array is found that is consistent with all of the measured low-resolution image intensities. Each measured low-resolution image intensity must agree with the images obtained by computing the Fourier transforms of a circularly (or whatever the shape of the individual telescopes) windowed portion of the estimated synthetic-aperture field (and taking the squared magnitude). Note that no measurements are made in the pupil plane in this case. The Fourier ptychography approach (also employing phase retrieval) is expected to have performance that is in the same ballpark as two-plane direct-detection phase retrieval.