A HIGH RELIABILITY CONTROL SYSTEM

J. Callahan, J. Collins, A. Qualls, IUCF, 2401 Milo B. Sampson Lane, Bloomington, IN 47408
W. Hunt, Indiana University Computer Science Dept., Bloomington, IN 47405
VME Master Crate - 6U
Abstract TX

This paper describes part of an innovative control system FO

Ports
developed for the Cooler Injector Synchrotron (CIS). RX

This system achieved specifications oriented to VMIC

5531M
accelerator applications which could not be met with
DEC Alpha
commercial hardware and hence may be of interest to
other laboratories. The system displays high levels of
performance, reliability, maintainability, and VME Slave Crate - 3U Power Supply
repeatability. It is VME based and employs fiber optic TX
CH0
DAC DAC
(DAC)
data transmission for high noise rejection; actively FO CH1
ADC ADC
Ports (ADC)
redundant modules with automatic switch over for high CH2
RX (DAC)
reliability; and built in test and self diagnosis with CH3
(ADC)
centralized failure and system health monitoring for rapid VMIC HI-REL DAC/ADC
CH 1 CH 2
5531S Module
maintenance. Several modules, designed and
manufactured at IUCF, are described. This paper focuses Figure 1 Beamline DAC/ADC - Overall System Design
on the Non-ramping Controls Subsystem.

1 OVERALL SYSTEM 2 REPEATABILITY

There were four major goals which drove the CIS control Repeatability refers to being able to reproduce runs,
system design: precision, repeatability, reliability, and perhaps years later. We frequently run different energy
maintainability. The overall system architecture was levels and different particles and, historically, it has taken
driven by the requirement for a precise control system. many hours to re-tune the cyclotrons and cooler after
Electromagnetic Interference (EMI), which is intense in each energy/particle change. There are several reasons for
our environment, limits the precision of DACs and ADCs this including magnet hysteresis, but a significant cause is
connected to power supplies over long copper cables. To drift in the DAC/ADCs and the power supplies, as well as
combat this we designed the system as a fiber optic star calibration differences when DACs and power supplies
network with the DACs and ADCs located near or in the are replaced after failures. There are two kinds of drift
power supplies themselves. (Figure 1). Locally, each which are important here. One is temperature drift
DAC/ADC module has opto-isolation to avoid interaction induced by changing temperatures, the other is long term
among power supplies. The central computer, a DEC drift induced by changes in component characteristics
Alpha, drives two VME 6U communications crates over time. Our DAC/ADC modules display 3 ppm/°C of
located adjacent to the computer. The communications temperature drift. To attain this level, at reasonable cost,
crates are populated with VMIC 5231M fiber optic (FO) we were forced to design our own DAC/ADCs. These
communications modules which are linked to remote 3U modules operate by generating a ramp whose end points
and 6U crates by fiber optic cables. Each remote crate has are tied to voltage references of great temperature
a VMIC 5231S FO module which plugs into the local stability (3 ppm/°C). The input and output voltages are
VME backplane. The remote crates can hold up to ten HI- then compared (Figure 2) to the ramp in order to set the
REL DAC/ADC modules. Another consideration in output voltage (DAC) or measure the input voltage
providing precise control is to avoid interactions between (ADC). A byproduct of this design is that the same
power supplies. When DACs/ADCs controlling multiple channel can serve as either a DAC or an ADC depending
power supplies have large common mode signals on the on an external switch selection. The modules which we
control lines, unwanted interactions between the supplies have produced have four channels which can be set to
can result. To prevent this interaction, the DAC/ADC any combination of DAC or ADC (Figure 2).
modules were designed with optically isolated data and a
low capacitance power supply module which were also
designed and manufactured at IUCF. This approach
provides near total noise immunity (160dB @ 60 Hz) and
permits highly accurate control of power supplies and
other devices.

0-7803-4376-X/98/$10.00  1998 IEEE 2449

 12MHz 17 Bit
Down
16 Ultra Linear
module, backplane, or backplane power supply is
Oscillator
Ramp Generator
Counter
Reset
+10.000 detected by the module as a loss of communication. If the
V ref
Start
Point
To
FPGA detects a loss of communications it switches to the
V ref C o u n t Compare Integrator - VME
62768 Compare
+
16 Bit
Register
BUS
backup module. The master communications modules at
Ramp
Rate
the main computer are also located in two separate 6U
Zero
Count
Compare Integrator -
Compare 16 Bit
VME crates so that a failure of the primary crate results
32768
+ Register
in a switch over of all primary modules to their backups.
-10.000 -
The only common point in the system is the Alpha
V ref
Compare
+
16 Bit
Register
computer itself which is not duplexed.
From ADC in
VME CH 0
BUS 16 Bit
Register
Compare Integrator -
Compare 16 Bit
5 RELAIABILITY
DAC out
+ Register

ADC in
Another major design goal is reliability. Beam time is
CH 1
16 Bit
Register
Compare Integrator -
Compare
expensive. We wanted CIS to be highly reliable and we
16 Bit
DAC out
+ Register
were also laying ground work for the proposed Light Ion
CH 2
ADC in

-
Synchrotron and for possible medical applications of
16 Bit Compare Integrator
Register
DAC out
Compare
+
16 Bit
Register
proton therapy which require high reliability. The first
ADC in
step in “reliability by design” is to insure that individual
CH 3
16 Bit
Register
Compare Integrator -
Compare 16 Bit
modules have as high a reliability as possible. Good
DAC out
+ Register
design, careful component selection, and heavy parts
derating yield a high reliability design. The next step is
Figure 2 16 Bit, 4 Channel DAC/ADC Block Diagram
carefully controlled manufacture to insure that the design
reliability is actually achieved. At IUCF, we developed an
3 REDUNDANCY ISO9000 compliant production facility (as yet unaudited)
To combat long term drift, we take advantage of a feature with full Electrostatic Discharge (ESD) protection to
which was originally designed for reliability reasons, as produce these modules. We have had approximately 50
will be described presently. Each DAC can be read back units in service for 9 months with no failures to date.
by the central computer and its output compared to its 5.1 MEAN TIME BETWEEN FAILURE (MTBF)
input, providing a direct measure of drift. If the DAC is in
calibration, the computer examines the output of the However, when large numbers of modules are used in a
power supply as measured by the ADC. If drift is system, the overall system MTBF is additive and hence,
detected here, we know that either the ADC or the power can be quite low, even when the individual MTBFs are
supply is in error, but not which. However, each power high. One of the best ways to improve system MTBF is to
supply is controlled by two DAC/ADC modules in an provide active redundancy. Active redundancy provides a
actively redundant configuration. The central computer back up module which takes over automatically when the
therefore has the ability to compare the output of two primary module fails. What makes active redundancy so
separate ADCs. If both agree, to within calibration attractive is that the probability of failure of two devices,
accuracy, we know that the power supply has probably in a given interval, is the product of the individual failure
drifted or malfunctioned. If the ADC’s differ, we know rates. When those rates are low, the product is extremely
the problem is with the ADCs. If the problem is with the low. If failures are detected immediately and the failed
ADC, the central computer has the ability to switch the unit replaced quickly, the system MTBF can approach
module to it’s backup and the primary module can then years. In CIS, critical devices are controlled by two
be removed and repaired or recalibrated. This feature also DAC/ADC modules. The primary module is normally in
permits us to automatically scan the system and control. The DAC/ADC module has a field
determine that all DACs/ADCs and power supplies are programmable gate array (FPGA) designated the “health”
functioning and are within specifications. A key FPGA which continuously monitors fourteen (14)
requirement of this approach is to calibrate power parameters within the module. If these parameters go out
supplies and DACs/ADCs which are nearing their of established ranges, a failure is detected. The primary
calibration limits. DAC/ADC module then relinquishes control to the
backup. The switch over occurs in less than a millisecond
4 COMMUNICATIONS LINK and will not usually be noticed by the controlled device at
all. The failure information is sent to the main computer.
The communication modules which link the DAC/ADC
The primary unit is also monitored by it’s backup module
modules to the main computer do not have automatic
via a “heartbeat” line. If the primary unit fails
switch over. To protect against communications failure,
catastrophically or loses power, the back up unit takes
the primary and backup DAC/ADC modules are located
over automatically. As mentioned previously, the main
in different crates. A failure in the communications

2450
computer can also direct a changeover to the backup
module if the primary module drifts out of calibration.

5.2 MEAN TIME TO REPAIR (MTTR)

Maintainability determines mean time to repair and
consists of two components: diagnosis and repair time.
Typically time spent diagnosing failures contributes 90%
of MTTR. In order to minimize MTTR, we designed in a
number of built in test features to facilitate diagnoses of
problems. For active redundancy to be effective, it is
necessary to detect failures and replace the offending
module. The DAC/ADC modules report their health and
primary/backup status to the main computer and display
it on front panel LEDs. The failed unit can then be
replaced. The modules have a “hot swap” capability
which allows us to replace a failed module without
turning off power to the crate. We expect an order of
magnitude reduction of MTTR from approximately 3.1
hours to typically 15 minutes.

6 CONCLUSION
The non-ramping control system displays high reliability
and very low drift. Other portions of the CIS control
system are described in other papers.

2451