21, rue d’Artois, F-75008 PARIS B3-202 CIGRE 2016

http : //www.cigre.org

Brown Field Implementation of an IEC 61850 Based Integrated Protection and

Automation System

M. PIMENTA, D. BURKART, M. SEITER, D. PRATT,

O. OMELCHENKO, I. ANDINO, T. MCMULLEN
Consolidated Edison Company of New York
USA

SUMMARY

This paper discusses the design and implementation decisions made to successfully upgrade an
existing, in service, high voltage substation to a modern IEC 61850 based protection and automation
system. The discussions will delve into the design strategies, practical issues encountered, lessons
learned, and recommendations for future projects.

On the positive side, there are the well-known actual and potential benefits of IEC 61850 standard
based systems. These include interoperability, reduced installation and life cycle costs, and future-
proof design, among others. On the other hand, there are obstacles and practical considerations that
erode some of these benefits and create unforeseen project risks and problems in the design and
implementation of such a system.

The high voltage substation that is being upgraded was heavily damaged by a flooding event. This
meant that much of the legacy copper wiring used for the protection and automation systems was
compromised and rendered unreliable. A primary driver for selecting an IEC 61850 solution was an
immediate savings of approximately 75% in the cost of replacing all the damaged and compromised
copper wiring and going with a fibre optic based Local Area Network instead. Ongoing reliability
benefits also accrue by virtue of the optical fibre invulnerability to interference, corrosion, electrical
short circuits and grounding issues.

The modernized substation uses two fully separate and independent lines of relay protection. That
means there are two separate and independent LANs, two sets of Protection Relays, and separate and
independent PTs and CTs. This project is the first instance where the Company has installed IEC
61850 based automation and protection on both lines. In a number of previous installations, IEC
61850 was used in only one of the two lines of protection.

The Company decided to implement IEC 61850 at the Station Bus level only. Process Bus
implementation was considered and ultimately discarded due to risk concerns related to the current
state of market maturity. In our judgement, current vendor solutions are relatively recent entries into
the Process Bus market and lack a sufficiently solid record of multiple years of in-service reliability.

[email protected]
An added challenge for a Brown Field installation is that the substation must provide uninterrupted
service to all power customers in the area while undergoing this complete renovation. This greatly
increases the complexity of the installation as compared to a green field project: The intermediate
transitions have to be carefully planned. Interface equipment has to be installed to provide the
connection between the new Automation and Protection system and the legacy equipment remaining
in adjacent sections of the system. The interface equipment must be configured to accommodate
adjacent section upgrades. To minimize the risks attendant to making drastic changes to the system
each time a new section is upgraded, the Vendor was required to deliver a system which is fully
programmed for its ultimate configuration. Also, no modifications to the database, HMI graphics or
any other core system functions are permitted once the first stage is commissioned and becomes
operational. Similarly, when cutting in a new upgraded section, no configuration changes are allowed
to any IEDs or protection devices installed in previous stages.

KEYWORDS

IEC 61850, Relay Protection, SCADA, Brown Field, High Voltage Substation

1
1. Basic System Requirements

General Requirements

The system design had to adhere to a number of core design principles.

1. Redundancy: No single component failure could result in a loss of function. This requirement
is applied very stringently to the Protection System and less so to other, less critical functions.

2. Modularity: To the greatest extent possible, the system should be comprised of identical
modules. This extended to the various types of equipment cabinets, but also to the network
architecture and the Relay Protection zones. (identical cabinets, connectorized panels,
modular fibre optic ring subnetworks, etc.)

3. Stability: The project schedule calls for installing various sections of the system in stages over
a time span of multiple years. That means the system will be in operation after the first section
is installed. To avoid the risks associated with major changes in data base and re-configuration
of Protection Relays, which could affect those sections already in operation, the system had to
be designed and delivered fully ready for its final configuration. No database changes or Relay
Protection configuration changes are allowed to any portion of the production system already
in operation.

4. Security: The system must be impervious to external cyber threats while allowing normal
SCADA interaction and providing corporate LAN access to the valuable information collected
by the advanced IEDs. No routable external connections were allowed. This was
accomplished via the use of Data Diodes, non-routable serial SCADA link and dedicated test
and maintenance laptops.

Specific Requirements:
Due to a number of underlying considerations having to do with Company standards and
recommended practices, the overall design was further constrained by certain requirements: The final
design had to include Vendor specific hardware such as Protection Relays, Data Diodes, and Windows
Servers; It had to support IEEE 1588 also known as Precision Time Protocol (PTP); And it had to be
sized for future growth such that all critical system resources would have 100% spare capacity (in
terms of computing resources, LAN bandwidth utilization, database sizing, etc.)

Initial Concerns:
At the outset, one of our primary concerns was Interoperability among the various system components.
A single Vendor solution, where all system components are manufactured by a single company
sidesteps this issue. For this project however, due to the pre-selected Relay Protection IEDs and other
system components, a single Vendor solution was not an option.

Even though Interoperability is a major goal of the IEC 61850 standard, our research had shown that
other Utilities had run into issues when attempting to integrate systems with devices from multiple
manufacturers. This is a natural result of the ongoing maturation process for the IEC 61850 standard
and the fact that fairly wide latitude is given as to how each Vendor implements the required
functionality in their products. To mitigate this risk, we specified that interoperability tests were to be
done very early in the project design cycle and complete compatibility had to be demonstrated by the
system Integrator.

A second major concern was how the system would perform in high activity or stressed conditions. It
is critical that the Protection GOOSE message timing must not be affected or degraded by lower
priority LAN traffic. Preliminary calculations showed that LAN bandwidth utilization, for example,
should not exceed 4% to 5% under any but the most extreme circumstances. However, it was essential
to demonstrate actual performance during system integration testing at the FAT. For this purpose, the

2
Specifications included a requirement to perform a stress test at 30% bandwidth utilization. During the
actual test, we ran the LAN traffic all the way to 60% bandwidth utilization (way beyond any expected
real world conditions) and observed absolutely no impact on the timing of the Relay Protection
functions.

Figure 1: Simplified System Block Diagram

Migration was also among the highest priority concerns for us. As explained earlier, the system is
being installed in stages over multiple outages over a period of years. During each outage, a single
bank of transformers is scheduled for cutover from the old protection to the new IEC 61850 based
system. The Vendor had to deliver a solution that required no changes to the Relay Protection
configuration, and no major changes to the system database, HMI displays, networking or custom
logic. To minimize the chances of adversely affecting the portions of the system already online, this
entire process needed to be automated and tested in advance.

2. Design and Implementation Choices

A complete upgrade of the entire substation including all 138kV ring bus protection was considered
during the project scoping phase. However, as the main justification for this project was storm
hardening the substation against flooding events, this was considered as a “target of opportunity” or
secondary objective. The additional estimated costs to do a complete overhaul of the entire station
were outside of the available budget. Because of budget considerations and the fact that the 138KV
ring bus is on an elevated platform and above the flood line, the scope of the upgrade was limited to
345kV protection, Low-Side feeder protection, and Bus protection in sections with upgraded feeder
protection. This created the requirement to integrate the existing electro-mechanical protection to the
communication-based architecture being installed. Additionally, the existing metering and protection
alarms needed to be retained and migrated to the MMS protocol.

The most significant issue with relay protection was determining how to transition from existing
breaker failure schemes. The supervision and directionality of the existing schemes presented
numerous challenges during the migration design. The first was that the existing bus tie breaker failure
schemes were current-supervised and bi-directional, while the current company standard when
installing new breaker failure was to use directional, current-supervised protection.

3
For simplicity, the Company decided to interface the existing protection schemes at the Self-reset
Lock-Out Relays (SR/LOR). This approach minimized the need for retesting the existing protection
because the CT and Trip circuits remained intact throughout migration or until retired/upgraded. These
also provided abundant spare decks allowing the existing trips, alarms, and target indicating relays to
persist, where necessary or desirable.

Existing metering would be replaced with values from new protection IED where possible. Otherwise
the existing mA outputs would be acquired by a device that could communicate engineering values to
the Data Concentrator in client-server protocols. Similarly, existing alarms would be acquired and
communicated to the Data Concentrator.

Outage Sequence: Outages to install upgraded 138kV bus protection are dictated by the outages for the
345kV transmission feeders. These outages are planned to minimize system impact and are therefore
dictated by System Operations. Engineering based the design on a fixed outage sequence supported by
System Operations. Each bus section will be migrated to the new protection scheme based on the
outage sequence.

There is a commonality to each section in terms of the number of outages required to migrate it. Each
section would experience a main outage, when its bus protection was upgraded or migrated completely
to the communication based system, and two side outages, when the section to either side experienced
its main outage. During a main outage where all protection equipment is upgraded on the bus section,
the adjacent bus side outages install protection interface devices (with final configuration) to connect
to the legacy protection. When the adjacent bus undergoes a main outage wiring is removed from
interface devices for legacy equipment removed from service. The GOOSE message structure is left
intact, but the initiating hardwired contacts are removed. This allows the interface device to be
deployed with a single configuration that will require no change during subsequent outages.

From an operational point of view, the people most affected by the IEC 61850 based architecture are
the Protection technicians responsible for configuring, testing and maintaining the equipment.
Therefore the testing considerations and the final decision on how to approach and implement testing
became a central design issue. Along with this, other fundamental design options were carefully
evaluated:

IED Test Mode: The testing procedures and methodologies are very different from the legacy
systems that they have been trained on and are accustomed to. The Company devised a testing
approach based on opening a Test Switch to place individual IEDs in Test Mode. This was consistent
with the way testing was performed at other non-IEC 61850 stations. Once in Test Mode, a device will
only respond or react to GOOSE messages if they originate from another IED which is also in Test
Mode. The common format for GOOSE message Control Block was designed to include this “Test
Mode GOOSE Bit”. This design, although similar in some respects to the Test Mode operation which
is part of the IEC 61850 Edition 2 implementation, is a custom solution developed by Con Edison to
be as compatible as possible to current operational procedures.

Parallel Path Logic: Additionally, the Company implemented a parallel logic path that allows running
full simulated tests on the Protection Relay devices where each test can involve from 3 to 6 or more
IEDs.

The parallel path logic was created to assist with the initial testing and future maintenance testing of
the GOOSE protection system. The HMI GOOSE screens provide the operator with a global view of
all GOOSE statuses and diagnostics. Each IED has Target LEDs configured to Alarm “Publisher in
Test Mode” and “GOOSE Bad Quality”. Using test equipment or a Simulator (i.e. Power System
Simulator) as a GOOSE simulator involves programming the relays with a logical OR gate for parallel
path messages. Parallel path GOOSE messages allow for the field device to remain fully in service

4
while simultaneously having the capability to perform testing without the need for any device
reprogramming.

The engineer/operator has the ability to test individual status bits or use custom scheme algorithms.
The Test Mode flag for the Simulator is always enabled when testing. See figure below for a high-
level overview of how parallel path GOOSE testing operates. The consistent design of parallel path
logic with fixed (reserved) blocks of received virtual bits eliminates the need for complicated testing
procedures while preserving test method flexibility. The testing of the primary device transmitting
functionality cannot be verified with simulation. Traditional test set methods can verify pickup
settings.

Figure 2: Parallel Path Logic

VB001:= Test Mode; VB002:= Quality Flag; VB003:= TRIP (BF Initiation);
VB061:= Simulated Test Mode; VB062:= Simulated Quality Flag;
VB063:= Simulated TRIP (BF Initiation)

Edition 2: Consideration was given to going with Edition 2 IEC 61850 for this project. We opted
however to stay with Edition 1, mainly due to the fact that at the time of procurement the Relay
Protection IEDs selected for this project were not available on the market with Edition 2 capability.

Process Bus: As mentioned in the Summary above, the Company decided against including IEC
61850 at the Process Bus level in the project scope for the initial implementation. This was based on
the limited availability of commercial devices with a proven long term track record of reliability in
real world conditions and also on limiting the risk exposure for the overall project to a comfortable
level. Given that the Company had already embarked on a number of “first time” solutions for this
project and opted for a completely new system architecture, it was felt that implementing a full
Process Bus solution was taking on too much risk. We are planning however, to install and test
Merging Units in the future at strategic points in the substation and gradually transitioning the system
over depending on the results of the pilot installations.

PRP and HSR: Both options: Parallel Redundancy Protocol (PRP) and High-availability Seamless
Redundancy (HSR) were researched and considered for this project, as would be expected for a system
where Redundancy was a deliberately chosen design principle. The rationale for excluding these as
system requirements was based on the system having two complete and separate lines of relay
protection. Given this fact, we then calculated the probability of completely losing relay protection on
both lines during a fault event. If we assume a fault event occurs on average once a day (let’s assume,
for the sake of using round numbers, the event lasts 100 msec), and that the Substation LAN also
reconfigures itself once a day (obviously both assumptions here are far beyond expected worst case
conditions) and that the LAN reconfiguration requires 100 msec to recover. The probability of a

5
system fault event coinciding with a LAN reconfiguration event is (100msec/24 hours) X (100msec/24
hours) which is on the order of 10-12. Then when we consider that even if this unlikely coincidence
should occur and the first line of protection is not available when needed during a fault, we have a
second line of protection that would clear the fault, it becomes evident that the additional marginal
reliability provided by a zero loss solution does not justify the extra costs or increased complexity.

Virtual LANs: We also looked closely at VLANs as a mechanism to reduce and isolate network
traffic. The choice of multicast filtering was between VLAN based filtering and Multicast (MAC)
Address filtering. IEC61850 requires several attributes be defined for all multicast messages (GOOSE
and MV), one being VLAN, another being Multicast Address. The Multicast address is set in the
Substation Configuration Description (SCD) file and should not be confused with Media Access
Control address, which is specific to a network interface and set in hardware (though they are
confusingly abbreviated the same). Also note that Multicast addresses do not have to be unique. Under
VLAN filtering, ports are members of one or more VLANs. If a multicast message is received at a port
that is not a member of that VLAN, the port does not forward that message on that port. Under MAC
filtering, a port registers to receive Multicast messages from zero or more Multicast addresses.

Engineering determined that relatively few devices required Multicast filtering. The switches selected
for the project implemented a block-by-default policy for both filtering types. This meant that a table
entry was required to register each membership for each port. However, the switch configuration
allowed ports to not participate in MAC filtering and simply forward all messages. By judicious
selection of Multicast addresses, it was possible to limit the table entries to only the port of interest
and few entries per port. It should also be noted that the protocol used for MAC filtering (GMRP) can
be automatically used to discover paths through the network and thus requires no configuration as long
as the end devices comply to GMRP, which is not common in IEDs.

DMS: The Disturbance Monitoring System (DMS) was one of the main parts of the project. It
automatically collects data, voltage and current graphs, Sequence of Events (SOE) information, and
protection relay operations during a station event. This data is critical for forensics analysis of the
events to determine root causes and confirm the system responded appropriately, as expected. The
vendor proposed a DMS built on proprietary, advanced disturbance monitoring and asset management
software. With support for IEC61850 protection relays and the capability to talk to legacy equipment
through the File Transfer Protocol, it interrogates relays around the network. From those relays it
gathers recorded files and analyses the incoming data to generate fault reports. It utilizes both Line 1
and Line 2 Protection IEDs for DFR (Digital Fault Recorder), DDR (Dynamic Disturbance
Recording), and SER (Sequence of Events Recorder) functions. The original Coned specification
required for each DFR, DDR, and SER to have cross triggering capabilities for the respective line of
protection via IEC 61850. This enables recording not only in the device involved in the event, but also
adjacent zones, even neighboring substations.

3. Unexpected Problems and Other Issues

Interoperability: The Parallel Path Logic approach ran into a problem that was ultimately traced to an
implementation decision by one of the IED Vendors. One of the IED types used (and the only type of
device that exhibited this problem) has an internal alarm called “GOOSE IED Absent” which comes
up if any subscribed IEDs are not present in the network.
Under normal operating conditions there should be no simulator devices connected to the network.
This causes the GOOSE IED Absent alarm to be continuously asserted in the relay. There is no
internal setting to suppress this alarm. To resolve this issue we decided to add a small device to
simulate a constant Test Bit. This device has to be disabled prior to performing regular system tests.

6
Breaker Interface Devices (BID): In our design the breaker interface device was intended to collect
breaker status, alarms, and other digital inputs, and perform automation operations such as monitoring
and supervisory controls. Additionally the breaker interface device performs protection operation
Trips as when triggered by the new protective relays. One of the problems we encountered was the
low current breaking capacity of the IED output relay contacts. This makes them unsuitable for direct
breaker tripping. Hence, at the late stage of the project we were forced to bring back the auxiliary
tripping relays eliminated earlier.

Cross Triggering: Cross Triggering turned out to be a much more complex process than originally
envisioned. As we started working in detail on the implementation it became apparent that the
additional functionality would come at great cost in terms of configuration complexity, system
performance, and long term testing and maintainability. It was decided instead to use analog type
triggers – Overcurrent and Under voltage.

IEEE 1588: We ran into a couple of issues when implementing this requirement. Many of the Relay
Protection IEDs used in the design did not support the IEEE 1588 protocol. To compensate, we had to
include conversion modules that connect to the substation LAN and generate IRIG-B outputs for use
by those IEDs. We also had to return an entire set of LAN switches because, even though the specs on
these devices clearly stated they support Precision Time Protocol (PTP), it turned out that they only
support PTP in a two-step Peer-to-Peer mode and we needed a one-step Peer-to-Peer implementation.
The LAN switches were successfully replaced with a model from a different manufacturer.

4. Lessons Learned

Specifications are critical:

System Specifications must accurately describe all aspects of functionality. However, when dealing
with new or unfamiliar technologies, it is extremely difficult to know what to specify, and what
priority to assign to various requirements. The natural tendency in this situation is to keep some
aspects of the Specifications purposely general and to trust that the Vendor has sufficient expertise to
make the right implementation decisions. This approach is fraught with potential risks and is not
recommended. It is much better to gain sufficient familiarity up front with the technology to
understand in great detail what is being specified.

Prototyping:
A requirement that paid significant dividends in minimizing the number of design and build errors was
that the equipment cabinet assemblies had to be prototyped and then evaluated by the utility personnel.
Although this had a significant schedule impact on the project, it resulted in the final cabinet
assemblies being built correctly. Had we not taken this approach the schedule impact to the project
would have been much greater and the repair of all the cabinet assemblies would have been very
costly in terms of labor. The lesson here is that when dealing with large numbers of components and
large complex projects, prototyping should always be included in the project plan.

Cooperation is key:
This project required a closer than usual level of cooperation between the Company and the Vendor. A
successful project outcome is highly dependent on the Customer and the Vendor seeing themselves as
part of a team sharing the same goal or objective. Open and honest communication is the best way to
avoid potential confrontation. When confrontation is unavoidable, it should be tightly isolated to the
single issue in question and quickly resolved. Every effort should be made to limit any potential
interference with the rest of the project.

Vendor provided tools are not always the best solution:

Although every IED manufacturer has a graphical user interface for configuration and programmable
logic, each tends to be quite different from one vendor to another in terms of format, symbols,
language, etc. We saw a need for a common graphical format for designing and creating logic

7
diagrams for each IED in the system. This format became a central tool during the project for
communicating our protection philosophy precisely to the Vendor and for disseminating this
information to the various different groups of people involved in the project, as well as for reference in
future troubleshooting.

5. Conclusions and Recommendations

Current state of IEC 61850 standard:

As part of the market maturation process, a great deal of progress has been made in implementing the
IEC 61850 standard so that it can, in fact, deliver the benefits promised. However, there is much work
that still needs to be done. Interoperability is still not guaranteed. Users must run interoperability tests
to be sure that the components selected for a system will perform as expected. Most importantly, there
is great need for vendor agnostic tools that allow users to use a system oriented, top-down design
approach irrespective of the mix of devices from different manufacturers. Users should not have to
switch back and forth between working on a system level SCD file and editing individual IED
configuration files using vendor proprietary software.

The importance of Prototyping:

As mentioned before, the recommended approach is to require the Vendor to build prototypes of each
type of cabinet assembly used in the system. However, we feel that prototyping should be applied
more generally when building large complex systems. Prototyping should be applied as a general
principle in terms of creating a small scale version for all critical system components such as
protection logic, HMI typical operator interface screens, network configuration, and so on. Prototyping
benefits the end Customer as well as the Vendor by catching problems early on and eliminating re-
design and rework late in the project cycle.

Factory Acceptance Testing:

The system described herein could not be tested in its ultimate configuration. However, there is great
value in making every effort to perform Factory Acceptance Testing with the system configured as
closely as possible to its real-world final configuration. This means that the end devices and any
sections of the system that are planned to be built and delivered in the future should be simulated to
the greatest possible extent.

8
BIBLIOGRAPHY

[1] Hubert Kirrmann. “Seamless Redundancy – Bumpless Ethernet Redundancy for Substations
with IEC 61850” (ABB Review Special Report)
[2] Klaus-Peter Brand, Wolfgang Wimmer “Modeling Interoperable Protection and Control devices
for Substation Automation According to IEC 61850”
[3] Dave Dolezilek “IEC 61850: What You Need to Know About Functionality and Practical
Implementation”. (Schweitzer Engineering Laboratories, Inc., Pullman, WA USA).
[4] Eric A. Udren, Paul T. Myrda “System-Wide Replacement Strategy for Substation Protection
and Automation” (Proceedings of the 39th Hawaii International Conferences of System Sciences
- 2006).
[5] Malcolm W. Stevens “An Implementation of an Optical Data Diode” (Defense Science and
Technology Organization DSTO-TR-0785)
[6] Austin Scott “Tactical Data Diodes in Industrial Automation and Control Systems” (The SANS
Insitute).