IMPROVING MANAGEMENT OF ELECTRIC DISTRIBUTION SYSTEM AT

PACIFIC GAS AND ELECTRIC COMPANY THROUGH SYSTEM

INTEGRATION UPGRADES
Gary Cassilagio, PG&E
Scott Neumann, UISOL
Barney Speckman, Nexant
Siri Varadan, Nexant
Ali Vojdani, UISOL

Abstract
In 2003 Pacific Gas and Electric Company (PG&E) upgraded and integrated the computer and
communications systems it uses to operate its electric distribution system. These system
enhancements will enable PG&E to better manage its electricity distribution system during
severe outage conditions such as storms. The project was completed prior to the winter storm
season in California. This paper presents the motivation for the project, the technical
architecture of the integration upgrades, and the lessons learned.
The project developed a multi-year roadmap for implementing the upgrades as the first step. The
project team then began developing software for integrating PG&E's existing outage information
system with its field automation system. Now that this integration is complete, power outage
orders can be sent to field personnel automatically using a wireless communications network and
work status or damage information can be sent back to the distribution operator over the same
wireless network. By enabling PG&E to better manage the troubleshooting and restoration
process-particularly during severe storm conditions-customers can find out when PG&E repair
crews will be able to troubleshoot and repair an outage and turn the power back on.
PG&E also integrated the supervisory control and data acquisition (SCADA) systems used to
manage the distribution network with the outage information system. This integration will allow
data received by the SCADA systems to be automatically communicated to the outage
information system at the operator's direction, thereby increasing the accuracy and timeliness of
the information and streamlining power restoration efforts.
The project also included developing a new distribution operator dashboard and integrated
logging information system using state-of-the-art portal technology. This web-based intranet
system will bring together many aspects of the workflow in a way that will enable distribution
operators to more efficiently and effectively perform their duties.
The project tied together a number of existing independent systems using cutting-edge enterprise
architecture integration (EAI) software [1].

Introduction
In 2002, PG&E initiated an Emergency Management Improvement Process to identify ways to
improve emergency response performance. During the pendency of this process, northern
California experienced severe storms in December 2002 that caused a large number of outages in
the PG&E distribution network, leaving many customers without electricity for several hours.
These outage events confirmed the need for improving PG&E’s outage management business
process. In response to this need, PG&E undertook an Electric Operations Technology
Improvement Project (EOTI).
The goal of the EOTI project was to improve distribution operations at PG&E particularly during
storm conditions. A critical consideration was to implement certain critical improvements by
November of 2003, prior to the next storm season.

Specifically, the EOTI project objectives were to:

– Improve business processes through technology
– Improve the efficiency and effectiveness of the Distribution Operator (DO)
– Improve timeliness and accuracy of outage information provided to customers
– Reduce outage restoration time
– Capture data required to measure and improve performance
This paper presents the solution approach developed by PG&E to devise a technology
improvement roadmap, implementation of the roadmap, a description of the integration project,
the results obtained thus far, and the next steps.

The Solution Approach

To achieve the stated objectives, PG&E along with a team of external consultant, first devised a
five steps approach depicted in Figure 1, to develop a roadmap for implementation of EOTI. The
five steps approach included:
1. As-Is Assessment – This step included conducting an assessment of the currently used
business processes and their associated supporting systems (software tools and
technologies). The step included Field visits to four distribution control centers, each in a
different geographical location. During each of these visits, interviews with field
personnel were conducted along with an on-site inspection of various processes and tools
used at each control center.

2. To-Be State Description – In this step, the “ideal” future (or “To Be”) states were
identified and described after discussions with PG&E Operations personnel. Having
understood the broader set of operating constraints and existing technologies used, the
“ideal” state was modified to be a “practically achievable” To Be state.

3. Gap Analysis – Having identified the As Is or current state and the To Be or future state,
the third step identified the gaps in business processes and technologies that needed to be
addressed.

4. Integration Approach –This step developed a high level technical architecture for how to
address the gaps identified in step 3 through upgrading system integration at PG&E.
 5. Business Plan development: This step developed a manageable plan for implementing
the needed integration upgrades.

Figure 1: Methodology for developing a technology roadmap

Systems Impacted

There are several key systems that are used for PG&E outage processes, as well as a number of
supporting systems. The central system to outage processes is the Outage Information System
(OIS), which is based on ABB CADOPS and used by operators in seventeen control rooms,
using a distributed architecture with control center servers that coordinate with a central server.
The Outage Management Tool (OMT) developed by PG&E permits outage information to be
viewed by anyone on the PG&E corporate network. OMT periodically obtained snapshots from
OIS of the current outages.

The PG&E Field Automation System (FAS) is an implementation of the mobile solution offered
by MDSI. Prior to this project there was no integration between the OIS and FAS systems. The
FAS focus was primarily for processing of service orders. In the event of an outage, trouble
tickets received resulted in the creation of service orders, which could be dispatched to a
troubleman. However, a key point is that no correlation of a service order to an outage and
predicted outage device was available. The lack of direct integration between OIS and FAS
forced heavy reliance on voice communications between distribution operators and troublemen.
Where the OIS was used for the correlation of outages, the capture of information to meet outage
logging requirements was performed using the PG&E developed Distribution Operation Logging
Information Program (DOLIP). This was a PC-based application which saved entered data into a
separate database for each control center. During major storms, time constraints resulted in
logging being done sometimes days after the fact.

PG&E uses the DC Systems SCADA for monitoring of the distribution network. There was no
integration with OIS or other systems. Due to the lack of integration, much information needed
to be entered redundantly into more than one system. For example, a device operation needed to
be acknowledged in the SCADA, entered into the OIS and then logged into DOLIP.

There are a variety of supporting systems related to distribution operations. These include
systems for managing distribution diagrams, interactive voice response and call handling. The
call handling systems provided trouble tickets for outage correlation. Information from the OIS
was also used to indicate expected outage durations to customers who called.

The “As Is” State

The As Is assessment was facilitated with field visits, on-site inspection of business processes
and supporting systems, and meetings with key Operations personnel. The focus of the
assessment was kept on the outage management process and the core systems involved in the
process: OIS, OMT, SCADA and DOLIP. It became apparent that the Distribution Operator
(DO), who plays the central role in the outage management process, used systems that were not
well integrated and consequently required duplicate manual entry of data, making the job
particularly difficult to manage during storm condition.

More specifically, the As Is state was characterized by:

1. Manual determination of outage location, cause, and restoration plan
2. Heavy reliance on verbal communications
3. Manual assignment of field resources
4. Loosely integrated (or not integrated) systems requiring multiple entry of the same data
one or more times
5. Multiple screens presenting data in various formats
6. Multiple organizations addressing outage management depending on the severity level of
the outage

In effect the DO played the role of the “integrator” by performing tasks that included:
1. Verify the outage
2. Determine the approximate location of the Outage using one or more tools including
SCADA and the OIS inference engine.
3. Dispatch field crew to the outage location.
4. Stay in constant touch with the field crew to provide instructions and information from
trouble calls etc.
 5. Refer to off-line feeder calculations and generate a suitable switching plan once the cause
of the outage was determined.
6. Update OIS information so that the customers calling in would have access to the most
recent information on outages.
7. Provide switching instructions and ensure power is restored when repairs are all done
satisfactorily.
8. Log all appropriate information and close out outage.

Figure 2 summarizes the As Is state for the resolution of an unplanned outage. The relative
numbering on the flows in Figure 2 show the order of actions beginning with the initiation of the
outage with a telephone call. The dashed lines in this figure show all the processes involving the
DO as part of this process. The information flows are complex, all verbal and required the DO to
interface with multiple systems that were not integrated, thus leading to multiple data entry for
logging purposes, repetition of tasks and leaving much real-time analysis for the DO to perform
instead of presenting ready-made solutions, burdening the DO who was already hard-pressed for
time during emergency situations. Verbal communications could lead to inaccuracies in
instructions and communications with the customers.

C ED ESA C us tom er/

Trans f orm er
relations hips (batc h) Serv ic e Serv ic e
R eques t R eports
(5) (9)
C orD aptix F AS

View of MDT
C C MS2
Adv antex s c reen
As s ignm ent (~6)
View of
C all EAI (1) of T-Man
H azard C alls MD T Orders (10)
(8)
(2)
C alls (4) ETA
no delay
non-hazard
If H azard
10m in D elay
H andof f
SO
R es toration
Switc hing
(18)
C all EAI (1)

ETOR T-Man
IVR C aus e Verbal
(16) de-as s ign
OIS Predic tions
R eports (19)
ETOR /C hange (17) (3) Verbal
Ons ite
As s ignm ent
C us tom er (11) ETOR (15)
View of (~7)
Outage
C allEAI(1)
View of (13)
Predic tions
Verif y (~6)
Outage Mark as
arriv ed
Sy nc hronize (~12): (12)
arriv als c aus e ETOR
TFC SC AD A
Predic tions View of Alarm
(26) (26)
R es toration
Mark as Switc hing
arriv ed (18)
R es toration
(12) DO Switc hing
Ex trac t View of OMT (18)
(26)
View of OMT
D e-as s ign (27)
(19) Get Switc h/
OMT
Outage
N um ber (14)
ETA

DOLIP

Figure 2: Unplanned outage process before integration upgrades.

The “To Be” State, Gap Analysis, and Integration Approach

Working with PG&E personnel who presented their “wish” list of improvements they “like to
have” and they “must have” improvements, the To Be or future state was developed. Noting that
the DO is central to control center Operations, the focus remained on making the DO more
efficient in conducting Operations. We focused on processes and systems that enable these
processes as part of this step. Six main areas of improvement emerged:
1. Customer Information and Interaction
2. Outage Information and Control for Operations
3. Outage Information for Management
4. Work Flow Management Systems
5. Historical Logging
6. Maps and Mapping

In case of each of the above six area, a high level summary of the To Be state is characterized by
the following:
1. Need for quick and efficient means to verify outages.
2. Integration of multiple systems and uniform presentation of data from multiple systems
3. A uniform presentation of data in the form of meaningful information, all in one
location/screen.
4. Automation of trivial and routine tasks involving data entry and logging
5. Automated flow of data in electronic format to/from DO to the Field personnel, and less
reliance on verbal communications.
6. Need for timely update of maps and related circuit calculations to provide accurate
representation of field conditions.

Based on the requirements listed above, a To Be architecture that addresses all the critical issues
identified in the As Is state was developed, as shown in Figure 3. Using EAI (Enterprise
Application Integration) middleware, the Figure shows the facilitation of data flows among all
the different applications, critical to Operations.

The proposed To Be state aims to:

1. Reduce/Eliminate multiple data entry
2. Consolidate Data and Present it through a Single User Interface – Using Portal
Technology
3. Improve Outage Communication and Restoration Process
4. Improve Distribution Operator Efficiency and consequently improve the bottom-line for
Distribution Operations – to improve SAIDI statistics.
Figure 3: Technology “To Be” state

Development of Roadmap

Having conducted the As Is assessment, the To Be state determination, Gap Analysis, and
identified integration approach, this step of the process included the development of a roadmap
that would serve as a practical and logical guide for the implementation activities. Tasks were
identified based on groupings and appropriately prioritized. Figure 4 shows the prioritized
roadmap along with an implementation schedule.

The following items form the core of the roadmap. Please refer to Figure 4 for implementation
schedule and timing.
1. Emergency Management Systems Improvements/Integration Phase I. The initial focus of
the plan was on implementing enhancements aimed at improving systems supporting
PG&E’s emergency management at all Control Centers prior to the 2003 storm season.
These activities included integration of several systems with the OIS and the
establishment of a Distribution Operator’s Desktop (DOD) that would allow the DO’s
access to most of the core operating tools via a common interface.
2. OIS Electronic Map Granularity Improvements. During the first year of the plan, four
multi-year programs were scheduled to begin. The first of these is the OIS Electronic
Map Granularity Improvements, which are scheduled to begin with one Control Center
and then carried on to all remaining Control Centers.
3. SCADA review and Upgrades effort. The second multi-year program is the SCADA
review and Upgrades effort. It is envisioned that over a three year time frame of this plan
a detailed review of SCADA deployment in each jurisdiction would be completed.
 4. Outage Metering Review and Upgrades. This program is targeted at evaluating the
options available to PG&E to directly detect outages instead of relying on customer calls
to detect outages and pinpoint their locations.
5. Operations Database Population. This program involves setting up databases for all
jurisdictions to deal with operations transactions for purposes of logging, presentation
and integration.
6. Emergency Management Improvements Phase II. This program would include further
integration of existing systems.
7. Advanced Applications and Productivity Improvements. This group of activities would
include a number of processes that are currently paper based. For example, the clearance
processing and switch log process.
8. Distribution Control Center Metrics and Measurement. This program aims to create key
performance indicators or metrics that can help benchmark standard process. The idea is
to be able to perform periodic tracking and for comparison purposes across different
electric utilities.
9. Work Flow Management. This activity involves the implementation of a workflow
management tool that would support management processes involving distribution
control centers.

Figure 4: Development Roadmap

Roadmap recommendations for Year 1 activities involving technology implementation included

the following items:
1. Consolidation of Control Center Applications using a Distribution Operators Dashboard
(DOD) – To present results of various applications in one central location (one screen)
2. One Way RT-SCADA to OIS Integration. To enable automatic (and DO assisted) outage
verification.
3. Improved Logging (ILIS – Integrated Logging Information System). To enhance logging
functionality to reduce multiple data entry efforts.
 4. OIS-FAS Integration – To minimize verbal communications and relieve the DO from
routine data entry tasks created as a result of being central to the outage management
process.

Implementation of the Roadmap

As stated earlier, a critical consideration in the EOTI project was the need to implement the
critical First Year improvements by November of 2003, prior to the next storm season. This
consideration posed an extremely tight “time box” for the implementation of the First Year
improvements. To that end, a relatively large project team was quickly assembled to take on the
implementation. The project team included a core project management team organized in a
Program Office that closely planned and coordinated the work of a large number of
implementation teams working in parallel on different aspect of the implementation including:

• Functional specifications
• Technical specifications and design
• Software development and enhancements
• Integration, Functional, and Load Testing
• Training
• Implementation – Go Live
Considering the team-members working on a part time basis, close to one hundred people were
involved in the implementation.

The Integration Project

The OIS-FAS integration objectives were to improve the handling of unplanned outages through
direct integration between OIS and FAS. This integration provided a two way data flow, where
DOs could assign outage work to troublemen, and troublemen could enter estimated time of
arrival (ETA), estimated time of repair (ETOR) and completion information electronically on
their mobile data terminals (MDT). The electronic capture of this information would be used to
automate the update of outage statuses and associated logging.
When an outage is inferred by OIS, the DO can decide to assign a specific troubleman. When
this is done, an outage order is automatically constructed and sent to the MDT of the troubleman
using FAS. The outage order includes information which can aid the troubleman in identifying
the actual location of the outage, including:
• The level of the outage, which could be customer, transformer, device or substation
• The circuit and outage device or transformer, as predicted by OIS
• Cause codes which were taken from call, which might indicate hazards such as wire
downs
 • Number of customers affected and number of hazards reported
• The street address of the device
• Up to six trouble calls that were correlated to the outage, where the calls selected are
prioritized by the information provided, such as hazard codes and comments
Figure 5 shows one of the screens that are used on the MDT for the handling of outage orders.

Figure 5: MDT Screen Layout

When the troubleman goes on enroute to an outage, he enters an ETA, which is returned by the
interface to automatically update the outage status in OIS and OMT. The troubleman can revise
the ETA as necessary. When the troubleman arrives onsite, this is also automatically reflected in
the outage status through the interface.
Aside from the information provided by the outage order, the troubleman can perform a variety
of host inquiries to obtain other information on an ad-hoc basis which would assist in finding the
location of an outage, such as all customers affected by an outage with address and phone
numbers or outage history.
Once the troubleman has found the location of an outage, he can verify it on the MDT, with the
status then being updated automatically in OIS and OMT. When verifying the outage, the
troubleman reports the estimated time of restoration or estimated time to repair. This also
provides the ability to identify whether or not repair crews are needed.
The MDT has several screens which can be used to capture information related to the outage.
The information that can be captured includes:
• Damaged equipment, identifying the types and quantities of material that would be
needed by crews for repairs
• Materials used, identify the types and quantities of materials actually used for repairs
• Switching operations performed
• Actual cause
• Repairs required with identification of required crew types
• Inoperable equipment
• Comments

All information collected through the interface was logged in a central Operations Database. This
information is then readily available for logging, reporting and calculation of outage statistics
and performance metrics. Figure 6 shows the revised process, where the addition automated
flows eliminate many of the verbal flows that were present in Figure 2.

C ED ESA C u s t o m e r/
T ra n s f o rm e r S e r v ic e
r e la t io n s h ip s ( b a t c h ) S e r v ic e R e p o rt s
R equest (9 )
(5 )
C o rD a p t ix FAS E TA
E TO R

V ie w o f MD T
C C MS2
A d v a n t e x s c re e n
A s s ig n m e n t (~ 6 )
V ie w o f
C a ll E A I ( 1 ) o f T -M a n
H a z a r d C a lls (8 ) M D T O rd e rs (1 0 )
(2 ) E TA
n o d e la y C a l ls ( 4 ) E TO R E TA
n o n -h a za rd E TO R
I f H a za rd
1 0 m i n D e la y
H andof f
SO
R e s t o r a t io n
A s s ig n o f
S w it c h i n g
TM a n
(1 8 )
C a ll E A I ( 1 )
C o m m u n ic a t i o n
t o T -M a n T -M a n
IVR ( S w it c h L o g s )
O IS P r e d ic t io n s
E T O R / C h a n g e (1 7 ) (3 )
C us tom er
V ie w o f S w it c h
C a ll E A I ( 1 ) O utage S tatus
V ie w o f
(1 3 )
P r e d ic t io n s
V e r if y (~ 6 )
O utage

S y n c h ro n ize (~ 1 2 ):
TF C a r r iv a l s c a u s e E T O R SCAD A
P r e d ic t io n s V ie w o f A l a r m
(2 6 ) (2 6 )
R e s t o r a t io n
S w i t c h in g
(1 8 )
DO R e s t o r a t io n
E x t ra c t V ie w o f O M T S w it c h i n g
(2 6 ) (1 8 )

O MT
C o m p le t e / C l o s e

O IS N um ber
I n ia li z in g D O L I P
R e p o r t in g D O LIP

Figure 6: Unplanned outage process after integration

Figure 7 describes the information flows that were implemented to integrate OIS, OMT, FAS
and the DO Dashboard, automating outage order creation, assignment and status updates.
 OIS – Outage Information System
OMT – Outage Management Tool
ODT – Outage Dispatch Tool
FAS – Field Automation System
DOD – DO Dashboard

Figure 7: Outage Order creation and assignment

PG&E uses the SeeBeyond product for implementation as an Enterprise Application

Architecture (EAI). However, not all products could natively integrate with EAI and required
the development of adapters which leveraged the integration capabilities of their products. In the
case of ABB CADOPS, interfaces were implemented using Oracle triggers and stored
procedures, as well as Talarian messages. The RT SCADA systems also utilized Talarian for
integration. In the case of FAS, the MDSI Adventex Host Interface is based upon MQ Series,
which required adapters to be implemented to EAI. It was very important, especially in the case
of the modifications to FAS, to get it right the first time.

The DOLIP logging program was functionally reimplemented and improved using web
technology, where the new application was called ‘ILIS’. Aside from the benefits of a web user
interface, ILIS used the Operations Database for storage of logs, where much information related
to an outage is populated automatically as a consequence of the OIS-FAS integration. This
resulted in the elimination of much of the redundant data entry as previously needed using
DOLIP, which improved both efficiency and data quality, where now logging was being
completed concurrent with the closure of the outage instead of after the fact.

Another key part of the project was the implementation of a Distribution Operator Dashboard
shown in Figure 8. Using portal technology, the dashboard provided the means to integrate a
wide variety of information and functionality needed by distribution operators in a manner which
provided for more efficient access by the user. Examples of information integrated by the DO
Dashboard includes:
• OMT generated system overview
 • Outage summary
• Outage statuses
• Outage details
• ILIS (logging application)
• Switch logs
• On call list
• Contact list
• Fire index
• Weather
• Diagrams
• Reference library

Figure 8: The Distribution Operator Dashboard Home page

Field Deployment
Despite the extremely tight schedule, the schedule provided for a pilot of critical software at
PG&E’s Oakland Distribution Operations Center to prove out the software before a system wide
roll out was under taken. This pilot deployment was used to validate the accuracy of the design
and development process and proved to be a valuable part of the project as several minor but
important changes resulted from the pilot deployment.
Following the pilot, software was rolled out system wide to the following 16 control centers. The
enhancements were rolled out in two groupings. The first group which included functions
necessary to automate the dispatch of filed personnel was rolled out over a 3 month period. The
Distribution Operator Desktop (DOD and ILIS) group was rolled out over an 8 month period.

During the roll out of the of two software groups, personnel at 17 distribution operations control
centers and over 1200 field personnel were trained to use the new software and business
processes. This significant training effort could only have been possible by utilizing a dedicated
core of trainers working full at a dedicated training facility to implement the training for the
distribution operators personnel and a roving dedicated training team to train the field personnel
on how to use computer equipment in their filed vehicles. In both cases training was tightly
scheduled to be performed “just in time” prior to their Go Live date.

In addition to the extensive training effort required, an onsite support team was involved with the
roll outs and remained on site for approximately a week to help with the initial use of these two
groups of changes to help the new users with what amounted to significant changes to the way
they performed tasks critical to their success.

PG&E chose to use experienced operating personnel to perform the training and onsite support.
These same individuals were involved with the functional and performance testing of the system
and prior to that were involved with the function design of the system.

Results
In all respects, PG&E considers the project a success. In establishing the project it was very
critical to roll out new field dispatch functions to improve performance during the 2003/2004
storm season and to roll out the improved logging and distribution operator coordination
functions as soon as possible. It was well known that the project schedule was extremely
aggressive from the start allowing only 7 months to develop and test all software and test the
software in an integrated environment with several other tightly coupled customer and operations
computer systems. Despite the tight schedule the filed dispatch functions were ready and roll out
began on schedule on November 1, 2003. The logging a distribution operator coordination group
was ready for roll out in late December 2003.

The changes that were implemented in this project allowed for a significant increase in the
effectiveness of the processes used manage outage related work especially during storm
conditions. Field personnel are dispatched, and managed without the need for verbal
communication via phone or radio. Field personnel can communicate ETA, ETOR, conditions
found on site, equipment needed to make repairs and when and how many customers have been
restored all with verbal communication. As a result, the customers get better information about
the outage that is affecting them, the repair crews can be better prepared when they arrive on site
to make repairs and outages are shorter than they otherwise would be. In addition, the DOD and
ILIS software which are integrated with the other operating functions, make it possible for the
distribution operators to stay abreast of logging functions during severe storm conditions
improving the accuracy of storm and outage information.
The new system also provides for improved monitoring of the outage management process since
it captures data that describes the steps and timing followed for restoring every outage. This date
is invaluable in measuring and improving the outage management process.

Lessons learned
Looking back on the project there are several lessons learned that were critical to the project’s
success as described below:
1. The project’s scope was managed very aggressively through a formal process that
evaluated the impact of every change including the risks to schedule. Very few changes
were approved during the course of the project.
2. The project was managed using an experienced dedicated project team that was able to
focus solely on the project. The project team used formal project management processes
and procedures to keep this time boxed project on schedule.
3. Detailed functional and design specifications were used to facilitate communication with
the client and among the 10 groups involved with design, development, testing and roll
out.
4. A dedicated team of operations personnel were involved with the entire process from
functional design to testing, training, roll out and field support. With out this team, the
project would have not been able to deliver on schedule.
5. Having a change management program is critical to the success of a project of this nature.

Next Steps
PG&E is considering a number of next steps in their processing of further improving their
distribution operations and outage management business processes including:
1. Using the data captured with the new systems, develop more comprehensive measures
for tracking performance of these processes.
2. Develop ways to use the materials list that is provided by filed personnel in a more
comprehensive and automated manner.
3. In the mid term, integrate these applications with the company’s business processes in
their SAP system and roll out the field functions to other file crews.
4. In the longer term, further optimize these business processes and the integration of
systems in the PG&E distribution operations centers.

Acknowledgments
The authors wish to acknowledge and thank the members of the PG&E project teams associated
with implementation for their contribution of ideas and to the success of this project.

References
[1] Vojdani, Ali, “Tools for real-time business integration and collaboration”, invited paper, IEEE PES
Vol. 18, No. 2, Special issue on Tools for Managing Restructured Energy Systems, pp 555-562, May
2003.