2019 IEEE 3rd Information Technology,Networking,Electronic and Automation Control Conference (ITNEC 2019)

Research on Clock Synchronization Mechanism of

Distributed Simulation System based on DDS
Wu Yongliang1 , Luo Peng2, Chen Xiaoping2, Xiong Lichuan1,2
1.Chengdu Aircraft Design & Research Institute of AVIC, Chengdu,China
2. College of Aeronautics and Astronautics,University of Electronic Science and Technology of China,Chengdu, China ;
[email protected], [email protected], [email protected], [email protected]

Abstract—Flight control system simulation test is a key In the distributed system of flight control system simulation
process in aircraft design stage. Distributed system test, the efficient and timely data interaction between each node
architecture is often used in flight control simulation test. is a key point of the system design. For distributed simulation
Clock synchronization has always been a research hotspot of test system, DDS is an excellent distributed message
distributed simulation system. Data distribution service (DDS) middleware [2]. It set up a system composed of release node and
is widely used in distributed simulation system for its good subscription node structure, providing a very efficient data
real-time performance, high efficiency and low coupling. transmission mechanism and rich and practical QOS
However DDS does not provide a excellent clock management functions. It is applied to resources field which
synchronization mechanism. Aiming at the demand of clock requiring high performance and predictability and effective use
of the key tasks.
synchronization of distributed simulation test system, an
improved PTP clock synchronization algorithm is proposed in However DDS lacks the implementation and management
this paper, and a virtual clock of distributed simulation system of time synchronization mechanism of each node. What’s more,
is constructed on the basis of the propulsion timing model. clock synchronization between nodes is more important
Experimental data show that the model can be applied to between distributed systems,. It needs to make sure that all the
simulation nodes stably, and the improved algorithm can also nodes in distributed simulation system of the time difference in
achieve clock synchronization between nodes of distributed a certain range. This paper studied on the basis of the DDS
system. clock synchronization mechanism of distributed systems.
Keywords—Data distribution services; Clock synchronization;
II. DATA DISTRIBUTION SERVICE
Virtual clock
DDS is a software design specification and standard
developed by OMG (object management organization) [3],
I. INTRODUCTION which can achieve publish/subscribe communication between
transport layer and application layer while meeting the real-
In the flight control system simulation test for near space time requirements. This specification standardizes the interface
UAV[1], in order to make the simulation structure more flexible and behavior of data publishing, transmission and subscribing
and the operation of each subsystem more efficient, the test in distributed real-time system. It can enhance the system
mostly adopts the distributed architecture. Each node has its dynamics, and improving communication efficiency and
own operating environment. There are four nodes in the present system flexibility.
system: flight control computer, AC flight package, ground
station and airborne system. The simulation structure for flight RTI DDS(Formerly known as NDDS[4]) is developed and
control simulation test is as Fig.1. implemented by Real-time Innovation Company, which was
the first commercial product to support the DDS specification.
)OLJKWFRQWURO $&
FRPSXWHU SDFNHWV DDS is designed for data publishing/subscription requirements
of real-time systems[5]. DDS supports multiple data distribution
patterns and data distribution strategies, and supports QoS
configuration in the form of XML. It is possible to change the
communication mode and performance of the system without
changing the source code of the nodes, thus reducing the
''69LUWXDO%86
coupling between nodes [6] .
0DLQFRQWURO
V\VWHP DDS data publish-subscribe model establishes the concept
of "global data space", as in Fig.2. When data is exchanged
*URXQG $LUERUQH between nodes, Publisher creates the DataWriter with the
VWDWLRQ V\VWHPV specified Topic and then writes the data to the global data
space through the DataWriter. The Subscriber creates the
Fig.1. Structure of flight control simulation system DataReader of the corresponding Topic, which retrieves data

978-1-5386-6243-4/19/$31.00 ©2019 IEEE 2224

orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap
 from the global data space. And Topic is the corresponding count runs at the set frequency and the virtual clock starts
relationship between publisher and subscriber. running. The functions listed in the table above are also
provided.
TABLE 1.INTERFACE TO THE VIRTUAL CLOCK
1RGHB$ 1RGHB%
&ODVV 9LUWXD&ORFN
* OREDO'DWD6SDFH ,QLWLDOL]H 
1RGHB& 1RGHB' *HW7LPH%HDW 
6HW)UHTXHQF\ 
,QWHUIDFH
&DOLEUDWH7LPH 
Fig.2.DDS communication model 5HVHW%HDW 

DDS also provide a request reply mode based on publish-

subscribe mode for different application scenarios. The IV. IMPROVED PTP CLOCK SYNCHRONIZATION ALGORITHM
request-reply pattern has two roles: Requester sends the request A. PTP clock synchronization protocol
message and waits for the reply message. Replier receives the PTP clock synchronization protocol [9] is a kind of clock
request message and responds with a reply message.
synchronization protocol applied to distributed measurement
Requesters and Repliers manage all interactions on behalf of
and control system through network communication, local
the application. Requester and Replier discover each other
computation and distributed objects. It’s also called ieee1588.
automatically based on the service name specified by the
application. When the application invokes the function, PTP protocol has two assumptions as follows:
Requester sends a message to the Replier informing the
receiving application. The application uses the Replier to 1) The network delay has symmetry. That means the
receive the request and send a reply message in turn. transmission network delay of sending and receiving between
the same two nodes is equal.
III. PROPULSION TIME MODEL 2) The sending processing time of each packet is ignored.
In the process of the correction of clock synchronization,
PTP protocol clock synchronization model implementation
the local clock is generally modified with the calibration results. process is as in Fig.3.
However, the change of node local time will also affect other
processes running in the system, and the periodic modification 0DVWHU 6ODYH

of the local clock will also make the system run abnormal. To 7P
avoid this situation, this paper constructs a virtual timing clock
referencing computer physical clock timing model. The clock
frequency of it even can be adjusted. 76
The virtual clock is implemented with the method of the
software by a separate higher-priority thread. The clock is 7P

maintained by each node of the distributed system according to 76

the internal synchronization mechanism of the simulation
system, isolating with physical environment of each node.
7P
Achieve the effect of software clock synchronization [6]. This
is called the propulsive clock model [8].
ܲ݇ǡ݊൅ͳ ൌ ܲ݇ǡ݊ ൅ ‫ ݇ܨ‬ሺܶ‫ݏ‬൅ͳ െ ܶ‫ ݏ‬ሻ  
7P
In the formula:
ܲ௞ǡ௡ is the calculation value of the propulsion timing
clock model.
k = 1, 2, 3,... , represents the node number of the Fig.3. PTP clock synchronization model
distributed system.
n =1,2,3... , represents the number of cycles. 1) The master clock sends out sync message and records the
exact time(Tm1) when the message leaves the master node;
‫ܨ‬௞ǡ௡ represents the clock frequency of the propulsion
timing clock model. 2) The master clock encapsulates the accurate sending time
ܶ௦ାଵ െ ܶ௦ represents the logical time interval. Tm1 into follow_up message and send it to the slave node;
In this model, logical time synchronization means that the 3) The slave node record the exact arrival time (Ts1) of
calculation values of each node are the same. If the calculation sync message from the master node;
values of each node at a certain time are all P, it can be
considered that they are at the same time of logical time T. 4) The slave node send delay_req message and record the
The implementation interface of the virtual clock is as in exact sending time(Ts2);
Tabale1. When the clock is initialized, a new thread with the 5) The master node records the exact arrival time of
highest priority is opened to run the clock. The virtual clock delay_req message to the master clock(Tm2);

2225
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap
 6) The master node sends delay_resp messages with 4) After receiving the reply form the slave node, the master
accurate timestamp information(Tm2)to the slave node; node calculates the network delay and send the calculated
delay to the slave node as the virtual time calibration command;
Offset of master and slave clocks can be calculated by Tm1,
Tm2, Ts1 and Ts2: 5) After receiving the calibration command from the master
node , the slave node calibrate it’s clock by making a
ሺ் ି் ሻିሺ் ି் ሻ
‫ ݐ݁ݏ݂݂݋‬ൌ  ೞభ ೘భ ೘మ ౩మ    backward adjustment(Δ T);
ଶ
After getting the offset from the node, the slave node can The virtual clock of slave clock get more accurate time
synchronize the time by modifying the local virtual clock. correction by the synchronization process .
B. Improved PTP protocol based on virtual clock ୘ ି୘ ି୘
‫ ݐ݁ݏ݂݂݋‬ൌ ೘మ ೘భ ೞభ   
ଶ
(1) Initial alignment time
The calibration result is the slave clock adjusting ‘ˆˆ•‡– ൅
The PTP protocol has its own clock for master and slave ௠ଵ forward of the virtual clock of this node. In this way, the
nodes, and the clock is always running. This paper has time of receiving message from the master node in the above
presented a propulsive timing model in which each node process can be understand as ‘ˆˆ•‡– ൅ ௠ଵ , which achieves the
maintains a virtual clock that can be invoked by command. clock synchronization in the virtual time domain.
Therefore, there is no clock offset in the improved clock
PTP protocol is a clock synchronization based on the
synchronization process. The master node wakes up the slave
assumption that the sending processing time is ignored. After
node's clock through the first packet. After receiving the
the improvement, the last calibration command (CalibrateCmd)
message from the clock, it is logically separated from the
only needs to tell the slave node to adjust how much time. It
physical environment where each node is located. The
does not need to be calculated into the synchronization model.
implementation process is shown in the Fig.4.
So, only two packages are used for computing the sending time
0DVWHU 6ODYH of data. In some way, it reduces the error causing by ignoring
processing time.
In the above synchronization process, as in Fig.5, only one
'HOD\

set of data is taken for clock synchronization time correction.

The big problem is the contingency of data because of the
fluctuation of the collected data. In order to ensure that the
76  uncertainty error has the least impact on the calibration effect
76 of clock synchronization, the slave clock returns multiple sets
of clock synchronization data when it make the reply to the
'HOD\

76 master node.

&ORFNVWDUW
7P FRPPDQGDQG
V\QFKURQRXV V\QFKURQRXV &DOLEUDWLRQ
UHSO\  UHSO\ Q FRPPDQG
V\QFUHTXHVW
76Q
7P 1RGHQ



1RGH
7PQ

&DOLEUDWH 1RGH

1RGH

Fig.4. Virtual clock synchronization initialization

7
Synchronization process: Fig.5. Multi-node clock initialization process
1) The master node send the start command (StartCmd) to When the master node has received the clock
the slave node for requesting clock synchronization, and synchronization data, it continues estimating of network delay
records the sending time Tm1; by Least Square Method:
2) After a certain network delay, the slave node receives the
startup command of the master node and starts the virtual clock › ൌ σ௡௜ୀଵሺʹ‫ ݐ݁ݏ݂݂݋‬െ ௠ଶ௜ ൅ ௠ଵ ൅ ௦ଵ௜ ሻଶ   
of the node; That is,
3) The slave node send multiple replies to the request of the σ೙ ் ିσ೙ ் ்
‫ ݐ݁ݏ݂݂݋‬ൌ  ೔సభ ೘మ೔ ೔సభ ೞభ೔ െ ೘భ  
master node, and add their own timestamp TS2 in the reply ଶ௡ ଶ
packet; i represents the ID number of each node of the distributed
system.

2226
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap
 At the end of period of clock synchronization initialization, process of simulating the master and slave clock initialization
the clock master node calculates the time correction of each of the system.
node according to the response packet of each slave node to
clock synchronization request. And the master node sends the Initial alignment time is complete. The virtual clock
running of each node is affected by the physical characteristics
clock modification amount of each node ο ൌ ‘ˆˆ•‡–݅ ൅ ݉ͳ of the node itself. So in order to improve the stability of clock
to each slave node in the later period of clock synchronization synchronization, it have to do periodic correction after
Initialization phase. initialization process.
When the Master Node makes a clock synchronization (2) Periodic time calibration
request, it creates a synchronous requester. Materializing
requests by using the data types of requests and replies, and The timing process in the previous section has been
initialize the requester with requester_params. Setting the completed. The soft clock of each slave node has been started
node ID of the master node of the synchronization system is 0, by the master node and preliminarily adjusted. In order to
and the startup control command to the macro StartCmd. And maintain the stability of clock synchronization, it is necessary
set the timestamp of the packet. The requester's send_request() to calibrate each clock periodically. Therefore, in each
function is called to send the synchronization request. Then it simulation cycle of the distributed simulation system, the
enters the ready receiving state of the reply packet. master clock also needs to publish its own clock information,
The packet structure of the master node publishing for and each slave node uses the received master node clock
requesting clock synchronization is as follows: information to calibrate the clock of the slave node, for
maintaining the stability of clock synchronization effect.
struct TimeSyncRequester{ In previous studies, the periodic frequency correction of the
unsigned long NodeID; clock mostly calculate by the way of one node i to collect all
unsigned long StartCmd; other nodes j= 1,2... i-1, i+1... , the clock difference of n, and
TimeStamp MasterSendTime; the local clock frequency calibration can be obtained after
}; average or weighted average processing[10]. However this
Each Slave Node creates a replier at startup and materialize method has a disadvantage, that is the local virtual clock
requests by using the data types of requests and replies . Then frequency is the physical environment and the local is
initializing the replier with replier_params. concerned, if the local clock frequency correction introduced
other nodes of the clock signal, it can hardly optain very good
When the slave node receive the master node clock correction effect. And each node of the physical environment
synchronization request, it parse the packet and judge whether to virtual clock clock frequency influence still exists, and it
the packet is master clock node (NodeID = 0), and ensure gets worse and worse as the virtual clock runs. According to
whether it is the start command for clock (StartCmd). If the the analysis, in the design process of clock frequency
clock synchronization request packet and request command is calibration module in the paper, it is done separately by each
from master clock node, each slave node invoke the Initialize ()
node . The node calibrates its virtual clock count and frequency
command of SoftClock to start a local virtual clock. At the
same time, creating the reply packet, and set the local node ID based on its clock run offset.
in the reply packet, take out the master node publishing time The calibration process is as in Fig.6. The master node
from the request packet and set it to the reply packet. Before publishes the virtual clock of this node to each slave node
sending the reply packet, add the time stamp of the slave node's through the write() interface of DDS. In order to save network
virtual timestamp to the packet. bandwidth, DDS transmission mode is configured to broadcast
int the paper.
Reply packet structure from slave clock:
&\FOH  &\FOH  &\FOH  &\FOH Q
struct TimeSyncRequester{ 0DVWHU 7 0 70 70 70 70Q
,QLWLDOL]DWLRQ

unsigned long NodeID;

Time calibra tion

Time calibra tion

Time calibra tion

TimeStamp
MasterSendTime;
TimeStamp SlaveSendTime;
}; 6ODYH 76 76 76 76 76 76Q

After the master node receives the reply from each node, Fig.6. All processes of clock synchronizes
the synchronization algorithm designed above is used to
In the Fig.6, the master node send the virtual clock
calculate the network delay of the synchronization process for
information of slave node (TMn) at the beginning of each
each packet. And the modified results of each slave node
synchronization cycle. And the slave node also records the
calculated are sent to each node through the write() function of
DDS's publish and subscribe mode. After each slave node local time 76Q after receiving a clock message from the master
receives the correction time. packet, it modifies the local node each cycle . Assuming that each time the machine virtual
virtual clock. So as to complete the slave clock startup and the clock messages sent from the master node to node of one-way
network latency is equal. This hypothesis has less influence on

2227
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap
 the error than the previous bidirectional network delay . It is and the maximum error is 3%. So the slave clock frequency is
closer to the actual situation. relatively stable.
The time information in figure 5 has the following 1.04
x 10
4

relationship: 1.03

1.02

ܶெ௜ െ ܶெሺ௜ିଵሻ ൌ ܶௌ௜ െ ܶௌሺ௜ିଵሻ ൅ ߠ௜ 㸪݅ ൌ ͳǡʹ ǥ ݊ ǥ  

Frequency of slave clock

1.01

Where,ߠ݅ is the clock offset of the slave node to the virtual 0.99

clock of the master node that takes place in the cycle ݅ .Then 0.98

each time the frequency correction is carried out, the 0.97

proportional relation of clock frequency is: 0.96

0 20 40 60 80 100 120 140 160 180
Simulation cycle

ிೖǡ೔ ்ೄ೔ ି்ೄሺ೔షభሻ

ൌ ˈ݅ ൌ ͳǡʹ ǥ ݊ ǥ   Fig.7. Frequency of the slave virtual clock
ிబǡ೔ ்ಾ೔ ି்ಾሺ೔షభሻ
In this model, the clock frequency of the master node is On the other hand, clock comparison before and after
always set to a constant value. That is, ‫Ͳܨ‬ǡ݅ ൌ  ‫ Ͳܨ‬. algorithm improvement is carried out on one node. The
simulation results are as in Fig.8.
So the new correction frequency is:
8
்ಾ೔ ି்ಾሺ೔షభሻ The improved

‫ܨ‬௞ǡ௜ାଵ ൌ ‫ܨ‬଴ ൈ ˈ݅ ൌ ͳǡʹ ǥ ݊ ǥ   7

No improvement

The clock difference between slave and master nodes(ms)

்ೄ೔ ି்ೄሺ೔షభሻ 6

Therefore, after the slave node receives the periodic clock 5

release of the master node, it can calculate the frequency to be 4

corrected by the above formula. By modifying the clock 3

frequency, the master and slave nodes can reach the same 1

speed. Then the offset in the above formula is calculated and 0

corrected. -1
0 100 200 300 400 500 600 700 800
Simulation time(s)

In each time calibration cycle, the offset of the period

Fig.8. Clock difference between slave-master nodes
length is still caused by the offset of the clock frequency, so the
mathematical relationship involved is as follows: The simulation results show that before the algorithm
improves, due to the periodic calibration of slave clock, the
‫ Ͳܨ‬ൈ ο‫ ݐ‬െ ‫݇ܨ‬ǡ݅ ൈ ο‫ ݐ‬ൌ  ߠ݅    average clock difference between master and slave clock is not
Meanwhile, the proportionality between ο‫ ݐ‬and the even large, but it's very volatile. After the algorithm is
previous clock frequency is: improved, the clock frequency of slave node is adjusted
periodically. In this way, slave nodes can adjust their clock
்ೄ೔ ି்ೄሺ೔షభሻ ்ಾ೔ ି்ಾሺ೔షభሻ
ο‫ ݐ‬ൌ  ൌ    speed according to the clock difference with the master node.
ிೖǡ೔ ிబ
Figure 8 shows, this improvement is very effective, clock
The clock offset from the slave node to the master node difference between the master and slave nodes is very stable.
during each clock synchronization period can be derived from
formula(9) and formula(10). Deploy the designed clock model and synchronization
algorithm to multiple nodes for verification. The results of time
‫݇ܨ‬ǡ݅
ߠ݅ ൌ ሺܶ‫ ݅ܯ‬െ ܶ‫ܯ‬ሺ݅െͳሻ ሻሺͳ െ ሻ  data collection are as in Fig.9.
‫Ͳܨ‬
‫Ͳܨ‬
ߠ݅ ൌ ሺܶܵ݅ െ ܶܵሺ݅െͳሻ ሻ ൈ ሺ െ ͳሻ   5
Node-1

‫݇ܨ‬ǡ݅ 4.5
Node-2
Node-3
Node-4

After the above analysis, after the slave node receiving the 4

periodic synchronization time packet from the master node, the

Clock difference with the master node(ms)

3.5

clock offset of it’s local virtual clock is calculated. The slave 2.5

node uses the calculated results to calibrate the local virtual 2

clock. The frequency calibration and offset calibration of 1.5

periodic calibration are completed. 0.5

V. THE SIMULATION VERIFICATION -0.5

0 5 10 15 20 25 30
Simulation time(min)

This part is the simulation and verification of virtual clock

model and synchronization algorithm. The master node clock Fig.9. Clock difference with the master node
frequency is stable to 10 KHz, which means for every 0.1ms, According to the simulation results, when the clock model
virtual clock count plus 1. Virtual clock frequency of slave and synchronization algorithm are deployed to each node, the
node calibrates itself according to the difference form the clock error of each node is within milliseconds and mainly
master clock each cycle. The clock frequency of the slave node within 3ms. It has good clock synchronization accuracy.
can be obtained as in Fig.7, sampling every second. As can be
seen from the Fig.7, most of the time the error is within 1.5% After the previous research and design, the internal time
domain of the simulation system was established on the basis

2228
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap
 of the propulsion timing model. The clock synchronization REFERENCES
algorithm is used to carry out initialization process and [1] Y.L Wu,Y.L Cai,H.X Baoyin,L.C Xiong,et al,”Design of redundant
periodic calibration of the clock of the master and slave nodes. fault-tolerant navigation system for near space UAV”.Acta Aeronautica
The calibration process also included frequency correction and et Astronautica Sinica.2016,37(S1):pp91-98.
clock offset calibration, so that each simulation node achieves [2] PardoCastellote,Gerardo.OMGDataDistributionService(DDS):Architect
good clock synchronization accuracy. In this way, even if the ural Overview. Real-Time Innovations, Inc. :REAL-TIME
INNOVATIONS INC SUNNYVALE CA, 2011, 1.
gap between each node and the local physical clock is larger
and larger, the clock synchronization of the entire simulation [3] OMG, Data distribution service for real-time systems
specification.Version 1.2. http://www.omg.org/spec/DDS/ Jan.2007.
time domain can be guaranteed to be within a certain accuracy.
[4] Pardo-Castellote G, Schneider S, Hamilton M. “NDDS: The real-time
publish-subscribe middleware”.Real-Time Innovations, Inc, 1999.
[5] Gerardo PC. “OMG Data-Distribution Service (DDS): Architectural
VI. CONCLUSIONS Update”. 2004 IEEE Military Communications Conference, 2004,
pp961-967.
In the distributed simulation test system, each functional [6] J Han,Y.S Wang,” Design and Implementation of Sensor Data
module runs on different terminals respectively. In order to Distribution System Based on DDS”.Ship Electronic Engineering,
solve the practical need of clock synchronization, the virtual 2018,Vol.38 No.10,pp130-133.
clock which promotes the timing model is adopted to avoid [7] N Sun,W Xiong,Y.Z Ding,”Study and Application of Clock
frequent modification of the local clock of the terminal. Then, Synchronization” , Computer Engineering and Applications | Comput
Eng Appl.2003.27,pp177-185.
on the basis of distributed simulation test system
[8] Y.X.Ling,L.J Fan,T.Wang,G.H Zhang,”ock Synchronization Algorithm
communication middleware DDS, the PTP clock Based on Parametric Difference for Distributed Node Clusters,” ACTA
synchronization algorithm is improved and implemented. ARMAMENTARII, V0l.34 No.1,pp.51-58,2013
Through the analysis of test data results, the virtual clock [9] IEEE Std 1588-2008.IEEE standard for a precision clock
model and synchronization algorithm in this paper have a high synchronization protocol for networked measurement and control
accuracy, which meets the requirements of the existing system systems. New York:Institute of Electrical and Electronic
Engineers ,2008.
for time synchronization accuracy. [10] Y.N Ren,B Zheng,H.L Guo,” Research and development of IEEE 1588
in distributed wireless network”, Information Technology.
2018.11,pp103-110.

2229
orized licensed use limited to: AMRITA VISHWA VIDYAPEETHAM AMRITA SCHOOL OF ENGINEERING. Downloaded on January 22,2024 at 09:38:27 UTC from IEEE Xplore. Restrictions ap