Developing a Private MetaAPI Python Lib for MetaTrader Trading

HangukQuant1, 2 *

October 19, 2022

Get After It!

David Goggins, probably

1 2

Abstract

In the previous work [1] we shared Python code for building a barebone order executor that sets up
an execution window, sets up passive limit orders at the beginning of the window and forces unfilled
trades at the end of the window. We discussed how this might help to reduce transaction costs. The
underlying API calls were to the MetaAPI endpoints, which in turn routes our packets to a MetaTrader
terminal. However in that code - our code was sorrowfully lacking in both structure and functionality
· · · we were only able to submit a limited number of order instructions and idle when we hit the rate
limit. The code also fails when we load it with any reasonable workload (of ∼ 60 orders).

Using the asynchronous credit semaphore [2] we built previously, we develop a more sophisticated
MetaAPI service package that allows us to more efficiently handle rate requests and perform order
execution on a thousand trades (& probably alot more) without issue. The focus of this paper is on de-
veloping that MetaAPI service package (on top of the official MetaAPI SDK) - and our next publication
will work on the order executor using the MetaAPI service we built here.

1 *1: hangukquant@gmail.com, hangukquant.substack.com

2 *2: DISCLAIMER: the contents of this work are not intended as investment, legal, tax or any other advice, and is for
informational purposes only. It is illegal to make unauthorized copies, forward to an unauthorized user or to post this article
electronically without express written consent by HangukQuant.

1
Contents

1 Introduction 2

2 Desired Behavior and Code Principles 3

2.1 MetaAPI Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Implementing Synchronizers 10

4 Implementing Broker Manager 15

5 Implementing Data Manager 30

5.1 A Bonus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

1 Introduction

As before, the same caveat applies - this is a result of weekly musings - there are bound to be mistakes.
Please email all errors/issues to hangukquant@gmail.com or tweet me at @HangukQuant. There is absolutely
no warranty or guarantee implied with this product. Use at your own risk. I provide no guarantee that it
will be functional, destructive or constructive in any sense of the word. Use at your own risk. Trading is a
risky operation.

Here we aim to be as succinct as possible - we recognise that the content provided herein is of a rather
niche subject, even for my readers. Some who already trade using a well developed out-of-the-box API that
works smoothly with the brokerage might find the content somewhat inapplicable. However, the reason I am
sharing this work is that there will (i) inevitably also be many traders who find this extremely useful and
(ii) there are some excellent practices in software development we uncover here. Whether you trade using a
MetaTrader terminal or not - I encourage you to give this paper the whole read. As much as possible, the
library we develop will not have dependency on the underlying brokerage, at least in terms of the interactions
between software components within our trading system.

We aim to wrap up these domain-specific discussions within the next week or the one after - and the
entire series of papers in the past couple of months will almost wrap up our quant toolkit development for the
time-being. We hope to go into discussions on market research, giving quant engineering/implementation
a break so that readers can catch up with the somewhat massive code/contents we have been putting out.
Although market research will naturally have less code, there will be more interesting insights and ideas for

2
alpha trading. It will also be more inclusive and useful to the broad rather than the narrow.

2 Desired Behavior and Code Principles

Sometimes, code implementation logic can be done in multiple different places/classes, all with ‘correct’ logic
and functionality. However, they are not equivalent choices. When unsure of where-to-code-what, it is likely
you do not have the correct abstraction in the first place. Each piece of software within our code should
have a well defined task. For example, numpy does mathematical computations. That numpy can be used
to for linear regressions does not mean we should create a linreg function in our numpy package.

Another question you should have before designing a software package should be the desired behavior,
in the most optimal of terms. Many developers think of the code first, and see what behavior they can
achieve. It should be the other way round. Start developing the anything is possible mentality. Any
‘optimal’ behavior you dream of can likely be coded. Think of what you want your software piece to achieve,
and then start with the code. In our case, we want to develop a MetaAPI brokerage service. But actually,
what we want from the highest abstraction is just a brokerage service. That we are using MetaAPI is a
state of affairs unimportant to our trading system. So what should a brokerage service achieve? It should
contain the functionalities to analyse current positions, open orders, open new trades, get quotes et cetera.
In addition, it should be error-tolerant, efficient, and a bunch of other nice things. We want our brokerage
service to, well, not crash every time there is an issue with our requests. The official MetaAPI Python SDK
https://github.com/agiliumtrade-ai/metaapi-python-sdk gives us the basic needs for interacting with our
broker. This is already very useful! It abstracts away the details of having to construct data packets from
our request parameters and handling socket communications from the MetaAPI server and back. We do not
have to learn the whole stack, and network programming is a big headache. We are given nice, asynchronous
native Python functions to work with. It does not however, solve alot of the problems specific to working
with the MetaTrader terminal. I am not quite experienced with the quirks of MetaTrader, but there seems
to be a subscribe-request-retrieve from memory cache kinda thing with MetaTrader. That means when first
seeking for some bid-ask data for AMD, if our MetaTrader does not have it on hand, it will first throw a
specificed symbol tick not found error and then cause our application to crash. Well, all we had to do was just
request for it again, and this time it would be available! This is not a MetaAPI issue - it is a MetaTrader
quirk and our MetaAPI is passing on the error baton back to us. But our trading system - once again
- should not have to worry about these softwares ‘caught lackin’. There are also MetaAPI issues - when
submitting many requests, the MetaAPI server complains with a 429 TooManyRequestsError - and our code
will crash again. This could also have been solved by simply pacing our data submit/requests, a detail our
system should not be concerned with. If our software components could talk, they would say this:

3
 1. Trading System: make me an order for 25 contracts of AAPL at bid. Tell me if there are issues with
the account/order such as market being closed, or not enough money to transact. But don’t tell me
about your network issues or stuff like that. I am not interested. Just tell me what the fill is and the
final position.

2. (MetaAPI) Brokerage Service: give me some orders to make, and I will make the right calls for you.
If you spam my service, that’s fine - I will just pace them out so we don’t crash. If the calls throw an
error, I will handle them for you. If the error just needs me to reattempt, sure I’ll do that. If the issue
is abit bigger than that - like you having not enough money or trying to trade on invalid configurations,
I will alert you. I will also try to make your requests as efficient and cheap as possible.

3. MetaAPI (Official) Python SDK: I know communication with our servers are complicated, and web-
sockets are confusing. I give you some functionalities you can call, and we will transform your request
parameters into bytes and send them to our server using unblocking sockets. We will sort the responses
out and get back to you. But if the server response is funky - that’s all yours. Don’t spam me though,
or I will stop working.

It is fairly clear that each of these components have different responsibilities. We are provided with the
official SDK - here we will work on the brokerage service. Let’s dive right in.

First, our file directories.

metaapi_service
synchronizers
synchronization_manager.py
utils
exceptions.py
metaapi_utilities.py
config.json
metaapi_broker_manager.py
metaapi_data_manager.py
metaapi_master.py

2.1 MetaAPI Service

We will be using the official package from MetaAPI, installed with

pip install metaapi-cloud-sdk

4
 Here is there documentation, https://metaapi.cloud/docs/client/, and code examples
https://github.com/agiliumtrade-ai/metaapi-python-sdk.

The first thing we want to do is to create a master service file, which allows us to obtain access to the dif-
ferent sub-services. This is the same pattern we observed in the database service code in our previous work [3].
In general, there are 4 different connection types exposed to us via the metaapi-cloud-sdk, namely the (1) RPC
connection, (2) Stream connection, (3) Synchronizers and (4) REST connections. These connections have
different properties in terms of the network latency. Additionally, the MetaAPI server follows a credit system,
in which different requests and data packets cost different ‘credits’ and each application/account is granted
3
x credits per y time. Here is their rate limitation specs https://metaapi.cloud/docs/client/rateLimiting/.

Our master service file would then juggle these 4 different connections and pass these connection objects
around to the service files which use these connections to get/submit data packets. We initialize two credit
semaphores, which represent two ‘credit pools’ that we are able to work with. All our requests to the MetaAPI
server will consume some credits, and we need to distribute this credit between the competing requests. The
classical semaphore acts as a lock that allows ‘up to z threads/processes’ to execute a logic. We extended that
variant such that each entrant logic can have variable costs, which will be refunded after a custom amount
of time. For those who are interested in semaphores, I recommend the not-as-benign-as-it-sounds book ‘The
Little Book of Semaphores’. The book is open-source - https://greenteapress.com/wp/semaphores/.

To get our custom semaphore, run

pip install credit-semaphore

1 import os
2 import asyncio
3 import pytz
4 import time
5 import json
6 import pathlib
7

8 from pprint import pprint

9 from dateutil import parser
10 from datetime import datetime
11 from datetime import timedelta
12 from asyncio import Lock
13 from asyncio import Semaphore

3 Interestingly, when reading their source code, even though it is said that ‘stream’ connections do not consume the same
credits as the ‘RPC’ connections, some of the stream functionalities inherit the same parent methods as the RPC functionalities
and hence consume credits. We will hence assume that the stream credits are costly in our paper, since their is some unclear
details with regards to this aspect.

5
14

15 from m e t a a p i _ c l o u d _ s d k import MetaApi

16

17 from c r e d i t _ s e m a p h o r e . a s y n c _ c r e d i t _ s e m a p h o r e import A s y n c C r e d i t S e m a p h o r e
18 from m et aa pi _ se rv i ce . synchronizers . s y n c h r o n i z a t i o n _ m a n a g e r import S y n c h r o n i z a t i o n M a n a g e r
19 from m et aa pi _ se rv i ce . m e t a a p i _ d a t a _ m a n a g e r import M e t a A p i D a t a M a n a g e r
20 from m et aa pi _ se rv i ce . m e t a a p i _ b r o k e r _ m a n a g e r import M e t a A p i B r o k e r M a n a g e r
21

22

23 class MetaApiMaster () :
24

25 def __init__ ( self , c o n f i g _ f i l e _ p a t h = str ( pathlib . Path ( __file__ ) . parent . resolve () ) + " /
config . json " ) :
26 with open ( config_file_path , " r " ) as f :
27 config = json . load ( f )
28 os . environ [ ’ META_KEY ’] = config [ " token " ]
29 os . environ [ ’ META_ACCOUNT ’] = config [ " login " ]
30 os . environ [ ’ META_DOMAIN ’] = config [ " domain " ]
31 os . environ [ ’ GMT_OFFSET ’] = config [ " gmtOffset " ]
32

33 self . metaapi = None

34 self . account = None
35 self . s t r e a m _ c o n n e c t i o n = None
36 self . rpc _connec tion = None
37

38 self . m e t a a p i _ d a t a _ m a n a g e r = None
39 self . m e t a a p i _ b r o k e r _ m a n a g e r = None
40

41 self . res t_semap hore = A s y n c C r e d i t S e m a p h o r e (500)

42 self . s o c k e t _ s e m a p h o r e = A s y n c C r e d i t S e m a p h o r e (500)
43 self . s y n c h r o n i z a t i o n _ m a n a g e r = None
44

45 async def e n s u r e _ t i m e _ s y n c h r o n i z e d ( self ) :

46 timings = await ( await self . g e t _ r p c _ c o n n e c t i o n () ) . ge t_ se r ve r_ t im e ()
47 server = timings [ " time " ]
48 broker = parser . parse ( timings [ " brokerTime " ])
49 gmt_offset = float ( os . environ [ " GMT_OFFSET " ])
50 b r ok e r _ g m t _ t o _ u t c = broker - timedelta ( hours = gmt_offset )
51 serverdiff = ( datetime . now ( pytz . utc ) - server ) / timedelta ( minutes =1)
52 brokerdiff = ( datetime . now ( pytz . utc ) - pytz . utc . localize ( b r o k e r _ g m t _ t o _ u t c ) ) /
timedelta ( minutes =1)
53 return serverdiff < 0.5 and brokerdiff < 0.5
54

6
55 def g e t _ s y n c h r o n i z a t i o n _ m a n a g e r ( self ) :
56 if not self . s y n c h r o n i z a t i o n _ m a n a g e r :
57 self . s y n c h r o n i z a t i o n _ m a n a g e r = S y n c h r o n i z a t i o n M a n a g e r ()
58 return self . s y n c h r o n i z a t i o n _ m a n a g e r
59

60 async def g e t _ m e t a a p i _ d a t a _ m a n a g e r ( self ) :

61 if not self . m e t a a p i _ d a t a _ m a n a g e r :
62 self . m e t a a p i _ d a t a _ m a n a g e r = M e t a A p i D a t a M a n a g e r (
63 account = await self . get_account () ,
64 s y n c h r o n i z a t i o n _ m a n a g e r = self . g e t _ s y n c h r o n i z a t i o n _ m a n a g e r () ,
65 rpc_ connecti on = await self . g e t _ r p c _ c o n n e c t i o n () ,
66 s t r e a m _ c o n n e c t i o n = await self . g e t _ s t r e a m _ c o n n e c t i o n () ,
67 c l e a n _ c o n n e c t i o n s = self . clean_connections ,
68 s o c k e t _ s e m a p h o re = self . socket_semaphore ,
69 rest _semapho re = self . re st_sema phore
70 )
71 print ( " made data manager " )
72 return self . m e t a a p i _ d a t a _ m a n a g e r
73

74 async def g e t _ m e t a a p i _ b r o k e r _ m a n a g e r ( self ) :

75 if not self . m e t a a p i _ b r o k e r _ m a n a g e r :
76 self . m e t a a p i _ b r o k e r _ m a n a g e r = M e t a A p i B r o k e r M a n a g e r (
77 account = await self . get_account () ,
78 s y n c h r o n i z a t i o n _ m a n a g e r = self . g e t _ s y n c h r o n i z a t i o n _ m a n a g e r () ,
79 rpc_ connecti on = await self . g e t _ r p c _ c o n n e c t i o n () ,
80 s t r e a m _ c o n n e c t i o n = await self . g e t _ s t r e a m _ c o n n e c t i o n () ,
81 c l e a n _ c o n n e c t i o n s = self . clean_connections ,
82 s o c k e t _ s e m a p h o re = self . socket_semaphore ,
83 rest _semapho re = self . re st_sema phore
84 )
85 print ( " made broker manager " )
86 return self . m e t a a p i _ b r o k e r _ m a n a g e r
87

88 def get_metaapi ( self ) :

89 if not self . metaapi :
90 self . metaapi = MetaApi ( os . environ [ ’ META_KEY ’] , { ’ domain ’: os . environ [ ’
META_DOMAIN ’ ]})
91 return self . metaapi
92

93 async def get_account ( self ) :

94 if not self . account :
95 self . metaapi = self . get_metaapi ()
96 self . account = await self . metaapi . m e t a t r a d e r _ a c c o u n t _ a p i . get_account ( os . environ [

7
 ’ META_ACCOUNT ’ ])
97

98 print ( f " account status : { self . account . state } " )

99 print ( f " account connection status : { self . account . c o n n e c t i o n _ s t a t u s } " )
100 if self . account . state not in [ " DEPLOYED " , " DEPLOYING " ]:
101 print ( " deploying account ... " )
102 await self . account . deploy ()
103 print ( " deployed account ... " )
104 if self . account . c o n n e c t i o n _ s t a t u s != ’ CONNECTED ’:
105 print ( " connecting account ... " )
106 await self . account . wait _connect ed ()
107 print ( " connected account ... " )
108 return self . account
109

110 async def g e t _ s t r e a m _ c o n n e c t i o n ( self ) :

111 if not self . s t r e a m _ c o n n e c t i o n :
112 account = await self . get_account ()
113 self . s t r e a m _ c o n n e c t i o n = account . g e t _ s t r e a m i n g _ c o n n e c t i o n ()
114 print ( " connecting stream ... add listeners " )
115 listeners = self . g e t _ s y n c h r o n i z a t i o n _ m a n a g e r () . g e t _ m a n a g e d _ l i s t e n e r s ()
116 for listener in listeners :
117 self . s t r e a m _ c o n n e c t i o n . a d d _ s y n c h r o n i z a t i o n _ l i s t e n e r ( listener = listener )
118 await self . s t r e a m _ c o n n e c t i o n . connect ()
119 print ( " connected stream ... " )
120 print ( " synchronizing stream ... " )
121 await self . s t r e a m _ c o n n e c t i o n . w a i t _ s y n c h r o n i z e d ({ ’ t i m e o u t I n S e c o n d s ’: 60 * 2})
122 print ( " synchronized stream ... " )
123 return self . s t r e a m _ c o n n e c t i o n
124

125 async def g e t _ r p c _ c o n n e c t i o n ( self ) :

126 if not self . rpc_conn ection :
127 account = await self . get_account ()
128 self . rp c_connec tion = account . g e t _ r p c _ c o n n e c t i o n ()
129 print ( " connecting rpc ... " )
130 await self . rp c_conne ction . connect ()
131 print ( " connected rpc ... " )
132 print ( " synchronizing rpc ... " )
133 await self . rp c_conne ction . w a i t _ s y n c h r o n i z e d ()
134 print ( " synchronized rpc ... " )
135 return self . rpc_ connecti on
136

137 async def c l e a n _ c o n n e c t i o n s ( self ) :

138 if self . s t r e a m _ c o n n e c t i o n :

8
139 await self . s t r e a m _ c o n n e c t i o n . close ()
140 if self . rp c_conne ction :
141 await self . rp c_conne ction . close ()
142 if self . account :
143 await self . account . undeploy ()
144 return True

Listing 1: metaapi master.py

We need to take a look at these files.

from metaapi_service.synchronizers.synchronization_manager import SynchronizationManager

from metaapi_service.metaapi_data_manager import MetaApiDataManager
from metaapi_service.metaapi_broker_manager import MetaApiBrokerManager

First, let’s make light discussions on what the behavior of these 3 classes should do. When working with
the MetaTrader terminal, the terminal acts as both a datasource and a broker. We can execute trades using
the broker logic and get historical data such as OHLCV from the data manager. This is how we organized
the code, based on the functionality. We can then pass these manager objects around in our trading system
code to help us. There are also different ways to get data - for instance it is a common requirement for us
to get information on our open positions and orders. This can be made available by our RPC connection,
by making the request
1 await self . rp c_connec tion . get_positions ()

inside the broker manager. However, this has a few downsides· · · A quote from their website

Below is the table of the cost of each websocket/SDK API request in credits.
Please note that only RPC API requests consume CPU credits.
Real-time streaming API does not consume any credits.

Path Credits
getPositions or get_positions 50
getPosition or get_position 50
getOrders or get_orders 50
getOrder or get_order 50

These requests are costly! When we are doing things like order execution, we need to make frequent calls
to these functions, while also requesting for market quotes and submitting trade request packets. There are
multiple competing requests, and making these ‘adjacent’ calls slow down our trade execution since they

9
 compete for the same credit pool. Additionally, think about the network trips - we first go ‘hey give us the
positions’ and the server goes ‘here are your positions’ each time we need them. Since there is no local sense
of terminal state, subsequent requests made even when there are no differences consume the same credits
and network costs.

What if we just go, ‘hey from now on we want to be alerted to all changes to positions and orders’.
In the future, when our book changes, there is a single network trip from the server to us, and this occurs
even if we are busy with other work. Additionally, it is always ‘in synchronization’ with our account status
- each time we need positions, we can just peek at this synchronized state without having to make any
network trips. In the background, the synchronization manager ensures that our internal table of positions
and orders is equivalent to the one we see on the physical MetaTrader terminal. This makes getting positions
(almost) instantaneous and also convenient! We could have done this manually by creating a thread that
polls positions every a seconds and updates our state, and sleeps otherwise. But this is both inefficient (in
credit cost) and suffers from sleep-latency (in compute cost). Markedly so, adding a socket listener pushes
the task down to the operating system level, and is hence efficient. That is what the synchronization manager
will do.

3 Implementing Synchronizers

MetaAPI SDK allows this functionality through the addition of synchronization listeners. We can subscribe
for synchronization to account details, position details, order details and pricing details. We will implement
functionalities such that we can maintain an internal state for the table of positions and table of orders.

1 import pytz
2

3 from pprint import pprint

4 from datetime import datetime
5 from collections import deque
6 from collections import defaultdict
7

8 from m e t a a p i _ c l o u d _ s d k . clients . metaApi . s y n c h r o n i z a t i o n L i s t e n e r import

SynchronizationListener
9

10 class S y n c h r o n i z a t i o n M a n a g e r () :
11

12 def __init__ ( self ) :

13 self . ac co u nt _m an a ge r = Account Manager ()
14 self . p o s i t i o n _ m a n a g e r = P o si ti on M an ag er ()
15 self . order_manager = OrderManager ()

10
16 self . listeners = [ self . account_manager , self . order_manager , self . p o s i t i o n _ m a n a g e r ]
17

18 def g e t _ a c c o u n t _ m a n a g e r ( self ) :
19 return self . a c co un t_ m an ag er
20

21 def g e t _ p o s i t i o n s _ m a n a g e r ( self ) :
22 return self . p o s i t i o n _ ma n a g e r
23

24 def g e t _ o r d e r _ m a n a g e r ( self ) :
25 return self . order_manager
26

27 def g e t _ m a n a g e d _ l i s t e n e r s ( self ) :
28 return self . listeners

Listing 2: synchronization manager.py

Like the master metaapi service file, this synchronization manager class acts as the master service file
for our synchronizers. Let’s implement the synchronizers.

1 class Acc ountMana ger ( S y n c h r o n i z a t i o n L i s t e n e r ) :

2

3 def __init__ ( self , bufferlen =1000) :

4 super () . __init__ ()
5 self . equity_hist = deque ( maxlen = bufferlen )
6

7 async def o n _ a c c o u n t _ i n f o r m a t i o n _ u p d a t e d ( self , instance_index , a c c o u n t _ i n f o r m a t i o n ) :

8 self . equity_hist . append (
9 ( datetime . now ( pytz . utc ) , a c c o u n t _ i n f o r m a t i o n [ " equity " ])
10 )
11

12 def get_equity ( self ) :

13 assert len ( self . equity_hist ) != 0
14 return self . equity_hist [ -1]
15

16 class Po si ti o nM an a ge r ( S y n c h r o n i z a t i o n L i s t e n e r ) :
17 def __init__ ( self , bufferlen =1000) :
18 super () . __init__ ()
19 self . pos itions_ dict = defaultdict ( lambda : defaultdict ( dict ) )
20 self . po si d _t o_ sy m bo l = {}
21

22 @staticmethod
23 def m a k e _ p o s i t i o n _ s c h e m a ( position ) :
24 position_type = position [ " type " ]
25 volume = position [ " volume " ]

11
26 sl = position [ " stopLoss " ] if " stopLoss " in position else 0
27 tp = position [ " takeProfit " ] if " takeProfit " in position else 0
28 return {
29 " type " : position_type ,
30 " volume " : volume ,
31 " tp " : tp ,
32 " sl " : sl ,
33 " meta " : position
34 }
35

36 async def o n _ p o s i t i o n s _ r e p l a c e d ( self , instance_index , positions ) :

37 posi tions_di ct = defaultdict ( lambda : defaultdict ( dict ) )
38 p osid _t o _s ym b ol = {}
39 for position in positions :
40 id = position [ " id " ]
41 symbol = position [ " symbol " ]
42 posi tions_di ct [ symbol ][ id ] = P os i ti on Ma n ag er . m a k e _ p o s i t i o n _ s c h e m a ( position )
43 p os id _t o _s ym bo l [ id ] = symbol
44 self . pos itions_ dict = p osition s_dict
45 self . po si d _t o_ sy m bo l = p os id _ to _s ym b ol
46

47 async def o n _ p o s i t i o n s _ s y n c h r o n i z e d ( self , instance_index , s y n c h r o n i z a t i o n _ i d ) :

48 pass
49

50 async def o n _ p o s i t i o n _ u p d a t e d ( self , instance_index , position ) :

51 id = position [ " id " ]
52 symbol = position [ " symbol " ]
53 self . pos itions_ dict [ symbol ][ id ] = P o si ti o nM an ag e r . m a k e _ p o s i t i o n _ s c h e m a ( position )
54 self . po si d _t o_ sy m bo l [ id ] = symbol
55

56 async def o n _ p o s i t i o n _ r e m o v e d ( self , instance_index , position_id ) :

57 del self . positio ns_dict [ self . p o si d_ to _ sy mb ol [ position_id ]][ position_id ]
58

59 def get_positions ( self ) :

60 return self . posi tions_di ct
61

62 def g e t _ p o s i t i o n s _ s u m m a r y ( self ) :
63 summary = defaultdict ( lambda : 0.0)
64 for symbol , symbol_dict in self . p osition s_dict . items () :
65 pos = 0.0
66 for posid , posdict in symbol_dict . items () :
67 scalar = 1 if posdict [ " type " ] == " P O S I T I O N _ T Y P E _ B U Y " \
68 else ( -1 if posdict [ " type " ] == " P O S I T I O N _ T Y P E _ S E L L " else 0)

12
69 assert ( abs ( scalar ) == 1)
70 pos += scalar * posdict [ " volume " ]
71 summary [ symbol ] = pos
72 return summary
73

74 class OrderManager ( S y n c h r o n i z a t i o n L i s t e n e r ) :
75

76 def __init__ ( self ) :

77 super () . __init__ ()
78 self . last_replaced = None
79 self . orders_dict = defaultdict ( lambda : defaultdict ( dict ) )
80 self . or di d _t o_ sy m bo l = {}
81

82 self . ord_cancelled = {}
83 self . ord_filled = {}
84 return
85

86 def get_orders ( self ) :

87 return self . orders_dict
88

89 def g e t _ o r d e r s _ s u m m a r y ( self ) :
90 summary = {}
91 for symbol , symbol_dict in self . orders_dict . items () :
92 ord_sums = []
93 for ordid , orddict in symbol_dict . items () :
94 ord_sum = {}
95 scalar = 1 if orddict [ " order_type " ] == " O R D E R _ T Y P E _ B U Y _ L I M I T " \
96 else ( -1 if orddict [ " order_type " ] == " O R D E R _ T Y P E _ S E L L _ L I M I T " else 0)
97 assert abs ( scalar ) == 1
98 ord_sum [ " order " ] = orddict [ " left_volume " ] * scalar
99 ord_sum [ " limit " ] = orddict [ " open_at " ]
100 ord_sum [ " state " ] = orddict [ " state " ]
101 ord_sum [ " sl " ] = orddict [ " sl " ]
102 ord_sum [ " tp " ] = orddict [ " tp " ]
103 ord_sums . append ( ord_sum )
104 summary [ symbol ] = ord_sums
105 return summary
106

107 def g e t _ f i l l e d _ o r d e r s ( self ) :

108 return self . ord_filled
109

110 def g e t _ o r d _ c a n c e l l e d ( self ) :

111 return self . ord_cancelled

13
112

113 @staticmethod
114 def m a k e _ o r d e r _ s c h e m a ( order ) :
115 left_volume = order [ " currentVolume " ]
116 total_volume = order [ " volume " ]
117 filled_volume = total_volume - left_volume
118 expiration = order [ " expirati onType " ]
119 open_at = order [ " openPrice " ] if " openPrice " in order else 0
120 state = order [ " state " ]
121 sl = order [ " stopLoss " ] if " stopLoss " in order else 0
122 tp = order [ " takeProfit " ] if " takeProfit " in order else 0
123 order_type = order [ " type " ]
124 return {
125 " left_volume " : left_volume ,
126 " total_volume " : total_volume ,
127 " filled_volume " : filled_volume ,
128 " expiration " : expiration ,
129 " open_at " : open_at ,
130 " state " : state ,
131 " sl " : sl ,
132 " tp " : tp ,
133 " order_type " : order_type ,
134 " meta " : order
135 }
136

137 async def o n _ p e n d i n g _ o r d e r s _ r e p l a c e d ( self , instance_index , orders ) :

138 self . last_replaced = datetime . now ( pytz . utc )
139 orders_dict = defaultdict ( lambda : defaultdict ( list ) )
140 o rdid _t o _s ym b ol = {}
141 for order in orders :
142 id = order [ " id " ]
143 symbol = order [ " symbol " ]
144 orders_dict [ symbol ][ id ] = OrderManager . m a k e _ o r d e r _ s c h e m a ( order )
145 o rd id _t o _s ym bo l [ id ] = symbol
146 self . orders_dict = orders_dict
147 self . or di d _t o_ sy m bo l = o rd id _ to _s ym b ol
148

149 async def o n _ p e n d i n g _ o r d e r _ u p d a t e d ( self , instance_index , order ) :

150 id = order [ " id " ]
151 symbol = order [ " symbol " ]
152 self . orders_dict [ symbol ][ id ] = OrderManager . m a k e _ o r d e r _ s c h e m a ( order )
153 self . or di d _t o_ sy m bo l [ id ] = symbol
154

14
155 async def o n _ p e n d i n g _ o r d e r _ c o m p l e t e d ( self , instance_index , order_id ) :
156 del self . orders_dict [ self . or di d _t o_ sy m bo l [ order_id ]][ order_id ]
157

158 async def o n _ p e n d i n g _ o r d e r s _ s y n c h r o n i z e d ( self , instance_index , s y n c h r o n i z a t i o n _ i d ) :

159 pass
160

161 async def o n _ h i s t o r y _ o r d e r _ a d d e d ( self , instance_index , history_order ) :

162 id = history_order [ " id " ]
163 symbol = history_order [ " symbol " ]
164 h i s t o r y _ o r d e r _ s t a t e = history_order [ " state " ]
165 if h i s t o r y _ o r d e r _ s t a t e == " O R D E R _ S T A T E _ C A N C E L E D " :
166 self . ord_cancelled [ symbol ] = OrderManager . m a k e _ o r d e r _ s c h e m a ( history_order )
167 elif h i s t o r y _ o r d e r _ s t a t e == " O R D E R _ S T A T E _ F I L L E D " :
168 self . ord_filled [ symbol ] = OrderManager . m a k e _ o r d e r _ s c h e m a ( history_order )
169 else :
170 pass

At all times, the positions dict and orders dict will be synchronized to our terminal state. Every time
the orders or positions are created/modified/deleted, the MetaAPI server will send us a data packet with
the updates, which will trigger the asynchronous functions to be scheduled on the event loop. The functions
will then run, and our managers keep track of the updates to keep the local state 1 : 1 to the terminal state.

4 Implementing Broker Manager

Recall that the broker manager receives the following objects/parameters.

1 import os
2 import pytz
3 import asyncio
4 from dateutil import parser
5 from datetime import datetime
6 from datetime import timedelta
7

8 class M e t a A p i B r o k e r M a n a g e r () :
9

10 def __init__ ( self ,

11 account ,
12 synchronization_manager ,
13 rpc_connection ,
14 stream_connection ,
15 clean_connections ,
16 socket_semaphore ,

15
17 rest _semapho re
18 ):
19 self . account = account
20 self . synman = s y n c h r o n i z a t i o n _ m a n a g e r
21 self . rpc _connec tion = r pc_conn ection
22 self . s t r e a m _ c o n n e c t i o n = s t r e a m _ c o n n e c t i o n
23 self . c l e a n _ c o n n e c t i o n s = c l e a n _ c o n n e c t i o n s
24 self . s o c k e t _ s e m a p h o r e = s o c k e t _ se m a p h o r e
25 self . res t_semap hore = r est_sem aphore

Listing 3: metaapi broker manager.py

We can then make calls such as

1 async def get_tick ( self , symbol , k e e p _ s u b s c r i p t i o n = True ) :

2 return await
3 self . rp c_connec tion
4 . get_tick ( symbol = symbol , k e e p _ s u b s c r i p t i o n = k e e p _ s u b s c r i p t i o n )

on our MetaApiBrokerManager object. But we can simply ‘break’ this service by doing

1 while True :
2 await manager . get_tick ( " EURUSD " )

Recall that we want our service to be error-tolerant in this aspect! This costs us some credits on the MetaAPI
server, so do calls like get order book and so on. How do we distribute the resources fairly and efficiently
from our client code? We can do this with the credit semaphore alluded to earlier. But it will be quite
annoying to have to do this every time we interact with the API. We have to do something like this each
time (pseudo-code):

1 async def get_tick ( self , symbol , k e e p _ s u b s c r i p t i o n = True ) :

2 job = self . r pc_conn ection . get_tick ( symbol = symbol , k e e p _ s u b s c r i p t i o n = k e e p _ s u b s c r i p t i o n )
3 semaphore = self . c r e d i t_ s e m a p h o r e
4 result = await semaphore . transact ( coroutine = job , costs =50 , refund_in =1)
5 return result

What if we want to do other things such as error handling for the connection requests? We then have
to extend our code to something like this.

1 async def get_tick ( self , symbol , k e e p _ s u b s c r i p t i o n = True ) :

2 try :
3 job = self . r pc_conne ction . get_tick (
4 symbol = symbol , k e e p _ s u b s c r i p t i o n = k e e p _ s u b s c r i p t i o n )
5 semaphore = self . c r ed i t _ s e m a p h o r e
6 result = await semaphore . transact ( coroutine = job , costs =50 , refund_in =1)

16
 7 return result
8 except Exception as err :
9 if str ( err ) == " Tick symbol is not found , try again " :
10 return await self . get_tick ( symbol , k e e p _ s u b s c r i p t i o n = k e e p _ s u b s c r i p t i o n )
11 if str ( err ) == " Server Error " :
12 raise ServerError ()
13 if str ( err ) == " Account Disconnected " :
14 raise Co nn e ct io nE r ro r ()

This sounds like a terrible ordeal to go through each time. We can use the magic of decorators and
wrapper functions to deal with this elegantly. We can indicate that we are using a particular function to
make API reqeusts, which in turn handles both the rate limits and common error handling. Let’s first explore
some of the errors, which are specified in their documentations herein:

1. https://metaapi.cloud/docs/client/websocket/errors/

2. https://metaapi.cloud/docs/client/models/error/

3. https://metaapi.cloud/docs/client/models/tooManyRequestsError/

4. https://metaapi.cloud/docs/client/models/webSocketError/

5. https://metaapi.cloud/docs/client/models/metatraderTradeResponse/

Let’s create custom exceptions to help classify our error logic better.

1 class A c c o un t E x c e p t i on ( Exception ) :
2 pass
3

4 class R e q u o t e L i m i t E x c e p t i o n ( Exception ) :
5 pass
6

7 class OrderNotFound ( Exception ) :

8 pass
9

10 class TradeRejected ( Exception ) :

11 pass
12

13 class Tra deCancel led ( Exception ) :

14 pass
15

16 class T r a d eE r r o r U n k n o w n ( Exception ) :
17 pass
18

17
19 class Tra deModeEr ror ( Exception ) :
20 pass
21

22 class Tr ad eS p ec sE r ro r ( Exception ) :
23 pass
24

25 class T r a d eS e r v e r E r r or ( Exception ) :
26 pass
27

28 class U n h a n d l e d T r a d e d E r r o r ( Exception ) :
29 pass
30

31 class D y n a mi c T r a d e E r r o r ( Exception ) :
32 pass
33

34 class M a r k et C l o s e d E r r o r ( Exception ) :
35 pass
36

37 class T r a d eT i m e o u t E r r o r ( Exception ) :
38 pass

Listing 4: exceptions.py

We will then create a decorator function in a utility file. The decorator function is first and foremost, a
wrapper. But the wrapper function can also take parameters! Read this blog here https://realpython.com/primer-
on-python-decorators/ for decorator functions with parameters. Frankly, they can be quite confusing - es-
sentially, we are taking variable arguments and variable positional arguments in our wrapped function, but
we need to defer calling them, since we want to (further) wrap it in a credit semaphore transaction. The
asynchronous function that is wrapped is passed in as the parameter func, which is an arbitrary method
indicated as a coroutine function with the keyword async. The main concern is whether our asynchronous
function is using the REST connection or the socket connection, determining which of the semaphores we
should use to transact. Next, we also need to know how long it takes for the credits to be refunded. For
instance, if we see the section ‘Number of CPU credits available to be spent on a single account’ on rate
limiting https://metaapi.cloud/docs/client/rateLimiting/, every 10 seconds we are given 5000 credits. This
translates to a 500 credits per second. This is our rate limit, which is why our semaphore was awarded 500
credits at initialization! The refund in parameter corresponding to this would be just 1 second, and the
costs would depend on the API call made inside our tagged asynchronous function. Last but not least, we
want to add exception handlers to deal with the various exceptions thrown by the MetaAPI to make the
API calls in our service class error-tolerant and more informative. We also add a timeout ability for any
unstable network connections that potentially hang infinitely.

18
 1 import pytz
2 import time
3 import asyncio
4

5 from dateutil import parser

6 from functools import wraps
7 from datetime import datetime
8

9 from asyncio . exceptions import TimeoutError as A s y n c i o T i m e o u t E x c e p t i o n

10

11 from m e t a a p i _ c l o u d _ s d k . clients . t i m e o u t E x c e p t i on import T i m e o u t Ex c e p t i o n as

MetaTimeoutException
12 from m e t a a p i _ c l o u d _ s d k . clients . errorHandler import N o t F o u n d E x c e p t i o n
13 from m e t a a p i _ c l o u d _ s d k . clients . errorHandler import V a l i d a t i o n E x c e p t i o n
14 from m e t a a p i _ c l o u d _ s d k . clients . errorHandler import T o o M a n y R e q u e s t s E x c e p t i o n
15 from m e t a a p i _ c l o u d _ s d k . clients . metaApi . trade Exceptio n import Tra deExcep tion
16

17 from m et aa pi _ se rv i ce . utils . exceptions import R e q u o t e L i m i t E x c e p t i o n

18 from m et aa pi _ se rv i ce . utils . exceptions import A c c ou n t E x c e p t i o n
19 from m et aa pi _ se rv i ce . utils . exceptions import OrderNotFound
20 from m et aa pi _ se rv i ce . utils . exceptions import TradeRejected
21 from m et aa pi _ se rv i ce . utils . exceptions import TradeC ancelled
22 from m et aa pi _ se rv i ce . utils . exceptions import T r a d e E r r o r U n k n o w n
23 from m et aa pi _ se rv i ce . utils . exceptions import T r a d e T i m e o u t E r r o r
24 from m et aa pi _ se rv i ce . utils . exceptions import TradeM odeError
25 from m et aa pi _ se rv i ce . utils . exceptions import T ra d eS pe cs E rr or
26 from m et aa pi _ se rv i ce . utils . exceptions import T r a de S e r v e r E r r o r
27 from m et aa pi _ se rv i ce . utils . exceptions import U n h a n d l e d T r a d e d E r r o r
28 from m et aa pi _ se rv i ce . utils . exceptions import D y n a m i c T r a d e E r r o r
29 from m et aa pi _ se rv i ce . utils . exceptions import M a r k e t C l o s e d E r r o r
30

31 def u se _c o nn ec ti o ns ( _func = None , * ,

32 timeout =1000 ,
33 rpc_costs = -1 ,
34 stream_costs = -1 ,
35 rest_costs = -1 ,
36 refund_in =10
37 ):
38

39 def sync_wrapper ( func ) :

40 @wraps ( func )
41 async def c o n n e c t i o n _ h a n d l e r ( self , * args , ** kwargs ) :
42 try :

19
43 verbose = False
44 assert (
45 [ rpc_costs >= 0 , stream_costs >= 0 , rest_costs >= 0]. count ( True ) <= 1
46 )
47 c on ne ct i on _u se d = " rpc " if rpc_costs >= 0 else \
48 ( " stream " if stream_costs >= 0 else \
49 ( " rest " if rest_costs >= 0 else " " )
50 )
51 work = None
52 if co nn e ct io n_ u se d == " rpc " or co n ne ct io n _u se d == " stream " :
53 s o c k e t _ s e m a p h o r e = self . s o c k e t _ s em a p h o r e
54 costs = stream_costs if stream_costs >= 0 else rpc_costs
55 assert ( costs >= 0)
56 work = s o c k e t _ s e m a p h o r e . transact (
57 coroutine = func ( self , * args , ** kwargs ) ,
58 credits = costs ,
59 refund_time = refund_in ,
60 trans action_ id = args ,
61 verbose = verbose
62 )
63 elif c on ne ct i on _u se d == " rest " :
64 rest_ semapho re = self . re st_semap hore
65 assert ( rest_costs >= 0)
66 work = res t_semap hore . transact (
67 coroutine = func ( self , * args , ** kwargs ) ,
68 credits = rest_costs ,
69 refund_time = refund_in ,
70 trans action_ id = args ,
71 verbose = verbose
72 )
73 elif c on ne ct i on _u se d == " " :
74 work = func ( self , * args , ** kwargs )
75

76 assert ( work )
77

78 result = None
79

80 if not timeout :
81 result = await work
82 else :
83 result = await asyncio . wait_for (
84 asyncio . create_task ( work ) , timeout = timeout )
85

20
86 return result
87

88 except ( AsyncioTimeoutException , M e t a T i m e o u t E x c e p t i o n ) as e :
89 raise e
90

91 except N o t F o u n d E x c e p t i o n as e :
92 retryable , err = n o t _ f o u n d _ e x c e p t i o n _ h a n d l e r ( e )
93 if retryable :
94 return await c o n n e c t i o n _ h a n d l e r ( self , * args , ** kwargs )
95 raise err
96

97 except T o o M a n y R e q u e s t s E x c e p t i o n as e :
98 retryable , err = e x c e p t i o n _ 4 2 9 _ h a n d l e r ( e )
99 if retryable :
100 return await c o n n e c t i o n _ h a n d l e r ( self , * args , ** kwargs )
101 raise err
102

103 except TradeE xceptio n as e :

104 retryable , err = t r a d e _ e x c e p t i o n _ h a n d l e r ( e )
105 if retryable :
106 return await c o n n e c t i o n _ h a n d l e r ( self , * args , ** kwargs )
107 raise err
108

109 return c o n n e c t i o n _ h a n d l e r
110

111 if _func is None :

112 return sync_wrapper
113 else :
114 return sync_wrapper ( _func )

Listing 5: metaapi utilities.py

We need to implement the additional error handlers. We categorize errors into the retryable and un-
retryable type. Suppose a network request fails on first request when the data is not on cache, and only
successfully returns data from the second request packet onward, we can just retry the request without
throwing the error back to the caller.

1 def t r a d e _ e x c e p t i o n _ h a n d l e r ( exc ) :
2 code = exc . stringCode
3 t h r o w a bl e s _ c o d e s = [
4 " TRADE_RETCODE_ORDER_TYPE_NOT_SUPPORTED ",
5 " TRADE_RETCODE_UNKNOWN ",
6 " TRADE_RETCODE_REQUOTE ",
7 " TRADE_RETCODE_TIMEOUT ",

21
 8 " TRADE_RETCODE_PRICE_CHANGED ",
9 " TRADE_RETCODE_PRICE_OFF ",
10 " TRADE_RETCODE_INVALID_EXPIRATION ",
11 " TRADE_RETCODE_ORDER_CHANGED ",
12 " TRADE_RETCODE_TOO_MANY_REQUESTS ",
13 " TRADE_RETCODE_SERVER_DISABLES_AT ",
14 " TRADE_RETCODE_CLIENT_DISABLES_AT ",
15 " TRADE_RETCODE_LOCKED ",
16 " TRADE_RETCODE_FROZEN ",
17 " TRADE_RETCODE_FIFO_CLOSE ",
18 " TRADE_RETCODE_CLOSE_ORDER_EXIST ",
19 " TRADE_RETCODE_REJECT_CANCEL "
20 ]
21 if code == " T R A D E _ R E T C O D E _ T I M E O U T " :
22 return True , T r a d e T i m e o u t E r r o r ( str ( exc ) )
23 if code == " T R A D E _ R E T C O D E _ N O _ M O N E Y " :
24 return False , A c co u n t E x c e p t i o n ( str ( exc ) )
25 if code == " E R R _ T R A D E _ O R D E R _ N O T _ F O U N D " :
26 return False , OrderNotFound ( str ( exc ) )
27 if code == " T R A D E _ R E T C O D E _ R E J E C T " :
28 return False , TradeRejected ( str ( exc ) )
29 if code == " T R A D E _ R E T C O D E _ C A N C E L " :
30 return False , TradeC ancelled ( str ( exc ) )
31 if code == " T R A D E _ R E T C O D E _ E R R O R " :
32 return False , T r a d e E r r o r U n k n o w n ( str ( exc ) )
33 if code == " T R A D E _ R E T C O D E _ M A R K E T _ C L O S E D " :
34 return False , M a r k e t C l o s e d E r r o r ( str ( exc ) )
35

36 if code in [
37 " TRADE_RETCODE_LONG_ONLY ",
38 " TRADE_RETCODE_SHORT_ONLY ",
39 " TRADE_RETCODE_CLOSE_ONLY ",
40 " TRADE_RETCODE_TRADE_DISABLED "
41 ]:
42 return False , TradeM odeError ( str ( exc ) )
43

44 if code in [
45 " TRADE_RETCODE_INVALID_PRICE ",
46 " TRADE_RETCODE_INVALID_STOPS "
47 ]:
48 return False , R e q u o t e L i m i t E x c e p t i o n ( str ( exc ) )
49

50 if code in [

22
51 " TRADE_RETCODE_INVALID_FILL ",
52 " TRADE_RETCODE_INVALID_ORDER ",
53 " TRADE_RETCODE_INVALID_VOLUME ",
54 " TRADE_RETCODE_INVALID_CLOSE_VOLUME "
55 ]:
56 return False , T ra de S pe cs Er r or ( str ( exc ) )
57

58 if code in [
59 " TRADE_RETCODE_CONNECTION ",
60 " TRADE_RETCODE_ONLY_REAL ",
61 " TRADE_RETCODE_LIMIT_ORDERS ",
62 " TRADE_RETCODE_LIMIT_VOLUME ",
63 " TRADE_RETCODE_LIMIT_POSITIONS "
64 ]:
65 return False , T r ad e S e r v e r E r r o r ( str ( exc ) )
66

67 if code in [
68 " TRADE_RETCODE_INVALID ",
69 " TRADE_RETCODE_POSITION_CLOSED ",
70 " TRADE_RETCODE_TRADE_POSITION_NOT_FOUND ",
71 " ERR_TRADE_POSITION_NOT_FOUND "
72 ]:
73 return False , D y n a m i c T r a d e E r r o r ( str ( exc ) )
74

75 if code in t h r o w a b l e s _ c o d e s :
76 return False , U n h a n d l e d T r a d e d E r r o r ( str ( exc ) )
77

78 return False , exc

79

80 import warnings
81 def e x c e p t i o n _ 4 2 9 _ h a n d l e r ( exc ) :
82 r e c o m m en d e d _ w a i t = exc . metadata [ " r e c o m m e n d e d R e t r y T i m e " ]
83 wait_datetime = parser . parse ( r e c o m m e n d e d _ w a i t )
84 wait_for = ( wait_datetime - datetime . now ( pytz . utc ) ) . total_seconds ()
85 if wait_for > 240:
86 return False , exc
87 if wait_for > 0:
88 warnings . warn ( f " blocking sleep - { wait_for } " )
89 time . sleep ( wait_for )
90 return True , exc
91

92 def n o t _ f o u n d _ e x c e p t i o n _ h a n d l e r ( exc ) :
93 if " Specified symbol tick not found " in str ( exc ) :

23
94 return True , exc
95 if " Specified symbol price not found " in str ( exc ) :
96 return True , exc
97 return False , exc

Listing 6: metaapi utilities.py

Decoding the error messages take some time, and some of them are not as immediately obvious. For
instance, trying to close a limit order that has already been hit will return a ‘TRADE RETCODE INVALID’
error code. Submitting limit orders with guaranteed instantaneous fill (such as those with market prices
already moving past the limit) would throw an ‘TRADE RETCODE INVALID PRICE’ error. An example
of when such an error might arise is if we first take a peek at the bid-ask and make a limit at bid for a long
trade. If the price in the meantime (in between the network trips) move below the bid, the limit would be
considered invalid.

Now that we have handled the error, we can liberally mark the API function calls to the MetaAPI server
with the @use connections decorator so that the coroutine is performed in a semaphore transaction! This is
all done without altering the ‘business logic’ of the original function.

Let’s take a look at the broker manager class.

1 import os
2 import pytz
3 import asyncio
4 from dateutil import parser
5 from datetime import datetime
6 from datetime import timedelta
7

8 from m et aa pi _ se rv i ce . utils . m e t a a p i _ u t i l i t i e s import us e_ c on ne ct i on s

9

10 class M e t a A p i B r o k e r M a n a g e r () :
11

12 def __init__ ( self ,

13 account ,
14 synchronization_manager ,
15 rpc_connection ,
16 stream_connection ,
17 clean_connections ,
18 socket_semaphore ,
19 rest _semapho re
20 ):
21 self . account = account
22 self . synman = s y n c h r o n i z a t i o n _ m a n a g e r

24
23 self . rpc _connec tion = r pc_conn ection
24 self . s t r e a m _ c o n n e c t i o n = s t r e a m _ c o n n e c t i o n
25 self . c l e a n _ c o n n e c t i o n s = c l e a n _ c o n n e c t i o n s
26 self . s o c k e t _ s e m a p h o r e = s o c k e t _ se m a p h o r e
27 self . res t_semap hore = r est_sem aphore
28

29 """
30 SECTION :: UTILITIES
31 """
32 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)
33 async def ge t _s er ve r _t im e ( self ) :
34 result = await self . rp c_connec tion . g et _s er v er _t im e ()
35 server_utc = result [ " time " ] # utc
36 broker_gmt = result [ " brokerTime " ] # gmt + x
37 broker_gmt = parser . parse ( broker_gmt )
38 return server_utc , broker_gmt
39

40 def g e t _ t e r m i n a l _ s p e c s ( self , symbol ) :

41 terminalState = self . s t r e a m _ c o n n e c t i o n . t erminal_ state
42 return terminalState . speci ficatio ns
43

44 def g e t _ c o n t r a c t _ s p e c s ( self , symbol ) :

45 terminalState = self . s t r e a m _ c o n n e c t i o n . t erminal_ state
46 return terminalState . specification ( symbol = symbol )
47

48 """
49 SECTION :: ACCOUNT
50 """
51 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)
52 async def g e t _ a c c o u n t _ i n f o ( self ) :
53 result = await self . rp c_connec tion . g e t _ a c c o u n t _ i n f o r m a t i o n ()
54 return result
55

56 @ u s e _ c on n e c t i o n s ( rpc_costs =500 , refund_in =1)

57 async def g e t _ a l l _ i n s t r u m e n t s ( self ) :
58 result = await self . rpc_co nnectio n . get_symbols ()
59 return result
60

61 @ u s e _ c on n e c t i o n s ( rpc_costs =500 , refund_in =1)

62 async def get_inst_spec ( self , symbol ) :
63 result = await self . rpc_co nnectio n . g e t _ s y m b o l _ s p e c i f i c a t i o n ( symbol )
64 return result
65

25
66 """
67 SECTION :: PORTFOLIO
68 """
69 # @ u s e _ co n n e c t i o n s ( rpc_costs =50 , refund_in =1)
70 async def g e t _ a c c o u n t _ e q u i t y ( self ) :
71 equity = self . synman . g e t _ a c c o u n t _ m a n a g e r () . get_equity ()
72 return equity
73

74 async def get_positions ( self , symbol = None , summary = True ) :

75 posman = self . synman . g e t _ p o s i t i o n s _ m a n a g e r ()
76 positions = posman . g e t _ p o s i t i o n s _ s u m m a r y () if summary else posman . get_positions ()
77 return positions [ symbol ] if symbol else positions
78

79 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)

80 async def g e t _ p o s i t i o n _ w i t h _ i d ( self , position_id ) :
81 result = await self . rp c_connec tion . get_position ( position_id )
82 return result
83

84 async def wi pe_posi tions ( self ) :

85 posi tions_di ct = await self . get_positions ( summary = False )
86 close_tasks = []
87 for symbol , symbol_dict in p ositions _dict . items () :
88 for posid , posdict in symbol_dict . items () :
89 close_task = asyncio . create_task (
90 self . c l o s e _ f u l l _ p o s i t i o n _ a t _ m a r k e t _ w i t h _ i d ( position_id = posid )
91 )
92 close_tasks . append ( close_task )
93 await asyncio . gather (* close_tasks )
94 return
95

96 async def get_orders ( self , symbol = None , summary = True ) :

97 ordman = self . synman . g e t _ o r d e r _ m a n a g e r ()
98 orders = ordman . g e t _ o r d e r s _ s u m m a r y () if summary else ordman . get_orders ()
99 return orders [ symbol ] if symbol else orders
100

101 async def wipe_orders ( self ) :

102 orders_dict = await self . get_orders ( summary = False )
103 cancel_tasks = []
104 for symbol , symbol_dict in orders_dict . items () :
105 for ordid , orddict in symbol_dict . items () :
106 cancel_task = asyncio . create_task (
107 self . cancel_order ( order_id = ordid )
108 )

26
109 cancel_tasks . append ( cancel_task )
110 await asyncio . gather (* cancel_tasks )
111 return
112

113 """
114 SECTION :: TRADE
115 """
116 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)
117 async def m a k e _ n e w _ m a r k e t _ o r d e r ( self , symbol , contracts , sl = None , tp = None ) :
118 contracts = float ( contracts )
119 assert contracts != 0
120 if contracts > 0:
121 result = await self . rpc _connec tion . c r e a t e _ m a r k e t _ b u y _ o r d e r ( symbol = symbol , volume
= contracts , stop_loss = sl , take_profit = tp )
122 if contracts < 0:
123 result = await self . rpc _connec tion . c r e a t e _ m a r k e t _ s e l l _ o r d e r ( symbol = symbol ,
volume = abs ( contracts ) , stop_loss = sl , take_profit = tp )
124 return result [ " orderId " ]
125

126 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

127 async def m a k e _ n e w _ l i m i t _ o r d e r ( self , symbol , contracts , open , sl = None , tp = None ) :
128 contracts = float ( contracts )
129 assert contracts != 0
130 if contracts > 0:
131 result = await self . rpc _connec tion . c r e a t e _ l i m i t _ b u y _ o r d e r ( symbol = symbol , volume =
contracts , open_price = open , stop_loss = sl , take_profit = tp )
132 if contracts < 0:
133 result = await self . rpc _connec tion . c r e a t e _ l i m i t _ s e l l _ o r d e r ( symbol = symbol , volume
= abs ( contracts ) , open_price = open , stop_loss = sl , take_profit = tp )
134 return result [ " orderId " ]
135

136 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

137 async def c l o s e _ p a r t i a l _ p o s i t i o n _ a t _ m a r k e t _ w i t h _ i d ( self , position_id , contracts ) :
138 assert contracts != 0
139 if contracts > 0:
140 result = await self . rpc _connec tion . c l o s e _ p o s i t i o n _ p a r t i a l l y ( position_id =
position_id , volume = contracts )
141 if contracts < 0:
142 result = await self . rpc _connec tion . c l o s e _ p o s i t i o n _ p a r t i a l l y ( position_id =
position_id , volume = abs ( contracts ) )
143 return result
144

145 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

27
146 async def c l o s e _ f u l l _ p o s i t i o n _ a t _ m a r k e t _ w i t h _ i d ( self , position_id ) :
147 result = await self . rp c_connec tion . close_po sition ( position_id = position_id )
148 return result [ " positionId " ]
149

150 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

151 async def c l o s e _ f u l l _ p o s i t i o n _ a t _ m a r k e t _ w i t h _ s y m b o l ( self , symbol ) :
152 result = await self . rp c_connec tion . c l o s e _ p o s i t i o n s _ b y _ s y m b o l ( symbol = symbol )
153 return result
154

155 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

156 async def m o d i f y _ o p e n _ p o s i t i o n ( self , position_id , sl = None , tp = None ) :
157 result = await self . rp c_connec tion . m od if y_ p os it io n ( position_id = position_id ,
stop_loss = sl , take_profit = tp )
158 return result [ " positionId " ]
159

160 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

161 async def m o d i f y _ p e n d i n g _ o r d e r ( self , order_id , open , sl = None , tp = None ) :
162 result = await self . rp c_connec tion . modify_order ( order_id = order_id , open_price = open ,
stop_loss = sl , take_profit = tp )
163 return result
164

165 @ u s e _ c on n e c t i o n s ( rpc_costs =10 , refund_in =1)

166 async def cancel_order ( self , order_id ) :
167 result = await self . rp c_connec tion . cancel_order ( order_id = order_id )
168 return result [ " orderId " ]
169

170 """
171 SECTION :: HISTORY
172 """
173 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)
174 async def o r d e r _ h i s t o r y _ b y _ t i c k e t ( self , ticket_id ) :
175 result = await self . rp c_connec tion . g e t _ h i s t o r y _ o r d e r s _ b y _ t i c k e t ( ticket = ticket_id )
176 return result
177

178 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)

179 async def o r d e r _ h i s t o r y _ b y _ p o s i t i o n ( self , position_id ) :
180 result = await self . rp c_connec tion . g e t _ h i s t o r y _ o r d e r s _ b y _ p o s i t i o n ( position_id =
position_id )
181 return result
182

183 @ u s e _ c on n e c t i o n s ( rpc_costs =75 , refund_in =1)

184 async def o r d e r _ h i s t o r y _ i n _ r a n g e ( self , start , end , offset =0 , limit =1000) :
185 result = await self . rp c_connec tion . g e t _ h i s t o r y _ o r d e r s _ b y _ t i m e _ r a n g e ( start_time = start

28
 , end_time = end , offset = offset , limit = limit )
186 return result
187

188 """
189 SECTION :: DATA
190 """
191 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)
192 async def get_tick ( self , symbol , k e e p _ s u b s c r i p t i o n = True ) :
193 return await self . rpc_con nection . get_tick ( symbol = symbol , k e e p _ s u b s c r i p t i o n =
keep_subscription )
194

195 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)

196 async def ge t_order _book ( self , symbol , k e e p _ s u b s c r i p t i o n = False ) :
197 return await self . rpc_con nection . get_book ( symbol = symbol , k e e p _ s u b s c r i p t i o n =
keep_subscription )
198

199 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)

200 async def ge t _l as t_ c an dl e ( self , symbol , granularity , k e e p _ s u b s c r i p t i o n = False ) :
201 assert ( granularity in [ " 1 m " , " 2 m " , " 3 m " , " 4 m " , " 5 m " , " 6 m " , " 10 m " , \
202 " 12 m " , " 15 m " , " 20 m " , " 30 m " , " 1 h " , " 2 h " , " 3 h " , " 4 h " , " 6 h " , " 8 h " , " 12 h " , " 1 d " , " 1 w
" , " 1 mn " ])
203 return await self . rpc_con nection . get_candle ( symbol = symbol , timeframe = granularity ,
keep_subscription = keep_subscription )
204

205 @ u s e _ c on n e c t i o n s ( rpc_costs =50 , refund_in =1)

206 async def ge t_last_ price ( self , symbol , k e e p _ s u b s c r i p t i o n = False ) :
207 return await self . rpc_con nection . g e t _ sy m b o l _ p r i c e ( symbol = symbol , k e e p _ s u b s c r i p t i o n =
keep_subscription )
208

209 async def get_last_tick ( self , symbol ) :

210 tick = await self . get_tick ( symbol )
211 bid = tick [ " bid " ] if " bid " in tick else None
212 ask = tick [ " ask " ] if " ask " in tick else None
213 utc_tick = tick [ " time " ]
214

215 m i n _ s i n c e _ l a s t _ t i c k = ( datetime . now ( pytz . utc ) - utc_tick ) / timedelta ( minutes =1)

216

217 if m i n _ s i n c e _ l a s t _ t i c k >= 1:
218 return None , None
219

220 while ( m i n _ s i n c e _ l a s t _ t i c k < 1) and ( not bid or not ask ) :

221 tick = await self . get_tick ( symbol )
222 if not bid :

29
223 bid = tick [ " bid " ] if " bid " in tick else None
224 if not ask :
225 ask = tick [ " ask " ] if " ask " in tick else None
226 utc_tick = tick [ " time " ]
227 m i n _ s i n c e _ l a s t _ t i c k = ( datetime . now ( pytz . utc ) - utc_tick ) / timedelta ( minutes =1)
228 return bid , ask

Listing 7: metaapi broker manager.py

Notice how we commented out the line

#@use_connections(rpc_costs=50, refund_in=1)
async def get_account_equity(self):
equity = self.synman.get_account_manager().get_equity()
return equity

This is one the functionalities provided by our synchronization manager! We can get the equity without
having to make an API request since we were keeping track of it in the background with the AccountManager
inside the SynchronizationManager. Also, another interesting fact is that the function does not need to be
asynchronous at all, but we kept the async signature - we want to maintain a flexible caller interface because
(i) the internal implementation can vary and (ii) we can use the same caller interface in another package,
say IBKRBrokerManager.

That’s great! Now, if we do a getTick call and the MetaTrader throws a symbol not found error (because
it is loading the data), then we get a NotFoundException which is routed to our 429 handler, which checks
the error message, marks this as a retryable exception and requests for the data packet again.

Awesome. Let’s move on to the data manager.

5 Implementing Data Manager

The data manager allows us to get historical data from our MetaTrader terminal.

1 import sys
2 import pytz
3 import asyncio
4 import pandas as pd
5

6 from collections import deque

7 from datetime import datetime
8

30
 9 from m et aa pi _ se rv i ce . utils . m e t a a p i _ u t i l i t i e s import us e_ c on ne ct i on s
10

11 class M e t a A p i D a t a M a n a g e r () :
12

13 def __init__ ( self ,

14 account ,
15 synchronization_manager ,
16 rpc_connection ,
17 stream_connection ,
18 clean_connections ,
19 socket_semaphore ,
20 rest _semapho re
21 ):
22 self . account = account
23 self . synman = s y n c h r o n i z a t i o n _ m a n a g e r
24 self . rpc _connec tion = r pc_conn ection
25 self . s t r e a m _ c o n n e c t i o n = s t r e a m _ c o n n e c t i o n
26 self . c l e a n _ c o n n e c t i o n s = c l e a n _ c o n n e c t i o n s
27 self . s o c k e t _ s e m a p h o r e = s o c k e t _ se m a p h o r e
28 self . res t_semap hore = r est_sem aphore
29

30 @ u s e _ c on n e c t i o n s ( timeout =240 , rest_costs =0.3 , refund_in =1)

31 async def _get_ticks ( self , ticker , start , offset , tries =0) :
32 try :
33 print ( f " { ticker } on try { tries } " )
34 result = await asyncio . wait_for ( asyncio . create_task (
35 self . account . g e t _ h i s t o r i c a l _ t i c k s ( ticker , start , offset )
36 ) , timeout =60)
37 return result
38 except asyncio . exceptions . TimeoutError as err :
39 if tries < 5:
40 print ( f " retry { ticker } ( tries { tries }) " )
41 return await self . _get_ticks ( ticker , start , offset , tries = tries + 1)
42 print ( f " ERROR { sys . _getframe () . f_code . co_name } { err } for { ticker } " )
43 return []
44

45 @ u s e _ c on n e c t i o n s ( timeout =240 , rest_costs =0.5 , refund_in =0.7)

46 async def _get_ohlcv ( self , ticker , granularity , end , tries =0) :
47 assert granularity in [ " 1 m " , " 2 m " , " 3 m " , " 4 m " , " 5 m " , " 6 m " , \
48 " 10 m " , " 12 m " , " 15 m " , " 20 m " , " 30 m " , " 1 h " , " 2 h " , " 3 h " , \
49 " 4 h " , " 6 h " , " 8 h " , " 12 h " , " 1 d " , " 1 w " , " 1 mn "
50 ]
51 try :

31
52 print ( f " { ticker } on try { tries } " )
53 result = await asyncio . wait_for ( asyncio . create_task (
54 self . account . g e t _ h i s t o r i c a l _ c a n d l e s ( ticker , granularity , end )
55 ) , timeout =60)
56 return result
57

58 except asyncio . exceptions . TimeoutError as err :

59 if tries < 5:
60 print ( f " retry { ticker } ( tries { tries }) " )
61 return await self . _get_ohlcv ( ticker , granularity , end , tries = tries + 1)
62 print ( f " ERROR { sys . _getframe () . f_code . co_name } { err } for { ticker } " )
63 return []
64

65 async def a s y n _ b a t c h _ g e t _ o h l c v ( self , tickers , tickermetas , period_starts , period_ends ,

granularity ) :
66 tasks = []
67 for ticker , tickermeta , period_start , period_end in zip ( tickers , tickermetas ,
period_starts , period_ends ) :
68 task = asyncio . create_task (
69 self . get_ohlcv (
70 ticker = ticker ,
71 granularity = granularity ,
72 period_start = period_start ,
73 period_end = period_end
74 )
75 )
76 tasks . append ( task )
77 return await asyncio . gather (* tasks )
78

79 async def get_ohlcv ( self , ticker , granularity , period_start , period_end ) :

80 ohlcvs = await self . _get_ohlcv ( ticker , granularity , period_end )
81 if not ohlcvs :
82 return pd . DataFrame ()
83 ohlcvss = deque ()
84 ohlcvss . appendleft ( ohlcvs )
85 end_time = ohlcvs [0][ " time " ]
86 while ohlcvs and end_time > period_start :
87 ohlcvs = await self . _get_ohlcv ( ticker , granularity , end_time )
88 if ohlcvs and len ( ohlcvs ) > 1:
89 end_time = ohlcvs [0][ " time " ]
90 ohlcvss . appendleft ( ohlcvs )
91 else :
92 break

32
93 subdfs = [ pd . DataFrame . from_records ( ohlcvs ) for ohlcvs in ohlcvss ]
94 series_df = pd . concat ( subdfs , axis =0) . d ro p_ du p li ca te s ( " time " )
95 series_df = series_df . loc [ series_df [ " time " ] >= period_start ]. reset_index ( drop = True )
96 series_df = series_df . rename ( columns ={ " time " : " datetime " , " brokerTime " : " brokerGMT
" })
97 series_df [ " brokerGMT " ] = pd . to_datetime ( series_df [ " brokerGMT " ])
98 series_df = series_df [[ " symbol " , " datetime " , " brokerGMT " , " open " , " high " , " low " , "
close " , " tickVolume " ]]
99 series_df = series_df . rename ( columns ={ " close " : " adj_close " , " tickVolume " : " volume "
})
100 if granularity == " 1 d " :
101 series_df [ " datetime " ] = series_df [ " datetime " ]. apply (
102 lambda dt : datetime ( dt . year , dt . month , dt . day , tzinfo = pytz . utc )
103 )
104 return series_df
105

106 async def get_ticks ( self , ticker , period_start , period_end ) :

107 offset = 0
108 ticks = await self . _get_ticks ( ticker , period_start , offset )
109 results = []
110 results . extend ( ticks )
111 start_time = ticks [ -1][ " time " ]
112 while ticks :
113 ticks = await self . _get_ticks ( ticker , start_time , 1)
114 if ticks and len ( ticks ) :
115 start_time = ticks [ -1][ " time " ]
116 results . extend ( ticks )
117 else :
118 break
119 series_df = pd . DataFrame . from_records ( results ) . rename ( columns ={ " time " : " datetime " ,
" brokerTime " : " brokerGMT " })
120 series_df [ " brokerGMT " ] = pd . to_datetime ( series_df [ " brokerGMT " ])
121 series_df = series_df [[ " symbol " , " datetime " , " brokerGMT " , " bid " , " ask " ]]
122 series_df = series_df . fillna ( method = " ffill " ) . fillna ( method = " bfill " )
123 return series_df

Listing 8: metaapi data manager.py

5.1 A Bonus

Recall in our previous work [4] on improving the database service, there was a code section like this:

1 async def a s y n _ b a t c h _ g e t _ o h l c v (

33
 2 self , tickers , read_db , insert_db , granularity , engine , tickermetas = None , period_start =
None , period_end = None , duration = None , chunksize =100
3 ):
4 assert ( engine in [ " e o d h i s t o r i c a l d a t a " , " DWX - MT5 " , " yfinance " ])
5

6 if engine == " e o d h i s t o r i c a l d a t a " :

7 datapoller = eod_wrapper
8 dformat = " spot "
9 elif engine == " DWX - MT5 " :
10 datapoller = await self . data_clients [ " meta_client " ]. g e t _ m e t a a p i _ d a t a _ m a n a g e r ()
11 dformat = " CFD "
12 elif engine == " yfinance " :
13 datapoller = y f i n a n c e _ w ra p p e r
14 dformat = " spot "
15

16 return await w r a p p e r _ a s y n _ b a t c h _ g e t _ o h l c v (
17 tickers = tickers ,
18 read_db = read_db ,
19 insert_db = insert_db ,
20 granularity = granularity ,
21 db_service = self . db_service ,
22 datapoller = datapoller ,
23 engine = engine ,
24 dtype = Equities . dtype ,
25 dformat = dformat ,
26 tickermetas = tickermetas ,
27 period_start = period_start ,
28 period_end = period_end ,
29 duration = duration ,
30 chunksize = chunksize ,
31 results =[] ,
32 tasks =[] ,
33 batch_id =0
34 )

And we talked about how the datapoller does not care about the datasource, but rather that the function
signature

async def asyn_batch_get_ohlcv(

self, tickers, tickermetas, period_starts, period_ends, granularity
)

is implemented in the datapoller object. Note that our MetaApiDataManager does in fact implement this

34
signature, and can act as a datapoller (we can make this cleaner and enforce this by using an abstract class).
Now we can do read/writes to our database without writing any additional code - all we have to do is to pass in
the MetaApiDataManager object as a datapoller, and since it meets the ‘function specifications’, specifying
the engine ‘DWX-MT5’ would retrieve the MetaTrader terminal data, specifying ‘yfinance’ retrieves the
Yahoo Finance data, and engine = ‘eodhistoricaldata’ gets data from our eodhistoricaldata database. That’s
neat.

35
References

[1] HangukQuant. A Barebone Implementation of an Order Executor. https://hangukquant.substack.

com/p/a-barebone-order-executor-w-code.

[2] HangukQuant. Credit Semaphores. https://open.substack.com/pub/hangukquant/p/

new-python-libraries-credit-semaphores.

[3] HangukQuant. Design and Implementation of a Quant Database. https://hangukquant.substack.

com/p/117-pages-design-and-implementation?utm_source=substack.

[4] HangukQuant. Improving our Database Service. https://hangukquant.substack.com/p/

improving-our-database-service.

36