the

Availability Digest
September 2008

Cellular Provider Goes Active/Active for Prepaid Calls

The Pan-African Cellular Service Provider

One of the largest cellular telephone service providers in Africa holds more than 50% of the cellphone market share in its country. It manages a cell network made up of thousands of cell sites, which provide coverage to over 95% of the South African population. It is a testament to the great open space in South Africa that this coverage only needs to extend to 6% of South Africas total geographical area. The company also provides cell service to several other African countries, including Tanzania, Mozambique, Lesotho, and the Democratic Republic of the Congo. In total, over 20 million customers in these countries are serviced by this provider. The prepaid calling card market is the fastest growing cellular service segment in Africa. From a processing viewpoint, prepaid cards generate about 1.5 to 2 million updates per hour against a six-terabyte database. To handle this volume, the company uses in its production environment three NonStop servers with XP storage along with some other ancillary systems. If the prepaid calling card service is unavailable, much of Africas cellular service comes to a halt. To ensure the continuous availability of prepaid card service, the heart of this production system is run as an active/active configuration.

The Prepaid Calling Card

Prepaid calling cards are purchased by subscribers at stores throughout the companys service area. Each card carries a unique identification number (PIN) and entitles the purchaser to a specified amount of additional money on his call account. A subscriber activates his additional money by calling the companys prepaid calling card service and entering the card identification number. A subscriber can also recharge his time without having to purchase a prepaid calling card by recharging at an automated teller machine (ATM) or through his banks web interface. Subscribers call accounts are kept on Intelligent Network (IN) platforms. These platforms are not supplied by HP, so communication with these platforms is over TCP/IP using a proprietary message layout.

2008 Sombers Associates, Inc., and W. H. Highleyman www.availabilitydigest.com

The Prepaid Recharge System

Prepaid calling cards are generally called vouchers since they represent a monetary value rather than call time. The IN platforms deduct the cost of a call from the subscribers call account. The prepaid calling card service is provided by the companys Prepaid Recharge System, which manages money recharges by the companys subscribers. This system provides facilities for subscribers to add money to their call accounts by entering a prepaid voucher PIN. Typically, users will redeem vouchers via their cell phones. However, recharge facilities are also available via bank ATMs or at point-of-sale (POS) terminals in the bigger supermarket chains. Web services also allow the subscriber to purchase SMS or data bundles out of his call account. Using the web services, subscribers can also view their remaining credit and transfer money from one cell-phone number to another. When a user first turns on his cell phone, the cell phone logs him into the network via the closest 1 cell-phone tower. Each user is assigned an IN Home Location Register (HLR), which is an active/active NonStop system that tracks the current location of the user and his account profile. The users HLR is determined from the first six digits of his cell-phone number. Upon logon, IN will obtain the users profile from his HLR. This profile information includes his remaining call time. When a user dials a number, IN will establish a connection between him and his called party. Upon completion of the call, the length of the call is deducted from his remaining minutes.

recharge request dial 100

Prepaid Recharge System

Redeem vouchers Transfer funds between IN accounts Recharges via ATMs SMS and data bundle purchases Query remaining funds

subscriber browser access

Recharge IN Platforms

To add money to his account via his cell phone, a subscriber dials a special number. This connects him to the companys Prepaid Recharge System through an automatic voice response system. This system will lead him through a menu from which he picks his appropriate service. If he chooses to redeem a prepaid calling card voucher that he has purchased, he enters the voucher PIN. This adds the amount of money represented by the voucher that he has purchased. The Prepaid Recharge System will forward this recharge time to the IN system, which will update the users profile to add the additional money to his account. It will also log the transaction to a

HPs Active/Active Home Location Register, Availability Digest; November 2006.

2008 Sombers Associates, Inc., and W. H. Highleyman www.availabilitydigest.com

Recharge Log for later reconciliation between the Prepaid Recharge System and IN as well as update the transaction display on the web interface.

The Active/Active Heart

The heart of this system is the Prepaid Front End, or PPFE. It is the PPFE that interacts with the user to recharge his account. Should the PPFE fail, no subscriber can add money to his account. If he has run out of money, he no longer has cell-phone service.

recharge request

recharge request

Shadowbase NonStop server Pretoria data replication NonStop server Midrand IN Platforms

Prepaid Front End ( PPFE)

Therefore, the PPFE must be up virtually 100% of the time. If it should go down, subscribers cannot recharge; and cell-phone service will be denied to a large segment of cell-phone subscribers. To ensure extreme availability for prepaid card service, the company has implemented the PPFE as a pair of NonStop nodes in an active/active architecture. One node is located in Pretoria, South Africa, and the other is located in Midrand, South Africa. The two nodes carry identical databases. The databases are kept in synchronism via bidirectional asynchronous replication provided by Gravic, Inc.s Shadowbase replication engine (www.gravic.com). The primary information that is replicated is that contained in the Recharge Logs. Should a node fail, all call transactions are routed to the surviving node, which continues to provide prepaid card services for all subscribers in the companys network. When the failed node is returned to service, its database is resynchronized with the active node; and normal replicated service is returned. The service provider did not initially use the system in a true active/active configuration, in which both nodes are actively processing transactions, because of concerns of fraudulent activity. For instance, two subscribers could use the same voucher simultaneously; and if their requests were routed to different nodes, they could succeed. In a single-node setup, the voucher record is locked by the first attempt; and the second attempt is blocked. Therefore, the company used the PPFE active/active system in a sizzling hot standby configuration. All transactions were handled by one node, the primary node, and were replicated to the other node, the backup node. The backup node had all applications running and the database opened so that it could take over instantly in the event of the failure of the primary node.
2008 Sombers Associates, Inc., and W. H. Highleyman www.availabilitydigest.com

However, the company has since resolved this concern and is now in the process of cutting over to a true active/active operation. To avoid fraudulent use and data collisions, one system will 2 handle odd-numbered vouchers; and the other will handle even-numbered vouchers .

Ancillary Systems
The PPFE feeds several ancillary systems important to the provision of prepaid card services. These systems are run as monolithic systems since their availability is not so critical. Though there is no need to synchronize the PPFE database with the ancillary systems, Shadowbase is used to transfer information between these systems by replicating database transactions from one system to another.
Billing/Administration
Shadowbase

Web Access
Shadowbase

Unix/Oracle Prepaid Front End

Windows/SQL Server

Shadowbase

NonStop server

Stock Card Reporting

Ancillary Systems
During normal operation, the ancillary system support is split between the two nodes in the PPFE. Should one node go down, its ancillary-system links are switched to the surviving node. Upon recovery of the failed node, the links are switched back. Web Access Subscriber web access to prepaid card data is provided by a web server running on a Windows/SQL Server platform. By using his browser, a subscriber can view the available money left in his account. He can add money to a card and can transfer money from one cell number to another. Updates such as these are replicated back to the PPFE database. Bidirectional replication is used by the web server to access data from the PPFE to support a subscribers requests and to replicate updated data back to the PPFE database.

Avoiding collisions by routing specific cards to specific nodes can be easily extended to more than two nodes by assigning more of the digits in the card number to the routing table. For instance, the entire last digit can be used to support 10 active/active nodes. Assuming the cards are randomly distributed, this can aid load-balancing efforts without need for more complex algorithms.
2008 Sombers Associates, Inc., and W. H. Highleyman www.availabilitydigest.com

Billing and Administration The Billing and Administration system is resident on a Unix system that runs under Oracle. Shadowbase is used to replicate the Recharge Logs to this system to charge the users accounts for time and other services purchased and to report on the recharge activity. Stock Card Reporting The Stock Card Reporting System tracks the prepaid vouchers that have been sent to stores for subscriber purchase. It tracks all cards that have been printed since 1996, even those that have been used or that have expired. Currently, there are over two billion cards in the Stock Card Reporting database, of which 50 million or more are active at any one time. The Stock Card Reporting System is implemented on a NonStop server and is fed from the PPFE via Shadowbase unidirectional replication.

The Result
By implementing the critical Prepaid Front End as an active/active system, the company has ensured continuous subscriber account recharging, calling time transfer, and calling time query services to its subscribers. It has also taken advantage of efficiencies by imposing reasonable availability requirements on its ancillary systems. These are all monolithic systems fed via data replication from the central PPFE. NonStop servers are judiciously used for those ancillary systems requiring greater availability. The cellular providers Prepaid Recharge System is an excellent example of a heterogeneous system mixing active/active technologies with high-availability systems, all connected by a common data-replication engine, Shadowbase in this case.

2008 Sombers Associates, Inc., and W. H. Highleyman www.availabilitydigest.com