Best Practices for Migrating

MySQL to the Cloud

Juan Pablo Arruti
Percona
Agenda
● IaaS vs DBaaS
● Migrating Data
● Replication between On-Premises and Cloud
● Testing Cloud Environments
● High Availability
● Monitoring
● Backups

2
IaaS vs DBaaS
What is IaaS?
Infrastructure as a Service (IaaS)
● Fundamentals Compute Resources
○ Servers, Storage, Network
● Provisioned and managed over the internet
● Complete control of servers
● AWS Services
○ EC2, EBS, VPC

4
And DBaaS?
Database as a Service (DBaaS)
● Provides database service (Instance or Cluster)
● Targeted to easily
○ Setup, Operate, Scale
● Manages common administration tasks
○ Backups, Patching, Failure Detection, Failover
● No OS access
● No Super privilege
● AWS
○ RDS, DynamoDB, Redshift
5
Pros and Cons of IaaS
Pros
● More control and flexibility
● Wide Instance Types
● Cheaper than RDS

Cons
● More operational work

6
Pros and Cons of DBaaS
Pros
● Easy to Manage
● Less operational tasks

Cons
● Less control and flexibility
● More expensive
● Limited Instance types

7
Which Do You Choose?
IaaS
● Database needs specific tuning or feature
● Available resources for operational tasks

DBaaS
● Need focus on data and code
● Generic setups are okay

8
Migrating Data
Best Practices
● Make it simple
● Migrate to same or higher minor version
● Avoid major version upgrades

1
0
Migrating Data to IaaS
● Similar to on-premises databases
● Use Physical Backups for large databases
○ XtraBackup, Cold Backups

● Logical Backups for small databases

1
1
Migrating Data to DBaaS
● Logical Backups
○ Access through MySQL Client
○ mysqldump, mysqlpump, mydumper

● Physical Backups are possible

○ XtraBackup*

Only available for AWS RDS

1
2
Migrating Data to DBaaS
Available MySQL Client Tools
● mysqldump
○ Most adopted tool
○ Single-threaded
● mysqlpump
○ Introduced in MySQL 5.7
○ Parallel backups
○ Restores are Single-threaded
● mydumper/myloader
○ Parallel backups and restores
1
3
Migrating Data to DBaaS
Best Practices for MySQL Clients
● Export all objects first
○ --no-data --routines --events --triggers
● Then export only data
○ --no-create-info --no-create-db
○ --routines=no --events=no --triggers=no
● Enable log_bin_trust_function_creators if log_bin=1
● Change object definer
○ DEFINER=`user`@`host`
● Force load and check all errors
○ --force
1
4
Migrating Data to RDS
Best Practices for MySQL Clients
● Increase max_allowed_packet (Default 4 MB)

● time_zone can be modified in parameter group (Default UTC)

○ RDS uses mysql schema Time Zone Tables
○ Recommended
■ Set session time_zone to match source database

1
5
Migrating Data to RDS
Speeding Up Logical Restore
● EC2 and RDS in same AZ
● Disable Multi-AZ
● Increase IOPS
● Modify Default Settings
○ Relax Durability
■ sync_binlog != 1
■ innodb_flush_log_at_trx_commit != 1
○ Tune InnoDB
■ Increase innodb_log_file_size (Default 128 MB)
■ Increase innodb_buffer_pool_size (Default
DBInstanceClassMemory*3/4)
1
6
Restore Amazon RDS from Xtrabackup
● Overview
○ Take backup from database
○ Upload into S3 bucket
○ Create new instance from the backup
■ Amazon MySQL RDS
■ Amazon Aurora MySQL

1
7
Restore Amazon RDS from Xtrabackup
● Limitations
○ Supported MySQL 5.6 and 5.7
○ Source/Target major versions must match
■ Target minor version must be higher
○ Source tables defined within default datadir
○ 6 TB database size limit
○ Source database can't be encrypted
○ User accounts, functions, stored procedures and
time zone info are not imported automatically

1
8
 Replication Between
On-Premises and Cloud
Replicating to the Cloud
● IaaS
○ Same as replication in on-premises
● DBaaS
○ Implementation and its limitations depends on the
cloud provider

2
0
Replicating to the Cloud
Best Practices

● If latency is high
○ Use compression for Master/Slave protocol
■ slave_compressed_protocol=1
○ Monitor replication lag with pt-heartbeat

● Ensure tables have Primary Key

○ binlog_format = ROW

2
1
External Master on AWS RDS
● Easy to set
● NO binlog_format constraints (MIXED, ROW,
STATEMENT)
○ Recommended ROW to avoid time_zone mismatch
● Log File Position or GTID based
● No filtered Replication is allowed
● Replication administration using procedures
○ mysql.rds_set_external_master
○ mysql.rds_start_replication
...

2
2
Testing Cloud Environments
Why Benchmark Cloud?
● Cloud resources may not map directly
● Validate if cloud instance is able to handle traffic
● Choose between IaaS and DBaaS
● Available tools
○ sysbench, pt-upgrade, Query Playback, ProxySQL

2
4
Query Playback
Key aspects
● Percona Labs GitHub repository
○ No active development
● Executes Queries in logs
○ Slow Query log, General log
● Compares execution results with Log
● Servers data should be consistent
● NO read-only option
● Multi-threaded
○ Queries executed at arrival time
2
5
Query Playback
Example Report
SELECT c FROM sbtest37 WHERE id=505; -->
thread 67 slower query was run in 86 microseconds instead of 34

Detailed Report
----------------
SELECTs : 1858522 queries (19297 faster, 1839225 slower)
...

Report
------
Executed 2161872 queries
Spent 00:09:08.631886 executing queries versus an expected 00:04:43.697328 time.
23610 queries were quicker than expected, 2138262 were slower
A total of 0 queries had errors.
Expected 40606531 rows, got 40606533 (a difference of 2)
Number of queries where number of rows differed: 2.

Average of 113782.74 queries per connection (19 connections).

2
6
ProxySQL
What is ProxySQL?
● GPL High Performance MySQL Proxy
● MySQL Protocol Aware
○ Clients connect to ProxySQL
○ Requests are evaluated
○ Actions are performed
■ Routing, Re-write, Mirroring

2
7
ProxySQL Query Mirroring
How does it work?
● Each client executes a query
● ProxySQL receives each query
● Query Processor identifies if the query is Mirrored
● Associates the Query to a Thread Pool
● Executes each Query in the Pool

2
8
ProxySQL Query Mirroring

2
9
ProxySQL Query Mirroring
Limitations
● May execute Queries in different order
● Some Queries may not be executed
● Data consistency is not guaranteed
● Adds load to ProxySQL process
● Prepared statements are not supported
● No report is provided

3
0
Query Mirroring Example

3
1
Query Mirroring Example
mysql> select * from stats_mysql_query_digest where digest_text like 'SELECT c FROM sbtest1 %' ORDER BY
hostgroup \G
*************************** 1. row ***************************
hostgroup: 1
schemaname: sbtest
username: jptest
digest: 0x290B92FD743826DA
digest_text: SELECT c FROM sbtest1 WHERE id BETWEEN ? AND ?
count_star: 48864
first_seen: 1558761934
last_seen: 1558765725
sum_time: 14788745778
min_time: 2877
max_time: 3733095
*************************** 2. row ***************************
hostgroup: 2
schemaname: sbtest
username: jptest
digest: 0x290B92FD743826DA
digest_text: SELECT c FROM sbtest1 WHERE id BETWEEN ? AND ?
count_star: 48832
first_seen: 1558761936
last_seen: 1558765725
sum_time: 16477562501
min_time: 2786
max_time: 4651554
3
2
High Availability
Load Balancers
Why are they useful?
● Phased migration approach
● Routing read-only traffic
● Monitor the new environment
● Adjust it if necessary
● Confirm the environment is stable
● Minimize downtime
● Avoid modifying App connection string
● Available Load Balancers
○ Cloud Load Balancing
○ ProxySQL
3
4
Cloud Load Balancing
● TCP Load Balancer
● Distributes traffic
● AWS Elastic Load Balancer
○ RDS not supported
○ Routes traffic only across Availability Zones
○ Custom MySQL health checks can be used

3
5
AWS ELB - Phased Migration

3
6
ProxySQL
● SQL Connection Endpoint
● Routes traffic to MySQL databases
● Splits read/write traffic
○ Based on read_only value
● Supports DBaaS

3
7
ProxySQL - Query Routing

3
8
DBaaS High Availability
How is it achieved?
● Data redundancy
● Automatic failover

How is it implemented?
● Each cloud vendor implements it differently

3
9
AWS RDS - Multi-AZ
Key aspects
● Synchronous Standby Replica - DRBD
● Block replicated to Different Availability Zone than
Primary
● Secondary is used for backups
● Failover takes place by internal DNS change

Limitations
● Reads on Secondary are not possible
● DML Overhead
4
0
AWS RDS Read-Replicas
Key aspects
● MySQL asynchronous replication
● Scale-out Reads
● Promote to stand-alone database
● Within Same AZ, Cross AZ, Cross Region
● Easy to implement

4
1
Monitoring
Monitoring MySQL in the Cloud
Best practices
● Establish a baseline
● Measure workload under different conditions
● Compare cloud instances and on-premises

Available tools
● Cloud Monitoring System
● Percona Monitoring and Management

4
3
AWS CloudWatch
What is CloudWatch?
● Monitors AWS resources
● Metrics collected automatically
● Custom dashboards
● Create alarms and send notifications based on metrics
● Launch additional resources or stop them
● High-level Database metrics
○ BinLogDiskUsage
○ DatabaseConnections
○ ReplicaLag
4
4
Percona Monitoring and Management

● Open source monitoring platform

● MySQL, MongoDB, PostgreSQL and ProxySQL
● Detailed metrics for DB Servers
● Query Analytics shows current queries and stats
● Supports AWS MySQL RDS and Aurora

4
5
PMM Architecture

4
6
Monitoring RDS in PMM
Recommendations
● PMM Server in same AZ than RDS
● Don't use T2 instances
● Use Elastic IP address
● Don't use RDS admin user

4
7
Monitoring RDS in PMM
● Enable performance_schema for Query Analytics

● Enable enhanced monitoring

○ OS level metrics

4
8
PMM MySQL Overview Dashboard

4
9
PMM Query Analytics

5
0
Backups
Cloud Storage
Key aspects
● Object storage - AWS S3
● Designed for durability - 99.999999999%
● Highly available
● Scalable
● Secure
● Range of storage classes
○ Standard, Infrequent Access, Glacier
○ Lifecycle policies
● Off-site storage
5
2
MySQL Backups to AWS S3
mysqldump - example
● Copy dump file with aws cli
shell> mysqldump --all-databases > mysql.dump.sql

shell> gzip mysql.dump.sql

shell> aws s3 cp mysql.dump.sql.gz s3://mysqlbucketprod/mysql.dump.sql.gz

upload: ./mysql.dump.sql.gz to s3://mysqlbucketprod/mysql.dump.sql.gz

or

shell> mysqldump --all-databases | gzip | aws s3 cp - \

s3://mysqlbucketprod/mysql.dump.sql.gz

5
3
MySQL Backups to AWS S3
XtraBackup - example
● Stream files with xbstream
● Upload stream with xbcloud
shell> xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp \
--target-dir=/tmp | \
xbcloud put --storage=s3 \
--s3-endpoint='s3-us-west-2.amazonaws.com' \
--s3-access-key='AKIAJ6HPUNXNZPTL2AAA' \
--s3-secret-key='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' \
--s3-bucket='mysqlbucketprod' \
--parallel=10 \
$(date +"%Y%m%d%H%M%S")-full_backup

5
4
RDS Backups
Automated backups
● Defined backup window
● Stored in S3
● InnoDB tables
● Elevated latency and IO freeze - except for Multi-AZ
● Point in time recovery
● Retention period = 1 - 35 days

Database snapshots
● Manually initiated
● No point in time recovery 5
5
Questions
Thank You to Our Sponsors
Rate My Session

58
Thank you!