Hi,
I try to setup mysql multi-master. The performance goes extremly down, if I run sysbench against a replicated database. From 1554.35 transactions/s without, to 56.51 transactions/s with replication.
Both server are configured as follow:
HP DL 370, 8x146GB 15K SAS Raid10, 24GB RAM, 2x Intel E5645
OS: Centos 6.2
Mysql: 5.5.20
my.cnf:
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
user=mysql
datadir=/var/lib/mysql
symbolic-links=0
sync_binlog = 1
binlog_cache_size = 1M
max_binlog_size = 10M
slave_compressed_protocol = 1
delay-key-write = ALL
skip-external-locking
skip-name-resolve
skip-external-locking
skip-show-database
safe-user-create = 1
#default_table_type = INNODB
default_storage_engine=InnoDB
transaction_isolation = REPEATABLE-READ
max_allowed_packet = 16M
max_heap_table_size = 64M
tmp_table_size = 64M
key_buffer_size = 384M
sort_buffer_size = 8M
join_buffer_size = 8M
read_buffer_size = 8M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 32M
myisam_sort_buffer_size = 64M
myisam_max_sort_file_size = 1G
myisam_repair_threads = 1
thread_cache_size = 8
table_cache = 512
query_cache_type = 1
query_cache_size = 32M
query_cache_limit = 2M
thread_concurrency = 8
local-infile = 0
long_query_time = 1
slow_query_log
log-short-format
myisam_recover
wait_timeout=68400
interactive_timeout = 68400
old_passwords=1
# innodb settings
innodb_thread_concurrency = 8
innodb_buffer_pool_size = 18G
innodb_additional_mem_pool_size = 20M
innodb_data_home_dir = /var/lib/mysql
innodb_log_group_home_dir = /var/lib/mysql
innodb_data_file_path = ibdata1:1500M;ibdata2:1500M:autoextend
innodb_log_file_size = 768M
innodb_log_buffer_size = 64M
innodb_log_files_in_group = 2
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 60
max_connections = 500
innodb_buffer_pool_instances=8
# master-master setup
auto_increment_increment=2
auto_increment_offset=2
log-bin=mydb2-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
binlog-ignore-db=performance_schema
server-id = 2
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=20000 # 20 second
rpl_semi_sync_slave_enabled=1
[safe_mysqld]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
I run sysbench two times. First against the test db, which is not replicated and then against "lolo" db, which is replicated.
[root@mydb1 ~]# sysbench --db-driver=mysql --test=oltp --oltp-table-size=2000000 --mysql-db=test --mysql-user=root --mysql-password=root123 --max-time=300 --max-requests=0 --num-threads=16 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 15 times)
Done.
OLTP test statistics:
queries performed:
read: 6528508
write: 2331610
other: 932644
total: 9792762
transactions: 466322 (1554.35 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 8860118 (29532.62 per sec.)
other operations: 932644 (3108.70 per sec.)
Test execution summary:
total time: 300.0112s
total number of events: 466322
total time taken by event execution: 4795.7616
per-request statistics:
min: 2.73ms
avg: 10.28ms
max: 108.98ms
approx. 95 percentile: 20.64ms
Threads fairness:
events (avg/stddev): 29145.1250/141.19
execution time (avg/stddev): 299.7351/0.00
[root@mydb1 ~]# sysbench --db-driver=mysql --test=oltp --oltp-table-size=2000000 --mysql-db=lolo --mysql-user=root --mysql-password=root123 --max-time=300 --max-requests=0 --num-threads=16 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 15 times)
Done.
OLTP test statistics:
queries performed:
read: 237398
write: 84785
other: 33914
total: 356097
transactions: 16957 (56.51 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 322183 (1073.60 per sec.)
other operations: 33914 (113.01 per sec.)
Test execution summary:
total time: 300.0948s
total number of events: 16957
total time taken by event execution: 4801.2059
per-request statistics:
min: 22.92ms
avg: 283.14ms
max: 975.20ms
approx. 95 percentile: 425.31ms
Threads fairness:
events (avg/stddev): 1059.8125/4.64
execution time (avg/stddev): 300.0754/0.01
The last one is extremly slow. It doesnt matter, if I have Semi-sync replication on or off. Replication works fine in both directions. I have absolutly no idea, what happen.
System usage at server1 during sysbench run 2:
17:50:00 CPU %user %nice %system %iowait %steal %idle
17:50:01 all 1,83 0,00 0,83 5,90 0,00 91,44
17:50:02 all 1,67 0,00 1,08 5,25 0,00 92,01
17:50:03 all 1,62 0,00 0,85 5,62 0,00 91,91
17:50:04 all 1,55 0,00 0,82 5,64 0,00 91,99
17:50:05 all 1,32 0,00 0,74 5,84 0,00 92,11
17:50:06 all 1,54 0,00 1,03 6,16 0,00 91,27
17:50:07 all 1,62 0,00 0,73 5,42 0,00 92,23
17:50:08 all 1,57 0,00 0,99 5,45 0,00 91,98
17:50:09 all 1,64 0,00 1,07 4,85 0,00 92,44
17:50:10 all 1,66 0,00 0,91 5,64 0,00 91,79
17:50:11 all 1,58 0,00 0,75 5,90 0,00 91,78
17:50:12 all 1,69 0,00 0,76 5,65 0,00 91,90
17:50:13 all 1,28 0,00 0,77 5,79 0,00 92,17
17:50:14 all 1,49 0,00 0,83 5,64 0,00 92,04
17:50:15 all 1,95 0,00 0,80 5,49 0,00 91,76
17:50:16 all 1,40 0,00 0,99 5,50 0,00 92,12
17:50:17 all 2,02 0,00 1,01 4,90 0,00 92,07
17:50:18 all 1,63 0,00 0,77 5,76 0,00 91,83
17:50:19 all 1,86 0,00 0,89 5,25 0,00 92,00
17:50:20 all 1,64 0,00 0,82 5,91 0,00 91,63
17:50:21 all 1,43 0,00 0,71 7,23 0,00 90,63
The %iowait on server1 during sysbench run 1 is roundabout 0,50.
System usage at server2 during sysbench run 2:
17:50:09 CPU %user %nice %system %iowait %steal %idle
17:50:10 all 0,47 0,00 0,20 0,94 0,00 98,39
17:50:11 all 0,38 0,00 0,13 0,25 0,00 99,24
17:50:12 all 0,31 0,00 0,12 0,18 0,00 99,39
17:50:13 all 0,63 0,00 0,25 0,13 0,00 99,00
17:50:14 all 0,19 0,00 0,12 0,06 0,00 99,63
17:50:15 all 0,73 0,00 0,15 0,44 0,00 98,68
17:50:16 all 0,23 0,00 0,12 0,06 0,00 99,60
17:50:17 all 0,40 0,00 0,13 0,27 0,00 99,20
17:50:18 all 0,37 0,00 0,12 0,06 0,00 99,45
17:50:19 all 0,49 0,00 0,24 0,24 0,00 99,03
17:50:20 all 0,38 0,00 0,13 0,82 0,00 98,67
17:50:21 all 0,72 0,00 0,29 0,29 0,00 98,70
17:50:22 all 0,18 0,00 0,06 0,12 0,00 99,65
17:50:23 all 0,65 0,00 0,26 0,13 0,00 98,96
17:50:24 all 0,24 0,00 0,06 0,06 0,00 99,63
17:50:25 all 0,51 0,00 0,25 0,25 0,00 98,99
17:50:26 all 0,18 0,00 0,06 0,18 0,00 99,57
17:50:27 all 0,73 0,00 0,15 0,15 0,00 98,97
17:50:28 all 0,35 0,00 0,12 0,12 0,00 99,41
17:50:29 all 0,38 0,00 0,13 0,25 0,00 99,25
17:50:30 all 0,49 0,00 0,12 0,37 0,00 99,02
Thx for anny hints!
Regards,
Thomas
I try to setup mysql multi-master. The performance goes extremly down, if I run sysbench against a replicated database. From 1554.35 transactions/s without, to 56.51 transactions/s with replication.
Both server are configured as follow:
HP DL 370, 8x146GB 15K SAS Raid10, 24GB RAM, 2x Intel E5645
OS: Centos 6.2
Mysql: 5.5.20
my.cnf:
[client]
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
user=mysql
datadir=/var/lib/mysql
symbolic-links=0
sync_binlog = 1
binlog_cache_size = 1M
max_binlog_size = 10M
slave_compressed_protocol = 1
delay-key-write = ALL
skip-external-locking
skip-name-resolve
skip-external-locking
skip-show-database
safe-user-create = 1
#default_table_type = INNODB
default_storage_engine=InnoDB
transaction_isolation = REPEATABLE-READ
max_allowed_packet = 16M
max_heap_table_size = 64M
tmp_table_size = 64M
key_buffer_size = 384M
sort_buffer_size = 8M
join_buffer_size = 8M
read_buffer_size = 8M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 32M
myisam_sort_buffer_size = 64M
myisam_max_sort_file_size = 1G
myisam_repair_threads = 1
thread_cache_size = 8
table_cache = 512
query_cache_type = 1
query_cache_size = 32M
query_cache_limit = 2M
thread_concurrency = 8
local-infile = 0
long_query_time = 1
slow_query_log
log-short-format
myisam_recover
wait_timeout=68400
interactive_timeout = 68400
old_passwords=1
# innodb settings
innodb_thread_concurrency = 8
innodb_buffer_pool_size = 18G
innodb_additional_mem_pool_size = 20M
innodb_data_home_dir = /var/lib/mysql
innodb_log_group_home_dir = /var/lib/mysql
innodb_data_file_path = ibdata1:1500M;ibdata2:1500M:autoextend
innodb_log_file_size = 768M
innodb_log_buffer_size = 64M
innodb_log_files_in_group = 2
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 60
max_connections = 500
innodb_buffer_pool_instances=8
# master-master setup
auto_increment_increment=2
auto_increment_offset=2
log-bin=mydb2-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
binlog-ignore-db=performance_schema
server-id = 2
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=20000 # 20 second
rpl_semi_sync_slave_enabled=1
[safe_mysqld]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
I run sysbench two times. First against the test db, which is not replicated and then against "lolo" db, which is replicated.
[root@mydb1 ~]# sysbench --db-driver=mysql --test=oltp --oltp-table-size=2000000 --mysql-db=test --mysql-user=root --mysql-password=root123 --max-time=300 --max-requests=0 --num-threads=16 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 15 times)
Done.
OLTP test statistics:
queries performed:
read: 6528508
write: 2331610
other: 932644
total: 9792762
transactions: 466322 (1554.35 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 8860118 (29532.62 per sec.)
other operations: 932644 (3108.70 per sec.)
Test execution summary:
total time: 300.0112s
total number of events: 466322
total time taken by event execution: 4795.7616
per-request statistics:
min: 2.73ms
avg: 10.28ms
max: 108.98ms
approx. 95 percentile: 20.64ms
Threads fairness:
events (avg/stddev): 29145.1250/141.19
execution time (avg/stddev): 299.7351/0.00
[root@mydb1 ~]# sysbench --db-driver=mysql --test=oltp --oltp-table-size=2000000 --mysql-db=lolo --mysql-user=root --mysql-password=root123 --max-time=300 --max-requests=0 --num-threads=16 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 15 times)
Done.
OLTP test statistics:
queries performed:
read: 237398
write: 84785
other: 33914
total: 356097
transactions: 16957 (56.51 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 322183 (1073.60 per sec.)
other operations: 33914 (113.01 per sec.)
Test execution summary:
total time: 300.0948s
total number of events: 16957
total time taken by event execution: 4801.2059
per-request statistics:
min: 22.92ms
avg: 283.14ms
max: 975.20ms
approx. 95 percentile: 425.31ms
Threads fairness:
events (avg/stddev): 1059.8125/4.64
execution time (avg/stddev): 300.0754/0.01
The last one is extremly slow. It doesnt matter, if I have Semi-sync replication on or off. Replication works fine in both directions. I have absolutly no idea, what happen.
System usage at server1 during sysbench run 2:
17:50:00 CPU %user %nice %system %iowait %steal %idle
17:50:01 all 1,83 0,00 0,83 5,90 0,00 91,44
17:50:02 all 1,67 0,00 1,08 5,25 0,00 92,01
17:50:03 all 1,62 0,00 0,85 5,62 0,00 91,91
17:50:04 all 1,55 0,00 0,82 5,64 0,00 91,99
17:50:05 all 1,32 0,00 0,74 5,84 0,00 92,11
17:50:06 all 1,54 0,00 1,03 6,16 0,00 91,27
17:50:07 all 1,62 0,00 0,73 5,42 0,00 92,23
17:50:08 all 1,57 0,00 0,99 5,45 0,00 91,98
17:50:09 all 1,64 0,00 1,07 4,85 0,00 92,44
17:50:10 all 1,66 0,00 0,91 5,64 0,00 91,79
17:50:11 all 1,58 0,00 0,75 5,90 0,00 91,78
17:50:12 all 1,69 0,00 0,76 5,65 0,00 91,90
17:50:13 all 1,28 0,00 0,77 5,79 0,00 92,17
17:50:14 all 1,49 0,00 0,83 5,64 0,00 92,04
17:50:15 all 1,95 0,00 0,80 5,49 0,00 91,76
17:50:16 all 1,40 0,00 0,99 5,50 0,00 92,12
17:50:17 all 2,02 0,00 1,01 4,90 0,00 92,07
17:50:18 all 1,63 0,00 0,77 5,76 0,00 91,83
17:50:19 all 1,86 0,00 0,89 5,25 0,00 92,00
17:50:20 all 1,64 0,00 0,82 5,91 0,00 91,63
17:50:21 all 1,43 0,00 0,71 7,23 0,00 90,63
The %iowait on server1 during sysbench run 1 is roundabout 0,50.
System usage at server2 during sysbench run 2:
17:50:09 CPU %user %nice %system %iowait %steal %idle
17:50:10 all 0,47 0,00 0,20 0,94 0,00 98,39
17:50:11 all 0,38 0,00 0,13 0,25 0,00 99,24
17:50:12 all 0,31 0,00 0,12 0,18 0,00 99,39
17:50:13 all 0,63 0,00 0,25 0,13 0,00 99,00
17:50:14 all 0,19 0,00 0,12 0,06 0,00 99,63
17:50:15 all 0,73 0,00 0,15 0,44 0,00 98,68
17:50:16 all 0,23 0,00 0,12 0,06 0,00 99,60
17:50:17 all 0,40 0,00 0,13 0,27 0,00 99,20
17:50:18 all 0,37 0,00 0,12 0,06 0,00 99,45
17:50:19 all 0,49 0,00 0,24 0,24 0,00 99,03
17:50:20 all 0,38 0,00 0,13 0,82 0,00 98,67
17:50:21 all 0,72 0,00 0,29 0,29 0,00 98,70
17:50:22 all 0,18 0,00 0,06 0,12 0,00 99,65
17:50:23 all 0,65 0,00 0,26 0,13 0,00 98,96
17:50:24 all 0,24 0,00 0,06 0,06 0,00 99,63
17:50:25 all 0,51 0,00 0,25 0,25 0,00 98,99
17:50:26 all 0,18 0,00 0,06 0,18 0,00 99,57
17:50:27 all 0,73 0,00 0,15 0,15 0,00 98,97
17:50:28 all 0,35 0,00 0,12 0,12 0,00 99,41
17:50:29 all 0,38 0,00 0,13 0,25 0,00 99,25
17:50:30 all 0,49 0,00 0,12 0,37 0,00 99,02
Thx for anny hints!
Regards,
Thomas