MySQL slow performance if use replication (1 reply)

MySQL 5.1.53-community
Configure master-master.
Without replication, benchmark sysbench report about ~10000 query per sec:
$ sysbench $hostinfo --mysql-table-engine=innodb --test=oltp cleanup
$ sysbench $hostinfo --mysql-table-engine=innodb --test=oltp --oltp-table-size=1000000 prepare
$ sysbench $hostinfo --mysql-table-engine=innodb --num-threads=5 --max-requests=1000000 --test=oltp --oltp-table-size=1000000 --oltp-nontrx-mode=insert run

---- Config BEGIN -----:
[mysqld]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
user = mysql
symbolic-links = 0
skip-name-resolve

# Replica:
server-id = 1
log-bin = mysql-bin
sync_binlog = 1
expire_logs_days = 3
binlog-format = MIXED
binlog-do-db = cltest
replicate-do-db = cltest
#auto_increment_increment = 10
#auto_increment_offset = 1
#log-slave-updates

# Per connection:
join_buffer_size = 512k
read_rnd_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 2M

# Global:
key_buffer_size = 512k
concurrent_insert = 1
max_connections = 500
#max_heap_table_size = 8192M
query_cache_size = 10M
thread_cache_size = 200
thread_concurrency = 10
tmp_table_size = 512M
table_open_cache = 512
table_definition_cache = 512
table_cache = 512
#long_query_time = 2
#slow_query_log = 1
#slow_query_log_file = /var/log/mysql-slow.log
#log-queries-not-using-indexes

# InnoDB:
innodb_log_file_size = 256M
innodb_log_buffer_size = 100M
innodb_buffer_pool_size = 4096M
innodb_thread_concurrency = 10
innodb_autoinc_lock_mode = 1
innodb_flush_log_at_trx_commit = 1
#innodb_flush_method = O_DSYNC
#innodb_concurrency_tickets = 500
#innodb_commit_concurrency = 10

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

---- Config END-----:

Enable replication on next commands:
$ mysql
SLAVE STOP;
CHANGE MASTER TO
MASTER_HOST='192.168.1.11',
MASTER_USER='repluser',
MASTER_PASSWORD='pass',
MASTER_PORT=3306,
MASTER_LOG_FILE='mysql-bin.000001',
MASTER_LOG_POS=106,
MASTER_CONNECT_RETRY=10;
SLAVE START;

----

On second server, changes:
server_id
auto_increment_increment = 10
auto_increment_offset = 2
MASTER_HOST='192.168.1.10'

----

sysbench show 380-400 query per sec. If enable one node - speed also be slowly.

sql-bench also demonstrate degradation of performance if replica enabled:

----

Benchmark DBD suite: 2.15
Arguments: --small-test --small-tables
Limits from: mysql
Server version: MySQL 5.1.53 community
Optimization: None

Operation seconds usr sys cpu tests
TOTALS 54.00 3.68 1.15 4.83 78237 (w/o replica)
TOTALS 767.00 4.40 1.50 5.90 78237 (w replica)


alter-table: Total time: 12 wallclock secs ( 0.02 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.03 CPU)
ATIS: Total time: 5 wallclock secs ( 0.38 usr 0.14 sys + 0.00 cusr 0.00 csys = 0.52 CPU)
big-tables: Total time: 1 wallclock secs ( 0.07 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.08 CPU)
connect: Total time: 3 wallclock secs ( 0.27 usr 0.13 sys + 0.00 cusr 0.00 csys = 0.40 CPU)
create: Total time: 9 wallclock secs ( 0.02 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.03 CPU)
insert: Total time: 5 wallclock secs ( 0.69 usr 0.18 sys + 0.00 cusr 0.00 csys = 0.87 CPU)
select: Total time: 7 wallclock secs ( 1.27 usr 0.23 sys + 0.00 cusr 0.00 csys = 1.50 CPU)
transactions: Test skipped because the database doesn't support transactions
wisconsin: Total time: 14 wallclock secs ( 1.01 usr 0.46 sys + 0.00 cusr 0.00 csys = 1.47 CPU)

All 9 test executed successfully

Totals per operation:
Operation seconds usr sys cpu tests
alter_table_add 7.00 0.01 0.00 0.01 92
alter_table_drop 4.00 0.01 0.00 0.01 46
connect 0.00 0.02 0.01 0.03 100
connect+select_1_row 0.00 0.04 0.01 0.05 100
connect+select_simple 0.00 0.04 0.01 0.05 100
count 0.00 0.01 0.00 0.01 100
count_distinct 0.00 0.02 0.00 0.02 100
count_distinct_2 0.00 0.00 0.00 0.00 100
count_distinct_big 0.00 0.01 0.00 0.01 30
count_distinct_group 0.00 0.01 0.00 0.01 100
count_distinct_group_on_key 0.00 0.01 0.00 0.01 100
count_distinct_group_on_key_parts 0.00 0.01 0.01 0.02 100
count_distinct_key_prefix 0.00 0.01 0.01 0.02 100
count_group_on_key_parts 0.00 0.01 0.00 0.01 100
count_on_key 3.00 0.41 0.08 0.49 5100
create+drop 1.00 0.00 0.00 0.00 10
create_MANY_tables 1.00 0.00 0.00 0.00 10
create_index 1.00 0.00 0.00 0.00 8
create_key+drop 7.00 0.02 0.01 0.03 100
create_table 2.00 0.00 0.00 0.00 31
delete_all_many_keys 0.00 0.01 0.00 0.01 1
delete_big 0.00 0.00 0.00 0.00 1
delete_big_many_keys 0.00 0.01 0.00 0.01 128
delete_key 0.00 0.00 0.00 0.00 100
delete_range 0.00 0.00 0.00 0.00 12
drop_index 0.00 0.00 0.00 0.00 8
drop_table 0.00 0.00 0.00 0.00 28
drop_table_when_MANY_tables 0.00 0.00 0.00 0.00 10
insert 18.00 0.79 0.61 1.40 44768
insert_duplicates 0.00 0.02 0.02 0.04 1000
insert_key 0.00 0.01 0.00 0.01 100
insert_many_fields 0.00 0.02 0.01 0.03 200
insert_select_1_key 0.00 0.00 0.00 0.00 1
insert_select_2_keys 0.00 0.00 0.00 0.00 1
min_max 0.00 0.01 0.00 0.01 60
min_max_on_key 3.00 0.59 0.10 0.69 7300
multiple_value_insert 0.00 0.00 0.00 0.00 1000
once_prepared_select 0.00 0.02 0.02 0.04 1000
order_by_big 0.00 0.03 0.00 0.03 10
order_by_big_key 0.00 0.03 0.00 0.03 10
order_by_big_key2 0.00 0.03 0.00 0.03 10
order_by_big_key_desc 0.00 0.04 0.00 0.04 10
order_by_big_key_diff 0.00 0.03 0.00 0.03 10
order_by_big_key_prefix 0.00 0.04 0.00 0.04 10
order_by_key2_diff 0.00 0.00 0.00 0.00 10
order_by_key_prefix 0.00 0.00 0.00 0.00 10
order_by_range 0.00 0.00 0.00 0.00 10
outer_join 0.00 0.00 0.00 0.00 10
outer_join_found 0.00 0.01 0.00 0.01 10
outer_join_not_found 0.00 0.00 0.00 0.00 10
outer_join_on_key 0.00 0.00 0.00 0.00 10
prepared_select 1.00 0.07 0.00 0.07 1000
select_1_row 0.00 0.03 0.01 0.04 1000
select_1_row_cache 1.00 0.02 0.01 0.03 1000
select_2_rows 0.00 0.03 0.02 0.05 1000
select_big 0.00 0.04 0.00 0.04 17
select_big_str 0.00 0.04 0.03 0.07 100
select_cache 0.00 0.09 0.01 0.10 1000
select_cache2 1.00 0.07 0.02 0.09 1000
select_column+column 0.00 0.02 0.01 0.03 1000
select_diff_key 0.00 0.01 0.00 0.01 10
select_distinct 0.00 0.04 0.00 0.04 80
select_group 0.00 0.04 0.01 0.05 391
select_group_when_MANY_tables 0.00 0.00 0.00 0.00 10
select_join 0.00 0.01 0.00 0.01 10
select_key 0.00 0.00 0.00 0.00 20
select_key2 0.00 0.00 0.00 0.00 20
select_key2_return_key 0.00 0.00 0.00 0.00 20
select_key2_return_prim 0.00 0.00 0.00 0.00 20
select_key_prefix 0.00 0.01 0.00 0.01 20
select_key_prefix_join 0.00 0.06 0.00 0.06 10
select_key_return_key 0.00 0.00 0.01 0.01 20
select_many_fields 1.00 0.05 0.00 0.05 200
select_range 0.00 0.03 0.00 0.03 41
select_range_key2 1.00 0.03 0.00 0.03 505
select_range_prefix 0.00 0.03 0.00 0.03 505
select_simple 0.00 0.02 0.01 0.03 1000
select_simple_cache 1.00 0.01 0.01 0.02 1000
select_simple_join 0.00 0.01 0.00 0.01 50
update_big 0.00 0.00 0.00 0.00 10
update_of_key 0.00 0.01 0.01 0.02 500
update_of_key_big 0.00 0.00 0.00 0.00 13
update_of_primary_key_many_keys 0.00 0.00 0.00 0.00 256
update_with_key 1.00 0.04 0.04 0.08 3000
update_with_key_prefix 0.00 0.03 0.02 0.05 1000
wisc_benchmark 0.00 0.51 0.03 0.54 34
TOTALS 54.00 3.68 1.15 4.83 78237

AND

alter-table: Total time: 16 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
ATIS: Total time: 133 wallclock secs ( 0.53 usr 0.22 sys + 0.00 cusr 0.00 csys = 0.75 CPU)
big-tables: Total time: 5 wallclock secs ( 0.08 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.09 CPU)
connect: Total time: 3 wallclock secs ( 0.27 usr 0.15 sys + 0.00 cusr 0.00 csys = 0.42 CPU)
create: Total time: 14 wallclock secs ( 0.02 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.03 CPU)
insert: Total time: 124 wallclock secs ( 0.79 usr 0.27 sys + 0.00 cusr 0.00 csys = 1.06 CPU)
select: Total time: 21 wallclock secs ( 1.37 usr 0.23 sys + 0.00 cusr 0.00 csys = 1.60 CPU)
transactions: Test skipped because the database doesn't support transactions
wisconsin: Total time: 467 wallclock secs ( 1.40 usr 0.64 sys + 0.00 cusr 0.00 csys = 2.04 CPU)

All 9 test executed successfully

Totals per operation:
Operation seconds usr sys cpu tests
alter_table_add 8.00 0.00 0.00 0.00 92
alter_table_drop 5.00 0.00 0.00 0.00 46
connect 0.00 0.03 0.00 0.03 100
connect+select_1_row 0.00 0.04 0.01 0.05 100
connect+select_simple 0.00 0.04 0.02 0.06 100
count 0.00 0.01 0.00 0.01 100
count_distinct 0.00 0.01 0.00 0.01 100
count_distinct_2 0.00 0.01 0.00 0.01 100
count_distinct_big 0.00 0.02 0.00 0.02 30
count_distinct_group 0.00 0.01 0.00 0.01 100
count_distinct_group_on_key 0.00 0.01 0.00 0.01 100
count_distinct_group_on_key_parts 0.00 0.01 0.00 0.01 100
count_distinct_key_prefix 0.00 0.01 0.01 0.02 100
count_group_on_key_parts 0.00 0.02 0.00 0.02 100
count_on_key 3.00 0.44 0.08 0.52 5100
create+drop 1.00 0.00 0.00 0.00 10
create_MANY_tables 1.00 0.00 0.00 0.00 10
create_index 0.00 0.00 0.00 0.00 8
create_key+drop 11.00 0.02 0.01 0.03 100
create_table 2.00 0.00 0.00 0.00 31
delete_all_many_keys 2.00 0.01 0.00 0.01 1
delete_big 0.00 0.00 0.00 0.00 1
delete_big_many_keys 2.00 0.00 0.00 0.00 128
delete_key 1.00 0.00 0.00 0.00 100
delete_range 1.00 0.00 0.00 0.00 12
drop_index 1.00 0.01 0.00 0.01 8
drop_table 0.00 0.00 0.00 0.00 28
drop_table_when_MANY_tables 1.00 0.00 0.00 0.00 10
insert 650.00 1.35 0.91 2.26 44768
insert_duplicates 1.00 0.03 0.01 0.04 1000
insert_key 1.00 0.01 0.00 0.01 100
insert_many_fields 4.00 0.02 0.01 0.03 200
insert_select_1_key 0.00 0.00 0.00 0.00 1
insert_select_2_keys 0.00 0.00 0.00 0.00 1
min_max 0.00 0.01 0.00 0.01 60
min_max_on_key 4.00 0.62 0.09 0.71 7300
multiple_value_insert 0.00 0.00 0.00 0.00 1000
once_prepared_select 1.00 0.03 0.01 0.04 1000
order_by_big 0.00 0.04 0.00 0.04 10
order_by_big_key 0.00 0.04 0.00 0.04 10
order_by_big_key2 0.00 0.04 0.00 0.04 10
order_by_big_key_desc 0.00 0.03 0.00 0.03 10
order_by_big_key_diff 0.00 0.02 0.01 0.03 10
order_by_big_key_prefix 0.00 0.03 0.00 0.03 10
order_by_key2_diff 0.00 0.00 0.00 0.00 10
order_by_key_prefix 0.00 0.00 0.00 0.00 10
order_by_range 0.00 0.00 0.00 0.00 10
outer_join 0.00 0.00 0.00 0.00 10
outer_join_found 0.00 0.00 0.00 0.00 10
outer_join_not_found 0.00 0.01 0.00 0.01 10
outer_join_on_key 0.00 0.00 0.00 0.00 10
prepared_select 0.00 0.06 0.02 0.08 1000
select_1_row 0.00 0.02 0.02 0.04 1000
select_1_row_cache 1.00 0.02 0.01 0.03 1000
select_2_rows 0.00 0.02 0.02 0.04 1000
select_big 0.00 0.03 0.00 0.03 17
select_big_str 0.00 0.03 0.04 0.07 100
select_cache 0.00 0.09 0.02 0.11 1000
select_cache2 1.00 0.09 0.02 0.11 1000
select_column+column 0.00 0.03 0.01 0.04 1000
select_diff_key 0.00 0.00 0.00 0.00 10
select_distinct 0.00 0.03 0.00 0.03 80
select_group 0.00 0.04 0.01 0.05 391
select_group_when_MANY_tables 0.00 0.00 0.00 0.00 10
select_join 0.00 0.01 0.00 0.01 10
select_key 0.00 0.01 0.00 0.01 20
select_key2 0.00 0.00 0.01 0.01 20
select_key2_return_key 0.00 0.00 0.00 0.00 20
select_key2_return_prim 0.00 0.01 0.00 0.01 20
select_key_prefix 0.00 0.00 0.00 0.00 20
select_key_prefix_join 1.00 0.06 0.01 0.07 10
select_key_return_key 0.00 0.00 0.00 0.00 20
select_many_fields 0.00 0.06 0.00 0.06 200
select_range 0.00 0.03 0.00 0.03 41
select_range_key2 0.00 0.03 0.00 0.03 505
select_range_prefix 0.00 0.04 0.00 0.04 505
select_simple 0.00 0.02 0.01 0.03 1000
select_simple_cache 1.00 0.02 0.01 0.03 1000
select_simple_join 0.00 0.00 0.00 0.00 50
update_big 0.00 0.00 0.00 0.00 10
update_of_key 7.00 0.01 0.01 0.02 500
update_of_key_big 0.00 0.00 0.00 0.00 13
update_of_primary_key_many_keys 3.00 0.00 0.01 0.01 256
update_with_key 40.00 0.07 0.07 0.14 3000
update_with_key_prefix 12.00 0.05 0.02 0.07 1000
wisc_benchmark 1.00 0.54 0.01 0.55 34
TOTALS 767.00 4.40 1.50 5.90 78237