Жил да был себе один mysql-сервер с версией 5.5.13. В один прекрасный солнечный весенний день возникла необходимость обновить его до версии 5.6.10. Остановил mysql, удалил старые rpm-ки, поставил новые. Но вот после команды "/sbin/service mysql start" ждал меня следующий облом в error.log-е:
130319 11:10:54 mysqld_safe Number of processes running now: 0
130319 11:10:54 mysqld_safe mysqld restarted
2013-03-19 11:10:57 9319 [Note] Plugin 'FEDERATED' is disabled.
2013-03-19 11:10:57 9319 [Note] InnoDB: The InnoDB memory heap is disabled
2013-03-19 11:10:57 9319 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-03-19 11:10:57 9319 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-03-19 11:10:57 9319 [Note] InnoDB: CPU does not support crc32 instructions
2013-03-19 11:10:57 9319 [Note] InnoDB: Using Linux native AIO
2013-03-19 11:10:57 9319 [Note] InnoDB: Initializing buffer pool, size = 150.0G
2013-03-19 11:11:11 9319 [Note] InnoDB: Completed initialization of buffer pool
2013-03-19 11:11:11 9319 [Note] InnoDB: Highest supported file format is Barracuda.
2013-03-19 11:11:14 9319 [Note] InnoDB: Log scan progressed past the checkpoint lsn 810321211776
2013-03-19 11:11:14 9319 [Note] InnoDB: Database was not shutdown normally!
2013-03-19 11:11:14 9319 [Note] InnoDB: Starting crash recovery.
2013-03-19 11:11:14 9319 [Note] InnoDB: Reading tablespace information from the .ibd files...
2013-03-19 11:11:14 9319 [Note] InnoDB: Restoring possible half-written data pages
2013-03-19 11:11:14 9319 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 810321211796
InnoDB: Last MySQL binlog file position 0 864691679, file name ./serv4-bin.000427
2013-03-19 11:11:20 9319 [Note] InnoDB: 128 rollback segment(s) are active.
2013-03-19 11:11:20 9319 [Note] InnoDB: Waiting for purge to start
2013-03-19 11:11:20 9319 [Note] InnoDB: 1.2.10 started; log sequence number 810321211796
2013-03-19 11:11:20 9319 [Note] Recovering after a crash using serv4-bin
2013-03-19 11:11:20 9319 [Note] Starting crash recovery...
2013-03-19 11:11:20 9319 [Note] Crash recovery finished.
2013-03-19 11:11:20 9319 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3309
2013-03-19 11:11:20 9319 [Note] - '0.0.0.0' resolves to '0.0.0.0';
2013-03-19 11:11:20 9319 [Note] Server socket created on IP: '0.0.0.0'.
2013-03-19 11:11:20 9319 [Warning] Info table is not ready to be used. Table 'mysql.slave_master_info' cannot be opened.
2013-03-19 11:11:20 9319 [Warning] Info table is not ready to be used. Table 'mysql.slave_relay_log_info' cannot be opened.
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.events_waits_current is wrong. Expected 19, found 16. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [Warning] Slave SQL: If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.events_waits_history is wrong. Expected 19, found 16. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.events_waits_history_long is wrong. Expected 19, found 16. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_host_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Incorrect definition of table performance_schema.events_waits_summary_by_thread_by_event_name: expected column 'THREAD_ID' at position 0 to have type bigint(20), found type int(11).
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_user_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_waits_summary_by_account_by_event_name' has the wrong structure
11:11:20 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=0
max_threads=10000
thread_count=2
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61607924 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2ad3330edb90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 43cd0930 thread_stack 0x30000
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.file_summary_by_event_name is wrong. Expected 23, found 5. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.file_summary_by_instance is wrong. Expected 25, found 6. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'host_cache' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Incorrect definition of table performance_schema.mutex_instances: expected column 'LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8ff785]
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'objects_summary_global_by_type' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Incorrect definition of table performance_schema.rwlock_instances: expected column 'WRITE_LOCKED_BY_THREAD_ID' at position 2 to have type bigint(20), found type int(11).
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure
/usr/sbin/mysqld(handle_fatal_signal+0x3e8)[0x66e9b8]
/lib64/libpthread.so.0[0x304880eb10]
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'setup_objects' has the wrong structure
/usr/sbin/mysqld[0x694995]
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_table' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'table_lock_waits_summary_by_table' has the wrong structure
/usr/sbin/mysqld(_Z11mysql_grantP3THDPKcR4ListI11st_lex_userEmbb+0x525)[0x6987c5]
2013-03-19 11:11:20 9319 [ERROR] Column count of mysql.threads is wrong. Expected 14, found 3. Created with MySQL 50513, now running 50610. Please use mysql_upgrade to fix this error.
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_current' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_history' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_history_long' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_thread_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_account_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_user_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_summary_by_host_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_stages_summary_global_by_event_name' has the wrong structure
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5a85)[0x6f0545]
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_current' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_history' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_history_long' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_thread_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_account_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_user_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_host_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_global_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'events_statements_summary_by_digest' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'users' has the wrong structure
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x318)[0x6f1858]
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'accounts' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'hosts' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'socket_instances' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'socket_summary_by_instance' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'socket_summary_by_event_name' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'session_connect_attrs' has the wrong structure
2013-03-19 11:11:20 9319 [ERROR] Native table 'performance_schema'.'session_account_connect_attrs' has the wrong structure
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0x60b)[0x89676b]
/usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x74)[0x8907a4]
/usr/sbin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x18a)[0x8d33fa]
/usr/sbin/mysqld[0x8d3f68]
/usr/sbin/mysqld(handle_slave_sql+0xd14)[0x8d5314]
/usr/sbin/mysqld(pfs_spawn_thread+0x13b)[0x9790cb]
/lib64/libpthread.so.0[0x304880673d]
/lib64/libc.so.6(clone+0x6d)[0x30480d3f6d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2ad3330f87b5): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
И так по кругу – mysqld пытается взлететь, у него не получается, он в конвульсиях пишет в лог всё вышепроцитированное, дохнет, mysqld_safe пытается перезапустить его заново... Поскольку основная ругань была на базу performance_schema, то я попробовал запустить mysql вот так:
/usr/sbin/mysqld --defaults-file=/etc/my4.cnf --basedir=/usr --datadir=/var/lib/mysql4 --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql4/error.log --open-files-limit=65536 --pid-file=/var/lib/mysql4/mysql-4.pid --socket=/var/lib/mysql4/mysql.sock --port=3309 --skip-performance-schema --read-only --skip-networking --skip-slave-start
Параметров так много, потому что на сервере крутится несколько экземпляров mysql на разных портах, каждый со своими базами. Ключевым параметром здесь является --skip-performance-schema. После этого mysql таки успешно взлетел и у меня появилась возможность выполнить команду
mysql_upgrade --socket=/var/lib/mysql4/mysql.sock
которая починила все ошибки в таблицах, связанные с изменением их структуры в новой версии. Далее тушим аккуратно этот экземпляр:
mysqladmin shutdown --socket=/var/lib/mysql4/mysql.sock
И запускаем его уже штатным образом:
И наслаждаемся в полной мере новыми фичами версии 5.6 :)