cassandra集群问题之：schema version mismatch detected，cassandraschema

cassandra集群问题之：schema version mismatch detected

问题重现

今天打算在cassandra集群山创建一个新的keyspace的时候报错：

[cassandra@cassandra01]  $ /opt/database/apache-cassandra/bin/cqlsh -u cassandra 10.175.20.102
Password: 
Connected to cassandra-cluster01 at 10.175.20.102:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh> CREATE KEYSPACE and_fleet WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '2'}  AND durable_writes = true;
Warning: schema version mismatch detected; check the schema versions of your nodes in system.local and system.peers.

问题诊断

检查发现虽然爆出警告但keyspace还是创建成功了：

dbuser@cqlsh> desc keyspaces;

and_fleet 

检查集群网络状况：

[cassandra@cassandra01]  $ /opt/database/apache-cassandra/bin/nodetool -u cassandra -pw cassandra -h 10.175.20.102 status
xss =  -ea -javaagent:./../lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6144m -Xmx6144m -Xmn512m -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
DN  10.175.20.122  48.07 GiB  256          ?       49d6e598-df8e-4841-91d7-1333bab94357  RAC1
UN  10.175.20.102  46.48 GiB  256          ?       7b26cff8-a809-483e-8a21-62aa39d3025b  RAC1
UN  10.175.20.128  48.44 GiB  256          ?       1ecebc86-b710-4a89-bca5-ceb20bfa1505  RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
You have new mail in /var/spool/mail/cassandra

可观察到10.175.20.122端集群状态为Down，说明此节点已断开集群。

问题修复

需要重启10.175.20.122端的cassandra：

关闭cassandra：

pgrep -u cassandra  -f cassandra | xargs kill -9

启动cassandra：

/opt/database/apache-cassandra/bin/cassandra

由于需要进行数据同步，花费时间较长

再次查看集群状态：

[cassandra@cassandra01]  $ /opt/database/apache-cassandra/bin/nodetool -u cassandra -pw cassandra -h 10.175.20.102 status
xss =  -ea -javaagent:./../lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6144m -Xmx6144m -Xmn512m -XX:+HeapDumpOnOutOfMemoryError -Xss256k
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns    Host ID                               Rack
UN  10.175.20.122  48.07 GiB  256          ?       49d6e598-df8e-4841-91d7-1333bab94357  RAC1
UN  10.175.20.102  46.48 GiB  256          ?       7b26cff8-a809-483e-8a21-62aa39d3025b  RAC1
UN  10.175.20.128  48.44 GiB  256          ?       1ecebc86-b710-4a89-bca5-ceb20bfa1505  RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

集群状态已修复。