Jinyun's Notes 🚀

没什么天赋,爱好也不多,但愿坚持做些喜欢的事情

0%

Redis 集群搭建

Redis 是一个 K-V 存储系统,是一个高性能的 K-V 数据库,是目前大规模使用的缓存中间件。单节点的Redis 已经就达到了很高的性能,为了提高可用性我们可以使用 Redis 集群。

Redis Website

集群简介

Redis 集群是一个可以在多个 Redis 节点之间进行数据共享的设施(installation)。

Redis 集群不支持那些需要同时处理多个键的 Redis 命令, 因为执行这些命令需要在多个 Redis 节点之间移动数据, 并且在高负载的情况下, 这些命令将降低 Redis 集群的性能,并导致不可预测的行为。

Redis 集群通过分区(partition)来提供一定程度的可用性(availability):即使集群中有一部分节点失效或者无法进行通讯,集群也可以继续处理命令请求。

Redis 集群提供了以下两个好处:

  • 将数据自动切分(split)到多个节点的能力
  • 当集群中的一部分节点失效或者无法进行通讯时,仍然可以继续处理命令请求的能力

集群说明

集群数据共享

Redis 集群使用数据分片(sharding)而非一致性哈希(consistency hashing)来实现:一个 Redis 集群包含 16384 个哈希槽(hash slot),数据库中的每个键都属于这 16384 个哈希槽的其中一个,集群使用公式 CRC16(key) % 16384 来计算键 key 属于哪个槽,其中 CRC16(key) 语句用于计算键 keyCRC16 校验和。

集群中的每个节点负责处理一部分哈希槽

一个集群可以有三个哈希槽,其中:

  • 节点 A 负责处理 0 号至 5500 号哈希槽
  • 节点 B 负责处理 5501 号至 11000 号哈希槽
  • 节点 C 负责处理 11001 号至 16384 号哈希槽

集群中的主从复制

为了使得集群在一部分节点下线或者无法与集群的大多数(majority)节点进行通讯的情况下,仍然可以正常运作,Redis 集群对节点使用了主从复制功能:集群中的每个节点都有 1 个至 N 个复制品(replica),其中一个复制品为主节点(master),而其余的 N-1 个复制品为从节点(slave

集群 TCP 端口

每个 Redis Cluster 节点都需要 2TCP 连接打开,正常的 Redis TCP 端口被服务于客户端,例如:6379,加上通过增加 10000 到数据端口获取的端口,例如:16379

第二个高端口被用来 Redis Cluster Bus,也就是使用二进制协议的一个节点到节点的通信信道。Redis Cluster Bus 被节点用于故障检测,配置更新,故障转移授权等等。客户端不应该尝试与 Redis Cluster Bus 端口进行通信,而是始终使用正常的 Redis 命令端口,然而要确保在防火墙中打开两个端口,否则 Redis Cluster 节点将无法通信。

命令端口和 Redis Cluster Bus 端口偏移是固定的,始终为 10000

注意,为了使 Redis Cluster 正常工作,你需要为每个节点:

  • 用于客户端进行通信的普通客户端通信端口(通常使用 6379)对所有需要到达集群的客户端,以及所有其它集群节点(使用客户端端口进行密钥迁移)都是开放的
  • Redis Cluster Bus 端口(客户端端口 +10000)必须可从所有其它集群节点访问

如果不打开两个 TCP 端口,集群将无法按预期工作

Redis Cluster Bus 使用不同的二进制协议进行节占到节点的数据交换,这更适合于使用少量的带宽和处理时间交换节点之间的信息

安装依赖

1
2
3
4
5
6
7
8
$ sudo gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
$ sudo curl -sSL https://get.rvm.io | bash -s stable
$ sudo source ~/.rvm/scripts/rvm
$ sudo echo "ruby_url=https://cache.ruby-china.org/pub/ruby" > ~/.rvm/user/db
$ rvm -v
$ sudo rvm install 2.4.1
$ sudo gem update
$ sudo gem install redis

集群配置

安装 Redis

1
2
3
4
5
$ cd ~
$ wget http://download.redis.io/releases/redis-4.0.9.tar.gz
$ tar xzf redis-4.0.9.tar.gz
$ cd redis-4.0.9
$ sudo make

创建配置目录

1
$ sudo mkdir -pv /usr/local/redis-cluster/{bin,etc,log,run,var}

创建配置文件

1
$ sudo cp ~/redis-4.0.9/redis.conf /usr/local/redis-cluster/etc/redis-7001.conf

复制命令文件

1
2
$ cd ~/redis-4.0.9/src/
$ sudo cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-sentinel redis-cli redis-server /usr/local/redis-cluster/bin

修改配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo vim /usr/local/redis-cluster/etc/redis-7001.conf
bind 192.168.33.110
port 7001
daemonize yes

pidfile /user/local/redis-cluster/run/7001.pid
logfile /usr/local/redis-cluster/log/7001.log
dbfilename dump-7001.rdb

dir /usr/local/redis-cluster/var

appendonly yes
appendfilename "appendonly-7001.aof"

cluster-enabled yes
cluster-config-file nodes-7001.conf
cluster-node-timeout 15000

复制配置文件

将复制后的文件中的 7001 换成相应的名称

1
2
3
4
5
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7006.conf

集群部署

启动节点

1
2
3
4
5
6
7
// 启动配置的所有节点
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7001.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7006.conf

检查服务

1
2
3
4
5
6
7
$ ps -ef | grep redis
root 20077 1 0 May18 ? 00:01:04 /usr/local/bin/redis-server 192.168.33.110:7001 [cluster]
root 20079 1 0 May18 ? 00:01:03 /usr/local/bin/redis-server 192.168.33.110:7002 [cluster]
root 20081 1 0 May18 ? 00:01:03 /usr/local/bin/redis-server 192.168.33.110:7003 [cluster]
root 20089 1 0 May18 ? 00:01:03 /usr/local/bin/redis-server 192.168.33.110:7004 [cluster]
root 20091 1 0 May18 ? 00:01:03 /usr/local/bin/redis-server 192.168.33.110:7005 [cluster]
root 20096 1 0 May18 ? 00:01:03 /usr/local/bin/redis-server 192.168.33.110:7006 [cluster]

停止节点

1
$ sudo /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001 shutdown

节点加入集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
$ sudo cp ~/redis-4.0.9/src/redis-trib.rb /usr/local/redis-cluster/bin/redis-trib
$ sudo chmod a+x /usr/local/redis-cluster/bin/redis-trib
$ sudo /usr/local/redis-cluster/bin/redis-trib create --replicas 1 \
> 192.168.33.110:7001 \
> 192.168.33.110:7002 \
> 192.168.33.110:7003 \
> 192.168.33.110:7004 \
> 192.168.33.110:7005 \
> 192.168.33.110:7006 \
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7006 to 127.0.0.1:7002
Adding replica 127.0.0.1:7004 to 127.0.0.1:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
slots:0-5460 (5461 slots) master
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
slots:5461-10922 (5462 slots) master
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
slots:10923-16383 (5461 slots) master
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7001)
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
slots: (0 slots) slave
replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
slots: (0 slots) slave
replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
slots: (0 slots) slave
replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到,创建集群这个过程比较繁琐,可以创建脚本来完成这些操作

  • 启动脚本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ sudo vim /usr/local/redis-cluster/bin/start.sh
#!/usr/bin/env bash

RedisServerPath=/usr/local/redis-cluster/bin
RedisConfPath=/usr/local/redis-cluster/etc

for i in 1 2 3 4 5 6; do
$RedisServerPath/redis-server $RedisConfPath/redis-700$i.conf
done

/usr/local/redis-cluster/bin/redis-trib.rb create --replicas 1 \
192.168.33.110:7001 \
192.168.33.110:7002 \
192.168.33.110:7003 \
192.168.33.110:7004 \
192.168.33.110:7005 \
192.168.33.110:7006 \
$ sudo chmod a+x /usr/local/redis-cluster/bin/start.sh
  • 停止脚本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ sudo vim /usr/local/redis-cluster/bin/stop.sh
#!/usr/bin/env bash

RedisClusterPath=/usr/local/redis-cluster
RedisCliPath=/usr/local/redis-cluster/bin
ip=192.168.33.110

for i in 1 2 3 4 5 6; do
$RedisCliPath/redis-cli -c -h $ip -p 700$i shutdown
done

rm -rf $RedisClusterPath/log/*
rm -rf $RedisClusterPath/run/*
rm -rf $RedisClusterPath/var/*
$ sudo chmod a+x /usr/local/redis-cluster/bin/stop.sh

查看信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> cluster nodes # 查看节点信息
5521b06a46910c873e7ee8a274f2cb9fc41ffed1 192.168.33.110:7003@17003 master - 0 1526681994008 3 connected 10923-16383
8153a1b35e0fab1e48f0171653f6cffd82cbc5a6 192.168.33.110:7004@17004 slave d7e0395152496f41fe73fbbdabc8499c79f6632e 0 1526681996033 4 connected
a2ff29661185f6ad17c99232d0490b3814ba0164 192.168.33.110:7006@17006 slave affc237bf177fd4ac591314736a1358e16545d0c 0 1526681995020 6 connected
d7e0395152496f41fe73fbbdabc8499c79f6632e 192.168.33.110:7002@17002 master - 0 1526681993000 2 connected 5461-10922
8b613e4b5abbdf3c2486dfcb8bc1e9dde1897820 192.168.33.110:7005@17005 slave 5521b06a46910c873e7ee8a274f2cb9fc41ffed1 0 1526681993000 5 connected
affc237bf177fd4ac591314736a1358e16545d0c 192.168.33.110:7001@17001 myself,master - 0 1526681994000 1 connected 0-5460
192.168.33.110:7001> cluster info # 查看集群信息
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:59501
cluster_stats_messages_pong_sent:61755
cluster_stats_messages_sent:121256
cluster_stats_messages_ping_received:61750
cluster_stats_messages_pong_received:59501
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:121256

相关命令

命令说明
CLUSTER INFO打印集群信息
CLUSTER NODES列出集群当前已知的所有节点信息
CLUSTER MEET <ip> <port>ipport 所指定的节点添加到集群当中
CLUSTER FORGET <node_id>从集群中移除 node_id 指定的节点
CLUSTER REPLICATE <node_id>将当前节点设置为 node_id 指定的节点的从节点
CLUSTER SAVECONFIG将节点的配置文件保存到硬盘里面
CLUSTER ADDSLOTS <slot> [slot ...]将一个或多个槽(slot)指派(assign)给当前节点
CLUSTER DELSLOTS <slot> [slot ...]移除一个或多个槽对当前节点的指派
CLUSTER FLUSHSLOTS移除指派给当前节点的所有槽,让当前节点变成一个没有指派任何槽的节点
CLUSTER SETSLOT <slot> NODE <node_id>将槽 slot 指派给 node_id 指定的节点,如果槽已经指派给另一个节点,那么先让另一个节点删除该槽,然后再进行指派
CLUSTER SETSLOT <slot> MIGRATING <node_id>将本节点的槽 slot 迁移到 node_id 指定的节点中
CLUSTER SETSLOT <slot> IMPORTING <node_id>node_id 指定的节点中导入槽 slot 到本节点
CLUSTER SETSLOT <slot> STABLE取消对槽 slot 的导入(import)或者迁移(migrate
CLUSTER KEYSLOT <key>计算键 key 应该被放置在哪个槽上
CLUSTER COUNTKEYSINSLOT <slot>返回槽 slot 目前包含的键值对数量
CLUSTER GETKEYSINSLOT <slot> <count>返回 countslot 槽中的键

实例操作

客户端操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// 从某个节点设置值后,去不同的节点获取设置的值
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> set foo bar
-> Redirected to slot [12182] located at 192.168.33.110:7003
OK
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7002
192.168.33.110:7002> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7003
192.168.33.110:7003> get foo
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7004
192.168.33.110:7004> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7005
192.168.33.110:7005> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7006
192.168.33.110:7006> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"

程序操作

下载安装

1
2
3
4
5
6
$ cd ~
$ mkdir test
$ cd test
$ composer init
$ composer install
$ composer require predis/predis

测试代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<?php

require __DIR__ . '/vendor/autoload.php';

for ($i = 0; $i < 6; $i++) {
$parameters[] = [
'host' => '192.168.33.110',
'port' => '700' . $i,
'password' => null,
'database' => 0,
'timeout' => 5,
'read_write_timeout' => 5,
];
}
$options = ['cluster' => 'redis'];

try {
$cluster = new \Predis\Client($parameters, $options);
} catch (\Predis\PredisException $e) {
echo $e->getMessage();
exit();
}

for ($i = 1; $i < 100; $i++) {
$format = 'No.%s is a RedisCluster test.';
$cluster->set("$i", sprintf($format, $i));
echo $cluster->get("$i") . PHP_EOL;
}

测试结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ php cluster.php
No.1 is a Redis cluster test.
No.2 is a Redis cluster test.
No.3 is a Redis cluster test.
...
No.99 is a Redis cluster test.

$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get 11
"No.11 is a RedisCluster test."
192.168.33.110:7001> get 23
-> Redirected to slot [9671] located at 192.168.33.110:7002
"No.23 is a RedisCluster test."
192.168.33.110:7002> get 33
"No.33 is a RedisCluster test."
192.168.33.110:7002> get 67
"No.67 is a RedisCluster test."
192.168.33.110:7002> get 88
-> Redirected to slot [15207] located at 192.168.33.110:7003
"No.88 is a RedisCluster test."
192.168.33.110:7003> get 99
-> Redirected to slot [6263] located at 192.168.33.110:7002
"No.99 is a RedisCluster test."
192.168.33.110:7002> get 100
-> Redirected to slot [339] located at 192.168.33.110:7001
(nil)
192.168.33.110:7001> get 17
-> Redirected to slot [12304] located at 192.168.33.110:7003
"No.17 is a RedisCluster test."
192.168.33.110:7003> get 42
-> Redirected to slot [8000] located at 192.168.33.110:7002
"No.42 is a RedisCluster test."
192.168.33.110:7002> get 56
-> Redirected to slot [11509] located at 192.168.33.110:7003
"No.56 is a RedisCluster test."
192.168.33.110:7003> get 61
-> Redirected to slot [2369] located at 192.168.33.110:7001
"No.61 is a RedisCluster test."
本笔记是笔者在学习和工作中的一些整理,如对您有用,请鼓励我继续写作