Redis 集群搭建

作者收录于 Redis

2020-01-16 约 3065 字预计阅读 7 分钟

Redis 是一个 K-V 存储系统，是一个高性能的 K-V 数据库，是目前大规模使用的缓存中间件。单节点的Redis 已经就达到了很高的性能，为了提高可用性我们可以使用 Redis 集群。

Redis Website

集群简介

Redis 集群是一个可以在多个 Redis 节点之间进行数据共享的设施（installation）。

Redis 集群不支持那些需要同时处理多个键的 Redis 命令，因为执行这些命令需要在多个 Redis 节点之间移动数据，并且在高负载的情况下，这些命令将降低 Redis 集群的性能，并导致不可预测的行为。

Redis 集群通过分区（partition）来提供一定程度的可用性（availability）：即使集群中有一部分节点失效或者无法进行通讯，集群也可以继续处理命令请求。

Redis 集群提供了以下两个好处：

将数据自动切分（split）到多个节点的能力
当集群中的一部分节点失效或者无法进行通讯时，仍然可以继续处理命令请求的能力

集群说明

集群数据共享

Redis 集群使用数据分片（sharding）而非一致性哈希（consistency hashing）来实现：一个 Redis 集群包含 16384 个哈希槽（hash slot），数据库中的每个键都属于这 16384 个哈希槽的其中一个，集群使用公式 CRC16(key) % 16384 来计算键 key 属于哪个槽，其中 CRC16(key) 语句用于计算键 key 的 CRC16 校验和。

集群中的每个节点负责处理一部分哈希槽

一个集群可以有三个哈希槽，其中：

节点 A 负责处理 0 号至 5500 号哈希槽
节点 B 负责处理 5501 号至 11000 号哈希槽
节点 C 负责处理 11001 号至 16384 号哈希槽

集群中的主从复制

为了使得集群在一部分节点下线或者无法与集群的大多数（majority）节点进行通讯的情况下，仍然可以正常运作，Redis 集群对节点使用了主从复制功能：集群中的每个节点都有 1 个至 N 个复制品（replica），其中一个复制品为主节点（master），而其余的 N-1 个复制品为从节点（slave）

集群 `TCP` 端口

每个 Redis Cluster 节点都需要 2 个 TCP 连接打开，正常的 Redis TCP 端口被服务于客户端，例如：6379，加上通过增加 10000 到数据端口获取的端口，例如：16379。

第二个高端口被用来 Redis Cluster Bus，也就是使用二进制协议的一个节点到节点的通信信道。Redis Cluster Bus 被节点用于故障检测，配置更新，故障转移授权等等。客户端不应该尝试与 Redis Cluster Bus 端口进行通信，而是始终使用正常的 Redis 命令端口，然而要确保在防火墙中打开两个端口，否则 Redis Cluster 节点将无法通信。

命令端口和 Redis Cluster Bus 端口偏移是固定的，始终为 10000。

注意，为了使 Redis Cluster 正常工作，你需要为每个节点：

用于客户端进行通信的普通客户端通信端口（通常使用 6379）对所有需要到达集群的客户端，以及所有其它集群节点（使用客户端端口进行密钥迁移）都是开放的
Redis Cluster Bus 端口（客户端端口 +10000）必须可从所有其它集群节点访问

如果不打开两个 TCP 端口，集群将无法按预期工作

Redis Cluster Bus 使用不同的二进制协议进行节占到节点的数据交换，这更适合于使用少量的带宽和处理时间交换节点之间的信息

安装依赖

1
2
3
4
5
6
7
8
$ sudo gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
$ sudo curl -sSL https://get.rvm.io | bash -s stable
$ sudo source ~/.rvm/scripts/rvm
$ sudo echo "ruby_url=https://cache.ruby-china.org/pub/ruby" > ~/.rvm/user/db
$ rvm -v
$ sudo rvm install 2.4.1
$ sudo gem update
$ sudo gem install redis

集群配置

安装 Redis

1
2
3
4
5
$ cd ~
$ wget http://download.redis.io/releases/redis-4.0.9.tar.gz
$ tar xzf redis-4.0.9.tar.gz
$ cd redis-4.0.9
$ sudo make

创建配置目录

1
$ sudo mkdir -pv /usr/local/redis-cluster/{bin,etc,log,run,var}

创建配置文件

1
$ sudo cp ~/redis-4.0.9/redis.conf /usr/local/redis-cluster/etc/redis-7001.conf

复制命令文件

1
2
$ cd ~/redis-4.0.9/src/
$ sudo cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-sentinel redis-cli redis-server /usr/local/redis-cluster/bin

修改配置文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ sudo vim /usr/local/redis-cluster/etc/redis-7001.conf
bind 192.168.33.110
port 7001
daemonize yes

pidfile /user/local/redis-cluster/run/7001.pid
logfile /usr/local/redis-cluster/log/7001.log
dbfilename dump-7001.rdb

dir /usr/local/redis-cluster/var

appendonly yes
appendfilename "appendonly-7001.aof"

cluster-enabled yes
cluster-config-file nodes-7001.conf
cluster-node-timeout 15000

复制配置文件

将复制后的文件中的 7001 换成相应的名称

1
2
3
4
5
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo cp /usr/local/redis-cluster/etc/redis-7001.conf /usr/local/redis-cluster/etc/redis-7006.conf

集群部署

启动节点

1
2
3
4
5
6
7
// 启动配置的所有节点
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7001.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7002.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7003.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7004.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7005.conf
$ sudo /usr/local/redis-cluster/bin/redis-cli /usr/local/redis-cluster/etc/redis-7006.conf

检查服务

1
2
3
4
5
6
7
$ ps -ef | grep redis
root     20077     1  0 May18 ?        00:01:04 /usr/local/bin/redis-server 192.168.33.110:7001 [cluster]
root     20079     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7002 [cluster]
root     20081     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7003 [cluster]
root     20089     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7004 [cluster]
root     20091     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7005 [cluster]
root     20096     1  0 May18 ?        00:01:03 /usr/local/bin/redis-server 192.168.33.110:7006 [cluster]

停止节点

1
$ sudo /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001 shutdown

节点加入集群

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
$ sudo cp ~/redis-4.0.9/src/redis-trib.rb /usr/local/redis-cluster/bin/redis-trib
$ sudo chmod a+x /usr/local/redis-cluster/bin/redis-trib
$ sudo /usr/local/redis-cluster/bin/redis-trib create --replicas 1 \
> 192.168.33.110:7001 \
> 192.168.33.110:7002 \
> 192.168.33.110:7003 \
> 192.168.33.110:7004 \
> 192.168.33.110:7005 \
> 192.168.33.110:7006 \
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7001
127.0.0.1:7002
127.0.0.1:7003
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7006 to 127.0.0.1:7002
Adding replica 127.0.0.1:7004 to 127.0.0.1:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7001)
M: 4ed7d8f54311fb883431c1c5a4715c1ae59d5215 127.0.0.1:7001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 11f577639b081c1514f28a74764c9330d819dcb8 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e
S: a6f79f4d1de8d7bef4911e934b802829ed29a2d3 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 31030e4c35659d5de1146941a2d1f2c1524b53e9
S: ff6d7ca3ea29036385c89a5dc93f807f91307871 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 4ed7d8f54311fb883431c1c5a4715c1ae59d5215
M: 31030e4c35659d5de1146941a2d1f2c1524b53e9 127.0.0.1:7002
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: 573752b8b0747bc2c7bb3e4a1edb0e467cfdef0e 127.0.0.1:7003
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

可以看到，创建集群这个过程比较繁琐，可以创建脚本来完成这些操作

启动脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
$ sudo vim /usr/local/redis-cluster/bin/start.sh
#!/usr/bin/env bash

RedisServerPath=/usr/local/redis-cluster/bin
RedisConfPath=/usr/local/redis-cluster/etc

for i in 1 2 3 4 5 6; do
    $RedisServerPath/redis-server $RedisConfPath/redis-700$i.conf
done

/usr/local/redis-cluster/bin/redis-trib.rb create --replicas 1 \
192.168.33.110:7001 \
192.168.33.110:7002 \
192.168.33.110:7003 \
192.168.33.110:7004 \
192.168.33.110:7005 \
192.168.33.110:7006 \
$ sudo chmod a+x /usr/local/redis-cluster/bin/start.sh

停止脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$ sudo vim /usr/local/redis-cluster/bin/stop.sh
#!/usr/bin/env bash

RedisClusterPath=/usr/local/redis-cluster
RedisCliPath=/usr/local/redis-cluster/bin
ip=192.168.33.110

for i in 1 2 3 4 5 6; do
    $RedisCliPath/redis-cli -c -h $ip -p 700$i shutdown
done

rm -rf $RedisClusterPath/log/*
rm -rf $RedisClusterPath/run/*
rm -rf $RedisClusterPath/var/*
$ sudo chmod a+x /usr/local/redis-cluster/bin/stop.sh

查看信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> cluster nodes # 查看节点信息
5521b06a46910c873e7ee8a274f2cb9fc41ffed1 192.168.33.110:7003@17003 master - 0 1526681994008 3 connected 10923-16383
8153a1b35e0fab1e48f0171653f6cffd82cbc5a6 192.168.33.110:7004@17004 slave d7e0395152496f41fe73fbbdabc8499c79f6632e 0 1526681996033 4 connected
a2ff29661185f6ad17c99232d0490b3814ba0164 192.168.33.110:7006@17006 slave affc237bf177fd4ac591314736a1358e16545d0c 0 1526681995020 6 connected
d7e0395152496f41fe73fbbdabc8499c79f6632e 192.168.33.110:7002@17002 master - 0 1526681993000 2 connected 5461-10922
8b613e4b5abbdf3c2486dfcb8bc1e9dde1897820 192.168.33.110:7005@17005 slave 5521b06a46910c873e7ee8a274f2cb9fc41ffed1 0 1526681993000 5 connected
affc237bf177fd4ac591314736a1358e16545d0c 192.168.33.110:7001@17001 myself,master - 0 1526681994000 1 connected 0-5460
192.168.33.110:7001> cluster info # 查看集群信息
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:59501
cluster_stats_messages_pong_sent:61755
cluster_stats_messages_sent:121256
cluster_stats_messages_ping_received:61750
cluster_stats_messages_pong_received:59501
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:121256

命令	说明
`CLUSTER INFO`	打印集群信息
`CLUSTER NODES`	列出集群当前已知的所有节点信息
`CLUSTER MEET <ip> <port>`	将 `ip` 和 `port` 所指定的节点添加到集群当中
`CLUSTER FORGET <node_id>`	从集群中移除 `node_id` 指定的节点
`CLUSTER REPLICATE <node_id>`	将当前节点设置为 `node_id` 指定的节点的从节点
`CLUSTER SAVECONFIG`	将节点的配置文件保存到硬盘里面
`CLUSTER ADDSLOTS <slot> [slot ...]`	将一个或多个槽（`slot`）指派（`assign`）给当前节点
`CLUSTER DELSLOTS <slot> [slot ...]`	移除一个或多个槽对当前节点的指派
`CLUSTER FLUSHSLOTS`	移除指派给当前节点的所有槽，让当前节点变成一个没有指派任何槽的节点
`CLUSTER SETSLOT <slot> NODE <node_id>`	将槽 `slot` 指派给 `node_id` 指定的节点，如果槽已经指派给另一个节点，那么先让另一个节点删除该槽，然后再进行指派
`CLUSTER SETSLOT <slot> MIGRATING <node_id>`	将本节点的槽 `slot` 迁移到 `node_id` 指定的节点中
`CLUSTER SETSLOT <slot> IMPORTING <node_id>`	从 `node_id` 指定的节点中导入槽 `slot` 到本节点
`CLUSTER SETSLOT <slot> STABLE`	取消对槽 `slot` 的导入（`import`）或者迁移（`migrate`）
`CLUSTER KEYSLOT <key>`	计算键 `key` 应该被放置在哪个槽上
`CLUSTER COUNTKEYSINSLOT <slot>`	返回槽 `slot` 目前包含的键值对数量
`CLUSTER GETKEYSINSLOT <slot> <count>`	返回 `count` 个 `slot` 槽中的键

实例操作

客户端操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// 从某个节点设置值后，去不同的节点获取设置的值
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> set foo bar
-> Redirected to slot [12182] located at 192.168.33.110:7003
OK
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7002
192.168.33.110:7002> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7003
192.168.33.110:7003> get foo
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7004
192.168.33.110:7004> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7005
192.168.33.110:7005> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"
$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7006
192.168.33.110:7006> get foo
-> Redirected to slot [12182] located at 192.168.33.110:7003
"bar"

程序操作

下载安装

Flexible and feature-complete Redis client for PHP and HHVM
Quick tour

1
2
3
4
5
6
$ cd ~
$ mkdir test
$ cd test
$ composer init
$ composer install
$ composer require predis/predis

测试代码

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<?php

require __DIR__ . '/vendor/autoload.php';

for ($i = 0; $i < 6; $i++) {
    $parameters[] = [
        'host'               => '192.168.33.110',
        'port'               => '700' . $i,
        'password'           => null,
        'database'           => 0,
        'timeout'            => 5,
        'read_write_timeout' => 5,
    ];
}
$options = ['cluster' => 'redis'];

try {
    $cluster = new \Predis\Client($parameters, $options);
} catch (\Predis\PredisException $e) {
    echo $e->getMessage();
    exit();
}

for ($i = 1; $i < 100; $i++) {
    $format = 'No.%s is a RedisCluster test.';
    $cluster->set("$i", sprintf($format, $i));
    echo $cluster->get("$i") . PHP_EOL;
}

测试结果

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ php cluster.php
No.1 is a Redis cluster test.
No.2 is a Redis cluster test.
No.3 is a Redis cluster test.
...
No.99 is a Redis cluster test.

$ /usr/local/redis-cluster/bin/redis-cli -c -h 192.168.33.110 -p 7001
192.168.33.110:7001> get 11
"No.11 is a RedisCluster test."
192.168.33.110:7001> get 23
-> Redirected to slot [9671] located at 192.168.33.110:7002
"No.23 is a RedisCluster test."
192.168.33.110:7002> get 33
"No.33 is a RedisCluster test."
192.168.33.110:7002> get 67
"No.67 is a RedisCluster test."
192.168.33.110:7002> get 88
-> Redirected to slot [15207] located at 192.168.33.110:7003
"No.88 is a RedisCluster test."
192.168.33.110:7003> get 99
-> Redirected to slot [6263] located at 192.168.33.110:7002
"No.99 is a RedisCluster test."
192.168.33.110:7002> get 100
-> Redirected to slot [339] located at 192.168.33.110:7001
(nil)
192.168.33.110:7001> get 17
-> Redirected to slot [12304] located at 192.168.33.110:7003
"No.17 is a RedisCluster test."
192.168.33.110:7003> get 42
-> Redirected to slot [8000] located at 192.168.33.110:7002
"No.42 is a RedisCluster test."
192.168.33.110:7002> get 56
-> Redirected to slot [11509] located at 192.168.33.110:7003
"No.56 is a RedisCluster test."
192.168.33.110:7003> get 61
-> Redirected to slot [2369] located at 192.168.33.110:7001
"No.61 is a RedisCluster test."

目录