Tutorial

Create Redis Replication with Sentinel using Docker on multiple Servers

Bule Edan

There are multiple tutorials that explains how to setup Redis with multiple replicas, connected to sentinel, however always on the same server. Event when using docker, all is in one server.
However here is a the issue: if the server goes down, all goes down, and the point of having replicas and sentinel goes to the trash, no more High Availability.

That’s where came the idea, how about having a network with multiple Servers, where each server has a Redis Node (Master or Slave), and a Redis Sentinel instance. So if one server goes down, we still can have High Availability, Sentinel taking care of the failover.

Now how to do it?

Our target for the tutorial

Here is our basic structure target

You will notice “ZeroTier“, what is it? It is a great and simple way to create and manage your own private network with any devices you want (even your laptop/phone/tablet). This allows you to close all sensitive ports, and only devices from this private network will be able to have access. So this is such a huge plus for security.

Requirements for the tutorial

  • Virtual Machine (in this case I use VMWare)
  • A ZeroTier account. Can make one for free on zerotier.com, you can have up to 25 devices on your network.
  • Docker installed on your VMs (Install Docker)

#1 – Disable Ports on the main interface

For security purpose, we will close the needed ports on the main interface:

  • 6379: Redis
  • 8001: Redis Insights
  • 26379: Redis Sentinel

Considering our main interface being ens160, here are the commands depending on your firewall:

IPTables

sudo iptables -A INPUT -i ens160 -p tcp -m tcp -m multiport ! --dports 6379,8001,26379 -j DROP

UFW

sudo ufw deny in on eth0 proto tcp to any port 6379,8001,26379

#2 – Have your VMs joining your ZeroTier network

On each of your VMs, type the command to join your ZeroTier network: 

sudo zerotier-cli join YOUR_ZT_NETWORK_ID

After you have enabled your devices from your ZeroTier admin, make sure they all have their IP assigned.
In our case we will consider:

  • VM 1: 10.147.19.1
  • VM 2: 10.147.19.2
  • VM 3: 10.147.19.3

#3 – Avoiding having memory alerts on our VMs using docker

Running our docker containers may trigger some alerts. At least I had it running it on VMWare, using Debian 11.
This command will save your life: 

sysctl vm.overcommit_memory=1

#4 – Setting up and Running the Master

On the VM1, create a directory for your project, and in the same time a subdirectory for Redis Persistence:

sudo mkdir -p /var/www/PROJECT_NAME/data

Then you create docker-compose.yml in your project directory /var/www/PROJECT_A

version: '2'

services:
  redis-master:
    image: redis/redis-stack-server:latest
    environment:
      - REDIS_ARGS=--port 6379 --requirepass Str0ngP4ssw0rdF0rRedis
    ports:
      - '6379:6379'
      - '8001:8001'
    volumes:
      - '/var/www/PROJECT_A/data:/data'
  redis-sentinel:
    image: 'bitnami/redis-sentinel:latest'
    environment:
      - REDIS_MASTER_HOST=10.147.19.1
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=Str0ngP4ssw0rdF0rRedis
      - REDIS_SENTINEL_PASSWORD=P4ssw0rdF0rSentinel
      - REDIS_SENTINEL_PORT_NUMBER=26379
      - REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=10000
      - REDIS_SENTINEL_FAILOVER_TIMEOUT=180000
      - REDIS_SENTINEL_ANNOUNCE_IP=10.147.19.1
      - REDIS_SENTINEL_ANNOUNCE_PORT=26379
      - REDIS_SENTINEL_QUORUM=2
    ports:
      - '26379:26379'
volumes:
  redis_data:
    driver: local

We are using 2 different docker images:

  • Redis Stack Server: which will allow us to have Redis with all its modules such as RediSearch, RedisJSON, RedisTimeSeries, …
    DockerHub Documentation: click here
  • Redis Sentinel by Bitnami: so far this is the best docker image that I’ve found. Very complete and ease for the configuration

Now let’s create our containers:

sudo docker compose up -d

IMPORTANT

  • Make sure to generate strong and different passwords for both Redis and Sentinel: Redis is so fast that brute force attacks is very efficient.
  • Make sure to keep the same password between all Redis master and slaves. This will avoid issues on status change from slave to master.
  • REDIS_SENTINEL_ANNOUNCE_IP and REDIS_SENTINEL_ANNOUNCE_PORT are very important. They will make sure to announce their ZeroTier IP and Port to other sentinels. If not, it will use the default VM IP.

#5 – Setting up and Running the Slaves

On the VM2 and VM3, create a directory for your project, and in the same time a subdirectory for Redis Persistence:

sudo mkdir -p /var/www/PROJECT_NAME/data

Then you create docker-compose.yml in your project directory /var/www/PROJECT_A

docker-compose.yml on VM2:

version: '2'

services:
  redis-slave:
    image: redis/redis-stack-server:latest
    environment:
      - REDIS_ARGS=--port 6379 --requirepass Str0ngP4ssw0rdF0rRedis --replicaof 10.147.19.1 6379 --masterauth Str0ngP4ssw0rdF0rRedis --replica-announce-ip 10.147.19.2 --replica-announce-port 6379
    ports:
      - '6379:6379'
      - '8001:8001'
    volumes:
      - '/var/www/PROJECT_A/data:/bitnami/redis/data'
  redis-sentinel:
    image: 'bitnami/redis-sentinel:latest'
    environment:
      - REDIS_MASTER_HOST=10.147.19.1
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=Str0ngP4ssw0rdF0rRedis
      - REDIS_SENTINEL_PASSWORD=P4ssw0rdF0rSentinel
      - REDIS_SENTINEL_PORT_NUMBER=26379
      - REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=10000
      - REDIS_SENTINEL_FAILOVER_TIMEOUT=180000
      - REDIS_SENTINEL_ANNOUNCE_IP=10.147.19.2
      - REDIS_SENTINEL_ANNOUNCE_PORT=26379
      - REDIS_SENTINEL_QUORUM=2
    ports:
      - '26379:26379'
volumes:
  redis_data:
    driver: local

docker-compose.yml on VM3:

version: '2'

services:
  redis-slave:
    image: redis/redis-stack-server:latest
    environment:
      - REDIS_ARGS=--port 6379 --requirepass Str0ngP4ssw0rdF0rRedis --replicaof 10.147.19.1 6379 --masterauth Str0ngP4ssw0rdF0rRedis --replica-announce-ip 10.147.19.3 --replica-announce-port 6379
    ports:
      - '6379:6379'
      - '8001:8001'
    volumes:
      - '/var/www/PROJECT_A/data:/bitnami/redis/data'
  redis-sentinel:
    image: 'bitnami/redis-sentinel:latest'
    environment:
      - REDIS_MASTER_HOST=10.147.19.1
      - REDIS_MASTER_PORT_NUMBER=6379
      - REDIS_MASTER_PASSWORD=Str0ngP4ssw0rdF0rRedis
      - REDIS_SENTINEL_PASSWORD=P4ssw0rdF0rSentinel
      - REDIS_SENTINEL_PORT_NUMBER=26379
      - REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=10000
      - REDIS_SENTINEL_FAILOVER_TIMEOUT=180000
      - REDIS_SENTINEL_ANNOUNCE_IP=10.147.19.3
      - REDIS_SENTINEL_ANNOUNCE_PORT=26379
      - REDIS_SENTINEL_QUORUM=2
    ports:
      - '26379:26379'
volumes:
  redis_data:
    driver: local

Now let’s create our containers:

sudo docker compose up -d

#6 – Checking our infra

Let’s have a looks at the logs of our sentinel in the VM1 using this command: 

sudo docker logs PROJECT_A-redis-sentinel-1

You should have something similar with this:

redis-sentinel 13:37:42.42 Welcome to the Bitnami redis-sentinel container
redis-sentinel 13:37:42.42 Subscribe to project updates by watching https://github.com/bitnami/containers
redis-sentinel 13:37:42.42 Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-sentinel 13:37:42.42
redis-sentinel 13:37:42.42 INFO ==> ** Starting Redis sentinel setup **
redis-sentinel 13:37:42.42 INFO ==> Initializing Redis Sentinel…
redis-sentinel 13:37:42.42 INFO ==> Configuring Redis Sentinel…
redis-sentinel 13:37:42.42 INFO ==> ** Redis sentinel setup finished! **

redis-sentinel 13:37:42.42 INFO ==> ** Starting Redis Sentinel **
1:X 31 Oct 2023 13:37:42.420 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:X 31 Oct 2023 13:37:42.420 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=1, just started
1:X 31 Oct 2023 13:37:42.420 * Configuration loaded
1:X 31 Oct 2023 13:37:42.420 * monotonic clock: POSIX clock_gettime
1:X 31 Oct 2023 13:37:42.420 * Running mode=sentinel, port=26379.
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk
1:X 31 Oct 2023 13:37:42.420 * Sentinel ID is 2f318fd092574cdc6e133b9e5b04d133730bc1f9
1:X 31 Oct 2023 13:37:42.420 # +monitor master mymaster 10.147.19.1 6379 quorum 2
1:X 31 Oct 2023 13:37:42.420 * +sentinel sentinel 82ac5bd621c3a0acea04762625fafb123bdd3893 10.147.19.2 26379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk
1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.2:6379 10.147.19.2 6379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk
1:X 31 Oct 2023 13:37:42.420 * +sentinel sentinel 59d9d2749d7dc1bd5221f8b06a12346c2cc94339 10.147.19.3 26379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk
1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.3:6379 10.147.19.3 6379 @ mymaster 10.147.19.1 6379

We can see the informations when it discovers the replicas and set them as slave:

1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.2:6379 10.147.19.2 6379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.3:6379 10.147.19.3 6379 @ mymaster 10.147.19.1 6379

And also we can see when it discovers the other sentinels:

1:X 31 Oct 2023 13:37:42.420 * +sentinel sentinel 82ac5bd621c3a0acea04762625fafb123bdd3893 10.147.19.2 26379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * +sentinel sentinel 59d9d2749d7dc1bd5221f8b06a12346c2cc94339 10.147.19.3 26379 @ mymaster 10.147.19.1 6379

If you check on the other VMs, you will have similar information in their sentinel logs.

#7 – Testing Fail Over

First, let’s follow the logs from any sentinel:

sudo docker logs -f PROJECT_A-redis-sentinel-1

Let’s try killing the Redis Master:

sudo docker kill PROJECT_A-redis-master-1

Then you should see these lines appearing, having sentinel discovering master redis went down, and selecting slave as a new master:

1:X 31 Oct 2023 13:37:42.420 # +sdown master mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 # +odown master mymaster 10.147.19.1 6379 #quorum 3/2
1:X 31 Oct 2023 13:37:42.420 # +new-epoch 1
1:X 31 Oct 2023 13:37:42.420 # +try-failover master mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk
1:X 31 Oct 2023 13:37:42.420 # +vote-for-leader 2f318fd092574cdc6e133b9e5b04d164930bc1f9 1
1:X 31 Oct 2023 13:37:42.420 * 82ac5bd621c3a0acea04762625fafb123bdd3893 voted for 82ac5bd621c3a0acea04762625fafb123bdd3893 1
1:X 31 Oct 2023 13:37:42.420 * 59d9d2749d7dc1bd5221f8b06a12346c2cc94339 voted for 82ac5bd621c3a0acea04762625fafb123bdd3893 1
1:X 31 Oct 2023 13:37:42.420 # +config-update-from sentinel 82ac5bd621c3a0acea04762625fafb123bdd3893 10.147.19.2 26379 @ mymaster 10.147.19.1 6379
1:X 31 Oct 2023 13:37:42.420 # +switch-master mymaster 10.147.17.1 6379 10.147.17.249 6379
1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.2:6379 10.147.19.2 6379 @ mymaster 10.147.19.3 6379
1:X 31 Oct 2023 13:37:42.420 * +slave slave 10.147.19.1:6379 10.147.19.1 6379 @ mymaster 10.147.19.3 6379
1:X 31 Oct 2023 13:37:42.420 * Sentinel new configuration saved on disk

What’s next?

We can scale this up by having a cluster of Redis replication for multiple projects on the same servers. For a better resource management, we can have the masters on different servers and go for an infra like this:

And voilà. Please share comments if you have any feedbacks about this, hope this has been helpful for you.

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

We use cookies to improve your experience on our website