Redis - Replication and high availability

Zakir Hussain -

In the previous blog post, we looked at how to set up replication and high availability with PostgreSQL. Now let's look at how to do the same for Redis database.

Introduction
At the base of Redis replication, there is a very simple to use and configure master-slave replication. It allows slave Redis instances to be exact copies of master instances. The slave will automatically reconnect to the master every time the link breaks, and will attempt to be an exact copy of it.

Redis Sentinel provides high availability for Redis. In practical terms, this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures.It also provides other tasks such as monitoring, notifications and acts as a configuration provider for clients.

Setup
Unlike PostgreSQL, setting and activating replication and high availability is pretty simple on Redis.

sudo add-apt-repository ppa:chris-lea/redis-server
sudo apt-get update
sudo apt-get install redis-sentinel

The above commands will install Redis server and Redis sentinel.

Configuration

You need 3 servers to setup Redis server and Redis sentinel. Let's assume
that we have the following servers.

  • Master Redis server (127.0.0.1)
  • Slave Redis server (127.0.0.2)
  • Sentinel server which acts like an arbiter (127.0.0.3)

Modify redis.conf file on master.

sudo vi /etc/redis/redis.conf

bind 0.0.0.0
protected-mode no

The above settings mean that anyone anywhere can connect to your Redis instance without a password. I did that because I wanted to keep the Redis configuration minimal. I have secured access to the Redis port using ufw (Ubuntu firewall) to allow access to only specific IPs. If you don't use ufw, please set the protected mode to yes and set a password, which will be used to access your Redis instance. The same applies for sentinel configuration as well. More information can be found here: https://redis.io/topics/security

Modify sentinel.conf file on master.

sudo vi /etc/redis/sentinel.conf

bind 0.0.0.0
protected-mode no

sentinel monitor mymaster 127.0.0.2 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1

sentinel announce-ip "127.0.0.1"

Let’s look at what these options do:

  1. Sentinel monitor mymaster 127.0.0.2 6379 2 : This line tells sentinel which master to monitor (mymaster). The sentinel will find the host at 127.0.0.2:6379. The quorum for this setup is 2. The quorum is only used to detect failures: This number of sentinels must agree about the fact that the master is not reachable. When a failure is detected, one sentinel is elected as the leader who authorizes the failover. This happens when the majority of sentinel processes vote for the leader.
  2. Sentinel down-after-milliseconds mymaster 5000: The time in milliseconds an instance is allowed be unreachable for a sentinel (not answering to pings or replies with an error). After this time, the master is considered to be down.
  3. Sentinel failover-timeout mymaster 10000: The timeout in milliseconds that sentinel will wait after a failover before initiating a new failover.
  4. Sentinel parallel-syncs mymaster 1: The number of slaves that can sync with the new master at the same time after a failover.
  5. Sentinel announce-ip "127.0.0.1" : This is the public IP address of your server.

Now modify the redis.conf and sentinel.conf files on the slave server (127.0.0.2). Then sentinel.conf file will have only one change - the announce IP. The only change in redis.conf file is that you have to add the following line:

slaveof 127.0.0.1 6379

In the arbiter server (127.0.0.3), just configure the sentinel.

If you have ufw configured, make sure that you allow access to the ports 6379 and 26379 from all the above mentioned servers.

That's it. All the servers are configured and ready to go.

Start Redis server and Redis sentinel services on all the servers using the following commands:

sudo service redis-server start
sudo service redis-sentinel start

The problem with Redis sentinel is that the client you use should provide support for it as well. Most of the clients do, some don't. If your client does not support connecting to sentinel directly, you can use Haproxy for high availability.

Setup Haproxy

Install Haproxy and edit it's configuration:

sudo apt-get install haproxy
sudo vi /etc/haproxy/haproxy.cfg

defaults REDIS
    mode tcp
    timeout connect 3s
    timeout server 6s
    timeout client 6s

frontend ft_redis
    bind *:6380 name redis
    default_backend bk_redis

backend bk_redis
    option tcp-check
    tcp-check connect
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    tcp-check send info\ replication\r\n
    tcp-check expect string role:master
    tcp-check send QUIT\r\n
    tcp-check expect string +OK
    server remote_redis 127.0.0.1:6379 check inter 1s
    server local_redis 127.0.0.2:6379 check inter 1s

Now use the haproxy port in your Redis connection url. Haproxy will decide where to forward your request to.