Redis - Replication and high availability
In the previous blog post, we looked at how to set up replication and high availability with PostgreSQL. Now let's look at how to do the same for Redis database.
At the base of Redis replication, there is a very simple to use and configure master-slave replication. It allows slave Redis instances to be exact copies of master instances. The slave will automatically reconnect to the master every time the link breaks, and will attempt to be an exact copy of it.
Redis Sentinel provides high availability for Redis. In practical terms, this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures.It also provides other tasks such as monitoring, notifications and acts as a configuration provider for clients.
Unlike PostgreSQL, setting and activating replication and high availability is pretty simple on Redis.
sudo add-apt-repository ppa:chris-lea/redis-server sudo apt-get update sudo apt-get install redis-sentinel
The above commands will install Redis server and Redis sentinel.
You need 3 servers to setup Redis server and Redis sentinel. Let's assume
that we have the following servers.
- Master Redis server (127.0.0.1)
- Slave Redis server (127.0.0.2)
- Sentinel server which acts like an arbiter (127.0.0.3)
Modify redis.conf file on master.
sudo vi /etc/redis/redis.conf
bind 0.0.0.0 protected-mode no
The above settings mean that anyone anywhere can connect to your Redis instance without a password. I did that because I wanted to keep the Redis configuration minimal. I have secured access to the Redis port using ufw (Ubuntu firewall) to allow access to only specific IPs. If you don't use ufw, please set the protected mode to yes and set a password, which will be used to access your Redis instance. The same applies for sentinel configuration as well. More information can be found here: https://redis.io/topics/security
Modify sentinel.conf file on master.
sudo vi /etc/redis/sentinel.conf
bind 0.0.0.0 protected-mode no sentinel monitor mymaster 127.0.0.2 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 10000 sentinel parallel-syncs mymaster 1 sentinel announce-ip "127.0.0.1"
Let’s look at what these options do:
- Sentinel monitor mymaster 127.0.0.2 6379 2 : This line tells sentinel which master to monitor (mymaster). The sentinel will find the host at 127.0.0.2:6379. The quorum for this setup is 2. The quorum is only used to detect failures: This number of sentinels must agree about the fact that the master is not reachable. When a failure is detected, one sentinel is elected as the leader who authorizes the failover. This happens when the majority of sentinel processes vote for the leader.
- Sentinel down-after-milliseconds mymaster 5000: The time in milliseconds an instance is allowed be unreachable for a sentinel (not answering to pings or replies with an error). After this time, the master is considered to be down.
- Sentinel failover-timeout mymaster 10000: The timeout in milliseconds that sentinel will wait after a failover before initiating a new failover.
- Sentinel parallel-syncs mymaster 1: The number of slaves that can sync with the new master at the same time after a failover.
- Sentinel announce-ip "127.0.0.1" : This is the public IP address of your server.
Now modify the redis.conf and sentinel.conf files on the slave server (127.0.0.2). Then sentinel.conf file will have only one change - the announce IP. The only change in redis.conf file is that you have to add the following line:
slaveof 127.0.0.1 6379
In the arbiter server (127.0.0.3), just configure the sentinel.
If you have ufw configured, make sure that you allow access to the ports 6379 and 26379 from all the above mentioned servers.
That's it. All the servers are configured and ready to go.
Start Redis server and Redis sentinel services on all the servers using the following commands:
sudo service redis-server start sudo service redis-sentinel start
The problem with Redis sentinel is that the client you use should provide support for it as well. Most of the clients do, some don't. If your client does not support connecting to sentinel directly, you can use Haproxy for high availability.
Install Haproxy and edit it's configuration:
sudo apt-get install haproxy sudo vi /etc/haproxy/haproxy.cfg
defaults REDIS mode tcp timeout connect 3s timeout server 6s timeout client 6s frontend ft_redis bind *:6380 name redis default_backend bk_redis backend bk_redis option tcp-check tcp-check connect tcp-check send PING\r\n tcp-check expect string +PONG tcp-check send info\ replication\r\n tcp-check expect string role:master tcp-check send QUIT\r\n tcp-check expect string +OK server remote_redis 127.0.0.1:6379 check inter 1s server local_redis 127.0.0.2:6379 check inter 1s
Now use the haproxy port in your Redis connection url. Haproxy will decide where to forward your request to.