When using Sidekiq and Redis, memory issues can occur due to several reasons.
Some common reasons include memory leaks, inefficient data structures, and
improper configuration settings. Debugging these issues requires a systematic
approach and careful analysis of logs and performance metrics. In this chapter,
we will see some redis-cli commands that will aid us in debugging
memory-related issues.
Open up the console and connect to Redis using the below command.
You can get the values for REDIS_HOST, REDIS_PORT and DB_NO by inspecting
the Sidekiq server log. It should have a log in the following format:
In this case, 127.0.0.1 is the value for REDIS_HOST, 6379 is the value for
REDIS_PORT and 1 is the DB_NO. You can test whether the connection is
alive by typing PING into the Redis console.
We can see the memory usage of Redis by typing the following into the console:
It is not uncommon to observe that the maxmemory_human metric is reported as
0B, while the used_memory_human metric indicates a non-zero value, such as
2.42M. This is a result of configuring the maxmemory parameter to a value of
zero, which indicates that there are no enforced memory constraints on the
system. This behavior is the default for 64-bit systems, while 32-bit systems
have an implicit memory limit of 3GB.
You can set the max memory limit using the CONFIG SET command.
Upon setting the maxmemory value, the system will operate within the memory
constraints defined by this value. However, once the specified amount of memory
is reached, the default behavior of the system will depend on the eviction
policies that have been configured.
If you observe the output of INFO memory command, you might see
maxmemory_policy set to noeviction. With noeviction policy, Redis will
stop responding after the maxmemory value is reached. However, no data will be
evicted. You can learn about other eviction policies from the
official docs.
Inspecting keys
Now assume that you encountered a memory overflow error while using Sidekiq. How
would you get to know what caused the error? Redis is an in-memory, key/value
store. So an intuitive way would be to inspect the keys and their memory usage.
Executing the above command will provide a list of all the keys currently stored
in the Redis database. In certain scenarios, an excessive memory load may be
attributed to a large number of stored keys. Redis facilitates efficient key
inspection and management by providing a command, KEYS *<substring>*, which
enables users to narrow down their search by filtering keys based on a specific
substring. For instance, if one wants to inspect keys containing the substring
failed, the following command can be utilized:
You can also see the value of these keys. For this, you need to inspect the type
of key first.
Depending on the response from the previous command you can use the following
commands to see the corresponding values.
- For "string":
get <key>
- For "hash":
hgetall <key>
- For "list":
lrange <key> 0 -1
- For "set":
smembers <key>
- For "zset":
zrange <key> 0 -1 withscores
Inspecting memory usage
Sometimes, the number of keys will be less but the memory may be overloaded. In
this case, you can use the MEMORY USAGE <key> command to check the memory
usage of a specific key. For example, to see the memory usage corresponding to
key cron_job:todo_notifications_job:enqueued use the following command:
An efficient method would be to use a bash script to list all the keys with
their memory usage. For doing this, you can quit the Redis console and type the
following into the terminal:
Make sure you enter proper values for REDIS_HOST, REDIS_PORT and DB_NO.
This would print all the keys with their memory usage.
Now by observing the memory consumed by the keys, you will be able to identify
which key is responsible for memory overload. Normally this happens when we try
to store a large amount of data in Redis. You should avoid this as it can
negatively impact performance. Instead, use Redis to store small amounts of
metadata, such as job identifiers or timestamps.