Add a Redis instance
Java Serialized Objects
Use Smaller Keys
Switch to 32 Bits
Upgrade Redis Version
Use Better Serializer
Combine Smaller Strings to Hashes
Switch from Set to Intset for Numeric IDs
Switch to Bloom Filter or HyperLogLog
Shard Big Hash to Small Hash
Convert Hashtable to Ziplist for Hashes
Convert to a List instead of Hash
Compress Field Names
Enable Compression for List
Avoid Dynamic Lua Script
Reclaim Expired Keys Memory Faster
How Memory Analysis Works
Here’s a brief description of what goes on under the hood when you analyze a snapshot:
analyze-memory button is clicked, it connects to the redis instance and takes a point-in-time snapshot of the database.
This can be done in two ways:
- Using the SYNC command.
This is the preferred approach and is used if possible.
Redis has a
SYNCcommand that slaves use to synchronize with the master. Our agent pretends to be a slave and sends the
SYNCcommand to the instance, which responds with all its data as it would to a slave trying to synchronize with it.
- Using the DUMP command.
Cloud providers do not support the
SYNCcommand, so that approach won’t work. But they do support the
DUMPcommand. This command serializes the value of a key in a redis-specific format and returns it. We scan all the keys iteratively, dump the values, and concatenate them to generate an RDB. There are a couple of caveats with this method, among them being that the serialization format is opaque and non-standard and that all the keys are not dumped at the exact same time, meaning that some keys’ values may have changed since the time the first key was dumped, making this not an exact point-in-time snapshot.
After we have the dump, following either of the approaches mentioned above, we perform analysis on the dump, computing memory statistics and discovering key patterns. What happens here is similar to what is done by the open source redis-rdb-tools. The result of this process is an RSNAP file (stands for redis snapshot), which contains the key names, memory statistics and other generated information about the dump, but importantly, not the values of the keys themselves. The dump file never really leaves the system the agent runs on.
After the RSNAP file is completely generated, it is uploaded to our servers where further analysis is done to generate recommendations. We have over 20 recommendations at this point which give you simple advice on how to optimize your redis database.
So that’s a very brief overview of what happens under the hood at Rdbtools. We are constantly working on improving our process and we’ve had quite a bit of back and forth about the exact mechanism of the entire process. It goes without saying that the process will keep evolving and might even look very different in the near future. We’ll try to keep this page updated with all significant changes, so check back here or follow our blog to stay updated.