How to analyze and optimize memory usage in Redis
Redis is an in-memory data store, which means that the entire dataset is stored in memory (RAM). While this is great for performance, when the size of the data starts growing, more and more RAM needs to be added to hold all that data. This can quickly become prohibitively expensive and before you know it, server costs start skyrocketing, leaving you worried about whether Redis can scale for you in a cost-effective way.
This is, of course, a common problem with any database, but hurts more in the case of in-memory databases like Redis, since RAM is more expensive than secondary storage like hard disks and SSDs. We must remember, however, that Redis is not meant to store terabytes of data for your application. Rather, it is meant to be used to store specifically that data that needs to be read (and to a lesser extent, written) at a very high rate, while the rest of your data can sit on disk-backed traditional databases. To that end, Redis is most often used as a cache, holding only the most active data with high read/write throughput requirements (think scoreboards and real-time chat messages). Hence, the main culprit for excessive memory usage with Redis is application behaviour. Your application may be storing unnecessary data that does not benefit from being in Redis, or even completely redundant data, i.e, data that’s never used for any purpose, as this user was.
To diagnose problems with excessive memory usage, we obviously need a way to find out which keys are using the most memory so we can begin to reason about and understand what behaviours of the application are causing the problem. Remember, premature optimization is the root of all evil, and Redis is no exception. We absolutely need visibility into the memory characteristics of our dataset to even begin thinking about where the problem lies and how we can go about fixing it. Below, we explore some of the options available to peek into our Redis data and find those areas that can benefit from optimization at the application level.
The MEMORY USAGE command
This aptly named command is available since Redis 4.0.0 and is the first step to debugging memory problems with Redis. From the docs:
The MEMORY USAGE command reports the number of bytes that a key and its value require to be stored in RAM. The reported usage is the total of memory allocations for data and administrative overheads that a key its value require.
Here’s an example:
MEMORY USAGE users_by_reputation (integer) 2923419
A simple integer representing the number of bytes used for storing the key, value and internal overheads is returned. For quickly finding the memory usage of a particular key, it doesn’t get simpler than this.
Two possible cons are:
- This command is only available in Redis 4.0.0 and up. This excludes most cloud provider.
- It only tells you the memory used by a single key, so unless you know in advance which key or keys are causing the problem, this command has very limited usefulness.
redis-rdb-tools is a nifty little command-line tool that goes beyond the humble
MEMORY USAGE command we discussed above. It is a popular and powerful memory profiling tool build specifically for Redis that parses a Redis dump (.rdb) file and generates a detailed memory profile for each key in the dump, including details like the memory used, data type, number of elements, etc. It also allows you to filter the keys which are processed, so you could optionally run the analysis on a particular key or a subset of keys that match a particular pattern. The output of the analysis is a CSV file with columns for the key, used memory, data type, number of elements, etc.
This tool is so great because, unlike the
MEMORY USAGE command, which works only on a single key, this tool analyzes the entire dataset and spits out the result in CSV format, which means you can load it into any SQL database and perform whatever queries you like, sorting it and filtering it as you wish, with all the power of SQL, which gives you great insight into the memory cost of your data. As a bonus, if you’re using Redis versions prior to 4.0.0, you can still use this tool to analyze single keys if you wish, since this tool works by parsing the binary dump that Redis produces, instead of relying on Redis to report memory usage. On the flip side, this difference in approach does mean that the calculated memory usage is more of an estimate than an exact figure, as mentioned in the project’s documentation. However, this is not really a problem when our goal is to identify memory usage problems, since we are more interested in relative sizes of the keys and identifying keys that use significantly more memory that they should be using, so a few bytes here or there is really irrelevant.
RDBTools is a web-based tool that offers several tools to manage and optimize Redis, with the main focus on memory optimization. It was born out of the open source redis-rdb-tools project discussed above. With it we can effortlessly analyze our Redis server’s memory usage by simply pointing to our instance and clicking “Analyze”. The tool downloads a dump from the running instance and analyzes it, presenting you with a slick UI for filtering and sorting your keys based on various parameters like memory usage, data type, encoding, number of elements, etc., similar to the command line tool, but without any of the manual effort involved in taking a dump, generating the memory profile, setting up a database, importing the profile, writing and running queries, etc.
Other than that, RDBTools also generates several charts and graphs from the memory profile allowing you even greater insight into your data, along with recommendations for optimizing the configuration of your Redis server, based on industry standards and years of experience optimizing Redis, to squeeze out even more memory savings. Apart from memory optimization, there a host of other features useful for anyone using Redis, like a built-in rich CLI, configuration editor with inline documentation, rich live data visualizer, etc.
RDBTools is available as a docker image, available in docker hub, and can be run on your own machine, so there’s no need to worry about leaking sensitive data. A trial version is available for free forever, with some usage limits. See the pricing page for more information on the various paid plans available.
Running out of memory is an inevitable problem when using in-memory databases like Redis. Like all optimization problems, the first step is to diagnose the problem correctly. It’s important to resist the temptation to quickly jump to conclusions based on unverified hypotheses. “Measure, measure, measure” is the name of the game when it comes to optimization, and there are various methods and tools for gaining insight into your Redis memory problems. Which one is best depends upon your specific needs and where you are in the development process. For someone just starting to use Redis, keeping an eye on individual keys could be enough to start, whereas for heavy users of Redis, it certainly makes sense to go for a dedicated GUI tool with all the bells and whistles to save pain later. Whatever your choice, always be sure to understand the root of your problem before trying to solve it, and remember, if you’re having memory problems with Redis, you’re probably using more memory than you really need.