Advanced Redis: Command Loading
People like making new commands for Redis. People often submit new commands for inclusion into the official Redis codebase. Sadly, the official Redis codebase doesn’t always need a “only return users with birth dates corresponding to the current Zodiac sign” command.
Adding Custom Commands [before]
How do people add custom commands directly into their Redis servers?
- Browse
redisCommandTable[]array inredis.cand discover out how commands are created. - Browse some command functions to see how they work.
- Add entries to
redisCommandTable[]along with your own command handling and support functions somewhere inside the Redis codebase. - Compile, test, debug, win.
That’s great! It’s easy, everybody loves the Redis codebase, but you have one giant problem: upgrades. Nobody enjoys merging their custom code into new releases. You eventually hit a merge conflict and end up having to double verify you didn’t break either Redis or your own code in the merge.
If you use distributed Redis, you have to make sure you only distribute your Redis or replication can break in new and creative ways when replicas receive commands not in their command table.
Using this approach, you probably don’t upgrade Redis very much (even though each release adds stability, speed, and feature awesomeness).
Adding Custom Commands [after]
What if you could preserve your custom commands across new installs of Redis?
Redis has two ways of making “custom commands”
- Built-in Lua scripting
- C functions bound to command names in
redisCommandTable[].
Lua Scripts
Writing a custom Lua script is the best solution to 80% of custom command needs.
Lua scripts are the preferred way to guarantee transactional semantics. Lua scripts are portable across Redis releases and scripts get automatically distributed to all replicas, so replicas never fail due to not having your custom command available. Redis also gives all Lua scripts access to fast built-in json and msgpack encoding and decoding.
But, custom Lua scripts only fit 80% of scenarios. For the other 20%, you’re diving into the Redis codebase and doing something clever. What if you didn’t have to fork Redis and build your custom command directly into it? What if you could load your custom Redis command as a library directly into Redis? What if we could do it right now?
Dynamic Command Loading
Everybody loves shared libraries. Who doesn’t remember their first time getting mod_perl and mod_ssl to both a.) compile and b.) load into Apache?
What if Redis had a module-add directive? You could ask Redis to load a shared library containing self-contained command functions with an export table of command name to function mappings.
You could write your custom command and maintain it outside of the Redis source tree. You should still recompile your module against each new version of Redis to verify the interfaces you use don’t change, but a quick recompile and re-automated-test is much simpler than merging a new upstream.
You could even distribute your custom command to others without needing to teach privileged patch/merge/compile incantations.
Maybe you can tell what I did this week?
Dynamic Redis
I added a dynamic module loading facility to Redis. It can be applied to existing and future Redis releases and allows you to write custom C commands (or C++ or anything generating a Unix shared object), load them into Redis, then use the commands directly as if they were built in directly.
How does it work?
redisCommandTable[] describes: every command, functions the commands run, and other details (number of arguments, if commands generate writes or are read-only, should be denied under certain situations, etc).
I made redisCommandTable[] extensible so we can append to it during runtime. You can load custom commands at server start-up using a config directive. You can load custom commands at runtime with a CONFIG SET directive. You can even reload your module at runtime so you can update/develop/debug custom command code without restarting.
Before you get more details, I must apply some giant warning labels here:
- You are interacting with Redis C code.
- You can easily add memory leaks, segfaults, and/or data corruption.
- Don’t do that.
- If you use replication, you must be extra careful about distributing your command modules and verifying they have the same version and behavior across instances.
More Details
You can find usage details at Dynamic Redis: Use Command Modules.
You can download a version of Redis with module loading built in at Dynamic Redis: Releases.
How do I make loadable commands?
Checkout krmt for examples and read Dynamic Redis: Create Command Modules for more details.
Conclusion
Try new things and let me know what you think.