Advanced Redis: Subscribe a script to a Pub/Sub channel

I’m not sure if this is a good idea. But I did it anyway.

First, some background.

Redis Pub/Sub Background

The first rule of using Redis Pub/Sub: you probably shouldn’t be using Redis Pub/Sub.

People get excited about Pub/Sub for some reason. Maybe it’s because Pub/Sub sounds like an alcoholic unterseebooten.

You try to be clever. You start plotting, “I’ll use Pub/Sub for distributed messaging, alerting, metrics, and updates!” Then, after a few days of programming your message passing infrastructure, reliable clients, and distributed Pub/Sub infrastructure, you find your system full of race conditions.

You patch your race condition infested Pub/Sub system. You start your Pub/Sub monstrosity. You quickly realize sometimes a PUBLISH happens when clients are disconnected, resulting in completely lost messages. Your home-grown messaging system suddenly evolves the requirement to store messages if no clients are around to process things.

You add monitoring and logging. It doesn’t help. All your stopgaps and hacks still result in lost messages. Sometimes.

You sit back and think about what works best for your problem at hand, and you have The Redis Pub/Sub Epiphany: “I shoulda just used a list and treated it as a persistent message queue.”

Using lists as a Pub/Sub replacement isn’t perfect though. The fundamental Pub/Sub hotness of multiple receivers all processing the same message in near real time goes away when you run a multi-consumer-from-single-list system.

Pub/Sub/Script

What if you could subscribe a Lua script to a Pub/Sub channel?

For example, if you subscribe a script to channel router.messages then send message HELLO, the script would receive ARGV[1] of router.message and ARGV[2] of HELLO.

The script receives all PUBLISH messages just like any normal subscribed client.

The magic is: the script is subscribed inside the Redis server itself. If you subscribe a script to a channel then disconnect, your script continues to process messages after you disconnect.

Doesn’t that sound neat? I thought it did. So, I got it working.

With my implementation, scripts can subscribe to a channel then get run after each PUBLISH to their channel. The prime directive of Redis Lua scripts still applies: scripts should be small, essentially creating compound commands and not performing excessive computation or analysis.

Lua scripts can run most Redis commands, including PUBLISH. You can create new and wonderful types of Pub/Sub topologies by re-broadcasting messages to other channels based on any criteria you can specify in a Lua script (incoming channel name, message contents, values of other redis keys specifying where matching messages should be re-broadcast, etc).

Sample Use Cases

Non-Lossy Channels

You can use a channel with a script to persist messages to an internal list if there are no other clients currently receiving messages.

The return value of the PUBLISH command is the number of clients (including server-side scripts) subscribed to receive the PUBLISH. Using the returned number of subscribers, you can count the receivers and if nobody received the message, just append the message to a list to process later (e.g. “If the receiver is only 1 (being me, the script), append the message to a list so an active client can de-queue it later”).

Historical Log

Likewise, you can use a script to persist every message sent to a channel to the Redis data structure of your choice (count them in a sorted set? keep them in a capped-sized list?).

Fun With Infinite Loops

You may be a bit ahead of me here, but, yes, you can also create a script to PUBLISH a message back to the receiving channel. It will work exactly as you expect. This is bad. Do not infinite loop Redis (unless you want to).

Internally Updated (and Bounded) Metrics

Redis 2.8.0 enables keyspace notifications. For every action altering any Redis key, you can configure Redis to broadcast a message to a Pub/Sub channel notifying you of the change (set/update/delete, notice: no reads. reads don’t modify the keyspace, so no keyspace notification is generated).

Since keyspace notifications use the standard Redis Pub/Sub mechanism, you can subscribe custom Lua scripts to keyspace notifications.

For example, subscribe a script to channel __keyevent@0__:set to store the top 50 keys you SET in your server. Or, be more generic and subscribe a script to __keyevent* and insert your top 100 keys into a sorted set (or in a list of fixed-size to see your 1000 most recent keys modified).

Commands Added

Note: these commands currently only exist in my Pub/Sub scripting branch. For more details, read the most recent commit messages there.

Naming

SCRIPT NAME [script-name] [existing-sha] Returns name you just set. Overrides any existing name.

SCRIPT DELNAME [script-name] Deletes name binding of script. Does not delete script or SHA.

SCRIPT GETNAME [script-name] Returns SHA of script.

EVALNAME [script-name or sha] [key-count] [keys …] [args …]

Pub/Sub Scripts

SSUBSCRIBE [channel] [script-name or sha] [script-name or sha …] Returns new count of all clients listening to channel.

SPSUBSCRIBE [channel-pattern] [script-name or sha] [script-name or sha …] Returns new count of all clients listening to channel.

SUNSUBSCRIBE [channel] [script-name or sha] [script-name or sha …] Returns number of scripts unsubscribed from channel.

SPUNSUBSCRIBE [channel-pattern] [script-name or sha] [script-name or sha …] Returns number of scripts unsubscribed from channel.

Decision Axes

Here are a few things I had to decide along the way (design decisions, as it were). There are usually reasons for the way I did it and the way I didn’t do it, but everybody will see things in different ways.

How to keep track of subscribing scripts to channels?

Introduce script naming support with new EVALNAME command. EVALNAME can still EVALSHA too.

Script names are local to the current node only (they aren’t replicated and won’t survive a server reboot) and exist in their own namespace. Script names have no interaction with the normal Redis keyspace.

You can re-point names at any time and anything using the name will execute using the new target value. For example: if you subscribe a channel to a script you called userMetrics, run it for two days, then update the name userMetrics to point to a new SHA of an updated script, all Pub/Sub calls will use the new SHA you’re pointing to without needing to unsubscribe or resubscribe on any channels.

If you use names with Pub/Sub, it gives you one less thing to track if you need to manage channel unsubscribes. (It’s easier to unsubscribe a short name you may store a persistent reference to than remember a 40 character sha1 that changes with every script update.)

How to integrate with Pub/Sub infrastructure?

I wanted to change as little code as possible, so Pub/Sub scripts act as fake clients inside the server. This way, the Pub/Sub code thinks the server-side script is just a normal client and doesn’t have to do double accounting duty for scripts versus actual clients of a Pub/Sub channel.

When to execute the script?

We could execute the Lua script during a PUBLISH

Clean, but means you can’t call PUBLISH inside of a Lua script executed as part of a Pub/Sub broadcast. (Redis functions are decidedly non-reentrant.)

Mock a normal client and process results in the client event loop

Too complicated. Since these clients never receive any live network traffic, you’d have to manually tell the event loop to process the fake client. Also, you’d have to manually de-parse the result buffer and reassemble the results into calls for the Lua script.

Insert delayed Lua script calls into a “do this soon, but not now” list

Instead of running the Lua script directly in the PUBLISH command, just make the PUBLISH command append the task “run this subscribed Lua script with these two or three arguments on this DB” to a global list of things to do. Redis has an internal cron running, by default, 10 times per second. The internal cron now has an additional job of checking if there are any Lua scripts waiting to run. If so, it executes them all.

How are keys defined?

Spoiler: they aren’t.

Redis Lua scripts receive arguments either in KEYS[] or ARGS[]. KEYS[] exists to tell Redis you are only accessing Redis keys defined in the KEYS[] arguments (which is specified at script call time, meaning you can only write to keys specified before the script even runs).

The KEYS[] mechanism isn’t currently enforced in Redis (it’ll be more useful for Redis Cluster), and I’m ignoring the KEYS[] problem entirely in this Pub/Sub script implementation. This incarnation of Redis Pub/Sub Lua Scripting Support probably won’t work in a cluster environment, but it will work fine in stand-alone or replicated Redis.

Conclusion

My implementation still has a few major flaws:

Flaws

there is no way to populate KEYS[] for any Pub/Sub scripts
- meaning scripts probably can’t run in the context of Redis Cluster
I’m not certain running the scripts delayed from the cron loop is proper
- I’d rather run them as a regular client, but I didn’t want to parse the command buffer (and I didn’t see how to trigger a fake client to wake up the outbound result sending event loop since it doesn’t have a real fd).
you can infinite loop Redis and there’s no way to break out of it except to realize it’s happening and either a.) unsubscribe the script from the looping channel or b.) kill the server.
- and you’re in big trouble if you’re infinite looping and appending to a list or doing something in the category of “appending data in an uncapped way.”
The return value of a Pub/Sub script will never be read.

But, my implementation has a few major awesomes too:

Awesomes

You can create interesting fan-out Pub/Sub topologies.
- You can re-route messages to other channels based on message contents or incoming channel (incoming channel can vary if you subscribe to pattern-based channels).
You can stop using lists when you really want Pub/Sub.
- Create a channel with only one script subscribed. PUBLISH to that channel. The script will re-PUBLISH your message to the actual channel, but if no clients receive the re-publish, your script can append the message to a list for processing when clients come back online.
You can track detailed in-server metrics with keyspace notifications.
You can use Pub/Sub scripts as a crazy way to let Redis Lua scripts call other Lua scripts.
- Normally, Redis Lua scripts are denied access to script-running commands (because the script running mechanism isn’t reentrant). But, Redis Lua scripts calling PUBLISH isn’t reentrant, it just appends your call to a list to process later. Yes, this means you can’t access the return value of your Pub/Sub script, but you can trigger other scripts to be called.
I’m pretty sure you can model all the core AMQP topologies somehow, but I haven’t looked into it yet.

That’s all for now.

Advanced Redis: Subscribe Script to Pub/Sub Channel