A Smarter Redis – DZone Database

Everyone loves Redis. It’s fast, easy to use from almost any programming language, and works exactly as it should. In addition, it’s got an impressive list of modules for additional functionality. It’s an excellent product.

Sometimes, though, we may want Redis to be a little smarter.

We want some clients to have a different view of the data; for instance, we might want to change how some data is stored or retrieved.

Redis does support scripting using the Lua language, but it does not seem to be used very much, I think for a few reasons:

  • Lua is an acceptable scripting language, but it’s a bit niche (used by less than 1% of programmers according to TIOBE, and mostly in games)

  • scripts have to be explicitly invoked and cannot modify the default behavior of Redis (unlike, say, a trigger in a SQL database)

  • a script is executed atomically — all server activities are blocked during its entire runtime, which can be daunting in high-performance environments

So if we want to change how our Redis database behaves, what are our options?

Redis is open source, so theoretically, we could modify it, requiring a significant amount of work.

We could also write a custom Redis module, but that is not a trivial task. Modules are written in C or anything that can produce a C-compatible shared library, and they have to be installed in the server, which is a non-starter in most cloud environments.

Enter the Proxy

A much easier option is to put a programmable database proxy between Redis and its clients. This will allow us to modify the requests and the responses as desired without making any changes to the Redis servers or the clients. In addition, by having that logic outside of Redis, we can decide which clients go through the proxy (and therefore are submitted to our logic) and which connect directly to Redis.

Does Redis Need to Be Smarter?

Redis does a wonderful job, but there are many scenarios in which additional smarts could make a big difference. For instance, we may want to:

  • integrate Redis into our enterprise authentication mechanism

  • reject specific requests given certain conditions

  • change how an existing application queries Redis without changing the application

  • add new types of commands to Redis

  • store some data using our form of encryption without affecting the clients

  • modify specific responses differ depending on who is asking

The list goes on. Let’s take a deeper look at a few examples.

Enterprise Authentication

Redis has its authentication mechanism, but it’s pretty simplistic. The password(s) can be specified in the Redis.conf file, or they can be added at runtime with a command. It’s better than nothing, but it does not integrate into well-known authentication systems (the Enterprise edition of Redis does support integration with LDAP, but the open-sourced version does not).

A proxy can add a layer of authentication that will be transparent to the applications by intercepting the AUTH command and performing authentication with a provider such as Active Directory, SAML, OpenID Connect, etc. If the credentials are valid, the proxy can let the client proceed. If the credentials are invalid, the proxy can respond with an error message and (optionally) close the connection.

Even more interesting, the proxy can retrieve the user’s profile (eg, roles and group memberships) from the identity provider and use that information to determine the user’s permissions.

Controlling Requests

Redis has the concept of access control lists (ACLs), which allow us to specify what access certain users get to certain objects. This mechanism is quite powerful, though its syntax is not necessarily obvious.

For instance, we can specify that user jdoe has permission to execute the GET and MGET commands on keys FOO* and BAR*:

ACL SETUSER jdoe on +GET ~FOO* ~BAR*
ACL SETUSER jdoe on +MGET ~FOO* ~BAR*

Redis 7.0 has extended this mechanism to make it possible to specify which keys are readable and which are writeable.

This mechanism works fine, but it only goes so far. In particular, Redis’ pattern matching is relatively limited, so using a complete regular expression engine in the proxy can be an advantage, especially when filtering responses.

More importantly, ACLs apply only to keys and not values. A programmable proxy, by contrast, can allow us to make sure that any object containing (for example) an attribute Classification: secret should not be visible to users not satisfying certain conditions.

With a programmable proxy, we can create arbitrary logic that can accept, reject or modify requests and responses based on any desired criteria, such as the user, the client IP address and location, the nature of the request or the response, the time of day, the day of the week, and so on.

A trivial example would be to record all request types during a period, then reject any request that has never been seen before — not appropriate for all databases, clearly, but a simple and effective security measure.

Redirecting Requests

Because the proxy receives every request before it gets to Redis, it can implement new commands, redirect some or all of these requests to a different Redis server, or translate them to queries for a completely different database, API, or whatever.

For example, the proxy can easily intercept requests such as fetch those values ​​from (say) a REST service and respond directly to the client without ever talking to Redis. This allows us to create virtual objects which do not exist in the Redis database but seem just as real to the clients.

Another example would be to create a new pseudo-command that is, in fact, not executed by Redis but rather by the proxy. However, as far as the clients are concerned, it looks and smells exactly like a new Redis command.

Custom Encryption

Redis has a module called redicrypt that provides new commands to store and retrieve data in encrypted form. This works fine, but it does require us to invoke encryption and decryption from our clients specifically.

A proxy can encrypt and decrypt data transparently as it moves between the clients and the servers, using our chosen encryption algorithms and keys. It can encrypt exactly what we need, which may be only certain objects or certain parts of certain objects. This is all done without any changes to the clients: as far as they’re concerned, nothing has changed at all.

Augmenting Pub/Sub

Redis has a simple but effective publish/subscribe model that is easy to use and works very nicely in a cluster.

A proxy can, of course, intercept any subscription commands and redirect them or extend them to other systems like MQ, Kafka, Akka, etc.

Correspondingly, a proxy can modify, hide or inject messages for clients to receive. This makes it a great way to integrate with other systems.

Renaming Commands

Just one last example. It’s rather specific but representative.

Redis allows us to rename commands (who knew?) For example, if we add the following to the Redis.conf file:

rename-command GET cf16263ba56e75f73232248251b9ba70

Then no one will be able to execute a GET unless they know the secret word that has replaced the command.

If your Redis server does this, you may have difficulties using some client libraries because most of them do not support this feature. A proxy can trivially change requests between clients and servers and make this problem disappear.

Conclusion

I hope these examples have piqued your interest. Redis is a hugely popular database for good reasons, but it can benefit from being (or seeming to be) even more capable in many situations.

If you’re curious, you can try the Redis tutorial for Gallium Data (a free database proxy) and get a feel for the possibilities this approach creates. Having an additional layer of control between your database clients and your database servers is an empowering experience — you don’t have to be limited by what your database can and cannot do out of the box.

.

Leave a Comment