Redis

Reids is a data structure store that can be used to store and communicate information needed during the execution of a program (or programs). It is commonly used as a caching layer in web servers, but it’s utility goes well beyond that example. Redis provides other memory storage operations that are useful in day to day development. In this post I will explore the “key-list” structure in Redis using the operations: rpop, lpop, rpush and lpush These operations easily provide queues and stacks.

Redis setup

There are several ways to get redis setup for use on your own system. In my case, I’m using a GCP Compute Engine node running a Debian based linux OS.1 Redis is available on may operating systems and package managers. In these posts, I will focus specifically on Linux Debian.

$ sudo apt-get install redis
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libatomic1 libhiredis0.14 libjemalloc2 liblua5.1-0 lua-bitop lua-cjson redis-server redis-tools
Suggested packages:
  ruby-redis
The following NEW packages will be installed:
  libatomic1 libhiredis0.14 libjemalloc2 liblua5.1-0 lua-bitop lua-cjson redis redis-server redis-tools
  
...TRUNCATED...

Created symlink /etc/systemd/system/redis.service → /lib/systemd/system/redis-server.service.
Created symlink /etc/systemd/system/multi-user.target.wants/redis-server.service → /lib/systemd/system/redis-server.service.
Setting up redis (5:5.0.3-4+deb10u2) ...
Processing triggers for systemd (241-7~deb10u5) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...

After this command has run, redis will be installed and will be running through Systemd:

$ systemctl status redis-server.service
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-01-25 13:13:50 UTC; 1h 27min ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 395 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
 Main PID: 683 (redis-server)
    Tasks: 4 (limit: 2322)
   Memory: 13.9M
   CGroup: /system.slice/redis-server.service
           └─683 /usr/bin/redis-server 127.0.0.1:6379

Jan 25 13:13:49 hugo-builder-images systemd[1]: Starting Advanced key-value store...
Jan 25 13:13:49 hugo-builder-images systemd[1]: redis-server.service: Can't open PID file /run/redis/redis-server.pid (yet?) after start: No such file or directory
Jan 25 13:13:50 hugo-builder-images systemd[1]: Started Advanced key-value store.

Redis has settings to protect the server from undesired access. If you plannin to use redis in a production environment, you would need to spend time to harden it. The default configuration for redis only binds to the localhost ip address, which is a good safe starting point. (See /etc/redis/redis.conf for details)

Redis repl

The installation command above includes redis-tools which provides the redis-cli command line tool. The redis-cli presents a simple REPL that you can use to interact with the redis database directly. To get help on the commands available, you can start by sending help

$ redis-cli
127.0.0.1:6379> help
redis-cli 5.0.3
To get help about Redis commands type:
      "help @<group>" to get a list of commands in <group>
      "help <command>" for help on <command>
      "help <tab>" to get a list of possible help topics
      "quit" to exit

To set redis-cli preferences:
      ":set hints" enable online hints
      ":set nohints" disable online hints
Set your preferences in ~/.redisclirc

The help command will give introductory information, but to get into actual commands, you will probably to specify a help group to get down to some specific actions. You can find more information on the redis commands available on the web: redis-cli information

The commands that we will be working with are list related so I send a help @list:

127.0.0.1:6379> help @list

...filtered...

  LLEN key
  summary: Get the length of a list
  since: 1.0.0

  LPOP key
  summary: Remove and get the first element in a list
  since: 1.0.0

  LPUSH key value [value ...]
  summary: Prepend one or multiple values to a list
  since: 1.0.0

  RPOP key
  summary: Remove and get the last element in a list
  since: 1.0.0

  RPUSH key value [value ...]
  summary: Append one or multiple values to a list
  since: 1.0.0

There are many redis list commands. I filtered the output above to only show the llen, lpop, lpush, rpop and rpush commands which I will be using in this post. For all of these commands, the key is a unique identifier for the list object that will be operating on.

For example, here is a REPL session:

  1. Report the length of the TEST list.

     127.0.0.1:6379> llen TEST
 (integer) 0

  2. Push “A” on to the left side of the TEST list.

     127.0.0.1:6379> lpush TEST A
 (integer) 1

  3. Push “B” on the left side of the TEST list.

     127.0.0.1:6379> lpush TEST B
 (integer) 2

  4. Report the length of the TEST list.

     127.0.0.1:6379> llen TEST
 (integer) 2

  5. Pop the “A” item off the right side of the TEST list.

     127.0.0.1:6379> rpop TEST
 "A"

  6. Pop the “B” item off the right side of the TEST list.

     127.0.0.1:6379> rpop TEST
 "B"

  7. Report the length of the TEST list.

     127.0.0.1:6379> llen TEST
 (integer) 0

Here’s a quick visual representation of the above operations:

svg

On all the push and pop commands the prefix l and r indicate which side of the list the item will be added, in the case of push, or removed in the case of pop. Unlike the push and pop operations, the l prefix on llen indicates that length operation is being performed on list.

Also note that conventionally, I will push onto the list from the left and pop off the right. This convention will be useful with list operations that I will go into in a subsequent post.

Stacks oh my! (quick diversion)

Just as a little diversion, the operations provided can also be used to create a stack rather than a queue. A stack can be achieved by popping off the same side as the push. For example lpush followed by a lpop. In this example, I push items on the left side and then pop them off the left side. The same result can be achieved using right side operations.

127.0.0.1:6379> lpush TEST A
(integer) 1
127.0.0.1:6379> lpush TEST B
(integer) 2
127.0.0.1:6379> lpush TEST C
(integer) 3
127.0.0.1:6379> lpop TEST
"C"
127.0.0.1:6379> lpop TEST
"B"
127.0.0.1:6379> lpop TEST
"A"
127.0.0.1:6379> lpop TEST
(nil)

Python examples

Ok so enough redis-cli repl work. Now I will demonstrate the queue operations using the redis python package. First up, getting redis installed.

Setup python

I find that my gcp Compute Engine is a little thin on necessary python stuff. Fortunately there is some reference material: gcp setup linux Also, I always use venv to isolate my python environment and within that virtual environment, I use pip to install python packages.

To install python3, venv and pip I used the following commands:

sudo apt update
sudo apt install python3 python3-dev python3-venv wget
wget https://bootstrap.pypa.io/get-pip.py
sudo python3 get-pip.py

Get redis package

Now that I have a working modern python with pip I can install the redis package.

$ python -m venv redis
$ source redis/bin/activate
(redis) $ pip install redis
Collecting redis
  Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 270 kB/s 
Installing collected packages: redis
Successfully installed redis-3.5.3

Now I should have the minimal environment necessary to run through redis examples:

(redis) $ pip freeze
pkg-resources==0.0.0
redis==3.5.3

Using the python REPL to use redis

Just like there’s a redis repl, there is a python repl. Here is an example of connecting to redis through python using the repl

  1. Start up the repl

     (redis) $ python3
 Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
 [GCC 8.3.0] on linux
 Type "help", "copyright", "credits" or "license" for more information.

  2. Import Redis class from the repl package and create an instance.

     >>> from redis import Redis
 >>> conn = Redis()
 >>> conn.ping()
 True

  3. Check to see if there are any existing keys.

     >>> conn.keys("*")
 []

  4. Push “A” onto the lefthand side of the TEST list, then similarly push “B”

     >>> conn.lpush("TEST","A")
 1
 >>> conn.lpush("TEST","B")
 2

  5. Check the size of the list then pop items off the list

     >>> conn.llen("TEST")
 2
 >>> conn.rpop("TEST")
 b'A'
 >>> conn.rpop("TEST")
 b'B'
 >>> conn.llen("TEST")
 0

  6. Note that items retrieved from redis are prefixed with a ‘b’ indicating that the data is returned as a binary. Here is an example of retrieving an item from the queue and decoding it as ‘ASCII’ for use.

     >>> conn.lpush("TEST","A")
 1

 >>> conn.rpop("TEST").decode("ASCII")
 'A'

  7. Building on this nuance a little, if you want to store json and retrieve json from Redis. You will need to decode the value.

     >>> import json
 >>> mydata = {"name":"something important", "value": 99}
 >>> json.dumps(mydata)
 '{"name": "something important", "value": 99}'
 >>> conn.lpush("TEST",json.dumps(mydata))
 1
 >>> result = conn.rpop("TEST")
 >>> result
 b'{"name": "something important", "value": 99}'

 >>> mydata_result = json.loads(result.decode("ASCII"))
 >>> mydata_result
 {'name': 'something important', 'value': 99}

  8. Finally close the Redis connection

     >>> conn.close()
 >>> conn.ping()
 True

The Python Code

Ok so now what does the code look like to push items onto a queue and then pull them off. Again, this is not a hardened production environment, just a workspace that will be brought down when not in use.

from redis import Redis
import os
import string
import logging

logger = logging.getLogger(__name__)
logging.basicConfig(level=os.getenv('LOGLEVEL', 'DEBUG'))


def main():
    # connect to redis database
    conn = Redis()

    # push several items onto the queue:
    letters_to_push = string.ascii_uppercase[5:10]
    for letter in letters_to_push:
        logger.debug("pushing letter: {}".format(letter))
        logger.debug("rpush result:{}".format(conn.rpush('pending', letter)))

    # pop all the letters off the queue
    while 1:
        item = conn.lpop('pending')
        logger.debug("lpop result:{}".format(item))
        if item is None:
            break

    logger.debug("done")


if __name__ == '__main__':
    main()

And finally, some output from the script:from redis import Redis import os import string import logging

logger = logging.getLogger(name) logging.basicConfig(level=os.getenv(‘LOGLEVEL’, ‘DEBUG’))

def main(): # connect to redis database conn = Redis() logger.debug(“Is there a connection? {}".format(conn.ping())) conn.close()

# push several items onto the queue:
letters_to_push = string.ascii_uppercase[5:10]
for letter in letters_to_push:
    logger.debug("pushing letter: {}".format(letter))
    logger.debug("rpush result:{}".format(conn.rpush('pending', letter)))

# pop all the letters off the queue
while 1:
    item = conn.lpop('pending')
    if item is None:
        break
    logger.debug("lpop result:{}".format(item.decode("ASCII")))

logger.debug("done")

if name == ‘main': main()

(redis) $ python RedisPushPop.py
DEBUG:__main__:Is there a connection? True
DEBUG:__main__:pushing letter: F
DEBUG:__main__:rpush result:1
DEBUG:__main__:pushing letter: G
DEBUG:__main__:rpush result:2
DEBUG:__main__:pushing letter: H
DEBUG:__main__:rpush result:3
DEBUG:__main__:pushing letter: I
DEBUG:__main__:rpush result:4
DEBUG:__main__:pushing letter: J
DEBUG:__main__:rpush result:5
DEBUG:__main__:lpop result:F
DEBUG:__main__:lpop result:G
DEBUG:__main__:lpop result:H
DEBUG:__main__:lpop result:I
DEBUG:__main__:lpop result:J
DEBUG:__main__:done

Final Word

Redis provides a lot of interesting capabilities and can provide a useful way to distribute tasks to independent workers. I hope that this post has provided you with a better understanding of the queue and stack capabilities available in Redis.

  1. Yadda Yadda.. all my other computers are dead or on Apple M1 and I don’t have the patience to get hugo/go working yet… ↩︎