Pub/Sub log handler example

This example will show you how to forward logs into a Redis Pub/Sub channel using rlh.RedisPubSubLogHandler.

Logger setup

import logging
from rlh import RedisPubSubLogHandler

# define the logger
logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

Default handler

By default, the RedisPubSubLogHandler will send the logs to a redis.Redis instance (running by default on localhost, port 6379) in a channel named “logs”.

Define a default Redis pub/sub handler and adding the handler to our logger

# define the default Redis pub/sub handler
handler = RedisPubSubLogHandler()
# add the handler to the logger
logger.addHandler(handler)

Subscribe to the log channel

Note that the logs emitted before subscription will be lost, so make sure to define a subscriber before publishing any logs.

from redis import Redis

r = Redis(decode_responses=True)
pubsub = r.pubsub()
pubsub.subscribe("logs")
pubsub.get_message(timeout=10)

{'type': 'subscribe', 'pattern': None, 'channel': 'logs', 'data': 1}

Emit some logs

logger.info("Some log message")
logger.info("Another log message")
logger.error("An error message!")

INFO:root:Some log message
INFO:root:Another log message
ERROR:root:An error message!

Retrieve the logs emited

for i in range(3):
    log = pubsub.get_message(timeout=10)
    print(f"Log {i} : ", log["data"])

Log 0 :  {"msg": "Some log message", "levelname": "INFO", "created": 1675346789.361666}
Log 1 :  {"msg": "Another log message", "levelname": "INFO", "created": 1675346789.3638494}
Log 2 :  {"msg": "An error message!", "levelname": "ERROR", "created": 1675346789.36544}

Custom channel name

By default, the logs are saved in a channel named “logs”, you can however change this by setting the channel_name parameter.

# handler with a custom stream name
handler = RedisPubSubLogHandler(channel_name="custom")

Change saved fields

By default the logs emitted are saved as a dict with the following fields:

• msg : the log message

• levelname : the log level

• created : the timestamp when the log has been created

But those fields can be tunned by specifying the fields parameter of RedisPubSubLogHandler. The fields specified must be valid LogRecord attributes (you can see the list of valid attributes in Python logging documentation).

# remove the previous handler
logger.removeHandler(handler)

# create a handler with fields msg, lineno and name
handler = RedisPubSubLogHandler(channel_name="custom_fields", fields=["msg", "lineno", "name"])
logger.addHandler(handler)

pubsub = r.pubsub()
pubsub.subscribe("custom_fields")
pubsub.get_message(timeout=10)

{'type': 'subscribe', 'pattern': None, 'channel': 'custom_fields', 'data': 1}

logger.info("Some log message")
logger.info("Another log message")
logger.error("An error message!")

INFO:root:Some log message
INFO:root:Another log message
ERROR:root:An error message!

for i in range(3):
    log = pubsub.get_message(timeout=10)
    print(f"Log {i} : ", log["data"])

Log 0 :  {"msg": "Some log message", "lineno": 1, "name": "root"}
Log 1 :  {"msg": "Another log message", "lineno": 2, "name": "root"}
Log 2 :  {"msg": "An error message!", "lineno": 3, "name": "root"}

The dict saved for each log now contains the custom fields specified earlier.

Save logs as pickle format

Rather than saving only some of the LogRecord attributes, you can save the whole object in their pickle format. This can be usefull if you need to re-use the logs in another Python program (pickle format is Python specific).

# remove the previous handler
logger.removeHandler(handler)

# create a handler that saves logs as pickle format
handler = RedisPubSubLogHandler(channel_name="pkl_logs", as_pkl=True)
logger.addHandler(handler)

r = Redis()
pubsub = r.pubsub()
pubsub.subscribe("pkl_logs")
pubsub.get_message(timeout=10)

{'type': 'subscribe', 'pattern': None, 'channel': b'pkl_logs', 'data': 1}

logger.info("Some log message")

INFO:root:Some log message

import pickle

record = pickle.loads(pubsub.get_message(timeout=10)["data"])
record.getMessage()

'Some log message'