Stream log handler example

This example will show you how to forward logs into a Redis Stream using rlh.RedisStreamHandler.

Logger setup

import logging
from rlh import RedisStreamLogHandler

# define the logger
logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

Default handler

By default, the RedisStreamLogHandler will send the logs to a redis.Redis instance (running by default on localhost, port 6379) in a stream named “logs”.

Define a default Redis stream handler and adding the handler to our logger

# define the default Redis stream handler
handler = RedisStreamLogHandler()
# add the handler to the logger
logger.addHandler(handler)

Emit some logs

logger.info("Some log message")
logger.info("Another log message")
logger.error("An error message!")

INFO:root:Some log message
INFO:root:Another log message
ERROR:root:An error message!

Retrieve the logs emited

from redis import Redis

r = Redis(decode_responses=True)
r.xrange("logs", "-", "+")

[('1676638733992-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638733.9901333'}),
 ('1676638733994-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638733.992978'}),
 ('1676638733996-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638733.9949841'}),
 ('1676638768379-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638768.377096'}),
 ('1676638768380-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638768.3795207'}),
 ('1676638768382-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638768.381374'}),
 ('1676638811791-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638811.7903416'}),
 ('1676638811792-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638811.7914383'}),
 ('1676638811793-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638811.7926123'}),
 ('1676638872008-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638872.007143'}),
 ('1676638872009-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638872.0087047'}),
 ('1676638872010-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638872.0101254'}),
 ('1676638903137-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638903.1366746'}),
 ('1676638903139-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638903.138543'}),
 ('1676638903140-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638903.1398375'}),
 ('1676638919216-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638919.2142808'}),
 ('1676638919219-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638919.2176497'}),
 ('1676638919220-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638919.2195425'}),
 ('1676638919373-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638919.3716154'}),
 ('1676638919376-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638919.3746507'}),
 ('1676638919378-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638919.3776107'}),
 ('1676638919517-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638919.5143547'}),
 ('1676638919519-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676638919.5185797'}),
 ('1676638919522-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676638919.5209208'}),
 ('1676638919639-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676638919.6361604'}),
 ('1676639089881-0',
  {'msg': 'Some log message',
   'levelname': 'INFO',
   'created': '1676639089.8804429'}),
 ('1676639089884-0',
  {'msg': 'Another log message',
   'levelname': 'INFO',
   'created': '1676639089.882255'}),
 ('1676639089886-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676639089.8847578'})]

Custom stream name

By default, the logs are saved in a stream named “logs”, you can however change this by setting the stream_name parameter.

# handler with a custom stream name
handler = RedisStreamLogHandler(stream_name="custom")

Set stream maximum length

You can set a maximum length for the stream with maxlen parameter, so it does not grow indefinitely. By default this maximum length is approximate, which means that if the maximum length is set to n the stream will contain at least n entries, but can contain more. You can however force the stream to contain exactly maxlen entries by setting approximate parameter to False; but this is not recommended as it can create performance issues as stated in Redis documentation (see Redis documentation for more information).

# remove the previous handler
logger.removeHandler(handler)

# handler with a fixed maximum length of 1
handler = RedisStreamLogHandler(stream_name="maxlen", maxlen=1, approximate=False)
logger.addHandler(handler)

# adding some logs
logger.info("Some log message")
logger.info("Another log message")
logger.error("An error message!")

INFO:root:Some log message
INFO:root:Another log message
ERROR:root:An error message!

# retrieve all the logs in Redis stream
r.xrange("maxlen", "-", "+")

[('1676639090037-0',
  {'msg': 'An error message!',
   'levelname': 'ERROR',
   'created': '1676639090.0358083'})]

Only one message has been kept in the Redis stream.

Change saved fields

By default the logs emitted are saved as a dict with the following fields:

• msg : the log message

• levelname : the log level

• created : the timestamp when the log has been created

But those fields can be tunned by specifying the fields parameter of RedisStreamLogHandler. The fields specified must be valid LogRecord attributes (you can see the list of valid attributes in Python logging documentation).

# remove the previous handler
logger.removeHandler(handler)

# create a handler with fields msg, lineno and name
handler = RedisStreamLogHandler(stream_name="custom_fields", fields=["msg", "lineno", "name"])
logger.addHandler(handler)

logger.info("Some log message")
logger.info("Another log message")
logger.error("An error message!")

INFO:root:Some log message
INFO:root:Another log message
ERROR:root:An error message!

r.xrange("custom_fields", "-", "+")

[('1676638919518-0',
  {'msg': 'Some log message', 'lineno': '1', 'name': 'root'}),
 ('1676638919520-0',
  {'msg': 'Another log message', 'lineno': '2', 'name': 'root'}),
 ('1676638919522-0',
  {'msg': 'An error message!', 'lineno': '3', 'name': 'root'}),
 ('1676639090209-0',
  {'msg': 'Some log message', 'lineno': '1', 'name': 'root'}),
 ('1676639090211-0',
  {'msg': 'Another log message', 'lineno': '2', 'name': 'root'}),
 ('1676639090213-0',
  {'msg': 'An error message!', 'lineno': '3', 'name': 'root'})]

The dict saved for each log now contains the custom fields specified earlier.

Save logs as pickle format

Rather than saving only some of the LogRecord attributes, you can save the whole object in their pickle format. This can be usefull if you need to re-use the logs in another Python program (pickle format is Python specific).

# remove the previous handler
logger.removeHandler(handler)

# create a handler that saves logs as pickle format
handler = RedisStreamLogHandler(stream_name="pkl_logs", as_pkl=True)
logger.addHandler(handler)

logger.info("Some log message")

INFO:root:Some log message

import pickle

r = Redis()
record = pickle.loads(r.xrange("pkl_logs", "-", "+")[0][1][b"pkl"])
record.getMessage()

'Some log message'