How can the Python built‑in logging module be configured and used to meet audit requirements in a financial data pipeline? What log levels and handlers would you recommend?

Question

Assisting AI · Accepted Answer

Audit logging in a financial pipeline requires immutable, tamper‑evident records that capture who did what and when. Configure the Python logging module with a dedicated audit logger:

```python
import logging, logging.config
config = {
    'version': 1,
    'formatters': {
        'audit': {
            'format': '%(asctime)s %(levelname)s %(name)s %(message)s'
        }
    },
    'handlers': {
        'audit_file': {
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/finance/audit.log',
            'maxBytes': 10485760,
            'backupCount': 5,
            'formatter': 'audit',
            'level': 'INFO'
        }
    },
    'loggers': {
        'audit': {
            'handlers': ['audit_file'],
            'level': 'INFO',
            'propagate': False
        }
    }
}
logging.config.dictConfig(config)
logger = logging.getLogger('audit')
```

Use INFO for normal data ingestion, WARNING for data anomalies, ERROR for processing failures, and CRITICAL for system outages. Include structured fields (e.g., transaction_id, user_id) in the message or use JSON formatting for easier parsing. Rotate logs to prevent disk exhaustion and consider writing to a write‑once storage (e.g., S3 with versioning) for compliance. This setup satisfies most audit requirements while keeping the codebase simple.

How can the Python built‑in logging module be configured and used to meet audit requirements in a financial data pipeline? What log levels and handlers would you recommend?

💡 Model Answer

🎤 Get questions like this answered in real-time