Combining Exception Handling with Logging

Exception handling and logging are two fundamental pillars of robust Python programming. While exception handling helps your program deal with unexpected situations gracefully, logging provides the visibility you need to understand what happened and why. When combined effectively, they create a powerful system for building reliable, maintainable applications.

Why Combine Exception Handling with Logging?

Simply catching exceptions isn't enough - you need to know when and why they occur. Logging provides the perfect mechanism to record exception details without disrupting your program's flow. Instead of just printing error messages to the console, which might get missed, logging allows you to: - Persist error information for later analysis - Capture contextual data around the exception - Control where and how errors are recorded - Add severity levels to different types of exceptions

Let's start with a basic example of exception handling without logging:

try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero!")

This approach has several limitations. The error message might get lost among other output, and you lose valuable debugging information like the traceback.

Basic Integration of Logging and Exceptions

Here's how you can enhance this with Python's built-in logging module:

import logging

# Configure basic logging
logging.basicConfig(level=logging.ERROR, filename='app.log')

try:
    result = 10 / 0
except ZeroDivisionError as e:
    logging.error("Division by zero attempted: %s", e)

This approach saves the error to a file, but we're still missing the traceback. Let's improve it:

import logging
import traceback

logging.basicConfig(level=logging.ERROR, filename='app.log')

try:
    result = 10 / 0
except ZeroDivisionError:
    logging.error("Division error occurred:\n%s", traceback.format_exc())

Logging Level	Usage with Exceptions	Typical Scenario
ERROR	Expected exceptions	Division by zero, file not found
WARNING	Recoverable issues	Deprecated API usage
CRITICAL	Unrecoverable errors	Database connection lost
INFO	Normal operation	Successful operations
DEBUG	Detailed debugging	Step-by-step execution

The real power comes when you use the exc_info parameter:

import logging

logging.basicConfig(level=logging.ERROR, filename='app.log')

try:
    result = 10 / 0
except ZeroDivisionError:
    logging.error("Division by zero error", exc_info=True)

This automatically captures the full traceback and includes it in the log.

Structured Exception Logging

For more complex applications, you'll want to create a more structured approach:

import logging

# Create a dedicated logger for exceptions
exception_logger = logging.getLogger('exception_handler')
exception_logger.setLevel(logging.ERROR)

# Create file handler
file_handler = logging.FileHandler('exceptions.log')
file_handler.setLevel(logging.ERROR)

# Create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)

# Add handler to logger
exception_logger.addHandler(file_handler)

def safe_divide(a, b):
    try:
        return a / b
    except ZeroDivisionError as e:
        exception_logger.error("Division by zero in safe_divide: %s/%s", a, b, exc_info=True)
        return None
    except TypeError as e:
        exception_logger.error("Type error in safe_divide: %s", e, exc_info=True)
        return None

Key benefits of using logging with exceptions: - Centralized error tracking across your application - Configurable output destinations (files, console, network) - Automatic timestamping of when errors occurred - Flexible formatting of error messages - Different log levels for different severity of issues

Contextual Logging with Exceptions

Often, you need more context than just the exception itself. Here's how to add contextual information:

import logging
from logging import LoggerAdapter

class ContextLogger:
    def __init__(self, logger, context):
        self.logger = logger
        self.context = context

    def error(self, msg, *args, **kwargs):
        if 'extra' not in kwargs:
            kwargs['extra'] = {}
        kwargs['extra'].update(self.context)
        self.logger.error(msg, *args, **kwargs)

# Usage
logger = logging.getLogger('app')
context_logger = ContextLogger(logger, {'user_id': 123, 'operation': 'file_processing'})

try:
    # Some operation that might fail
    process_file('data.txt')
except Exception as e:
    context_logger.error("Failed to process file: %s", e, exc_info=True)

Context Field	Example Value	Purpose
user_id	12345	Identify affected user
request_id	req-abc123	Track specific request
module	data_processor	Locate source of error
operation	file_upload	Identify failed operation
timestamp	2023-12-01T10:30:00	Exact time of failure

Advanced Exception Handling Patterns

For enterprise applications, consider these patterns:

import logging
from functools import wraps

def log_exceptions(logger):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            try:
                return func(*args, **kwargs)
            except Exception as e:
                logger.error(
                    "Exception in %s: %s", 
                    func.__name__, 
                    e, 
                    exc_info=True,
                    extra={'args': args, 'kwargs': kwargs}
                )
                raise  # Re-raise the exception
        return wrapper
    return decorator

# Usage
logger = logging.getLogger('app')

@log_exceptions(logger)
def process_data(data):
    # Your code here
    return data.upper()

Best practices for combining logging and exceptions: - Always include tracebacks for debugging - Add relevant context to help with investigation - Use appropriate log levels for different exception types - Don't log sensitive information in error messages - Consider using structured logging formats like JSON - Rotate log files to prevent disk space issues - Set up alerts for critical errors

Handling Specific Exception Types

Different exceptions should often be handled and logged differently:

import logging

logger = logging.getLogger('app')

def handle_file_operations(filename):
    try:
        with open(filename, 'r') as file:
            content = file.read()
            return process_content(content)

    except FileNotFoundError:
        logger.warning("File not found: %s", filename)
        return None

    except PermissionError:
        logger.error("Permission denied for file: %s", filename)
        raise  # Re-raise critical errors

    except UnicodeDecodeError as e:
        logger.error("Encoding error in file %s: %s", filename, e, exc_info=True)
        return None

    except Exception as e:
        logger.critical("Unexpected error processing file %s: %s", filename, e, exc_info=True)
        raise

Logging Configuration Best Practices

A well-configured logging setup is crucial:

import logging
import logging.config

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'detailed': {
            'format': '%(asctime)s %(name)-15s %(levelname)-8s %(message)s'
        }
    },
    'handlers': {
        'file': {
            'class': 'logging.FileHandler',
            'level': 'ERROR',
            'formatter': 'detailed',
            'filename': 'errors.log',
        },
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'INFO',
            'formatter': 'detailed',
        }
    },
    'root': {
        'level': 'INFO',
        'handlers': ['file', 'console']
    },
}

logging.config.dictConfig(LOGGING_CONFIG)

Critical considerations for production logging: - Separate error logs from regular application logs - Implement log rotation to manage file sizes - Use appropriate log levels for different environments - Include correlation IDs for distributed systems - Monitor log files for error patterns - Regularly review and clean up old log files

Real-World Example: Database Operations

Here's a practical example combining exception handling and logging for database operations:

import logging
import psycopg2
from psycopg2 import OperationalError, ProgrammingError

logger = logging.getLogger('database')

class DatabaseManager:
    def __init__(self, connection_string):
        self.connection_string = connection_string
        self.connection = None

    def connect(self):
        try:
            self.connection = psycopg2.connect(self.connection_string)
            logger.info("Database connection established")
            return True

        except OperationalError as e:
            logger.error("Failed to connect to database: %s", e, exc_info=True)
            return False

    def execute_query(self, query, params=None):
        try:
            with self.connection.cursor() as cursor:
                cursor.execute(query, params or ())
                result = cursor.fetchall()
                logger.debug("Query executed successfully: %s", query)
                return result

        except ProgrammingError as e:
            logger.error("SQL error in query %s: %s", query, e, exc_info=True)
            raise

        except OperationalError as e:
            logger.critical("Database connection lost: %s", e, exc_info=True)
            self.reconnect()
            raise

        except Exception as e:
            logger.error("Unexpected error executing query: %s", e, exc_info=True)
            raise

    def reconnect(self):
        logger.warning("Attempting to reconnect to database")
        self.connect()

This approach ensures that: - Connection errors are logged as errors - SQL syntax errors are logged with full tracebacks - Connection loss triggers critical logs and reconnection attempts - Successful operations are logged at debug level - Unexpected errors are captured with complete context

By combining exception handling with strategic logging, you create applications that not only handle errors gracefully but also provide the diagnostic information needed to fix issues quickly and prevent them from recurring.