Handling Exceptions in Web Requests

Handling Exceptions in Web Requests

Making web requests is a core part of modern Python programming, whether you’re consuming an API, scraping a website, or integrating with external services. But the internet is unpredictable—servers go down, connections time out, data formats change, and errors happen. If your code isn’t prepared, a single failed request can crash your entire application.

That’s where exception handling comes in. By anticipating and gracefully managing errors, you can build applications that are not only functional but resilient and user-friendly. Let’s look at how to properly handle exceptions when making web requests in Python.

Common Web Request Exceptions

When using the popular requests library, several exceptions can occur. Being familiar with them will help you write better error-handling code.

ConnectionError: This occurs when a network problem happens, like DNS failure or refused connection.

Timeout: The server didn’t respond in time.

HTTPError: For HTTP error responses like 404 or 500 status codes.

RequestException: The base exception for all requests-related errors.

Here’s a basic example of making a request without any error handling:

import requests

response = requests.get('https://api.example.com/data')
data = response.json()
print(data)

This code is brittle. If the request fails, an exception will be raised and likely crash your program.

Basic Exception Handling

Let’s improve the previous example by adding a try-except block:

import requests
from requests.exceptions import RequestException

try:
    response = requests.get('https://api.example.com/data')
    response.raise_for_status()  # Raises HTTPError for bad status codes
    data = response.json()
    print(data)
except RequestException as e:
    print(f"Request failed: {e}")

The raise_for_status() method is particularly useful—it will raise an HTTPError if the response status code indicates a failure (4xx or 5xx). This allows you to handle both network errors and HTTP errors in the same exception block.

Specific Exception Handling

For more granular control, you can catch specific exceptions:

import requests
from requests.exceptions import Timeout, ConnectionError, HTTPError

try:
    response = requests.get('https://api.example.com/data', timeout=5)
    response.raise_for_status()
    data = response.json()
    print(data)
except Timeout:
    print("The request timed out")
except ConnectionError:
    print("Network problem occurred")
except HTTPError as e:
    print(f"HTTP error occurred: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

This approach lets you handle different error types with specific responses, which is valuable for logging, user messaging, or implementing fallback strategies.

Working with JSON Responses

Even when a request succeeds, you might encounter issues with the response content. A common pitfall is assuming the response will always contain valid JSON:

import requests
from requests.exceptions import RequestException
import json

try:
    response = requests.get('https://api.example.com/data')
    response.raise_for_status()

    try:
        data = response.json()
        print(data)
    except json.JSONDecodeError:
        print("Received response is not valid JSON")

except RequestException as e:
    print(f"Request failed: {e}")

This nested try-except structure ensures you handle both request failures and content parsing issues.

Exception Type Common Causes Recommended Action
ConnectionError Network issues, DNS failure Retry with backoff, check network connection
Timeout Server too slow, network congestion Increase timeout, implement retry logic
HTTPError 404, 500 status codes Handle specific status codes appropriately
JSONDecodeError Invalid JSON response Validate response content, use fallback data

Implementing Retry Logic

For transient errors, implementing retry logic can significantly improve reliability. Here's a simple implementation:

import requests
from requests.exceptions import RequestException
import time

def make_request_with_retry(url, max_retries=3, backoff_factor=1):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.json()
        except RequestException as e:
            if attempt == max_retries - 1:
                raise e
            wait_time = backoff_factor * (2 ** attempt)
            print(f"Request failed, retrying in {wait_time} seconds...")
            time.sleep(wait_time)

    return None

# Usage
data = make_request_with_retry('https://api.example.com/data')
if data:
    print(data)

For production use, consider using the urllib3.util.retry module or the tenacity library for more sophisticated retry functionality.

Handling Different HTTP Status Codes

Different HTTP status codes often require different handling strategies. Here's how you might handle some common ones:

import requests
from requests.exceptions import RequestException

try:
    response = requests.get('https://api.example.com/user/123')
    response.raise_for_status()

except requests.HTTPError as e:
    if response.status_code == 404:
        print("User not found")
    elif response.status_code == 401:
        print("Authentication required")
    elif response.status_code == 403:
        print("Access forbidden")
    elif response.status_code == 429:
        print("Rate limited - too many requests")
    elif 500 <= response.status_code < 600:
        print("Server error - try again later")
    else:
        print(f"HTTP error: {e}")

except RequestException as e:
    print(f"Request failed: {e}")

Timeout Management

Always set timeouts on your requests to prevent them from hanging indefinitely:

import requests

# Timeout for entire request (connect + read)
try:
    response = requests.get('https://api.example.com/data', timeout=5.0)
except requests.Timeout:
    print("Request timed out")

# Separate connect and read timeouts
try:
    response = requests.get('https://api.example.com/data', timeout=(3.05, 27))
except requests.Timeout:
    print("Connect or read operation timed out")

The timeout parameter can be a single float (total timeout) or a tuple (connect timeout, read timeout).

Best practices for web request exception handling:

  • Always use explicit timeouts
  • Implement appropriate retry logic for transient errors
  • Handle different HTTP status codes specifically
  • Validate response content before processing
  • Use connection pooling for multiple requests
  • Log errors with sufficient context for debugging
  • Consider circuit breakers for frequently failing endpoints

Advanced Error Handling Patterns

For complex applications, you might want to implement more sophisticated patterns:

import requests
from requests.exceptions import RequestException
from functools import wraps

def handle_request_errors(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except requests.HTTPError as e:
            # Handle specific HTTP errors
            if e.response.status_code == 404:
                return {"error": "Resource not found"}
            elif e.response.status_code == 429:
                return {"error": "Rate limited"}
            else:
                return {"error": f"HTTP error: {e}"}
        except requests.Timeout:
            return {"error": "Request timed out"}
        except requests.ConnectionError:
            return {"error": "Network connection failed"}
        except RequestException as e:
            return {"error": f"Request failed: {e}"}
    return wrapper

@handle_request_errors
def fetch_data(url):
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return response.json()

# Usage
result = fetch_data('https://api.example.com/data')
if 'error' in result:
    print(f"Error: {result['error']}")
else:
    print(f"Data: {result}")

This decorator pattern allows you to reuse error handling logic across multiple request functions.

Working with Sessions and Persistent Connections

When making multiple requests to the same API, using sessions can improve performance and provide better error handling:

import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import RequestException
from urllib3.util.retry import Retry

def create_retry_session(retries=3, backoff_factor=0.3):
    session = requests.Session()
    retry_strategy = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=[429, 500, 502, 503, 504],
    )
    adapter = HTTPAdapter(max_retries=retry_strategy)
    session.mount("http://", adapter)
    session.mount("https://", adapter)
    return session

# Usage
session = create_retry_session()
try:
    response = session.get('https://api.example.com/data', timeout=5)
    response.raise_for_status()
    data = response.json()
except RequestException as e:
    print(f"Request failed after retries: {e}")

This approach automatically retries on specific HTTP status codes that often indicate temporary issues.

Handling Large Responses and Memory Errors

When working with large responses, you might encounter memory-related issues. Here's how to handle them gracefully:

import requests
from requests.exceptions import RequestException, ChunkedEncodingError

try:
    response = requests.get('https://api.example.com/large-data', stream=True)
    response.raise_for_status()

    # Process response in chunks to avoid memory issues
    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            # Process each chunk
            process_chunk(chunk)

except ChunkedEncodingError:
    print("Error in chunked transfer encoding")
except MemoryError:
    print("Insufficient memory to handle response")
except RequestException as e:
    print(f"Request failed: {e}")

The stream=True parameter and iter_content() method allow you to handle large responses without loading everything into memory at once.

When dealing with web requests, there are several critical considerations for proper exception handling. First, always validate your error handling with different types of failures to ensure your application behaves correctly under various error conditions. Second, implement comprehensive logging to capture enough context about failures for effective debugging. Third, consider user experience by providing appropriate feedback or fallback content when requests fail.

Error Scenario Detection Method Recommended Response
Network connectivity issues ConnectionError Retry with exponential backoff
Server timeouts Timeout exception Retry with increased timeout values
Rate limiting HTTP 429 status code Implement rate limit awareness and backoff
Server errors HTTP 5xx status codes Retry with caution, implement circuit breakers
Client errors HTTP 4xx status codes Review request parameters, don't retry

Testing Your Exception Handling

Proper testing is crucial for ensuring your error handling works as expected. Here's how you might test different error scenarios:

import unittest
from unittest.mock import patch
import requests
from requests.exceptions import Timeout, ConnectionError
from your_module import fetch_data

class TestRequestHandling(unittest.TestCase):

    @patch('your_module.requests.get')
    def test_timeout_handling(self, mock_get):
        mock_get.side_effect = Timeout("Request timed out")

        result = fetch_data('https://api.example.com/data')
        self.assertIn('error', result)
        self.assertIn('timed out', result['error'])

    @patch('your_module.requests.get')
    def test_connection_error_handling(self, mock_get):
        mock_get.side_effect = ConnectionError("Network problem")

        result = fetch_data('https://api.example.com/data')
        self.assertIn('error', result)
        self.assertIn('network', result['error'].lower())

    @patch('your_module.requests.get')
    def test_http_error_handling(self, mock_get):
        mock_response = unittest.mock.Mock()
        mock_response.raise_for_status.side_effect = requests.HTTPError("404 Client Error")
        mock_response.status_code = 404
        mock_get.return_value = mock_response

        result = fetch_data('https://api.example.com/data')
        self.assertIn('error', result)
        self.assertIn('not found', result['error'].lower())

if __name__ == '__main__':
    unittest.main()

This testing approach ensures your exception handling logic works correctly for various error conditions.

Real-World Implementation Considerations

In production applications, consider these additional factors for robust web request handling:

Circuit Breakers: Implement circuit breakers to stop making requests to failing services temporarily.

Fallback Content: Provide fallback data or cached responses when live requests fail.

Monitoring and Alerting: Set up monitoring for request failure rates and alert on abnormal patterns.

User Feedback: Provide clear, actionable error messages to end-users when appropriate.

Rate Limit Awareness: Implement logic to respect API rate limits and handle 429 responses gracefully.

By implementing comprehensive exception handling for your web requests, you'll create applications that are more reliable, user-friendly, and maintainable. The internet will always be unpredictable, but with proper error handling, your Python applications can handle whatever comes their way.