
Handling Exceptions in Web Requests
Making web requests is a core part of modern Python programming, whether you’re consuming an API, scraping a website, or integrating with external services. But the internet is unpredictable—servers go down, connections time out, data formats change, and errors happen. If your code isn’t prepared, a single failed request can crash your entire application.
That’s where exception handling comes in. By anticipating and gracefully managing errors, you can build applications that are not only functional but resilient and user-friendly. Let’s look at how to properly handle exceptions when making web requests in Python.
Common Web Request Exceptions
When using the popular requests
library, several exceptions can occur. Being familiar with them will help you write better error-handling code.
ConnectionError: This occurs when a network problem happens, like DNS failure or refused connection.
Timeout: The server didn’t respond in time.
HTTPError: For HTTP error responses like 404 or 500 status codes.
RequestException: The base exception for all requests-related errors.
Here’s a basic example of making a request without any error handling:
import requests
response = requests.get('https://api.example.com/data')
data = response.json()
print(data)
This code is brittle. If the request fails, an exception will be raised and likely crash your program.
Basic Exception Handling
Let’s improve the previous example by adding a try-except block:
import requests
from requests.exceptions import RequestException
try:
response = requests.get('https://api.example.com/data')
response.raise_for_status() # Raises HTTPError for bad status codes
data = response.json()
print(data)
except RequestException as e:
print(f"Request failed: {e}")
The raise_for_status()
method is particularly useful—it will raise an HTTPError if the response status code indicates a failure (4xx or 5xx). This allows you to handle both network errors and HTTP errors in the same exception block.
Specific Exception Handling
For more granular control, you can catch specific exceptions:
import requests
from requests.exceptions import Timeout, ConnectionError, HTTPError
try:
response = requests.get('https://api.example.com/data', timeout=5)
response.raise_for_status()
data = response.json()
print(data)
except Timeout:
print("The request timed out")
except ConnectionError:
print("Network problem occurred")
except HTTPError as e:
print(f"HTTP error occurred: {e}")
except Exception as e:
print(f"An unexpected error occurred: {e}")
This approach lets you handle different error types with specific responses, which is valuable for logging, user messaging, or implementing fallback strategies.
Working with JSON Responses
Even when a request succeeds, you might encounter issues with the response content. A common pitfall is assuming the response will always contain valid JSON:
import requests
from requests.exceptions import RequestException
import json
try:
response = requests.get('https://api.example.com/data')
response.raise_for_status()
try:
data = response.json()
print(data)
except json.JSONDecodeError:
print("Received response is not valid JSON")
except RequestException as e:
print(f"Request failed: {e}")
This nested try-except structure ensures you handle both request failures and content parsing issues.
Exception Type | Common Causes | Recommended Action |
---|---|---|
ConnectionError | Network issues, DNS failure | Retry with backoff, check network connection |
Timeout | Server too slow, network congestion | Increase timeout, implement retry logic |
HTTPError | 404, 500 status codes | Handle specific status codes appropriately |
JSONDecodeError | Invalid JSON response | Validate response content, use fallback data |
Implementing Retry Logic
For transient errors, implementing retry logic can significantly improve reliability. Here's a simple implementation:
import requests
from requests.exceptions import RequestException
import time
def make_request_with_retry(url, max_retries=3, backoff_factor=1):
for attempt in range(max_retries):
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
return response.json()
except RequestException as e:
if attempt == max_retries - 1:
raise e
wait_time = backoff_factor * (2 ** attempt)
print(f"Request failed, retrying in {wait_time} seconds...")
time.sleep(wait_time)
return None
# Usage
data = make_request_with_retry('https://api.example.com/data')
if data:
print(data)
For production use, consider using the urllib3.util.retry
module or the tenacity
library for more sophisticated retry functionality.
Handling Different HTTP Status Codes
Different HTTP status codes often require different handling strategies. Here's how you might handle some common ones:
import requests
from requests.exceptions import RequestException
try:
response = requests.get('https://api.example.com/user/123')
response.raise_for_status()
except requests.HTTPError as e:
if response.status_code == 404:
print("User not found")
elif response.status_code == 401:
print("Authentication required")
elif response.status_code == 403:
print("Access forbidden")
elif response.status_code == 429:
print("Rate limited - too many requests")
elif 500 <= response.status_code < 600:
print("Server error - try again later")
else:
print(f"HTTP error: {e}")
except RequestException as e:
print(f"Request failed: {e}")
Timeout Management
Always set timeouts on your requests to prevent them from hanging indefinitely:
import requests
# Timeout for entire request (connect + read)
try:
response = requests.get('https://api.example.com/data', timeout=5.0)
except requests.Timeout:
print("Request timed out")
# Separate connect and read timeouts
try:
response = requests.get('https://api.example.com/data', timeout=(3.05, 27))
except requests.Timeout:
print("Connect or read operation timed out")
The timeout parameter can be a single float (total timeout) or a tuple (connect timeout, read timeout).
Best practices for web request exception handling:
- Always use explicit timeouts
- Implement appropriate retry logic for transient errors
- Handle different HTTP status codes specifically
- Validate response content before processing
- Use connection pooling for multiple requests
- Log errors with sufficient context for debugging
- Consider circuit breakers for frequently failing endpoints
Advanced Error Handling Patterns
For complex applications, you might want to implement more sophisticated patterns:
import requests
from requests.exceptions import RequestException
from functools import wraps
def handle_request_errors(func):
@wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except requests.HTTPError as e:
# Handle specific HTTP errors
if e.response.status_code == 404:
return {"error": "Resource not found"}
elif e.response.status_code == 429:
return {"error": "Rate limited"}
else:
return {"error": f"HTTP error: {e}"}
except requests.Timeout:
return {"error": "Request timed out"}
except requests.ConnectionError:
return {"error": "Network connection failed"}
except RequestException as e:
return {"error": f"Request failed: {e}"}
return wrapper
@handle_request_errors
def fetch_data(url):
response = requests.get(url, timeout=10)
response.raise_for_status()
return response.json()
# Usage
result = fetch_data('https://api.example.com/data')
if 'error' in result:
print(f"Error: {result['error']}")
else:
print(f"Data: {result}")
This decorator pattern allows you to reuse error handling logic across multiple request functions.
Working with Sessions and Persistent Connections
When making multiple requests to the same API, using sessions can improve performance and provide better error handling:
import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import RequestException
from urllib3.util.retry import Retry
def create_retry_session(retries=3, backoff_factor=0.3):
session = requests.Session()
retry_strategy = Retry(
total=retries,
backoff_factor=backoff_factor,
status_forcelist=[429, 500, 502, 503, 504],
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)
return session
# Usage
session = create_retry_session()
try:
response = session.get('https://api.example.com/data', timeout=5)
response.raise_for_status()
data = response.json()
except RequestException as e:
print(f"Request failed after retries: {e}")
This approach automatically retries on specific HTTP status codes that often indicate temporary issues.
Handling Large Responses and Memory Errors
When working with large responses, you might encounter memory-related issues. Here's how to handle them gracefully:
import requests
from requests.exceptions import RequestException, ChunkedEncodingError
try:
response = requests.get('https://api.example.com/large-data', stream=True)
response.raise_for_status()
# Process response in chunks to avoid memory issues
for chunk in response.iter_content(chunk_size=8192):
if chunk:
# Process each chunk
process_chunk(chunk)
except ChunkedEncodingError:
print("Error in chunked transfer encoding")
except MemoryError:
print("Insufficient memory to handle response")
except RequestException as e:
print(f"Request failed: {e}")
The stream=True
parameter and iter_content()
method allow you to handle large responses without loading everything into memory at once.
When dealing with web requests, there are several critical considerations for proper exception handling. First, always validate your error handling with different types of failures to ensure your application behaves correctly under various error conditions. Second, implement comprehensive logging to capture enough context about failures for effective debugging. Third, consider user experience by providing appropriate feedback or fallback content when requests fail.
Error Scenario | Detection Method | Recommended Response |
---|---|---|
Network connectivity issues | ConnectionError | Retry with exponential backoff |
Server timeouts | Timeout exception | Retry with increased timeout values |
Rate limiting | HTTP 429 status code | Implement rate limit awareness and backoff |
Server errors | HTTP 5xx status codes | Retry with caution, implement circuit breakers |
Client errors | HTTP 4xx status codes | Review request parameters, don't retry |
Testing Your Exception Handling
Proper testing is crucial for ensuring your error handling works as expected. Here's how you might test different error scenarios:
import unittest
from unittest.mock import patch
import requests
from requests.exceptions import Timeout, ConnectionError
from your_module import fetch_data
class TestRequestHandling(unittest.TestCase):
@patch('your_module.requests.get')
def test_timeout_handling(self, mock_get):
mock_get.side_effect = Timeout("Request timed out")
result = fetch_data('https://api.example.com/data')
self.assertIn('error', result)
self.assertIn('timed out', result['error'])
@patch('your_module.requests.get')
def test_connection_error_handling(self, mock_get):
mock_get.side_effect = ConnectionError("Network problem")
result = fetch_data('https://api.example.com/data')
self.assertIn('error', result)
self.assertIn('network', result['error'].lower())
@patch('your_module.requests.get')
def test_http_error_handling(self, mock_get):
mock_response = unittest.mock.Mock()
mock_response.raise_for_status.side_effect = requests.HTTPError("404 Client Error")
mock_response.status_code = 404
mock_get.return_value = mock_response
result = fetch_data('https://api.example.com/data')
self.assertIn('error', result)
self.assertIn('not found', result['error'].lower())
if __name__ == '__main__':
unittest.main()
This testing approach ensures your exception handling logic works correctly for various error conditions.
Real-World Implementation Considerations
In production applications, consider these additional factors for robust web request handling:
Circuit Breakers: Implement circuit breakers to stop making requests to failing services temporarily.
Fallback Content: Provide fallback data or cached responses when live requests fail.
Monitoring and Alerting: Set up monitoring for request failure rates and alert on abnormal patterns.
User Feedback: Provide clear, actionable error messages to end-users when appropriate.
Rate Limit Awareness: Implement logic to respect API rate limits and handle 429 responses gracefully.
By implementing comprehensive exception handling for your web requests, you'll create applications that are more reliable, user-friendly, and maintainable. The internet will always be unpredictable, but with proper error handling, your Python applications can handle whatever comes their way.