bolster.utils.web

HTTP session utilities with retry and rate-limit handling.

Provides a pre-configured requests.Session (session) with:

• Automatic retry on transient server errors (500/502/503/504)

• Exponential backoff with jitter: 0s, 2–7s, 4–9s, 8–13s (~29s worst case)

• A consistent User-Agent header for polite scraping

• Helpers for downloading Excel files and ZIP archives in memory

All data-source modules should import session from here rather than calling requests.get() directly, so that retry logic is applied uniformly.

Example

>>> from bolster.utils.web import session
>>> type(session).__name__
'Session'

Attributes

logger
ua
session

Classes

RateLimitAwareRetry	Retry strategy that logs HTTP errors and connection failures for diagnosis.
CachingSession	requests.Session that caches GET and HEAD responses to disk with a TTL.

Functions

get_last_valid(url)	Get the last valid URL from Wayback Machine.
resilient_get(url, **kwargs)	Attempt a get, but if it fails, try using the wayback machine to get the last valid version and get that.
get_excel_dataframe(file_url[, requests_kwargs, ...])	Download and read Excel file into pandas DataFrame.
download_extract_zip(url)	Download a ZIP file and extract its contents in memory.

Module Contents

bolster.utils.web.logger

bolster.utils.web.ua

class bolster.utils.web.RateLimitAwareRetry(total=10, connect=None, read=None, redirect=None, status=None, other=None, allowed_methods=DEFAULT_ALLOWED_METHODS, status_forcelist=None, backoff_factor=0, backoff_max=DEFAULT_BACKOFF_MAX, raise_on_redirect=True, raise_on_status=True, history=None, respect_retry_after_header=True, remove_headers_on_redirect=DEFAULT_REMOVE_HEADERS_ON_REDIRECT, backoff_jitter=0.0, retry_after_max=DEFAULT_RETRY_AFTER_MAX)

Bases: urllib3.util.retry.Retry

Retry strategy that logs HTTP errors and connection failures for diagnosis.

increment(method=None, url=None, response=None, error=None, _pool=None, _stacktrace=None)

Override increment to track the last response status.

class bolster.utils.web.CachingSession

Bases: requests.Session

requests.Session that caches GET and HEAD responses to disk with a TTL.

GET: only caches un-parametered requests (no params= kwarg) whose Content-Type is a text-like format — specifically anything starting with “text/” (HTML, XML, plain text) or “application/json” or “application/rss+xml”. Binary downloads (Excel, ZIP, etc.) bypass the cache and should go through CachedDownloader instead. Calls passing params= are never cached: the cache key is derived from the base URL only, so caching a parametered request risks silently serving a different request’s response for the same cache slot (e.g. ?personId=1 vs ?personId=2) — see https://github.com/andrewbolster/bolster/pull/1948.

HEAD: caches the status code (no body) for un-parametered requests, including non-2xx codes — a module probing “does this direct file URL exist” before falling back to scraping a publication page would otherwise repeat the same live network round-trip on every retry even when the upstream is consistently down.

Cache lives in ~/.cache/bolster/_pages/ keyed by URL hash. TTL is controlled by _PAGE_CACHE_TTL_SECONDS (default: 1 hour).

Example

>>> from bolster.utils.web import session
>>> type(session).__name__
'CachingSession'

get(url, **kwargs)

Sends a GET request. Returns Response object.

Parameters:

• url – URL for the new Request object.

• params – (optional) Dictionary, list of tuples or bytes to send

in the query string for the Request. :param **kwargs: Optional arguments that request takes. :rtype: requests.Response

head(url, **kwargs)

Cache un-parametered HEAD status codes (no body to store).

Modules sometimes probe a candidate URL with HEAD before deciding whether to GET it (e.g. checking a direct-file URL exists before falling back to scraping a publication page). Without caching this, every retry/repeat call pays a full network round-trip just to learn the same status code as last time.

bolster.utils.web.session

bolster.utils.web.get_last_valid(url)

Get the last valid URL from Wayback Machine.

bolster.utils.web.resilient_get(url, **kwargs)

Attempt a get, but if it fails, try using the wayback machine to get the last valid version and get that.

If all else fails, raise a HTTPError from the inner “NoCDXRecordFound” exception.

bolster.utils.web.get_excel_dataframe(file_url, requests_kwargs=None, read_kwargs=None)

Download and read Excel file into pandas DataFrame.

bolster.utils.web.download_extract_zip(url)

Download a ZIP file and extract its contents in memory.

Yields (filename, file-like object) pairs. Shows a tqdm progress bar during download sized to Content-Length when the server provides it.