bolster.data_sources.daera_waste

DAERA NI Local Authority Collected (LAC) Municipal Waste Statistics.

Quarterly time-series data on local-authority-collected municipal waste management across Northern Ireland, published by the Department of Agriculture, Environment and Rural Affairs (DAERA).

Data Source:

Discovery page: https://www.daera-ni.gov.uk/publications/northern-ireland-local-authority-collected-municipal-waste-management-statistics-time-series-data

The module scrapes the DAERA publications page to auto-discover the current CSV URL (which changes with each release, e.g. 2026-04/...). It then downloads the time-series CSV and returns a tidy long-format DataFrame.

Update Frequency:

Quarterly (provisional) with finalised annual revisions. The current series runs from Q1 2006/07 to the most recent available quarter.

Geographic Coverage:

All NI council areas including both pre- and post-2015 boundaries, plus a Northern Ireland aggregate row. The 11 post-2015 LGD councils are: Antrim & Newtownabbey, Ards & North Down, Armagh City Banbridge & Craigavon, Belfast, Causeway Coast & Glens, Derry City & Strabane, Fermanagh & Omagh, Lisburn & Castlereagh, Mid & East Antrim, Mid Ulster, Newry Mourne & Down.

Example

>>> from bolster.data_sources import daera_waste
>>> df = daera_waste.get_latest_waste_statistics()
>>> 'council_area' in df.columns
True
>>> 'tonnes' in df.columns
True

Attributes

`logger`
`DAERA_PUBLICATION_PAGE`
`DAERA_BASE_URL`
`NI_COUNCILS_POST_2015`

Exceptions

`DAERADataNotFoundError`	DAERA data file or publication page could not be located.
`DAERAValidationError`	DAERA DataFrame failed validation checks.

Functions

`get_waste_publication_url`([prefer])	Scrape the DAERA publications page for the latest LAC waste CSV/Excel URL.
`parse_waste_file`(file_path)	Parse a DAERA LAC municipal waste time-series CSV file.
`get_latest_waste_statistics`([force_refresh])	Download and parse the latest DAERA LAC municipal waste statistics.
`validate_waste_data`(df)	Validate a DAERA LAC municipal waste DataFrame.

Module Contents

bolster.data_sources.daera_waste.logger[source]

bolster.data_sources.daera_waste.DAERA_PUBLICATION_PAGE = 'https://www.daera-ni.gov.uk/publications/northern-ireland-local-authority-collected-municipal-wa...[source]

bolster.data_sources.daera_waste.DAERA_BASE_URL = 'https://www.daera-ni.gov.uk'[source]

bolster.data_sources.daera_waste.NI_COUNCILS_POST_2015[source]

exception bolster.data_sources.daera_waste.DAERADataNotFoundError[source]

Bases: Exception

DAERA data file or publication page could not be located.

Initialize self. See help(type(self)) for accurate signature.

exception bolster.data_sources.daera_waste.DAERAValidationError[source]

Bases: Exception

DAERA DataFrame failed validation checks.

Initialize self. See help(type(self)) for accurate signature.

bolster.data_sources.daera_waste.get_waste_publication_url(prefer='csv')[source]

Scrape the DAERA publications page for the latest LAC waste CSV/Excel URL.

The URL contains a date component (e.g. 2026-04/) that changes with each release, so this function fetches the page and finds the current link.

Parameters:: prefer (str) – Preferred file type — "csv" (default) or "xlsx".
Returns:: Absolute URL of the latest time-series file.
Raises:: DAERADataNotFoundError – If the publication page cannot be fetched or no matching link is found.
Return type:: str

Example

>>> url = get_waste_publication_url()
>>> url.endswith(".csv") or url.endswith(".xlsx")
True
>>> "daera-ni.gov.uk" in url
True

bolster.data_sources.daera_waste.parse_waste_file(file_path)[source]

Parse a DAERA LAC municipal waste time-series CSV file.

Reads the CSV (which uses commas as thousands separators in numeric columns), renames columns to clean internal names, and returns a tidy long-format DataFrame.

Metadata columns (QuarterCode, QuarterName, FinancialYear, AreaCode, AreaName, WasteManagementGroup, DataStatus) are retained alongside all numeric waste metric columns.

Parameters:: file_path (str | pathlib.Path) – Path to a downloaded .csv waste time-series file.
Returns:: DataFrame with one row per (quarter, council area) and columns including financial_year, quarter_code, quarter_name, area_code, council_area, waste_management_group, data_status, plus numeric waste metrics.
Raises:: DAERAValidationError – If the file cannot be read or lacks the expected structure.
Return type:: pandas.DataFrame

Example

>>> import tempfile, pathlib
>>> # In practice, use get_latest_waste_statistics() instead
>>> # parse_waste_file(pathlib.Path("/path/to/download.csv"))

bolster.data_sources.daera_waste.get_latest_waste_statistics(force_refresh=False)[source]

Download and parse the latest DAERA LAC municipal waste statistics.

Scrapes the DAERA publications page for the current CSV URL (handling date-stamped paths that change with each release), downloads the file with 30-day caching, and returns a parsed DataFrame.

Parameters:

force_refresh (bool) – If True, bypass the local cache and re-download.

Returns:

DataFrame from parse_waste_file().

Raises:

DAERADataNotFoundError – If the publication page or file cannot be fetched.
DAERAValidationError – If the downloaded file cannot be parsed.

Return type:

pandas.DataFrame

Example

>>> df = get_latest_waste_statistics()
>>> 'council_area' in df.columns
True
>>> (df['lac_waste_arisings_tonnes'] >= 0).all()
True

bolster.data_sources.daera_waste.validate_waste_data(df)[source]

Validate a DAERA LAC municipal waste DataFrame.

Parameters:: df (pandas.DataFrame) – DataFrame from get_latest_waste_statistics() or parse_waste_file().
Returns:: True if all checks pass.
Raises:: DAERAValidationError – If the DataFrame is empty, missing required columns, has negative tonnage values, lacks expected NI councils, or covers an implausibly short time span.
Return type:: bool

Example

>>> df = get_latest_waste_statistics()
>>> validate_waste_data(df)
True