bolster.data_sources.psni.crime_statistics ========================================== .. py:module:: bolster.data_sources.psni.crime_statistics .. autoapi-nested-parse:: PSNI Police Recorded Crime Statistics. Provides access to police recorded crime statistics for Northern Ireland. Data includes: - Monthly crime counts by crime type and policing district - Geographic breakdown by 11 policing districts (aligned with LGDs) - Outcome data (charges, cautions, etc.) by district - Historical time series from April 2001 to December 2021 - Integration with NISRA datasets via LGD and NUTS3 codes Data Source: **Primary Source**: OpenDataNI - Police Recorded Crime in Northern Ireland https://www.opendatani.gov.uk/dataset/police-recorded-crime-in-northern-ireland **DATA LIMITATION — STALE SINCE JANUARY 2022**: The OpenDataNI dataset was last updated 27 January 2022 and only contains data through December 2021. PSNI stopped pushing updates to OpenDataNI after that date. The PSNI official statistics page publishes quarterly Excel files with current data, but psni.police.uk is protected by Cloudflare which blocks automated downloads. Calling ``get_latest_crime_statistics()`` will raise ``PSNIDataStaleError`` to make this limitation explicit. The historical data (Apr 2001–Dec 2021) remains accessible via ``get_historical_crime_statistics()``. For 2022+ data, consult PSNI directly: - Official stats page: https://www.psni.police.uk/about-us/our-publications-and-reports/official-statistics/police-recorded-crime-statistics - Contact: statistics@psni.police.uk Update Frequency: Quarterly (end of Jan, May, Jul, Oct) — **STALE SINCE 2022** Geographic Coverage: Northern Ireland (11 policing districts + NI total) Reference Date: Month of crime occurrence Time Coverage: April 2001 to December 2021 .. rubric:: Example >>> from bolster.data_sources.psni import crime_statistics >>> df = crime_statistics.get_historical_crime_statistics() >>> sorted(df.columns.tolist()) ['calendar_year', 'count', 'crime_type', 'data_measure', 'date', 'lgd_code', 'month', 'nuts3_code', 'nuts3_name', 'policing_district'] >>> belfast_lgd = crime_statistics.get_lgd_code('Belfast City') >>> belfast_lgd 'N09000003' Attributes ---------- .. autoapisummary:: bolster.data_sources.psni.crime_statistics.logger bolster.data_sources.psni.crime_statistics.CRIME_STATISTICS_URL bolster.data_sources.psni.crime_statistics.DATA_GUIDE_URL bolster.data_sources.psni.crime_statistics.PSNI_OFFICIAL_STATS_URL bolster.data_sources.psni.crime_statistics.PSNI_STATISTICS_EMAIL Functions --------- .. autoapisummary:: bolster.data_sources.psni.crime_statistics.get_data_source_info bolster.data_sources.psni.crime_statistics.parse_crime_statistics_file bolster.data_sources.psni.crime_statistics.get_latest_crime_statistics bolster.data_sources.psni.crime_statistics.get_historical_crime_statistics bolster.data_sources.psni.crime_statistics.validate_crime_statistics bolster.data_sources.psni.crime_statistics.filter_by_district bolster.data_sources.psni.crime_statistics.filter_by_crime_type bolster.data_sources.psni.crime_statistics.filter_by_date_range bolster.data_sources.psni.crime_statistics.get_total_crimes_by_district bolster.data_sources.psni.crime_statistics.get_crime_trends bolster.data_sources.psni.crime_statistics.get_outcome_rates_by_district bolster.data_sources.psni.crime_statistics.get_available_crime_types bolster.data_sources.psni.crime_statistics.get_available_districts Module Contents --------------- .. py:data:: logger .. py:data:: CRIME_STATISTICS_URL :value: 'https://admin.opendatani.gov.uk/dataset/80dc9542-7b2a-48f5-bbf4-ccc7040d36af/resource/6fd51851-d... .. py:data:: DATA_GUIDE_URL :value: 'https://admin.opendatani.gov.uk/dataset/80dc9542-7b2a-48f5-bbf4-ccc7040d36af/resource/51cd6a9e-6... .. py:data:: PSNI_OFFICIAL_STATS_URL :value: 'https://www.psni.police.uk/about-us/our-publications-and-reports/official-statistics/police-reco... .. py:data:: PSNI_STATISTICS_EMAIL :value: 'statistics@psni.police.uk' .. py:function:: get_data_source_info() Get information about crime statistics data sources. Returns a dictionary with URLs and contact information for accessing PSNI crime statistics. Use this when you need data beyond December 2021. :returns: - opendatani_url: OpenDataNI dataset URL (data through Dec 2021) - data_guide_url: PDF data guide URL - psni_official_url: PSNI official statistics page (current data) - contact_email: PSNI Statistics Branch email - data_limitation: Description of OpenDataNI data limitations - last_update: Last known update date for OpenDataNI :rtype: Dictionary with keys .. rubric:: Example >>> info = get_data_source_info() >>> sorted(info.keys()) ['contact_email', 'data_guide_url', 'data_limitation', 'last_update', 'opendatani_url', 'psni_official_url'] .. py:function:: parse_crime_statistics_file(file_path, add_geographic_codes = True) Parse PSNI crime statistics CSV file. The file is in long format with columns for year, month, district, crime type, data measure, and count. This function reads the CSV, cleans column names, adds date parsing, and optionally adds LGD and NUTS3 geographic codes for cross-dataset integration. :param file_path: Path to the crime statistics CSV file :param add_geographic_codes: If True, add LGD and NUTS3 code columns :returns: - calendar_year: int (year of crime) - month: str (month name: Apr, May, ..., Dec) - policing_district: str (district name or "Northern Ireland") - crime_type: str (Home Office crime classification) - data_measure: str (type of measure - crime count, outcome number, outcome rate) - count: float (value - can be count or percentage) - date: datetime (first day of month) - lgd_code: str (ONS LGD code, if add_geographic_codes=True) - nuts3_code: str (NUTS3 region code, if add_geographic_codes=True) - nuts3_name: str (NUTS3 region name, if add_geographic_codes=True) :rtype: DataFrame with columns :raises PSNIValidationError: If file structure is unexpected .. rubric:: Example >>> path = download_file(CRIME_STATISTICS_URL, cache_ttl_hours=24*7) >>> df = parse_crime_statistics_file(path) >>> 'crime_type' in df.columns True >>> len(df) > 0 True .. py:function:: get_latest_crime_statistics(force_refresh = False, add_geographic_codes = True) Raises PSNIDataStaleError — use get_historical_crime_statistics() instead. The OpenDataNI source was last updated January 2022. PSNI's official site publishes current data but is Cloudflare-protected and inaccessible to automated downloads. Use ``get_historical_crime_statistics()`` to access the data available (Apr 2001–Dec 2021). :raises PSNIDataStaleError: Always — this data source has no accessible update. .. py:function:: get_historical_crime_statistics(force_refresh = False, add_geographic_codes = True) Get historical police recorded crime statistics (April 2001 – December 2021). Downloads the crime statistics CSV from OpenDataNI. This dataset covers April 2001 through December 2021 and has not been updated since January 2022. For 2022+ data, consult PSNI directly. :param force_refresh: If True, bypass cache and download fresh data :param add_geographic_codes: If True, add LGD and NUTS3 code columns :returns: date, calendar_year, month, policing_district, crime_type, data_measure, count, lgd_code, nuts3_code, nuts3_name :rtype: DataFrame with columns :raises PSNIDataNotFoundError: If download fails :raises PSNIValidationError: If file structure is unexpected .. rubric:: Example >>> df = get_historical_crime_statistics() >>> sorted(df.columns.tolist()) ['calendar_year', 'count', 'crime_type', 'data_measure', 'date', 'lgd_code', 'month', 'nuts3_code', 'nuts3_name', 'policing_district'] >>> df['date'].max().year 2021 .. py:function:: validate_crime_statistics(df) Validate crime statistics data integrity. Performs sanity checks on the crime statistics data: - Non-negative crime counts - Reasonable date ranges - Expected policing districts present - No unexpected missing data :param df: DataFrame from parse_crime_statistics_file or get_latest_crime_statistics :returns: True if validation passes :raises PSNIValidationError: If validation fails .. rubric:: Example >>> df = get_latest_crime_statistics() >>> validate_crime_statistics(df) True .. py:function:: filter_by_district(df, district) Filter crime statistics to specific policing district(s). :param df: DataFrame from get_latest_crime_statistics :param district: District name(s) to filter (e.g., "Belfast City" or ["Belfast City", "Derry City & Strabane"]) :returns: Filtered DataFrame .. rubric:: Example >>> df = get_latest_crime_statistics() >>> belfast = filter_by_district(df, "Belfast City") >>> belfast['policing_district'].unique().tolist() ['Belfast City'] >>> >>> # Multiple districts >>> cities = filter_by_district(df, ["Belfast City", "Derry City & Strabane"]) >>> len(cities['policing_district'].unique()) == 2 True .. py:function:: filter_by_crime_type(df, crime_type) Filter crime statistics to specific crime type(s). :param df: DataFrame from get_latest_crime_statistics :param crime_type: Crime type(s) to filter (e.g., "Burglary" or ["Violence with injury", "Robbery"]) :returns: Filtered DataFrame .. rubric:: Example >>> df = get_latest_crime_statistics() >>> violence = filter_by_crime_type(df, "Violence with injury (including homicide & death/serious injury by unlawful driving)") >>> len(violence) > 0 True .. py:function:: filter_by_date_range(df, start_date = None, end_date = None) Filter crime statistics to a date range. :param df: DataFrame from get_latest_crime_statistics :param start_date: Start date (inclusive), e.g., "2020-01-01" or datetime :param end_date: End date (inclusive), e.g., "2021-12-31" or datetime :returns: Filtered DataFrame .. rubric:: Example >>> df = get_latest_crime_statistics() >>> # Get 2020 data >>> df_2020 = filter_by_date_range(df, "2020-01-01", "2020-12-31") >>> df_2020['calendar_year'].unique().tolist() [2020] >>> >>> # Get data from 2018 onwards >>> recent = filter_by_date_range(df, start_date="2018-01-01") >>> len(recent) > 0 True .. py:function:: get_total_crimes_by_district(df, year = None) Calculate total recorded crimes by policing district. :param df: DataFrame from get_latest_crime_statistics :param year: Optional year to filter (uses all years if None) :returns: policing_district, lgd_code, nuts3_code, total_crimes :rtype: DataFrame with columns .. rubric:: Example >>> df = get_latest_crime_statistics() >>> totals_2021 = get_total_crimes_by_district(df, year=2021) >>> sorted(totals_2021.columns.tolist()) ['lgd_code', 'nuts3_code', 'policing_district', 'total_crimes'] .. py:function:: get_crime_trends(df, crime_type = 'Total police recorded crime', district = 'Northern Ireland', measure = 'Police Recorded Crime') Get monthly crime trends for a specific crime type and district. :param df: DataFrame from get_latest_crime_statistics :param crime_type: Crime type to analyze (default: total crimes) :param district: Policing district (default: Northern Ireland total) :param measure: Data measure to use (default: Police Recorded Crime) :returns: date, calendar_year, month, count :rtype: DataFrame with columns .. rubric:: Example >>> df = get_latest_crime_statistics() >>> trends = get_crime_trends(df, district="Belfast City") >>> sorted(trends.columns.tolist()) ['calendar_year', 'count', 'date', 'month'] >>> len(trends) > 0 True .. py:function:: get_outcome_rates_by_district(df, year = None, crime_type = 'Total police recorded crime') Calculate crime outcome rates by policing district. Outcome rate represents the percentage of crimes with an outcome (charge, caution, community resolution, etc.) :param df: DataFrame from get_latest_crime_statistics :param year: Optional year to filter (uses all years if None) :param crime_type: Crime type to analyze (default: total crimes) :returns: policing_district, lgd_code, average_outcome_rate :rtype: DataFrame with columns .. rubric:: Example >>> df = get_latest_crime_statistics() >>> outcomes = get_outcome_rates_by_district(df, year=2021) >>> 'average_outcome_rate' in outcomes.columns True .. py:function:: get_available_crime_types(df) Get list of all crime types in the dataset. :param df: DataFrame from get_latest_crime_statistics :returns: Sorted list of crime type names .. rubric:: Example >>> df = get_latest_crime_statistics() >>> crime_types = get_available_crime_types(df) >>> isinstance(crime_types, list) True >>> 'Total police recorded crime' in crime_types True .. py:function:: get_available_districts(df) Get list of all policing districts in the dataset. :param df: DataFrame from get_latest_crime_statistics :returns: Sorted list of district names .. rubric:: Example >>> df = get_latest_crime_statistics() >>> districts = get_available_districts(df) >>> isinstance(districts, list) True >>> 'Northern Ireland' in districts True