bolster.utils.datatables
========================

.. py:module:: bolster.utils.datatables

.. autoapi-nested-parse::

   Generic utility for extracting DataTables data from HTML pages.

   Many Northern Ireland government statistics pages use R's flexdashboard/DT package
   to embed DataTables widgets. The data is stored as column-transposed JSON inside
   ``<script type="application/json">`` blocks with a ``{"x": {"data": [...], ...}}``
   structure, where ``x["data"]`` is a list of column arrays (not row arrays) and
   ``x["container"]`` holds the HTML table header with column names.

   .. rubric:: Example

   >>> from bolster.utils.datatables import datatables_to_dataframe
   >>> payload = {
   ...     "data": [["A", "B"], [1, 2]],
   ...     "container": "<table><thead><tr><th>Name</th><th>Value</th></tr></thead></table>",
   ... }
   >>> df = datatables_to_dataframe(payload)
   >>> list(df.columns)
   ['Name', 'Value']



Attributes
----------

.. autoapisummary::

   bolster.utils.datatables.logger


Exceptions
----------

.. autoapisummary::

   bolster.utils.datatables.DataTablesError


Functions
---------

.. autoapisummary::

   bolster.utils.datatables.fetch_datatables_json
   bolster.utils.datatables.datatables_to_dataframe
   bolster.utils.datatables.get_column_headers_from_url


Module Contents
---------------

.. py:data:: logger

.. py:exception:: DataTablesError

   Bases: :py:obj:`Exception`


   Raised when DataTables extraction fails.

   Initialize self.  See help(type(self)) for accurate signature.


.. py:function:: fetch_datatables_json(url, timeout = 30)

   Fetch an HTML page and extract the embedded DT widget JSON payload.

   The payload is the parsed content of the largest
   ``<script type="application/json">`` block whose ``x.data`` key is a
   column-transposed list (i.e. a list of lists).

   :param url: URL of the HTML page containing a DataTables widget.
   :param timeout: HTTP request timeout in seconds.

   :returns: The ``x`` sub-dict from the DT widget payload, containing at minimum
             ``"data"`` (list of column arrays) and ``"container"`` (HTML header).

   :raises DataTablesError: If the page cannot be fetched or no DT payload is found.

   .. rubric:: Example

   >>> from bolster.utils.datatables import DataTablesError
   >>> try:
   ...     fetch_datatables_json("https://example.com/data.html")
   ... except DataTablesError:
   ...     print("DataTablesError raised for invalid page")
   DataTablesError raised for invalid page

   A successful call returns a dict extracted from the page's DT widget.
   The shape mirrors what ``_extract_datatables_payload`` returns:

   >>> sample_html = (
   ...     '<script type="application/json">'
   ...     '{"x": {"data": [["Belfast", "Derry"], [1200, 800]],'
   ...     ' "container": "<thead><tr><th>City</th><th>Count</th></tr></thead>"}}'
   ...     "</script>"
   ... )
   >>> payload = _extract_datatables_payload(sample_html)
   >>> sorted(payload.keys())
   ['container', 'data']
   >>> len(payload["data"])
   2
   >>> payload["data"][0]
   ['Belfast', 'Derry']


.. py:function:: datatables_to_dataframe(payload)

   Convert a DT widget payload into a row-oriented DataFrame.

   The ``payload["data"]`` field is a list of column arrays (column-transposed).
   This function transposes it into a normal row-oriented DataFrame and uses
   column names from ``payload["container"]`` if available.

   :param payload: The ``x`` sub-dict from a DT widget JSON block, as returned by
                   :func:`fetch_datatables_json`.

   :returns: DataFrame with one row per record and columns named from the HTML header.

   :raises DataTablesError: If ``payload["data"]`` is missing or malformed.

   .. rubric:: Example

   >>> payload = {
   ...     "data": [["a", "b"], [1, 2]],
   ...     "container": "<table><thead><tr><th>Name</th><th>Value</th></tr></thead></table>",
   ... }
   >>> df = datatables_to_dataframe(payload)
   >>> list(df.columns)
   ['Name', 'Value']
   >>> len(df)
   2


.. py:function:: get_column_headers_from_url(url, timeout = 30)

   Fetch a DataTables page and return its column header names.

   Convenience helper for discovery.

   :param url: URL of the HTML page.
   :param timeout: HTTP request timeout in seconds.

   :returns: List of column header strings.