{ "cells": [ { "cell_type": "markdown", "id": "06fb8ff9-1e42-4382-b2ff-e2f8b189b4f8", "metadata": {}, "source": [ "# Migrating from `btrdb-python`\n", "\n", "This package is meant to supersede the previous [`BTrDB` Python](https://github.com/PingThingsIO/btrdb-python) API. This guide showcases how to migrate your old code to the PingThings Timeseries package." ] }, { "cell_type": "code", "execution_count": null, "id": "cbc5f10d-38ac-41bc-b13e-fce447c32ef9", "metadata": {}, "outputs": [], "source": [ "import os\n", "import re\n", "\n", "from IPython.display import Markdown, display" ] }, { "cell_type": "markdown", "id": "9082c0ad-734b-4f50-87cd-e6f01ecc7c3c", "metadata": {}, "source": [ "## Connecting to the Predictive Grid Platform" ] }, { "cell_type": "markdown", "id": "739d12cd-ecd6-4641-a3cc-9caefe866416", "metadata": {}, "source": [ "The first step is to import the APIs and connect to a database. This should be a drop-in replacement, substituting `pingthings.timeseries` (shortened to `pt`) for `btrdb`: " ] }, { "cell_type": "code", "execution_count": null, "id": "96b36f36-12a5-460c-bc88-5b1c107a9c2e", "metadata": {}, "outputs": [], "source": [ "import btrdb\n", "import pingthings as pt" ] }, { "cell_type": "markdown", "id": "1f033139-60f7-4331-9347-ce70645323a3", "metadata": {}, "source": [ "As before, you can connect by explicitly providing an endpoint and API key. And, as before, it's much better to have the API read in these values from the environment." ] }, { "cell_type": "code", "execution_count": null, "id": "310282f5-1525-4fa6-b261-d5ddeadacfe1", "metadata": {}, "outputs": [], "source": [ "# _ = btrdb.connect(conn_str=os.environ[\"BTRDB_ENDPOINTS\"], apikey=os.environ[\"BTRDB_API_KEY\"])\n", "bp_conn = btrdb.connect()\n", "\n", "# _ = pt.connect(endpoint=os.environ[\"BTRDB_ENDPOINTS\"], apikey=os.environ[\"BTRDB_API_KEY\"])\n", "pt_conn = pt.connect()" ] }, { "cell_type": "markdown", "id": "f87e5aa6-2f47-4eb0-a177-918a6ba0c9cd", "metadata": {}, "source": [ "_**Note:** in this example, we're connecting to and pulling data from the [NI4AI database](https://ni4ai.org/)._" ] }, { "cell_type": "markdown", "id": "e31416ac-baa9-43fd-a065-5f80ebf721e4", "metadata": {}, "source": [ "## List collections and streams\n", "\n", "A PingThings Client [`Client`](pingthings.Client) behaves very similarly to a BTrDB connection object. " ] }, { "cell_type": "code", "execution_count": null, "id": "a2a6832f-69ae-4eea-8ebd-b769eec6435c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['ami/lcl/MAC000002',\n", " 'ami/lcl/MAC000003',\n", " 'ami/lcl/MAC000004',\n", " 'ami/lcl/MAC000005',\n", " 'ami/lcl/MAC000006',\n", " 'ami/lcl/MAC000007',\n", " 'ami/lcl/MAC000008',\n", " 'ami/lcl/MAC000009',\n", " 'ami/lcl/MAC000010',\n", " 'ami/lcl/MAC000011']" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "['ami/lcl/MAC000002',\n", " 'ami/lcl/MAC000003',\n", " 'ami/lcl/MAC000004',\n", " 'ami/lcl/MAC000005',\n", " 'ami/lcl/MAC000006',\n", " 'ami/lcl/MAC000007',\n", " 'ami/lcl/MAC000008',\n", " 'ami/lcl/MAC000009',\n", " 'ami/lcl/MAC000010',\n", " 'ami/lcl/MAC000011']" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(bp_conn.list_collections()[:10])\n", "print()\n", "display(pt_conn.list_collections()[:10])" ] }, { "cell_type": "code", "execution_count": null, "id": "43b4d78e-add4-4841-9ddb-85235cf40011", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['sunshine/PMU1',\n", " 'sunshine/PMU2',\n", " 'sunshine/PMU3',\n", " 'sunshine/PMU4',\n", " 'sunshine/PMU5',\n", " 'sunshine/PMU6']" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "['sunshine/PMU1',\n", " 'sunshine/PMU2',\n", " 'sunshine/PMU3',\n", " 'sunshine/PMU4',\n", " 'sunshine/PMU5',\n", " 'sunshine/PMU6']" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(bp_conn.list_collections(starts_with=\"sunshine\"))\n", "print()\n", "display(pt_conn.list_collections(prefix=\"sunshine\"))" ] }, { "cell_type": "code", "execution_count": null, "id": "ffe01278-39ff-46ed-9b06-7ced60e7896a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.11/site-packages/btrdb/exceptions.py:52: FutureWarning: StreamSet will be the default return object for ``streams_in_collection`` in a future release.\n", " return fn(*args, **kwargs)\n" ] }, { "data": { "text/plain": [ "{'L1ANG': UUID('06e00f19-5ee2-4ae3-911e-6cedbb979d6b'),\n", " 'C2MAG': UUID('dbd38fba-4f22-46e1-87be-850f800ec024'),\n", " 'C2ANG': UUID('31273550-7d27-4f5e-89ba-e5cae0ddccf4'),\n", " 'L2ANG': UUID('5e820477-04bc-4562-a02e-90e3ec1fd083'),\n", " 'L3ANG': UUID('e51f6dad-dc3d-4bfd-9092-bb7e6f88f2f7'),\n", " 'L3MAG': UUID('a60aa7aa-9ad1-425b-bb2e-74b42a2cd3e0'),\n", " 'LSTATE': UUID('c8b4b009-738d-4a35-968b-20575b588429'),\n", " 'C3ANG': UUID('b9b0ec24-84c9-42de-afe4-4b19b1ab03b4'),\n", " 'L2MAG': UUID('b4de2e84-7816-4dc7-9702-f1c210586e1c'),\n", " 'C3MAG': UUID('2f077897-1ddf-4427-857f-53c64f69c1cb'),\n", " 'C1MAG': UUID('bd74aa49-8ccc-4f0b-8f83-7195b6c1818e'),\n", " 'L1MAG': UUID('edb36769-f56f-47c5-bbda-a177c424cc4f'),\n", " 'C1ANG': UUID('fa16194e-a4ac-460c-a75f-c8694a5ad0a9')}" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "{'L1ANG': UUID('06e00f19-5ee2-4ae3-911e-6cedbb979d6b'),\n", " 'C2MAG': UUID('dbd38fba-4f22-46e1-87be-850f800ec024'),\n", " 'C2ANG': UUID('31273550-7d27-4f5e-89ba-e5cae0ddccf4'),\n", " 'L2ANG': UUID('5e820477-04bc-4562-a02e-90e3ec1fd083'),\n", " 'L3ANG': UUID('e51f6dad-dc3d-4bfd-9092-bb7e6f88f2f7'),\n", " 'L3MAG': UUID('a60aa7aa-9ad1-425b-bb2e-74b42a2cd3e0'),\n", " 'LSTATE': UUID('c8b4b009-738d-4a35-968b-20575b588429'),\n", " 'C3ANG': UUID('b9b0ec24-84c9-42de-afe4-4b19b1ab03b4'),\n", " 'L2MAG': UUID('b4de2e84-7816-4dc7-9702-f1c210586e1c'),\n", " 'C3MAG': UUID('2f077897-1ddf-4427-857f-53c64f69c1cb'),\n", " 'C1MAG': UUID('bd74aa49-8ccc-4f0b-8f83-7195b6c1818e'),\n", " 'L1MAG': UUID('edb36769-f56f-47c5-bbda-a177c424cc4f'),\n", " 'C1ANG': UUID('fa16194e-a4ac-460c-a75f-c8694a5ad0a9')}" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display({stream.name: stream.uuid for stream in bp_conn.streams_in_collection(\"sunshine/PMU5\")})\n", "print()\n", "display({stream.name: stream.uuid for stream in pt_conn.streams_in_collection(\"sunshine/PMU5\")})" ] }, { "cell_type": "code", "execution_count": null, "id": "180b8907-68ec-4865-adf4-7ac08ef7086d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'sunshine/PMU5'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "'C3MAG'" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "'sunshine/PMU5'" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "'C3MAG'" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "bp_stream = bp_conn.stream_from_uuid(\"2f077897-1ddf-4427-857f-53c64f69c1cb\")\n", "display(bp_stream.collection, bp_stream.name)\n", "print()\n", "pt_stream = pt_conn.stream_from_uuid(\"2f077897-1ddf-4427-857f-53c64f69c1cb\")\n", "display(pt_stream.collection, pt_stream.name)" ] }, { "cell_type": "markdown", "id": "c0698f94-45dc-4589-8c42-a6e5eb6c25c2", "metadata": {}, "source": [ "## Utility Methods" ] }, { "cell_type": "markdown", "id": "77f0343b-4120-455c-adf3-01365a3e68e9", "metadata": {}, "source": [ "The utilities from the `timez` subpackage can now be found in [`pt.timeseries.utils`](pingthings.timeseries.utils)..." ] }, { "cell_type": "code", "execution_count": null, "id": "10b13e45-f0da-448d-ad40-a71216414d5d", "metadata": {}, "outputs": [], "source": [ "from btrdb.utils import timez" ] }, { "cell_type": "code", "execution_count": null, "id": "7d988490-5dc4-4a31-815c-dfb25b0fcd69", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1745880640757817088" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "1745880640758724826" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(timez.currently_as_ns())\n", "print()\n", "display(pt.utils.currently_as_ns())" ] }, { "cell_type": "code", "execution_count": null, "id": "5fbe3130-a060-4bb5-97fd-2e40771829b4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "datetime.datetime(2024, 10, 9, 15, 39, 27, 976883, tzinfo=)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "datetime.datetime(2024, 10, 9, 15, 39, 27, 976883, tzinfo=datetime.timezone.utc)" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(timez.ns_to_datetime(1728488367976882944))\n", "print()\n", "display(pt.utils.ns_to_datetime(1728488367976882944))" ] }, { "cell_type": "code", "execution_count": null, "id": "6beb0c6f-2c6e-4582-8fa6-a876aa1c7857", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "86400000000000" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "86400000000000" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(timez.ns_delta(days=1))\n", "print()\n", "display(pt.utils.ns_delta(days=1))" ] }, { "cell_type": "code", "execution_count": null, "id": "6300af8a-0442-4cb5-b2e3-62e98037729a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1728432000000000000" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "1728432000000000000" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(timez.to_nanoseconds(\"2024-10-09\"))\n", "print()\n", "display(pt.utils.to_nanoseconds(\"2024-10-09\"))" ] }, { "cell_type": "markdown", "id": "4dfde569-1bed-4bf9-9fd9-0fb2e567ee21", "metadata": {}, "source": [ "... with the addition of [a new helper method](pingthings.timeseries.utils.nearest_pointwidth) for finding the power of 2 ns closest to a desired duration." ] }, { "cell_type": "code", "execution_count": null, "id": "00cc67d8-8738-4efa-905d-f12cbce39fd7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The closest aligned width is 2**43, or 8796093022208, ns\n" ] } ], "source": [ "point_width = pt.utils.nearest_point_width(hours=3)\n", "print(f\"The closest aligned width is 2**{point_width.point_width}, or {point_width.ns}, ns\")" ] }, { "cell_type": "markdown", "id": "12c50dda-6cd4-4a3f-b48c-fe4357636bce", "metadata": {}, "source": [ "Moreover, all [valid point-widths]() are now available in a convenient lookup table as [`pt.constants.PW`](pingthings.timeseries.constants.PW). As a little bit of sugar, printing a point-width gives its representation as a `timedelta` object:" ] }, { "cell_type": "code", "execution_count": null, "id": "f74c160d-2f9b-414a-9423-ab8fc6338d89", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "datetime.timedelta(seconds=70368, microseconds=744178)\n" ] } ], "source": [ "print(pt.timeseries.constants.PW(46))" ] }, { "cell_type": "markdown", "id": "7c10d41e-4bf6-4b3f-9975-37113a9c09ac", "metadata": {}, "source": [ "## Working with Streams" ] }, { "cell_type": "markdown", "id": "b083c764-7798-4a81-b93d-0dcdfecec0c9", "metadata": {}, "source": [ "### Metadata" ] }, { "cell_type": "code", "execution_count": null, "id": "51f2b65a-7452-4ffb-8e08-96e07c8aeb89", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'unit': 'amps', 'ingress': '', 'distiller': '', 'name': 'C3MAG'}" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "({'location': 'substation',\n", " 'impedance': {'source': 'PMU5',\n", " 'target': 'PMU2',\n", " 'pos_sequence': '0.489+j0.59',\n", " 'neg_sequence': '0.971+j1.476'}},\n", " 2)" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "{'name': 'C3MAG', 'unit': 'amps', 'ingress': '', 'distiller': ''}" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "{'location': 'substation',\n", " 'impedance': '{\"source\": \"PMU5\", \"target\": \"PMU2\", \"pos_sequence\": \"0.489+j0.59\", \"neg_sequence\": \"0.971+j1.476\"}'}" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(bp_stream.tags(), bp_stream.annotations())\n", "print()\n", "display(pt_stream.tags(), pt_stream.annotations())" ] }, { "cell_type": "markdown", "id": "ad1cc2f8-5c22-4a81-838d-930ad0cce101", "metadata": {}, "source": [ "**Note** how [`pt.timeseries.Stream.annotations()`](pingthings.timeseries.Stream.annotations) _does not_ include the metadata version." ] }, { "cell_type": "markdown", "id": "3a3dafaf-aa4f-41e0-a5f6-b6cc3b6f4532", "metadata": {}, "source": [ "Getting the latest version is now more explicit:" ] }, { "cell_type": "code", "execution_count": null, "id": "8de2557e-9d93-4fc9-9226-2ad66b99c80c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "149229" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "149229" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(bp_stream.version())\n", "print()\n", "display(pt_stream.get_latest_version())" ] }, { "cell_type": "markdown", "id": "3c54cb65-8594-425c-ad15-7b45d740bec4", "metadata": {}, "source": [ "and the return signatues of `earliest`, `latest` have been simplified, no longer including the corresponding version (which can be fetched via the above if not explicitly set):" ] }, { "cell_type": "code", "execution_count": null, "id": "6c6136fb-32d5-4043-91de-44a4f5bab91e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(RawPoint(946684797008333000, 269.8155212402344), 149229)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "(RawPoint(1476816263666666000, 302.1278076171875), 149229)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "3704481931" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "Point(time: 946684797008333000, value: 269.8155212402344)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "Point(time: 1476816263666666000, value: 302.1278076171875)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "3704481931" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(bp_stream.earliest(), bp_stream.latest(), bp_stream.count())\n", "print()\n", "display(pt_stream.earliest(), pt_stream.latest(), pt_stream.count())" ] }, { "cell_type": "markdown", "id": "17a61261-ed26-4f70-9383-e4c01f70e679", "metadata": {}, "source": [ "### Raw Point Queries\n", "\n", "The main difference\u2014and the biggest performance improvement\u2014with the new API is that all data queries now return [PyArrow tables](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). Not only is this data returned much faster and more efficiently, but these tables can be quickly converted into Pandas DataFrames using built-in PyArrow methods." ] }, { "cell_type": "code", "execution_count": null, "id": "315f1ce4-7602-4939-8692-02ccc19d7d11", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": null, "id": "be96f63c-6658-47f8-827d-0bad322efba0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[(RawPoint(946684797008333000, 269.8155212402344), 149229),\n", " (RawPoint(946684797016666000, 269.4358825683594), 149229),\n", " (RawPoint(946684797024999000, 268.82281494140625), 149229),\n", " (RawPoint(946684797033333000, 269.13201904296875), 149229),\n", " (RawPoint(946684797041666000, 270.1548156738281), 149229),\n", " (RawPoint(946684797049999000, 270.3088073730469), 149229),\n", " (RawPoint(946684797058333000, 269.6369323730469), 149229),\n", " (RawPoint(946684797066666000, 269.450439453125), 149229),\n", " (RawPoint(946684797074999000, 269.6058044433594), 149229),\n", " (RawPoint(946684797083333000, 269.1439514160156), 149229),\n", " (RawPoint(946684797091666000, 268.5997619628906), 149229),\n", " (RawPoint(946684797099999000, 268.5179138183594), 149229)]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timevalue
01999-12-31 23:59:57.008333269.815521
11999-12-31 23:59:57.016666269.435883
21999-12-31 23:59:57.024999268.822815
31999-12-31 23:59:57.033333269.132019
41999-12-31 23:59:57.041666270.154816
51999-12-31 23:59:57.049999270.308807
61999-12-31 23:59:57.058333269.636932
71999-12-31 23:59:57.066666269.450439
81999-12-31 23:59:57.074999269.605804
91999-12-31 23:59:57.083333269.143951
101999-12-31 23:59:57.091666268.599762
111999-12-31 23:59:57.099999268.517914
\n", "
" ], "text/plain": [ " time value\n", "0 1999-12-31 23:59:57.008333 269.815521\n", "1 1999-12-31 23:59:57.016666 269.435883\n", "2 1999-12-31 23:59:57.024999 268.822815\n", "3 1999-12-31 23:59:57.033333 269.132019\n", "4 1999-12-31 23:59:57.041666 270.154816\n", "5 1999-12-31 23:59:57.049999 270.308807\n", "6 1999-12-31 23:59:57.058333 269.636932\n", "7 1999-12-31 23:59:57.066666 269.450439\n", "8 1999-12-31 23:59:57.074999 269.605804\n", "9 1999-12-31 23:59:57.083333 269.143951\n", "10 1999-12-31 23:59:57.091666 268.599762\n", "11 1999-12-31 23:59:57.099999 268.517914" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "---" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "pyarrow.Table\n", "time: timestamp[ns, tz=UTC] not null\n", "value: float not null\n", "----\n", "time: [[1999-12-31 23:59:57.008333000Z,1999-12-31 23:59:57.016666000Z,1999-12-31 23:59:57.024999000Z,1999-12-31 23:59:57.033333000Z,1999-12-31 23:59:57.041666000Z,...,1999-12-31 23:59:57.074999000Z,1999-12-31 23:59:57.083333000Z,1999-12-31 23:59:57.091666000Z,1999-12-31 23:59:57.099999000Z,1999-12-31 23:59:57.108333000Z]]\n", "value: [[269.81552,269.43588,268.8228,269.13202,270.15482,...,269.6058,269.14395,268.59976,268.5179,268.71313]]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timevalue
01999-12-31 23:59:57.008333+00:00269.815521
11999-12-31 23:59:57.016666+00:00269.435883
21999-12-31 23:59:57.024999+00:00268.822815
31999-12-31 23:59:57.033333+00:00269.132019
41999-12-31 23:59:57.041666+00:00270.154816
51999-12-31 23:59:57.049999+00:00270.308807
61999-12-31 23:59:57.058333+00:00269.636932
71999-12-31 23:59:57.066666+00:00269.450439
81999-12-31 23:59:57.074999+00:00269.605804
91999-12-31 23:59:57.083333+00:00269.143951
101999-12-31 23:59:57.091666+00:00268.599762
111999-12-31 23:59:57.099999+00:00268.517914
121999-12-31 23:59:57.108333+00:00268.713135
\n", "
" ], "text/plain": [ " time value\n", "0 1999-12-31 23:59:57.008333+00:00 269.815521\n", "1 1999-12-31 23:59:57.016666+00:00 269.435883\n", "2 1999-12-31 23:59:57.024999+00:00 268.822815\n", "3 1999-12-31 23:59:57.033333+00:00 269.132019\n", "4 1999-12-31 23:59:57.041666+00:00 270.154816\n", "5 1999-12-31 23:59:57.049999+00:00 270.308807\n", "6 1999-12-31 23:59:57.058333+00:00 269.636932\n", "7 1999-12-31 23:59:57.066666+00:00 269.450439\n", "8 1999-12-31 23:59:57.074999+00:00 269.605804\n", "9 1999-12-31 23:59:57.083333+00:00 269.143951\n", "10 1999-12-31 23:59:57.091666+00:00 268.599762\n", "11 1999-12-31 23:59:57.099999+00:00 268.517914\n", "12 1999-12-31 23:59:57.108333+00:00 268.713135" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "start = timez.to_nanoseconds(\"1999-12-31 23:59:57\")\n", "end = start + timez.ns_delta(seconds=0.1)\n", "\n", "bp_stream_data = bp_stream.values(start, end)\n", "\n", "display(bp_stream_data)\n", "\n", "as_array: list[tuple[pd.Timestamp, float]] = []\n", "for raw_point, _ in bp_stream_data:\n", " as_array.append((pd.Timestamp(raw_point.time), raw_point.value))\n", "display(pd.DataFrame(as_array, columns=[\"time\", \"value\"]))\n", "\n", "display(Markdown(\"---\"))\n", "\n", "start = pt.utils.to_nanoseconds(\"1999-12-31 23:59:57\")\n", "end = start + pt.utils.ns_delta(seconds=0.1)\n", "\n", "pt_stream_data = pt_stream.raw_values(946684797008333000, 946684797008333000 + 1e8)\n", "display(pt_stream_data)\n", "display(pt_stream_data.to_pandas())" ] }, { "cell_type": "markdown", "id": "8615fd1a-0117-42a5-a130-fdf4558c3b3f", "metadata": {}, "source": [ "For large amounts of data, you should see significantly faster query times with significantly less CPU overhead." ] }, { "cell_type": "code", "execution_count": null, "id": "59297bbb-3728-4310-8940-a81598164ba6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 day of btrdb values comprises 10327421 values\n", "CPU times: user 17.3 s, sys: 1.23 s, total: 18.6 s\n", "Wall time: 18.7 s\n" ] }, { "data": { "text/markdown": [ "---" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "1 day of btrdb values consumes 10327421 values\n", "CPU times: user 150 ms, sys: 396 ms, total: 545 ms\n", "Wall time: 1.34 s\n" ] } ], "source": [ "start = timez.to_nanoseconds(\"2016-5-1\")\n", "end = timez.to_nanoseconds(\"2016-5-2\")\n", "\n", "%time print(f\"1 day of btrdb values comprises {len(bp_stream.values(start, end))} values\")\n", "\n", "display(Markdown(\"---\"))\n", "\n", "start = pt.utils.to_nanoseconds(\"2016-5-1\")\n", "end = pt.utils.to_nanoseconds(\"2016-5-2\")\n", "\n", "%time print(f\"1 day of btrdb values consumes {pt_stream.raw_values(start, end).shape[0]} values\")" ] }, { "cell_type": "markdown", "id": "35b9d80b-24ff-43ae-b232-d2794a55a10f", "metadata": {}, "source": [ "### Statistical Queries\n", "\n", "[\"StatPoint\"]() queries also now return as PyArrow tables, making them significantly easier to work with. The [new API has also been streamlined](pingthings.timeseries.Stream.windowed_values), taking `width` in ns or as an explicit `PW`." ] }, { "cell_type": "code", "execution_count": null, "id": "47f52b78-4a9d-4e6e-a4fc-fdcaa08531a2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The nearest aligned width to 1 week is datetime.timedelta(days=6, seconds=44549, microseconds=953421) (pointwith=49)\n" ] } ], "source": [ "week_ish = pt.utils.nearest_point_width(days=7)\n", "print(f\"The nearest aligned width to 1 week is {week_ish} (pointwith={week_ish.point_width})\")" ] }, { "cell_type": "code", "execution_count": null, "id": "04f33794-0de0-4728-8dfd-3c7908ce8a98", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((StatPoint(1461981029035147264, 166.16246032714844, 245.77867209651018, 351.5482482910156, 65474938, 30.139268588781327),\n", " 149229),\n", " (StatPoint(1462543978988568576, 194.8922119140625, 268.5108455921535, 373.44482421875, 67508423, 28.56277400251457),\n", " 149229),\n", " (StatPoint(1463106928941989888, 173.6748504638672, 258.19540845109805, 361.1649475097656, 67516769, 32.87283851829869),\n", " 149229),\n", " (StatPoint(1463669878895411200, 0.0009941004682332277, 213.16805834630688, 349.7587890625, 67494393, 106.41148944767052),\n", " 149229))" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timeminmeanmaxcountstddev
02016-04-30 01:50:29.035147264166.162460245.778672351.5482486547493830.139269
12016-05-06 14:12:58.988568576194.892212268.510846373.4448246750842328.562774
22016-05-13 02:35:28.941989888173.674850258.195408361.1649486751676932.872839
32016-05-19 14:57:58.8954112000.000994213.168058349.75878967494393106.411489
\n", "
" ], "text/plain": [ " time min mean max count \\\n", "0 2016-04-30 01:50:29.035147264 166.162460 245.778672 351.548248 65474938 \n", "1 2016-05-06 14:12:58.988568576 194.892212 268.510846 373.444824 67508423 \n", "2 2016-05-13 02:35:28.941989888 173.674850 258.195408 361.164948 67516769 \n", "3 2016-05-19 14:57:58.895411200 0.000994 213.168058 349.758789 67494393 \n", "\n", " stddev \n", "0 30.139269 \n", "1 28.562774 \n", "2 32.872839 \n", "3 106.411489 " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "---" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "pyarrow.Table\n", "time: timestamp[ns, tz=UTC] not null\n", "min: float not null\n", "mean: float not null\n", "max: float not null\n", "count: uint64 not null\n", "stddev: float not null\n", "----\n", "time: [[2016-04-30 01:50:29.035147264Z,2016-05-06 14:12:58.988568576Z,2016-05-13 02:35:28.941989888Z,2016-05-19 14:57:58.895411200Z]]\n", "min: [[166.16246,194.89221,173.67485,0.0009941005]]\n", "mean: [[245.77867,268.51083,258.1954,213.16806]]\n", "max: [[351.54825,373.44482,361.16495,349.7588]]\n", "count: [[65474938,67508423,67516769,67494393]]\n", "stddev: [[30.139269,28.562775,32.872837,106.41149]]" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
timeminmeanmaxcountstddev
02016-04-30 01:50:29.035147264+00:00166.162460245.778671351.5482486547493830.139269
12016-05-06 14:12:58.988568576+00:00194.892212268.510834373.4448246750842328.562775
22016-05-13 02:35:28.941989888+00:00173.674850258.195404361.1649486751676932.872837
32016-05-19 14:57:58.895411200+00:000.000994213.168060349.75878967494393106.411491
\n", "
" ], "text/plain": [ " time min mean max \\\n", "0 2016-04-30 01:50:29.035147264+00:00 166.162460 245.778671 351.548248 \n", "1 2016-05-06 14:12:58.988568576+00:00 194.892212 268.510834 373.444824 \n", "2 2016-05-13 02:35:28.941989888+00:00 173.674850 258.195404 361.164948 \n", "3 2016-05-19 14:57:58.895411200+00:00 0.000994 213.168060 349.758789 \n", "\n", " count stddev \n", "0 65474938 30.139269 \n", "1 67508423 28.562775 \n", "2 67516769 32.872837 \n", "3 67494393 106.411491 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "start = timez.to_nanoseconds(\"2016-5-1\")\n", "end = timez.to_nanoseconds(\"2016-6-1\")\n", "\n", "bp_stats = bp_stream.aligned_windows(start, end, pointwidth=49)\n", "display(bp_stats)\n", "\n", "as_stat_array: list[tuple[pd.Timestamp, float, float, float, float, float]] = []\n", "for point, _ in bp_stats:\n", " as_stat_array.append(\n", " (pd.Timestamp(point.time), point.min, point.mean, point.max, point.count, point.stddev)\n", " )\n", "display(pd.DataFrame(as_stat_array, columns=[\"time\", \"min\", \"mean\", \"max\", \"count\", \"stddev\"]))\n", "\n", "display(Markdown(\"---\"))\n", "\n", "bp_stats = pt_stream.windowed_values(start, end, width=pt.constants.PW[49])\n", "display(bp_stats)\n", "\n", "display(bp_stats.to_pandas())" ] }, { "cell_type": "markdown", "id": "30743411-b6b8-4c6a-8180-fb5948f2d7bf", "metadata": {}, "source": [ "_If your workflow relies on precise-valued windowed queries (`bp_stream.windows()`), the new `windowed_values` queries accepts [`precise=True` as an keyword argument](pingthings.timeseries.Stream.windowed_values)._" ] }, { "cell_type": "markdown", "id": "74781c9e-6e33-4246-bdc9-a1aea775665e", "metadata": {}, "source": [ "## Working with Multiple Streams" ] }, { "cell_type": "markdown", "id": "b2e39f2a-ed20-4a37-bcb1-439337f54d7e", "metadata": {}, "source": [ "`StreamSet`s, are now more closely embedded with the `Client` API. Explicitly: `streams_in_collection` now returns the set of streams pre-wrapped as a `StreamSet`." ] }, { "cell_type": "code", "execution_count": null, "id": "68a2f616-1072-461e-bbb7-855dcc7b51e6", "metadata": {}, "outputs": [], "source": [ "from btrdb.stream import StreamSet" ] }, { "cell_type": "code", "execution_count": null, "id": "02f27f9d-c503-436d-876f-6e71997cc2a4", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.11/site-packages/btrdb/exceptions.py:52: FutureWarning: StreamSet will be the default return object for ``streams_in_collection`` in a future release.\n", " return fn(*args, **kwargs)\n" ] }, { "data": { "text/plain": [ "list" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "pingthings.timeseries.client.StreamSet" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(type(bp_conn.streams_in_collection(\"sunshine/PMU1\")))\n", "bp_pmu1 = StreamSet(bp_conn.streams_in_collection(\"sunshine/PMU1\"))\n", "print()\n", "pt_pmu1 = pt_conn.streams_in_collection(\"sunshine/PMU1\")\n", "display(type(pt_pmu1))" ] }, { "cell_type": "markdown", "id": "8851c51c-bf7f-4328-ae2a-1534473038ff", "metadata": {}, "source": [ "Though you can still manually create `StreamSet`s from `list`s of `Stream`s" ] }, { "cell_type": "code", "execution_count": null, "id": "f6207500-8f87-4a7d-8d90-b222a7f02d10", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "L1MAG\n", "L3MAG\n", "L2MAG\n" ] } ], "source": [ "pmu1_voltages = pt.timeseries.StreamSet(\n", " [stream for stream in pt_pmu1 if stream.tags()[\"unit\"] == \"volts\"]\n", ")\n", "for stream in pmu1_voltages:\n", " print(stream.name)" ] }, { "cell_type": "markdown", "id": "4829f8db-5e70-4602-98ef-ed399244cd32", "metadata": {}, "source": [ "As with `Stream`s, the new `StreamSet` queries also return data in the form of `PyArrow` tables, with the column names specifying the UUID of the individual streams. These columns can be renamed as desired directly using PyArrow." ] }, { "cell_type": "code", "execution_count": null, "id": "517ffb99-efae-4ab4-af9b-c1522fcbe0ba", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sunshine/PMU1/L1MAGsunshine/PMU1/L3MAGsunshine/PMU1/L2MAG
time
14516712000083330007227.0122077202.7241217243.580078
14516712000166660007226.9267587202.5693367243.549805
14516712000249990007227.1962897202.8129887243.640625
14516712000333330007227.3896487202.8144537243.552734
14516712000416660007227.2934577202.3378917243.318359
14516712000499990007227.1459967202.0371097243.214844
14516712000583330007227.1821297201.9267587243.302734
14516712000666660007227.2700207201.9975597243.347656
14516712000749990007227.1347667202.1333017243.130859
14516712000833330007226.8847667202.0190437242.918457
14516712000916660007226.9335947201.9018557243.006348
14516712000999990007227.2807627201.9272467243.362793
\n", "
" ], "text/plain": [ " sunshine/PMU1/L1MAG sunshine/PMU1/L3MAG \\\n", "time \n", "1451671200008333000 7227.012207 7202.724121 \n", "1451671200016666000 7226.926758 7202.569336 \n", "1451671200024999000 7227.196289 7202.812988 \n", "1451671200033333000 7227.389648 7202.814453 \n", "1451671200041666000 7227.293457 7202.337891 \n", "1451671200049999000 7227.145996 7202.037109 \n", "1451671200058333000 7227.182129 7201.926758 \n", "1451671200066666000 7227.270020 7201.997559 \n", "1451671200074999000 7227.134766 7202.133301 \n", "1451671200083333000 7226.884766 7202.019043 \n", "1451671200091666000 7226.933594 7201.901855 \n", "1451671200099999000 7227.280762 7201.927246 \n", "\n", " sunshine/PMU1/L2MAG \n", "time \n", "1451671200008333000 7243.580078 \n", "1451671200016666000 7243.549805 \n", "1451671200024999000 7243.640625 \n", "1451671200033333000 7243.552734 \n", "1451671200041666000 7243.318359 \n", "1451671200049999000 7243.214844 \n", "1451671200058333000 7243.302734 \n", "1451671200066666000 7243.347656 \n", "1451671200074999000 7243.130859 \n", "1451671200083333000 7242.918457 \n", "1451671200091666000 7243.006348 \n", "1451671200099999000 7243.362793 " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "---" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sunshine/PMU1/L1MAGsunshine/PMU1/L3MAGsunshine/PMU1/L2MAG
time
2016-01-01 18:00:00.008333+00:007227.0122077202.7241217243.580078
2016-01-01 18:00:00.016666+00:007226.9267587202.5693367243.549805
2016-01-01 18:00:00.024999+00:007227.1962897202.8129887243.640625
2016-01-01 18:00:00.033333+00:007227.3896487202.8144537243.552734
2016-01-01 18:00:00.041666+00:007227.2934577202.3378917243.318359
2016-01-01 18:00:00.049999+00:007227.1459967202.0371097243.214844
2016-01-01 18:00:00.058333+00:007227.1821297201.9267587243.302734
2016-01-01 18:00:00.066666+00:007227.2700207201.9975597243.347656
2016-01-01 18:00:00.074999+00:007227.1347667202.1333017243.130859
2016-01-01 18:00:00.083333+00:007226.8847667202.0190437242.918457
2016-01-01 18:00:00.091666+00:007226.9335947201.9018557243.006348
2016-01-01 18:00:00.099999+00:007227.2807627201.9272467243.362793
\n", "
" ], "text/plain": [ " sunshine/PMU1/L1MAG sunshine/PMU1/L3MAG \\\n", "time \n", "2016-01-01 18:00:00.008333+00:00 7227.012207 7202.724121 \n", "2016-01-01 18:00:00.016666+00:00 7226.926758 7202.569336 \n", "2016-01-01 18:00:00.024999+00:00 7227.196289 7202.812988 \n", "2016-01-01 18:00:00.033333+00:00 7227.389648 7202.814453 \n", "2016-01-01 18:00:00.041666+00:00 7227.293457 7202.337891 \n", "2016-01-01 18:00:00.049999+00:00 7227.145996 7202.037109 \n", "2016-01-01 18:00:00.058333+00:00 7227.182129 7201.926758 \n", "2016-01-01 18:00:00.066666+00:00 7227.270020 7201.997559 \n", "2016-01-01 18:00:00.074999+00:00 7227.134766 7202.133301 \n", "2016-01-01 18:00:00.083333+00:00 7226.884766 7202.019043 \n", "2016-01-01 18:00:00.091666+00:00 7226.933594 7201.901855 \n", "2016-01-01 18:00:00.099999+00:00 7227.280762 7201.927246 \n", "\n", " sunshine/PMU1/L2MAG \n", "time \n", "2016-01-01 18:00:00.008333+00:00 7243.580078 \n", "2016-01-01 18:00:00.016666+00:00 7243.549805 \n", "2016-01-01 18:00:00.024999+00:00 7243.640625 \n", "2016-01-01 18:00:00.033333+00:00 7243.552734 \n", "2016-01-01 18:00:00.041666+00:00 7243.318359 \n", "2016-01-01 18:00:00.049999+00:00 7243.214844 \n", "2016-01-01 18:00:00.058333+00:00 7243.302734 \n", "2016-01-01 18:00:00.066666+00:00 7243.347656 \n", "2016-01-01 18:00:00.074999+00:00 7243.130859 \n", "2016-01-01 18:00:00.083333+00:00 7242.918457 \n", "2016-01-01 18:00:00.091666+00:00 7243.006348 \n", "2016-01-01 18:00:00.099999+00:00 7243.362793 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "start = timez.to_nanoseconds(\"2016-1-1 18:00:00\")\n", "end = start + timez.ns_delta(seconds=0.1)\n", "display(bp_pmu1.filter(name=re.compile(\"L.MAG\"), start=start, end=end).to_dataframe())\n", "\n", "display(Markdown(\"---\"))\n", "\n", "display(\n", " pmu1_voltages.raw_values(start, end)\n", " .rename_columns([\"time\"] + [f\"{stream.collection}/{stream.name}\" for stream in pmu1_voltages])\n", " .to_pandas()\n", " .set_index(\"time\")\n", ")" ] }, { "cell_type": "markdown", "id": "cfbc0125-c744-4611-bc0a-a505e968959f", "metadata": {}, "source": [ "_Note also that the old API did not convert the time values from nanoseconds to Timestamps._" ] }, { "cell_type": "markdown", "id": "6f57ebd9-7608-4b6b-8a30-cbd195bc3c11", "metadata": {}, "source": [ "Statistical queries are similarly managed, with the additional step of needing to apply a `MultiIndex` if you want to match the result of `agg='all'`" ] }, { "cell_type": "code", "execution_count": null, "id": "2910d5e4-0d9e-41f5-9a9e-5f7c40b85363", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sunshine/PMU1
L1MAGL3MAGL2MAG
minmeanmaxcountstddevminmeanmaxcountstddevminmeanmaxcountstddev
2015-12-15 05:58:00.0132997126686.6684577162.1917207262.52148422623839934.1604196343.7026377117.5096857243.53125022623839939.3043266562.4599617144.4435957285.52099622623839945.038278
2016-02-05 08:57:59.640670208771.5824587154.2883647298.77246114967353434.191939809.0826427123.8681947309.24511714967353436.310027846.4526377148.0591647338.80224614967353438.897277
2016-03-02 10:27:59.4543554564686.4409187154.0528207305.97851627021626137.1794774284.3374027123.8810127324.55175827021627140.5783096265.9687507148.8670087367.45263727021629645.239399
2016-03-28 11:57:59.2680407046825.3710947157.5808687301.88525426959173936.5141876722.9111337125.5772447320.83203126959171242.5865206803.8520517156.3225887389.02294926959175347.147822
2016-04-23 13:27:59.0817259526580.9541027161.2330467300.86230526969215435.6145346822.6782237128.7461497281.97753926969205338.4503556805.8676767160.6596757314.78076226969220141.751122
2016-05-19 14:57:58.8954112006796.8339847160.4587367286.55127014738398234.1646866616.6831057132.7393187260.20214814738394234.9399156653.4916997158.4132777304.76904314738390738.659853
2016-06-14 16:27:58.7090964486964.9145517159.3780367325.37158220649705340.1492036907.3779307109.4776897304.16601620649705336.8259066928.1572277130.9535757326.69238320649705335.976938
2016-07-10 17:57:58.5227816965558.5742197167.4529267294.76318426981099239.2380896341.1083987113.3680687260.76269526981099236.5473276316.7919927131.9972807287.86572326981099238.203994
2016-08-05 19:27:58.3364669445780.0566417161.1743697307.21289126854905841.3216975629.0307627117.8348377252.26025426854905836.9489366436.2460947131.5317037271.19775426854905838.167856
2016-08-31 20:57:58.1501521926392.5209967161.8944067318.22851626898814639.9614106674.6235357126.2621917310.35791026898814636.3677026332.3105477142.1718417331.62939526898814637.958689
2016-09-26 22:27:57.9638374405955.3212897158.6709697300.72509825596254139.9770055741.3598637130.0955067287.33984425596254137.1624705688.1337897142.6807457298.87158225596254138.296652
2016-10-22 23:57:57.7775226885097.3129887156.9742347283.45361326878714936.3305413451.3640147127.4346717266.22119126878714936.1740465108.1953127151.0796727299.83984426878714137.499896
2016-11-18 01:27:57.5912079366591.2348637158.8551797281.74755927021597833.4856516529.4418957124.7363147247.68945327021597835.9028735967.0488287153.1319247305.47265627021597839.321384
\n", "
" ], "text/plain": [ " sunshine/PMU1 \\\n", " L1MAG \n", " min mean max \n", "2015-12-15 05:58:00.013299712 6686.668457 7162.191720 7262.521484 \n", "2016-02-05 08:57:59.640670208 771.582458 7154.288364 7298.772461 \n", "2016-03-02 10:27:59.454355456 4686.440918 7154.052820 7305.978516 \n", "2016-03-28 11:57:59.268040704 6825.371094 7157.580868 7301.885254 \n", "2016-04-23 13:27:59.081725952 6580.954102 7161.233046 7300.862305 \n", "2016-05-19 14:57:58.895411200 6796.833984 7160.458736 7286.551270 \n", "2016-06-14 16:27:58.709096448 6964.914551 7159.378036 7325.371582 \n", "2016-07-10 17:57:58.522781696 5558.574219 7167.452926 7294.763184 \n", "2016-08-05 19:27:58.336466944 5780.056641 7161.174369 7307.212891 \n", "2016-08-31 20:57:58.150152192 6392.520996 7161.894406 7318.228516 \n", "2016-09-26 22:27:57.963837440 5955.321289 7158.670969 7300.725098 \n", "2016-10-22 23:57:57.777522688 5097.312988 7156.974234 7283.453613 \n", "2016-11-18 01:27:57.591207936 6591.234863 7158.855179 7281.747559 \n", "\n", " \\\n", " L3MAG \n", " count stddev min mean \n", "2015-12-15 05:58:00.013299712 226238399 34.160419 6343.702637 7117.509685 \n", "2016-02-05 08:57:59.640670208 149673534 34.191939 809.082642 7123.868194 \n", "2016-03-02 10:27:59.454355456 270216261 37.179477 4284.337402 7123.881012 \n", "2016-03-28 11:57:59.268040704 269591739 36.514187 6722.911133 7125.577244 \n", "2016-04-23 13:27:59.081725952 269692154 35.614534 6822.678223 7128.746149 \n", "2016-05-19 14:57:58.895411200 147383982 34.164686 6616.683105 7132.739318 \n", "2016-06-14 16:27:58.709096448 206497053 40.149203 6907.377930 7109.477689 \n", "2016-07-10 17:57:58.522781696 269810992 39.238089 6341.108398 7113.368068 \n", "2016-08-05 19:27:58.336466944 268549058 41.321697 5629.030762 7117.834837 \n", "2016-08-31 20:57:58.150152192 268988146 39.961410 6674.623535 7126.262191 \n", "2016-09-26 22:27:57.963837440 255962541 39.977005 5741.359863 7130.095506 \n", "2016-10-22 23:57:57.777522688 268787149 36.330541 3451.364014 7127.434671 \n", "2016-11-18 01:27:57.591207936 270215978 33.485651 6529.441895 7124.736314 \n", "\n", " \\\n", " L2MAG \n", " max count stddev min \n", "2015-12-15 05:58:00.013299712 7243.531250 226238399 39.304326 6562.459961 \n", "2016-02-05 08:57:59.640670208 7309.245117 149673534 36.310027 846.452637 \n", "2016-03-02 10:27:59.454355456 7324.551758 270216271 40.578309 6265.968750 \n", "2016-03-28 11:57:59.268040704 7320.832031 269591712 42.586520 6803.852051 \n", "2016-04-23 13:27:59.081725952 7281.977539 269692053 38.450355 6805.867676 \n", "2016-05-19 14:57:58.895411200 7260.202148 147383942 34.939915 6653.491699 \n", "2016-06-14 16:27:58.709096448 7304.166016 206497053 36.825906 6928.157227 \n", "2016-07-10 17:57:58.522781696 7260.762695 269810992 36.547327 6316.791992 \n", "2016-08-05 19:27:58.336466944 7252.260254 268549058 36.948936 6436.246094 \n", "2016-08-31 20:57:58.150152192 7310.357910 268988146 36.367702 6332.310547 \n", "2016-09-26 22:27:57.963837440 7287.339844 255962541 37.162470 5688.133789 \n", "2016-10-22 23:57:57.777522688 7266.221191 268787149 36.174046 5108.195312 \n", "2016-11-18 01:27:57.591207936 7247.689453 270215978 35.902873 5967.048828 \n", "\n", " \n", " \n", " mean max count stddev \n", "2015-12-15 05:58:00.013299712 7144.443595 7285.520996 226238399 45.038278 \n", "2016-02-05 08:57:59.640670208 7148.059164 7338.802246 149673534 38.897277 \n", "2016-03-02 10:27:59.454355456 7148.867008 7367.452637 270216296 45.239399 \n", "2016-03-28 11:57:59.268040704 7156.322588 7389.022949 269591753 47.147822 \n", "2016-04-23 13:27:59.081725952 7160.659675 7314.780762 269692201 41.751122 \n", "2016-05-19 14:57:58.895411200 7158.413277 7304.769043 147383907 38.659853 \n", "2016-06-14 16:27:58.709096448 7130.953575 7326.692383 206497053 35.976938 \n", "2016-07-10 17:57:58.522781696 7131.997280 7287.865723 269810992 38.203994 \n", "2016-08-05 19:27:58.336466944 7131.531703 7271.197754 268549058 38.167856 \n", "2016-08-31 20:57:58.150152192 7142.171841 7331.629395 268988146 37.958689 \n", "2016-09-26 22:27:57.963837440 7142.680745 7298.871582 255962541 38.296652 \n", "2016-10-22 23:57:57.777522688 7151.079672 7299.839844 268787141 37.499896 \n", "2016-11-18 01:27:57.591207936 7153.131924 7305.472656 270215978 39.321384 " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/markdown": [ "---" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/opt/conda/lib/python3.11/site-packages/pingthings/timeseries/client.py:1636: UserWarning: Multistream Window and Aligned window queries do not currently support non-nullable fields. Will be replaced with Null values.\n", " warnings.warn(\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sunshine/PMU1
L1MAGL3MAGL2MAG
minmeanmaxcountstddevminmeanmaxcountstddevminmeanmaxcountstddev
time
2015-12-15 05:58:00.013299712+00:006686.6684577162.1918957262.52148422623839934.1604196343.7026377117.5097667243.53125022623839939.3043256562.4599617144.4433597285.52099622623839945.038277
2016-02-05 08:57:59.640670208+00:00771.5824587154.2885747298.77246114967353434.191940809.0826427123.8681647309.24511714967353436.310028846.4526377148.0590827338.80224614967353438.897278
2016-03-02 10:27:59.454355456+00:004686.4409187154.0527347305.97851627021626137.1794784284.3374027123.8808597324.55175827021627140.5783086265.9687507148.8671887367.45263727021629645.239399
2016-03-28 11:57:59.268040704+00:006825.3710947157.5810557301.88525426959173936.5141876722.9111337125.5771487320.83203126959171242.5865216803.8520517156.3227547389.02294926959175347.147823
2016-04-23 13:27:59.081725952+00:006580.9541027161.2329107300.86230526969215435.6145366822.6782237128.7460947281.97753926969205338.4503566805.8676767160.6596687314.78076226969220141.751122
2016-05-19 14:57:58.895411200+00:006796.8339847160.4584967286.55127014738398234.1646846616.6831057132.7392587260.20214814738394234.9399156653.4916997158.4130867304.76904314738390738.659855
2016-06-14 16:27:58.709096448+00:006964.9145517159.3779307325.37158220649705340.1492046907.3779307109.4775397304.16601620649705336.8259056928.1572277130.9536137326.69238320649705335.976936
2016-07-10 17:57:58.522781696+00:005558.5742197167.4531257294.76318426981099239.2380916341.1083987113.3681647260.76269526981099236.5473256316.7919927131.9970707287.86572326981099238.203995
2016-08-05 19:27:58.336466944+00:005780.0566417161.1743167307.21289126854905841.3216975629.0307627117.8349617252.26025426854905836.9489366436.2460947131.5317387271.19775426854905838.167854
2016-08-31 20:57:58.150152192+00:006392.5209967161.8945317318.22851626898814639.9614116674.6235357126.2622077310.35791026898814636.3677026332.3105477142.1718757331.62939526898814637.958691
2016-09-26 22:27:57.963837440+00:005955.3212897158.6708987300.72509825596254139.9770055741.3598637130.0957037287.33984425596254137.1624685688.1337897142.6806647298.87158225596254138.296650
2016-10-22 23:57:57.777522688+00:005097.3129887156.9741217283.45361326878714936.3305403451.3640147127.4345707266.22119126878714936.1740465108.1953127151.0795907299.83984426878714137.499897
2016-11-18 01:27:57.591207936+00:006591.2348637158.8549807281.74755927021597833.4856536529.4418957124.7363287247.68945327021597835.9028745967.0488287153.1318367305.47265627021597839.321384
\n", "
" ], "text/plain": [ " sunshine/PMU1 \\\n", " L1MAG \n", " min mean max \n", "time \n", "2015-12-15 05:58:00.013299712+00:00 6686.668457 7162.191895 7262.521484 \n", "2016-02-05 08:57:59.640670208+00:00 771.582458 7154.288574 7298.772461 \n", "2016-03-02 10:27:59.454355456+00:00 4686.440918 7154.052734 7305.978516 \n", "2016-03-28 11:57:59.268040704+00:00 6825.371094 7157.581055 7301.885254 \n", "2016-04-23 13:27:59.081725952+00:00 6580.954102 7161.232910 7300.862305 \n", "2016-05-19 14:57:58.895411200+00:00 6796.833984 7160.458496 7286.551270 \n", "2016-06-14 16:27:58.709096448+00:00 6964.914551 7159.377930 7325.371582 \n", "2016-07-10 17:57:58.522781696+00:00 5558.574219 7167.453125 7294.763184 \n", "2016-08-05 19:27:58.336466944+00:00 5780.056641 7161.174316 7307.212891 \n", "2016-08-31 20:57:58.150152192+00:00 6392.520996 7161.894531 7318.228516 \n", "2016-09-26 22:27:57.963837440+00:00 5955.321289 7158.670898 7300.725098 \n", "2016-10-22 23:57:57.777522688+00:00 5097.312988 7156.974121 7283.453613 \n", "2016-11-18 01:27:57.591207936+00:00 6591.234863 7158.854980 7281.747559 \n", "\n", " \\\n", " L3MAG \n", " count stddev min \n", "time \n", "2015-12-15 05:58:00.013299712+00:00 226238399 34.160419 6343.702637 \n", "2016-02-05 08:57:59.640670208+00:00 149673534 34.191940 809.082642 \n", "2016-03-02 10:27:59.454355456+00:00 270216261 37.179478 4284.337402 \n", "2016-03-28 11:57:59.268040704+00:00 269591739 36.514187 6722.911133 \n", "2016-04-23 13:27:59.081725952+00:00 269692154 35.614536 6822.678223 \n", "2016-05-19 14:57:58.895411200+00:00 147383982 34.164684 6616.683105 \n", "2016-06-14 16:27:58.709096448+00:00 206497053 40.149204 6907.377930 \n", "2016-07-10 17:57:58.522781696+00:00 269810992 39.238091 6341.108398 \n", "2016-08-05 19:27:58.336466944+00:00 268549058 41.321697 5629.030762 \n", "2016-08-31 20:57:58.150152192+00:00 268988146 39.961411 6674.623535 \n", "2016-09-26 22:27:57.963837440+00:00 255962541 39.977005 5741.359863 \n", "2016-10-22 23:57:57.777522688+00:00 268787149 36.330540 3451.364014 \n", "2016-11-18 01:27:57.591207936+00:00 270215978 33.485653 6529.441895 \n", "\n", " \\\n", " \n", " mean max count \n", "time \n", "2015-12-15 05:58:00.013299712+00:00 7117.509766 7243.531250 226238399 \n", "2016-02-05 08:57:59.640670208+00:00 7123.868164 7309.245117 149673534 \n", "2016-03-02 10:27:59.454355456+00:00 7123.880859 7324.551758 270216271 \n", "2016-03-28 11:57:59.268040704+00:00 7125.577148 7320.832031 269591712 \n", "2016-04-23 13:27:59.081725952+00:00 7128.746094 7281.977539 269692053 \n", "2016-05-19 14:57:58.895411200+00:00 7132.739258 7260.202148 147383942 \n", "2016-06-14 16:27:58.709096448+00:00 7109.477539 7304.166016 206497053 \n", "2016-07-10 17:57:58.522781696+00:00 7113.368164 7260.762695 269810992 \n", "2016-08-05 19:27:58.336466944+00:00 7117.834961 7252.260254 268549058 \n", "2016-08-31 20:57:58.150152192+00:00 7126.262207 7310.357910 268988146 \n", "2016-09-26 22:27:57.963837440+00:00 7130.095703 7287.339844 255962541 \n", "2016-10-22 23:57:57.777522688+00:00 7127.434570 7266.221191 268787149 \n", "2016-11-18 01:27:57.591207936+00:00 7124.736328 7247.689453 270215978 \n", "\n", " \\\n", " L2MAG \n", " stddev min mean \n", "time \n", "2015-12-15 05:58:00.013299712+00:00 39.304325 6562.459961 7144.443359 \n", "2016-02-05 08:57:59.640670208+00:00 36.310028 846.452637 7148.059082 \n", "2016-03-02 10:27:59.454355456+00:00 40.578308 6265.968750 7148.867188 \n", "2016-03-28 11:57:59.268040704+00:00 42.586521 6803.852051 7156.322754 \n", "2016-04-23 13:27:59.081725952+00:00 38.450356 6805.867676 7160.659668 \n", "2016-05-19 14:57:58.895411200+00:00 34.939915 6653.491699 7158.413086 \n", "2016-06-14 16:27:58.709096448+00:00 36.825905 6928.157227 7130.953613 \n", "2016-07-10 17:57:58.522781696+00:00 36.547325 6316.791992 7131.997070 \n", "2016-08-05 19:27:58.336466944+00:00 36.948936 6436.246094 7131.531738 \n", "2016-08-31 20:57:58.150152192+00:00 36.367702 6332.310547 7142.171875 \n", "2016-09-26 22:27:57.963837440+00:00 37.162468 5688.133789 7142.680664 \n", "2016-10-22 23:57:57.777522688+00:00 36.174046 5108.195312 7151.079590 \n", "2016-11-18 01:27:57.591207936+00:00 35.902874 5967.048828 7153.131836 \n", "\n", " \n", " \n", " max count stddev \n", "time \n", "2015-12-15 05:58:00.013299712+00:00 7285.520996 226238399 45.038277 \n", "2016-02-05 08:57:59.640670208+00:00 7338.802246 149673534 38.897278 \n", "2016-03-02 10:27:59.454355456+00:00 7367.452637 270216296 45.239399 \n", "2016-03-28 11:57:59.268040704+00:00 7389.022949 269591753 47.147823 \n", "2016-04-23 13:27:59.081725952+00:00 7314.780762 269692201 41.751122 \n", "2016-05-19 14:57:58.895411200+00:00 7304.769043 147383907 38.659855 \n", "2016-06-14 16:27:58.709096448+00:00 7326.692383 206497053 35.976936 \n", "2016-07-10 17:57:58.522781696+00:00 7287.865723 269810992 38.203995 \n", "2016-08-05 19:27:58.336466944+00:00 7271.197754 268549058 38.167854 \n", "2016-08-31 20:57:58.150152192+00:00 7331.629395 268988146 37.958691 \n", "2016-09-26 22:27:57.963837440+00:00 7298.871582 255962541 38.296650 \n", "2016-10-22 23:57:57.777522688+00:00 7299.839844 268787141 37.499897 \n", "2016-11-18 01:27:57.591207936+00:00 7305.472656 270215978 39.321384 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "start = timez.to_nanoseconds(\"2016-1-1 18:00:00\")\n", "end = timez.to_nanoseconds(\"2017-1-1 18:00:00\")\n", "pw = pt.utils.nearest_point_width(days=30)\n", "\n", "bp_df = (\n", " bp_pmu1.filter(name=re.compile(\"L.MAG\"), start=start, end=end)\n", " .aligned_windows(pointwidth=pw.point_width)\n", " .to_dataframe(agg=\"all\")\n", ")\n", "bp_df.index = [pd.Timestamp(i) for i in bp_df.index] # manually convert index to timestamp\n", "display(bp_df)\n", "\n", "display(Markdown(\"---\"))\n", "\n", "renamer: dict[str, tuple[str]] = {}\n", "for stream in pmu1_voltages:\n", " for metric in (\"min\", \"mean\", \"max\", \"count\", \"stddev\"):\n", " renamer[f\"{stream.uuid}/{metric}\"] = (stream.collection, stream.name, metric)\n", "\n", "pt_df = (\n", " pmu1_voltages.windowed_values(start, end, width=pw)\n", " .to_pandas()\n", " .set_index(\"time\")\n", " .rename(columns=renamer)\n", ")\n", "display(pt_df.reindex(pd.MultiIndex.from_tuples(pt_df.columns), axis=1))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }