Your first aggregation¶
ETHOS.TSAM compresses a long time series into a handful of typical periods — representative day-shapes that stand in for the whole record. In this tutorial you take six weeks of hourly data and shrink it to 6 typical days with a single call, then check that the compressed version still behaves like the original.
By the end you will be able to:
- run an aggregation with
tsam.aggregate(), - read the result — the typical periods, how many real days each represents, and the accuracy,
- reconstruct the original timeline from the typical periods and judge the error, and
- trade size for accuracy by letting tsam search for good configurations.
Everything here works the same on a full multi-year dataset — we use a six-week slice only so the plots stay readable. You don't need to understand how the clustering works yet. For that, see the how-aggregation-works explanation series.
import pandas as pd
import plotly.express as px
import plotly.io as pio
import tsam
pio.renderers.default = "notebook_connected"
raw = pd.read_csv("../data/testdata.csv", index_col=0, parse_dates=True)
data = raw.loc["2010-01-01":"2010-02-11"] # six weeks of hourly data
# Inferred units for this example dataset (adjust if, e.g., your Load is in kW).
UNITS = {"GHI": "W/m²", "T": "°C", "Wind": "m/s", "Load": "MW"}
data
| GHI | T | Wind | Load | |
|---|---|---|---|---|
| 2010-01-01 00:30:00 | 0 | -2.8 | 8.6 | 364.541326 |
| 2010-01-01 01:30:00 | 0 | -3.3 | 9.7 | 357.416844 |
| 2010-01-01 02:30:00 | 0 | -3.2 | 9.8 | 350.191306 |
| 2010-01-01 03:30:00 | 0 | -3.2 | 9.4 | 345.161449 |
| 2010-01-01 04:30:00 | 0 | -3.2 | 10.0 | 340.678216 |
| ... | ... | ... | ... | ... |
| 2010-02-11 19:30:00 | 0 | 5.1 | 2.0 | 557.214697 |
| 2010-02-11 20:30:00 | 0 | 5.1 | 1.0 | 537.347911 |
| 2010-02-11 21:30:00 | 0 | 5.1 | 2.0 | 516.011213 |
| 2010-02-11 22:30:00 | 0 | 5.3 | 4.0 | 484.996060 |
| 2010-02-11 23:30:00 | 0 | 5.3 | 3.0 | 461.348843 |
1008 rows × 4 columns
Step 1 — Look at the data¶
The dataset has four attributes — GHI (irradiance), T (temperature), Wind, and Load. We will follow Load through the whole tutorial. Here it is across the full six weeks. You can already make out the daily up-and-down rhythm and a slow seasonal drift from left to right.
fig = px.line(
data.reset_index(names="time"),
x="time",
y="Load",
title="Original Load — six weeks of hourly data",
)
fig.update_traces(name="Load", showlegend=True)
fig.update_yaxes(title_text=f"Load [{UNITS['Load']}]")
fig.update_xaxes(title_text="time")
fig.update_layout(legend_title_text="")
fig.show()
Step 2 — Aggregate to typical days¶
With a single call we reduce the six weeks to 6 typical days. n_clusters is how many
typical days you want. period_duration="1D" says each period is one day.
result = tsam.aggregate(data, n_clusters=6, period_duration="1D")
That one call runs the feature-based merging workflow from the time-series-aggregation review by Hoffmann et al. (2020): it normalizes each attribute, slices the series into daily periods, clusters them under a chosen distance metric and representation, and optionally rescales to preserve attribute means. In that review's taxonomy tsam sits in the clustering branch — and also offers segmentation and extreme periods, while never changing the number of attributes. Notation and equations gives the equations and the Pipeline Guide the full flow.
Step 3 — From members to representatives¶
Aggregation works by grouping similar days. Each of the 42 days is assigned to one of the 6 clusters, and every day in a cluster is a member. Here are the same six weeks as above, now colored by the cluster each day was assigned to.
result.plot.clusters_over_time(
columns=["Load"], units=UNITS, title="Six weeks of Load, colored by cluster"
)
The colors show which cluster each day landed in. Now look inside one cluster. Each cluster's member days are the faint lines and the bold line is the single representative that stands in for them, both drawn in that cluster's color. Use the slider to step through the clusters.
result.plot.cluster_members(columns=["Load"], units=UNITS)
The six representatives on one axis, each drawn in its cluster color and labelled with how many real days it stands for:
result.plot.cluster_representatives(
columns=["Load"], units=UNITS, title="The six typical days"
)
Step 4 — Did it hold up?¶
To check the compression, expand the typical days back over the original six weeks. Each real day is replaced by its representative, so every day is shaded with its cluster color. The dashed line is the original Load laid on top, so you can see where the reconstruction differs.
result.plot.clusters_over_time(
columns=["Load"],
reconstructed=True,
overlay_original=True,
units=UNITS,
title="Reconstructed Load, colored by cluster (original dashed)",
)
Now zoom into a single week to see the fine detail:
week = slice("2010-01-11", "2010-01-17")
result.plot.compare(
columns=["Load"],
time_slice=week,
color="source",
units=UNITS,
title="One week: original vs. reconstructed Load",
)
And the error as a single number per column:
result.accuracy.rmse
GHI 0.070131 T 0.110487 Wind 0.188102 Load 0.092193 Name: RMSE, dtype: float64
Step 5 — How small can you go?¶
Six typical days was a guess. The real question is the trade-off: how much accuracy do
you give up at each level of compression? find_pareto_front searches combinations of
typical periods (how many typical days) and segments (how many timesteps per day),
and keeps only the configurations that are Pareto-optimal — nothing smaller is also more
accurate. We let it run all the way up to full resolution, where the error reaches zero
because nothing has been aggregated.
There is no single right answer on this curve. You read it the other way around: decide
how much error you can accept, then take the smallest configuration that stays under it.
pareto.find_by_rmse(0.05) does exactly that. Hover any point to see the configuration
behind it.
from tsam.tuning import find_pareto_front
pareto = find_pareto_front(
data,
period_duration="1D",
timesteps=[6, 24, 48, 96, 168, 336, 504, 720, 1008],
numerical_tolerance=1e-9,
show_progress=False,
)
pareto.plot()
Recap, and where to go next¶
You ran the whole core workflow: you aggregated six weeks into 6 typical days with one
call, saw how each typical day represents a group of real days, reconstructed the
full timeline and checked the error, and traced the size-vs-accuracy trade-off.
To turn a Pareto point into a usable aggregation, pareto.summary lists the points as a
table and pareto.find_by_rmse(0.05) or pareto.find_by_timesteps(100) hand you back a
ready-to-use result.
From here:
Read this next
- Choosing a method — the four decisions
aggregate()just made for you, what each one throws away, and how to read a configuration off what you are modelling. It is the map the rest of the documentation hangs off.
Make it smaller
- Segmentation — also merge timesteps within each day.
- How small can you go? — the full guide to searching the periods x segments grid.
Control what it preserves
- Clustering methods — choose how days are grouped.
- Representations — choose what each typical day keeps (mean, distribution, min/max).
- Extreme periods — force the peak day to survive.
Put it to work
- Optimization workflow — hand the representatives, counts, and
assignments to a downstream model, and map results back with
disaggregate().
Know what it costs
- How long will this take? — clustering runtime by method and dataset size, and the parameter that moves it most.
Understand how it works
- How aggregation works — the explanation series that traces every step by hand on a tiny dataset.