← Overview
Extreme periods¶
Clustering finds typical behavior, and representation condenses each group into one profile. Both steps work against the rare period: a peak that happens once is, by construction, not typical, and averaging is exactly what removes it. For capacity sizing that one period is often the only one that matters.
Extreme periods are a clustering extension: after the groups are formed, a chosen period is forced into the representative set so it cannot be averaged away.
| In | a finished clustering, plus a rule for which period must survive |
| Inside | one of three ways to fit that period into the existing cluster set |
| Out | an amended set of representatives, assignments and counts |
import numpy as np
import pandas as pd
import plotly.io as pio
import tsam
from tsam import ClusterConfig, ExtremeConfig
pio.renderers.default = "notebook_connected"
ATTRS = ["solar", "load"]
UNITS = {"solar": "W/m²", "load": "MW"}
N_TIMESTEPS = 4
# The tiny six-day series and its period matrix D, both from 01_preprocessing.
tiny = pd.read_csv("../../data/tiny.csv", index_col=0, parse_dates=True)
D = pd.read_csv("../../data/tiny_periods.csv", header=[0, 1], index_col=0)
# This step starts from a finished clustering, so take k=3 Ward as given —
# the same partition the clustering notebooks trace.
base = tsam.aggregate(
tiny,
n_clusters=3,
period_duration="1D",
cluster=ClusterConfig(method="hierarchical"),
preserve_column_means=False, # rescaling is 05's topic; keep the raw profiles here
)
base_assignments = [int(c) for c in base.cluster_assignments]
print("k=3 Ward assignments, one per day:", base_assignments)
print("cluster counts:", dict(base.cluster_counts))
k=3 Ward assignments, one per day: [1, 1, 2, 2, 0, 0]
cluster counts: {0: 2.0, 1: 2.0, 2: 2.0}
1 What comes in: a clustering that lost the peak¶
The tiny series peaks at 10 MW on day5, t3. Ward at k=3 groups day5 with day4, and the
default medoid representation picks one of the two to speak for both. Whichever it picks, the
cluster's profile is a real day — but it is not day5, and the peak is gone.
peak_day = int(
D["load"].max(axis=1).idxmax()
) # the period holding the highest load timestep
print(f"original peak load: {tiny['load'].max()} {UNITS['load']} (day{peak_day})")
print(
f"highest load in any representative: "
f"{base.cluster_representatives['load'].max()} {UNITS['load']}"
)
print(
f"\nday{peak_day} sits in cluster {base_assignments[peak_day]}, whose representative is:"
)
base.cluster_representatives.loc[base_assignments[peak_day]]
original peak load: 10 MW (day5) highest load in any representative: 7.0 MW day5 sits in cluster 0, whose representative is:
| solar | load | |
|---|---|---|
| timestep | ||
| 0 | 0.0 | 6.0 |
| 1 | 1.0 | 6.0 |
| 2 | 1.0 | 7.0 |
| 3 | 0.0 | 7.0 |
2 Choosing which period is "extreme"¶
Extreme is not a property of the data — it is a question you ask of it, and tsam offers four ways to ask. Two look at a single timestep, two look at the period as a whole:
| Criterion | Selects the period containing… |
|---|---|
max_value |
the single highest timestep of that column |
min_value |
the single lowest timestep of that column |
max_period |
the highest average over the period |
min_period |
the lowest average over the period |
They do not have to agree. A day with one enormous spike wins max_value; a day that is
relentlessly high all through wins max_period.
# The same four questions tsam asks internally, on the normalized period matrix.
criteria = {
"max_value(load)": D["load"].max(axis=1).idxmax(),
"min_value(solar)": D["solar"].min(axis=1).idxmin(),
"max_period(load)": D["load"].mean(axis=1).idxmax(),
"min_period(solar)": D["solar"].mean(axis=1).idxmin(),
}
for name, day in criteria.items():
print(f" {name:20s} -> day{day}")
print("\nThis notebook uses max_value(load) — the day holding the 10 MW spike.")
EXTREME_DAY = criteria["max_value(load)"]
max_value(load) -> day5 min_value(solar) -> day0 max_period(load) -> day5 min_period(solar) -> day5 This notebook uses max_value(load) — the day holding the 10 MW spike.
3 Inside: three ways to fit it in¶
All three start from the same place — the extreme period's profile — and differ in what they do to the cluster set.
append — give it its own cluster¶
The extreme profile is added as a new cluster, and only the extreme period itself is reassigned to it. Every other period keeps the cluster it had. The cluster count rises by one per extreme.
new_cluster — its own cluster, and pull in the neighbors¶
Identical to append, plus one extra pass: every period is asked whether it is closer to the
extreme profile than to its own cluster's center, and if so it is moved into the extreme's
cluster. So new_cluster differs from append only when some period prefers the extreme —
otherwise the two are the same operation.
def medoid_center(matrix):
"""The member with the smallest total distance to its cluster-mates (03's rule)."""
dist = np.sqrt(((matrix[:, None, :] - matrix[None, :, :]) ** 2).sum(-1))
return matrix[int(np.argmin(dist.sum(axis=0)))]
# The k=3 centers, in the normalized space where tsam measures.
centers = {
c: medoid_center(
D.loc[[d for d, a in enumerate(base_assignments) if a == c]].values
)
for c in sorted(set(base_assignments))
}
extreme_profile = D.loc[EXTREME_DAY].values
# The exact test new_cluster runs for every period (squared distances, as in tsam).
print(
f"new_cluster's pull test — is a day closer to day{EXTREME_DAY} than to its own center?\n"
)
for day in range(len(D)):
if day == EXTREME_DAY:
continue
to_own = ((D.loc[day].values - centers[base_assignments[day]]) ** 2).sum()
to_extreme = ((D.loc[day].values - extreme_profile) ** 2).sum()
verdict = "PULLED" if to_extreme < to_own else "stays"
print(
f" day{day}: own center {to_own:.4f} vs day{EXTREME_DAY} {to_extreme:.4f} -> {verdict}"
)
new_cluster's pull test — is a day closer to day5 than to its own center? day0: own center 0.0000 vs day5 2.6751 -> stays day1: own center 0.0517 vs day5 2.8996 -> stays day2: own center 0.0000 vs day5 0.7985 -> stays day3: own center 0.0925 vs day5 1.0134 -> stays day4: own center 0.0000 vs day5 0.2401 -> stays
Nothing is pulled — so on this dataset new_cluster and append do exactly the same thing.
That is not a quirk to skip past; it is the clearest statement of what the knob does. The peak
day is an outlier: being far from everything is why it was selected as the extreme, and that
same distance is why no other day prefers it. new_cluster earns its name when the extreme is
less lonely — a peak day flanked by several near-peak days will pull those neighbors in, and
the original cluster shrinks. On six days with two-member clusters, that never arises.
replace — overwrite an existing center¶
The third rule adds no cluster at all. It takes the cluster the extreme period already belongs to and overwrites that center's values — but only in the column the criterion named. The cluster count is unchanged.
results = {
method: tsam.aggregate(
tiny,
n_clusters=3,
period_duration="1D",
cluster=ClusterConfig(method="hierarchical"),
extremes=ExtremeConfig(method=method, max_value=["load"]),
preserve_column_means=False,
)
for method in ["append", "new_cluster", "replace"]
}
home = base_assignments[EXTREME_DAY]
print(f"Cluster {home} — the one day{EXTREME_DAY} was already in.\n")
print("before (medoid of the cluster):")
print(base.cluster_representatives.loc[home].to_string())
print("\nafter `replace`:")
print(results["replace"].cluster_representatives.loc[home].to_string())
before, after = (
base.cluster_representatives.loc[home],
results["replace"].cluster_representatives.loc[home],
)
print()
for attr in ATTRS:
changed = not np.allclose(before[attr].values, after[attr].values)
print(f" {attr:6s} changed: {changed}")
Cluster 0 — the one day5 was already in.
before (medoid of the cluster):
solar load
timestep
0 0.0 6.0
1 1.0 6.0
2 1.0 7.0
3 0.0 7.0
after `replace`:
solar load
timestep
0 0.0 6.0
1 1.0 7.0
2 1.0 8.0
3 0.0 10.0
solar changed: False
load changed: True
replace spliced two different days together. The load column is now day5's — peak intact —
while the solar column is still the medoid's. The result is a profile that never happened:
it pairs one day's demand with another day's sunshine.
That is the cost the name does not advertise. Representation drew a
line between representatives that are real periods and ones that are constructed; replace
takes a real period and makes it constructed, breaking the correlation between attributes that
made it realistic. If a downstream model cares that high demand coincided with low sun, replace
quietly discards exactly that.
append and new_cluster do not have this problem: they insert the extreme period whole.
4 What comes out¶
The three rules leave the pipeline in visibly different states. Read the assignment vector as "which representative stands in for each calendar day":
rows = [
{
"strategy": "none (baseline)",
"n_clusters": base.n_clusters,
"assignments": str(base_assignments),
"counts": str(dict(base.cluster_counts)),
"peak load in reps": round(
float(base.cluster_representatives["load"].max()), 2
),
}
]
for method, result in results.items():
rows.append(
{
"strategy": method,
"n_clusters": result.n_clusters,
"assignments": str([int(c) for c in result.cluster_assignments]),
"counts": str(dict(result.cluster_counts)),
"peak load in reps": round(
float(result.cluster_representatives["load"].max()), 2
),
}
)
pd.DataFrame(rows).set_index("strategy")
| n_clusters | assignments | counts | peak load in reps | |
|---|---|---|---|---|
| strategy | ||||
| none (baseline) | 3 | [1, 1, 2, 2, 0, 0] | {0: 2.0, 1: 2.0, 2: 2.0} | 7.0 |
| append | 4 | [1, 1, 2, 2, 0, 3] | {0: 1.0, 1: 2.0, 2: 2.0, 3: 1.0} | 10.0 |
| new_cluster | 4 | [1, 1, 2, 2, 0, 3] | {0: 1.0, 1: 2.0, 2: 2.0, 3: 1.0} | 10.0 |
| replace | 3 | [1, 1, 2, 2, 0, 0] | {0: 2.0, 1: 2.0, 2: 2.0} | 10.0 |
All three recover the 10 MW peak; what they charge for it differs.
append/new_clusterbuy a fourth cluster. day5 leaves cluster 0 and stands alone with an occurrence count of 1 — it now represents only itself, which is exactly what a once-in-a-series event should do. The model pays with one more typical period to solve over.replacekeeps three clusters and pays in realism instead: no extra period, but cluster 0's profile is now a splice, and its count of 2 means that spliced day is claimed to occur twice.
A last consequence, easy to miss: an extreme cluster holding a single period has an occurrence count of 1, so its representative contributes almost nothing to the column totals — which is why rescaling deliberately leaves extreme clusters alone when it corrects the means.
Up next:
- Rescaling — restore the column totals a non-mean representative distorts (extreme clusters excluded), and return the profiles to physical units
See also:
- Representation — where the peak was lost in the first place, and the
maxoid/distribution_minmaxrules that push back on it without adding a cluster - Extreme periods how-to — choosing a strategy on a realistic series