← Overview
Agglomerative clustering: hierarchical and contiguous¶
Both methods in this notebook use Ward agglomerative clustering — bottom-up merging that minimizes the increase in within-cluster variance at each step. They differ only in whether merges are restricted to temporally adjacent periods.
Relationship to segmentation:
contiguous(period-level) andSegmentConfig(timestep-level within a period) are the same algorithm applied at different granularities. See Segmentation for the timestep-level view.
1 Hierarchical clustering (unconstrained Ward)¶
Mechanism: bottom-up agglomeration using Ward linkage:
- Start with every period in its own cluster.
- Find the pair of clusters whose merge increases total within-cluster variance the least (Ward criterion).
- Merge them.
- Repeat until $k$ clusters remain.
The Ward merge cost between clusters $A$ and $B$ is:
$$ \Delta(A, B) = \frac{|A| \cdot |B|}{|A| + |B|} \| \bar{x}_A - \bar{x}_B \|^2 $$
where $|A|$, $|B|$ are cluster sizes and $\bar{x}_A$, $\bar{x}_B$ are their centroids.
1.1 In — what Ward's hierarchical algorithm receives¶
Exactly one thing: the input period matrix $D$. To showcase the algorithm we use the
normalized (6, 8) array of floats created in the preprocessing
notebook — one row per period (6), one column per (attribute, timestep) pair (2 × 4 = 8
coordinates).
Note what is not in there: the algorithm receives an anonymous float array. No dates, no column names, no ordering information — nothing that tells it day 0 came before day 1. Ward cannot favor neighboring days because it cannot see which days are neighbors. That fact is what section 2 has to work around.
import numpy as np
import pandas as pd
import plotly.express as px
import plotly.figure_factory as ff
import plotly.graph_objects as go
import plotly.io as pio
import scipy.cluster.hierarchy as sch
from sklearn.cluster import AgglomerativeClustering
from sklearn.metrics import adjusted_rand_score
# tsam's *own* clustering entry point — the same function tsam.aggregate calls
# internally to turn the period matrix D into cluster labels.
from tsam.algorithms.clustering import assign_clusters
pio.renderers.default = "notebook_connected"
# Preprocessed period matrix D (normalized + unstacked) from 01_preprocessing.
tiny_period_df = pd.read_csv(
"../../../data/tiny_periods.csv", header=[0, 1], index_col=0
)
tiny_period_array = tiny_period_df.values # shape (6, 8): six periods, eight features
N_PERIODS = tiny_period_array.shape[0]
N_ATTRS, N_TIMESTEPS = 2, 4
DAYS = [f"day_{p}" for p in range(N_PERIODS)]
# D as the estimator receives it: floats only, no time, no names.
tiny_period_df.round(4)
| solar | load | |||||||
|---|---|---|---|---|---|---|---|---|
| TimeStep | 0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 |
| PeriodNum | ||||||||
| 0 | 0.0 | 1.000 | 0.750 | 0.0 | 0.0000 | 0.0000 | 0.1429 | 0.2857 |
| 1 | 0.0 | 0.875 | 0.875 | 0.0 | 0.0000 | 0.0000 | 0.1429 | 0.1429 |
| 2 | 0.0 | 0.375 | 0.250 | 0.0 | 0.1429 | 0.2857 | 0.2857 | 0.4286 |
| 3 | 0.0 | 0.250 | 0.375 | 0.0 | 0.1429 | 0.1429 | 0.4286 | 0.2857 |
| 4 | 0.0 | 0.125 | 0.125 | 0.0 | 0.4286 | 0.4286 | 0.5714 | 0.5714 |
| 5 | 0.0 | 0.125 | 0.000 | 0.0 | 0.4286 | 0.5714 | 0.7143 | 1.0000 |
1.2 Middle — the merge history¶
To showcase how the algorithm works we apply Ward and look at the merge history.
assign_clusters asks for a fixed n_clusters, so it only ever sees one cut of the tree
and throws the rest away. But the estimator computes the whole thing, and if you fit it with
distance_threshold=0 instead, it keeps every merge and exposes the history on two attributes:
children_— the pair merged at each step. Ids0…5are the original days; id6 + jis the cluster created at stepj. So the tree is written in terms of its own history.distances_— the Ward cost $\Delta(A, B)$ of that merge, i.e. how much within-cluster variance it cost to join the pair.
Five merges take six singleton clusters down to one. That table is the entire intermediate state of the algorithm — everything else in this section is a different rendering of it.
# The same estimator assign_clusters builds, but fitted to keep *every* merge
# (distance_threshold=0) instead of stopping at k. Same data, same Ward criterion —
# only the stopping point differs.
tree = AgglomerativeClustering(
n_clusters=None, distance_threshold=0, linkage="ward", compute_distances=True
).fit(tiny_period_array)
# Unroll that history into a readable table. `inside[i]` = the days sitting in cluster id i.
days_inside_column = {p: {p} for p in range(N_PERIODS)}
history = []
for step, ((period_1_merged, period_2_merged), cost) in enumerate(
zip(tree.children_, tree.distances_)
):
new_id = N_PERIODS + step # ids 0..5 are days; 6+ are clusters born here
days_inside_column[new_id] = (
days_inside_column[period_1_merged] | days_inside_column[period_2_merged]
)
history.append(
{
"step": step,
"merged": f"{period_1_merged} + {period_2_merged}",
"ward cost": round(float(cost), 4),
"-> new cluster id": new_id,
"days inside it": sorted(days_inside_column[new_id]),
"clusters left": N_PERIODS - step - 1,
}
)
pd.DataFrame(history).set_index("step")
| merged | ward cost | -> new cluster id | days inside it | clusters left | |
|---|---|---|---|---|---|
| step | |||||
| 0 | 0 + 1 | 0.2273 | 6 | [0, 1] | 5 |
| 1 | 2 + 3 | 0.3041 | 7 | [2, 3] | 4 |
| 2 | 4 + 5 | 0.4900 | 8 | [4, 5] | 3 |
| 3 | 7 + 8 | 1.0259 | 9 | [2, 3, 4, 5] | 2 |
| 4 | 6 + 9 | 1.9124 | 10 | [0, 1, 2, 3, 4, 5] | 1 |
Reading the history. Ward merges the two sunny days first (cost 0.23), then the two overcast days (0.30), then the two cloudy days (0.49) — the three shape-pairs, cheapest first. Only then, having run out of cheap merges, does it start joining pairs to each other, and the price jumps: 0.49 → 1.03. That jump is the argument for cutting at k=3. Below it, merges are nearly free because the days really are alike; above it, every further merge forces genuinely different days together.
The two views that follow are the same five rows: first the cost of each merge in order, which is what decides the merge sequence, then the tree those merges build.
# View 1 — the merge cost at each step. Ward always takes the cheapest merge available,
# so this curve *is* the merge order: reading it left to right replays the algorithm.
steps = np.arange(len(tree.distances_))
labels_merged = [
f"{sorted(days_inside_column[a] | days_inside_column[b])}"
for a, b in tree.children_
]
fig_cost = go.Figure()
fig_cost.add_trace(
go.Scatter(
x=steps,
y=tree.distances_,
mode="lines+markers+text",
text=[f"{c:.2f}" for c in tree.distances_],
textposition="top left",
marker={"size": 11, "color": "#1f4ea1"},
line={"color": "#1f4ea1"},
customdata=labels_merged,
hovertemplate="step %{x}: cost %{y:.4f}<br>days now together: %{customdata}<extra></extra>",
showlegend=False,
)
)
# The jump between step 2 and step 3 is the k=3 argument, made visible.
fig_cost.add_annotation(
x=2.5,
y=(tree.distances_[2] + tree.distances_[3]) / 2,
ax=70,
ay=0,
text="<b>the jump</b><br>cheap merges run out<br>→ cut here, k=3",
showarrow=True,
arrowhead=2,
font={"color": "#EF553B"},
arrowcolor="#EF553B",
)
fig_cost.add_vrect(
x0=-0.3,
x1=2.3,
fillcolor="#00CC96",
opacity=0.10,
line_width=0,
annotation_text="merging alike days",
annotation_position="top left",
)
fig_cost.add_vrect(
x0=2.7,
x1=4.3,
fillcolor="#EF553B",
opacity=0.10,
line_width=0,
annotation_text="merging unlike days",
annotation_position="top right",
)
fig_cost.update_layout(
title=(
"Ward cost of each merge, in the order they happen<br>"
"<sup>Every step takes the cheapest merge still available, so the curve only "
"rises. Where it rises <i>sharply</i> is where the natural groups end.</sup>"
),
xaxis_title="merge step",
yaxis_title="Ward cost Δ(A, B)",
xaxis={"tickmode": "array", "tickvals": steps},
height=380,
)
fig_cost.show()
# View 2 — the dendrogram: the same five merges drawn as the tree they build.
# Merge cost is the y-axis here too, so the k=3 jump is the tall gap.
#
# scipy renders the picture; the tree it draws is the same one sklearn just built.
linkage = sch.ward(tiny_period_array)
print(
"scipy's tree == the sklearn tree tsam uses:",
np.allclose(linkage[:, :3], np.column_stack([tree.children_, tree.distances_])),
)
fig_dendro = ff.create_dendrogram(
tiny_period_array,
labels=DAYS,
linkagefun=lambda _pdist: linkage,
colorscale=px.colors.qualitative.Set1[:6],
)
fig_dendro.update_layout(
title=(
"Ward dendrogram — the merge history as a tree<br>"
"<sup>Bar height = ward cost. The three shape-pairs join cheaply near the "
"floor; the tall gap above them is where k=3 sits.</sup>"
),
yaxis_title="Ward cost of the merge",
xaxis_title="Period",
)
fig_dendro.show()
scipy's tree == the sklearn tree tsam uses: True
1.3 Out — what comes back¶
One array of six integers: labels[p] is the cluster of day p. That is the whole output
contract of the clustering step — the tree, the costs and the merge order are all discarded,
and only one cut of the dendrogram above survives into the rest of the pipeline.
Below, the k=3 cut is obtained two ways: by cutting the tree we just drew, and by calling
tsam's own assign_clusters on $D$. The two must agree, and they do.
# (a) The tree above, cut at k=3.
labels_from_tree = sch.fcluster(linkage, t=3, criterion="maxclust")
# (b) tsam's clustering step, called directly on the period matrix D.
labels_assign = assign_clusters(
tiny_period_array, n_clusters=3, cluster_method="hierarchical"
)
print("(a) tree cut at k=3 :", labels_from_tree)
print("(b) assign_clusters(D, 3):", labels_assign)
# The cluster *numbers* differ; the *grouping* must not. A rand score of 1.0 means the two
# labellings partition the days identically, whatever the groups happen to be called.
print(
"\n(a) vs (b) identical partition:",
adjusted_rand_score(labels_from_tree, labels_assign) == 1.0,
)
(a) tree cut at k=3 : [1 1 2 2 3 3] (b) assign_clusters(D, 3): [1 1 2 2 0 0] (a) vs (b) identical partition: True
Reading the assignment array. Entry p is the cluster of tiny day p. Both routes
group days 0–1 (sunny), 2–3 (overcast) and 4–5 (cloudy) — the same three shape-pairs that
k-means and k-medoids find in the partitional notebook,
reached here by a completely different route: cheapest-merge-first, as the history table
showed.
The cluster numbers differ between the routes (fcluster numbers by tree position,
scikit-learn by merge order), which is why the check uses the rand score rather than ==:
what a clustering asserts is which days share a cluster, never what the group is called.
Downstream code must never depend on the numbering.
2 Contiguous clustering — Ward with a temporal-adjacency constraint¶
Common misconception corrected:
contiguousis sometimes described as "time-based". This is wrong. Algorithmically it is Ward agglomerative clustering with an adjacency-matrix connectivity constraint — identical tohierarchical, except the connectivity matrix restricts merges to immediately adjacent periods only. It is therefore:
- Feature-based (uses Ward / within-cluster variance, i.e. feature similarity)
- With a time-contiguity constraint (can only merge neighbors)
It sits in the feature-based row of the Hoffmann taxonomy.
Section 1.1 made the point that the input period matrix $D$ carries no time information, so
Ward cannot possibly favor neighboring days. Time therefore has to arrive as a second
input: the bidiagonal adjacency matrix $I(|i-j| = 1)$, passed to the same estimator as
connectivity. That single extra argument is the entire difference between the two methods —
same $D$, same Ward criterion, same shape of output. Only the set of merges Ward is allowed
to consider changes.
Connection to segmentation: contiguous and
segmentation are the same algorithm
(Ward + adjacency constraint) applied at different granularities:
contiguousmerges periods (rows of the D matrix)- segmentation merges timesteps within each period
# The second input, exactly as assign_clusters builds it (clustering.py):
adj = np.eye(N_PERIODS, k=1) + np.eye(N_PERIODS, k=-1) # bidiagonal: I(|i - j| = 1)
print("In (1): D, unchanged ", tiny_period_array.shape)
print(
"In (2): adjacency ", adj.shape, "— 1 where two periods are neighbors in time"
)
print(pd.DataFrame(adj.astype(int), index=DAYS, columns=DAYS), "\n")
# Same function, same D, one word changed: cluster_method="contiguous". Internally that
# passes `adj` to AgglomerativeClustering as `connectivity` and changes nothing else.
labels_contiguous = assign_clusters(
tiny_period_array, n_clusters=3, cluster_method="contiguous"
)
print("Out: same shape as before —", labels_contiguous)
In (1): D, unchanged (6, 8)
In (2): adjacency (6, 6) — 1 where two periods are neighbors in time
day_0 day_1 day_2 day_3 day_4 day_5
day_0 0 1 0 0 0 0
day_1 1 0 1 0 0 0
day_2 0 1 0 1 0 0
day_3 0 0 1 0 1 0
day_4 0 0 0 1 0 1
day_5 0 0 0 0 1 0
Out: same shape as before — [1 1 2 2 0 0]
The constrained tree, drawn the same way. The clearest way to see what the constraint did
is to build the whole tree again — this time with connectivity=adj — and put its dendrogram
next to the unconstrained one from section 1.
AgglomerativeClustering reports its tree as children_ + distances_; scipy wants those as
a four-column linkage matrix, so the helper below adds the one column scikit-learn omits: the
number of original days sitting under each merge.
def linkage_from_tree(model, n_samples):
"""scikit-learn's (children_, distances_) as a scipy 4-column linkage matrix.
scipy also wants the size of each merged cluster, which scikit-learn does not
store, so count it by walking the tree bottom-up.
"""
counts = np.zeros(model.children_.shape[0])
for i, (left, right) in enumerate(model.children_):
counts[i] = sum(
1 if child < n_samples else counts[child - n_samples]
for child in (left, right)
)
return np.column_stack([model.children_, model.distances_, counts]).astype(float)
# The same fit as section 1, plus the one extra argument.
tree_contiguous = AgglomerativeClustering(
n_clusters=None,
distance_threshold=0,
linkage="ward",
connectivity=adj,
compute_distances=True,
).fit(tiny_period_array)
linkage_contiguous = linkage_from_tree(tree_contiguous, N_PERIODS)
fig_dendro_c = ff.create_dendrogram(
tiny_period_array,
labels=DAYS,
linkagefun=lambda _pdist: linkage_contiguous,
colorscale=px.colors.qualitative.Set1[:6],
)
fig_dendro_c.update_layout(
title=(
"Contiguous Ward dendrogram — only neighbors were allowed to merge<br>"
"<sup>Compare with the unconstrained tree above: on this dataset they are "
"identical, because every merge Ward wanted was already between neighbors.</sup>"
),
yaxis_title="Ward cost of the merge",
xaxis_title="Period",
)
fig_dendro_c.show()
# Did the constraint change anything at all here?
same_tree = np.allclose(tree.distances_, tree_contiguous.distances_) and np.array_equal(
tree.children_, tree_contiguous.children_
)
print("unconstrained merge costs:", np.round(tree.distances_, 4))
print("contiguous merge costs:", np.round(tree_contiguous.distances_, 4))
print("\nIdentical tree?", same_tree)
print(
"Identical k=3 partition?",
adjusted_rand_score(labels_assign, labels_contiguous) == 1.0,
)
unconstrained merge costs: [0.2273 0.3041 0.49 1.0259 1.9124] contiguous merge costs: [0.2273 0.3041 0.49 1.0259 1.9124] Identical tree? True Identical k=3 partition? True
Why the tiny set cannot show the difference. The two trees come out identical — same merges, same costs, same partition. That is not a bug, and it is worth understanding before you read anything into it.
The constraint only bites when Ward wants a merge that the adjacency matrix forbids. Here it never does: the six days were deliberately laid out so that similar shapes are already neighbors (sunny 0–1, overcast 2–3, cloudy 4–5), so every cheapest-available merge happened to be between adjacent days anyway. The constraint was satisfied for free, and forbidding non-neighbor merges cost nothing.
Scramble the calendar and the two methods separate immediately: unconstrained Ward would still
find the three shape-pairs, while contiguous — forbidden from reaching across the series —
would be forced to merge dissimilar neighbors and pay a higher $J$ for it. That is the real
trade contiguous makes: it gives up some accuracy to guarantee that every cluster is a
solid block of calendar time, which is what you need when a downstream model reasons about
chronology (seasonal storage, for instance). See
Comparing clustering methods for that
gap measured on a realistic, unordered series.
Up next:
- Extremal-prototype selection — k-maxoids, the spread-maximizing method
See also:
- Averaging — the one time-based grouping method (positional blocks)
- Segmentation — the timestep-level analogue of
contiguous