Clustering & k-NN Analysis Results — HKTVMall Contact Lens Reviews

Author: Composer (Cursor AI Agent)
Date: 2026-06-26
Script: cluster_reviews.py
Outputs: cluster_output/review_clusters.parquet, cluster_output/cluster_report.json

Sources

Methodology

Feature engineering

Each review is encoded as a fixed-length vector, StandardScaler-normalized:

Canonical topics: comfort, quality, price, packaging, delivery, repurchase, prescription, brand, appearance, customer_service, eye_health, expiry, fit_sizing, other

Algorithms

Two-pass design

Pass A — Full VoC Segmentation

Best k: 5 (silhouette 0.469)
k-NN coherence (k=15): mean 0.997, median 1.000, 99.5% of reviews have ≥80% same-cluster neighbors

k-selection table

Cluster profiles

Sentiment mix by cluster

Top topics by cluster

Top brands by cluster

Centroid signatures (top z-scored features)

Prototype reviews (nearest to centroid)

Cluster 0 — Silent promoters

Cluster 1 — Articulate advocates

Cluster 2 — Vocal detractors

Cluster 3 — Ambivalent critics

Pass A interpretation

The market splits into ~82% promoters (clusters 0 + 1) with two behavioral modes:

• Silent (40%) — ultra-short 5★ reviews, corrective-lens topic bias
• Articulate (42%) — comfort/appearance-led praise, OLENS-heavy

~10% hard detractors (cluster 2) write long fit + eye_health complaints at ★1.8.
~8% ambivalent (cluster 3) are structurally mixed — trade-off language at ★3.6.

Clusters 0 and 1 share high stars but differ in verbosity and topic emphasis, not sentiment.

Pass B — Topic Archetypes (no sentiment features)

Best k: 4 (silhouette 0.414)
k-NN coherence (k=15): mean 0.973, median 1.000, 95.6% with ≥80% same-cluster neighbors

k-selection table

Cluster profiles

Sentiment mix by cluster (within topic archetypes)

Top topics by cluster

Top brands by cluster

Centroid signatures

Prototype reviews

Cluster 0 — Value loyalists

Cluster 1 — Corrective lens buyers

Cluster 2 — Failure bundle

Cluster 3 — Cosmetic enthusiasts

Pass B interpretation

Three distinct happy paths converge on ~4.8★ for different reasons:

1. Corrective (40%) — fit, prescription, expiry, brand trust
2. Value (27%) — price, repurchase, packaging, delivery
3. Cosmetic (21%) — comfort, appearance, quality; OLENS over-indexed

Failure bundle (12%) is topic-structural: fit + packaging + delivery co-occur with low ratings and high complaint volume. It captures 74% negative AND 23% mixed — the mixed reviews here are ops/fit problems expressed ambivalently, not price trade-offs.

Cross-tab: Pass A (tone) × Pass B (experience)

Rows = kmeans_cluster (Pass A), columns = topic_cluster (Pass B):

Cross-tab reads

k-NN validation summary

Pass A coherence is near-perfect because sentiment one-hot features dominate separation. Pass B's slightly lower coherence (97.3%) reflects softer boundaries between value and cosmetic happy paths — expected and healthy.

HDBSCAN sanity check (Pass A vectors)

HDBSCAN over-segments for executive use. KMeans with k=4–5 yields interpretable archetypes; HDBSCAN useful for micro-SKU anomaly detection later.

Strategic actions by segment

Reproducing

cd /home/sidmishra/ontologer/subprojects/review_sequencing_sentiment
source venv/bin/activate
python cluster_reviews.py

Outputs:

• cluster_output/review_clusters.parquet — all reviews with kmeans_cluster, topic_cluster, hdbscan_cluster
• cluster_output/cluster_report.json — full numeric results and prototypes

Join key: review_id → base cl_reviews.parquet or enrichment file.

End of clustering results.

Updated: 2026-06-26 13:55 via generate_clustering_results.py

Report block convention

Each analytical section uses up to three executable layers:

Prose How we found this (math) gives the formulas. Cached artifacts live in cluster_output/.

Pass A — Full VoC Segmentation (auto-generated blocks)

[result:review_cluster_k_selection] | k | Silhouette | | --- | --- | | 4 | 0.4658 | | 5 | 0.4691 | | 6 | 0.4126 | | 7 | 0.4163 | | 8 | 0.4255 | | 9 | 0.4036 | | 10 | 0.3646 | | 11 | 0.3560 | | 12 | 0.3967 |

k-NN coherence (k=15): 0.997 [/result]

[babysql:polars] SELECT k, silhouette, calinski_harabasz, davies_bouldin FROM read_csv('/home/sidmishra/ontologer/subprojects/review_sequencing_sentiment/cluster_output/review_k_selection.csv') ORDER BY k [/babysql]

[babysql:polars] SELECT kmeans_cluster AS cluster, COUNT() AS n, ROUND(AVG(rating), 2) AS avg_rating, ROUND(100.0 * SUM(CASE WHEN m_sentiment = 'positive' THEN 1 ELSE 0 END) / COUNT(), 1) AS pos_pct FROM '/home/sidmishra/ontologer/subprojects/review_sequencing_sentiment/cluster_output/review_clusters.parquet' GROUP BY kmeans_cluster ORDER BY cluster [/babysql]

[ml:sklearn-kmeans] library: scikit-learn script: cluster_reviews.py :: run_clustering unit: review (one row per review_id) features: 15 topic flags + sentiment one-hot + rating/m_score/verbosity preprocess: StandardScaler model: KMeans(n_init=10, random_state=42) selection: k in 4..12, maximize silhouette validation: knn_coherence(k=15) artifact: cluster_output/review_clusters.parquet [/ml]

How we found this (math): KMeans on z-scored features; silhouette s=(b−a)/max(a,b).

Pass B — Topic archetypes (auto-generated blocks)

[result:topic_cluster_k_selection] | k | Silhouette | | --- | --- | | 4 | 0.4139 | | 5 | 0.4059 | | 6 | 0.3190 | | 7 | 0.3056 | | 8 | 0.3014 | | 9 | 0.2944 | | 10 | 0.2880 | | 11 | 0.3179 | | 12 | 0.3206 |

Best k=4; k-NN coherence 0.973 [/result]

[ml:sklearn-kmeans] library: scikit-learn script: cluster_reviews.py :: run_clustering(include_sentiment=False) features: topics + behavior only (no sentiment one-hot) [/ml]