Experiments Validating Linear Centroids Hypothesis
The paper validates the effectiveness of the Linear Centroids Hypothesis (LCH) through a set of experiments showing that (i) centroids align into linear directions when inputs share a feature, and (ii) using centroids...
The paper validates the effectiveness of the Linear Centroids Hypothesis (LCH) through a set of experiments showing that (i) centroids align into linear directions when inputs share a feature, and (ii) using centroids (instead of latent activations) improves multiple interpretability pipelines (feature dictionaries, probes, circuit localization, saliency). [:cite[1]{ln=2}], [:cite[2]{ln=2}], [:cite[3]{ln=5}]. Experiments that provide evidence for / validate the LCH 1) 2D toy task: inside vs. outside a star shaped polygon (Figure 1) A deep network is trained to classify whether 2D points lie inside or outside a star shaped polygon. [:cite[4]{ln=1}]. The paper then samples points from the polygon’s edges and observes that the centroids for local experts on these edges “separate into linear directions,” presented as evidence for the LCH and confirming their theorem based characterization. [:cite[5]{ln=4}], [:cite[5]{ln=6}], [:cite[2]{ln=2}], [:cite[2]{ln=3}]. 2) Pretrained vision model: ResNet50 + DTD textures; PCA separation (Figure 2) For a pretrained ResNet50, the authors study centroids from the fourth layer and run PCA on those centroids for inputs from two texture classes in the DTD dataset. [:cite[3]{ln=6}], [:cite[3]{ln=7}]. They report that centroids corresponding to distinct features separate into distinct linear directions under PCA, presented as “The LCH holds true for pre trained vision models on ImageNet.” [:cite[3]{ln=4}], [:cite[3]{ln=5}], [:cite[2]{ln=4}]. 3) Spurious feature test: FashionMNIST with color label correlation; linear probe (Figure 3 left) To compare LCH vs LRH susceptibility to spurious features, they train a CNN on FashionMNIST while artificially coloring images with varying correlation to the label (fully correlated → not spurious; random → spurious). [:cite[6]{ln=2}], [:cite[6]{ln=3}], [:cite[6]{ln=4}]. They then train a linear probe to predict color from either latent activations or centroids, measuring probe accuracy. [:cite[6]{ln=5}]. The key outcome: centroids represent the color feature as linear directions only insofar as the feature is not spurious, whereas latent activations remain linear even when the feature is spurious. [:cite[7]{ln=1}]. The figure caption summarizes this as: “Centroids are not susceptible to identifying spurious features, unlike latent activations.” [:cite[8]{ln=1}], [:cite[8]{ln=2}]. 4) Cross model dictionary alignment: DINOv2 vs DINOv3 sparse autoencoders (Figure 3 right) They train sparse autoencoders on either latent activations or centroids (from the last MLP block) for DINOv2 and DINOv3. [:cite[9]{ln=1}]. They compare the learned dictionaries by taking each DINOv2 feature and computing its maximum cosine similarity with the DINOv3 dictionary. [:cite[9]{ln=2}]. Because DINOv3 refines DINOv2’s features, they argue the dictionary should align highly; they report this is observed for centroid based dictionaries , while latent based dictionaries show many features with cosine similarity around 0 (suggesting poor cross model correlation). [:cite[9]{ln=3}], [:cite[9]{ln=4}], [:cite[9]{ln=5}], [:cite[9]{ln=7}]. 5) Downstream transfer/generalization of feature dictionaries: Imagenette (Figure 4) They compare sparse autoencoder feature dictionaries trained on either centroids or latents, then evaluate: Linear probe accuracy on Imagenette test set , where probes are trained on train set feature decompositions and tested on test set decompositions. [:cite[10]{ln=1}], [:cite[11]{ln=2}]. Feature firing frequency on unseen inputs (test set) through the sparse autoencoder. [:cite[10]{ln=2}], [:cite[11]{ln=3}]. They report centroid derived dictionaries show greater generalization to test set features and more features fire on test inputs , arguing centroid derived SAE features are less likely to be spurious. [:cite[12]{ln=1}], [:cite[12]{ln=2}], [:cite[12]{ln=3}]. 6) Qualitative semantic coherence of learned features: nearest neighbors in activation pattern space (Figure 5) They compare dictionaries by computing each input’s binary activation pattern over features, then retrieving nearest neighbors via Jaccard distance on those patterns. [:cite[13]{ln=1}], [:cite[13]{ln=2}]. They observe centroid derived dictionaries yield neighbors with clear semantic commonality (example: “white golf ball with black text”), unlike LRH derived dictionaries. [:cite[13]{ln=3}]. The figure caption states centroid trained SAE features are more semantically coherent than latent trained ones. [:cite[14]{ln=1}]. 7) Circuit discovery / neuron attribution in GPT2 Large (Figure 6), plus robustness checks (Figure 15) They define a centroid based neuron attribution score (Equation (2)) and apply it to GPT2 Large by sampling a neighborhood around an embedding and computing normalized attribution values for neurons in a specific MLP block layer. [:cite[15]{ln=2}], [:cite[15]{ln=3}], [:cite[15]{ln=5}]. They ...