Evidence Supporting Neural Network Convergence

Evidence supporting the convergence/alignment of neural networks’ representations (as argued and surveyed in The Platonic Representation Hypothesis ) includes: Alignment across domains and objectives over time: The paper states that, over time and across multiple domains , “the ways by which different neural networks represent data are becoming more aligned.”[‌:cite[1]{ln=2}‌] It frames this as being trained models “progressively” toward a shared representation.[‌:cite[2]{ln=4}‌] Cross modal convergence as models get larger (vision ↔ language): The paper reports evidence that “as vision models and language models get larger, they measure distance between datapoints in a more and more alike way.”[‌:cite[1]{ln=3}‌] It also describes its hypothesis that model embeddings derived from a common underlying reality become “aligned” as scale increases.[‌:cite[3]{ln=5}‌], [‌:cite[3]{ln=6}‌] Model weight basin convergence for same architectures: For the question of whether convergence extends to weights, the paper notes “ample evidence” that models “with the same architecture” often “converge to the same basin of weights” (citing Nagarajan & Kolter; Garipov et al.; Lubana et al.).[‌:cite[4]{ln=3}‌] Kernel/activation alignment increases with scale and performance (observational evidence): It cites observations that “alignment increases with model scale and dataset size,” including Kornblith et al. and Roeder et al., plus a CIFAR 10 result that “larger models exhibit greater alignment… compared to smaller ones.”[‌:cite[5]{ln=3}‌], [‌:cite[5]{ln=5}‌] It also adds a theoretical claim that “models with similar outputs… also have similar internal activations.”[‌:cite[5]{ln=6}‌] Empirical measurement: stronger models form tighter representation clusters: The paper evaluates transfer performance of 78 vision models and reports that high transfer models form “a tightly clustered set of representations,” while weak models are more variable.[‌:cite[6]{ln=4}‌] It visualizes this with UMAP, concluding that “models that are competent all represent data in a similar way.”[‌:cite[6]{ln=6}‌] Representational similarity can be tested via “model stitching”: It describes “model stitching” as evidence of compatible representations at intermediate layers: if a stitched model with an affine “stitching layer” achieves good performance, it indicates compatibility “up to the transform h.”[‌:cite[7]{ln=3}‌] It summarizes prior findings that such alignment exists across datasets (e.g., ImageNet vs Places 365) while maintaining good performance.[‌:cite[8]{ln=1}‌] Zero shot stitching / cross model usefulness: The paper notes work showing “zero shot model stitching without learning a stitching layer,” where models trained on different text modalities still yield similar embeddings, with kernel defined by learned representations acting as a “bridge” between models.[‌:cite[9]{ln=2}‌], [‌:cite[9]{ln=4}‌] Cross modal semantic alignment beyond stitching: It also surveys other cross modal evidence, e.g., auditory models “roughly aligned with LLMs up to a linear transformation,”[‌:cite[10]{ln=3}‌] and work showing “visual concept” word embeddings can be “isometrically mapped” to corresponding image embeddings.[‌:cite[11]{ln=4}‌] Direct definition of the alignment/convergence notion via kernels: The paper defines representational alignment in terms of similarity between induced “kernels,” i.e., similarity structures in representation spaces.[‌:cite[12]{ln=3}‌] It points to the use of kernel alignment metrics (like CKA, SVCCA, and nearest neighbor metrics) as tools to quantify representational convergence.[‌:cite[13]{ln=1}‌], [‌:cite[13]{ln=2}‌] Missing Content (scope limitation): Your question asks generally “what evidence supports convergence of neural networks?” The repository provided contains only this single paper.[files list 1] Therefore, the evidence list above is restricted to the types of convergence evidence this paper surveys and demonstrates; it may not include other literature on optimization convergence (loss/gradient convergence) or formal theoretical guarantees beyond what is referenced here.