Visualizing 3D tensor batch representation, temporal patch projection, multi-crop view extractions, and joint self-supervised distillation for ECG representation learning.
xECG (Lunelli et al., 2025)arXiv:2509.10151bench-xecg Official Architecture
1
Input Tensor Representation in 3D Space
Mini-batch: \(B=128\) ECGs • Visualized for \(B=3\)
Standard 12-lead electrocardiography records simultaneous electrical potential differences across the myocardium from 12 distinct anatomical viewpoints. In a typical training mini-batch on GPU, the input data forms a 3-dimensional tensor \(\mathbf{X} \in \mathbb{R}^{B \times 12 \times T}\) (e.g. \([128, 12, 5000]\) samples for 10-second continuous recordings at \(500\,\mathrm{Hz}\); illustrated below for \(B=3\) patient sheets):
Pipeline Flow 1 → 2Transition to Signal Domain Augmentations: The network does not train on the unedited 10-second recording directly. In the DataLoader (PretrainDataset), each continuous 12-lead signal \([12, T]\) is augmented with physical sensor variations and sampled into multi-scale temporal crops (Global & Local views).
In the xECG self-supervised distillation framework, multi-crop view sampling and physical waveform augmentations operate directly on the continuous 12-lead raw signal \([12, T]\) in the data loader before any temporal patch tokenization takes place:
Adds zero-mean high-frequency jitter simulating skeletal muscle tremor and electrical noise during telemetry.
↕ Amplitude Scaling
Multiplies leads by scale factor \(\alpha \in [0.8, 1.2]\) simulating chest geometry and skin conductance differences.
Recurrent Alignment & Flatline Zero-Padding in Continuous DomainContinuous Raw Signals · NO Patches Yet
Original xECG Paper & Codebase Specification: Each extracted view immediately starts at index 0 (\(t=0\)) as an autonomous continuous waveform. Shape alignment padding is applied strictly on the right / end only as flatline zero voltage (\(0.0\,\mathrm{mV}\)). There is no front/left padding to preserve the source recording's offset.
✓ Adopted in xECG: Start at \(t=0\) + Right-Pad End with Zero Voltage (\(0\,\mathrm{mV}\))Paper & Official Code
t = 0.0s (Real ECG signal begins immediately)t = 4.0s (Crop End)t = 8.0s (0.0 mV Zero-Padding)
The continuous crop is treated as a fresh autonomous sub-sequence starting at \(t=0\). The xLSTM recurrent memory states \((h_0, c_0)\) initialize directly on real cardiac waveform, learning translation-invariant representations. Implemented via RandomCrop + _pad_time_end in bench-xecg.
✗ NOT Used: Pad Beginning & End (Absolute Position Alignment)Breaks Recurrence
Dummy 0.0 mV Front Pad\(t_k = t_{\mathrm{start}}\) (Offset)Dummy 0.0 mV End Pad
Padding the beginning to preserve the crop's absolute position offset forces forward recurrent xLSTM cells to step through 1000+ dummy zero samples before encountering cardiac signal, corrupting memory states. In contrast, adopted crops start immediately on signal.
Why do Global Views fill 100% and Local Crops fill 50% here? (Batch Slot Normalization)
In PyTorch, all 6 views must share the same tensor length \(T_{\max} = 4000\) samples (\(8.0\,\mathrm{s}\)) to assemble a dense mini-batch \([B, 6, 12, 4000]\):
Global Views: Cropped to \(\mathbf{80\%}\) of 10s recording \(= 4000\) samples.
→ Fills \(\mathbf{100\%}\) of 4000 batch slot (\(\mathbf{0}\) padding).
Local Crops: Cropped to \(\mathbf{40\%}\) of 10s recording \(= 2000\) samples.
→ Fills \(\mathbf{50\%}\) of 4000 batch slot (\(2000\) real \(+ \mathbf{2000}\) zero-pad).
Global View 1
[12 leads × 4000 samples]80% crop (4000 real) • 0 pad
4000 Real • 0 Pad
Global View 2
[12 leads × 4000 samples]80% crop (4000 real) • 0 pad
4000 Real • 0 Pad
Local Crop 1
[12 leads × 4000 samples]40% crop + 2000 pad
0.0 mV Flatline Pad (+2000 smp)
2000 Real • 2000 Pad
Local Crop 2
[12 leads × 4000 samples]40% crop + 2000 pad
0.0 mV Flatline Pad (+2000 smp)
2000 Real • 2000 Pad
Local Crop 3
[12 leads × 4000 samples]40% crop + 2000 pad
0.0 mV Flatline Pad (+2000 smp)
2000 Real • 2000 Pad
Local Crop 4
[12 leads × 4000 samples]40% crop + 2000 pad
0.0 mV Flatline Pad (+2000 smp)
2000 Real • 2000 Pad
⚡ Important Architectural Boundary: All signals shown above are raw continuous multi-channel waveforms \([12, T]\). Slicing into discrete temporal patch tokens \((P_1, P_2, \dots, P_N)\) occurs next in Step 3.
Pipeline Flow 2 → 3Transition to Temporal Tokenization: We now have autonomous 12-lead view tensors \([12, T_{\mathrm{crop}}]\). In Step 3, these multi-channel waveforms are sliced along the time dimension across all 12 leads into discrete temporal patch tokens \((P_1, P_2, \dots, P_N)\).
3
3D Temporal Patching Across All 12 Leads
Multichannel Waveform Tokenization
Unlike 2D images in Vision Transformers (which patch across height and width), a 12-lead ECG is an electrophysiological recording where the 12 leads represent simultaneous spatial projections of a single heart dipole vector \(\vec{D}(t)\). Slicing across leads would destroy spatial anatomical coherence. Therefore, xECG uses 1D temporal patching spanning all 12 leads:
Temporal Slicing of Extracted 12-Lead View (Window Length \(P_s = 25\) samples \(= 50\,\mathrm{ms}\)):Slicing Planes: \(\Delta t = 25\) samples at 500 Hz
Lead I
Lead II
... (12 L)
Leads III, aVR, aVL, aVF, V1, V2, V3, V4, V5 (Simultaneously Sliced at every Δt)
2D Stack: Leads are stacked vertically as rows in the patch matrix. Unrolling them horizontally into a 1D vector happens next in Step 4.
Why Multi-Lead Temporal Slicing?
During any \(50\,\mathrm{ms}\) interval, myocardial electrical activation generates an instantaneous 3D vector dipole in space. Slicing across all 12 leads captures simultaneous frontal and horizontal plane projections, preserving the complete cardiac dipole geometry for that window.
In this step, patch \(\mathbf{P}_i\) is still a 2D physical matrix of shape \([12 \text{ leads} \times 25 \text{ samples}]\). Each lead provides a horizontal row of 25 consecutive voltages.
→ In Step 4, this matrix is unrolled row-by-row into a single \(1 \times 300\) row vector to be mapped into the latent space via the linear projection matrix.
Pipeline Flow 3 → 4Transition to Linear Patch Projection: Each patch \(\mathbf{P}_i \in \mathbb{R}^{12 \times 25}\) is currently a 2D matrix of 12 lead rows. In Step 4, it is unrolled row-by-row into a 300-dimensional row vector \(\mathbf{x}_i \in \mathbb{R}^{1 \times 300}\) and projected through learned linear matrix \(\mathbf{W}_{\mathrm{proj}} \in \mathbb{R}^{300 \times 1024}\) to produce the token sequence \((\mathbf{z}_1, \dots, \mathbf{z}_N)\).
4
Patch Flattening & Linear Projection (Patch Embedding + Selective Masking)
Each multi-lead patch \(\mathbf{P}_i \in \mathbb{R}^{12 \times 25}\) produced in Step 3 represents a \(50\,\mathrm{ms}\) instantaneous temporal slice across all 12 anatomical leads (\(12 \times 25 = 300\) scalar voltage readings). To prepare these physical voltage matrices for the neural encoder, xECG performs three sequential mathematical operations on every patch, followed by selective masking for student global views:
Spatial-Temporal Unrolling (Flattening): Concatenates the 12 lead rows into a contiguous 300-dimensional voltage vector \(\mathbf{x}_i \in \mathbb{R}^{300}\).
Learnable Linear Projection: Multiplies by weight matrix \(\mathbf{W}_{\mathrm{proj}} \in \mathbb{R}^{300 \times 1024} + \mathbf{b}_{\mathrm{proj}}\) to project raw microvolts into the latent representation space \(D=1024\).
1D Positional Embedding Summation: Injects temporal sequence order by adding a learned positional vector \(\mathbf{E}_i \in \mathbb{R}^{1024}\), yielding embedded token \(\mathbf{z}_i = \mathbf{h}_i + \mathbf{E}_i\).
Stacks as row \(i\) in sequence matrix \(\mathbf{Z} \in \mathbb{R}^{160 \times 1024}\)
4.2 THE OUTPUT
Full Batch of Patch Token Embeddings
Every temporal patch across all 12 leads, all 6 views, and all 128 batch recordings is now a dense continuous 1024-dimensional latent embedding vector \(\mathbf{z} \in \mathbb{R}^{1024}\).
On the GPU accelerator, projection is executed as a single batched tensor operation across all \(B=128\) patient recordings and all \(V=6\) views simultaneously. In PyTorch, this corresponds to:
tokens = self.proj(patches.flatten(-2, -1)) + self.pos_embed[:, :seq_len, :]
1. Multi-Crop Views[128, 6, 160, 12, 25]All views right-padded to equal maximum length \(N=160\).
2. Flatten Leads & Samples[768, 160, 300]Collapses \(12 \times 25 \to 300\) contiguous raw voltage scalars per patch.
3. Linear GEMM Kernel[768, 160, 1024]Matrix multiply with \(\mathbf{W}_{\mathrm{proj}} \in \mathbb{R}^{300 \times 1024}\) in a single hardware kernel.
4. Broadcast Positional \(\mathbf{E}\)[768, 160, 1024]Learnable 1D temporal matrix broadcasted across all 768 sequences.
Selective Masking Strategy: Asymmetric Information Flow
Student Global Masked • Teacher & Local Unmasked
SimDINOv2 unifies masked autoencoding (MIM) with multi-crop self-distillation. Crucially, masking is applied asymmetrically: only Student Global views have patches replaced with the shared learnable [MASK] token vector \(\mathbf{e}_{\mathrm{mask}} \in \mathbb{R}^{1024}\). Local crops and all Teacher views remain 100% unmasked. All 8 individual views are stacked vertically below to compare token sequences, stochastic mask distributions, and trailing padding directly:
Student Local Crops (\(v_1, v_2, v_3, v_4\)) — 4.0 s Context • 80 Real + 80 Right-Pad
0% Signal Masking • Flatline Padding Masked in Attention
v₁
Student Local 1t ∈ [0.8, 4.8] s • QRS Complex
z₁z₂z₃z₄z₅z₆…z₇₉z₈₀padpadpad…pad
80 Real + 80 Pad • 0% Masked
v₂
Student Local 2t ∈ [2.2, 6.2] s • ST Segment
z₁z₂z₃z₄z₅z₆…z₇₉z₈₀padpadpad…pad
80 Real + 80 Pad • 0% Masked
v₃
Student Local 3t ∈ [3.6, 7.6] s • T Wave Peak
z₁z₂z₃z₄z₅z₆…z₇₉z₈₀padpadpad…pad
80 Real + 80 Pad • 0% Masked
v₄
Student Local 4t ∈ [5.1, 9.1] s • Late Cycle
z₁z₂z₃z₄z₅z₆…z₇₉z₈₀padpadpad…pad
80 Real + 80 Pad • 0% Masked
Teacher Global Views (\(u_1, u_2\)) — 8.0 s Context • 160 Tokens
100% Pristine Ground-Truth Targets • Zero Gradient (EMA)
u₁
Teacher Global 1t ∈ [0.0, 8.0] s • Pristine Target
z₁z₂z₃z₄z₅z₆z₇z₈z₉z₁₀z₁₁z₁₂…z₁₅₉z₁₆₀
100% Pristine • Target for u₁
u₂
Teacher Global 2t ∈ [2.0, 10.0] s • Pristine Target
z₁z₂z₃z₄z₅z₆z₇z₈z₉z₁₀z₁₁z₁₂…z₁₅₉z₁₆₀
100% Pristine • Target for u₂
Student Global Strategy (\(u_1, u_2\))
Masking Ratio: ~30–40% of patch tokens randomly substituted with learnable vector \(\mathbf{e}_{\mathrm{mask}} + \mathbf{E}_i\).
Supervisory Role: Drives the patch reconstruction objective \(\mathcal{L}_{\mathrm{patch}}\).
Physiological Rationale: Forces the student to reconstruct occluded QRS complexes or ST elevations from surrounding rhythm context.
Student Local Crops Strategy (\(v_1 \dots v_4\))
Masking Ratio:0% masked. All 80 real signal tokens remain pristine.
Trailing Padding: Tokens 81–160 are right-padded flatline zeros, completely masked from attention pooling.
Why No Masking? Local crops span only \(4.0\,\mathrm{s}\). Additional masking would destroy critical cardiac morphology needed for part-to-whole alignment.
Teacher Global Strategy (\(u_1, u_2\))
Masking Ratio:0% masked. Complete 160 active tokens fed to EMA teacher.
Supervisory Target: Unmasked teacher representations provide the ground-truth targets for \(\mathcal{L}_{\mathrm{patch}}\) and \(\mathcal{L}_{\mathrm{comp}}\).
Anti-Corruption Guarantee: Masking the teacher would provide corrupted, noisy regression targets and destabilize distillation.
Reconstruct occluded wave complexes while aligning global semantics.
Student Local (\(v_1 \dots v_4\))
\(4.0\,\mathrm{s}\) (\(40\%\))
\(80\) real + \(80\) pad
0% masked (unmasked)
\(\mathcal{L}_{\mathrm{comp}}\) (student local → teacher global)
Enforce part-to-whole view invariance across temporal shifts.
Teacher Global (\(u_1, u_2\))
\(8.0\,\mathrm{s}\) (\(80\%\))
\(160\) active tokens
0% masked (100% pristine)
EMA Regression Target (no grad)
Provides stable, noise-free semantic targets for the student network.
Pipeline Flow 4 → 5Sequence Ready for xLSTM Backbone: With raw voltages converted to normalized \(D=1024\) latent embeddings and positional coordinates injected, the sequence \((\mathbf{z}_1, \dots, \mathbf{z}_N) \in \mathbb{R}^{N \times 1024}\) is passed directly into the 9-block Bidirectional xLSTM encoder. Unlike ViTs, no artificial [CLS] token is prepended.
5
Bidirectional xLSTM Encoder Backbone
Batch: Parallel Across All \(B \times 6 = 768\) View Sequences
In contrast to Vision Transformers (ViT), which prepend an artificial [CLS] token at index 0, xECG inputs pure temporal patch tokens \((z_1, \dots, z_N) \in \mathbb{R}^{N \times 1024}\). Prepending dummy tokens into recurrent architectures like xLSTM alters initial cell states \((h_0, c_0)\) and introduces artificial initialization bias. Illustrated below for a single sequence; on the GPU, all \(B \times 6 = 768\) view sequences in the mini-batch are processed concurrently through 9 bidirectional xLSTM blocks alternating between sLSTM (scalar memory with exponential gating) and mLSTM (matrix memory with associative storage and retrieval):
sLSTM 1⇄ BiDir
→
sLSTM 2⇄ BiDir
→
mLSTM 3⇄ BiDir
→
mLSTM 4⇄ BiDir
→
sLSTM 5⇄ BiDir
→
sLSTM 6⇄ BiDir
→
mLSTM 7⇄ BiDir
→
mLSTM 8⇄ BiDir
→
sLSTM 9⇄ BiDir
sLSTM (Scalar Memory Blocks: 1, 2, 5, 6, 9)
Exponential gating with memory mixing stabilizes long-range tracking of rhythmic baseline features, heart rate variability, and recurrent cardiac wave cycles.
Backbone Output: The 9th block outputs contextualized sequence representations \(\mathbf{s} = (s_1, s_2, \dots, s_N) \in \mathbb{R}^{N \times 1024}\). Each token \(s_i\) encapsulates both forward past context and backward future context of the 12-lead ECG.
Sequence Aggregation to Unit Hypersphere \(\mathbb{S}^{D-1}\)
Instead of using an ad-hoc token (like [CLS]) or simple mean-pooling (which dilutes localized cardiac events like ventricular ectopy across quiet intervals), xECG condenses the contextualized patch sequence \((s_1, \dots, s_N) \in \mathbb{R}^{N \times 1024}\) through a 2-stage representation head (defined in bench_xecg/models/pooling.py):
6.1Attention Pooling Block (AttentionPooling)
Query Cross-Attention + Residual MLP
Learned query parameter probe \(\mathbf{q} \in \mathbb{R}^{1 \times 1024}\) performs multi-head cross-attention across all patch tokens \((s_1 \dots s_N)\) with zero-weight padding masking.
Physiological Grounding: The learned probe automatically focuses attention mass on the high-voltage QRS complex (\(\approx 54\%\)) and T-wave repolarization (\(\approx 21\%\)), while the attention padding mask strictly forces \(\alpha_{\mathrm{pad}} = 0.00\) on all trailing zero-pad tokens.
6.2\(\ell_2\) Hypersphere Normalization
Unit Hypersphere \(\mathbb{S}^{D-1}\)
Projects the pooled vector \(\mathbf{z}_{\mathrm{pool}}\) onto the unit hypersphere \(\mathbb{S}^{D-1}\), bounding vector norm to \(\|\hat{\mathbf{z}}\|_2 = 1.0\).
Why \(\ell_2\) Normalization is Mandatory: On the unit hypersphere, Euclidean distance exactly equals Cosine distance (\(1 - \cos\)). This eliminates representation magnitude drift, stabilizes the batch covariance matrix \(\operatorname{Cov}(\hat{\mathbf{z}})\), and allows SimDINOv2 to compute the coding rate \(R_\varepsilon\) directly.
⚡ Vectorized Mini-Batch Scope: Computed simultaneously across all \(B \times 6 = 768\) views on GPU \(\implies \hat{\mathbf{z}} \in \mathbb{R}^{768 \times 1024}\).
Pipeline Flow 6 → 7Normalized Representations Ready for Dual Distillation: With each view encoded into a unit-normalized vector \(\hat{\mathbf{z}} \in \mathbb{S}^{1023}\) and constituent token sequence \(\mathbf{s} \in \mathbb{R}^{N \times 1024}\), the pipeline bifurcates into an asymmetric Student-Teacher architecture. The Student learns via backpropagation gradients, while the Teacher maintains a noise-free historical target via Exponential Moving Average (EMA).
7
Teacher-Student Dual Network Dynamics & Exponential Moving Average (EMA)
Self-supervised representation learning without negative examples faces a catastrophic collapse hazard: both networks can trivially output a constant zero or static bias vector. SimDINOv2 eliminates this through structural network asymmetry and momentum parameter updates. The Student network \(\mathbf{f}_{\theta_s}\) processes all views (including masked and cropped variants) with active gradient tracking, while the Teacher network \(\mathbf{f}_{\theta_t}\) evaluates only pristine global signals under torch.no_grad() and updates its weights strictly through an Exponential Moving Average (EMA):
SStudent Network \(\mathbf{f}_{\theta_s}\)
Backprop Gradients Enabled
Input Scope: All 6 views per patient • \(2\) Global Views (30–40% masked) + \(4\) Local Crops (0% masked). Total mini-batch throughput on GPU: \(B \times 6 = 768\) sequences.
The EMA Momentum Wire & Polyak-Ruppert Averaging Effect
Cosine Momentum Schedule: 0.99 → 1.00
The teacher never computes gradients. Instead, after each backward step, the Student's updated weights are exponentially blended into the Teacher:
\[\lambda_t = 1 - (1 - \lambda_0) \cdot \frac{1 + \cos\left(\frac{\pi t}{T}\right)}{2}, \quad \lambda_0 = 0.99 \to \lambda_T = 1.00\]
Polyak-Ruppert Temporal Ensemble: The teacher behaves as an ensemble over thousands of preceding student parameter states: \(\theta_t \approx \sum_{k=0}^t w_k \theta_{s,k}\).
Trivial Collapse Prevention: Because \(\nabla_{\theta_t} \mathcal{L} = 0\), the optimization cannot solve the loss by simultaneously collapsing both networks into a trivial state \(\theta_s = \theta_t = \mathbf{0}\).
One-way momentum wire • No backpropagation into Teacher
Intermediate Checkpoint: Student vs. Teacher Mini-Batch Allocation (\(B=128\))
Network Branch
Input Crops per ECG
Total Sequences (\(B=128\))
Selective Masking
Forward Mode
Memory (Activations)
Parameter Update
Student \(\mathbf{f}_{\theta_s}\)
\(2\) Globals + \(4\) Locals (\(=6\))
\(128 \times 6 = \mathbf{768}\) sequences
\(30\text{--}40\%\) masked (Globals)
Train (Autograd graph built)
\(\approx 3.8\,\mathrm{GB}\) (gradients stored)
AdamW with Layerwise LR Decay
Teacher \(\mathbf{f}_{\theta_t}\)
\(2\) Globals only (\(=2\))
\(128 \times 2 = \mathbf{256}\) sequences
\(0\%\) masked (100% clean)
Eval (torch.no_grad())
\(\approx 0.3\,\mathrm{GB}\) (no graph stored)
Exponential Moving Average (EMA)
Pipeline Flow 7 → 8Representations Assembled for Objective Evaluation: With Student latents \(\hat{\mathbf{z}}_s \in \mathbb{R}^{768 \times 1024}\) and Teacher targets \(\hat{\mathbf{z}}_t \in \mathbb{R}^{256 \times 1024}\) residing in GPU VRAM on the unit hypersphere \(\mathbb{S}^{1023}\), the system executes the core breakthrough of SimDINOv2: closed-form Maximal Coding Rate Reduction (MCR) without heuristic centering or sharpening.
8
Multi-Component SimDINOv2 Loss Formulation: The MCR Objective
Pushes batch representations apart: Maximizes the volume (coding rate) of the mini-batch covariance ellipsoid across all \(1024\) dimensions to prevent collapse.
Local waveform reconstruction: iBOT-style token distillation forcing the student to reconstruct masked temporal intervals from contextual leads.
8.1 Compression Loss (\(\mathcal{L}_{\mathrm{comp}}\)): Cross-View Pairwise Alignment
Scaled by \(\mathrm{coeff} = 1.0\)
For each ECG recording in the batch, we compute the cosine similarity between every Student view (\(S_{g1}, S_{g2}, S_{l1}, S_{l2}, S_{l3}, S_{l4}\)) and each Teacher global view (\(T_{g1}, T_{g2}\)). Crucially, to prevent trivial shortcut learning, matching a view against its exact identical twin (e.g. \(S_{g1} \leftrightarrow T_{g1}\)) is strictly excluded by zeroing out the diagonal in PyTorch (sim.view(-1, ...)[::(len+1), :].fill_(0)):
Cardiovascular Grounding: Forces local crops containing isolated arrhythmias (e.g. premature ventricular complexes in Lead V1) to project near the global representation of the entire 12-lead recording, learning robust part-to-whole diagnostic invariance.
If we only minimize compression loss, the entire network will collapse to a single point \(\hat{\mathbf{z}} \equiv \mathbf{c}\) on the hypersphere (where \(\cos(\mathbf{c}, \mathbf{c}) = 1.0 \implies \mathcal{L}_{\mathrm{comp}} = 0.0\)). SimDINOv2 prevents this by directly maximizing the Coding Rate \(R_\varepsilon(\mathbf{Z})\): the volume of the feature space spanned by the mini-batch on the unit hypersphere:
⚠️ Collapsed Subspace (Low Rank)Rank \(\ll 1024\)
Zero Volume • Determinant \(\approx 1.0\) • \(R_\varepsilon \to 0\)
Representations cluster along a 1D line. The covariance matrix has only one non-zero eigenvalue (\(\lambda_1 \gg 0, \lambda_{2\dots 1024} = 0\)). High expansion penalty!
Max Volume • Uniform Eigenvalues • \(R_\varepsilon\) Maximized
Representations spread uniformly across all 1024 orthogonal axes of \(\mathbb{S}^{1023}\). Maximal entropy and maximal information capacity. Minimal expansion penalty!
1. Expansion Feature Mixture (expa_type = 1):
\[\mathbf{Z}_{\mathrm{expa}} = \frac{\hat{\mathbf{Z}}_{s,\mathrm{globals}} + \hat{\mathbf{Z}}_{t,\mathrm{globals}}}{2} \in \mathbb{R}^{m \times D}, \quad m = B \times 2 = 256, \quad D = 1024\]
2. Feature Covariance & Fast Cholesky Log-Determinant:
Numerical Stability Guarantee: Computing \(\log \det\) via Cholesky decomposition \(\mathbf{L}\) runs entirely in GPU Tensor Cores in \(\approx 0.8\,\mathrm{ms}\) and is completely immune to float underflow. Negating the coding rate turns volume maximization into standard gradient descent minimization.
Global attention pooling captures macro patient-level rhythm, but subtle localized morphological features (e.g. ST-elevation in Lead V2, bifid P-waves, pacemaker spikes) require dense temporal supervision. In the 2 Student Global views, 30–40% of patch tokens were replaced by learnable [MASK] embeddings. The Teacher processed the exact same temporal locations in pristine form. \(\mathcal{L}_{\mathrm{patch}}\) forces the Student to reconstruct these occluded tokens:
Padding Safety: The vectorized padding mask \(\mathbf{M}_{\mathrm{pad}}\) strictly excludes trailing flatline padding tokens, guaranteeing that gradients are generated only from authentic cardiac depolarization intervals.
✦ The Complete SimDINOv2 Master Objective
Official bench-xecg Objective
The total loss combines cross-view compression, anti-collapse coding rate expansion, and masked patch distillation into a single scalar minimized by the AdamW optimizer:
Compression Term (\(\alpha = 1.0\))Pulls student global and local views toward teacher global targets on \(\mathbb{S}^{1023}\).
Expansion Term (\(\gamma \cdot R_\varepsilon\))Expands the volume of the feature covariance ellipsoid across all 1024 dimensions.
Dense Patch Term (\(\mathcal{L}_{\mathrm{patch}}\))Reconstructs occluded cardiac wave complexes at masked temporal intervals.
Pipeline Flow 8 → 9From Loss Evaluation to System Synthesis: With all mathematical operations defined from raw voltage waveforms to scalar loss backpropagation, we now aggregate the complete system architecture into an end-to-end hardware ledger and comparative paradigm matrix.
The diagram and ledger below trace the complete life-cycle of a mini-batch of \(B=128\) 12-lead ECGs as it moves through the entire GPU training pipeline—from raw millivolts on disk to backpropagation gradients and EMA momentum parameter updates: