Self-supervised methods for Martian terrain analysis struggle with multispectral data due to pixel-level reconstruction biases and high computational cost. We introduce Spatio-Spectral JEPA (SS-JEPA), a joint embedding predictive architecture that applies independent masking across spectral channels to learn cross-band correlations. SS-JEPA achieves 0.865 mIoU on landslide segmentation using only 20M parameters, rivaling state-of-the-art ensembles with 5.6x more parameters.