Mismatches between samples and their respective channel or target commonly arise in several real-world applications. For instance, whole-brain calcium imaging of freely moving organisms, multiple-target tracking or multi-person contactless vital sign monitoring may be severely affected by mismatched sample-channel assignments. To address this issue systematically, we frame it as a signal reconstruction problem where correspondences between samples and channels are lost. Assuming a sensing matrix for the signals, we show the problem's equivalence to a highly structured unlabeled sensing problem and establish conditions for unique recovery. This is crucial since existing unlabeled sensing theory is inapplicable and results for reconstructing shuffled multi-channel signals do not yet exist. Our results extend to continuous-time sparse signals, and we derive conditions for reconstructing shuffled sparse signals. For the two-channel case, we provide a first reconstruction method, which combines sparse signal recovery with robust linear regression, outperforming existing unlabeled sensing methods in numerical experiments. Additionally, we showcase its effectiveness in a real-world application involving calcium imaging traces. Our theory marks a significant initial step in addressing this challenging signal reconstruction problem, with potential extensions to diverse signal representations encountered in real-world problems with imprecise measurement or channel assignment.