A Dataset is Worth 1 MB: Efficient Dataset Distribution via Pseudo-Labels

Paper: 2602.23358 Authors: Elad Kimchi Shoshani, Leeyam Gabay, Yedid Hoshen Categories: cs.LG, cs.CV

Abstract

This paper introduces PLADA (Pseudo-Labels as Data), a novel approach to dataset distribution that radically reduces communication costs. Instead of transmitting raw pixel data or distilled datasets, PLADA assumes clients have access to a large, unlabeled reference dataset (like ImageNet) and transmits only class labels for selected images. The method incorporates an intelligent pruning mechanism to filter the reference dataset, retaining only semantically relevant images for the target task. Experimental validation across 10 datasets shows that PLADA can transfer complete task knowledge with payloads under 1 MB while maintaining competitive classification accuracy, representing a breakthrough in efficient dataset serving.

Key Contributions

• Zero-pixel transmission paradigm: Completely eliminates the need to transmit image data by leveraging pre-loaded reference datasets
• Semantic pruning mechanism: Intelligently filters reference datasets to select only the most relevant images for target tasks, balancing efficiency and accuracy
• Sub-megabyte payloads: Achieves dataset distribution with less than 1 MB of transmitted data across diverse classification tasks
• Scalability to high-resolution data: Overcomes limitations of traditional dataset distillation methods that struggle with large, high-resolution images

Methodology: Pseudo-Labels as Data

The PLADA framework operates on a simple but powerful premise: rather than transmitting pixels, transmit semantic information. The approach consists of three key components:

Reference Dataset Assumption: Clients are assumed to have access to a large, generic, unlabeled dataset $\mathcal{D}_{\text{ref}}$ (e.g., ImageNet-1K with 1.2M images or ImageNet-21K with 14M images). This one-time preloading cost is amortized across multiple task distributions.

Label Assignment: For a target task with classes $\mathcal{C}_{\text{target}}$, the server assigns pseudo-labels from $\mathcal{C}_{\text{target}}$ to images in $\mathcal{D}_{\text{ref}}$. The transmitted payload consists solely of tuples $(i, y)$ where $i$ is an image index and $y$ is the assigned class label.

Semantic Pruning: To address distribution mismatch between reference and target datasets, PLADA employs a pruning strategy that selects a subset $\mathcal{D}_{\text{pruned}} \subset \mathcal{D}_{\text{ref}}$ of the most semantically relevant images. This selection process uses feature-based similarity metrics to identify images that best represent the target task’s semantic space, simultaneously reducing payload size and improving training efficiency.

The pruning mechanism is critical: transmitting labels for all 1.2M ImageNet images would still require significant bandwidth, while random sampling would include many irrelevant images. PLADA’s intelligent selection ensures that only the most informative pseudo-labels are transmitted.

Experimental Results and Analysis

The authors evaluate PLADA across 10 diverse datasets spanning different domains, image resolutions, and classification complexities. Key findings include:

Compression Performance: PLADA consistently achieves payloads under 1 MB across all tested datasets. For context, transmitting even a single high-resolution image can exceed this size, yet PLADA encodes entire task-specific training signals within this budget.

Accuracy Retention: Despite the extreme compression, PLADA maintains competitive classification accuracy compared to training on full target datasets. The semantic pruning mechanism proves effective at identifying reference images that capture the essential visual patterns needed for target tasks.

Scalability: Unlike dataset distillation methods that struggle with high-resolution images due to computational and memory constraints, PLADA’s label-only approach scales effortlessly to any image resolution since pixels are never processed or transmitted.

Cross-domain Transfer: The method demonstrates robustness across domain shifts between reference and target datasets, suggesting that large generic datasets like ImageNet contain sufficient visual diversity to support diverse downstream tasks through appropriate label assignment.

Implications and Future Directions

PLADA represents a paradigm shift in how we think about dataset distribution. The implications extend beyond mere bandwidth savings:

Federated Learning: In federated settings where communication is the primary bottleneck, PLADA could enable more frequent model updates and support for resource-constrained edge devices.

Privacy Considerations: By transmitting only labels rather than raw images, PLADA inherently provides a degree of privacy protection, though the reference dataset itself must still be carefully curated.

Environmental Impact: Reducing data transmission from gigabytes to kilobytes translates to significant energy savings at scale, particularly for cloud-based ML services serving millions of clients.

Limitations: The approach requires clients to store large reference datasets locally, which may be prohibitive for extremely resource-constrained devices. Additionally, performance depends on the semantic coverage of the reference dataset relative to target tasks.

Future work could explore dynamic reference dataset selection, where different clients use different reference datasets based on their storage capabilities, or investigate methods to generate synthetic reference datasets optimized for label-based transfer.

Takeaways

1. PLADA achieves dataset distribution with sub-1MB payloads by transmitting only class labels for pre-loaded reference datasets, eliminating pixel transmission entirely
2. Semantic pruning is essential for selecting relevant reference images, balancing transmission efficiency with training effectiveness
3. The approach scales to high-resolution data where traditional dataset distillation methods fail, offering a practical solution for real-world deployment
4. Experimental validation across 10 datasets confirms that extreme compression (1000x+ reduction) is achievable while maintaining competitive accuracy
5. The method opens new possibilities for efficient federated learning, privacy-preserving dataset sharing, and environmentally sustainable ML infrastructure