ACM MM 2026

Direct Speech-to-Image Synthesis via
Cross-Modal Latent Transformers

Generate Images Directly from Human Speech

Adhithyaa Natarajan, Balamurugan Palaniappan

Department of Industrial Engineering and Operations Research
Indian Institute of Technology Bombay, Mumbai, India

Accepted at 35th ACM International Conference on Multimedia (ACM MM 2026)

Paper Code (Coming Soon) BibTeX (Coming Soon)
SPIMSyn Teaser

Direct Speech-to-Image Synthesis framework.

Abstract

Generating images directly from spoken language without intermediate text transcription remains a fundamental challenge in cross-modal perception. Existing approaches cascade Automatic Speech Recognition (ASR) with Text-to-Image (T2I) models, introducing compounding transcription errors and latency that may degrade image quality. We eliminate this bottleneck by proposing SPIMSyn and SPIMSyn-D, two SPeech to IMage Synthesis frameworks that adapt Stable Diffusion 3.5 for direct Speech-to-Image (S2I) generation. A novel 3-Branch Transformer Decoder predicts conditioning embeddings corresponding to the three text encoders of Stable Diffusion 3.5: CLIP-L, CLIP-G, and T5-XXL.

To obtain speech representations, SPIMSyn employs a pretrained Whisper Encoder, while SPIMSyn-D uses training-free discrete mel-filterbank tokenization through dMel. Extensive experiments demonstrate that both SPIMSyn and SPIMSyn-D eliminate explicit ASR decoding, thereby reducing cascading errors, improving image generation fidelity, and achieving lower end-to-end latency compared with conventional cascaded Speech-to-Image pipelines.

Architecture

Overview of the proposed SPIMSyn and SPIMSyn-D frameworks for direct speech-to-image synthesis.

SPIMSyn Architecture
A) Proposed 3-Branch Transformer Decoder Architecture. B) Frozen Stable Diffusion 3.5 (Medium). C) Decoder Block Components consisting of self-attention, followed by cross-attention and an MLP with residual connections. D) Loss components used in the proposed 3-Branch Transformer Decoder. E) Computation of Centroids. Text captions are encoded into embeddings, clustered using K-means, and the resulting cluster centroids are stored and used as the ground truth for cross-entropy and contrastive losses.

Experimental Results

We evaluate our proposed SPIMSyn and SPIMSyn-D on synthesized and real-world speech datasets to demonstrate direct speech-to-image generation without intermediate ASR transcription.

Synthesized LAION-5B

We first evaluate our framework on a synthesized speech version of the LAION-5B dataset. Compared with cascaded Speech-to-Image baselines, SPIMSyn and SPIMSyn-D generate images that more faithfully preserve the semantic content of the spoken input while avoiding transcription errors introduced by Automatic Speech Recognition (ASR).

Synthesized LAION-5B

Spoken COCO

We further evaluate the proposed framework on the Spoken COCO dataset containing natural human speech. Although trained on synthesized speech, SPIMSyn and SPIMSyn-D generalize effectively to real spoken descriptions without additional fine-tuning.

Spoken COCO

FLEURS

To evaluate robustness on long and complex spoken descriptions, we conduct experiments on FLEURS benchmark. SPIMSyn and SPIMSyn-D accurately capture semantic information even for lengthy speech inputs.

FLEURS

Common Voice

We evaluate our approach on Common Voice dataset containing diverse speakers recorded under different acoustic conditions such as speech in board room meetings and in other environments. The proposed framework demonstrates strong robustness across speaker variability while maintaining semantic fidelity.

Common Voice

LibriSpeech Generalization

To evaluate robustness on unconstrained spoken language, we conduct experiments on the LibriSpeech benchmark. The proposed SPIMSyn and SPIMSyn-D frameworks generate meaningful images directly from long-form speech without relying on intermediate Automatic Speech Recognition. These results demonstrate strong generalization beyond the synthesized training corpus.

LibriSpeech Results

Accent Robustness

We evaluate the proposed framework across multiple English accents including American, British, Indian, German, French, Chinese, and Japanese speakers. SPIMSyn and SPIMSyn-D consistently generate meaningful images irrespective to accent variations.

Accent Robustness

Prosody Robustness

We investigate whether variations in speaking style, including neutral, happy, angry, sad, and whispered speech, influence the generated images.

Prosody Results

Paralinguistic Speech Understanding

Beyond spoken words, human speech often contains paralinguistic cues such as laughter, sighs, and non-verbal vocalizations. We evaluate whether these signals influence the generated images. The proposed framework demonstrates robustness to such acoustic variations of the spoken content.

Paralinguistic Results

Extension to Speech-to-Video Generation

Although our framework is trained for direct Speech-to-Image generation, the predicted multimodal conditioning embeddings can also be integrated into modern diffusion-based video generation models. The following examples demonstrate preliminary results, illustrating the potential of extending toward direct Speech-to-Video synthesis.

Whisper + SD Baseline
Proposed SPIMSyn
Proposed SPIMSyn-D

Citation

@article{coming soon}