Concatenative Neural Speech Synthesis

Karl El Hajal1,2, Sanat Kumar Agrawal1,2,3, Enno Hermann1, Mathew Magimai.-Doss1

1Idiap Research Institute, Martigny, Switzerland    2EPFL, Lausanne, Switzerland    3IIT Bombay, Mumbai, India

Abstract

Speech synthesis has evolved from knowledge-driven approaches, such as concatenative speech synthesis, to data-driven approaches, such as neural speech synthesis. Both approaches can yield commercial speech synthesis systems. In this paper, we propose a novel speech synthesis approach that combines concatenative speech synthesis with a recently proposed neural speech synthesis approach based on self-supervised learning representations. Through experiments on multiple voices in the CMU ARCTIC corpus, we show that the proposed approach opens new ways to exploit prior knowledge and auxiliary data to reduce target data requirements and develop interpretable and controllable neural-based speech synthesis systems.

Diagram of the proposed target-domain training-free TTS framework combining concatenative text-to-SSL synthesis, kNN-VC, and a neural vocoder.

Experiment 1: Concatenative Synthesis Comparisons

cmu_us_awb_arctic

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cmu_us_clb_arctic

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cmu_us_jmk_arctic

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cmu_us_rms_arctic

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cmu_us_slt_arctic

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cstr_us_awb_arctic_multisyn

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

cstr_us_jmk_arctic_multisyn

Reference
Transcript Festival TTS Festival TTS + kNN-VC NeuralConcat NeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

Diphone Inventory: cmu_us_kal_diphone

Inventory recording

The inventory consists of 1619 diphones covering 229 s (0.064 h) of speech.

TranscriptFestival TTSFestival TTS Enc/DecFestival TTS + kNN-VCNeuralConcatNeuralConcat + kNN-VC
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

Experiment 2: Zero-Shot Multi-Speaker Comparisons

LibriSpeech speaker 7127

Reference
Transcript HierSpeech++ XTTS GlowkNN-TTS NeuralConcat (jmk_multisyn) NeuralConcat (kal_diphone)
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

LibriSpeech speaker 7729

Reference
Transcript HierSpeech++ XTTS GlowkNN-TTS NeuralConcat (jmk_multisyn) NeuralConcat (kal_diphone)
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

LibriSpeech speaker 6829

Reference
Transcript HierSpeech++ XTTS GlowkNN-TTS NeuralConcat (jmk_multisyn) NeuralConcat (kal_diphone)
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.

LibriSpeech speaker 8555

Reference
Transcript HierSpeech++ XTTS GlowkNN-TTS NeuralConcat (jmk_multisyn) NeuralConcat (kal_diphone)
he built a wifi door bell, he said.
however, the driver sustained serious injuries to the head.
think of the skiing route as of a similar hiking route.