Recent advances in Text-to-Speech (TTS) synthesis have enabled the generation of highly intelligible and natural-sounding speech. However, most conventional TTS systems still produce neutral or reading-style speech and therefore lack the expressiveness required for applications such as audiobook narration, virtual assistants, and human–machine interaction. While expressive speech synthesis has traditionally focused on prosodic variation, growing evidence suggests that pronunciation variation also plays an important role in conveying speaking style and emotion.
This study investigates the contribution of pronunciation variation to expressive speech synthesis in French and explores whether deep neural networks can automatically transform canonical pronunciations into speech-style-dependent pronunciations. Particular attention is given to pronunciation phenomena characteristic of expressive French speech, including the realization or omission of the mute e (schwa-like vowel) and variations in pause realization between words.
The research formulates the problem as a grapheme-to-phoneme (G2P) conversion task and develops a series of speech-style-aware pronunciation models based on a sequence-to-sequence encoder–decoder recurrent neural network (RNN) architecture implemented in OpenNMT. Several modeling strategies are evaluated, including the use of shorter phoneme sequences, explicit word-boundary information for pause prediction, and transfer learning through model pre-training. Model performance is assessed using the phoneme error rate (PER) on six neutral-to-emotion pronunciation conversion tasks in a French expressive speech corpus.
The experimental results show that deep neural network-based G2P conversion can successfully model pronunciation variation associated with expressive speech and consistently improves pronunciation prediction accuracy. The findings demonstrate that pronunciation variation should be considered alongside prosodic modeling in expressive speech synthesis and provide a practical framework for speech-style-aware grapheme-to-phoneme conversion in French.
Table of Contents
1 Introduction
1.1 Host institution
1.2 Research purpose
1.3 Background
1.3.1 Speech style in speech synthesis
1.3.2 Grapheme-to-Phoneme (G2P) conversion
1.4 Organization of the report
2 Sequence-to-sequence (Seq2Seq) encoder-decoder architecture for speech-style-aware G2P conversion
2.1 An overview of deep learning (DL) and neural networks
2.1.1 Feedfoward neural networks
2.1.2 Recurrent neural networks (RNNs) and long short-term memory (LSTM)
2.2 Sequence-to-sequence (Seq2Seq) encoder-decoder framework
2.3 Attention mechanism in encoder-decoder framework
2.4 Adaptation of Seq2Seq architecture for speech-style-aware G2P conversion
3 Experiments and results
3.1 Preliminary experiments
3.1.1 Experiment setup
3.1.2 Results
3.2 Using short segments
3.2.1 Constructing short segments and model configuration
3.2.2 Results
3.3 Architecture selection
3.4 Impact of word boundaries
3.5 Impact of pre-training
3.6 Evaluation for various emotions
4 Conclusion
A CMUdict for English results
Research Objectives and Topics
The main objective of this research is to investigate the contribution of segmental pronunciation variations to expressivity in speech synthesis for the French language. The research focuses on developing deep neural network models capable of transforming a neutral or canonical pronunciation into a specific pronunciation that corresponds to a target speech style or emotional state, thereby overcoming the limitations of standard neutral speech synthesis.
- Investigation of pronunciation variations (segmental) as a cue for emotional expressivity in French.
- Application of Grapheme-to-Phoneme (G2P) conversion to model specific speech styles and emotions.
- Utilization of sequence-to-sequence encoder-decoder architectures (specifically RNN/LSTM).
- Evaluation of techniques such as short segment construction, word boundary marking, and model pre-training.
- Improvement of Phoneme Error Rate (PER) performance in expressive TTS systems.
Excerpt from the Book
Grapheme-to-Phoneme (G2P) conversion
As stated previously, our major research purpose is to investigate if deep neural networks architecture can help to develop a model that can transform a canonical pronunciation into a specific pronunciation corresponding to a chosen speech style in terms of pronunciation variations, and in particular for French, pronunciation variations mainly concerns phenomena such as addition or omission of mute "e" (i.e. schwa-like vowels) and the realization or elimination of pauses between words. Therefore, such a model would have to learn how to modify certain sounds, to delete or add mute "e", and to deal with liaison consonants and pauses. Another, more global approach will be to learn a model that directly converts a text into a sequence of phonemes (and pauses) for a given speech style. Such a model should therefore learn both the conversion of letter sequences into phoneme sequences and the specificities of each speech style. To fulfill those goals, the scheme of grapheme-to-phoneme (G2P) conversion is apparently a good choice.
The G2P conversion system is a key component in both automatic speech recognition (ASR) and text-to-speech synthesis (TTS) systems as it describes how words are pronounced. The G2P conversion can be viewed as a process of translating an input sequence of graphemes (letters) to an output sequence of phonemes. Usually, the grapheme and phoneme sequences have been aligned to form joint grapheme-phoneme units. In these alignments, a grapheme may correspond to a single phoneme, a null phoneme with no pronunciation, or a compound phoneme that is a concatenation of two or more phonemes. It’s also possible that two or more graphemes correspond to a single phoneme.
Summary of Chapters
1 Introduction: Provides an overview of the host institution and the research motivation, highlighting the need for more expressive TTS systems beyond neutral speaking styles.
2 Sequence-to-sequence (Seq2Seq) encoder-decoder architecture for speech-style-aware G2P conversion: Reviews the deep learning fundamentals and the specific Seq2Seq encoder-decoder framework used to address the G2P conversion task.
3 Experiments and results: Details the experimental setup using the Caroline corpus, including the investigation of short segments, word boundaries, and the impact of pre-training on model performance.
4 Conclusion: Summarizes the key findings of the project and suggests potential directions for future research to further improve model expressivity and performance.
A CMUdict for English results: Presents the evaluation results obtained by applying the proposed model architecture to the English CMUdict dataset.
Keywords
Deep Learning, Neural Networks, Speech Synthesis, Text-To-Speech (TTS), Grapheme-to-Phoneme (G2P), Expressive Speech, Pronunciation Variation, French, Sequence-to-Sequence, LSTM, Phoneme Error Rate, Attention Mechanism, OpenNMT, Emotional Expressivity, Mute "e".
Frequently Asked Questions
What is the primary research goal of this work?
The primary goal is to investigate how pronunciation variations (segmental level) contribute to the perception of emotions in French speech synthesis and whether deep neural networks can model these variations effectively to enhance expressivity.
What are the central thematic fields?
The study centers on Text-To-Speech synthesis, Grapheme-to-Phoneme conversion, deep learning, sequence-to-sequence modeling, and the phonetic analysis of emotional and expressive speech in French.
Which scientific method is used to develop the models?
The research uses an encoder-decoder sequence-to-sequence (Seq2Seq) architecture based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) cells, implemented within the OpenNMT toolkit.
What does the main part of the report cover?
The main part documents extensive experiments, starting with whole-sentence level training and progressing to optimized short-segment models, incorporating word boundary information and pre-training techniques to minimize the Phoneme Error Rate.
What do the most significant keywords describe?
The keywords highlight the intersection of advanced deep learning techniques (like RNNs/LSTMs) and linguistic specificities, specifically focusing on expressive speech synthesis and G2P conversion for the French language.
What is the overall conclusion regarding the research?
The work demonstrates that using short phoneme segments and pre-trained models leads to significant improvements in predicting expressive pronunciation, thereby validating the proposed architecture for expressive speech synthesis.
How does the usage of short segments impact model performance?
The experiments showed that splitting sequences at interword pauses into shorter segments significantly reduces the Phoneme Error Rate (PER) compared to training on long, whole-sentence sequences, as it makes the learning process more manageable for the model.
What role does the "Caroline" corpus play?
The Caroline expressive speech corpus provides the necessary labeled data (neutral and 6 emotional expressions) for the French language, enabling the actress-based "exercise in style" data to be used for training the G2P conversion models.
- Quote paper
- Ruoxiao Yang (Author), 2020, Speech-Style-Aware Grapheme-to-Phoneme (G2P) Conversion for Expressive Speech Synthesis, Munich, GRIN Verlag, https://www.grin.com/document/1742322
