Natural Language Processing | Ziyang Lin

Modern ASR Technology Analysis: From Traditional Models to LLM-Driven New Paradigms

Sat, 28 Jun 2025 13:00:00 +0000

1. Background

1.1 Pain Points of Traditional ASR Models

Traditional Automatic Speech Recognition (ASR) models, such as those based on Hidden Markov Models-Gaussian Mixture Models (HMM-GMM) or Deep Neural Networks (DNN), perform well in specific domains and controlled environments but face numerous challenges:

Data Sparsity: Heavy dependence on large-scale, high-quality labeled datasets, resulting in poor generalization to low-resource languages or specific accents.
Insufficient Robustness: Performance drops dramatically in noisy environments, far-field audio capture, multi-person conversations, and other real-world scenarios.
Lack of Contextual Understanding: Models are typically limited to direct mapping from acoustic features to text, lacking understanding of long-range context, semantics, and speaker intent, leading to recognition errors (such as homophone confusion).
Limited Multi-task Capabilities: Traditional models are usually single-task oriented, supporting only speech transcription without simultaneously handling speaker diarization, language identification, translation, and other tasks.

1.2 Large Language Model (LLM) Driven ASR New Paradigm

In recent years, end-to-end large ASR models represented by Whisper have demonstrated unprecedented robustness and generalization capabilities through pretraining on massive, diverse unsupervised or weakly supervised data. These models typically adopt an Encoder-Decoder architecture, treating ASR as a sequence-to-sequence translation problem.

Typical Process:

graph TD
A["Raw Audio Waveform"] --> B["Feature Extraction (e.g., Log-Mel Spectrogram)"]
B --> C["Transformer Encoder"]
C --> D["Latent Representation"]
D --> E["Transformer Decoder"]
E --> F["Text Sequence Output"]

This approach not only simplifies the complex pipeline of traditional ASR but also learns rich acoustic and linguistic knowledge through large-scale data, enabling excellent performance even in zero-shot scenarios.

2. Analysis of ASR Model Solutions

2.1 Whisper-large-v3-turbo

Whisper is a pretrained ASR model developed by OpenAI, with its large-v3 and large-v3-turbo versions being among the industry-leading models.

2.1.1 Whisper Design

Structural Modules:

graph TD
A["Audio Input (30s segment)"] --> B["Log-Mel Spectrogram"]
B --> C["Transformer Encoder"]
C --> D["Encoded Representation"]
D --> E["Transformer Decoder"]
E --> F["Predicted Text Tokens"]
subgraph "Multi-task Processing"
E --> G["Transcription"]
E --> H["Translation"]
E --> I["Language Identification"]
end

Features:

Large-scale Weakly Supervised Training: Trained on 680,000 hours of multilingual, multi-task data, covering a wide range of accents, background noise, and technical terminology.
End-to-end Architecture: A unified Transformer model directly maps audio to text, without requiring external language models or alignment modules.
Multi-task Capability: The model can simultaneously handle multilingual speech transcription, speech translation, and language identification.
Robustness: Through carefully designed data augmentation and mixing, the model performs excellently under various challenging conditions.
Turbo Version: large-v3-turbo is an optimized version of large-v3, potentially offering improvements in inference speed, computational efficiency, or specific task performance, with approximately 798M parameters.

2.1.2 Problems Solved

Target Problem	Whisper's Solution
Poor Generalization	Large-scale pretraining on massive, diverse datasets covering nearly a hundred languages.
Insufficient Robustness	Training data includes various background noise, accents, and speaking styles, enhancing performance in real-world scenarios.
Weak Contextual Modeling	Transformer architecture captures long-range dependencies in audio signals.
Complex Deployment	Provides multiple model sizes (from `tiny` to `large`), with open-sourced code and model weights, facilitating community use and deployment.

2.1.3 Production Defect Analysis

2.1.3.1 Hallucination Issues

In segments with no speech or noise, the model sometimes generates meaningless or repetitive text, a common issue with large autoregressive models.
This phenomenon is particularly noticeable in long audio processing and may require additional post-processing logic for detection and filtering.

2.1.3.2 Limited Timestamp Precision

The model predicts word-level timestamps, but their precision may not meet the stringent requirements of certain applications (such as subtitle alignment, speech editing).
Timestamp accuracy decreases during long periods of silence or rapid speech flow.

2.1.3.3 High Computational Resource Requirements

The large-v3 model contains 1.55 billion parameters, and the turbo version has nearly 800 million parameters, demanding significant computational resources (especially GPU memory), making it unsuitable for direct execution on edge devices.
Although optimization techniques like quantization exist, balancing performance while reducing resource consumption remains a challenge.

2.1.3.4 Real-time Processing Bottlenecks

The model processes 30-second audio windows, requiring complex sliding window and caching mechanisms for real-time streaming ASR scenarios, which introduces additional latency.

2.2 SenseVoice

SenseVoice is a next-generation industrial-grade ASR model developed by Alibaba DAMO Academy's speech team. Unlike Whisper, which focuses on robust general transcription, SenseVoice emphasizes multi-functionality, real-time processing, and integration with downstream tasks.

2.2.1 SenseVoice Design

Structural Modules:

graph TD
A["Audio Stream"] --> B["FSMN-VAD (Voice Activity Detection)"]
B --> C["Encoder (e.g., SAN-M)"]
C --> D["Latent Representation"]
D --> E["Decoder"]
E --> F["Text Output"]
subgraph "Multi-task and Control"
G["Speaker Diarization"] --> C
H["Emotion Recognition"] --> C
I["Zero-shot TTS Prompt"] --> E
end

Features:

Unified End-to-end Model: Integrates acoustic model, language model, and punctuation prediction, achieving end-to-end output from speech to punctuated text.
Multi-task Learning: The model not only performs speech recognition but also simultaneously outputs speaker diarization, emotional information, and can even generate acoustic prompts for zero-shot TTS.
Streaming and Non-streaming Integration: Supports both streaming and non-streaming modes through a unified architecture, meeting the needs of real-time and offline scenarios.
TTS Integration: One innovation of SenseVoice is that its output can serve as a prompt for TTS models like CosyVoice, enabling voice cloning and transfer, closing the loop between ASR and TTS.

2.2.2 Problems Solved

Target Problem	SenseVoice's Solution
Single-task Limitation, Integration Difficulties	Designed as a multi-task model, natively supporting speaker diarization, emotion recognition, etc., simplifying dialogue system construction.
Poor Real-time Performance	Adopts efficient streaming architecture (such as SAN-M), combined with VAD, achieving low-latency real-time recognition.
Lack of Coordination with Downstream Tasks	Output includes rich meta-information (such as speaker, emotion) and can generate TTS prompts, achieving deep integration between ASR and TTS.
Punctuation Restoration Dependent on Post-processing	Incorporates punctuation prediction as a built-in task, achieving joint modeling of text and punctuation.

2.2.3 Production Defect Analysis

2.2.3.1 Model Complexity and Maintenance

As a complex model integrating multiple functions, its training and maintenance costs are relatively high.
Balancing multiple tasks may require fine-tuning to avoid performance degradation in any single task.

2.2.3.2 Generalization of Zero-shot Capabilities

Although it supports zero-shot TTS prompt generation, its voice cloning effect and stability when facing unseen speakers or complex acoustic environments may not match specialized voice cloning models.

2.2.3.3 Open-source Ecosystem and Community

Compared to Whisper's strong open-source community and rich ecosystem tools, SenseVoice, as an industrial-grade model, may have limited open-source availability and community support, affecting its popularity in academic and developer communities.

3. Conclusion

Whisper: Through large-scale weakly supervised learning, it has pushed the robustness and generalization capabilities of ASR to new heights. It is a powerful general-purpose speech recognizer, particularly suitable for processing diverse, uncontrolled audio data. Its design philosophy is “trading scale for performance,” excelling in zero-shot and multilingual scenarios.
SenseVoice: Represents the trend of ASR technology developing towards multi-functionality and integration. It is not just a recognizer but a perceptual frontend for conversational intelligence, aimed at providing richer, more real-time input for downstream tasks (such as dialogue systems, TTS). Its design philosophy is “fusion and collaboration,” emphasizing ASR's pivotal role in the entire intelligent interaction chain.

In summary, Whisper defines the performance baseline for modern ASR, while SenseVoice explores broader possibilities for ASR in industrial applications. Future ASR technology may develop towards combining the strengths of both: having both the robustness and generalization capabilities of Whisper and the multi-task collaboration and real-time processing capabilities of SenseVoice.

2020 Assessing the Funniness of Edited News Headlines

Mon, 27 Jul 2020 04:56:23 +0100

This project aims to develop potential solutions for the tasks rised by the competition Assessing the Funniness of Edited News Headlines (SemEval-2020) on the platform CodaLab

As of July 26, 2020, the test result of my trained model (‘bert-base-uncased’ from the Huggingface transformers) ranked third in the ranking of Post Evaluation Task 1 on the CodaLab

Tasks Description
Data Preprocessing
Models Choices & Design
Design of Training Processes (for task two only)
Optimizer & Learning Rate Scheduler
Prime Hyperparameters
Results
Discussion
Prospective
License

Tasks Description

Task one - Given one edited headline, design a regression model to predict how funny it is
Task two - Given the original headline and two manually edited versions, design a model to predict which edited version is the funnier of the two

Data Preprocessing

Task One

Convert original headlines into normal sentences (Remove < and /> by applying RE)
Get the edited version of headlines by doing word substitution using RE
Do tokenization and lowercasing for each edited-original headlines pair

Data preprocessing for pre-trained LMs (BERT-liked LMs):
- Version 1 - Concatenate original headlines and new headlines
- Version 2 - Concatenate new headlines and new words
- Version 3 – Contain only new headlines

Task Two

There are 3 versions of data preprocessing:

The normal version
The headlines truncated version
The punctuation removal version

Models Choices & Design

Task One

Two Inputs FFNN
Two Inputs CNN
Two Inputs RNN
Two Inputs Concatenated RNN
Pre-trained LM + a regression layer (LMs applied: BERT, ALBERT, XLNet, ELECTRA)

Two Inputs FFNN

This model is a two inputs’ feed forward neural network in which two input matrices representing all the original headlines and their corresponding edited headlines respectively are passed simultaneously to the first so called embedding layer of the model to get the word embedding of the fixed dimension for each word in the headline. Following above the model will do averaging for each headline to get the ‘document representation (vector)’ for each headline. Then these headlines’ vector representations are passed to a combination of three concatenated fully connected layers where the information about “how humour they are” are encoded. The Relu activation is applied after output from each of the first two hidden layers to prevent gradient vanishing and gradient exploding. Finally, all weighted sums or all vector products between the n-th row of the original matrix and the n-th row of the edited matrix are computed such that a vector with the size (origin_headlines_num, 1) is returned.

Two Inputs CNN

This model uses text CNN architecture with single windows size instead of FFNN for the regression task. The original headlines tensor and the edited headlines tensor are taken as the two inputs. In the output layer, unlike the normal matrix multiplication, all weighted sums or all vector products between the n-th row of the original matrix and the n-th row of the edited matrix are computed such that a vector with the size (origin_headlines_num, 1) is returned.

Two Inputs RNN

This model uses single layer bidirectional RNN architecture for the regression task. It is again the same as Two Inputs CNN that takes two tensors as its inputs and does a row-wise weighted summation in the output layer.

Two Inputs Concatenated RNN

This model is all the same as Two Inputs RNN except it concatenates the two last hidden states for the original headlines and the edited headlines to form a single representation and do a normal matrix multiplication in the output layer.

Pre-trained LM + a regression layer (LMs applied: BERT, ALBERT, XLNet, ELECTRA)

Version 1 - Concatenate original headlines and new headlines

Version 2 - Concatenate new headlines and new words

Version 3 – Contain only new headlines

Task Two

Pre-trained LM + a classification layer

Concatenate edited headline 1 and edited headline 2

Design of Training Processes (for task two only)

Version 1:

Training the model “Pre-trained LM + a classification layer” straightly for the real classification task

Version 2 (Fake Task + Real Task):

Firstly, training the model “Pre-trained LM + a regression layer” for a fake regression task on the training dataset
After training well, get rid of the regression layer and add an initialized classification layer on top of the pre-trained LM
Finally training the model for the real classification task

Optimizer & Learning Rate Scheduler

For FFNN, CNN, RNN:

The optimizer AdamW and the scheduler CosineAnnealingLR provided by pytorch

For pre-trained LMs (BERT-liked LMs):

The optimizer AdamW and the scheduler get_linear_schedule_with_warmup from Huggingface transformers

Prime Hyperparameters

Learning Rate
Fine-tuning Rate
Adam Epsilon
Weight Decay
Warmup Ratio
Number of Steps

Results

Task One

Best performance achieved by Two Inputs FFNN

EPOCHS	LRATE	EMBEDDING_DIM	HIDDEN_DIM_1	HIDDEN_DIM_2	HIDDEN_DIM_3	Train Loss	Val. Loss	Test Loss
100	0.145	300	100	50	10	0.575	0.581	0.576

Best performance achieved by Two Inputs CNN

EPOCHS	LRATE	EMBEDDING_DIM	FC_OUT_DIM	N_OUT_CHANNELS	WINDOW_SIZE	DROPOUT	Train Loss	Val. Loss
500	5e-3	50	25	100	3	0.7	0.624	0.661

Best performance achieved by Two Inputs RNN

EPOCHS	LRATE	EMBEDDING_DIM	HIDDEN_DIM	FC_OUTPUT_DIM	BIDIRECTIONAL	DROPOUT	Train Loss	Val. Loss	Test Loss
30	1e-4	50	128	32	Ture	0.3	0.586	0.576	0.571

Best performance achieved by Pre-trained LMs

Without Data Augmentation
- Model: bert_base_uncased
- Inputs structure: new headlines + new words
- Test loss: 0.52937
With Data Augmentation (add “funlines” training dataset)
- Model: bert_base_uncased
- Inputs structure: new headlines + new words
- Test loss: 0.52054 (Best performance achieved among all trials)


T1 Pre-trained LMs Log

Task Two

Version 1: Straightly training the model for the real task


T2 Log 1


T2 Log 2

Version 2: Fake Task Training + Real Task Training


T2 Fake Task Log


T2 Real Task Log

Discussion

Task One

The performance of Two Inputs RNN is just slightly better compared with that of the Two Inputs FFNN (0.5759702196 vs. 0.5751694002) while the time complexity of the Two Inputs RNN is much higher than the Two Inputs FFNN, at some point the current version of Two Inputs RNN is resources-wasted.
The Two Inputs CNN with a single window size performs worse than the Two Inputs FFNN and the Two Inputs RNN, and one of possible reasons is that it only looks at one size of n-gram and hence ignores the knowledge of n-grams with different lengths.

Task Two

For different preprocessing methods, the headlines truncated version and the punctuation removal version have the same performance as the normal one except that truncating headlines will reduce the training time for a single epoch.
The issue of overfitting on the training dataset is hard to overcome when applying BERT-liked pre-trained LMs (Although several methods, such as data augmentation, weight decay and dropout increase have been tried to mitigate this problem.
Surprisingly, the fake task training for pre-trained LMs does not help to improve the performance of the model in real task even a little bit.
With the same hyperparameters setting for the certain task, the performance of the newly proposed pre-trained LM is not necessarily the best.

Prospective

Construct a pretrain LM to do a binary classification task in which the model learns to decide whether a word from the edited new headline is original or edited. Take the embeddings out of the pretrain model and use it to initialize the model for the real regression task. By doing so we expect the embeddings can be informed some knowledge about the relationship between original headlines and edited headlines.
Build up a pretrain LM to do a text translation task on the training dataset and use the embeddings of this model to initialize the model for the real regression task. (Aim to learn the semantics of funniness).
Intuitively thinking the performance of the Two Inputs CNN might be improved by increasing the number of the window sizes (different n-gram filters).
Applying the pre-trained LM Longformer rather than other BERT-liked models for the task two, in which the Longformer has the ‘global attention mask’ and it can probably better model the relationship between the edited word and the other words in a headline (e.g. How important is the edited word for the whole headline in order to make it funnier? / How does the edited word contribute to the meaning of the whole sentence?).

License

This project following MIT License as written in the LICENSE file.

Natural Language Processing | Ziyang Lin

Modern ASR Technology Analysis: From Traditional Models to LLM-Driven New Paradigms

1. Background

1.1 Pain Points of Traditional ASR Models

1.2 Large Language Model (LLM) Driven ASR New Paradigm

2. Analysis of ASR Model Solutions

2.1 Whisper-large-v3-turbo

2.1.1 Whisper Design

2.1.2 Problems Solved

2.1.3 Production Defect Analysis

2.1.3.1 Hallucination Issues

2.1.3.2 Limited Timestamp Precision

2.1.3.3 High Computational Resource Requirements

2.1.3.4 Real-time Processing Bottlenecks

2.2 SenseVoice

2.2.1 SenseVoice Design

2.2.2 Problems Solved

2.2.3 Production Defect Analysis

2.2.3.1 Model Complexity and Maintenance

2.2.3.2 Generalization of Zero-shot Capabilities

2.2.3.3 Open-source Ecosystem and Community

3. Conclusion

2020 Assessing the Funniness of Edited News Headlines

Contents

Tasks Description

Data Preprocessing

Task One

Task Two

Models Choices & Design

Task One

Two Inputs FFNN

Two Inputs CNN

Two Inputs RNN

Two Inputs Concatenated RNN

Pre-trained LM + a regression layer (LMs applied: BERT, ALBERT, XLNet, ELECTRA)

Version 1 - Concatenate original headlines and new headlines

Version 2 - Concatenate new headlines and new words

Version 3 – Contain only new headlines

Task Two

Pre-trained LM + a classification layer

Concatenate edited headline 1 and edited headline 2

Design of Training Processes (for task two only)

Version 1:

Version 2 (Fake Task + Real Task):

Optimizer & Learning Rate Scheduler

For FFNN, CNN, RNN:

For pre-trained LMs (BERT-liked LMs):

Prime Hyperparameters

Results

Task One

Best performance achieved by Two Inputs FFNN

Best performance achieved by Two Inputs CNN

Best performance achieved by Two Inputs RNN

Best performance achieved by Pre-trained LMs

Task Two

Version 1: Straightly training the model for the real task

Version 2: Fake Task Training + Real Task Training

Discussion

Task One

Task Two

Prospective

License