PeerTube/packages/transcription-devtools/README.md

64 lines
2.6 KiB
Markdown
Raw Normal View History

2024-06-13 09:23:12 +02:00
# Transcription DevTools
Includes:
* __JiWER__ CLI NodeJS wrapper
* Benchmark tool to test multiple transcription engines
* TypeScript classes to evaluate word-error-rate of files generated by the transcription
## Build
```sh
npm run build
```
## Benchmark
A benchmark of available __transcribers__ might be run with:
```sh
npm run benchmark
```
```
┌────────────────────────┬───────────────────────┬───────────────────────┬──────────┬────────┬───────────────────────┐
│ (index) │ WER │ CER │ duration │ model │ engine │
├────────────────────────┼───────────────────────┼───────────────────────┼──────────┼────────┼───────────────────────┤
│ 5yZGBYqojXe7nuhq1TuHvz │ '28.39506172839506%' │ '9.62457337883959%' │ '41s' │ 'tiny' │ 'openai-whisper' │
│ x6qREJ2AkTU4e5YmvfivQN │ '29.75206611570248%' │ '10.46195652173913%' │ '15s' │ 'tiny' │ 'whisper-ctranslate2' │
└────────────────────────┴───────────────────────┴───────────────────────┴──────────┴────────┴───────────────────────┘
```
The benchmark may be run with multiple model builtin sizes:
```sh
MODELS=tiny,small,large npm run benchmark
```
## Jiwer
> *JiWER is a python tool for computing the word-error-rate of ASR systems.*
> https://jitsi.github.io/jiwer/cli/
__JiWER__ serves as a reference implementation to calculate errors rates between 2 text files:
- WER (Word Error Rate)
- CER (Character Error Rate)
### Usage
```typescript
const jiwerCLI = new JiwerClI('./reference.txt', './hypothesis.txt')
// WER as a percentage, ex: 0.03 -> 3%
console.log(await jiwerCLI.wer())
// CER as a percentage: 0.01 -> 1%
console.log(await jiwerCLI.cer())
// Detailed comparison report
console.log(await jiwerCLI.alignment())
```
## Resources
- https://jitsi.github.io/jiwer/
- https://github.com/rapidfuzz/RapidFuzz