Improve Spanish Anki vocabulary tooling

This commit is contained in:
Pawel Sarkowicz
2026-06-13 19:26:45 -04:00
parent d06ce15eed
commit 622f4899ac
6 changed files with 2027 additions and 16 deletions
+72 -1
View File
@@ -199,7 +199,78 @@ hablar 9
見る (見た) 6
```
### YouTube
### Words from TSV Export
Extract vocabulary from an Anki TSV export file instead of using AnkiConnect:
```shell
saiki words --lang es --input Español.txt --field 2 --output words_es_content.txt --debug words_es_debug.tsv
```
Additional file-based options:
```shell
saiki words --lang es --input Español.txt --field 2 --include-proper-nouns
saiki words --lang es --input Español.txt --field 2 --function-words
saiki words --lang es --input Español.txt --field 2 --lemma-corrections my_fixes.tsv
saiki words --lang es --input Español.txt --field 2 --bad-lemma-file bad_lemmas.txt
```
When `--input` is provided, `--field` specifies the 1-based column index of the
Spanish text (default 2). The audio column (index 1) and tags column are
automatically skipped. File-based NLP extraction is currently Spanish-specific;
the older AnkiConnect-based `saiki words jp` flow is unchanged.
The file-based pipeline:
- Parses Anki `#` header lines (`#separator:tab`, `#html:true`, `#tags column:N`)
- Uses Python's `csv` module for robust TSV parsing
- Removes `[sound:...mp3]` markers, HTML tags, and English glosses after
`<br><br>` from the field text
- Applies safe Spanish lemmatisation (multi-word lemmas like `ayudar yo` or
`lavar él` are rejected; bad lemmas like `comar` -> `comer` are corrected)
- Tracks POS counts, surface forms, example sentences, and source line numbers
- Tracks original spaCy lemmas and correction/fallback reasons for debugging
Output files produced:
- `words_es_content.txt` — cleaned content vocabulary (NOUN, VERB, ADJ, ADV)
- `words_es_debug.tsv` — per-lemma debug info
- `words_es_proper_nouns.txt` — proper nouns (only with `--include-proper-nouns`)
- `words_es_function_words.txt` — function words (only with `--function-words`)
- `words_es_suspicious_tokens.txt` — lemmas that required correction
Debug TSV example:
```text
lemma count pos_counts top_surface_forms example_sentences source_lines original_lemmas lemma_statuses status
comer 8 VERB:8 como, come, comen Yo como manzanas.; Ustedes los comen con arroz. 70,979 comar, comer corrected:comar->comer:1, ok:7
```
### Lint Spanish Cards
Check an Anki TSV export for suspicious or awkward Spanish:
```shell
saiki lint-anki-es --input Español.txt --field 2 --output suspicious_cards_es.tsv
```
Detects known errors and rule-based suspicious patterns and produces a TSV
report with: source line, original text, cleaned text, reason, and suggested
fix.
### Compare
Compare deck vocabulary against a target list:
```shell
saiki compare --deck-words words_es_content.txt --target-words target_es_top_1000.txt --output missing_from_deck.txt
saiki compare --deck-words words_es_content.txt --target-words target_es_top_1000.txt --min-frequency 3
```
Normalises case and optionally strips accents for matching, so `cómo` and
`como` are treated as the same word. Use `--min-frequency` to ignore accidental
low-frequency words in the deck.
Mine vocabulary or sentence rows from YouTube subtitles.