New scope reading the API instead of scraping HTML: one search path per run,
passed raw through --single-uri, and the answer stored as-is into its own
SQLite file created from the embedded schema.
Command line: --single-uri carries the path after /v1/search/
(category:flac/date:today/page:1) and must name a category; --sleep sets the
pause between pages, with the config value as fallback. A day named on the
command line is always fetched again and its dump rewritten, so cron can
refresh today hourly and yesterday daily.
Date placeholders: date:auto walks backwards to the newest day in
[srrdb.floor..yesterday] that has no dump yet, closing holes instead of
declaring the archive covered. date:today, date:yesterday and date:tomorrow
stand for the neighbouring days. Today is never taken by the walk, because a
day still being filled would be frozen by its own dump after the first run.
Dumps are the cursor as well as the record, so a missing dump directory is an
error instead of "nothing harvested yet": a renamed, unmounted or mistyped
directory must not send the run silently back to the beginning. An empty day is
a normal result and is remembered as harvested; API warnings never are, and
they reach the exit code.
Writes go through one transaction, so a day lands completely or not at all.
group and category come from the release line and the search path; status, nfo,
content and synced_at stay NULL. srrdb.log.jsonl records one line per finished
run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind
field.
Also: the console command contract returns (string, error), so a failing scope
reaches the process exit code and cron can tell a failed run from a successful
one. config/scope/srrdb.yaml is untracked because its paths differ per host.
modernc.org/sqlite replaces the archived libsql client, which only ever talked
to a server. pkg/repository/srrdb.go is split by responsibility and covered by
table-driven tests.
Three-level core resolution with priority overrides → dictionaries/model → suspects:
1. dicts/overrides.txt (highest priority)
Point fixes via exact lookup on Raw BEFORE the positional logic.
Format: <original line> => artist | title. An override fixes
Artist/Title and shifts the dictionary tail zone by the actual core
length (overrideCoreLength, fallback 2 tokens).
2. dicts/artists.txt
Whitelist of hyphenated artists (Iku-Turso, Post-Mortem, ...): a
[0]-[1] hit → Artist = 2 tokens, Title = [2], tail zone shifted.
One entry covers all releases of the artist automatically.
3. Suspected title continuation
Unknown tokens between the title zone and the tail (followed later by
dictionary markers) no longer pollute UnknownTokens; they go to a
dedicated TitleExtra field + warning "suspected title continuation" —
a reviewer hint that the core is probably longer than 2 tokens.
Catalogue numbers (parenthesized tokens) are resolved with FIRST
priority and never land in suspects.
Type field (normalized base release type):
- new normalization map dicts/types.txt (<Base> = variants):
EP = EP, CDS, CDM, 7INCH_VINYL, CDEP, ..., REMASTERED_EP; Demo = ...;
Split; Single; Compilation; Collaboration (curator-extendable)
- tokens matched against edition/source are additionally normalized to
the base type (first match wins); edition.txt/source.txt untouched,
Source stays raw
- TypeDistribution map[string]int added to GroupStats and the report
(console/JSON)
- OST/INSTRUMENTAL are release traits, not types; excluded from the map
Tests: old cases updated to the new suspect semantics; acceptance tests
added for overrides (Milicia_Oscura-Anti-Cristo), artist dictionary
(Iku-Turso), the suspect section, Type normalization (CDEP → EP with
raw Source=CDEP, first-match-wins) and TypeDistribution.
go build / go vet / go test ./internal/releaseparser/... — all green.
Generalise the parser: the MOONBLOOD group profile is gone, the parser and
its dictionaries are now group-agnostic under internal/releaseparser/groups.
Positional model:
- [0]=Artist, [1]=Title are fixed and never matched against dictionaries, so
titles like "Split"/"Demo"/"De" are never eaten.
- [len-1]=Group is always read from the line, not from the query key.
- [len-2]=Year when it is a 4-digit 19xx|20xx token; otherwise the slot is
resolved as a middle-zone token (e.g. FLAC is still picked up as format).
- The middle zone is resolved via case-insensitive dictionaries in the order
format -> source -> bit-depth -> resolution -> edition -> language ->
catalogue number; stored values keep their original case.
Catalogue numbers: any "(...)" token in the middle zone is a catalogue number
regardless of content (digits, roman numerals, dots, underscores, mixed case);
splitting never breaks tokens inside parentheses.
Dictionaries moved to internal/releaseparser/groups/resources/dicts and
embedded via go:embed; dead profile fields (Name, ExpectedTailOrder) removed;
profile comments rewritten in British English.
File-based AnalyzeGroup/TestParseMoonblood removed — titles now come from the
database (cmd/srrdb: srrdb <releaser>..., config/secret.yaml DSN).
Validation output (cmd/srrdb):
- tmp/releaseparser/<group>.json — per-group parsed ParsedRelease records
- tmp/releaseparser/<releaser>.report.json — per-request aggregate (same as console)
- review_unknown_tokens.txt — unknown/warning rows for manual review
Tests updated for the positional model; added
TestParseCatalogNumberWithDotAndRoman for (DYN.3140-2) and (GOWP_DCCCLXXXIX).
go build, go vet and go test all pass.
Parse releases with an explicit fixed positional model and document the
whole logic in British English doc-blocks.
Model:
- [0]=Artist, [1]=Title are fixed and never matched against dictionaries,
so titles like "Split"/"Demo"/"De" are never eaten.
- [len-1]=Group is always read from the line, not from the query key.
- [len-2]=Year when it is a 4-digit 19xx|20xx token; otherwise the slot is
resolved as a middle-zone token (e.g. FLAC still picked up as format).
- The middle zone is resolved via case-insensitive dictionaries in the order
format -> source -> bit-depth -> resolution -> edition -> language ->
catalogue number. Position within the zone is irrelevant.
Catalogue numbers:
- Any "(...)" token in the middle zone is a catalogue number regardless of
content (digits, roman numerals, dots, underscores, mixed case). Removed
the hasDigit requirement and character whitelist that rejected real
catalogues like (DYN.3140-2) and (GOWP_DCCCLXXXIX).
- Splitting never breaks tokens inside parentheses.
Case handling: lookups are case-insensitive; stored values keep their
original case from the line.
Documentation: replaced Ukrainian comments in parser.go and analyzer.go with
British-English doc-blocks describing the model, tail resolution order,
catalogue rule and confidence scoring.
Tests:
- Add TestParseCatalogNumberWithDotAndRoman for (DYN.3140-2)/(GOWP_DCCCLXXXIX).
- Update case-insensitive, split-title, no-year and language expectations.
- go build, go vet and go test all pass.
Yevhen Odynets
root
yevhen