releaseparser: move group-agnostic parser to groups/, GB doc-blocks, per-group JSON reports
Generalise the parser: the MOONBLOOD group profile is gone, the parser and its dictionaries are now group-agnostic under internal/releaseparser/groups. Positional model: - [0]=Artist, [1]=Title are fixed and never matched against dictionaries, so titles like "Split"/"Demo"/"De" are never eaten. - [len-1]=Group is always read from the line, not from the query key. - [len-2]=Year when it is a 4-digit 19xx|20xx token; otherwise the slot is resolved as a middle-zone token (e.g. FLAC is still picked up as format). - The middle zone is resolved via case-insensitive dictionaries in the order format -> source -> bit-depth -> resolution -> edition -> language -> catalogue number; stored values keep their original case. Catalogue numbers: any "(...)" token in the middle zone is a catalogue number regardless of content (digits, roman numerals, dots, underscores, mixed case); splitting never breaks tokens inside parentheses. Dictionaries moved to internal/releaseparser/groups/resources/dicts and embedded via go:embed; dead profile fields (Name, ExpectedTailOrder) removed; profile comments rewritten in British English. File-based AnalyzeGroup/TestParseMoonblood removed — titles now come from the database (cmd/srrdb: srrdb <releaser>..., config/secret.yaml DSN). Validation output (cmd/srrdb): - tmp/releaseparser/<group>.json — per-group parsed ParsedRelease records - tmp/releaseparser/<releaser>.report.json — per-request aggregate (same as console) - review_unknown_tokens.txt — unknown/warning rows for manual review Tests updated for the positional model; added TestParseCatalogNumberWithDotAndRoman for (DYN.3140-2) and (GOWP_DCCCLXXXIX). go build, go vet and go test all pass.
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
package groups
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"strings"
|
||||
|
||||
"git.kplus.net.ua/yevhen/resource-scraper/internal/releaseparser"
|
||||
)
|
||||
|
||||
// The token dictionaries live under resources/dicts/*.txt — one token (in any
|
||||
// case) per line; blank lines and lines starting with "#" are ignored. They
|
||||
// can be edited without recompiling the parsing logic: go:embed pulls them
|
||||
// into the binary at build time.
|
||||
|
||||
//go:embed resources/dicts/*.txt
|
||||
var dictFS embed.FS
|
||||
|
||||
func loadDict(name string) map[string]bool {
|
||||
data, err := dictFS.ReadFile("resources/dicts/" + name + ".txt")
|
||||
if err != nil {
|
||||
panic("release parser: cannot load dictionary " + name + ": " + err.Error())
|
||||
}
|
||||
out := make(map[string]bool)
|
||||
for line := range strings.SplitSeq(string(data), "\n") {
|
||||
line = strings.ToUpper(strings.TrimSpace(line))
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
out[line] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Profile holds the shared dictionaries for the release parser. The parser is
|
||||
// group-agnostic: the group name is read from each release line, never from
|
||||
// this profile, so there is a single set of dictionaries covering all groups.
|
||||
var Profile = releaseparser.GroupProfile{
|
||||
FormatTokens: loadDict("format"),
|
||||
SourceTokens: loadDict("source"),
|
||||
BitDepthTokens: loadDict("bitdepth"),
|
||||
ResolutionTokens: loadDict("resolution"),
|
||||
LanguageMarkers: loadDict("language"),
|
||||
EditionTokens: loadDict("edition"),
|
||||
CommentTokens: loadDict("comment"),
|
||||
}
|
||||
Reference in New Issue
Block a user