releaseparser: Artist/Title boundary resolution + normalized Type field
Three-level core resolution with priority overrides → dictionaries/model → suspects: 1. dicts/overrides.txt (highest priority) Point fixes via exact lookup on Raw BEFORE the positional logic. Format: <original line> => artist | title. An override fixes Artist/Title and shifts the dictionary tail zone by the actual core length (overrideCoreLength, fallback 2 tokens). 2. dicts/artists.txt Whitelist of hyphenated artists (Iku-Turso, Post-Mortem, ...): a [0]-[1] hit → Artist = 2 tokens, Title = [2], tail zone shifted. One entry covers all releases of the artist automatically. 3. Suspected title continuation Unknown tokens between the title zone and the tail (followed later by dictionary markers) no longer pollute UnknownTokens; they go to a dedicated TitleExtra field + warning "suspected title continuation" — a reviewer hint that the core is probably longer than 2 tokens. Catalogue numbers (parenthesized tokens) are resolved with FIRST priority and never land in suspects. Type field (normalized base release type): - new normalization map dicts/types.txt (<Base> = variants): EP = EP, CDS, CDM, 7INCH_VINYL, CDEP, ..., REMASTERED_EP; Demo = ...; Split; Single; Compilation; Collaboration (curator-extendable) - tokens matched against edition/source are additionally normalized to the base type (first match wins); edition.txt/source.txt untouched, Source stays raw - TypeDistribution map[string]int added to GroupStats and the report (console/JSON) - OST/INSTRUMENTAL are release traits, not types; excluded from the map Tests: old cases updated to the new suspect semantics; acceptance tests added for overrides (Milicia_Oscura-Anti-Cristo), artist dictionary (Iku-Turso), the suspect section, Type normalization (CDEP → EP with raw Source=CDEP, first-match-wins) and TypeDistribution. go build / go vet / go test ./internal/releaseparser/... — all green.
This commit is contained in:
@@ -10,13 +10,13 @@ import (
|
||||
// GroupStats summarises how a set of release lines parsed: field
|
||||
// distributions, unknown tokens and warnings aggregated by occurrence, and
|
||||
// the average parsing confidence. The slices are sorted by descending count.
|
||||
|
||||
type GroupStats struct {
|
||||
Total int
|
||||
YearDistribution map[int]int
|
||||
SourceDistribution map[string]int
|
||||
BitDepthDistribution map[string]int
|
||||
ResolutionDistribution map[string]int
|
||||
TypeDistribution map[string]int
|
||||
LanguageDistribution map[string]int
|
||||
EditionDistribution map[string]int
|
||||
CommentDistribution map[string]int
|
||||
@@ -25,14 +25,13 @@ type GroupStats struct {
|
||||
AvgConfidence float64
|
||||
}
|
||||
|
||||
|
||||
// AnalyzeTitles parses a set of release lines (for example fetched from the
|
||||
// database by a particular releaser key) and aggregates the results into a
|
||||
// GroupStats summary.
|
||||
|
||||
func AnalyzeTitles(titles []string) GroupStats {
|
||||
stats := GroupStats{
|
||||
YearDistribution: make(map[int]int),
|
||||
TypeDistribution: make(map[string]int),
|
||||
SourceDistribution: make(map[string]int),
|
||||
BitDepthDistribution: make(map[string]int),
|
||||
ResolutionDistribution: make(map[string]int),
|
||||
@@ -43,12 +42,10 @@ func AnalyzeTitles(titles []string) GroupStats {
|
||||
WarningStats: []releaseparser.UnknownTokenStats{},
|
||||
}
|
||||
|
||||
|
||||
totalConfidence := 0.0
|
||||
unknownMap := make(map[string]*releaseparser.UnknownTokenStats)
|
||||
warningMap := make(map[string]*releaseparser.UnknownTokenStats)
|
||||
|
||||
|
||||
for _, title := range titles {
|
||||
result := Parse(title)
|
||||
stats.Total++
|
||||
@@ -77,6 +74,10 @@ func AnalyzeTitles(titles []string) GroupStats {
|
||||
}
|
||||
}
|
||||
|
||||
if result.Type != "" {
|
||||
stats.TypeDistribution[result.Type]++
|
||||
}
|
||||
|
||||
for _, unknown := range result.UnknownTokens {
|
||||
if _, exists := unknownMap[unknown]; !exists {
|
||||
unknownMap[unknown] = &releaseparser.UnknownTokenStats{
|
||||
@@ -127,7 +128,6 @@ func AnalyzeTitles(titles []string) GroupStats {
|
||||
return stats
|
||||
}
|
||||
|
||||
|
||||
// AnalyzeByGroup parses a set of release lines and returns per-group
|
||||
// statistics keyed by the actual group read from each line (not from the
|
||||
// query key, which may cover several groups). It is used to validate parsing
|
||||
@@ -146,7 +146,6 @@ func AnalyzeByGroup(titles []string) map[string]GroupStats {
|
||||
return out
|
||||
}
|
||||
|
||||
|
||||
// ParseByGroup parses a set of release lines into a per-group list of fully
|
||||
// populated ParsedRelease records, keyed by the actual group read from each
|
||||
// line. It is used to validate parsing quality field-by-field against the
|
||||
|
||||
Reference in New Issue
Block a user