Compare commits

..

26 Commits

Author SHA1 Message Date
Yevhen Odynets 39aa8e2e81 fixed a new prescene layout 2026-09-27 01:12:52 +03:00
Yevhen Odynets 347f525801 releaseparser: Artist/Title boundary resolution + normalized Type field
Three-level core resolution with priority overrides → dictionaries/model → suspects:

1. dicts/overrides.txt (highest priority)
   Point fixes via exact lookup on Raw BEFORE the positional logic.
   Format: <original line> => artist | title. An override fixes
   Artist/Title and shifts the dictionary tail zone by the actual core
   length (overrideCoreLength, fallback 2 tokens).

2. dicts/artists.txt
   Whitelist of hyphenated artists (Iku-Turso, Post-Mortem, ...): a
   [0]-[1] hit → Artist = 2 tokens, Title = [2], tail zone shifted.
   One entry covers all releases of the artist automatically.

3. Suspected title continuation
   Unknown tokens between the title zone and the tail (followed later by
   dictionary markers) no longer pollute UnknownTokens; they go to a
   dedicated TitleExtra field + warning "suspected title continuation" —
   a reviewer hint that the core is probably longer than 2 tokens.
   Catalogue numbers (parenthesized tokens) are resolved with FIRST
   priority and never land in suspects.

Type field (normalized base release type):
- new normalization map dicts/types.txt (<Base> = variants):
  EP = EP, CDS, CDM, 7INCH_VINYL, CDEP, ..., REMASTERED_EP; Demo = ...;
  Split; Single; Compilation; Collaboration (curator-extendable)
- tokens matched against edition/source are additionally normalized to
  the base type (first match wins); edition.txt/source.txt untouched,
  Source stays raw
- TypeDistribution map[string]int added to GroupStats and the report
  (console/JSON)
- OST/INSTRUMENTAL are release traits, not types; excluded from the map

Tests: old cases updated to the new suspect semantics; acceptance tests
added for overrides (Milicia_Oscura-Anti-Cristo), artist dictionary
(Iku-Turso), the suspect section, Type normalization (CDEP → EP with
raw Source=CDEP, first-match-wins) and TypeDistribution.

go build / go vet / go test ./internal/releaseparser/... — all green.
2026-09-01 13:18:40 +03:00
Yevhen Odynets f2629f002c releaseparser: move group-agnostic parser to groups/, GB doc-blocks, per-group JSON reports
Generalise the parser: the MOONBLOOD group profile is gone, the parser and
its dictionaries are now group-agnostic under internal/releaseparser/groups.

Positional model:
- [0]=Artist, [1]=Title are fixed and never matched against dictionaries, so
  titles like "Split"/"Demo"/"De" are never eaten.
- [len-1]=Group is always read from the line, not from the query key.
- [len-2]=Year when it is a 4-digit 19xx|20xx token; otherwise the slot is
  resolved as a middle-zone token (e.g. FLAC is still picked up as format).
- The middle zone is resolved via case-insensitive dictionaries in the order
  format -> source -> bit-depth -> resolution -> edition -> language ->
  catalogue number; stored values keep their original case.

Catalogue numbers: any "(...)" token in the middle zone is a catalogue number
regardless of content (digits, roman numerals, dots, underscores, mixed case);
splitting never breaks tokens inside parentheses.

Dictionaries moved to internal/releaseparser/groups/resources/dicts and
embedded via go:embed; dead profile fields (Name, ExpectedTailOrder) removed;
profile comments rewritten in British English.

File-based AnalyzeGroup/TestParseMoonblood removed — titles now come from the
database (cmd/srrdb: srrdb <releaser>..., config/secret.yaml DSN).

Validation output (cmd/srrdb):
- tmp/releaseparser/<group>.json         — per-group parsed ParsedRelease records
- tmp/releaseparser/<releaser>.report.json — per-request aggregate (same as console)
- review_unknown_tokens.txt              — unknown/warning rows for manual review

Tests updated for the positional model; added
TestParseCatalogNumberWithDotAndRoman for (DYN.3140-2) and (GOWP_DCCCLXXXIX).
go build, go vet and go test all pass.
2026-09-01 11:28:19 +03:00
Yevhen Odynets feef080d80 fix(file) Remove Printline 2026-09-01 09:19:17 +03:00
Yevhen Odynets 736921ae0b cosmetic changes 2026-09-01 09:05:27 +03:00
Yevhen Odynets 31febb9f56 releaseparser: adopt fixed positional model, auto-detect catalogues, add GB doc-blocks
Parse releases with an explicit fixed positional model and document the
whole logic in British English doc-blocks.

Model:
- [0]=Artist, [1]=Title are fixed and never matched against dictionaries,
  so titles like "Split"/"Demo"/"De" are never eaten.
- [len-1]=Group is always read from the line, not from the query key.
- [len-2]=Year when it is a 4-digit 19xx|20xx token; otherwise the slot is
  resolved as a middle-zone token (e.g. FLAC still picked up as format).
- The middle zone is resolved via case-insensitive dictionaries in the order
  format -> source -> bit-depth -> resolution -> edition -> language ->
  catalogue number. Position within the zone is irrelevant.

Catalogue numbers:
- Any "(...)" token in the middle zone is a catalogue number regardless of
  content (digits, roman numerals, dots, underscores, mixed case). Removed
  the hasDigit requirement and character whitelist that rejected real
  catalogues like (DYN.3140-2) and (GOWP_DCCCLXXXIX).
- Splitting never breaks tokens inside parentheses.

Case handling: lookups are case-insensitive; stored values keep their
original case from the line.

Documentation: replaced Ukrainian comments in parser.go and analyzer.go with
British-English doc-blocks describing the model, tail resolution order,
catalogue rule and confidence scoring.

Tests:
- Add TestParseCatalogNumberWithDotAndRoman for (DYN.3140-2)/(GOWP_DCCCLXXXIX).
- Update case-insensitive, split-title, no-year and language expectations.
- go build, go vet and go test all pass.
2026-09-01 04:26:06 +03:00
Yevhen Odynets 3f3598c446 add nultie gpouring 2026-09-01 00:08:03 +03:00
Yevhen Odynets 1b2521d4ac refactored & audited, real analizer added 2026-08-31 22:24:38 +03:00
Yevhen Odynets e11c82bd76 Fixed rutracker after changing stucture of the atom file of theirs own 2026-08-31 20:24:07 +03:00
Yevhen Odynets fedb470da6 minor changes 2026-08-31 19:00:09 +03:00
Yevhen Odynets 167d98ac9c updated go version 2026-04-29 19:12:36 +03:00
Yevhen Odynets fcf98f20cb Merge branch 'master' of git.kplus.net.ua-yevhen:yevhen/resource-scraper 2026-01-18 01:27:48 +02:00
root 1565c066f7 Make install.sh executable 2026-01-18 01:27:29 +02:00
Yevhen Odynets 52fbcaebbf upgrade packages 2026-01-18 01:22:01 +02:00
Yevhen Odynets 0979bf5ad8 tuned default precene fetch 2026-01-18 01:07:51 +02:00
yevhen 8eeb962016 changed handlers summoning 2025-11-12 12:56:22 +02:00
yevhen c6d1a8d455 fixed mails moving 2025-10-28 13:36:53 +02:00
yevhen c8745e1f60 some mirror fixed 2025-10-28 12:59:54 +02:00
yevhen ac4b0ef47b added to ignore STONERD in prescene 2025-10-28 11:38:00 +02:00
yevhen 879b33d385 all fixed 2025-10-28 11:36:11 +02:00
yevhen 54aaa80f23 migrate to 1.25.3 2025-10-28 10:16:56 +02:00
yevhen a41ccec4a0 change modelu name to git.kplus.net.ua 2025-10-06 12:30:33 +03:00
yevhen b26c43ee4b change git imports 2025-10-06 12:28:15 +03:00
yevhen 1544b943fc cahnge git server and fixed time parsing 2025-10-06 12:21:22 +03:00
yevhen 970f6c2bfd change done folder 2025-03-04 11:39:56 +02:00
yevhen 1c17e73e2d stb config changed 2025-03-04 08:47:42 +02:00
46 changed files with 1998 additions and 392 deletions
+267
View File
@@ -0,0 +1,267 @@
# =============================================================
# .gitattributes — Universal JetBrains / Multi-Stack Template
# Covers: PHP, Go, TypeScript, JavaScript, Rust, Dart/Flutter,
# Python, Ruby, HCL/Terraform, SQL, HTML, CSS, DevOps
# =============================================================
# ─────────────────────────────────────────────────────────────
# DEFAULTS: LF everywhere, CRLF only for Windows-specific files
# ─────────────────────────────────────────────────────────────
* text=auto eol=lf
*.bat text eol=crlf
*.cmd text eol=crlf
*.ps1 text eol=crlf
*.psm1 text eol=crlf
*.psd1 text eol=crlf
# ─────────────────────────────────────────────────────────────
# PHP
# ─────────────────────────────────────────────────────────────
*.php text diff=php
*.phtml text diff=php
*.php8 text diff=php
# ─────────────────────────────────────────────────────────────
# Go
# ─────────────────────────────────────────────────────────────
*.go text diff=golang
go.sum text -diff
go.work.sum text -diff
# ─────────────────────────────────────────────────────────────
# TypeScript / JavaScript
# ─────────────────────────────────────────────────────────────
*.ts text diff=typescript
*.tsx text diff=typescript
*.js text diff=javascript
*.jsx text diff=javascript
*.mjs text diff=javascript
*.cjs text diff=javascript
# ─────────────────────────────────────────────────────────────
# Rust
# ─────────────────────────────────────────────────────────────
*.rs text diff=rust
Cargo.lock text -diff
# ─────────────────────────────────────────────────────────────
# Dart / Flutter
# ─────────────────────────────────────────────────────────────
*.dart text diff=dart
pubspec.lock text -diff
# ─────────────────────────────────────────────────────────────
# Lua
# ─────────────────────────────────────────────────────────────
*.lua text diff=lua
# ─────────────────────────────────────────────────────────────
# Python
# ─────────────────────────────────────────────────────────────
*.py text diff=python
*.pyi text diff=python
Pipfile.lock text -diff
# ─────────────────────────────────────────────────────────────
# Ruby
# ─────────────────────────────────────────────────────────────
*.rb text diff=ruby
Gemfile.lock text -diff
# ─────────────────────────────────────────────────────────────
# Web: HTML / CSS / Templates
# ─────────────────────────────────────────────────────────────
*.html text diff=html
*.htm text diff=html
*.blade.php text diff=html
*.templ text diff=html
*.twig text diff=html
*.njk text diff=html
*.ejs text diff=html
*.hbs text diff=html
*.mustache text diff=html
*.vue text diff=html
*.svelte text diff=html
*.css text diff=css
*.scss text diff=css
*.sass text diff=css
*.less text diff=css
# ─────────────────────────────────────────────────────────────
# Data / Config formats
# ─────────────────────────────────────────────────────────────
*.json text diff=json
*.jsonc text diff=json
*.json5 text diff=json
*.yaml text diff=yaml
*.yml text diff=yaml
*.toml text
*.xml text diff=html
*.svg text diff=html
*.env text
*.env.* text
*.ini text
*.cfg text
*.conf text
*.properties text
# ─────────────────────────────────────────────────────────────
# HCL / Terraform / HashiCorp stack
# ─────────────────────────────────────────────────────────────
*.hcl text diff=terraform
*.tf text diff=terraform
*.tfvars text diff=terraform
*.tfvars.json text diff=json
.terraform.lock.hcl text -diff merge=ours
# ─────────────────────────────────────────────────────────────
# SQL / Database
# ─────────────────────────────────────────────────────────────
*.sql text diff=sql
*.pgsql text diff=sql
*.mysql text diff=sql
# ─────────────────────────────────────────────────────────────
# Shell / Scripts
# ─────────────────────────────────────────────────────────────
*.sh text eol=lf
*.bash text eol=lf
*.zsh text eol=lf
*.fish text eol=lf
Makefile text eol=lf
*.mk text eol=lf
Dockerfile text eol=lf
Dockerfile.* text eol=lf
*.dockerfile text eol=lf
*.containerfile text eol=lf
# ─────────────────────────────────────────────────────────────
# Documentation
# ─────────────────────────────────────────────────────────────
*.md text diff=markdown
*.mdx text diff=markdown
*.rst text
*.txt text
*.adoc text
LICENSE text
AUTHORS text
CODEOWNERS text
# ─────────────────────────────────────────────────────────────
# Lock files — no diff noise, merge=ours
# ─────────────────────────────────────────────────────────────
*.lock text -diff merge=ours
*.lock.* text -diff merge=ours
package-lock.json text -diff merge=ours
yarn.lock text -diff merge=ours
pnpm-lock.yaml text -diff merge=ours
composer.lock text -diff merge=ours
bun.lockb binary
# ─────────────────────────────────────────────────────────────
# Generated files — hide from diff + GitHub stats
# ─────────────────────────────────────────────────────────────
*_templ.go linguist-generated=true -diff
*.pb.go linguist-generated=true -diff
*.gen.go linguist-generated=true -diff
*.generated.go linguist-generated=true -diff
*.generated.ts linguist-generated=true -diff
*.generated.php linguist-generated=true -diff
*_test.snap linguist-generated=true -diff
# ─────────────────────────────────────────────────────────────
# Binary: Images
# ─────────────────────────────────────────────────────────────
*.png binary
*.jpg binary
*.jpeg binary
*.gif binary
*.bmp binary
*.ico binary
*.webp binary
*.avif binary
*.tiff binary
*.tif binary
*.psd binary
*.ai binary
*.eps binary
# ─────────────────────────────────────────────────────────────
# Binary: Fonts
# ─────────────────────────────────────────────────────────────
*.ttf binary
*.otf binary
*.woff binary
*.woff2 binary
*.eot binary
# ─────────────────────────────────────────────────────────────
# Binary: Audio / Video
# ─────────────────────────────────────────────────────────────
*.mp3 binary
*.mp4 binary
*.wav binary
*.flac binary
*.ogg binary
*.opus binary
*.aac binary
*.m4a binary
*.webm binary
*.avi binary
*.mov binary
*.mkv binary
# ─────────────────────────────────────────────────────────────
# Binary: Archives / Executables
# ─────────────────────────────────────────────────────────────
*.zip binary
*.tar binary
*.gz binary
*.bz2 binary
*.xz binary
*.7z binary
*.rar binary
*.exe binary
*.dll binary
*.so binary
*.dylib binary
*.a binary
*.o binary
*.wasm binary
# ─────────────────────────────────────────────────────────────
# Binary: Documents / Data files
# ─────────────────────────────────────────────────────────────
*.pdf binary
*.doc binary
*.docx binary
*.xls binary
*.xlsx binary
*.ppt binary
*.pptx binary
*.db binary
*.sqlite binary
*.sqlite3 binary
# ─────────────────────────────────────────────────────────────
# Terraform state — never should be committed, but just in case
# ─────────────────────────────────────────────────────────────
*.tfstate binary
*.tfstate.backup binary
# ─────────────────────────────────────────────────────────────
# git archive export-ignore
# (files excluded from GitHub release tarballs)
# ─────────────────────────────────────────────────────────────
README.md export-ignore
CHANGELOG.md export-ignore
CONTRIBUTING.md.md export-ignore
SECURITY.md export-ignore
.editorconfig export-ignore
.env.example export-ignore
.gitattributes export-ignore
.gitignore export-ignore
.github export-ignore
docs export-ignore
tests export-ignore
+8 -21
View File
@@ -1,34 +1,21 @@
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
.idea/**/aws.xml
.idea/**/contentModel.xml
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
.idea/**/gradle.xml
.idea/**/libraries
.idea/
cmake-build-*/
.idea/**/mongoSettings.xml
*.iws
out/
.idea_modules/
atlassian-ide-plugin.xml
.idea/replstate.xml
.idea/sonarlint/
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
.idea/httpRequests
.idea/caches/build_file_checksums.ser
.env
tmp/
vendor/
config/secret.yaml
*.md
mdb
internal/releaseparser/groups/resources/*
review_unknown_tokens.txt
srrdb.exe
**/*.exe
/**/*.exe
+6
View File
@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="DataSourcePerFileMappings">
<file url="file://$PROJECT_DIR$/pkg/repository/table/external_sources.go" value="e3965414-f0c4-4a05-af48-11e028c626da" />
</component>
</project>
+11
View File
@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="GoImports">
<option name="excludedPackages">
<array>
<option value="github.com/pkg/errors" />
<option value="golang.org/x/net/context" />
</array>
</option>
</component>
</project>
-1
View File
@@ -12,7 +12,6 @@
<option value="gosimple" />
</list>
</option>
<option name="goLinterExe" value="C:\Users\theAmok\go\bin\golangci-lint.exe" />
<option name="linterSelected" value="true" />
</component>
</project>
+4 -1
View File
@@ -1,8 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="WEB_MODULE" version="4">
<component name="Go" enabled="true" />
<component name="GoModuleSettings" enabled="true" />
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$" />
<content url="file://$MODULE_DIR$">
<excludeFolder url="file://$MODULE_DIR$/tmp" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
+3 -3
View File
@@ -6,9 +6,9 @@ import (
_ "github.com/go-sql-driver/mysql"
scraper "git.amok.space/yevhen/resource-scraper/internal"
"git.amok.space/yevhen/resource-scraper/internal/config"
"git.amok.space/yevhen/resource-scraper/internal/logging"
scraper "git.kplus.net.ua/yevhen/resource-scraper/internal"
"git.kplus.net.ua/yevhen/resource-scraper/internal/config"
"git.kplus.net.ua/yevhen/resource-scraper/internal/logging"
)
var (
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"strings"
"github.com/jmoiron/sqlx"
"github.com/spf13/viper"
_ "github.com/go-sql-driver/mysql"
"git.kplus.net.ua/yevhen/resource-scraper/internal/releaseparser/groups"
)
func main() {
if len(os.Args) < 2 {
log.Fatal("usage: srrdb <releaser> [<releaser>...]")
}
releasers := os.Args[1:]
// Читаємо config/secret.yaml (без змісту в коді — тільки потрібні ключі)
viper.SetConfigFile("config/secret.yaml")
if err := viper.ReadInConfig(); err != nil {
log.Fatal("failed to read config/secret.yaml: ", err)
}
drv := viper.GetString("db.driver")
dsn := viper.GetString("db.data_source_name")
dbase, err := sqlx.Connect(drv, dsn)
if err != nil {
panic("Failed to connect to the database: " + err.Error())
}
defer dbase.Close()
const outDir = "tmp/releaseparser"
if err := os.MkdirAll(outDir, 0o755); err != nil {
log.Fatal(err)
}
out := make(map[string]groups.GroupStats, len(releasers))
for _, releaser := range releasers {
var titles []string
err = dbase.Select(&titles,
`SELECT title FROM theamok_mdb.external_sources
WHERE type = 'prescene' AND releaser = ?`, releaser)
if err != nil {
log.Fatal(err)
}
out[releaser] = groups.AnalyzeTitles(titles)
// Per-group parsed records keyed by the actual group read from each
// line: every release with all resolved fields for manual validation.
for group, parsed := range groups.ParseByGroup(titles) {
writeJSON(filepath.Join(outDir, safeName(group)+".json"), parsed)
}
// Per-request report identical to what is printed to the console.
writeJSON(filepath.Join(outDir, safeName(releaser)+".report.json"), out[releaser])
}
data, err := json.MarshalIndent(out, "", " ")
if err != nil {
log.Fatal(err)
}
// Ручна ревізія: всі невідомі токени — у файл для правки бази.
review, err := os.OpenFile("review_unknown_tokens.txt",
os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
if err != nil {
log.Fatal(err)
}
defer review.Close()
for releaser, stats := range out {
for _, unk := range stats.UnknownTokens {
for _, rel := range unk.Releases {
fmt.Fprintf(review, "%s\t%s\t%s\n", releaser, unk.Token, rel)
}
}
for _, warn := range stats.WarningStats {
for _, rel := range warn.Releases {
fmt.Fprintf(review, "%s\tWARN\t%s\t%s\n", releaser, warn.Token, rel)
}
}
}
fmt.Println(string(data))
}
// writeJSON writes value as an indented JSON document to the given path.
func writeJSON(path string, value any) {
data, err := json.MarshalIndent(value, "", " ")
if err != nil {
log.Fatalf("marshal %s: %v", path, err)
}
if err := os.WriteFile(path, data, 0o644); err != nil {
log.Fatalf("write %s: %v", path, err)
}
}
// safeName strips characters that are unsafe in file names.
func safeName(s string) string {
repl := func(r rune) rune {
switch {
case r >= 'a' && r <= 'z':
return r
case r >= 'A' && r <= 'Z':
return r
case r >= '0' && r <= '9':
return r
case r == '-' || r == '_' || r == '.':
return r
default:
return '_'
}
}
return strings.Map(repl, s)
}
+2 -2
View File
@@ -2,5 +2,5 @@ scope:
allow: [info, web, rutracker, prescene, stb, metal-archives, console]
default: console
mail:
amok.space:
dial-tls: mail.amok.space:993
kplus.net.ua:
dial-tls: s21.thehost.com.ua:993
View File
+1 -1
View File
@@ -2,7 +2,7 @@ role: console
endpoint: https://scnlog.me/music/flac/
endpoint-next: page/%s/
sleep-before-next-iteration: 5s
levels-to-scrape: 5
levels-to-scrape: 6
groups:
tags:
neformat:
+6 -6
View File
@@ -1,16 +1,16 @@
role: console
stb:
endpoint: stb@amok.space
#mailboxes: ["INBOX"]
endpoint: stb@kplus.net.ua
mailboxes: ["INBOX"]
#mailboxes: ["INBOX", "Junk Mail", "Done"]
mailboxes: ["Junk Mail"]
#mailboxes: ["Junk Mail"]
#mailboxes: ["Done"]
search-criteria: ["Your Share The Brutality digest", "noreply@sharethebrutality.info"]
move-processed-to-mailbox:
succeed: Processed/Succeed
succeed: Succeed
#succeed: Done
failed: Processed/Failed
suspicious: Processed/Suspicious
failed: Failed
suspicious: Suspicious
sender:
mailbox: noreply
host: sharethebrutality.info
+25 -30
View File
@@ -1,47 +1,42 @@
module git.amok.space/yevhen/resource-scraper
module git.kplus.net.ua/yevhen/resource-scraper
go 1.24
go 1.27
require (
github.com/emersion/go-imap/v2 v2.0.0-beta.5
github.com/go-chi/chi/v5 v5.2.1
github.com/go-chi/cors v1.2.1
github.com/davecgh/go-spew v1.1.1
github.com/dromara/carbon/v2 v2.6.17
github.com/emersion/go-imap/v2 v2.0.0-beta.8
github.com/go-chi/chi/v5 v5.3.2
github.com/go-chi/cors v1.2.2
github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c
github.com/go-sql-driver/mysql v1.9.0
github.com/golang-module/carbon/v2 v2.4.1
github.com/go-sql-driver/mysql v1.10.1
github.com/iancoleman/strcase v0.3.0
github.com/jmoiron/sqlx v1.4.0
github.com/logrusorgru/aurora/v4 v4.0.0
github.com/mewkiz/flac v1.0.12
github.com/spf13/pflag v1.0.6
github.com/spf13/viper v1.19.0
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa
golang.org/x/net v0.35.0
github.com/mewkiz/flac v1.0.14
github.com/spf13/pflag v1.0.10
github.com/spf13/viper v1.21.0
golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba
golang.org/x/net v0.59.0
)
require (
filippo.io/edwards25519 v1.1.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
filippo.io/edwards25519 v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.5 // indirect
github.com/emersion/go-message v0.18.2 // indirect
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 // indirect
github.com/fsnotify/fsnotify v1.8.0 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/icza/bitio v1.1.0 // indirect
github.com/magiconair/properties v1.8.9 // indirect
github.com/mewkiz/pkg v0.0.0-20241223220703-7f3c7df797ff // indirect
github.com/mewkiz/pkg v0.0.0-20260703220044-4fb89b18cc87 // indirect
github.com/mewpkg/term v0.0.0-20241026122259-37a80af23985 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
github.com/sagikazarmark/locafero v0.7.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.12.0 // indirect
github.com/spf13/cast v1.7.1 // indirect
github.com/pelletier/go-toml/v2 v2.4.3 // indirect
github.com/sagikazarmark/locafero v0.12.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/spf13/cast v1.10.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/text v0.22.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
)
+49 -101
View File
@@ -1,38 +1,37 @@
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/d4l3k/messagediff v1.2.2-0.20190829033028-7e0a312ae40b/go.mod h1:Oozbb1TVXFac9FtSIxHBMnBCq2qeH/2KkEQxENCrlLo=
filippo.io/edwards25519 v1.2.0 h1:crnVqOiS4jqYleHd9vaKZ+HKtHfllngJIiOpNpoJsjo=
filippo.io/edwards25519 v1.2.0/go.mod h1:xzAOLCNug/yB62zG1bQ8uziwrIqIuxhctzJT18Q77mc=
github.com/andybalholm/cascadia v1.3.5 h1:RLjq12WJy58dN6eCIQrz0bAGZkztHWsEPFxP53Y7Ms8=
github.com/andybalholm/cascadia v1.3.5/go.mod h1:BLRmbRjpEtNKieZOCCvYj4RqN+KRA41GBe/5O+G93kM=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/emersion/go-imap/v2 v2.0.0-beta.5 h1:H3858DNmBuXyMK1++YrQIRdpKE1MwBc+ywBtg3n+0wA=
github.com/emersion/go-imap/v2 v2.0.0-beta.5/go.mod h1:BZTFHsS1hmgBkFlHqbxGLXk2hnRqTItUgwjSSCsYNAk=
github.com/dromara/carbon/v2 v2.6.17 h1:QL1LVcp5A/NQtIPbc1vrL9qIXxPNWnaDOBaFlmP1iDg=
github.com/dromara/carbon/v2 v2.6.17/go.mod h1:NGo3reeV5vhWCYWcSqbJRZm46MEwyfYI5EJRdVFoLJo=
github.com/emersion/go-imap/v2 v2.0.0-beta.8 h1:5IXZK1E33DyeP526320J3RS7eFlCYGFgtbrfapqDPug=
github.com/emersion/go-imap/v2 v2.0.0-beta.8/go.mod h1:dhoFe2Q0PwLrMD7oZw8ODuaD0vLYPe5uj2wcOMnvh48=
github.com/emersion/go-message v0.18.2 h1:rl55SQdjd9oJcIoQNhubD2Acs1E6IzlZISRTK7x/Lpg=
github.com/emersion/go-message v0.18.2/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA=
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 h1:oP4q0fw+fOSWn3DfFi4EXdT+B+gTtzx8GC9xsc26Znk=
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.8.0 h1:dAwr6QBTBZIkG8roQaJjGof0pp0EeF+tNV7YBP3F/8M=
github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/go-chi/chi/v5 v5.2.1 h1:KOIHODQj58PmL80G2Eak4WdvUzjSJSm0vG72crDCqb8=
github.com/go-chi/chi/v5 v5.2.1/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/cors v1.2.1 h1:xEC8UT3Rlp2QuWNEr4Fs/c2EAGVKBwy/1vHx3bppil4=
github.com/go-chi/cors v1.2.1/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.3.2 h1:5YQkICvTCSZ25hoRsyJazN0scjzKGiu4VAUc7H1o1nY=
github.com/go-chi/chi/v5 v5.3.2/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-chi/cors v1.2.2 h1:Jmey33TE+b+rB7fT8MUy1u0I4L+NARQlK6LhzKPSyQE=
github.com/go-chi/cors v1.2.2/go.mod h1:sSbTewc+6wYHBBCW7ytsFSn836hqM7JxpglAy2Vzc58=
github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c h1:wpkoddUomPfHiOziHZixGO5ZBS73cKqVzZipfrLmO1w=
github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c/go.mod h1:oVDCh3qjJMLVUSILBRwrm+Bc6RNXGZYtoh9xdvf1ffM=
github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg=
github.com/go-sql-driver/mysql v1.9.0 h1:Y0zIbQXhQKmQgTp44Y1dp3wTXcn804QoTptLZT1vtvo=
github.com/go-sql-driver/mysql v1.9.0/go.mod h1:pDetrLJeA3oMujJuvXc8RJoasr589B6A9fwzD3QMrqw=
github.com/go-sql-driver/mysql v1.10.1 h1:arlSnNLq6a5yxGxV7qg9lF4j0C+KwD6NbQyKr9QL6ME=
github.com/go-sql-driver/mysql v1.10.1/go.mod h1:M+cqaI7+xxXGG9swrdeUIoPG3Y3KCkF0pZej+SK+nWk=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f h1:3BSP1Tbs2djlpprl7wCLuiqMaUh5SJkkzI2gDs+FgLs=
github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f/go.mod h1:Pcatq5tYkCW2Q6yrR2VRHlbHpZ/R4/7qyL1TCF7vl14=
github.com/golang-module/carbon/v2 v2.4.1 h1:cYUD8T+rHeX+qIybGYpnJ8I90F10dvyEF67VNOO+zZM=
github.com/golang-module/carbon/v2 v2.4.1/go.mod h1:1jP9AZ4k2+lmfgY/wZgmtsN52VcHC5YuPM6varKDTkM=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/iancoleman/strcase v0.3.0 h1:nTXanmYxhfFAMjZL34Ov6gkzEsSJZ5DbhxWjvSASxEI=
github.com/iancoleman/strcase v0.3.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
github.com/icza/bitio v1.1.0 h1:ysX4vtldjdi3Ygai5m1cWy4oLkhWTAi+SyO6HC8L9T0=
@@ -41,7 +40,6 @@ github.com/icza/mighty v0.0.0-20180919140131-cfd07d671de6 h1:8UsGZ2rr2ksmEru6lTo
github.com/icza/mighty v0.0.0-20180919140131-cfd07d671de6/go.mod h1:xQig96I1VNBDIWGCdTt54nHt6EeI639SmHycLYL7FkA=
github.com/jmoiron/sqlx v1.4.0 h1:1PLqN7S1UYp5t4SrVVnt4nUVNemrDAtxlulVe+Qgm3o=
github.com/jmoiron/sqlx v1.4.0/go.mod h1:ZrZ7UsYB/weZdl2Bxg6jCRO9c3YHl8r3ahlKmRT4JLY=
github.com/jszwec/csvutil v1.5.1/go.mod h1:Rpu7Uu9giO9subDyMCIQfHVDuLrcaC36UA4YcJjGBkg=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -50,124 +48,74 @@ github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/logrusorgru/aurora/v4 v4.0.0 h1:sRjfPpun/63iADiSvGGjgA1cAYegEWMPCJdUpJYn9JA=
github.com/logrusorgru/aurora/v4 v4.0.0/go.mod h1:lP0iIa2nrnT/qoFXcOZSrZQpJ1o6n2CUf/hyHi2Q4ZQ=
github.com/magiconair/properties v1.8.9 h1:nWcCbLq1N2v/cpNsy5WvQ37Fb+YElfq20WJ/a8RkpQM=
github.com/magiconair/properties v1.8.9/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/mewkiz/flac v1.0.12 h1:5Y1BRlUebfiVXPmz7hDD7h3ceV2XNrGNMejNVjDpgPY=
github.com/mewkiz/flac v1.0.12/go.mod h1:1UeXlFRJp4ft2mfZnPLRpQTd7cSjb/s17o7JQzzyrCA=
github.com/mewkiz/pkg v0.0.0-20230226050401-4010bf0fec14/go.mod h1:QYCFBiH5q6XTHEbWhR0uhR3M9qNPoD2CSQzr0g75kE4=
github.com/mewkiz/pkg v0.0.0-20241223220703-7f3c7df797ff h1:ts1OPi6zzzHE4nosvaXfkvhVNHeBghtxwwJL/PBsVHk=
github.com/mewkiz/pkg v0.0.0-20241223220703-7f3c7df797ff/go.mod h1:dpOqQ/SkbEbPklAuIQqDGHc6K6kygt0L1XYa/IMlQeI=
github.com/mewkiz/flac v1.0.14 h1:hyRGAM8NCKznoPmIi9zz2jyO+nfmxY2ErqBnHZ+gxh4=
github.com/mewkiz/flac v1.0.14/go.mod h1:HfPYDA+oxjyuqMu2V+cyKcxF51KM6incpw5eZXmfA6k=
github.com/mewkiz/pkg v0.0.0-20260703220044-4fb89b18cc87 h1:fpWNLIg7nX2f+D+6ZveQ5hfkTjq5WgMBtYbz5Nb1w0Y=
github.com/mewkiz/pkg v0.0.0-20260703220044-4fb89b18cc87/go.mod h1:wohiqwTwXA5twjDES74xIJLGjAxo3/0pHSbTXLbSDW8=
github.com/mewpkg/term v0.0.0-20241026122259-37a80af23985 h1:h8O1byDZ1uk6RUXMhj1QJU3VXFKXHDZxr4TXRPGeBa8=
github.com/mewpkg/term v0.0.0-20241026122259-37a80af23985/go.mod h1:uiPmbdUbdt1NkGApKl7htQjZ8S7XaGUAVulJUJ9v6q4=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pelletier/go-toml/v2 v2.4.3 h1:GTRvJQutkOSftxIFD5xw9aepkYNuPWmVJpffdDPYVpY=
github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/sagikazarmark/locafero v0.7.0 h1:5MqpDsTGNDhY8sGp0Aowyf0qKsPrhewaLSsFaodPcyo=
github.com/sagikazarmark/locafero v0.7.0/go.mod h1:2za3Cg5rMaTMoG/2Ulr9AwtFaIppKXTRYnozin4aB5k=
github.com/sagikazarmark/slog-shim v0.1.0 h1:diDBnUNK9N/354PgrxMywXnAwEr1QZcOr6gto+ugjYE=
github.com/sagikazarmark/slog-shim v0.1.0/go.mod h1:SrcSrq8aKtyuqEI1uvTDTK1arOWRIczQRv+GVI1AkeQ=
github.com/sourcegraph/conc v0.3.0 h1:OQTbbt6P72L20UqAkXXuLOj79LfEanQ+YQFNpLA9ySo=
github.com/sourcegraph/conc v0.3.0/go.mod h1:Sdozi7LEKbFPqYX2/J+iBAM6HpqSLTASQIKqDmF7Mt0=
github.com/spf13/afero v1.12.0 h1:UcOPyRBYczmFn6yvphxkn9ZEOY65cpwGKb5mL36mrqs=
github.com/spf13/afero v1.12.0/go.mod h1:ZTlWwG4/ahT8W7T0WQ5uYmjI9duaLQGy3Q2OAl4sk/4=
github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.19.0 h1:RWq5SEjt8o25SROyN3z2OrDB9l7RPd3lwTWU8EcEdcI=
github.com/spf13/viper v1.19.0/go.mod h1:GQUN9bilAbhU/jgc1bKs99f/suXKeUMct8Adx5+Ntkg=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/sagikazarmark/locafero v0.12.0 h1:/NQhBAkUb4+fH1jivKHWusDYFjMOOKU88eegjfxfHb4=
github.com/sagikazarmark/locafero v0.12.0/go.mod h1:sZh36u/YSZ918v0Io+U9ogLYQJ9tLLBmM4eneO6WwsI=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/spf13/cast v1.10.0 h1:h2x0u2shc1QuLHfxi+cTJvs30+ZAHOGRic8uyGTDWxY=
github.com/spf13/cast v1.10.0/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.21.0 h1:x5S+0EU27Lbphp4UKm1C+1oQO+rKx36vfCoaVebLFSU=
github.com/spf13/viper v1.21.0/go.mod h1:P0lhsswPGWD/1lZJ9ny3fYnVqxiegrlNrEmgLjbTCAY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa h1:t2QcU6V556bFjYgu4L6C+6VrCPyJZ+eyRsABUPs1mz4=
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa/go.mod h1:BHOTPb3L19zxehTsLoJXVaTktb06DFgmdW6Wb9s8jqk=
golang.org/x/image v0.5.0/go.mod h1:FVC7BI/5Ym8R25iw5OLsgshdUBbT1h5jZTpA+mvAdZ4=
golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba h1:Ck8QetSgk912qxWLMCKxd0in+aiyBQyDSMae6e/xmpU=
golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba/go.mod h1:50RgIsmK7OwqzTTeqcSXQW8SswW0o8fRcDxmqGluJ8E=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+1 -1
View File
@@ -3,7 +3,7 @@ package sugar
import (
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func IsDev() bool {
+79 -15
View File
@@ -2,32 +2,43 @@ package sugar
import (
"fmt"
"log"
"log/slog"
"os"
"path/filepath"
"strconv"
)
// EnsureDir joins the given path segments and creates the directory
// (including all parent directories) if it does not already exist.
// Returns the resolved path or an error if creation fails.
func EnsureDir(dirPath ...string) (string, error) {
outPath := filepath.Join(dirPath...)
if len(outPath) < 3 {
return "", fmt.Errorf("the path is short to be a folder: %s", outPath)
if outPath == "" {
return "", fmt.Errorf("path must not be empty")
}
if _, err := os.Stat(outPath); os.IsNotExist(err) {
var dirMod uint64
if dirMod, err = strconv.ParseUint("0775", 8, 32); err == nil {
err = os.Mkdir(outPath, os.FileMode(dirMod))
}
if err != nil && !os.IsExist(err) {
slog.Error("error creating tmp dir", "err", err)
}
if err := os.MkdirAll(outPath, os.FileMode(0775)); err != nil {
return "", fmt.Errorf("failed to ensure directory %q: %w", outPath, err)
}
return outPath, nil
}
// ExecDir returns the directory of the running binary.
// Works correctly even with symlinks and -trimpath builds.
func ExecDir() (string, error) {
exe, err := os.Executable()
if err != nil {
return "", err
}
// EvalSymlinks is critical — os.Executable can return a symlink path
resolved, err := filepath.EvalSymlinks(exe)
if err != nil {
return "", err
}
return filepath.Dir(resolved), nil
}
func WriteDataToTmpFile(content, fp string) (int, string, error) {
dir, err := EnsureDir(os.TempDir(), "mdb")
if err != nil {
@@ -58,6 +69,59 @@ func WriteDataToTmpFile(content, fp string) (int, string, error) {
return filesize, fp, nil
}
/*
REMOVED_FROM_MMT_DUE_TO_REDOING_OF_THE_EVENT_LOG
*/
// ConfigPaths returns candidate directories for config lookup.
// Priority: XDG_CONFIG_HOME → binary dir → CWD → /etc/appname
func ConfigPaths(appName string) []string {
var candidates []string
// 1. Explicit env override (highest priority)
if envPath := os.Getenv("APP_CONFIG_DIR"); envPath != "" {
candidates = append(candidates, envPath)
}
// 2. Relative to binary (best for production deployments)
if exe, err := os.Executable(); err == nil {
if resolved, err := filepath.EvalSymlinks(exe); err == nil {
candidates = append(candidates, filepath.Dir(resolved))
}
}
// 3. CWD (convenient for `go run` and development)
if cwd, err := os.Getwd(); err == nil {
candidates = append(candidates, cwd)
}
// 4. XDG standard (Linux/Unix)
if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" {
candidates = append(candidates, filepath.Join(xdg, appName))
} else if home, err := os.UserHomeDir(); err == nil {
candidates = append(candidates, filepath.Join(home, ".config", appName))
}
// 5. System-wide fallback
candidates = append(candidates, filepath.Join("/etc", appName))
return candidates
}
// FindConfig searches all candidate dirs for the given filename.
func FindConfig(appName, filename string) (string, error) {
for _, dir := range ConfigPaths(appName) {
path := filepath.Join(dir, filename)
if _, err := os.Stat(path); err == nil {
return path, nil
}
}
return "", fmt.Errorf("config %q not found in any candidate directory", filename)
}
// DebugPath Debug helper — remove before production
func DebugPath(path string) {
info, err := os.Stat(path)
log.Printf("path=%q exists=%v err=%v mode=%v", path, err == nil, err, func() string {
if info != nil {
return info.Mode().String()
}
return "n/a"
}())
}
Executable
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
#go build -trimpath -ldflags="-s -w" -o /usr/local/bin/mdb ./cmd
go build -trimpath -ldflags="-s -w" -o /opt/sources/resource-scraper/mdb ./cmd
+5 -5
View File
@@ -8,11 +8,11 @@ import (
"github.com/spf13/viper"
"golang.org/x/exp/slices"
"git.amok.space/yevhen/resource-scraper/internal/db"
"git.amok.space/yevhen/resource-scraper/pkg/handler"
"git.amok.space/yevhen/resource-scraper/pkg/repository"
"git.amok.space/yevhen/resource-scraper/pkg/service"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/internal/db"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/handler"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/repository"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/service"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func isAllowScope() bool {
+12 -2
View File
@@ -5,17 +5,21 @@ import (
"log"
"log/slog"
"path/filepath"
"runtime"
"slices"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/helper/sugar"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func New() {
configFilePath := viper.GetString(constant.FlagConfigFile)
configName := filepath.Base(configFilePath)
sugar.DebugPath(configFilePath)
dir := filepath.Dir(configFilePath)
if dir == configName && dir == "." {
configName = "default"
@@ -23,7 +27,13 @@ func New() {
configDir, _ := filepath.Abs(dir)
viper.AddConfigPath(configDir)
switch runtime.GOOS {
case "windows":
viper.AddConfigPath(configDir)
case "linux":
viper.AddConfigPath("/opt/sources/resource-scraper/config")
}
viper.SetConfigName(configName)
err := viper.ReadInConfig()
if err != nil { // Handle errors reading the config file
+2 -2
View File
@@ -9,7 +9,7 @@ import (
"github.com/spf13/pflag"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
const usage = `Music Database (MDB) server/cli craftware'
@@ -24,7 +24,7 @@ Commands:
-section -s Select section e.g.: rutracker, bandcamp etc.
The Go runtime version: %s
Report bugs to https://git.amok.space/yevhen/resource-scraper/issues`
Report bugs to https://git.kplus.net.ua/yevhen/resource-scraper/issues`
const (
version = "0.1"
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/go-chi/chi/v5"
"github.com/go-chi/cors"
"git.amok.space/yevhen/resource-scraper/helper/web"
"git.kplus.net.ua/yevhen/resource-scraper/helper/web"
)
/*type Server struct {
+23 -2
View File
@@ -10,7 +10,7 @@ import (
"github.com/logrusorgru/aurora/v4"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/helper/sugar"
"git.kplus.net.ua/yevhen/resource-scraper/helper/sugar"
)
type EmailService struct {
@@ -57,6 +57,7 @@ func (s *EmailService) Connect() {
s.client = conn
s.pass = mail[s.User]
s.success("connected to " + mail["dial-tls"])
//defer s.Logout()
@@ -67,8 +68,12 @@ func (s *EmailService) Login() {
err := s.client.Login(s.User, s.pass).Wait()
if s.CheckErr("Login", err) {
fmt.Println("========================================")
fmt.Println(err.Error())
fmt.Println("========================================")
return
}
s.success(s.User + " logged")
}
@@ -116,6 +121,7 @@ func (s *EmailService) ListMessages(mailboxes []string, criteria *imap.SearchCri
}
messages, err := s.client.Fetch(seqSet, fetchOptions).Collect()
if s.CheckErr("Fetch", err) {
continue
}
@@ -132,7 +138,20 @@ func (s *EmailService) LogOut() {
}
func (s *EmailService) CreateMailbox(mailboxName string) {
s.client.Create(mailboxName, nil)
if err := s.client.Create(mailboxName, nil); err != nil {
log.Println("Помилка створення папки:", mailboxName, err)
} else {
log.Println("Створено папку:", mailboxName)
}
}
func (s *EmailService) DeleteMailbox(mailboxName string) {
// Вибираємо папку
if err := s.client.Delete(mailboxName); err != nil {
log.Println("Помилка видалення папки:", mailboxName, err)
} else {
log.Println("Папка видалена:", mailboxName)
}
}
func (s *EmailService) MailboxesList() {
@@ -162,7 +181,9 @@ func (s *EmailService) MoveMessageToMailbox(msg *client.FetchMessageBuffer, stat
movable.AddNum(msg.SeqNum)
mailbox := viper.GetStringMapString("stb.move-processed-to-mailbox")
wait, err := s.client.Move(movable, mailbox[status]).Wait()
if s.CheckErr("Moving to archive", err) {
fmt.Println("Error moving to mail:", mailbox[status], "Error:", err.Error())
return false
}
+160
View File
@@ -0,0 +1,160 @@
package groups
import (
"sort"
"strings"
"git.kplus.net.ua/yevhen/resource-scraper/internal/releaseparser"
)
// GroupStats summarises how a set of release lines parsed: field
// distributions, unknown tokens and warnings aggregated by occurrence, and
// the average parsing confidence. The slices are sorted by descending count.
type GroupStats struct {
Total int
YearDistribution map[int]int
SourceDistribution map[string]int
BitDepthDistribution map[string]int
ResolutionDistribution map[string]int
TypeDistribution map[string]int
LanguageDistribution map[string]int
EditionDistribution map[string]int
CommentDistribution map[string]int
UnknownTokens []releaseparser.UnknownTokenStats
WarningStats []releaseparser.UnknownTokenStats
AvgConfidence float64
}
// AnalyzeTitles parses a set of release lines (for example fetched from the
// database by a particular releaser key) and aggregates the results into a
// GroupStats summary.
func AnalyzeTitles(titles []string) GroupStats {
stats := GroupStats{
YearDistribution: make(map[int]int),
TypeDistribution: make(map[string]int),
SourceDistribution: make(map[string]int),
BitDepthDistribution: make(map[string]int),
ResolutionDistribution: make(map[string]int),
LanguageDistribution: make(map[string]int),
EditionDistribution: make(map[string]int),
CommentDistribution: make(map[string]int),
UnknownTokens: []releaseparser.UnknownTokenStats{},
WarningStats: []releaseparser.UnknownTokenStats{},
}
totalConfidence := 0.0
unknownMap := make(map[string]*releaseparser.UnknownTokenStats)
warningMap := make(map[string]*releaseparser.UnknownTokenStats)
for _, title := range titles {
result := Parse(title)
stats.Total++
if result.Year != 0 {
stats.YearDistribution[result.Year]++
}
if result.Source != "" {
stats.SourceDistribution[result.Source]++
}
if result.BitDepth != "" {
stats.BitDepthDistribution[result.BitDepth]++
}
if result.Resolution != "" {
stats.ResolutionDistribution[result.Resolution]++
}
if result.Language != "" {
stats.LanguageDistribution[result.Language]++
}
if result.Edition != "" {
stats.EditionDistribution[result.Edition]++
}
if result.Comment != "" {
for _, c := range strings.Split(result.Comment, ",") {
stats.CommentDistribution[c]++
}
}
if result.Type != "" {
stats.TypeDistribution[result.Type]++
}
for _, unknown := range result.UnknownTokens {
if _, exists := unknownMap[unknown]; !exists {
unknownMap[unknown] = &releaseparser.UnknownTokenStats{
Token: unknown,
Count: 0,
Releases: []string{},
}
}
unknownMap[unknown].Count++
unknownMap[unknown].Releases = append(unknownMap[unknown].Releases, title)
}
for _, warning := range result.Warnings {
if _, exists := warningMap[warning]; !exists {
warningMap[warning] = &releaseparser.UnknownTokenStats{
Token: warning,
Count: 0,
Releases: []string{},
}
}
warningMap[warning].Count++
warningMap[warning].Releases = append(warningMap[warning].Releases, title)
}
totalConfidence += result.Confidence
}
// Convert the aggregates from maps to slices and order them by count
// (most frequent first).
for _, tokenStats := range unknownMap {
stats.UnknownTokens = append(stats.UnknownTokens, *tokenStats)
}
sort.Slice(stats.UnknownTokens, func(i, j int) bool {
return stats.UnknownTokens[i].Count > stats.UnknownTokens[j].Count
})
for _, w := range warningMap {
stats.WarningStats = append(stats.WarningStats, *w)
}
sort.Slice(stats.WarningStats, func(i, j int) bool {
return stats.WarningStats[i].Count > stats.WarningStats[j].Count
})
if stats.Total > 0 {
stats.AvgConfidence = totalConfidence / float64(stats.Total)
}
return stats
}
// AnalyzeByGroup parses a set of release lines and returns per-group
// statistics keyed by the actual group read from each line (not from the
// query key, which may cover several groups). It is used to validate parsing
// quality against the original data, grouped by the real releaser name.
func AnalyzeByGroup(titles []string) map[string]GroupStats {
byGroup := make(map[string][]string)
for _, title := range titles {
group := Parse(title).Group
byGroup[group] = append(byGroup[group], title)
}
out := make(map[string]GroupStats, len(byGroup))
for group, lines := range byGroup {
out[group] = AnalyzeTitles(lines)
}
return out
}
// ParseByGroup parses a set of release lines into a per-group list of fully
// populated ParsedRelease records, keyed by the actual group read from each
// line. It is used to validate parsing quality field-by-field against the
// original data.
func ParseByGroup(titles []string) map[string][]releaseparser.ParsedRelease {
byGroup := make(map[string][]releaseparser.ParsedRelease)
for _, title := range titles {
result := Parse(title)
byGroup[result.Group] = append(byGroup[result.Group], result)
}
return byGroup
}
+310
View File
@@ -0,0 +1,310 @@
package groups
import (
"regexp"
"strconv"
"strings"
"git.kplus.net.ua/yevhen/resource-scraper/internal/releaseparser"
)
var yearRegexp = regexp.MustCompile(`^(19|20)\d{2}$`)
// Parse splits a release line into tokens and maps them onto a fixed
// positional model while the interior is resolved solely through the
// group profile's dictionaries.
//
// The release line is split on "-" (but never inside curved parentheses, so
// that a catalogue number such as (BRC30-001-CD-RRR108) stays one token).
// The positional layout is:
//
// [0] Artist - fixed; never matched against dictionaries.
// [1] Title - fixed; never matched against dictionaries.
// [len-2] Year - if the token is a 4-digit year (19xx | 20xx).
// [len-1] Group - always read from the line, never from the query key.
//
// Everything between [1] and the reserved trailing slots ([2 .. len-3], or
// [2 .. len-2] when there is no year) is the middle zone. Each token there is
// looked up in the dictionaries (format, source, bit-depth, resolution,
// language, edition) and, if it is a parenthesized token, as a catalogue
// number. Position within the middle zone is irrelevant. Tokens that match
// nothing are NOT silently consumed — they are reported in UnknownTokens so
// that they can be reviewed and added to a dictionary.
//
// Why [0]/[1] are fixed and never matched against the dictionaries: titles
// such as "Split" (Alone_In_The_Morgue-Split-...) or "Demo"
// (Mystifier-Demo-Nized-...) are legitimate words that collide with edition
// markers. Since index 1 is always the title, regardless of case or
// dictionary membership, such words are never eaten.
//
// Values are preserved in their original case (as they appear in the line);
// the dictionary lookups themselves are case-insensitive.
func Parse(raw string) releaseparser.ParsedRelease {
result := releaseparser.ParsedRelease{
Raw: strings.TrimSpace(raw),
Warnings: []string{},
UnknownTokens: []string{},
}
tokens, comments := extractComments(split(raw))
if len(comments) > 0 {
result.Comment = strings.Join(comments, ",")
}
if len(tokens) < 5 {
result.Warnings = append(result.Warnings, "too few tokens")
result.Confidence = 0.2
return result
}
// Left core boundary: after the resolved Artist (2 tokens by default,
// 3 with an artist dictionary hit, or fixed by an override).
start := 2
// Level 1: exact-lookup override table fixes Artist/Title only;
// the tail zone is parsed as usual below. The override is
// authoritative about the core: leftover tokens are plain unknowns,
// not part of the title.
if o, ok := Profile.Overrides[result.Raw]; ok {
result.Artist, result.Title = o.Artist, o.Title
start = overrideCoreLength(tokens, o)
}
// Level 2: artist dictionary — a hyphenated artist may span ANY number
// of leading tokens ("Shadu-Nar-Mattaru" = 3). The LONGEST whitelisted
// prefix wins; the title is the token right after it. At least one
// title token and one tail token must remain.
artistDict := false
if result.Artist == "" {
for k := len(tokens) - 2; k >= 2; k-- {
if Profile.ArtistTokens[strings.ToUpper(strings.Join(tokens[:k], "-"))] {
artistDict = true
start = k + 1
break
}
}
}
result.Group = tokens[len(tokens)-1]
end := len(tokens) - 1
if len(tokens) >= 3 && yearRegexp.MatchString(tokens[len(tokens)-2]) {
result.Year, _ = strconv.Atoi(tokens[len(tokens)-2])
end = len(tokens) - 2
}
// Right core boundary: the first dictionary-known token after the
// resolved artist. The boundary is contextual, but the title itself
// stays a single token ([1], or [2] with an artist dictionary hit).
// Unknown tokens inside the core region ([start .. tailStart)) are
// REPORTED in UnknownTokens — never eaten, never glued into the title.
tailStart := end
for i := start; i < end; i++ {
if isKnownToken(tokens[i]) {
tailStart = i
break
}
}
// Resolve Artist/Title from the core unless the override already did.
if result.Artist == "" {
if artistDict {
// The artist spans tokens [0 .. start-2]; re-join them.
result.Artist = strings.Join(tokens[:start-1], "-")
} else {
result.Artist = tokens[0]
}
if len(tokens) > start-1 {
result.Title = tokens[start-1]
}
}
// Core-region leftovers (e.g. the rest of a wildcards title after an
// override, or a hyphenated title fragment without an artist entry):
// visible in UnknownTokens for review.
for _, tok := range tokens[start:tailStart] {
result.UnknownTokens = append(result.UnknownTokens, tok)
}
// Level 3: dictionary-driven tail zone, starting after the resolved core.
parseTailZone(tokens[tailStart:end], &result)
score(&result)
return result
}
// split breaks a raw release line into tokens on "-", but never inside
// curved parentheses, so that a catalogue number such as
// (BRC30-001-CD-RRR108) remains a single token.
func split(raw string) []string {
s := strings.TrimSpace(raw)
out := make([]string, 0, 8)
var cur strings.Builder
inParen := false
for _, r := range s {
switch {
case r == '(':
inParen = true
cur.WriteRune(r)
case r == ')':
inParen = false
cur.WriteRune(r)
case r == '-' && !inParen:
if t := strings.TrimSpace(cur.String()); t != "" {
out = append(out, t)
}
cur.Reset()
default:
cur.WriteRune(r)
}
}
if t := strings.TrimSpace(cur.String()); t != "" {
out = append(out, t)
}
return out
}
// extractComments removes service comments (CommentTokens) from the token
// list and returns the cleaned list together with the removed comments.
func extractComments(tokens []string) ([]string, []string) {
out := make([]string, 0, len(tokens))
var comments []string
for _, tok := range tokens {
if Profile.CommentTokens[strings.ToUpper(tok)] {
comments = append(comments, tok)
continue
}
out = append(out, tok)
}
return out, comments
}
// catalogNumber reports whether the token is a parenthesized catalogue
// number. In the positional model any token wrapped in curved parentheses
// that sits in the middle zone ([2..]) is a catalogue number regardless of
// its content (digits, letters, roman numerals, dots, underscores, mixed
// case). Parenthesized tokens at [0]/[1] are handled as artist/title, not
// here. It returns the parenthesized content, or "" when the token is not
// a "(...)" group.
func catalogNumber(token string) string {
if len(token) < 3 || token[0] != '(' || token[len(token)-1] != ')' {
return ""
}
return token[1 : len(token)-1]
}
// isKnownToken reports whether the token belongs to the tail zone: it
// matches one of the profile dictionaries or is a parenthesized catalogue
// number. The first such token after the core marks the right core
// boundary.
func isKnownToken(tok string) bool {
up := strings.ToUpper(tok)
return Profile.FormatTokens[up] || Profile.SourceTokens[up] ||
Profile.BitDepthTokens[up] || Profile.ResolutionTokens[up] ||
Profile.EditionTokens[up] || Profile.LanguageMarkers[up] ||
catalogNumber(tok) != ""
}
// parseTailZone resolves middle-zone tokens against the dictionaries.
// The first match wins for each field; catalogue numbers are recognized
// from parenthesized tokens, and anything unrecognized is appended to
// UnknownTokens so that it can be reviewed and curated into a dictionary.
func parseTailZone(zone []string, result *releaseparser.ParsedRelease) {
for _, tok := range zone {
up := strings.ToUpper(tok)
switch {
case Profile.FormatTokens[up]:
if result.Format == "" {
result.Format = tok
}
case Profile.SourceTokens[up]:
if result.Source == "" {
result.Source = tok
}
applyType(tok, result)
case Profile.BitDepthTokens[up]:
if result.BitDepth == "" {
result.BitDepth = tok
}
case Profile.ResolutionTokens[up]:
if result.Resolution == "" {
result.Resolution = tok
}
case Profile.EditionTokens[up]:
if result.Edition == "" {
result.Edition = tok
}
applyType(tok, result)
case Profile.LanguageMarkers[up]:
if result.Language == "" {
result.Language = tok
}
default:
if cat := catalogNumber(tok); cat != "" && result.CatalogNo == "" {
result.CatalogNo = cat
} else {
result.UnknownTokens = append(result.UnknownTokens, tok)
}
}
}
}
// applyType normalizes a matched edition/source token into ParsedRelease.Type
// via the types.txt variant map; first match wins. The token still fills its
// own raw field (Source stays raw).
func applyType(tok string, result *releaseparser.ParsedRelease) {
if base, ok := Profile.TypeVariants[strings.ToUpper(tok)]; ok && result.Type == "" {
result.Type = base
}
}
// overrideCoreLength returns the number of leading tokens covered by the
// override's Artist and Title (each may span several hyphen-separated
// tokens), falling back to the default 2-token core when the raw tokens
// do not reconstruct the override fields.
func overrideCoreLength(tokens []string, o releaseparser.Override) int {
for a := 1; a <= len(tokens); a++ {
if strings.Join(tokens[:a], "-") != o.Artist {
continue
}
for t := a + 1; t <= len(tokens); t++ {
if strings.Join(tokens[a:t], "-") == o.Title {
return t
}
}
}
return 2
}
// score computes a heuristic confidence in [0,1] for a parsed release based
// on how many core fields (group, year, format, source, artist) were resolved
// and whether any warnings or unknown tokens are present.
func score(result *releaseparser.ParsedRelease) {
score := 0.0
if result.Group != "" {
score += 0.25
}
if result.Year != 0 {
score += 0.20
}
if result.Format != "" {
score += 0.15
}
if result.Source != "" {
score += 0.15
}
if result.Artist != "" {
score += 0.15
}
if len(result.Warnings) == 0 {
score += 0.10
}
if len(result.UnknownTokens) > 0 {
score -= 0.10
}
if score > 1 {
score = 1
}
if score < 0 {
score = 0
}
result.Confidence = score
}
@@ -0,0 +1,494 @@
package groups
import (
"testing"
)
func TestParseFullTail(t *testing.T) {
r := Parse("Fane-Dragged_Through_the_Unending-24BIT-WEB-FLAC-2025-MOONBLOOD")
if r.Artist != "Fane" {
t.Errorf("Artist = %q, want %q", r.Artist, "Fane")
}
if r.Title != "Dragged_Through_the_Unending" {
t.Errorf("Title = %q, want %q", r.Title, "Dragged_Through_the_Unending")
}
if r.Year != 2025 || r.Format != "FLAC" || r.Source != "WEB" || r.BitDepth != "24BIT" {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
if r.Confidence < 0.99 {
t.Errorf("Confidence = %.2f, want ~1.0", r.Confidence)
}
}
func TestParseEditionAndDemo(t *testing.T) {
r := Parse("Satans_Grind-Rotten_Demo-DEMO-16BIT-WEB-FLAC-2016-MOONBLOOD")
if r.Edition != "DEMO" {
t.Errorf("Edition = %q, want %q", r.Edition, "DEMO")
}
if r.Artist != "Satans_Grind" || r.Title != "Rotten_Demo" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseArtistWithHyphen(t *testing.T) {
// Artist сам містить дефіс, але Ultra-Violence ще не в словнику:
// core контекстуальний — все між артистом і першим словниковим
// маркером (EP) є назвою, зібраною назад через "-".
r := Parse("Ultra-Violence-All_My_Life-EP-16BIT-WEB-FLAC-2026-MOONBLOOD")
if r.Artist != "Ultra" {
t.Errorf("Artist = %q, want Ultra", r.Artist)
}
if r.Title != "Violence" {
t.Errorf("Title = %q, want Violence", r.Title)
}
if r.Edition != "EP" || r.BitDepth != "16BIT" || r.Source != "WEB" || r.Year != 2026 {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 1 || r.UnknownTokens[0] != "All_My_Life" {
t.Errorf("UnknownTokens = %v, want [All_My_Life]", r.UnknownTokens)
}
}
func TestParseLanguageAndCDSource(t *testing.T) {
r := Parse("Burn_Thou_in_Effigy-Memorias_Postumas_do_Imperador-BR-CD-FLAC-2026-MOONBLOOD")
if r.Language != "BR" || r.Source != "CD" || r.Year != 2026 || r.Format != "FLAC" {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseNoYear(t *testing.T) {
// Without a year the tail still parses; group stays the last token.
r := Parse("Fane-Gallant_Shroud-EP-16BIT-WEB-FLAC-MOONBLOOD")
if r.Group != "MOONBLOOD" || r.Year != 0 {
t.Errorf("Group/Year = %q/%d", r.Group, r.Year)
}
if r.Edition != "EP" || r.Source != "WEB" || r.Format != "FLAC" {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Fane" || r.Title != "Gallant_Shroud" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
}
func TestParseResolutionSlot(t *testing.T) {
r := Parse("Ice_Nine_Kills-Last_Chance_To_Make_Amends-REMASTERED-24BIT-44KHZ-WEB-FLAC-2024-FLACCiD")
if r.Group != "FLACCiD" {
t.Errorf("Group = %q, want FLACCiD", r.Group)
}
if r.Resolution != "44KHZ" {
t.Errorf("Resolution = %q, want 44KHZ", r.Resolution)
}
if r.BitDepth != "24BIT" || r.Edition != "REMASTERED" || r.Source != "WEB" || r.Year != 2024 {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Ice_Nine_Kills" || r.Title != "Last_Chance_To_Make_Amends" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseVinylIsSource(t *testing.T) {
r := Parse("ROTH-Alles_Endet-DE-VINYL-FLAC-2026-TOTENKVLT")
if r.Language != "DE" {
t.Errorf("Language = %q, want DE", r.Language)
}
if r.Source != "VINYL" {
t.Errorf("Source = %q, want VINYL", r.Source)
}
if r.Edition != "" {
t.Errorf("Edition = %q, want empty (VINYL is source, not edition)", r.Edition)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseCaseInsensitiveDictionaries(t *testing.T) {
r := Parse("Artist-Some_Title-16bit-web-flac-2020-moonblood")
// Значення зберігаються в оригінальному регістрі (як у рядку),
// а звірка зі словниками йде регістронезалежно.
if r.BitDepth != "16bit" || r.Source != "web" || r.Format != "flac" {
t.Errorf("case-insensitive lookup failed: %+v", r)
}
if r.Artist != "Artist" || r.Title != "Some_Title" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
}
func TestParseUnknownTailTokenReported(t *testing.T) {
// Невідомий токен ПІСЛЯ останнього словникового маркера (за ним уже
// нічого відомого немає) → у UnknownTokens.
r := Parse("Artist-Some_Title-16BIT-WEB-FLAC-FOO-2025-MOONBLOOD")
if len(r.UnknownTokens) != 1 || r.UnknownTokens[0] != "FOO" {
t.Errorf("UnknownTokens = %v, want [FOO]", r.UnknownTokens)
}
if len(r.TitleExtra) != 0 {
t.Errorf("TitleExtra = %v, want empty", r.TitleExtra)
}
if r.Edition != "" || r.BitDepth != "16BIT" || r.Source != "WEB" || r.Year != 2025 {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Artist" || r.Title != "Some_Title" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
}
func TestParseContextualCoreAbsorbsTitleTail(t *testing.T) {
// Права межа core контекстуальна (перший словниковий маркер), але
// невідомі токени всередині core-зони — в UnknownTokens, не з'їдаються.
r := Parse("Artist-Some_Title-Part_Two-Part_Three-16BIT-WEB-FLAC-2025-MOONBLOOD")
if r.Artist != "Artist" {
t.Errorf("Artist = %q, want Artist", r.Artist)
}
if r.Title != "Some_Title" {
t.Errorf("Title = %q, want Some_Title", r.Title)
}
if r.BitDepth != "16BIT" || r.Source != "WEB" || r.Format != "FLAC" || r.Year != 2025 {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 2 || r.UnknownTokens[0] != "Part_Two" ||
r.UnknownTokens[1] != "Part_Three" {
t.Errorf("UnknownTokens = %v, want [Part_Two Part_Three]", r.UnknownTokens)
}
}
func TestParseCoreTokensNeverReported(t *testing.T) {
// Артист/тайтл ніколи не потрапляють в UnknownTokens.
r := Parse("Fane-Dragged_Through_the_Unending-24BIT-WEB-FLAC-2025-MOONBLOOD")
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
if r.Artist != "Fane" || r.Title != "Dragged_Through_the_Unending" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
}
func TestParseAnyGroupNoWarning(t *testing.T) {
cases := []string{
"Jetsex-Down_Down_Down-24BIT-WEB-FLAC-2025-FLACCiD",
"Sabaton-The_Last_Stand-24BIT-96KHZ-WEB-FLAC-2016-FLACCiD",
"Wind_Walkers-What_If_I_Break-16BIT-WEB-FLAC-2023-FLACCiD",
}
for _, line := range cases {
r := Parse(line)
if r.Group != "FLACCiD" {
t.Errorf("%s: Group = %q, want FLACCiD", line, r.Group)
}
if len(r.Warnings) != 0 {
t.Errorf("%s: Warnings = %v, want empty", line, r.Warnings)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("%s: UnknownTokens = %v, want empty", line, r.UnknownTokens)
}
if r.Year == 0 || r.Format != "FLAC" || r.Source != "WEB" || r.BitDepth == "" {
t.Errorf("%s: tail fields wrong: %+v", line, r)
}
}
}
func TestParseCommentExtracted(t *testing.T) {
r := Parse("Rain_City_Drive-Rain_City_Drive-DELUXE_EDITION-REPACK-16BIT-WEB-FLAC-2023-FLACCiD")
if r.Comment != "REPACK" {
t.Errorf("Comment = %q, want REPACK", r.Comment)
}
if r.Edition != "DELUXE_EDITION" || r.BitDepth != "16BIT" || r.Source != "WEB" || r.Year != 2023 {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Rain_City_Drive" || r.Title != "Rain_City_Drive" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 || len(r.Warnings) != 0 {
t.Errorf("UnknownTokens=%v Warnings=%v, want empty", r.UnknownTokens, r.Warnings)
}
}
func TestParseNumericArtistNotAYear(t *testing.T) {
// [0]=Artist — навіть якщо це число-рік (1914, 1928), а не "рік" токен.
for _, line := range []string{
"1914-Eschatology_of_War-24BIT-WEB-FLAC-2015-MOONBLOOD",
"1928-Lappel_du_Spectre-FR-CD-FLAC-2025-MOONBLOOD",
} {
r := Parse(line)
if r.Artist != "1914" && r.Artist != "1928" {
t.Errorf("%s: Artist = %q, want numeric artist", line, r.Artist)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("%s: UnknownTokens = %v, want empty", line, r.UnknownTokens)
}
}
}
func TestParseCatalogNumber(t *testing.T) {
r := Parse("Antidot-Beyond_Trance_And_Time-(GLOBAL006)-REPACK-EP-FLAC-2026-AFTERHOUR")
if r.CatalogNo != "GLOBAL006" {
t.Errorf("CatalogNo = %q, want GLOBAL006", r.CatalogNo)
}
if r.Comment != "REPACK" || r.Edition != "EP" || r.Format != "FLAC" || r.Year != 2026 {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Antidot" || r.Title != "Beyond_Trance_And_Time" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseCatalogNumberWithDashesInside(t *testing.T) {
r := Parse("Murder_Intentions-Excessive_Display_Of_Human_Nature-(BRC30-001-CD-RRR108)-CD-FLAC-2018-86D")
if r.CatalogNo != "BRC30-001-CD-RRR108" {
t.Errorf("CatalogNo = %q, want BRC30-001-CD-RRR108", r.CatalogNo)
}
if r.Source != "CD" || r.Format != "FLAC" || r.Year != 2018 || r.Group != "86D" {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Murder_Intentions" || r.Title != "Excessive_Display_Of_Human_Nature" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseCatalogNumberWithDotAndRoman(t *testing.T) {
// Катало у дужках у середній зоні — незалежно від вмісту
// (крапка, римські, букви, дефіси): все валідне.
cases := []struct{ line, catno string }{
{"Nervosa-Perpetual_Chaos-(DYN.3140-2)-LIMITED_EDITION-CD-FLAC-2021-MOONBLOOD", "DYN.3140-2"},
{"Black_Ceremonial_Kult-Crowned_in_Chaos-(GOWP_DCCCLXXXIX)-CDEP-FLAC-2021-MOONBLOOD", "GOWP_DCCCLXXXIX"},
}
for _, c := range cases {
r := Parse(c.line)
if r.CatalogNo != c.catno {
t.Errorf("CatalogNo = %q, want %q (release: %s)", r.CatalogNo, c.catno, c.line)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty (release: %s)", r.UnknownTokens, c.line)
}
}
}
func TestParseSplitKeepsParensAndSplitsTitles(t *testing.T) {
// Evolution — не словниковий токен у core-зоні → в UnknownTokens,
// не з'їдається. (SMG409) — катало, перший відомий маркер.
r := Parse("Ferosity-De-Evolution-(SMG409)-CD-FLAC-2025-86D")
if r.CatalogNo != "SMG409" {
t.Errorf("CatalogNo = %q, want SMG409", r.CatalogNo)
}
if r.Artist != "Ferosity" || r.Title != "De" {
t.Errorf("Artist/Title = %q/%q, want Ferosity / De", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 1 || r.UnknownTokens[0] != "Evolution" {
t.Errorf("UnknownTokens = %v, want [Evolution]", r.UnknownTokens)
}
}
func TestParseLanguageRequiresKnownNeighbor(t *testing.T) {
// Мовний маркер валідний, коли праворуч — відомий хвостовий маркер.
r := Parse("ROTH-Alles_Endet-DE-VINYL-FLAC-2026-TOTENKVLT")
if r.Language != "DE" || r.Source != "VINYL" {
t.Errorf("Language/Source = %q/%q, want DE/VINYL", r.Language, r.Source)
}
}
func TestParseLanguageBetweenTitleAndFormat(t *testing.T) {
// A: мовний маркер у зоні між title і format — валідний,
// все ліворуч від нього — artist+title, нічого в unknown.
r := Parse("Kollektiver_Brechreiz-Heilig_Scheint_Scheinheilig-(NG207)-DE-CD-FLAC-2009-SHELTER")
if r.CatalogNo != "NG207" {
t.Errorf("CatalogNo = %q, want NG207", r.CatalogNo)
}
if r.Language != "DE" || r.Source != "CD" || r.Format != "FLAC" || r.Year != 2009 {
t.Errorf("tail fields wrong: %+v", r)
}
if r.Artist != "Kollektiver_Brechreiz" || r.Title != "Heilig_Scheint_Scheinheilig" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 0 || len(r.Warnings) != 0 {
t.Errorf("UnknownTokens=%v Warnings=%v, want empty", r.UnknownTokens, r.Warnings)
}
}
func TestParseLanguageNeverBeforeFormat(t *testing.T) {
// DE — словниковий токен у середині (language.txt), позиція неважлива:
// Language=DE.
r := Parse("Some_Artist-Some_Song-DE-FLAC-2020-SOMEGROUP")
if r.Language != "DE" {
t.Errorf("Language = %q, want DE", r.Language)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
if r.Title != "Some_Song" {
t.Errorf("Title = %q, want Some_Song", r.Title)
}
}
func TestParseCompoundEditionRecognized(t *testing.T) {
// REMASTERED_REISSUE — складений маркер у dicts/edition.txt, розпізнається
// як edition і не йде в UnknownTokens.
r := Parse("Ominous_Scriptures-Incarnation_of_the_Unheavenly-(LSR07)-REMASTERED_REISSUE-CD-FLAC-2021-86D")
if r.CatalogNo != "LSR07" {
t.Errorf("CatalogNo = %q, want LSR07", r.CatalogNo)
}
if r.Edition != "REMASTERED_REISSUE" {
t.Errorf("Edition = %q, want REMASTERED_REISSUE", r.Edition)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
if r.Artist != "Ominous_Scriptures" || r.Title != "Incarnation_of_the_Unheavenly" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if r.Source != "CD" || r.Format != "FLAC" || r.Year != 2021 {
t.Errorf("tail fields wrong: %+v", r)
}
}
func TestParseMixedCaseTitleNeverReported(t *testing.T) {
// Хвіст назви (_Sky_Burial) — не словник, але він ДО першого маркера
// (катало (RTM103)) → частина назви, не UnknownTokens.
r := Parse("Enthrallment-Against_the_Will_to_Live_-_Sky_Burial-(RTM103)-CD-FLAC-2021-86D")
if r.Artist != "Enthrallment" || r.Title != "Against_the_Will_to_Live_" {
t.Errorf("Artist/Title = %q/%q", r.Artist, r.Title)
}
if len(r.UnknownTokens) != 1 || r.UnknownTokens[0] != "_Sky_Burial" {
t.Errorf("UnknownTokens = %v, want [_Sky_Burial]", r.UnknownTokens)
}
if r.CatalogNo != "RTM103" {
t.Errorf("CatalogNo = %q, want RTM103", r.CatalogNo)
}
}
func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
func TestParseParenthesizedTitleNotCatNo(t *testing.T) {
// Дужки в тайтлі при WEB/BitDepth — не каталономер, лишається в тайтлі.
r := Parse("The_Amazons-21st_Century_Fiction_(Epilogue)-16BIT-WEB-FLAC-2026-ENViED")
if r.CatalogNo != "" {
t.Errorf("CatalogNo = %q, want empty", r.CatalogNo)
}
if r.Title != "21st_Century_Fiction_(Epilogue)" {
t.Errorf("Title = %q, want 21st_Century_Fiction_(Epilogue)", r.Title)
}
if r.Source != "WEB" || r.BitDepth != "16BIT" || r.Year != 2026 {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 {
t.Errorf("UnknownTokens = %v, want empty", r.UnknownTokens)
}
}
func TestParseArtistDictionaryCoversReleases(t *testing.T) {
// artists.txt: "Iku-Turso" у словнику → Artist = 2 токени,
// Title = [2], хвостова зона зсувається і парситься як завжди.
r := Parse("Iku-Turso-Ikuinen_Steelfest_MMXXII_Live-24BIT-WEB-FLAC-2016-MOONBLOOD")
if r.Artist != "Iku-Turso" {
t.Errorf("Artist = %q, want Iku-Turso", r.Artist)
}
if r.Title != "Ikuinen_Steelfest_MMXXII_Live" {
t.Errorf("Title = %q, want Ikuinen_Steelfest_MMXXII_Live", r.Title)
}
if r.BitDepth != "24BIT" || r.Source != "WEB" || r.Format != "FLAC" || r.Year != 2016 {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 || len(r.TitleExtra) != 0 || len(r.Warnings) != 0 {
t.Errorf("Unknown=%v Extra=%v Warnings=%v, want empty",
r.UnknownTokens, r.TitleExtra, r.Warnings)
}
}
func TestParseArtistDictionaryMultiTokenArtist(t *testing.T) {
// artists.txt: "Shadu-Nar-Mattaru" — артист з ТРЬОХ токенів.
// Найдовший словниковий префікс виграє: Artist = 3 токени,
// Title = наступний токен, хвіст парситься як завжди.
r := Parse("Shadu-Nar-Mattaru-On_Pathways_Macabre-24BIT-WEB-FLAC-2023-MOONBLOOD")
if r.Artist != "Shadu-Nar-Mattaru" {
t.Errorf("Artist = %q, want Shadu-Nar-Mattaru", r.Artist)
}
if r.Title != "On_Pathways_Macabre" {
t.Errorf("Title = %q, want On_Pathways_Macabre", r.Title)
}
if r.BitDepth != "24BIT" || r.Source != "WEB" || r.Format != "FLAC" || r.Year != 2023 {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 || len(r.TitleExtra) != 0 || len(r.Warnings) != 0 {
t.Errorf("Unknown=%v Extra=%v Warnings=%v, want empty",
r.UnknownTokens, r.TitleExtra, r.Warnings)
}
}
func TestParseOverrideWinsOverPositionalModel(t *testing.T) {
// overrides.txt (найвищий пріоритет): точний lookup по Raw
// фіксує Artist/Title; хвіст парситься як завжди.
raw := "Milicia_Oscura-Anti-Cristo-ES-16BIT-WEB-FLAC-2013-MOONBLOOD"
r := Parse(raw)
if r.Artist != "Milicia_Oscura" {
t.Errorf("Artist = %q, want Milicia_Oscura", r.Artist)
}
if r.Title != "Anti-Cristo" {
t.Errorf("Title = %q, want Anti-Cristo", r.Title)
}
if r.Language != "ES" || r.BitDepth != "16BIT" || r.Source != "WEB" ||
r.Format != "FLAC" || r.Year != 2013 || r.Group != "MOONBLOOD" {
t.Errorf("tail fields wrong: %+v", r)
}
if len(r.UnknownTokens) != 0 || len(r.TitleExtra) != 0 || len(r.Warnings) != 0 {
t.Errorf("Unknown=%v Extra=%v Warnings=%v, want empty",
r.UnknownTokens, r.TitleExtra, r.Warnings)
}
}
func TestParseTypeNormalizedSourceStaysRaw(t *testing.T) {
// CDEP у source.txt: Source лишається сирим, Type нормалізується в EP.
r := Parse("Black_Ceremonial_Kult-Crowned_in_Chaos-CDEP-FLAC-2021-MOONBLOOD")
if r.Type != "EP" {
t.Errorf("Type = %q, want EP", r.Type)
}
if r.Source != "CDEP" {
t.Errorf("Source = %q, want raw CDEP", r.Source)
}
}
func TestParseTypeFirstMatchWins(t *testing.T) {
// REMASTERED_EP → EP; наступний типовий токен не перезаписує.
r := Parse("Some_Artist-Some_Title-REMASTERED_EP-SINGLE-FLAC-2021-MOONBLOOD")
if r.Type != "EP" {
t.Errorf("Type = %q, want EP (first match wins)", r.Type)
}
if r.Edition != "REMASTERED_EP" {
t.Errorf("Edition = %q, want REMASTERED_EP", r.Edition)
}
}
func TestAnalyzeTypeDistribution(t *testing.T) {
stats := AnalyzeTitles([]string{
"Some_Artist-Some_Title-CDEP-FLAC-2021-MOONBLOOD",
"Other_Artist-Other_Title-EP-FLAC-2022-MOONBLOOD",
"Third_Artist-Third_Title-SPLIT-FLAC-2023-MOONBLOOD",
})
if stats.TypeDistribution["EP"] != 2 {
t.Errorf("TypeDistribution[EP] = %d, want 2", stats.TypeDistribution["EP"])
}
if stats.TypeDistribution["Split"] != 1 {
t.Errorf("TypeDistribution[Split] = %d, want 1", stats.TypeDistribution["Split"])
}
}
+104
View File
@@ -0,0 +1,104 @@
package groups
import (
"embed"
"strings"
"git.kplus.net.ua/yevhen/resource-scraper/internal/releaseparser"
)
// The token dictionaries live under resources/dicts/*.txt — one token (in any
// case) per line; blank lines and lines starting with "#" are ignored. They
// can be edited without recompiling the parsing logic: go:embed pulls them
// into the binary at build time.
//go:embed resources/dicts/*.txt
var dictFS embed.FS
func loadDict(name string) map[string]bool {
data, err := dictFS.ReadFile("resources/dicts/" + name + ".txt")
if err != nil {
panic("release parser: cannot load dictionary " + name + ": " + err.Error())
}
out := make(map[string]bool)
for line := range strings.SplitSeq(string(data), "\n") {
line = strings.ToUpper(strings.TrimSpace(line))
if line == "" || strings.HasPrefix(line, "#") {
continue
}
out[line] = true
}
return out
}
// loadOverrides parses the override table: "<raw line> => artist | title".
func loadOverrides(name string) map[string]releaseparser.Override {
data, err := dictFS.ReadFile("resources/dicts/" + name + ".txt")
if err != nil {
panic("release parser: cannot load dictionary " + name + ": " + err.Error())
}
out := make(map[string]releaseparser.Override)
for line := range strings.SplitSeq(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
head, rest, ok := strings.Cut(line, "=>")
if !ok {
continue
}
artist, title, ok := strings.Cut(strings.TrimSpace(rest), "|")
if !ok {
continue
}
out[strings.TrimSpace(head)] = releaseparser.Override{
Artist: strings.TrimSpace(artist),
Title: strings.TrimSpace(title),
}
}
return out
}
// loadTypeVariants parses types.txt: "<Base> = variant, variant, ..." and
// returns variant (uppercased) -> base as written in the file.
func loadTypeVariants(name string) map[string]string {
data, err := dictFS.ReadFile("resources/dicts/" + name + ".txt")
if err != nil {
panic("release parser: cannot load dictionary " + name + ": " + err.Error())
}
out := make(map[string]string)
for line := range strings.SplitSeq(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
base, rest, ok := strings.Cut(line, "=")
if !ok {
continue
}
base = strings.TrimSpace(base)
for v := range strings.SplitSeq(rest, ",") {
v = strings.ToUpper(strings.TrimSpace(v))
if v != "" {
out[v] = base
}
}
}
return out
}
// Profile holds the shared dictionaries for the release parser. The parser is
// group-agnostic: the group name is read from each release line, never from
// this profile, so there is a single set of dictionaries covering all groups.
var Profile = releaseparser.GroupProfile{
FormatTokens: loadDict("format"),
ArtistTokens: loadDict("artists"),
TypeVariants: loadTypeVariants("types"),
SourceTokens: loadDict("source"),
BitDepthTokens: loadDict("bitdepth"),
ResolutionTokens: loadDict("resolution"),
LanguageMarkers: loadDict("language"),
EditionTokens: loadDict("edition"),
Overrides: loadOverrides("overrides"),
CommentTokens: loadDict("comment"),
}
+63
View File
@@ -0,0 +1,63 @@
package releaseparser
type Override struct {
Artist string
Title string
}
type GroupProfile struct {
ArtistTokens map[string]bool // hyphenated artist whitelist
TypeVariants map[string]string // variant (upper) -> normalized Type
Overrides map[string]Override
FormatTokens map[string]bool
SourceTokens map[string]bool
BitDepthTokens map[string]bool
ResolutionTokens map[string]bool
LanguageMarkers map[string]bool
EditionTokens map[string]bool
CommentTokens map[string]bool
}
type ParsedRelease struct {
Raw string
Group string
Artist string
Year int
Title string
TitleExtra []string
Type string
Source string
Edition string
CatalogNo string
Language string
BitDepth string
Resolution string
Comment string
Format string
Core string
Override Override
UnknownTokens []string
Warnings []string
Confidence float64
}
type UnknownTokenStats struct {
Token string
Count int
Releases []string
}
type GroupStats struct {
Total int
TypeDistribution map[string]int
YearDistribution map[int]int
SourceDistribution map[string]int
BitDepthDistribution map[string]int
ResolutionDistribution map[string]int
LanguageDistribution map[string]int
EditionDistribution map[string]int
CommentDistribution map[string]int
UnknownTokens []UnknownTokenStats
WarningStats []UnknownTokenStats
AvgConfidence float64
}
-59
View File
@@ -1,59 +0,0 @@
scope prescene
FLOAT: 33.000000, 12.000000, 3
false
====================== 1 ==============================
{Id:825647 Type:prescene TypeId:4313240 Title:Phaethon-Wielder_Of_The_Steel-24BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/phaethon-wielder_of_the_steel-24bit-web-flac-2024-moonblood/ Created:2024-09-10 15:09:25 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 2 ==============================
{Id:825648 Type:prescene TypeId:4313238 Title:Helvetets_Port-Warlords-24BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/helvetets_port-warlords-24bit-web-flac-2024-moonblood/ Created:2024-09-10 15:08:33 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 4 ==============================
{Id:825649 Type:prescene TypeId:4313236 Title:Circle_of_Ouroborus-Ajattoman_Uni-FI-16BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/circle_of_ouroborus-ajattoman_uni-fi-16bit-web-flac-2024-moonblood/ Created:2024-09-10 15:07:20 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 5 ==============================
{Id:825650 Type:prescene TypeId:4313234 Title:Blitzkrieg-Blitzkrieg-24BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/blitzkrieg-blitzkrieg-24bit-web-flac-2024-moonblood/ Created:2024-09-10 15:06:36 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 6 ==============================
{Id:825651 Type:prescene TypeId:4313233 Title:Antioch-Onward_With_Obliteration-EP-16BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/antioch-onward_with_obliteration-ep-16bit-web-flac-2024-moonblood/ Created:2024-09-10 15:05:27 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 8 ==============================
{Id:825631 Type:prescene TypeId:4313221 Title:Wretchgod-Suffering_Upon_Suffering-EP-24BIT-WEB-FLAC-2024-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/wretchgod-suffering_upon_suffering-ep-24bit-web-flac-2024-moonblood/ Created:2024-09-10 14:07:36 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 9 ==============================
{Id:825632 Type:prescene TypeId:4313220 Title:Wretchgod-Stygian_Blood_Ritual-EP-16BIT-WEB-FLAC-2022-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/wretchgod-stygian_blood_ritual-ep-16bit-web-flac-2022-moonblood/ Created:2024-09-10 14:06:51 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 11 ==============================
{Id:825633 Type:prescene TypeId:4313188 Title:1349-The_Infernal_Pathway-REPACK-Deluxe_Edition-16BIT-WEB-FLAC-2019-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-the_infernal_pathway-repack-deluxe_edition-16bit-web-flac-2019-moonblood/ Created:2024-09-10 13:34:09 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 15 ==============================
{Id:825634 Type:prescene TypeId:4313177 Title:1349-The_Infernal_Pathway-Deluxe_Edition-16BIT-WEB-FLAC-2019-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-the_infernal_pathway-deluxe_edition-16bit-web-flac-2019-moonblood/ Created:2024-09-10 13:17:42 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 16 ==============================
{Id:825635 Type:prescene TypeId:4313175 Title:1349-Revelations_of_the_Black_Flame-16BIT-WEB-FLAC-2009-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-revelations_of_the_black_flame-16bit-web-flac-2009-moonblood/ Created:2024-09-10 13:16:03 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 17 ==============================
{Id:825636 Type:prescene TypeId:4313173 Title:1349-Massive_Cauldron_of_Chaos-16BIT-WEB-FLAC-2014-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-massive_cauldron_of_chaos-16bit-web-flac-2014-moonblood/ Created:2024-09-10 13:15:22 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 19 ==============================
{Id:825637 Type:prescene TypeId:4313171 Title:1349-Liberation-16BIT-WEB-FLAC-2003-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-liberation-16bit-web-flac-2003-moonblood/ Created:2024-09-10 13:14:38 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 21 ==============================
{Id:825638 Type:prescene TypeId:4313168 Title:1349-Hellfire-16BIT-WEB-FLAC-2005-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-hellfire-16bit-web-flac-2005-moonblood/ Created:2024-09-10 13:13:29 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 22 ==============================
{Id:825639 Type:prescene TypeId:4313166 Title:1349-Demonoir-REISSUE-16BIT-WEB-FLAC-2011-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-demonoir-reissue-16bit-web-flac-2011-moonblood/ Created:2024-09-10 13:12:45 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 23 ==============================
{Id:825640 Type:prescene TypeId:4313164 Title:1349-Beyond_the_Apocalypse-16BIT-WEB-FLAC-2004-MOONBLOOD TypeSubsectionId:0 Releaser:MOONBLOOD ExSource:https://scnlog.me/music/1349-beyond_the_apocalypse-16bit-web-flac-2004-moonblood/ Created:2024-09-10 13:12:00 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
Sleeping... 0
false
====================== 1 ==============================
{Id:825652 Type:prescene TypeId:4300530 Title:Zuckerbaby--Zuckerbaby-CD-FLAC-1997-MEOWZiK TypeSubsectionId:0 Releaser:MEOWZiK ExSource:https://scnlog.me/music/zuckerbaby-zuckerbaby-cd-flac-1997-meowzik/ Created:2024-08-31 18:17:03 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 2 ==============================
{Id:825653 Type:prescene TypeId:4300529 Title:Silmarils--Vegas_76-FR-CD-FLAC-2000-MEOWZiK TypeSubsectionId:0 Releaser:MEOWZiK ExSource:https://scnlog.me/music/silmarils-vegas_76-fr-cd-flac-2000-meowzik/ Created:2024-08-31 18:16:29 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 3 ==============================
{Id:825654 Type:prescene TypeId:4300527 Title:Michael_Kaeshammer--Days_Like_These-CD-FLAC-2007-MEOWZiK TypeSubsectionId:0 Releaser:MEOWZiK ExSource:https://scnlog.me/music/michael_kaeshammer-days_like_these-cd-flac-2007-meowzik/ Created:2024-08-31 18:14:25 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 4 ==============================
{Id:825655 Type:prescene TypeId:4300521 Title:Butch_Walker--Letters-CD-FLAC-2004-MEOWZiK TypeSubsectionId:0 Releaser:MEOWZiK ExSource:https://scnlog.me/music/butch_walker-letters-cd-flac-2004-meowzik/ Created:2024-08-31 18:09:22 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 16 ==============================
{Id:825656 Type:prescene TypeId:4300320 Title:Embodiment_Elimination-Metamorphosis_Incarnate_Through_Genetic_Devastation-(ISR071-24)-CD-FLAC-2024-86D TypeSubsectionId:0 Releaser:86D ExSource:https://scnlog.me/music/embodiment_elimination-metamorphosis_incarnate_through_genetic_devastation-isr071-24-cd-flac-2024-86d/ Created:2024-08-31 11:19:45 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 19 ==============================
{Id:825657 Type:prescene TypeId:4300309 Title:Putrified_Degradation-Manufractured-(ISR072-24)-CD-FLAC-2024-86D TypeSubsectionId:0 Releaser:86D ExSource:https://scnlog.me/music/putrified_degradation-manufractured-isr072-24-cd-flac-2024-86d/ Created:2024-08-31 10:45:04 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 20 ==============================
{Id:825658 Type:prescene TypeId:4300307 Title:Post_Mortal_Possession-The_Dead_Space_Between_the_Stars-(LORD091)-CD-FLAC-2024-86D TypeSubsectionId:0 Releaser:86D ExSource:https://scnlog.me/music/post_mortal_possession-the_dead_space_between_the_stars-lord091-cd-flac-2024-86d/ Created:2024-08-31 10:43:09 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 21 ==============================
{Id:825659 Type:prescene TypeId:4300306 Title:Invirulant-Necrothuggin-(ISR070-24)-CDEP-FLAC-2024-86D TypeSubsectionId:0 Releaser:86D ExSource:https://scnlog.me/music/invirulant-necrothuggin-isr070-24-cdep-flac-2024-86d/ Created:2024-08-31 10:41:04 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
====================== 22 ==============================
{Id:825660 Type:prescene TypeId:4300304 Title:Dead_and_Dripping-Disillusioned_by_Excessive_Human_Consumption-(LORD092)-DEMO_REISSUE-CD-FLAC-2024-86D TypeSubsectionId:0 Releaser:86D ExSource:https://scnlog.me/music/dead_and_dripping-disillusioned_by_excessive_human_consumption-lord092-demo_reissue-cd-flac-2024-86d/ Created:2024-08-31 10:38:13 Modified: Contents: A: H: Fingerprint: FsFingerprint: Vid:0 G: ExternalSourcesCol: Error:<nil>}
Sleeping... 1
scope: prescene
[1 45]
init console console: ps
Execution took 12.6276275s
+29 -11
View File
@@ -1,31 +1,49 @@
package handler
import (
"reflect"
"log" // Додано імпорт log
"github.com/iancoleman/strcase"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/pkg/service"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/service"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
// consoleCommandHandler визначає тип для функцій-обробників консольних команд.
type consoleCommandHandler func() string
type Handler struct {
services *service.Service
// consoleCommands зберігає мапу назв команд до відповідних функцій-обробників.
consoleCommands map[string]consoleCommandHandler
}
func New(services *service.Service) *Handler {
return &Handler{services: services}
h := &Handler{services: services}
// Ініціалізація мапи команд.
h.consoleCommands = map[string]consoleCommandHandler{
"console": h.Console,
"info": h.Info,
"metal-archives": h.MetalArchives,
"prescene": h.Prescene,
"rutracker": h.Rutracker,
"stb": h.STB,
//"web": h.Web,
}
return h
}
func (h *Handler) InitConsole() string {
methodName := strcase.ToCamel(viper.GetString(constant.FlagScopeEnable))
commandName := viper.GetString(constant.FlagScopeEnable)
immutable := reflect.ValueOf(h)
method := immutable.MethodByName(methodName)
v := method.Call(nil)
return methodName + " launched, " + v[0].String() + "\n"
if cmdFunc, ok := h.consoleCommands[commandName]; ok {
return commandName + " launched, " + cmdFunc() + "\n"
} else {
log.Printf("Error: unknown console command '%s'", commandName)
return "Error: unknown console command"
}
}
func (h *Handler) InitRoutes() string {
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"github.com/mewkiz/flac"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func (h *Handler) Info() string {
+14 -14
View File
@@ -9,33 +9,33 @@ import (
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func (h *Handler) Prescene() string {
pagesToScrape := []string{"1"}
currMinute := time.Now().Minute()
levels := viper.GetInt(constant.CfgKeyLevelsToScrape)
initPage := strconv.Itoa(currMinute/(60/levels) + 1)
pagesToScrape := []string{initPage}
if levels > 1 {
var q = (60.0 / 100.0) * (100.0 / viper.GetFloat64(constant.CfgKeyLevelsToScrape))
minute := float64(time.Now().Minute())
if minute < 1 {
minute = 1
}
var q = (60.0 / 100.0) * (100.0 / float64(levels))
minute := float64(currMinute + 1)
fmt.Printf("FLOAT: %f, %f, %v\n", minute, q, math.Ceil(minute/q))
hour := time.Now().Hour()
if hour == 1 {
hour = 25
} else if hour == 0 {
hour = 1
}
hour := time.Now().Hour() + levels
page := hour * int(math.Ceil(minute/q))
pagesToScrape = append(pagesToScrape, strconv.Itoa(page))
}
_, err := h.services.Prescene.GetPage(pagesToScrape)
_, err := h.services.Prescene.GetSrrDb()
if err != nil {
slog.Error("error occurred while getting page: ", "err", err.Error())
}
_, err = h.services.Prescene.GetPage(pagesToScrape)
if err != nil {
slog.Error("error occurred while getting page: ", "err", err.Error())
}
-2
View File
@@ -17,7 +17,5 @@ func (h *Handler) Rutracker() string {
slog.Error("error occurred while getting topic: ", "err", err.Error())
}
//fmt.Printf("%+v", rt)
return "rt"
}
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
)
func (h *Handler) STB() string {
+32 -15
View File
@@ -10,21 +10,33 @@ import (
"strings"
"time"
"github.com/davecgh/go-spew/spew"
"github.com/dromara/carbon/v2"
"github.com/go-shiori/dom"
"github.com/golang-module/carbon/v2"
"github.com/jmoiron/sqlx"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/helper/parser"
"git.amok.space/yevhen/resource-scraper/pkg/repository/table"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/helper/parser"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/repository/table"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type Prescene struct {
db *sqlx.DB
}
func (s *Prescene) GetSrrDb() ([]model.ExternalSources, error) {
entries := make([]model.ExternalSources, 0)
now := time.Now()
yesterday := now.Add(-24 * time.Hour)
todayStr := now.Format("2006-01-02")
yesterdayStr := yesterday.Format("2006-01-02")
spew.Dump(entries, todayStr, yesterdayStr)
return []model.ExternalSources{}, nil
}
func NewPresceneRepository(db *sqlx.DB) *Prescene {
return &Prescene{db: db}
}
@@ -44,7 +56,7 @@ func (s *Prescene) GetPage(pageNumbers []string) ([]model.ExternalSources, error
endpoint += fmt.Sprintf(viper.GetString(constant.CfgKeyEndpointNext), t)
}
//doc, err := parser.HTMLSourceFromURL("https://mdb.amok.space/$/scnlog.html")
//doc, err := parser.HTMLSourceFromURL("https://mdb.kplus.net.ua /$/scnlog.html")
if result, err := parseUrl(endpoint, s.db); err == nil {
entries = append(entries, result...)
} else {
@@ -73,16 +85,25 @@ func parseUrl(endpoint string, db *sqlx.DB) ([]model.ExternalSources, error) {
}
var validID = regexp.MustCompile(`-\d+\/$`)
for i, item := range dom.QuerySelectorAll(doc, ".post.type-post.category-flac.category-music") {
for i, item := range dom.QuerySelectorAll(doc, "li.row") {
var es model.ExternalSources
columns := []string{"`type`", "type_id", "title", "eXsource", "releaser", "created"}
// TypeId тепер напряму з data-id на <li>, без regex по class
idAttr := dom.GetAttribute(item, "data-id")
es.TypeId, _ = strconv.Atoi(idAttr)
title := dom.QuerySelector(item, ".title")
if title != nil {
anchor := dom.QuerySelector(title, "h1 > a")
anchor := dom.QuerySelector(title, "a")
if anchor != nil {
es.Type = constant.ScopePrescene
es.Title = dom.GetAttribute(anchor, "title")
titleSpan := dom.QuerySelector(anchor, "span.title-start")
if titleSpan != nil {
es.Title = dom.TextContent(titleSpan)
}
if es.Title == "Auto Draft" {
slog.Info("Skipped", "title", es.Title)
continue
@@ -111,17 +132,13 @@ func parseUrl(endpoint string, db *sqlx.DB) ([]model.ExternalSources, error) {
continue
}
localtime := dom.QuerySelector(title, "small > span.localtime")
localtime := dom.QuerySelector(item, "span.localtime")
if localtime != nil {
lc := dom.GetAttribute(localtime, "data-lttime")
es.Created = carbon.Parse(lc)
es.Created = carbon.Parse(lc, "Europe/Kyiv")
}
}
cls := dom.GetAttribute(item, "class")
pattern := regexp.MustCompile(`(?s)^post-(\d+)\spost`)
es.TypeId, _ = strconv.Atoi(pattern.FindStringSubmatch(cls)[1])
esModel := table.ExternalSources{Columns: columns}
entry := esModel.InsertOnDuplicate(es, db)
entries = append(entries, entry)
+1 -1
View File
@@ -3,7 +3,7 @@ package repository
import (
"github.com/jmoiron/sqlx"
"git.amok.space/yevhen/resource-scraper/types/interface"
"git.kplus.net.ua/yevhen/resource-scraper/types/interface"
)
type Repository struct {
+36 -15
View File
@@ -9,14 +9,14 @@ import (
"net/url"
"strconv"
"github.com/golang-module/carbon/v2"
"github.com/dromara/carbon/v2"
"github.com/jmoiron/sqlx"
"github.com/spf13/viper"
"git.amok.space/yevhen/resource-scraper/pkg/repository/table"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.amok.space/yevhen/resource-scraper/types/resource"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/repository/table"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/types/resource"
)
type Rutracker struct {
@@ -29,6 +29,7 @@ func NewRutrackerRepository(db *sqlx.DB) *Rutracker {
func (s *Rutracker) GetTopic(topics []string) ([]model.ExternalSources, error) {
endpoint := viper.GetString(constant.CfgKeyEndpoint)
entries := make([]model.ExternalSources, 0)
columns := []string{"`type`", "type_id", "title", "type_subsection_id", "releaser", "created"}
@@ -41,19 +42,17 @@ func (s *Rutracker) GetTopic(topics []string) ([]model.ExternalSources, error) {
for _, e := range topic.Entry {
var es model.ExternalSources
u, _ := url.Parse(e.Link.Href)
es.Type = constant.ScopeRuTracker
es.TypeId, _ = strconv.Atoi(u.Query().Get("t"))
es.TypeId = topicId(e)
es.Title = e.Title
es.TypeSubsectionId, _ = strconv.Atoi(t)
es.Releaser = e.Author.Name
es.Created = carbon.Parse(e.Updated)
es.Created = carbon.Parse(e.Updated, "Europe/Kyiv")
esModel := table.ExternalSources{Columns: columns}
entry := esModel.InsertOnDuplicate(es, s.db)
entries = append(entries, entry)
//fmt.Printf("%+v\n\n\n", entry)
}
}
@@ -63,22 +62,44 @@ func (s *Rutracker) GetTopic(topics []string) ([]model.ExternalSources, error) {
func fetch(endpoint string) (*resource.RutrackerAtomTopic, error) {
resp, err := http.Get(endpoint)
if err != nil {
slog.Error("couldn't fetch data", endpoint, err.Error())
slog.Error("couldn't fetch data", "endpoint", endpoint, "err", err.Error())
return nil, err
}
defer func(Body io.ReadCloser) {
err = Body.Close()
if err != nil {
fmt.Println("Body.Close")
if cerr := Body.Close(); cerr != nil {
slog.Error("couldn't close response body", "endpoint", endpoint, "err", cerr.Error())
}
}(resp.Body)
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("unexpected status code %d from %s", resp.StatusCode, endpoint)
}
topic := &resource.RutrackerAtomTopic{}
if err = xml.NewDecoder(resp.Body).Decode(topic); err != nil {
slog.Error("couldn't decode feed", "endpoint", endpoint, "err", err.Error())
return nil, err
}
return topic, nil
}
// topicId resolves the rutracker topic id preferring the page link query (?t=...),
// falling back to the entry id tag (tag:rto.feed,...:/t/5702203).
func topicId(e resource.RutrackerAtomEntry) int {
if link, ok := e.PageLink(); ok {
if u, err := url.Parse(link.Href); err == nil {
if id, err := strconv.Atoi(u.Query().Get("t")); err == nil {
return id
}
}
}
id, err := e.TopicId()
if err != nil {
slog.Error("couldn't resolve topic id", "entry_id", e.ID, "err", err.Error())
return 0
}
return id
}
+15 -13
View File
@@ -7,19 +7,19 @@ import (
"strconv"
"strings"
"github.com/dromara/carbon/v2"
"github.com/emersion/go-imap/v2"
"github.com/go-shiori/dom"
"github.com/golang-module/carbon/v2"
"github.com/jmoiron/sqlx"
"github.com/spf13/viper"
"golang.org/x/net/html"
"git.amok.space/yevhen/resource-scraper/helper/parser"
"git.amok.space/yevhen/resource-scraper/helper/sugar"
"git.amok.space/yevhen/resource-scraper/internal/mail"
_table "git.amok.space/yevhen/resource-scraper/pkg/repository/table"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/helper/parser"
"git.kplus.net.ua/yevhen/resource-scraper/helper/sugar"
"git.kplus.net.ua/yevhen/resource-scraper/internal/mail"
_table "git.kplus.net.ua/yevhen/resource-scraper/pkg/repository/table"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type ShareTheBrutality struct {
@@ -44,12 +44,10 @@ func (s *ShareTheBrutality) GetMail(email string) ([]model.ExternalSources, *mai
searchCriteria := &imap.SearchCriteria{
Text: viper.GetStringSlice(criteria),
}
//now := carbon.Now()
s.EmailService.ListMessages(viper.GetStringSlice(mailboxes), searchCriteria)
//box.CreateMailbox("INBOX/Processed")
//s.EmailService.CreateMailbox("Processed/Succeed")
//s.EmailService.CreateMailbox("Processed/Failed")
//s.EmailService.CreateMailbox("Processed/Suspicious")
//s.EmailService.CreateMailbox("Succeed")
//s.EmailService.DeleteMailbox("Succeed")
//s.EmailService.MailboxesList()
entries := s.Processing(viper.GetStringMapString(fmt.Sprintf("%s.sender", s.scope)))
@@ -131,7 +129,8 @@ func (s *ShareTheBrutality) Processing(sender map[string]string) []model.Externa
}
es.Releaser = sourceDataMatch[1]
es.Created = carbon.Parse(msg.Envelope.Date.String())
es.Created = carbon.Parse(msg.Envelope.Date.String(), "Europe/Kyiv")
es.Type = dbType
genre := strings.ToLower(sourceDataMatch[2])
@@ -140,6 +139,9 @@ func (s *ShareTheBrutality) Processing(sender map[string]string) []model.Externa
entries = append(entries, es)
}
//fmt.Println(entries)
//os.Exit(0)
result, status := _table.BatchInsertOnDuplicate(entries, s.db, columns)
if status != constant.StatusFailed {
entriesBatched = append(entriesBatched, result...)
+10 -4
View File
@@ -9,9 +9,9 @@ import (
"github.com/jmoiron/sqlx"
"github.com/logrusorgru/aurora/v4"
"git.amok.space/yevhen/resource-scraper/helper/thither"
"git.amok.space/yevhen/resource-scraper/types/constant"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/helper/thither"
"git.kplus.net.ua/yevhen/resource-scraper/types/constant"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type ExternalSources struct {
@@ -41,7 +41,7 @@ func BatchInsertOnDuplicate(entries []model.ExternalSources, db *sqlx.DB, column
var status string
errCount := 0
for i := 0; i < len(entries); i++ {
for i := range entries {
entry := es.InsertOnDuplicate(entries[i], db)
if entry.Error != nil {
slog.Error("insert/update entry", "err", entry.Error)
@@ -67,9 +67,15 @@ func BatchInsertOnDuplicate(entries []model.ExternalSources, db *sqlx.DB, column
}
func (f *ExternalSources) GetTypeIds(entries []model.ExternalSources, db *sqlx.DB) []int {
if len(entries) == 0 {
fmt.Printf("Entries are empty")
return nil // або логування помилки
}
var typeIds []int
ids := thither.FieldValueToStrSlice(entries, "TypeId")
query := fmt.Sprintf("SELECT type_id FROM %s WHERE `type` = '%s' AND type_id IN (%s) LIMIT %d", constant.ExternalSourcesTable, entries[0].Type, strings.Join(ids, ","), len(ids))
fmt.Println(query)
err := db.Select(&typeIds, query)
if err != nil {
slog.Error("getting type ids", "err", err)
+6 -2
View File
@@ -1,8 +1,8 @@
package service
import (
"git.amok.space/yevhen/resource-scraper/types/interface"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/types/interface"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type PresceneService struct {
@@ -16,3 +16,7 @@ func NewPresceneService(repo _interface.Prescene) *PresceneService {
func (p *PresceneService) GetPage(pageNumbers []string) ([]model.ExternalSources, error) {
return p.repo.GetPage(pageNumbers)
}
func (p *PresceneService) GetSrrDb() ([]model.ExternalSources, error) {
return p.repo.GetSrrDb()
}
+2 -2
View File
@@ -1,8 +1,8 @@
package service
import (
"git.amok.space/yevhen/resource-scraper/types/interface"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/types/interface"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type RutrackerService struct {
+2 -2
View File
@@ -1,8 +1,8 @@
package service
import (
"git.amok.space/yevhen/resource-scraper/pkg/repository"
"git.amok.space/yevhen/resource-scraper/types/interface"
"git.kplus.net.ua/yevhen/resource-scraper/pkg/repository"
"git.kplus.net.ua/yevhen/resource-scraper/types/interface"
)
type Service struct {
+3 -3
View File
@@ -1,9 +1,9 @@
package service
import (
"git.amok.space/yevhen/resource-scraper/internal/mail"
"git.amok.space/yevhen/resource-scraper/types/interface"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/internal/mail"
"git.kplus.net.ua/yevhen/resource-scraper/types/interface"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type ShareTheBrutalityService struct {
+6 -3
View File
@@ -1,8 +1,8 @@
package _interface
import (
"git.amok.space/yevhen/resource-scraper/internal/mail"
"git.amok.space/yevhen/resource-scraper/types/model"
"git.kplus.net.ua/yevhen/resource-scraper/internal/mail"
"git.kplus.net.ua/yevhen/resource-scraper/types/model"
)
type Rutracker interface {
@@ -11,12 +11,15 @@ type Rutracker interface {
type Prescene interface {
GetPage(pageNumbers []string) ([]model.ExternalSources, error)
GetSrrDb() ([]model.ExternalSources, error)
}
type ShareTheBrutality interface {
GetMail(email string) ([]model.ExternalSources, *mail.EmailService)
}
type MetalArchiveInterface interface{}
type MetalArchiveInterface interface {
GetSearchResults(searchResults []model.ExternalSources) ([]model.ExternalSources, error)
}
type Info interface{}
+18 -18
View File
@@ -1,7 +1,7 @@
package model
import (
"github.com/golang-module/carbon/v2"
"github.com/dromara/carbon/v2"
)
type Type string
@@ -21,22 +21,22 @@ type Type string
)*/
type ExternalSources struct {
Id int64 `json:"id" db:"id"`
Type string `json:"type" db:"type"`
TypeId int `json:"type_id" db:"type_id"`
Title string `json:"title" db:"title"`
TypeSubsectionId int `json:"type_subsection_id" db:"type_subsection_id"`
Releaser string `json:"releaser" db:"releaser"`
ExSource string `json:"ex_source" db:"eXsource"`
Created carbon.Carbon `json:"created_at" db:"created"`
Modified carbon.Carbon `json:"modified_at" db:"modified"`
Contents string `json:"contents" db:"contents"`
A string `json:"a" db:"a"`
H string `json:"h" db:"h"`
Fingerprint string `json:"fingerprint" db:"fingerprint"`
FsFingerprint string `json:"fs_fingerprint" db:"fs_fingerprint"`
Vid int `json:"vid" db:"vid"`
G string `json:"g" db:"g"`
ExternalSourcesCol string `json:"external_sources_column" db:"external_sourcescol"`
Id int64 `json:"id" db:"id"`
Type string `json:"type" db:"type"`
TypeId int `json:"type_id" db:"type_id"`
Title string `json:"title" db:"title"`
TypeSubsectionId int `json:"type_subsection_id" db:"type_subsection_id"`
Releaser string `json:"releaser" db:"releaser"`
ExSource string `json:"ex_source" db:"eXsource"`
Created *carbon.Carbon `json:"created_at" db:"created"`
Modified carbon.Carbon `json:"modified_at" db:"modified"`
Contents string `json:"contents" db:"contents"`
A string `json:"a" db:"a"`
H string `json:"h" db:"h"`
Fingerprint string `json:"fingerprint" db:"fingerprint"`
FsFingerprint string `json:"fs_fingerprint" db:"fs_fingerprint"`
Vid int `json:"vid" db:"vid"`
G string `json:"g" db:"g"`
ExternalSourcesCol string `json:"external_sources_column" db:"external_sourcescol"`
Error error
}
+1 -1
View File
@@ -1,6 +1,6 @@
package resource
import "github.com/golang-module/carbon/v2"
import "github.com/dromara/carbon/v2"
/*https://www.metal-archives.com/search/ajax-advanced/searching/albums/?bandName=&releaseTitle=&releaseYearFrom=2024&releaseMonthFrom=09&releaseYearTo=&releaseMonthTo=&country=&location=&releaseLabelName=&releaseCatalogNumber=&releaseIdentifiers=&releaseRecordingInfo=&releaseDescription=&releaseNotes=&genre=&sEcho=6&iColumns=4&sColumns=&iDisplayStart=1000&iDisplayLength=200&mDataProp_0=0&mDataProp_1=1&mDataProp_2=2&mDataProp_3=3&_=1726217541027*/
+56 -30
View File
@@ -1,35 +1,61 @@
package resource
import "encoding/xml"
import (
"encoding/xml"
"fmt"
"strconv"
"strings"
)
// RutrackerAtomTopic represents the atom feed of a rutracker forum subsection.
type RutrackerAtomTopic struct {
XMLName xml.Name `xml:"feed"`
Text string `xml:",chardata"`
Xmlns string `xml:"xmlns,attr"`
ID string `xml:"id"`
Link struct {
Text string `xml:",chardata"`
Href string `xml:"href,attr"`
} `xml:"link"`
Updated string `xml:"updated"`
Title string `xml:"title"`
Entry []struct {
Text string `xml:",chardata"`
ID string `xml:"id"`
Link struct {
Text string `xml:",chardata"`
Href string `xml:"href,attr"`
} `xml:"link"`
Updated string `xml:"updated"`
Title string `xml:"title"`
Author struct {
Text string `xml:",chardata"`
Name string `xml:"name"`
} `xml:"author"`
Category struct {
Text string `xml:",chardata"`
Term string `xml:"term,attr"`
Label string `xml:"label,attr"`
} `xml:"category"`
} `xml:"entry"`
XMLName xml.Name `xml:"feed"`
Xmlns string `xml:"xmlns,attr"`
ID string `xml:"id"`
Link []AtomLink `xml:"link"`
Updated string `xml:"updated"`
Title string `xml:"title"`
Entry []RutrackerAtomEntry `xml:"entry"`
}
// AtomLink is a generic atom link element. An entry contains two links:
// a topic page link (no rel) and a magnet link (rel="enclosure").
type AtomLink struct {
Href string `xml:"href,attr"`
Rel string `xml:"rel,attr"`
Type string `xml:"type,attr"`
}
type RutrackerAtomEntry struct {
ID string `xml:"id"`
Link []AtomLink `xml:"link"`
Updated string `xml:"updated"`
Title string `xml:"title"`
Author struct {
Name string `xml:"name"`
} `xml:"author"`
Category struct {
Term string `xml:"term,attr"`
Label string `xml:"label,attr"`
} `xml:"category"`
}
// PageLink returns the link to the topic page (the one without rel attribute).
func (e RutrackerAtomEntry) PageLink() (AtomLink, bool) {
for _, l := range e.Link {
if l.Rel == "" {
return l, true
}
}
return AtomLink{}, false
}
// TopicId extracts the topic id from the entry id tag: "tag:rto.feed,2026-08-29:/t/5702203".
func (e RutrackerAtomEntry) TopicId() (int, error) {
const marker = ":/t/"
i := strings.LastIndex(e.ID, marker)
if i < 0 {
return 0, fmt.Errorf("unexpected entry id format: %s", e.ID)
}
return strconv.Atoi(e.ID[i+len(marker):])
}