Yevhen Odynets
346369ca11
New scope reading the API instead of scraping HTML: one search path per run, passed raw through --single-uri, and the answer stored as-is into its own SQLite file created from the embedded schema. Command line: --single-uri carries the path after /v1/search/ (category:flac/date:today/page:1) and must name a category; --sleep sets the pause between pages, with the config value as fallback. A day named on the command line is always fetched again and its dump rewritten, so cron can refresh today hourly and yesterday daily. Date placeholders: date:auto walks backwards to the newest day in [srrdb.floor..yesterday] that has no dump yet, closing holes instead of declaring the archive covered. date:today, date:yesterday and date:tomorrow stand for the neighbouring days. Today is never taken by the walk, because a day still being filled would be frozen by its own dump after the first run. Dumps are the cursor as well as the record, so a missing dump directory is an error instead of "nothing harvested yet": a renamed, unmounted or mistyped directory must not send the run silently back to the beginning. An empty day is a normal result and is remembered as harvested; API warnings never are, and they reach the exit code. Writes go through one transaction, so a day lands completely or not at all. group and category come from the release line and the search path; status, nfo, content and synced_at stay NULL. srrdb.log.jsonl records one line per finished run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind field. Also: the console command contract returns (string, error), so a failing scope reaches the process exit code and cron can tell a failed run from a successful one. config/scope/srrdb.yaml is untracked because its paths differ per host. modernc.org/sqlite replaces the archived libsql client, which only ever talked to a server. pkg/repository/srrdb.go is split by responsibility and covered by table-driven tests.
52 lines
2.0 KiB
Modula-2
52 lines
2.0 KiB
Modula-2
module git.kplus.net.ua/yevhen/resource-scraper
|
|
|
|
go 1.27
|
|
|
|
require (
|
|
github.com/davecgh/go-spew v1.1.1
|
|
github.com/dromara/carbon/v2 v2.6.17
|
|
github.com/emersion/go-imap/v2 v2.0.0-beta.8
|
|
github.com/go-chi/chi/v5 v5.3.2
|
|
github.com/go-chi/cors v1.2.2
|
|
github.com/go-shiori/dom v0.0.0-20230515143342-73569d674e1c
|
|
github.com/go-sql-driver/mysql v1.10.1
|
|
github.com/iancoleman/strcase v0.3.0
|
|
github.com/jmoiron/sqlx v1.4.0
|
|
github.com/logrusorgru/aurora/v4 v4.0.0
|
|
github.com/mewkiz/flac v1.0.14
|
|
github.com/spf13/pflag v1.0.10
|
|
github.com/spf13/viper v1.21.0
|
|
golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba
|
|
golang.org/x/net v0.59.0
|
|
modernc.org/sqlite v1.60.0
|
|
)
|
|
|
|
require (
|
|
filippo.io/edwards25519 v1.2.0 // indirect
|
|
github.com/andybalholm/cascadia v1.3.5 // indirect
|
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
github.com/emersion/go-message v0.18.2 // indirect
|
|
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 // indirect
|
|
github.com/fsnotify/fsnotify v1.10.1 // indirect
|
|
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
|
github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f // indirect
|
|
github.com/google/uuid v1.6.0 // indirect
|
|
github.com/icza/bitio v1.1.0 // indirect
|
|
github.com/mattn/go-isatty v0.0.24 // indirect
|
|
github.com/mewkiz/pkg v0.0.0-20260703220044-4fb89b18cc87 // indirect
|
|
github.com/mewpkg/term v0.0.0-20241026122259-37a80af23985 // indirect
|
|
github.com/ncruces/go-strftime v1.0.0 // indirect
|
|
github.com/pelletier/go-toml/v2 v2.4.3 // indirect
|
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
|
github.com/sagikazarmark/locafero v0.12.0 // indirect
|
|
github.com/spf13/afero v1.15.0 // indirect
|
|
github.com/spf13/cast v1.10.0 // indirect
|
|
github.com/subosito/gotenv v1.6.0 // indirect
|
|
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
|
golang.org/x/sys v0.48.0 // indirect
|
|
golang.org/x/text v0.42.0 // indirect
|
|
modernc.org/libc v1.77.1 // indirect
|
|
modernc.org/mathutil v1.7.1 // indirect
|
|
modernc.org/memory v1.12.1 // indirect
|
|
)
|