srrdb: harvest the srrDB search API into a separate SQLite database

New scope reading the API instead of scraping HTML: one search path per run,
passed raw through --single-uri, and the answer stored as-is into its own
SQLite file created from the embedded schema.

Command line: --single-uri carries the path after /v1/search/
(category:flac/date:today/page:1) and must name a category; --sleep sets the
pause between pages, with the config value as fallback. A day named on the
command line is always fetched again and its dump rewritten, so cron can
refresh today hourly and yesterday daily.

Date placeholders: date:auto walks backwards to the newest day in
[srrdb.floor..yesterday] that has no dump yet, closing holes instead of
declaring the archive covered. date:today, date:yesterday and date:tomorrow
stand for the neighbouring days. Today is never taken by the walk, because a
day still being filled would be frozen by its own dump after the first run.

Dumps are the cursor as well as the record, so a missing dump directory is an
error instead of "nothing harvested yet": a renamed, unmounted or mistyped
directory must not send the run silently back to the beginning. An empty day is
a normal result and is remembered as harvested; API warnings never are, and
they reach the exit code.

Writes go through one transaction, so a day lands completely or not at all.
group and category come from the release line and the search path; status, nfo,
content and synced_at stay NULL. srrdb.log.jsonl records one line per finished
run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind
field.

Also: the console command contract returns (string, error), so a failing scope
reaches the process exit code and cron can tell a failed run from a successful
one. config/scope/srrdb.yaml is untracked because its paths differ per host.
modernc.org/sqlite replaces the archived libsql client, which only ever talked
to a server. pkg/repository/srrdb.go is split by responsibility and covered by
table-driven tests.
This commit is contained in:
Yevhen Odynets
2026-09-30 09:27:33 +03:00
parent 332d143f1f
commit 346369ca11
35 changed files with 1653 additions and 53 deletions
+1
View File
@@ -6,6 +6,7 @@ const (
ScopeWeb string = "web"
ScopeInfo string = "info"
ScopeShareTheBrutality string = "stb"
ScopeSrrdb string = "srrdb"
)
const (
+2
View File
@@ -10,6 +10,8 @@ const (
FlagScopeEnable = "scope-enable"
FlagSingleUri = "single-uri"
FlagEnv = "env"
FlagCreate = "create"
FlagSleep = "sleep"
DefaultEnvProd = "prod"
DefaultEnvDev = "devel"
+4
View File
@@ -22,4 +22,8 @@ type MetalArchiveInterface interface {
GetSearchResults(searchResults []model.ExternalSources) ([]model.ExternalSources, error)
}
type Srrdb interface {
Search(uri string) (int, error)
}
type Info interface{}
+22
View File
@@ -0,0 +1,22 @@
package model
// SrrdbRelease is one row of the srrdb_releases table described by
// internal/db/srrdb.sql. Release is the primary key, therefore a repeated
// search rewrites the previous row of the same release instead of inserting
// a copy. Columns left nil are stored as NULL.
type SrrdbRelease struct {
Release string `db:"release"`
Date string `db:"date"`
HasNfo string `db:"has_nfo"`
HasSrs string `db:"has_srs"`
IsForeign string `db:"is_foreign"`
ImdbID *string `db:"imdb_id"`
Size int64 `db:"size"`
GroupName string `db:"group_name"`
Category string `db:"category"`
Status *string `db:"status"`
Nfo []byte `db:"nfo"`
Content *string `db:"content"`
CreatedAt string `db:"created_at"`
SyncedAt *string `db:"synced_at"`
}
+26
View File
@@ -0,0 +1,26 @@
package resource
import "encoding/json"
// SrrdbSearchResponse is the payload of GET https://api.srrdb.com/v1/search/{path}.
// Query stays raw because the API returns a heterogeneous array there: category
// ids as numbers, search terms as strings.
type SrrdbSearchResponse struct {
Results []SrrdbResult `json:"results"`
ResultsCount int `json:"resultsCount"`
Warnings []string `json:"warnings"`
Query json.RawMessage `json:"query"`
}
// SrrdbResult is a single release as reported by the search endpoint.
// HasNFO, HasSRS and IsForeign arrive as the literal strings "yes" and "no";
// ImdbID is null for music releases.
type SrrdbResult struct {
Release string `json:"release"`
Date string `json:"date"`
HasNFO string `json:"hasNFO"`
HasSRS string `json:"hasSRS"`
IsForeign string `json:"isForeign"`
ImdbID *string `json:"imdbId"`
Size int64 `json:"size"`
}