Yevhen Odynets
346369ca11
New scope reading the API instead of scraping HTML: one search path per run, passed raw through --single-uri, and the answer stored as-is into its own SQLite file created from the embedded schema. Command line: --single-uri carries the path after /v1/search/ (category:flac/date:today/page:1) and must name a category; --sleep sets the pause between pages, with the config value as fallback. A day named on the command line is always fetched again and its dump rewritten, so cron can refresh today hourly and yesterday daily. Date placeholders: date:auto walks backwards to the newest day in [srrdb.floor..yesterday] that has no dump yet, closing holes instead of declaring the archive covered. date:today, date:yesterday and date:tomorrow stand for the neighbouring days. Today is never taken by the walk, because a day still being filled would be frozen by its own dump after the first run. Dumps are the cursor as well as the record, so a missing dump directory is an error instead of "nothing harvested yet": a renamed, unmounted or mistyped directory must not send the run silently back to the beginning. An empty day is a normal result and is remembered as harvested; API warnings never are, and they reach the exit code. Writes go through one transaction, so a day lands completely or not at all. group and category come from the release line and the search path; status, nfo, content and synced_at stay NULL. srrdb.log.jsonl records one line per finished run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind field. Also: the console command contract returns (string, error), so a failing scope reaches the process exit code and cron can tell a failed run from a successful one. config/scope/srrdb.yaml is untracked because its paths differ per host. modernc.org/sqlite replaces the archived libsql client, which only ever talked to a server. pkg/repository/srrdb.go is split by responsibility and covered by table-driven tests.
39 lines
1.1 KiB
SQL
39 lines
1.1 KiB
SQL
CREATE TABLE IF NOT EXISTS srrdb_releases
|
|
(
|
|
"release" TEXT PRIMARY KEY NOT NULL,
|
|
date TEXT,
|
|
created_at TEXT NOT NULL,
|
|
synced_at TEXT,
|
|
category TEXT NOT NULL DEFAULT '',
|
|
status TEXT,
|
|
group_name TEXT,
|
|
has_nfo TEXT,
|
|
has_srs TEXT,
|
|
is_foreign TEXT,
|
|
imdb_id TEXT,
|
|
size INTEGER,
|
|
comment TEXT,
|
|
nfo BLOB,
|
|
content TEXT,
|
|
source TEXT,
|
|
CHECK (
|
|
status IN (
|
|
'on', 'hdd', 'hdd_e', 'catalogued',
|
|
'deferred', 'ext', 'cloud', 'burnt', 'updated',
|
|
'onhold', 'watch', 'hidden', 'duplicate', 'trash',
|
|
'closed', 'request', '2cat', '2compare', 'neformat',
|
|
'sf88', 'BATA', 'away', 'ready2burn', 'accurip',
|
|
'replace', '2burn', 'remove', 'paid', 'free', 'ftp',
|
|
'stream', 'ai'
|
|
)
|
|
)
|
|
);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_srrdb_date
|
|
ON srrdb_releases (date);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_srrdb_group
|
|
ON srrdb_releases (group_name);
|
|
|
|
CREATE INDEX IF NOT EXISTS idx_srrdb_status
|
|
ON srrdb_releases (status); |