Yevhen Odynets
346369ca11
New scope reading the API instead of scraping HTML: one search path per run, passed raw through --single-uri, and the answer stored as-is into its own SQLite file created from the embedded schema. Command line: --single-uri carries the path after /v1/search/ (category:flac/date:today/page:1) and must name a category; --sleep sets the pause between pages, with the config value as fallback. A day named on the command line is always fetched again and its dump rewritten, so cron can refresh today hourly and yesterday daily. Date placeholders: date:auto walks backwards to the newest day in [srrdb.floor..yesterday] that has no dump yet, closing holes instead of declaring the archive covered. date:today, date:yesterday and date:tomorrow stand for the neighbouring days. Today is never taken by the walk, because a day still being filled would be frozen by its own dump after the first run. Dumps are the cursor as well as the record, so a missing dump directory is an error instead of "nothing harvested yet": a renamed, unmounted or mistyped directory must not send the run silently back to the beginning. An empty day is a normal result and is remembered as harvested; API warnings never are, and they reach the exit code. Writes go through one transaction, so a day lands completely or not at all. group and category come from the release line and the search path; status, nfo, content and synced_at stay NULL. srrdb.log.jsonl records one line per finished run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind field. Also: the console command contract returns (string, error), so a failing scope reaches the process exit code and cron can tell a failed run from a successful one. config/scope/srrdb.yaml is untracked because its paths differ per host. modernc.org/sqlite replaces the archived libsql client, which only ever talked to a server. pkg/repository/srrdb.go is split by responsibility and covered by table-driven tests.
61 lines
1.6 KiB
Go
61 lines
1.6 KiB
Go
package db
|
|
|
|
import (
|
|
_ "embed"
|
|
"fmt"
|
|
"strings"
|
|
|
|
"github.com/jmoiron/sqlx"
|
|
"github.com/spf13/viper"
|
|
)
|
|
|
|
// srrdbSchema is the DDL of the srrdb database. Keeping it in a file lets the
|
|
// schema be read and edited without touching Go code; go:embed pulls it into the
|
|
// binary at build time.
|
|
//
|
|
//go:embed srrdb.sql
|
|
var srrdbSchema string
|
|
|
|
// NewSrrdb opens the separate SQLite database that holds srrdb_releases.
|
|
// The driver name is read from db.ext.driver (sqlite, modernc.org/sqlite), so
|
|
// the driver can be replaced in configuration without touching this file.
|
|
func NewSrrdb() *sqlx.DB {
|
|
driver := viper.GetString("db.ext.driver")
|
|
path := viper.GetString("srrdb.database")
|
|
|
|
db, err := sqlx.Connect(driver, path)
|
|
if err != nil {
|
|
panic("Failed to connect to the srrdb database: " + err.Error())
|
|
}
|
|
|
|
if err = db.Ping(); err != nil {
|
|
panic("Failed to ping the srrdb database: " + err.Error())
|
|
}
|
|
|
|
if err = srrdbCreate(db); err != nil {
|
|
panic("Failed to create the srrdb schema: " + err.Error())
|
|
}
|
|
|
|
return db
|
|
}
|
|
|
|
// srrdbCreate applies the embedded DDL statement by statement. Every statement is
|
|
// written to be repeatable, so an existing database is left alone: this creates
|
|
// missing objects, it does not alter existing ones.
|
|
//
|
|
// Splitting on ";" is safe for this schema because no statement holds a semicolon
|
|
// inside a literal.
|
|
func srrdbCreate(db *sqlx.DB) error {
|
|
for _, statement := range strings.Split(srrdbSchema, ";") {
|
|
if statement = strings.TrimSpace(statement); statement == "" {
|
|
continue
|
|
}
|
|
|
|
if _, err := db.Exec(statement); err != nil {
|
|
return fmt.Errorf("apply srrdb schema statement %q: %w", statement, err)
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|