Yevhen Odynets
346369ca11
New scope reading the API instead of scraping HTML: one search path per run, passed raw through --single-uri, and the answer stored as-is into its own SQLite file created from the embedded schema. Command line: --single-uri carries the path after /v1/search/ (category:flac/date:today/page:1) and must name a category; --sleep sets the pause between pages, with the config value as fallback. A day named on the command line is always fetched again and its dump rewritten, so cron can refresh today hourly and yesterday daily. Date placeholders: date:auto walks backwards to the newest day in [srrdb.floor..yesterday] that has no dump yet, closing holes instead of declaring the archive covered. date:today, date:yesterday and date:tomorrow stand for the neighbouring days. Today is never taken by the walk, because a day still being filled would be frozen by its own dump after the first run. Dumps are the cursor as well as the record, so a missing dump directory is an error instead of "nothing harvested yet": a renamed, unmounted or mistyped directory must not send the run silently back to the beginning. An empty day is a normal result and is remembered as harvested; API warnings never are, and they reach the exit code. Writes go through one transaction, so a day lands completely or not at all. group and category come from the release line and the search path; status, nfo, content and synced_at stay NULL. srrdb.log.jsonl records one line per finished run, srrdb.error_log.jsonl keeps failures and API warnings apart by a kind field. Also: the console command contract returns (string, error), so a failing scope reaches the process exit code and cron can tell a failed run from a successful one. config/scope/srrdb.yaml is untracked because its paths differ per host. modernc.org/sqlite replaces the archived libsql client, which only ever talked to a server. pkg/repository/srrdb.go is split by responsibility and covered by table-driven tests.
45 lines
966 B
Go
45 lines
966 B
Go
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"os"
|
|
"time"
|
|
|
|
_ "github.com/go-sql-driver/mysql"
|
|
_ "modernc.org/sqlite"
|
|
|
|
scraper "git.kplus.net.ua/yevhen/resource-scraper/internal"
|
|
"git.kplus.net.ua/yevhen/resource-scraper/internal/config"
|
|
"git.kplus.net.ua/yevhen/resource-scraper/internal/logging"
|
|
)
|
|
|
|
var (
|
|
st time.Time
|
|
)
|
|
|
|
//var tokenAuth *jwtauth.JWTAuth
|
|
|
|
func init() {
|
|
st = time.Now()
|
|
logging.New()
|
|
scraper.ParseFlags()
|
|
config.New()
|
|
|
|
/*verifyKey := []byte(viper.GetString("jwt.sign_key"))
|
|
tokenAuth = jwtauth.New("HS256", verifyKey, nil)
|
|
|
|
// For debugging/example purposes, we generate and print
|
|
// a sample jwt token with claims `user_id:123` here:
|
|
_, tokenString, _ := tokenAuth.Encode(map[string]interface{}{"user_id": 123})
|
|
slog.Info("jwt", "token", tokenString)*/
|
|
}
|
|
|
|
func main() {
|
|
code := scraper.Bootstrap()
|
|
|
|
fmt.Printf("\nExecution took %v\n", time.Since(st))
|
|
|
|
// os.Exit skips deferred calls, so the timing above is printed before it.
|
|
os.Exit(code)
|
|
}
|