Route SimpCity forum scraping through FlareSolverr + add turbo.cr resolver

DDoS-Guard now binds session cookies to the issuing browser's fingerprint, so
direct Node fetch returns 403 even with valid cookies. Page HTML for any
forum_site with stored cookies is now fetched via a FlareSolverr browser
session opened once per scrape job.

- Hybrid cookie refresh: FlareSolverr clears the DDoS-Guard captcha, those
  cookies seed undetected_chromedriver, Turnstile auto-solves in the real
  browser, login form submits, final cookies + browser UA persist to forum_sites
- Per-site user_agent column so subsequent scraper requests match the UA the
  cookies were issued for (DDoS-Guard rejects UA mismatches)
- XenForo search rewritten as proper CSRF POST /search/search → results page
  parse, replacing the broken ?q=... GET that only returned the search form
- Pagination regex fallback in detectMaxPage catches XenForo pages that
  cheerio's class-based selectors miss
- New scrapers/turbo.js handles turbo.cr /embed/ and /a/ URLs by rendering
  the page via FlareSolverr and grabbing the signed mp4 from the resolved
  <video src> attribute (gallery-dl can't extract these — obfuscated WASM)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Trey T
2026-04-29 19:33:54 -05:00
parent 236f36aae6
commit aa4f1157d1
6 changed files with 589 additions and 78 deletions
+4 -1
View File
@@ -137,6 +137,9 @@ if (!forumCols.includes('password')) {
if (!forumCols.includes('cookie_expires_at')) {
db.exec('ALTER TABLE forum_sites ADD COLUMN cookie_expires_at TEXT');
}
if (!forumCols.includes('user_agent')) {
db.exec("ALTER TABLE forum_sites ADD COLUMN user_agent TEXT DEFAULT ''");
}
export function getAuthConfig() {
const row = db.prepare('SELECT * FROM auth_config LIMIT 1').get();
@@ -768,7 +771,7 @@ export function createForumSite(name, baseUrl, cookies, username, password) {
}
export function updateForumSite(id, fields) {
const allowed = ['name', 'base_url', 'cookies', 'username', 'password', 'cookie_expires_at'];
const allowed = ['name', 'base_url', 'cookies', 'username', 'password', 'cookie_expires_at', 'user_agent'];
const sets = [];
const vals = [];
for (const [k, v] of Object.entries(fields)) {