Route SimpCity forum scraping through FlareSolverr + add turbo.cr resolver
DDoS-Guard now binds session cookies to the issuing browser's fingerprint, so direct Node fetch returns 403 even with valid cookies. Page HTML for any forum_site with stored cookies is now fetched via a FlareSolverr browser session opened once per scrape job. - Hybrid cookie refresh: FlareSolverr clears the DDoS-Guard captcha, those cookies seed undetected_chromedriver, Turnstile auto-solves in the real browser, login form submits, final cookies + browser UA persist to forum_sites - Per-site user_agent column so subsequent scraper requests match the UA the cookies were issued for (DDoS-Guard rejects UA mismatches) - XenForo search rewritten as proper CSRF POST /search/search → results page parse, replacing the broken ?q=... GET that only returned the search form - Pagination regex fallback in detectMaxPage catches XenForo pages that cheerio's class-based selectors miss - New scrapers/turbo.js handles turbo.cr /embed/ and /a/ URLs by rendering the page via FlareSolverr and grabbing the signed mp4 from the resolved <video src> attribute (gallery-dl can't extract these — obfuscated WASM) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
+4
-1
@@ -137,6 +137,9 @@ if (!forumCols.includes('password')) {
|
||||
if (!forumCols.includes('cookie_expires_at')) {
|
||||
db.exec('ALTER TABLE forum_sites ADD COLUMN cookie_expires_at TEXT');
|
||||
}
|
||||
if (!forumCols.includes('user_agent')) {
|
||||
db.exec("ALTER TABLE forum_sites ADD COLUMN user_agent TEXT DEFAULT ''");
|
||||
}
|
||||
|
||||
export function getAuthConfig() {
|
||||
const row = db.prepare('SELECT * FROM auth_config LIMIT 1').get();
|
||||
@@ -768,7 +771,7 @@ export function createForumSite(name, baseUrl, cookies, username, password) {
|
||||
}
|
||||
|
||||
export function updateForumSite(id, fields) {
|
||||
const allowed = ['name', 'base_url', 'cookies', 'username', 'password', 'cookie_expires_at'];
|
||||
const allowed = ['name', 'base_url', 'cookies', 'username', 'password', 'cookie_expires_at', 'user_agent'];
|
||||
const sets = [];
|
||||
const vals = [];
|
||||
for (const [k, v] of Object.entries(fields)) {
|
||||
|
||||
Reference in New Issue
Block a user