GitHub - techno-neighbour/focal-harvest: An interactive CLI research assistant that scrapes, sanitizes, and synthesizes web content locally or via Gemini/OpenAI/Claude, with Discord/Telegram alerts & built-in automation.

Focal Harvest is a lightweight Python CLI tool with two primary use-cases, featuring a zero-config local statistical fallback for offline operation:

Automated Research Pipeline: Query search engines, scrape and clean target pages, and synthesize findings into structured reports using Gemini, Claude, or GPT-4o-mini.
OSINT Collection Framework: Monitor topics across Reddit, Hacker News, Stack Overflow, arXiv, and 17+ platforms with daemon scheduling, Wayback Machine fallbacks, and Discord/Telegram alert dispatch.

Python • AI Agent • Web Research • OSINT • Multi-Engine • 3-Tier Archive Cascade • Zero-Config Offline Fallback

🚀 What's New in v1.6.0

📱 Bidirectional Remote Mobile Bot Listener (Telegram & Discord): Message your Focal Harvest instance directly from your mobile device via Telegram or Discord slash commands (/research, /deep, /url, /status, /help). Supports single-word and quoted multi-word topics (e.g. /research "robotics engineering"), multi-URL deep dives (e.g. /url 'https://site1.com', 'https://site2.com'), custom focus areas, User ID authorization whitelisting, compact mobile summaries, and native .pdf and .md report uploads. See BOT_LISTENER_GUIDE.md for full setup instructions.

⚡ Quick Start (Get Started in 30 Seconds)

Installation

Ensure you have Python 3.12+ installed. Run the platform-independent installer:

(Or run pip install -r requirements.txt for standard manual installation).

Minimal Working Example (CLI Launch)

Launch the interactive Terminal User Interface (TUI):

Programmatic Python API Usage

For custom automation scripts, you can run the scraping and synthesis pipeline programmatically in Python:

from scraper import search_duckduckgo, scrape_urls_concurrently
from analyzer import synthesize_topics

# 1. Search for a query
results = search_duckduckgo("Gemini 1.5 Flash vs Pro", max_results=3)
urls = [r["url"] for r in results]

# 2. Scrape target pages concurrently in parallel threads
scraped_data = scrape_urls_concurrently(urls)

# 3. Synthesize the findings into a structured Markdown report
report = synthesize_topics(
    scraped_data, 
    query="Gemini 1.5 Flash vs Pro", 
    spec_topic="pricing comparisons"
)
print(report)

🎯 The Problem & Our Solution

Every developer, researcher, and tech blogger repeats the same tedious manual workflow: Search Query ➔ Open 20 Browser Tabs ➔ Ignore Cookie Walls & Ads ➔ Copy-Paste Snippets ➔ Synthesize with LLM ➔ Save Report

Focal Harvest automates this entire process inside a lightweight, terminal-based pipeline.

📊 Performance Benchmarks (Typical Runs)

Metric / Operation	Manual Browser Sweep	Heavy Local LLM (Ollama/Llama3)	Focal Harvest
Setup Package Size	~150MB (Chrome)	~4.7GB (Llama3-8B)	<5.2MB (Zero-Dependency core)
Cold Startup Time	~1.5s	~12.5s (model load)	<0.15s (TUI instant-boot)
Scrape & Clean Speed	~45s per page	~1.2s per page (raw HTML)	<0.25s per page (parallel readability)
Context Density	High fluff (ads/nav)	Medium (all HTML tags)	High Density (Top 15 key sentences)
Resource Usage	~1.2GB RAM	~6.5GB RAM + dedicated GPU	<85MB RAM (runs on low-end hardware)

⚙️ How It Works

graph LR
    A["Focus Query"] --> B["Concurrent Scraper (utils.py)"]
    B --> C["Readability Engine (scraper.py)"]
    C --> D["Synthesis Engine (analyzer.py)"]
    D --> E["TUI Reports / Webhooks (notifier.py)"]
    
    style A fill:#e1f5fe,stroke:#03a9f4,stroke-width:2px,color:#000000
    style E fill:#e8f5e9,stroke:#4caf50,stroke-width:2px,color:#000000

Orchestrated Search: Resolves search parameters via Tavily or DuckDuckGo.
Concurrent Fetching: Scrapes pages in parallel threads with Cloudflare bypasses.
HTML Sanitization: Stubs out ads, widgets, and cookie overlays, keeping only clean prose.
Adaptive Replenishment: Drops blocked/empty pages and automatically fetches replacements to ensure high-density inputs.
AI/Local Synthesis: Summarizes content using Gemini, Claude, or OpenAI with automatic key failover. Automatically falls back to an offline PositionRank statistical ranker if no API keys are provided.
Instant Dispatch: Saves structured Markdown/JSON reports locally and pushes alerts to Discord/Telegram.

📋 Example Output

Here is a real example of the structured Markdown report generated when researching "Gemini 1.5 Flash vs Gemini 1.5 Pro":

✨ Features

📱 Bidirectional Remote Mobile Bot Listener

Remote Mobile Slash Commands: Control your Focal Harvest instance from your mobile device via Telegram or Discord slash commands (/research, /deep, /url, /status, /help). See BOT_LISTENER_GUIDE.md for full setup.
Strict Whitelist Protection: Enforces user ID authorization whitelisting (discord_allowed_user_ids & telegram_allowed_user_ids) to block unauthorized access.
Dual Document Delivery: Receives a compact mobile summary preview along with both .pdf and .md report document file uploads.

🎨 Dynamic Adaptive Domain Categories

Intent-Aware Categorization: AI synthesis automatically classifies queries into specialized 3-section frameworks tailored to the domain:
- Software & Tech: Architecture & System Design ➔ Performance & Benchmarks ➔ Tradeoffs & Best Practices
- Entertainment & Media: Overview & Plot Synopsis ➔ Production & Creative Execution ➔ Critical Reception & Impact
- Business & Finance: Industry Overview & Market ➔ Business Model & Financial Metrics ➔ Risk Factors & Strategic Outlook
- Science & Healthcare: Background & Hypotheses ➔ Methodology & Data Analysis ➔ Practical Applications & Future Scope
- News & History: Context & Timeline ➔ Key Developments & Main Findings ➔ Broader Implications & Impact
- General / Informational: Overview & Core Concepts ➔ Detailed Breakdown ➔ Key Takeaways

🔍 Multi-Engine Search Replenishment & 3-Tier Web Archive Cascade

Multi-Engine Cascading: Automatically detects search result shortfalls and cascades across Tavily → DuckDuckGo → Bing HTML → Brave Search → Google Aggregated with URL deduplication.
3-Tier Web Archive Cascade: Recovers blocked or deleted pages by cascading through Internet Archive Wayback Machine → Memento Web Federated API (Library of Congress & global archives) → Archive.today.

🛡️ Resilient Web Scraping & Proxy Integration

HTTP/SOCKS5 Proxy Support: Full proxy configuration with password masking (***) and Option 16 live diagnostic tester querying api.ipify.org.
Optional 403 Bypass (curl_cffi): Impersonates standard Chrome TLS/JA3 signatures and client hints to bypass Akamai and Cloudflare anti-bot blocks.
Wayback Machine Fallback: Automatically redirects protected SPAs (Reddit, Stack Overflow, Quora) to the Internet Archive to bypass login gates.
Conditional User-Agent Routing: Automatically swaps to Discordbot/2.0 when scraping guest pages, and shifts to a desktop Chrome UA when user cookies are configured to avoid bot-signature blocks.
Unified Cookie Ingestion: Supports Netscape config/cookies.txt files and universal domain-to-cookie maps to scrape pages behind login walls.

🧹 17+ Built-In Custom Plugins

Pre-configured parser modules designed to clean and extract structured layouts:

HN & Reddit: Captures nested comments threads and recursive tree hierarchies.
Stack Overflow: Cleans code blocks and identifies accepted answers.
arXiv & Google Scholar: Extracts academic metadata, abstracts, and authors metrics.
Finance (Yahoo/SEC EDGAR): Scrapes financial streams and converts data tables to Markdown tables.
ReadTheDocs & Dev.to: Strips navigation sidebars and headers.

🧠 Smart Synthesis & AI Routing

Multi-Provider Failover: Cascades automatically across API keys (Gemini ➡️ OpenAI ➡️ Claude) before dropping to the offline keyword PositionRanker.
Live Search Grounding: Integrates Gemini's live Google search grounding to retrieve cited real-time facts directly in the model.
Local-First Cache: Saves MD5-hashed results in reports/cache/ to skip network requests on recurring topics.

📄 Multi-Format Exporters & Document Delivery

Styled ReportLab PDF: Exports beautiful PDF documents with custom headers, callout boxes, and cell-normalized tables.
Microsoft Word (.docx): Generates structured DOCX reports formatted with headers and source tables.
Markdown & JSON: Generates clean raw Markdown reports and JSON data payloads for programmatic pipelines.

🛠️ Codebase Architecture & File Structure

System Execution Workflow

sequenceDiagram
    participant User
    participant CLI as main.py
    participant Scraper as scraper.py
    participant AI as analyzer.py
    participant Notifier as notifier.py
    
    User->>CLI: Selects Option 1 (Single Scrape)
    CLI->>Scraper: Search Query & Crawl Targets
    Scraper->>Scraper: Run concurrent thread scrapes
    Scraper->>CLI: Return clean list of page contents
    CLI->>AI: Synthesize contents (AI or Local)
    AI->>CLI: Return structured report
    CLI->>Notifier: Export Markdown/JSON & Send Webhooks
    Notifier->>User: Renders Report in Terminal + sends Alerts

Folder Layout

├── install.py            # Platform-independent setup script
├── main.py               # Interactive TUI Controller (Version v1.6.0)
├── config_manager.py     # Reads and writes config.json
├── scraper.py            # Multi-engine search replenishment, 3-tier archive cascade, crawler
├── analyzer.py           # Multi-LLM provider wrappers, dynamic category synthesis, and PositionRank
├── notifier.py           # Markdown/JSON/PDF/DOCX exporter and Discord/Telegram webhook dispatch
├── utils.py              # Proxy manager, URL slug topic extractor, safe_request retries
├── std_plugins/          # Built-in plugins (Hacker News, Reddit, Stack Overflow, YouTube, etc.)
└── tests/                # Isolated unit tests suite (123 unit tests, 100% green)

⚖️ The 5 Strict Design Boundaries (Do NOT Violate)

Focal Harvest adheres to the following core maintainer guardrails:

Lightweight Footprint: Application remains under 5MB and installs in seconds.
Strict TUI Only: Runs entirely in the console—no local React dashboards or local REST servers.
Low Hardware Requirements: Runs smoothly on low-spec student laptops.
No Local AI Downloads: Uses remote APIs (Gemini/OpenAI/Claude) to bypass multi-gigabyte Ollama/Llama downloads.
No SQL Databases: Stores reports in plain JSON and Markdown files for absolute filesystem transparency.

⚖️ Legal Disclaimer & Responsible Use

Focal Harvest is a command-line utility for personal and research purposes.

Public Scraping: Programmatic scraping of public data is legally protected under established U.S. case law. Respect target servers by running crawls with polite rates.
Authenticated Scraping: Do not use active personal accounts with session cookies. Use burner/dedicated accounts exclusively to prevent account suspensions.

📜 License

Distributed under the MIT License. See LICENSE for details.