📡 Signal Tracker
Open-source prediction tracking & accuracy scoring framework.
Track predictions from anyone — media, analysts, politicians, CEOs, AI models. Score accuracy over time. Build leaderboards. Hold the world accountable.
Why?
Everyone makes predictions. Almost nobody tracks them.
- Elon Musk said FSD would be ready "next year" — five years in a row
- Jim Cramer's stock picks are famously inverse-correlated with outcomes
- Media outlets make bold forecasts and quietly move on when they're wrong
Signal Tracker gives you the tools to track all of this systematically.
Created by Crene — the AI-powered prediction intelligence platform that tracks 420+ sources across tech, finance, politics, and geopolitics.
Install
pip install signal-tracker
Quick Start
from signal_tracker import SignalTracker from datetime import date # Initialize tracker = SignalTracker() # Add sources to track elon = tracker.add_source("Elon Musk", source_type="person", category="tech") cramer = tracker.add_source("Jim Cramer", source_type="person", category="finance") # Add predictions claim1 = tracker.add_claim( source=elon, text="Tesla will achieve full self-driving by end of 2025", target_date=date(2025, 12, 31), ) claim2 = tracker.add_claim( source=cramer, text="Netflix will hit $800 by Q2 2025", target_date=date(2025, 6, 30), ) # Verify when outcomes are known tracker.verify(claim1, outcome="wrong", reasoning="FSD not achieved by deadline") tracker.verify(claim2, outcome="correct", reasoning="Netflix reached $820 in May 2025") # Build leaderboard board = tracker.leaderboard() for entry in board.entries: print(f"{entry.rank}. {entry.source.name}: {entry.score.accuracy_score}%") # Save state tracker.save("my_tracker.json")
Features
🎯 Prediction Tracking
Track specific, verifiable claims with time bounds, measurable targets, and clear success criteria.
claim = tracker.add_claim( source=source, text="Bitcoin will reach $150k by end of 2025", target_date=date(2025, 12, 31), category="crypto", )
📊 Accuracy Scoring
Multiple scoring modes — simple, time-windowed, and recency-weighted.
score = tracker.score(source) print(f"{source.name}: {score.accuracy_score}% ({score.correct_claims}/{score.total_claims})")
🏆 Leaderboards
Automatic ranking with risers, fallers, and notable results.
board = tracker.leaderboard(min_claims=3) board.top_accurate # Best predictors board.worst_accurate # Worst predictors board.biggest_risers # Improving fast board.biggest_fallers # Getting worse board.notable_wrongs # High-profile misses
🔍 Claim Extraction
Extract predictions from text — rule-based (fast) or LLM-powered (accurate).
# Rule-based (no API needed) claims = tracker.extract_claims( text="Elon Musk said Tesla will achieve full self-driving by 2025.", source=elon, ) # LLM-powered (bring your own LLM function) def my_llm(prompt: str) -> str: return response.text tracker = SignalTracker(llm_fn=my_llm) claims = tracker.extract_claims(text, source=elon, use_llm=True)
✅ Multi-Model Verification
Consensus-based verification like Crene's 4-LLM system.
tracker.verify_with_consensus(claim, [ {"outcome": "correct", "verifier": "ai:claude", "confidence": 0.9}, {"outcome": "correct", "verifier": "ai:gpt-4", "confidence": 0.85}, {"outcome": "wrong", "verifier": "ai:gemini", "confidence": 0.6}, ])
📈 Claim Quality Scoring
Automatically rate how verifiable a prediction is.
from signal_tracker import QualityScorer scorer = QualityScorer() score = scorer.score(claim) # 0-100
💾 Persistence
JSON file or SQLite for larger datasets.
tracker.save("tracker.json") tracker = SignalTracker.load("tracker.json") from signal_tracker.storage import SQLiteBackend backend = SQLiteBackend("tracker.db")
Architecture
signal-tracker/
├── tracker.py # SignalTracker — main interface
├── models.py # Source, Claim, Verification, ScoreSnapshot
├── scoring.py # AccuracyScorer, QualityScorer
├── extractors.py # ClaimExtractor (rules + LLM)
├── leaderboard.py # Leaderboard engine
└── storage.py # SQLiteBackend
Design principles: Zero required dependencies (stdlib only), bring your own LLM, pluggable storage, plain dataclasses everywhere.
Use Cases
| Use Case | Who It's For |
|---|---|
| Track media accuracy | Journalists, researchers |
| Score analyst predictions | Finance professionals |
| Monitor political promises | Civic organizations |
| Track AI model forecasts | ML engineers |
| Build prediction markets | Developers |
| Personal prediction journal | Anyone |
Roadmap
- v0.1 — Core tracking, scoring, leaderboards, extraction
- v0.2 — REST API server (FastAPI)
- v0.3 — Auto-ingest from RSS, Twitter, YouTube transcripts
- v0.4 — Dashboard UI (React)
- v0.5 — Prediction market integrations (Polymarket, Kalshi)
- v0.6 — Blockchain anchoring for tamper-proof records
About Crene
Crene is an AI-powered prediction intelligence platform that tracks 420+ sources across tech, finance, politics, and geopolitics. We use a 4-LLM consensus system (Claude, GPT-4, Gemini, Grok) to verify claims and score credibility.
Signal Tracker is the open-source framework extracted from Crene's production system. The framework is free — the data is the moat.
Contributing
We welcome contributions! See CONTRIBUTING.md.
git clone https://github.com/Creneinc/signal-tracker.git cd signal-tracker pip install -e ".[all]" pytest
License
MIT License. See LICENSE.