rune/NewsAgent
1
0
Fork 0
Code Issues Pull Requests Activity
Files
main
NewsAgent/README.md
Rune Olsen 0ed89a7045 Cost control
2026-01-27 09:25:11 +01:00

8.9 KiB
Raw Permalink Blame History

News Agent

An AI-powered daily tech news aggregator that fetches articles from RSS feeds, filters them by relevance to your interests, generates AI summaries, and emails you a beautifully formatted digest every morning.

Features

  • RSS Aggregation: Fetches from 15+ tech news sources covering Development, Self-hosting, Enterprise Architecture, and Gadgets
  • AI Filtering: Uses OpenRouter AI to score articles based on your interests (0-10 scale)
  • Smart Summarization: Generates concise 2-3 sentence summaries of each relevant article
  • Beautiful Emails: HTML email with responsive design, categorized sections, and relevance scores
  • Cost Tracking: Automatic tracking and display of per-run and cumulative costs in emails
  • Deduplication: SQLite database prevents duplicate articles
  • Automated Scheduling: Runs daily at 07:00 Europe/Oslo time via systemd timer
  • Production Ready: Error handling, logging, resource limits, and monitoring

Architecture

news-agent/
├── src/
│   ├── aggregator/     # RSS feed fetching
│   ├── ai/             # OpenRouter client, filtering, summarization
│   ├── storage/        # SQLite database operations
│   ├── email/          # Email generation and sending
│   └── main.py         # Main orchestrator
├── config.yaml         # Configuration
├── .env               # Secrets (API keys)
└── systemd/           # Service and timer files

Prerequisites

  • Fedora Linux (or other systemd-based distribution)
  • Python 3.11+
  • SMTP Server (your own mail server or service like Gmail, Outlook, etc.)
  • OpenRouter API Key (get from https://openrouter.ai)

Installation

1. Clone/Copy Project

# Copy this project to your home directory
mkdir -p ~/news-agent
cd ~/news-agent

2. Install Python and Dependencies

# Install Python 3.11+ if not already installed
sudo dnf install python3.11 python3-pip

# Create virtual environment
python3.11 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -e .

3. Configure News Agent

# Run setup script to copy config files
./setup.sh

# OR manually copy files:
cp .env.example .env
cp config.yaml.example config.yaml

Edit .env file:

nano .env

Add your credentials (NO quotes needed):

# OpenRouter API Key
OPENROUTER_API_KEY=sk-or-v1-...your-key-here...

# SMTP Credentials for your mail server
SMTP_USERNAME=your-email@yourdomain.com
SMTP_PASSWORD=your-smtp-password

Edit config.yaml file:

nano config.yaml

Update the email section:

email:
  to: "your-email@example.com"  # Where to receive the digest
  from: "news-agent@yourdomain.com"  # Sender address
  smtp:
    host: "mail.yourdomain.com"  # Your mail server hostname
    port: 587  # 587 for TLS, 465 for SSL
    use_tls: true  # true for port 587
    use_ssl: false  # true for port 465

Common SMTP Settings:

  • Your own server: Use your mail server hostname and credentials
  • Gmail: smtp.gmail.com:587, use App Password
  • Outlook/Office365: smtp.office365.com:587
  • SendGrid: smtp.sendgrid.net:587, use API key as password

Important settings to adjust:

  • ai.model: Use openai/gpt-4o-mini (recommended, ~$0.05/day) or see MODELS.md
  • ai.filtering.min_score: Lower = more articles (5.5 recommended, was 6.5)
  • ai.filtering.max_articles: Maximum articles per digest (default: 15)
  • email.to: Your email address
  • email.from: Sender address
  • smtp: Your mail server settings
  • RSS sources (add/remove feeds)
  • Your interests for AI filtering

4. Test Run

# Activate virtual environment
source .venv/bin/activate

# Run manually to test
python -m src.main

Check:

  • Console output for progress
  • Logs in data/logs/news-agent.log
  • Your email inbox for the digest

5. Set Up Systemd Timer

# Copy systemd files to user systemd directory
mkdir -p ~/.config/systemd/user
cp systemd/news-agent.service ~/.config/systemd/user/
cp systemd/news-agent.timer ~/.config/systemd/user/

# Edit service file to update paths if needed
nano ~/.config/systemd/user/news-agent.service

# Reload systemd
systemctl --user daemon-reload

# Enable and start timer
systemctl --user enable news-agent.timer
systemctl --user start news-agent.timer

# Check timer status
systemctl --user list-timers
systemctl --user status news-agent.timer

Enable lingering (allows user services to run when not logged in):

sudo loginctl enable-linger $USER

Usage

Manual Run

cd ~/news-agent
source .venv/bin/activate
python -m src.main

Check Status

# Check timer status
systemctl --user status news-agent.timer

# View logs
journalctl --user -u news-agent.service -f

# Or check log file
tail -f data/logs/news-agent.log

Trigger Manually

# Run service immediately (without waiting for timer)
systemctl --user start news-agent.service

View Last Run

systemctl --user status news-agent.service

Configuration

RSS Sources

Add or remove sources in config.yaml:

sources:
  rss:
    - name: "Your Source"
      url: "https://example.com/feed.xml"
      category: "tech"  # tech, development, selfhosting, architecture, gadgets

AI Configuration

Models (from cheap to expensive):

  • google/gemini-flash-1.5-8b - Fast, cheap, good quality (recommended)
  • google/gemini-2.0-flash-exp:free - Free, experimental (good for testing)
  • meta-llama/llama-3.1-8b-instruct:free - Free, decent quality
  • anthropic/claude-3.5-haiku - Better quality, slightly more expensive
  • openai/gpt-4o-mini - Good quality, moderate price

See MODELS.md for detailed comparison and selection guide. Full list at: https://openrouter.ai/models

Filtering:

ai:
  filtering:
    enabled: true
    min_score: 6.5  # Articles below this score are filtered out
    max_articles: 15  # Maximum articles in daily digest

Interests:

ai:
  interests:
    - "Your interest here"
    - "Another topic"

Schedule

Change time in ~/.config/systemd/user/news-agent.timer:

[Timer]
OnCalendar=07:00  # 24-hour format

Then reload:

systemctl --user daemon-reload
systemctl --user restart news-agent.timer

Troubleshooting

No Email Received

  1. Check logs:

    journalctl --user -u news-agent.service -n 50

  2. Check Postfix:

    sudo systemctl status postfix
sudo tail -f /var/log/maillog

  3. Test email manually:

    echo "Test email" | mail -s "Test" your-email@example.com

API Errors

Error: "No endpoints found for [model]"

  • The model name is incorrect
  • Check correct model names in MODELS.md
  • Update config.yaml with correct model ID

Other API errors:

  1. Verify API key in .env
  2. Check OpenRouter credit balance: https://openrouter.ai/credits
  3. Check rate limits in logs
  4. Try a different model (see MODELS.md)

Service Not Running

# Check service status
systemctl --user status news-agent.service

# Check timer status
systemctl --user status news-agent.timer

# View detailed logs
journalctl --user -xe -u news-agent.service

Database Issues

# Reset database (WARNING: deletes all history)
rm data/articles.db
python -m src.main

Cost Estimation

Using google/gemini-flash-1.5-8b (recommended):

  • Daily: ~$0.05-0.15 (varies by article count)
  • Monthly: ~$1.50-4.50
  • Yearly: ~$18-54

Free Options:

  • google/gemini-2.0-flash-exp:free - Completely free (experimental)
  • meta-llama/llama-3.1-8b-instruct:free - Completely free

Factors affecting cost:

  • Number of new articles
  • Content length
  • Filtering threshold (lower = more articles = higher cost)

Maintenance

Update Dependencies

cd ~/news-agent
source .venv/bin/activate
pip install --upgrade -e .

View Statistics

# Check database
sqlite3 data/articles.db "SELECT COUNT(*) FROM articles;"
sqlite3 data/articles.db "SELECT category, COUNT(*) FROM articles GROUP BY category;"

Logs Rotation

Logs automatically rotate at 10MB with 5 backups (configured in config.yaml).

Advanced Features

Add API News Sources

Extend src/aggregator/api_fetcher.py to support NewsAPI, Google News API, etc.

Customize Email Template

Edit src/email/templates/daily_digest.html for different styling.

Web Dashboard

Add Flask/FastAPI to create a web interface for viewing past digests.

Contributing

This is a personal project template. Feel free to fork and customize to your needs.

License

MIT License - Free to use and modify

Support

For issues with:

Credits

Built with:

  • Python 3.11+
  • OpenRouter AI (https://openrouter.ai)
  • Feedparser, Jinja2, Pydantic, and other open-source libraries
Reference in New Issue View Git Blame Copy Permalink