📦

News Pipeline

Automated AI research digest with LLM-powered summarization

Active v1.0.0 MIT

[vm] localhost :N/A

Tech Stack

Go Temporal Claude CLI Mattermost systemd

Requirements

• Go 1.21+
• Temporal Server
• Claude CLI
• Mattermost (optional)

Features

✓ Multi-source paper fetching
✓ Haiku summarization
✓ Sonnet relevance ranking
✓ Mattermost channel posting
✓ Markdown digest generation
✓ Optional TTS audio
✓ Systemd timer scheduling

Staying Current with AI Research

The AI/ML field moves fast. Papers drop daily across multiple platforms:

arXiv dumps new preprints constantly
HuggingFace Daily Papers curates trending work
Papers With Code tracks implementations

Manually checking each source is tedious. News Pipeline automates the entire flow: fetch, summarize, rank, and deliver.

Architecture

A Temporal workflow orchestrates the daily pipeline:

┌─────────────────────────────────────────────────────────────┐
│                    TEMPORAL WORKFLOW                         │
│    Fetch Sources → Summarize → Rank → Post → Archive        │
└─────────────────────────────────────────────────────────────┘
        │                   │                   │
        ▼                   ▼                   ▼
   ┌─────────┐        ┌─────────┐        ┌─────────┐
   │  arXiv  │        │HuggingFace│       │PapersCode│
   │   API   │        │   Daily  │        │   API   │
   └─────────┘        └─────────┘        └─────────┘
                            │
                            ▼
                    ┌───────────────┐
                    │  Claude CLI   │
                    │ Haiku+Sonnet  │
                    └───────────────┘
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
        ┌─────────┐   ┌─────────┐   ┌─────────┐
        │Mattermost│   │ Markdown│   │  TTS    │
        │ Channels│   │ Digest  │   │ Audio   │
        └─────────┘   └─────────┘   └─────────┘

Workflow Steps

Step	Model	Purpose
Fetch	N/A	Pull papers from 3 sources
Summarize	Haiku	Generate concise summaries
Rank	Sonnet	Score relevance, add labels
Post Raw	N/A	Dump to Mattermost raw channel
Post Digest	N/A	Curated top items to digest channel
Archive	N/A	Save markdown to `~/.news-pipeline/digests/`

Deployment

Runs as a systemd service with timer-based scheduling:

# Install (builds + installs systemd units)
./install.sh

# Triggers at 6 AM and 6 PM daily
sudo systemctl status news-pipeline-trigger.timer

# Worker runs continuously
sudo systemctl status news-pipeline-worker

Manual trigger:

./bin/news-trigger --wait

Output

Digests are saved to ~/.news-pipeline/digests/ as YYYY-MM-DD.md:

# AI/ML News Digest - 2025-12-28

## Top Papers

### 1. [Paper Title](https://arxiv.org/abs/...)
**Relevance**: 9/10 | **Labels**: transformers, efficiency
**Summary**: One-paragraph summary from Haiku...

### 2. [Another Paper](https://huggingface.co/papers/...)
...

Optional TTS generates audio summaries for top items.

Configuration

Variable	Default	Description
`TEMPORAL_ADDRESS`	localhost:7233	Temporal server
`MATTERMOST_URL`	chat.homelab.com	Mattermost server
`MATTERMOST_TOKEN`	(required)	Auth token
`NEWS_DIGEST_DIR`	~/.news-pipeline/digests	Output directory
`CLAUDE_PATH`	~/.nvm/…/claude	Claude CLI path

Summary

Benefit	Description
Automated	No manual checking of multiple sources
Curated	LLM ranking surfaces relevant papers
Delivered	Mattermost integration for team sharing
Archived	Markdown digests for future reference