RSS Feed Aggregator

A Go-based RSS feed aggregator and API that lets users create accounts, subscribe to feeds, and receive recent posts from the feeds they follow. The app periodically fetches configured RSS sources, parses the items, and stores them in PostgreSQL for later retrieval.

Features

• Create users and generate an API key
• Authenticate requests using the Authorization: ApiKey <key> header
• Add new feeds to the system
• Follow and unfollow feeds
• Poll RSS feeds in the background and store posts
• Expose a JSON API for users and posts
• Use PostgreSQL with generated database access code

Tech Stack

• Go
• Chi router
• PostgreSQL
• sqlc-generated database layer
• Goose SQL migrations
• RSS XML parsing with Go's standard library

Project Structure

• main.go — application startup and router configuration
• scraper.go — background feed polling and RSS parsing logic
• handler_*.go — HTTP handlers for users, feeds, and follows
• models.go — API-facing model conversion helpers
• internal/auth/ — API key extraction logic
• internal/database/ — generated DB queries and models
• sql/SCHEMA/ — SQL migration files
• sql/queries/ — SQL for generated database access

Requirements

• Go 1.25+
• PostgreSQL database
• A valid .env file with runtime configuration

Configuration

Create a .env file in the project root:

PORT=8080
DB_URL=postgres://username:password@localhost:5432/rss_scraper?sslmode=disable

You can also set these directly in your environment before running the app.

Database Setup

Create the PostgreSQL database and apply the migration files in sql/SCHEMA/.

If you are using Goose, the migration flow is typically:

goose up

If you use a different migration workflow, make sure all schema files are applied in sequence:

1. 001_USERs.sql
2. 002_USERs_apikey.sql
3. 003_feeds.sql
4. 004_feeds_follows.sql
5. 005_feeds_lastfetchedat.sql
6. 006_posts.sql

Run the Project

go mod download
go run .

The app starts an HTTP server and launches a background scraper loop using StartScraping(db, 10, time.Minute).

API Endpoints

Base path: /v1

User

• POST /v1/users

  • Body:

    {
      "name": "Alice"
    }

  • Creates a user and returns the created record with an API key.

• GET /v1/users

  • Requires Authorization: ApiKey <api_key>
  • Returns the authenticated user's profile

Feeds

• POST /v1/feeds

  • Requires auth
  • Body:

    {
      "name": "Example Feed",
      "url": "https://example.com/feed.xml"
    }

• GET /v1/feeds

  • Lists feeds in the system

Feed Follows

• POST /v1/feed_follow

  • Requires auth
  • Body:

    {
      "feed_id": "<uuid>"
    }

• GET /v1/feed_follow

  • Requires auth
  • Returns the authenticated user's followed feeds

• DELETE /v1/feed_follow/{feedFollowID}

  • Requires auth
  • Unfollows a feed by its follow record ID

Posts

• GET /v1/posts
  • Requires auth
  • Returns up to 10 recent posts for the authenticated user

Example Request

curl -X POST http://localhost:8080/v1/users \
  -H "Content-Type: application/json" \
  -d '{"name":"Alice"}'

curl -X GET http://localhost:8080/v1/users \
  -H "Authorization: ApiKey YOUR_API_KEY"

curl -X POST http://localhost:8080/v1/feeds \
  -H "Authorization: ApiKey YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"Tech News","url":"https://example.com/rss"}'

How the Scraper Works

The background scraper loops on a timer and asks the database for the next feeds to fetch. For each feed URL it performs an HTTP GET, parses the RSS XML, and inserts posts into the posts table while skipping duplicates.

This logic lives in scraper.go and is started in main.go with:

go StartScraping(db, 10, time.Minute)

Notes

• Feed URLs are required to be unique per feed record.
• Post URLs are also unique to avoid duplicates.
• The project uses go-chi for routing and CORS and relies on environment-based configuration.

License

This project currently does not include a license file. Add one if you intend to distribute it publicly.