Skip to content

Block blacklisted IP addresses - #82

Merged
JackLewis-digirati merged 1 commit into
mainfrom
feature/blacklistIps
Nov 16, 2026
Merged

JackLewis-digirati merged 1 commit into
mainfrom
feature/blacklistIps

Conversation

@JackLewis-digirati

@JackLewis-digirati JackLewis-digirati commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this change?

Resolves #77

This PR matches prior work in protaginst to block blacklisted ip's, like the loopback address. It also adds the ability to allow ip's blocked by default if required for local testing.

Configuration Changes

Note

This PR introduces configuration changes.

Service AppSetting Required? Description Default
Builder, Search TextServices:IpBlocking:BlockedIpRanges N Additional ranges to block []
Builder, Search TextServices:IpBlocking:AllowedIpRanges N Ranges permitted despite being blocked; can't open up metadata addresses []

@JackLewis-digirati

Copy link
Copy Markdown
Contributor Author

How it works

Resolves #77. The approach follows the IP blocking added to Protagonist (private-protagonist#204), with the classes ported closely:

text-services Protagonist
IpBlockingSettings OriginStrategySettings
IIpAddressPolicy / IpAddressPolicy IOriginAddressPolicy / OriginAddressPolicy
ConnectionGuard OriginConnectionGuard
IpAddressBlockedException OriginAddressBlockedException

Where it applies

UseIpBlocking(configuration) is applied to the two named HttpClients that fetch caller-supplied content:

  • Builder API – "Resource": sourceUri Manifests, ALTO/hOCR/VTT text, annotation pages.
  • Search API – "pdf": canvas images and pdf-type pages (e.g. cover pages) when generating PDFs.

Other clients (e.g. Builder → Search API cache invalidation, which calls our own configured SearchApiBaseUrl) are unaffected.

The check

ConnectionGuard.ConnectAsync is the SocketsHttpHandler.ConnectCallback, so it runs every time a connection is opened rather than once against the request URL:

  1. Resolve the host (or parse it if it's an IP literal).
  2. If any resolved address is blocked, throw IpAddressBlockedException — a host can't mix allowed and blocked addresses in DNS and hope we pick the allowed one.
  3. Otherwise connect to the addresses that were checked, rather than the hostname, so a second DNS lookup can't send us somewhere else. TLS sits above the callback, so SNI and certificate validation still use the hostname.

Because this happens per connection it covers:

  • every hop of a redirect chain, e.g. http://example.org/redirect → http://127.0.0.1/
  • hostnames whose DNS points at internal addresses, e.g. internal.example.org → 127.0.0.1
  • DNS rebinding between validation and fetch

UseProxy = false on these clients, as connecting via a proxy would mean only the proxy's address is checked.

Blocked ranges

Range Description Can be allowed?
169.254.169.254/32, fd00:ec2::254/128 Cloud instance metadata No
127.0.0.0/8, ::1/128 Loopback Yes
169.254.0.0/16, fe80::/10 Link-local Yes
fc00::/7 IPv6 unique local Yes
0.0.0.0/8, ::/128 Unspecified (reaches loopback) Yes

IPv4-mapped IPv6 addresses (::ffff:127.0.0.1) are checked in both forms, against allowed and blocked ranges alike.

Precedence is: never-allowable → AllowedIpRanges → default + BlockedIpRanges → permitted.

Configuration

Both APIs read the same section; values are CIDR only.

{
  "TextServices": {
    "IpBlocking": {
      "BlockedIpRanges": [ "10.0.0.0/8" ],
      "AllowedIpRanges": [ "127.0.0.0/8", "::1/128" ]
    }
  }
}
Service AppSetting Required? Description Default
Builder, Search TextServices:IpBlocking:BlockedIpRanges N Additional ranges to block []
Builder, Search TextServices:IpBlocking:AllowedIpRanges N Ranges permitted despite being blocked; can't open up metadata addresses []
  • The policy is built at registration, so an invalid range fails startup.
  • A warning is logged if any AllowedIpRanges are configured.
  • Local dev serving fixtures over http://localhost will need the loopback AllowedIpRanges above.

Failure behaviour

The guard doesn't log; the host, resolved address and blocking range are carried on IpAddressBlockedException and logged by the caller (HttpClient wraps it in an HttpRequestException, so callers use IpAddressBlockedException.FindInChain):

  • ResourceFetcher logs a warning and rethrows the unwrapped exception, so the job records Host '{host}' resolves to an IP address that is not permitted — failing the job for a sourceUri, or as a per-page error for text resources.
  • PdfBuilder logs a warning and keeps its existing fallbacks: a blank page for an image, skipping an embedded PDF.

Jobs are still accepted at POST/PUT and fail when processed, as in Protagonist — redirects, DNS changes and URIs inside the Manifest can only be checked at fetch time.

Not in this PR

file:// and s3:// sources are still accepted by ResourceFetcher and PdfBuilder; this is unrelated to IP blocking and will be raised separately.

@JackLewis-digirati
JackLewis-digirati marked this pull request as ready for review November 12, 2026 17:32
@JackLewis-digirati
JackLewis-digirati requested a review from a team as a code owner November 12, 2026 17:32
@JackLewis-digirati
JackLewis-digirati merged commit 2cc6b8a into main Nov 16, 2026
6 of 7 checks passed
@JackLewis-digirati
JackLewis-digirati deleted the feature/blacklistIps branch November 16, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blacklist IPs when fetching content

2 participants