Skip to content

[polars] Fetch the FRED CSV with requests before parsing it - #647

Merged
mmcky merged 1 commit into
mainfrom
claude/funny-hamilton-3j8jve
Oct 13, 2026
Merged

mmcky merged 1 commit into
mainfrom
claude/funny-hamilton-3j8jve

Conversation

@mmcky

@mmcky mmcky commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Problem

The weekly cache build has failed since polars 2.0.0 was released on 2026-10-06. It passed on 2026-10-05 and failed on 2026-10-12. polars.md hits the 600 s cell timeout on this line:

data = pl.read_csv(fred_url, try_parse_dates=True)

The lecture runs !pip install --upgrade polars yfinance, so CI gets the newest polars. The zh-cn edition fails in the same place, and its execution report names this cell.

Cause

polars 2.0 reads a URL through its object store. It sends a HEAD request for the file size, then ranged GETs. FRED's fredgraph.csv answers HEAD with content-length: 0, although the body is 13,809 bytes. Ranged GETs work (206, content-range: bytes 0-3/13809). polars 2.0 then stalls instead of raising.

Run against the real URL:

polars pl.read_csv(fred_url, try_parse_dates=True)
1.44.2 ok, 918 rows, 0.23 s
2.0.0 still running when killed after 90 s

Fix

Download the CSV with requests and pass the bytes to pl.read_csv. This gives 918 rows with the observation_date column parsed as Date under both 1.44.2 and 2.0.0. The sentence above the cell said read_csv fetches the URL itself, so it now describes the download.

This also matches the approach the pandas lecture already takes in its FRED section.

Testing

I ran all 52 code cells of the patched lecture under polars 2.0.0 and yfinance 1.7.0, with live FRED and Yahoo Finance data. All passed. The GitHub-hosted test_pwt.csv cells, which are untouched, also read fine under 2.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wyb6EzbNzY5a8uuJPnDtUb


Generated by Claude Code

polars 2.0.0 (2026-10-06) reads a URL through its object store: it
sends a HEAD request for the file size, then ranged GETs. FRED's
fredgraph.csv answers HEAD with Content-Length: 0 (the body is ~14 kB),
and polars 2.0 then stalls instead of erroring. The lecture installs
the newest polars, so the FRED cell has hung until the 600 s cell
timeout since the release, failing the weekly cache build on
2026-10-12 (it passed on 2026-10-05).

Download the file with requests and pass the bytes to pl.read_csv.
This reads the same 918 rows with the date column parsed under both
polars 1.44.2 and 2.0.0. The sentence above the cell no longer says
read_csv fetches the URL itself.

All 52 code cells of the lecture run under polars 2.0.0 with live FRED
and Yahoo Finance data.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Wyb6EzbNzY5a8uuJPnDtUb
Copilot AI balanced review requested due to automatic review settings October 13, 2026 04:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The fix is correct, but the newly added import requests is placed mid-cell, deviating from the lecture's convention of grouping imports at the top.

1 open finding
What changed in this PR

This PR fixes the weekly cache build, which has been failing since polars 2.0.0 (released 2026-10-06). In the polars lecture, pl.read_csv(fred_url, ...) relies on polars reading the URL directly through its object store; under polars 2.0, FRED's fredgraph.csv returns content-length: 0 on HEAD, causing the read to stall until the 600 s cell timeout. The fix downloads the CSV with requests first and passes the bytes to pl.read_csv, which aligns with how the pandas lecture already fetches FRED data.

Changes:

  • Download the FRED CSV via requests.get(...) and pass r.content to pl.read_csv instead of letting polars fetch the URL.
  • Update the surrounding prose so it describes the requests download rather than read_csv fetching the URL.
File Description
lectures/​polars.md Fetch the FRED CSV with requests and pass the bytes to pl.read_csv; update the explanatory sentence accordingly.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lectures/polars.md
@github-actions

Copy link
Copy Markdown

@mmcky
mmcky merged commit b0c58ab into main Oct 13, 2026
5 checks passed
@mmcky
mmcky deleted the claude/funny-hamilton-3j8jve branch October 13, 2026 04:34
@mmcky

mmcky commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

✅ Translation sync completed (zh-cn)

Target repo: QuantEcon/lecture-python-programming.zh-cn
Translation PR: QuantEcon/lecture-python-programming.zh-cn#108
Files synced (1):

  • lectures/polars.md

@mmcky

mmcky commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

✅ Translation sync completed (fr)

Target repo: QuantEcon/lecture-python-programming.fr
Translation PR: QuantEcon/lecture-python-programming.fr#97
Files synced (1):

  • lectures/polars.md

@mmcky

mmcky commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

✅ Translation sync completed (fa)

Target repo: QuantEcon/lecture-python-programming.fa
Translation PR: QuantEcon/lecture-python-programming.fa#171
Files synced (1):

  • lectures/polars.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants