[pearu]

Rationale for this change

CSV columns typed as date32 or date64 are parsed only from the ISO-8601 format, and ConvertOptions::timestamp_parsers applies only to timestamp columns, so other date formats need reading as a timestamp and a cast.

What changes are included in this PR?

• Date columns try timestamp_parsers after the ISO-8601 parser fails, and floor the parsed timestamp to the day boundary, as a cast from timestamp to date does. Values outside the range of date32/date64 are rejected.
• strptime parsers without a date directive are skipped, so that e.g. "%H:%M:%S" does not turn a time into 1900-01-01.
• With the ISO-8601 parser in timestamp_parsers, fractional seconds are accepted and discarded.
• Type inference of date columns is unchanged and uses ISO-8601 only, so a value with a time of day is not inferred as a date.

This PR is the date part of #50146.

Are these changes tested?

Yes, Date32Conversion.UserDefinedParsers and Date64Conversion.UserDefinedParsers in C++, and test_dates_with_timestamp_parsers in Python.

Are there any user-facing changes?

Yes, timestamp_parsers now also applies to date32/date64 columns given in column_types. This is documented in the C++ CSV docs and the ConvertOptions docstring.

AI usage disclosure

Developed with AI assistance (Claude Code): the implementation, tests, documentation and this description were AI-generated under my direction and reviewed line by line by me. I own and can debug these changes. No copyrighted material is included.

• GitHub Issue: [C++][Python] Use timestamp_parsers as a fallback for CSV date columns #54650

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically closed because you currently have 4 open pull requests, which is more than the limit of 3.

Due to the increase in pull requests opened by AI bots, and in order to keep the review queue manageable, Apache Arrow limits contributors without repository access to at most 3 concurrently open pull requests. This helps make sure each pull request gets the attention it needs and that work in progress does not go stale.

Once one of your other open pull requests has been merged or closed, you are welcome to reopen this one.

See also:

• Contribution Guidelines - Limit on concurrent pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #54650 has been automatically assigned in GitHub to PR creator.

[pearu]

@kou @ianmcook could you re-open this PR? Once opened, I'll rebase and it will be ready for review.

[kou]

Could you rebase on main?

[pearu]

Could you rebase on main?

Done. Please review.

[kou]

If we have a separated date_parsers, can we simplify the current implementation?

[pearu]

I used an AI agent to analyze the suggested alternative for its pros and cons. Assuming the existing tests stay as representative samples of the feature, with a separate date_parsers:

Could go:

• StrptimeFormatCanProduce
• make_date_time_converter and the is_type_inference parameter of Converter::Make, but only if date_parsers is also used for type inference, which means a column matching a datetime format listed in both options would be inferred as a date (inference tries dates before timestamps); keeping them means date_parsers differs from timestamp_parsers in not affecting inference

Would come:

• the new option in C++ ConvertOptions and in pyarrow (constructor, property, equality, pickling, repr), with its tests and docs, likely later also in GLib and R
• if used for type inference: inference of non-ISO date columns as dates, with its own tests
• the same again with time_parsers for GH-54651: [C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54662

Everything else would stay, as the tests use datetime formats, ISO8601() with fractional seconds and custom parsers, which the new option would have to accept as timestamp_parsers does; only the tests that a format without a date directive is not tried (and their counterparts in #54662) would be dropped, as that case only arises when date and timestamp columns share one option. Users with formats that contain both a date and a time would list them in both options.

The API change is the more important point: a new public option that is documented and kept from then on. So the implementation would get somewhat simpler, at the cost of a larger API. On this balance I'd keep the current approach, unless you see an API reason for the separate option.

[kou]

Thanks for analyzing. I also analyzed the date_parsers approach with Claude Code. It seems that other systems such as DuckDB and Apache Spark have a separated date format option. (Time format option isn't reused.)

@pitrou @jorisvandenbossche Do you have any opinion how to process custom date format columns in CSV reader?

The following text is generated by Claude Code after my analysis. (I changed some generated parts manually.)

---

Proposal

• Add ConvertOptions::date_parsers. Its type is the same as timestamp_parsers: std::vector<std::shared_ptr<TimestampParser>>. We don't need a new parser class. pyarrow accepts format strings and ISO8601 like timestamp_parsers.
• date_parsers replaces the built-in ISO-8601 parser like timestamp_parsers does:
  • If date_parsers is empty, we use only the built-in ISO-8601 parser (YYYY-MM-DD) as before. No behavior change.
  • If date_parsers isn't empty, we use only date_parsers. Users can include ISO8601() to accept ISO-8601 values too, e.g. [ISO8601(), "%m/%d/%Y"]. ISO8601() also accepts date-only values such as 2000-01-01.
• A parsed timestamp is floored to the day boundary. The time part is truncated only when users put a parser that accepts a time part into date_parsers, e.g. "%m/%d/%Y %H:%M" or ISO8601(). With "%m/%d/%Y", 03/15/2020 14:30 is rejected because of the trailing characters.
• Values with a zone offset are rejected as the current implementation does.
• date_parsers is also used for type inference (the date stage) like timestamp_parsers is used for the timestamp stages.
• timestamp_parsers isn't changed.
• We can use the same approach for time columns (GH-54651: [C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54662) with time_parsers.

Rationale

1. Consistency with other systems: DuckDB (dateformat), Spark (dateFormat) and readr (locale(date_format) / col_date(format)) have a separate format option for dates and use it for type inference too. With a custom format, they truncate the time part only when the format has a time part. I couldn't find any system that applies timestamp formats to date columns. (DuckDB and Spark also truncate ISO-8601 date-time values for date columns by default. This proposal keeps the current stricter behavior for the built-in ISO-8601 parser.)
2. No change to an existing API: With the current approach, timestamp_parsers starts affecting date (and later time) columns, and values that were rejected are now accepted with the time part truncated. This can't be reverted after a release. A separate option is purely additive.
3. Consistency with the CSV reader: The CSV reader rejects lossy conversions for typed columns. For example, 2000-01-01 00:00:00.5 for timestamp[s], 12:00:00.5 for time32[s], 1.25 for decimal128(5, 1) and 300 for int8 are all rejected. Truncating the time part silently because of parsers specified for timestamp columns isn't consistent with this. Truncation explicitly requested via date_parsers is fine.
4. It covers the reported use cases: [C++] Use timestamp parsers for date32() CSV parsing #26224, [C++] Unable to read date64 or date32 in specific format from CSV #28303 and [Python] Allow ConvertOptions.timestamp_parsers for date types #33357 (date-only custom formats) work by putting the formats into date_parsers. [Python] ArrowInvalid: CSV conversion error to date32[day]: invalid value '2000-01-01 00:00:00' #37180 (2000-01-01 00:00:00 for a date column) works with ISO8601() or "%Y-%m-%d %H:%M:%S" in date_parsers.
5. Simpler implementation:
   • StrptimeFormatCanProduce isn't needed. The problem it solves is that parsers for timestamp columns are also applied to date columns. It doesn't happen with date_parsers. If users put a wrong format such as "%H:%M:%S" into date_parsers, they get what they specified.
   • is_type_inference isn't needed because type inference uses the same converter. We don't need to change the public Converter::Make().
   • SplitISO8601Subseconds isn't needed. ISO8601() rejects values with fractional seconds for a date column like it does for a timestamp[s] column.
   • We don't need to check TimestampParser::kind(). Custom TimestampParser implementations work in the same way.
   • The date converter can use the same structure as MakeTimestampConverter(): the built-in parser if no parsers are given, the given parsers otherwise.

Changes from the current PR

• Use date_parsers instead of timestamp_parsers for date columns, and replace the built-in parser instead of falling back from it.
• Add date_parsers to C++ ConvertOptions, pyarrow (constructor, property, equality, pickling, repr) and the docs. (We don't need to change R (CsvConvertOptions$create()) and GLib (GArrowCSVReadOptions) in this PR. We can work on it as separated tasks). pyarrow.dataset doesn't need any change because it uses ConvertOptions as-is.
• Add tests for type inference with date_parsers, and remove StrptimeFormatCanProduce, SplitISO8601Subseconds, is_type_inference and their tests.
• The flooring, the date32/date64 range checks, the zone offset rejection and most of the tests (custom formats, pre-epoch values, out of range values, leap seconds, impossible dates and so on) can be reused.

[pitrou]

@kou I didn't read the Claude-generated code but I agree that a separate option would be more flexible without adding any complication.

[pearu]

@kou @pitrou @ianmcook Thanks, I'll follow the date_parsers proposal. The plan:

• A new sub-issue of CSV reader cannot parse dates or times #41488 and a new PR from main for ConvertOptions::date_parsers (C++, pyarrow, docs; R and GLib as follow-ups), as proposed: same type as timestamp_parsers, replaces the built-in ISO-8601 parser when non-empty, parsed timestamps are floored to the day, zone offsets are rejected, and it is also used for type inference. timestamp_parsers stays as is.
• time_parsers follows in a second sub-issue and PR, depending on the feedback here.
• This PR will be closed when the new one is ready; GH-54651: [C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54662 and the issues [C++][Python] Use timestamp_parsers as a fallback for CSV date columns #54650 and [C++][Python] Use timestamp_parsers as a fallback for CSV time columns #54651 will be closed once the new PRs have landed.

Two behaviours differ from what @ianmcook asked for in #50146, so please object if you see a problem:

• ISO8601() in date_parsers rejects fractional seconds, as it does for timestamp[s] (GH-41488: [C++][Python] Apply timestamp_parsers as fallback when parsing CSV date and time columns #50146 accepts and discards them).
• In time_parsers, a date-only ISO-8601 value gives 00:00:00: the user gets what the option specifies. The retry in seconds for dates outside the time unit's range stays.

Since type inference uses date_parsers: with ISO8601() in it, ISO date-time columns are inferred as dates (dates are tried before timestamps; DuckDB does the same), and without it, ISO dates are inferred as timestamps. Both will be documented. Values are whitespace-trimmed, as with the built-in date parser.