datetime: min date (0001-01-01 00:00:00) can't be converted to local timestamp

[DaveJohansen]

BPO	31212
Nosy	@abalkin, @vstinner, @cjerdonek, @akulakov

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2017-08-15.17:26:32.338>
labels = ['extension-modules', 'type-bug', '3.8']
title = "datetime: min date (0001-01-01 00:00:00) can't be converted to local timestamp"
updated_at = <Date 2021-07-08.18:00:58.315>
user = 'https://bugs.python.org/DaveJohansen'

bugs.python.org fields:

activity = <Date 2021-07-08.18:00:58.315>
actor = 'andrei.avk'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2017-08-15.17:26:32.338>
creator = 'Dave Johansen'
dependencies = []
files = []
hgrepos = []
issue_num = 31212
keywords = []
message_count = 13.0
messages = ['300303', '300338', '300339', '300389', '300390', '300391', '300392', '300393', '300394', '300395', '300396', '385937', '397159']
nosy_count = 6.0
nosy_names = ['belopolsky', 'vstinner', 'chris.jerdonek', 'Dave Johansen', 'andrei.avk', 'skagan_NRAO']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31212'
versions = ['Python 3.8']

[DaveJohansen]

This worked in Python 3.6.0 and before:

from datetime import datetime
d = datetime(1, 1, 1, 0, 0, 0)
d.timestamp()

The error output is:

ValueError: year 0 is out of range

But it used to return -62135658000.0.

Appears to be related to https://bugs.python.org/issue29921

[vstinner]

This error is a side effect of the implementation of the PEP-495. In your timezone, datetime(1, 1, 1, 0, 0, 0).timestamp() creates an internal timestamp with year=0. The problem is that the internal function ymd_to_ord() doesn't support year=0:

/* This is incorrect if year <= 0; we really want the floor
 * here.  But so long as MINYEAR is 1, the smallest year this
 * can see is 1.
 */
assert (year >= 1);

This worked in Python 3.6.0 and before: (...)

The question is if the result was correct before?

[vstinner]

Note: My change was the commit 5b804b2fb0eaa2bacb4afcfe4cfa85b31475e87f which prevented a crash, bpo-29100.

[DaveJohansen]

That's a valid datetime (i.e. within the min and max values) and tzinfo is None so I think it's completely reasonable to assume that timestamp() will return the correct value.

[abalkin]

The question is whether -62135658000.0 is the "correct" or even meaningful value:

>>> datetime.utcfromtimestamp(-62135658000.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year is out of range

(I ran the above in Python 2.7 to avoid any 3.x datetime innovations.)

I don't see much of a value in allowing datetime.timestamp() to produce values that correspond to out of bounds datetimes in UTC. In most cases this will only result in error later on in the program, or worse an error passing undetected.

What is the use case for datetime.min.timestamp()? I suspect OP encountered this issue in an overly aggressive edge case testing and not in a real user scenario.

[abalkin]

On the second thought, a reasonable design can use datetime.min/max as placeholders for unknown times far in the past/future compensating for the lack datetime ±inf. In this use case, it may be annoying to see errors from timestamp() instead of some ridiculously large values.

I am -0 on making this change. If anyone is motivated to produce a patch - I'll review it, but I am not going to write it myself.

I am marking this "tests needed" because we need a test complete with setting an appropriate timezone that demonstrates this issue.

[DaveJohansen]

The use case was parsing user input of ISO 8601 date strings and converting them to UNIX epochs. The input "0001-01-01T00:00:00" is valid, parses to a valid datetime and it seems like a reasonable expectation that all of the functions should work on a valid datetime (especially when they worked in earlier versions of Python).

[DaveJohansen]

Ok, so I understand the issue now. timestamp() for naive datetime instances applies the local timezone offset ( https://docs.python.org/3.6/library/datetime.html#datetime.datetime.timestamp ). This is surprising because naive datetime instances usually are just that and don't make assumptions the timezone, but I guess that the behavior is documented and I was just unaware of it. It still seems like this shouldn't give an error (especially when the timezone of the local machine is UTC), but I guess that it is what it is.

[abalkin]

It your use case, the input "0001-01-01T00:00:00" was in what timezone?

I suspect what you want is

>>> datetime.min.replace(tzinfo=timezone.utc).timestamp()
-62135596800.0

And not -62135658000.0. If you work with naive datetimes and just want to roundtrip between datetimes and numbers, you should not use a timezone-dependent conversion - attach UTC timezone before calling .timestamp() as shown above. You probably don't want your program to produce different numbers for the same "0001-01-01T00:00:00" input depending on where it is run.

[abalkin]

BTW, I was originally against introducing .timestamp() method and this issue illustrates why it is problematic: people use it without understanding what it does. If you want the distance between datetime.min and datetime(1970,1,1) in seconds - compute it explicitly:

>>> (datetime.min - datetime(1970,1,1)) / timedelta(0, 1)
-62135596800.0

If you don't want to loose precision is roundtriping between datetimes and numbers - use microseconds:

>>> (datetime.min - datetime(1970,1,1)) // datetime.resolution
-62135596800000000

It is perfectly fine to work with naive datetimes and ignore the timezone issues when all your timestamps are in one timezones, but if you choose to ignore timezones - don't use .timestamp().

[abalkin]

It still seems like this shouldn't give an error (especially when the timezone of the local machine is UTC)

This is the part where I agree with you. I suspect the error in the UTC case is an artifact of PEP-495 fold calculations that require probing times ±24 hours around the time being converted. For the times before timezones were invented (late 1800s?) we can safely assume that fold=0 always. To be safe, I would set the fold cut-off at the year 1000.

[skaganNRAO]

I encountered this issue in Python 3.8. I consider it to be a bug because it's an instance of a class-defined constant (datetime.min) not working with a method of that class (timestamp) when all other values that the class could take work with the method AND the constant works with all of the class' other methods. And it has practical implications: As belopolsky said above, "a reasonable design can use datetime.min/max as placeholders for unknown times far in the past/future compensating for the lack [of] datetime ±inf."

Since datetime.min lies so close to the edges of datetime's value-space, perhaps it should be made offset-aware and placed in UTC so that it stops breaking timestamp() when the user is in the wrong timezone. Alternatively, there could be a note in the documentation for datetime.timestamp() about this edge case. Assuming similarly bizarre behavior happens at datetime.max for one or more datetime methods (and for consistency's sake), it should probably be put in UTC as well.

[akulakov]

Perhaps min and max should be moved 24 hours up and down respectively? This would make their names more accurately reflect valid range of the class and also allow them to be used as effective -inf +inf.

[jbhanks]

This just bit me and from a user experience perspective, I agree that it seems like a bug. At a minimum, it would probably make sense for datetime.min to automatically set to UTC. However I'm not sure it would make sense to have datetime.datetime(1, 1, 1, 0, 0, 0).timestamp() tack on a timezone, because that isn't a reasonable thing to expect either if one really means datetime(1, 1, 1, 0, 0, 0) in the system timezone.
Perhaps there could be a suggestion in the error, perhaps something like:

This datetime is out of range for the current system timezone. If this is intended as a placeholder date, you can avoid this error by specifying the timezone as UTC

[StanFromIreland]

This is also relevant for the Python implementation.

Error message has improved too, year must be in 1..9999, not 0.

There was no real conclusion previously, what do we want to do?

[StanFromIreland]

The problem isn't really even in datetime;

cpython/Lib/_pydatetime.py

Line 1991 in 563ab5c

y, m, d, hh, mm, ss = _time.localtime(u)[:6]

This returns a y=0 causing the error, not sure why though, I'm on UTC+0. If you either comment out the check in _check_date_fields or add a if y==0: y=1 then the correct timestamp is returned: -62167217679.0 (Verified on timestamp.online)

[rafa-br34]

This is a bit out of topic, but could some kind soul recommend me some way to "sanitize" invalid parameters? I'm trying to parse the DateAndTime 11-byte object from RFC 2579 (page 18) but the issue is that sometimes invalid data is returned from servers/devices. For example, sometimes I might get all zero values, other times completely random values.
I wish datetime automatically did the maths to accept a range of arbitrary values but it uses constants.
My current approach is to just verify that the month, day, and hour parameters are accurate and presume that the rest will also be accurate.