I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
import pandas as pd
left = pd.DataFrame({"key": pd.Series([1, None], dtype="datetime64[ns]")})
right = pd.DataFrame({"key": pd.Series([1], dtype="timedelta64[ns]")})
right_nat = pd.DataFrame({"key": pd.Series([None], dtype="timedelta64[ns]")})
dtypes = left.merge(right, on="key", how="right").dtypes # timedelta64[ns]
dtypes_nat = left.merge(right_nat, on="key", how="right").dtypes # object
print(dtypes.equals(dtypes_nat))
# False Issue Description
If both the left and right frames contain NaT it looks like the merge considers there to be a match between the left and right keys and therefore promotes the key column to a common "object" type. In contrast if there is no NaT, the right merge produces a timedelta64[ns] key column, which feels more correct to me.
Note that in contrast a "left" merge produces an object dtype key column in both cases.
I would expect the dtype to be preserved, arguably also for a "left" merge, irrespective of the presence of NaT.
I would also (perhaps preferably) accept raising when attempting to merge between datetime and timedelta, since they have different units. In the same way that attempting to merge between int and datetime or int and timedelta raises.
I was somewhat surprised that the NaT matches in this merge (no other datetime matches a timedelta), since my interpretation is that a "datetime" NaT represents an unknown point in time, and a "timedelta" NaT represents an unknown duration.
INSTALLED VERSIONS
------------------
commit : b864b8eb54605124827e6af84aa7fbc816cfc1e0
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 6.2.0-37-generic
Version : #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8
pandas : 2.1.0.dev0+1414.gb864b8eb54
numpy : 1.26.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.2.1
Cython : 3.0.0
pytest : 7.4.0
hypothesis : 6.82.2
sphinx : 6.2.1
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.9
jinja2 : 3.1.2
IPython : 8.14.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : 2023.7.0
fsspec : 2023.6.0
gcsfs : 2023.6.0
matplotlib : 3.7.2
numba : 0.58.1
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : 1.2.2
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2023.6.0
scipy : 1.11.1
sqlalchemy : 2.0.19
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.7.0
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.0
pyqt5 : None
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4