DPY-3032 when batch inserting a dataframe with None values

[bartverweire-ep]

1. What versions are you using?

Give your Oracle Database version, e.g.:

19.25.0.0.0

Give your Oracle Client version (if you are using Thick mode):

(19, 25, 0, 0, 0)

Python version:

platform.platform: Linux-5.15.0-210.163.7.el9uek.x86_64-x86_64-with-glibc2.34
sys.maxsize > 2**32: True
platform.python_version: 3.13.9

oracledb.version: 3.4.1

2. Is it an error or a hang or a crash?

I'm encountering an error when trying to batch insert a data frame into a table

3. What error(s) or behavior you are seeing?

oracledb.exceptions.NotSupportedError: DPY-3032: conversion from Apache Arrow format "n" to Oracle Database is not supported

It's not really clear how to deal with None values in a DataFrame.
Converting None values to empty strings resolves the issue, but probably (?) only for text-like (varchar2, ...) columns.

Output:

==> OracleDB clientversion
(19, 25, 0, 0, 0)
truncate table DEST
   A  B     C
0  X  0  None
1  X  1  None
<class 'pandas.core.frame.DataFrame'>
insert into DEST (A,B,C) values (:1,:2,:3)
Traceback (most recent call last):
  File "/home/oracle/python/tas/test.py", line 35, in <module>
    cursor.executemany(insert_stmt, df)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/oracle/python/tas/.venv/lib64/python3.13/site-packages/oracledb/cursor.py", line 916, in executemany
    manager = self._impl._prepare_for_executemany(
        self,
    ...<2 lines>...
        batch_size,
    )
  File "src/oracledb/impl/base/cursor.pyx", line 454, in oracledb.base_impl.BaseCursorImpl._prepare_for_executemany
  File "src/oracledb/impl/base/batch_load_manager.pyx", line 148, in oracledb.base_impl.BatchLoadManager.create_for_executemany
  File "src/oracledb/impl/base/batch_load_manager.pyx", line 70, in oracledb.base_impl.BatchLoadManager._create
  File "src/oracledb/impl/arrow/dataframe.pyx", line 61, in oracledb.arrow_impl.DataFrameImpl.from_arrow_stream
  File "src/oracledb/impl/arrow/schema.pyx", line 190, in oracledb.arrow_impl.ArrowSchemaImpl.populate_from_schema
  File "/home/oracle/python/tas/.venv/lib64/python3.13/site-packages/oracledb/errors.py", line 199, in _raise_err
    raise error.exc_type(error) from cause
oracledb.exceptions.NotSupportedError: DPY-3032: conversion from Apache Arrow format "n" to Oracle Database is not supported

4. Does your application call init_oracle_client()?

Yes

oracledb.init_oracle_client()

5. Include a runnable Python script that shows the problem.

DDL:

  CREATE TABLE "STAGING"."DEST"
   (    "A" VARCHAR2(15 CHAR),
        "B" NUMBER(1,0),
        "C" VARCHAR2(50 CHAR)
   ) ;

import os

from  dotenv import load_dotenv
import pandas as pd

import oracledb

load_dotenv()

oracledb.init_oracle_client()
print("==> OracleDB clientversion")
print(oracledb.clientversion())

ora_con = oracledb.connect(
    user=os.getenv("DB_USER"),
    password=os.getenv("DB_PW"),
    dsn=os.getenv("DB_DSN")
)


truncate_stmt = f"truncate table DEST"
print(truncate_stmt)
cursor = ora_con.cursor()
cursor.execute(truncate_stmt)

df = pd.DataFrame({"A":["X","X"],"B":[0,1],"C":[None,None]}, index=[0,1])

insert_stmt = f"insert into DEST (A,B,C) values (:1,:2,:3) "

print(df)
print(type(df))

print(insert_stmt)
with ora_con.cursor() as cursor:
    cursor.executemany(insert_stmt, df)

    ora_con.commit()

I didn't find documentation about how to treat None/Null values.
Thank you for your feedback

Bart