Snowpark has a problem / bug that it does not maintain the types between Pandas and Snowpark, nor does it allow to manually set its schema.
df1 = session.sql(sql).to_pandas() df2 = session.create_dataframe(df)
The timestamp field on
TimestampType has become a
I’ve also tried to store the schema and use it, but same results.
df1 = session.sql(sql) df1_schema = df1.schema df1 = df1.to_pandas() df2 = session.create_dataframe(df, df1_schema)
Has anyone managed to deal with it?
This stops me from being able to write the
DataFrame back to the table as it needs to be of
TimestampType rather than
Tried to recreate this in snowpark, it seems that the TimestampType is internally getting changed to LongType when pandas df is converted to snowpark df using create_dataframe() method.
Also, specifying schema parameter in create_dataframe() method w.r.t this scenario is not making any difference.
So, one way is to explicitly change the column to timestamp using to_timestamp() method.
from snowflake.snowpark.functions import sql_expr df1 = session.sql("select * from timestamp_test") df1 = df1.to_pandas() df2 = session.create_dataframe(df1) colCast = df2.withColumn("T", sql_expr("to_timestamp(T::string)")) colCast.show()