If you have a pandas DataFrame or Series, you can import it into Terality with the from_pandas method. You can also convert to Terality DataFrame to a pandas DataFrame with to_pandas.
This is useful when you want to import data with a format not yet supported by Terality: read it with pandas first, then call from_pandas.
import terality as te
import pandas as pd
# Terality does not support the XML format yet, but pandas does.
df_pd = pd.read_xml("data.xml")
# Convert the pandas DataFrame to a Terality DataFrame
df_te = te.from_pandas(df_pd)
# Convert back the Terality DataFrame to a pandas DataFrame
df_pd = df_te.to_pandas()
Using from_pandas is less performant than directly using read_parquet or read_csv, and requires to load the whole dataset into memory first. Prefer reading directly from a file when possible.
Supported storage services
Terality can read files from the local filesystem, and write files back to this local filesystem.
In this configuration, files are directly uploaded to the Terality servers by the Terality client. If you are on a metered or low bandwidth Internet connection, we recommend that you use S3 or Azure Data Lake, as described in the next sections.
The machine running the Terality code must have enough permissions to read from or write objects to the source or destination storage account, as well as generating a user delegation key. These permissions maps to the standard Azure roles "Storage Blob Data Contributor"," Storage Blob Data Owner" or "Storage Blob Data Reader".
This method copies files directly from Azure Data Lake to the Terality servers. Files are not downloaded to the machine running the Terality client.
To use the Azure integration, you first need to install the Terality Azure extras Python package:
Services that are not directly supported by Terality, such as Snowflake or Databricks, often offer an option to export data to an AWS S3 bucket or Azure Data Lake Storage filesystem. You can thus use S3 or Data Lake Storage to transfer data between Terality and many other services.