Documentation
From/to pandas
Blob
Building a Terality DataFrame from a pandas DataFrame
If you have a
pandas DataFrame or Series, you can import it into Terality with the from_pandas method. You can also convert to Terality DataFrame to a pandas DataFrame with to_pandas.This is useful when you want to import data with a format not yet supported by Terality: read it with pandas first, then call
from_pandas. 1
import terality as te
2
import pandas as pd
3
4
df_pd = pd.DataFrame({"a": ["he", "llo", "wor", "ld", "!"]})
5
df_te = te.DataFrame.from_pandas(df_pd)
Copied!
Loading a pandas DataFrame in memory with for example
pd.read_parquet and then using te.Dataframe.from_pandas is less performant than directly reading the file with Terality with for examplete.read_parquet. Prefer reading directly from a file when possible.1
import pandas as pd
2
import terality as te
3
4
path = "..."
5
6
# Read the DataFrame in memory locally...
7
df_pd = pd.read_parquet(path)
8
# ...then send it over the network to Terality.
9
df_te = te.DataFrame.from_pandas(df_pd)
10
11
# Better option:
12
# Files are read only on Terality cluster. Local memory is not used.
13
# If the files are stored on a cloud provider, does an efficient
14
# cloud to cloud transfer not limited by your own bandwidth.
15
df_te = te.read_parquet(path)
Copied!
Retrieving a Terality data structure as a pandas data structure in memory
If you want to continue working on your data in Python, for example doing Machine Learning, you can export back the Terality DataFrame to a pandas DataFrame locally and start using your favorite ML framework.
For this option, you must ensure that the DataFrame is small enough to fit in memory. You're leaving the world of Terality's unlimited memory!
1
# Convert back the Terality DataFrame to a pandas DataFrame
2
df_pd_2 = df_te.to_pandas()
3
4
# we can check that we recovered our original pandas DataFrame
5
pd.assert(df_pd, df_pd2)
Copied!
Last modified 6mo ago