Writing to multiple files
When dealing with big data, it is often inconvenient to store all the data in a single huge file of several GBs (or worse tens, or hundreds of GBs). To help with this issue we added some functions to allow you to save your DataFrame over several_files:
Here by setting
num_rows_per_filewe specify the number of rows of each resulting file, or you could also specify the number of files or the in-memory size per file.