I looking for ways to read data from multiple partitioned directories from s3 using python.

data_folder/serial_number=1/cur_date=20-12-2012/abcdsd0324324.snappy.parquet
data_folder/serial_number=2/cur_date=27-12-2012/asdsdfsd0324324.snappy.parquet

pyarrow&#39;s ParquetDataset module has the capabilty to read from partitions. So I have tried the following  code : 

    &gt;&gt;&gt; import pandas as pd
    &gt;&gt;&gt; import pyarrow.parquet as pq
    &gt;&gt;&gt; import s3fs
    &gt;&gt;&gt; a = &quot;s3://my_bucker/path/to/data_folder/&quot;
    &gt;&gt;&gt; dataset = pq.ParquetDataset(a)

It threw the following error :

    Traceback (most recent call last):
      File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
      File &quot;/home/my_username/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py&quot;, line 502, in __init__
        self.metadata_path) = _make_manifest(path_or_paths, self.fs)
      File &quot;/home/my_username/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py&quot;, line 601, in _make_manifest
        .format(path))
    OSError: Passed non-file path: s3://my_bucker/path/to/data_folder/

Based on documentation of pyarrow I tried using s3fs as the file system, ie :

    &gt;&gt;&gt; dataset = pq.ParquetDataset(a,filesystem=s3fs)

Which throws the following error : 

    Traceback (most recent call last):
      File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
      File &quot;/home/my_username/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py&quot;, line 502, in __init__
        self.metadata_path) = _make_manifest(path_or_paths, self.fs)
      File &quot;/home/my_username/anaconda3/lib/python3.6/site-packages/pyarrow/parquet.py&quot;, line 583, in _make_manifest
        if is_string(path_or_paths) and fs.isdir(path_or_paths):
    AttributeError: module &#39;s3fs&#39; has no attribute &#39;isdir&#39;

I am limited to use a ECS cluster, hence **spark/pyspark is not an option**.

Is there a way we can easily read  the parquet files easily, in python from such partitioned directories in s3 ? I feel that listing the all the directories and then reading the is not a good practise as suggested in this [link](https://stackoverflow.com/questions/45043554/how-to-read-a-list-of-parquet-files-from-s3-as-a-pandas-dataframe-using-pyarrow). I would need to convert the read data to a pandas dataframe for further processing &amp; hence prefer options related to fastparquet or pyarrow. I am open to other options in python as well.
 


All Python S3fs Solutions on Gang of Coders

Total of 1 Python S3fs Solutions

How to read partitioned parquet files from S3 using pyarrow in python