Adding filenames to output

[alevites]

Hi,

I am using flatterer to flatten json files to parquet. My question is regarding the input of multiple files. The files all have the same format (and run through fine) but I need a way to distinguish which file each record in the result set came from. Is there any way to do this, ie. add a field with the filename or some other unique ID? I have tried a few different ways with parquet , dataframes, and even thought about using the fields.csv but I'm not seeing a straightforward way to do it.

Any help would be much appreciated!

[kindly]

There is not a very easy way to do that.

The intension for such cases was to use the generator input source explained on this page:

https://flatterer.opendata.coop/library.html#usage

Your code will look something like the following if the input is an array of objects.

import flatterer
import ijson

def array_item_generator():
    for filename in ['list', 'of', 'filenames']:
        with open(filename, 'rb') as f:
            for item in ijson.items(f, 'item'):
                 item['filename'] = filename
                 yield item

output = flatterer.flatten(array_item_generator(), 'games_dir', parquet=True)

[alevites]

Unfortunately, I can't get the above solution to work (no parquet results at all) but I did find a way to add the filename as I download the file but before I process it. Thanks for your help!