Heya, I'm looking at replacing an Explorer pipeline with Dux.
It does some interesting things... namely transforming data by column with elixir functions and reconstructing the data frame (Explorer.Series.transform). There is no option to replace the elixir functions with SQL.
I think I can emulate the transformation, but the one challenge looks like it might be memory in that Dux can't stream results? Is that right?
Keen to take any advice on how I might use Dux to do this.
e.g.
# Vast simplification; would actually pull multiple columns and transform each in parallel in reality
# then reconstruct into a DataFrame.
"some_file.parquet"
|> DF.select([:column])
|> DF.pull(:column)
|> S.transform(fn value -> ... do something end)
Ideally I'd continue to do those transformations over a stream rather than materialising the entire dataset into Elixir first.
Thanks in advance!
Heya, I'm looking at replacing an Explorer pipeline with Dux.
It does some interesting things... namely transforming data by column with elixir functions and reconstructing the data frame (Explorer.Series.transform). There is no option to replace the elixir functions with SQL.
I think I can emulate the transformation, but the one challenge looks like it might be memory in that Dux can't stream results? Is that right?
Keen to take any advice on how I might use Dux to do this.
e.g.
Ideally I'd continue to do those transformations over a stream rather than materialising the entire dataset into Elixir first.
Thanks in advance!