not sure why it should be specific to parquet file. a thin UI wrapper around duckdb could do the trick but for majority of formats (pq, csv, json, sqlite, iceberg, deltalake)
Thanks for your idea. I first thought that simply viewing data might not be useful as many IDEs already handle basic data previews. One of core goals was to access stored metadata directly from the file without necessarily scanning the data.
Your idea of potentially mixing current implementation (using pyarrow) with DuckDB is very interesting:)
note: with duckdb you get network storage for free (ie delta table on s3 works the same as local) and i guess a smaller deps than pyarrow (might be wrong)
not sure why it should be specific to parquet file. a thin UI wrapper around duckdb could do the trick but for majority of formats (pq, csv, json, sqlite, iceberg, deltalake)
Depending on the filetype, you might be better off with a dedicated table tool like VisiData https://www.visidata.org/docs/formats/
Thanks for your idea. I first thought that simply viewing data might not be useful as many IDEs already handle basic data previews. One of core goals was to access stored metadata directly from the file without necessarily scanning the data. Your idea of potentially mixing current implementation (using pyarrow) with DuckDB is very interesting:)
note: with duckdb you get network storage for free (ie delta table on s3 works the same as local) and i guess a smaller deps than pyarrow (might be wrong)
I upvote this suggestion and add pickle and hd5 to the list.
I would definitely add it to my roadmap