Support geospatial statistics and bounding box metadata - #173
Conversation
7a7afef to
5d32d64
Compare
I couldn't find any /docs directory, but I can see /_designs. Should I add a new doc here explaining how to extend simple-datagen.py, excluding changed parquet files from commit and so on? |
|
It's here: https://github.com/hardwood-hq/hardwood/tree/main/docs. You may have to rebase to latest main. No need for a design doc, but updating those user-facing docs would be good. |
5d32d64 to
1fc5c33
Compare
|
@mghildiy, pushed a commit with some minor fixes as I was taking another look. Taking a step back though, what this PR is missing are the actual geo logical types; without those, reading the stats isn't really useful; i.e. we're taking step 2 before step 1 here. Can you amend this PR accordingly? |
|
Ok, would do that. But I have a query here. How is it different from
geospatial types we are currently reading?
Manish
…On Sun, 29 Mar, 2026, 7:29 pm Gunnar Morling, ***@***.***> wrote:
*gunnarmorling* left a comment (hardwood-hq/hardwood#173)
<#173?email_source=notifications&email_token=ACMXFVRTCJZ7HEUAMJC5THT4TE23ZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJVGAZDCNBZG4Y2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4150214971>
@mghildiy <https://github.com/mghildiy>, pushed a commit with some minor
fixes as I was taking another look.
Taking a step back though, what this PR is missing are the actual geo
logical types; without those, reading the stats isn't really useful; i.e.
we're taking step 2 before step 1 here. Can you amend this PR accordingly?
—
Reply to this email directly, view it on GitHub
<#173?email_source=notifications&email_token=ACMXFVRTCJZ7HEUAMJC5THT4TE23ZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJVGAZDCNBZG4Y2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4150214971>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACMXFVUG2FJYS27OQBAYA2L4TE23ZAVCNFSM6AAAAACXDISB4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCNJQGIYTIOJXGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
That's the thing, we don't read any geospatial types yet, we solely retrieve statistics. We'll need logical types for GEOMETRY and GEOGRAPHY so consumers know how to interpret them. Also a predicate type for applying bounding box filters on these types. I have pushed another commit with a design doc (authored using Claude Code) which should clear things up. |
Ok, would go through design docs and implement the requirements. |
|
@gunnarmorling, which exact design document has new specs? |
2676777 to
f51309c
Compare
169a4e7 to
3fe9a32
Compare
|
@mghildiy, what should we do with this one? DYT it's good for another review round (apart from the needed rebase)? |
We can have another round of review for it. I would commit the changes after rebase. |
3fe9a32 to
b19a1cf
Compare
1. Classes to capture geospatial metadata(BoundingBox.java, GeospatialStatistics.java) 2. Reader classes(BoundingBoxReader.java, GeospatialStatisticsReader.java) 3. Test for same(GeospatialStatisticsTest.java) 4. simple-datagen.py extended to generate test parquet file geospatial_stats_test.parquet 5. usage.md updated with sample code to access geospatial metadata
… with EdgeInterpolation enum. 2. Add LogicalTypeReader cases 17/18 for GEOMETRY/GEOGRAPHY with CRS and edge interpolation parsing. 3. Add geospatialStatistics field (field 7) to ColumnIndex and ColumnIndexReader. 4. Add IntersectsPredicate to FilterPredicate with intersects() factory method. 5. Add GeospatialPredicate to ResolvedPredicate. 6. Add geospatial case to FilterPredicateResolver with GEOMETRY/GEOGRAPHY validation. 7. Add row group bounding box evaluation in RowGroupFilterEvaluator with antimeridian wrapping. 8. Add page-level bounding box based filter evaluation in PageFilterEvaluator. 9. Add Geospatial utility class for shared x-axis overlap logic. 10. Add GeospatialEndToEndTest for full filter pushdown chain with JTS WKB decoding. 11. Add unit tests for spatial predicate evaluation including antimeridian and and/or composition. 12. Add geo-datagen.py for geospatial test data generation. 13. Update usage.md with geospatial column identification and intersects filter pushdown examples.
f6e6a33 to
73d62b5
Compare
2 participants
Added support for reading geospatial metadata #78 :