Skip to content

Commit 76b7b03

Browse files
gunnarmorlinggunnarmorling
committed
#144 Removing S3 SDK dependency
1 parent 0de6127 commit 76b7b03

27 files changed

Lines changed: 1823 additions & 433 deletions

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ Avoid fully-qualified class names within the code, always add imports.
2727
Avoid object access and boxing as much as possible. Always prefer primitive access also if it means several similar methods.
2828
Before writing new code, search for existing patterns in the same class/package that accomplish the same thing (e.g., the DRY principle). Extract repeated logic into helper methods within the same class rather than duplicating it. When a pattern appears multiple times, consider consolidating it into a single well-named method with overloads if needed.
2929
Be conservative with base class refactoring. Do not pull implementation details up into abstract base classes unless the logic is truly identical across all subclasses with no foreseeable divergence. Shared helpers are better than shared template methods when subclasses may need different control flow.
30+
Never use `var` syntax.
3031

3132
# Documentation
3233

‎README.md‎

Lines changed: 70 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -479,7 +479,7 @@ try (Hardwood hardwood = Hardwood.create();
479479

480480
### Reading from S3
481481

482-
The `hardwood-s3` module adds support for reading Parquet files directly from Amazon S3. Add it as a dependency alongside `hardwood-core`:
482+
The `hardwood-s3` module adds support for reading Parquet files from Amazon S3 and S3-compatible services (Cloudflare R2, GCP Cloud Storage via HMAC keys, MinIO).
483483

484484
```xml
485485
<dependency>
@@ -488,18 +488,21 @@ The `hardwood-s3` module adds support for reading Parquet files directly from Am
488488
</dependency>
489489
```
490490

491-
Read a single file from S3:
491+
Read a file with static credentials:
492492

493493
```java
494-
import dev.hardwood.s3.S3InputFile;
494+
import dev.hardwood.s3.S3Credentials;
495+
import dev.hardwood.s3.S3Source;
495496
import dev.hardwood.reader.ParquetFileReader;
496497
import dev.hardwood.reader.RowReader;
497-
import software.amazon.awssdk.services.s3.S3Client;
498498

499-
S3Client s3 = S3Client.create(); // uses default credential chain
499+
S3Source source = S3Source.builder()
500+
.region("us-east-1")
501+
.credentials(S3Credentials.of("AKIA...", "secret"))
502+
.build();
500503

501504
try (ParquetFileReader reader = ParquetFileReader.open(
502-
S3InputFile.of(s3, "my-bucket", "data/trips.parquet"))) {
505+
source.inputFile("s3://my-bucket/data/trips.parquet"))) {
503506
try (RowReader rows = reader.createRowReader()) {
504507
while (rows.hasNext()) {
505508
rows.next();
@@ -509,22 +512,44 @@ try (ParquetFileReader reader = ParquetFileReader.open(
509512
}
510513
```
511514

512-
Read multiple files with a shared `S3Client` and thread pool:
515+
For dynamic or refreshable credentials, implement the `S3CredentialsProvider` functional interface:
513516

514517
```java
515-
import dev.hardwood.Hardwood;
516-
import dev.hardwood.InputFile;
517-
import dev.hardwood.s3.S3InputFile;
518+
S3Source source = S3Source.builder()
519+
.region("us-east-1")
520+
.credentials(() -> fetchCredentialsFromVault())
521+
.build();
522+
```
518523

519-
S3Client s3 = S3Client.create();
520-
List<InputFile> files = List.of(
521-
S3InputFile.of(s3, "my-bucket", "data/part-001.parquet"),
522-
S3InputFile.of(s3, "my-bucket", "data/part-002.parquet"),
523-
S3InputFile.of(s3, "my-bucket", "data/part-003.parquet")
524-
);
524+
For the full AWS credential chain (env vars, `~/.aws/credentials`, EC2/ECS instance profile, SSO, web identity), add the optional `hardwood-aws-auth` module:
525+
526+
```xml
527+
<dependency>
528+
<groupId>dev.hardwood</groupId>
529+
<artifactId>hardwood-aws-auth</artifactId>
530+
</dependency>
531+
```
532+
533+
```java
534+
import dev.hardwood.aws.auth.SdkCredentialsProviders;
535+
536+
S3Source source = S3Source.builder()
537+
.region("us-east-1")
538+
.credentials(SdkCredentialsProviders.defaultChain())
539+
.build();
540+
```
541+
542+
Read multiple files in the same bucket:
543+
544+
```java
545+
import dev.hardwood.Hardwood;
525546

526547
try (Hardwood hardwood = Hardwood.create();
527-
MultiFileParquetReader parquet = hardwood.openAll(files);
548+
MultiFileParquetReader parquet = hardwood.openAll(
549+
source.inputFilesInBucket("my-bucket",
550+
"data/part-001.parquet",
551+
"data/part-002.parquet",
552+
"data/part-003.parquet"));
528553
MultiFileRowReader reader = parquet.createRowReader()) {
529554
while (reader.hasNext()) {
530555
reader.next();
@@ -533,6 +558,32 @@ try (Hardwood hardwood = Hardwood.create();
533558
}
534559
```
535560

561+
Read multiple files across buckets:
562+
563+
```java
564+
hardwood.openAll(source.inputFiles(
565+
"s3://bucket-a/events.parquet",
566+
"s3://bucket-b/events.parquet"));
567+
```
568+
569+
In order to use S3-compatible services, set a custom endpoint:
570+
571+
```java
572+
// Cloudflare R2
573+
S3Source source = S3Source.builder()
574+
.endpoint("https://<account-id>.r2.cloudflarestorage.com")
575+
.credentials(S3Credentials.of(accessKeyId, secretKey))
576+
.build();
577+
578+
// GCP Cloud Storage (HMAC keys)
579+
S3Source source = S3Source.builder()
580+
.endpoint("https://storage.googleapis.com")
581+
.credentials(S3Credentials.of(hmacAccessId, hmacSecret))
582+
.build();
583+
```
584+
585+
When a custom endpoint is set, region can be omitted — `S3Source` uses a default for signing since S3-compatible services don't use the region for routing.
586+
536587
Column projection, row group filtering, and all other reader features work transparently with S3 files. Hardwood minimizes S3 requests by pre-fetching the file footer on open and coalescing column chunk reads within each row group.
537588

538589
### Reading into Avro GenericRecord
@@ -935,7 +986,8 @@ Hardwood is organized into public API packages and internal implementation packa
935986
| `dev.hardwood.schema` | **Public API** | Schema representation: file schema, column schemas, and column projection. |
936987
| `dev.hardwood.row` | **Public API** | Value types for nested data access: structs, lists, and maps. |
937988
| `dev.hardwood.avro` | **Public API** | Avro GenericRecord support: schema conversion and row materialization (`hardwood-avro` module). |
938-
| `dev.hardwood.s3` | **Public API** | S3 object storage InputFile implementation (`hardwood-s3` module). |
989+
| `dev.hardwood.s3` | **Public API** | S3 object storage support: `S3Source`, `S3InputFile`, `S3Credentials`, `S3CredentialsProvider` (`hardwood-s3` module, zero external dependencies). |
990+
| `dev.hardwood.aws.auth` | **Public API** | Bridges the AWS SDK credential chain to Hardwood's `S3CredentialsProvider` (`hardwood-aws-auth` module, optional). |
939991
| `dev.hardwood.jfr` | **Public API** | JFR event types emitted during file reading, decoding, and pipeline operations. |
940992
| `dev.hardwood.internal.*` | **Internal** | Implementation details — not part of the public API and may change without notice. |
941993

@@ -1153,8 +1205,6 @@ The solution differs by codec:
11531205

11541206
`netty-buffer` (an optional dependency of `brotli4j`) is declared explicitly at compile scope so that GraalVM can resolve the `ByteBufUtil` reference in `brotli4j`'s `DirectDecompress` class during image analysis.
11551207

1156-
Container builds use Mandrel's `--link-at-build-time`, which applies stricter type resolution: all reachable bytecode must have its full type hierarchy resolvable at build time. Netty's `Log4JLoggerFactory` references `org.apache.log4j.Logger` (provided by the `log4j-1.2-api` bridge artifact), which in turn references `org.apache.logging.log4j.core.LogEvent` from `log4j-core`. Both are therefore declared at compile scope in `cli/pom.xml` so that GraalVM can resolve the full chain during container builds.
1157-
11581208
#### Manual tests of the native CLI binary
11591209

11601210
1. Start S3Mock and set environment

‎aws-auth/pom.xml‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
SPDX-License-Identifier: Apache-2.0
5+
6+
Copyright The original authors
7+
8+
Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
9+
10+
-->
11+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
12+
<modelVersion>4.0.0</modelVersion>
13+
14+
<parent>
15+
<groupId>dev.hardwood</groupId>
16+
<artifactId>hardwood-parent</artifactId>
17+
<version>1.0.0-SNAPSHOT</version>
18+
</parent>
19+
20+
<artifactId>hardwood-aws-auth</artifactId>
21+
22+
<name>Hardwood AWS Auth</name>
23+
<description>Bridges the AWS SDK credential chain to Hardwood's credential types</description>
24+
<url>https://github.com/hardwood-hq/hardwood</url>
25+
26+
<scm>
27+
<connection>scm:git:git://github.com/hardwood-hq/hardwood.git</connection>
28+
<developerConnection>scm:git:git@github.com:hardwood-hq/hardwood.git</developerConnection>
29+
<url>https://github.com/hardwood-hq/hardwood/</url>
30+
<tag>HEAD</tag>
31+
</scm>
32+
33+
<properties>
34+
<aws-sdk.version>2.42.17</aws-sdk.version>
35+
</properties>
36+
37+
<dependencyManagement>
38+
<dependencies>
39+
<dependency>
40+
<groupId>dev.hardwood</groupId>
41+
<artifactId>hardwood-bom</artifactId>
42+
<version>${project.version}</version>
43+
<type>pom</type>
44+
<scope>import</scope>
45+
</dependency>
46+
<dependency>
47+
<groupId>software.amazon.awssdk</groupId>
48+
<artifactId>bom</artifactId>
49+
<version>${aws-sdk.version}</version>
50+
<type>pom</type>
51+
<scope>import</scope>
52+
</dependency>
53+
</dependencies>
54+
</dependencyManagement>
55+
56+
<dependencies>
57+
<dependency>
58+
<groupId>dev.hardwood</groupId>
59+
<artifactId>hardwood-s3</artifactId>
60+
</dependency>
61+
<dependency>
62+
<groupId>software.amazon.awssdk</groupId>
63+
<artifactId>auth</artifactId>
64+
<exclusions>
65+
<!-- Exclude signing/HTTP/checksum modules — only credential resolution is needed -->
66+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>http-auth-aws</artifactId></exclusion>
67+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>http-auth-aws-eventstream</artifactId></exclusion>
68+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>http-auth</artifactId></exclusion>
69+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>http-auth-spi</artifactId></exclusion>
70+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>checksums</artifactId></exclusion>
71+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>checksums-spi</artifactId></exclusion>
72+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>metrics-spi</artifactId></exclusion>
73+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>endpoints-spi</artifactId></exclusion>
74+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>retries-spi</artifactId></exclusion>
75+
<exclusion><groupId>software.amazon.awssdk</groupId><artifactId>retries</artifactId></exclusion>
76+
<exclusion><groupId>software.amazon.eventstream</groupId><artifactId>eventstream</artifactId></exclusion>
77+
</exclusions>
78+
</dependency>
79+
80+
<!-- Allows for browser-based authentication flow via aws login -->
81+
<dependency>
82+
<groupId>software.amazon.awssdk</groupId>
83+
<artifactId>signin</artifactId>
84+
</dependency>
85+
</dependencies>
86+
</project>

‎aws-auth/src/main/java/dev/hardwood/aws/auth/SdkCredentialsProviders.java‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* Copyright The original authors
5+
*
6+
* Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
7+
*/
8+
package dev.hardwood.aws.auth;
9+
10+
import dev.hardwood.s3.S3Credentials;
11+
import dev.hardwood.s3.S3CredentialsProvider;
12+
import software.amazon.awssdk.auth.credentials.AwsCredentials;
13+
import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
14+
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
15+
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
16+
17+
/// Bridges the AWS SDK credential chain to Hardwood's types.
18+
///
19+
/// This class requires `software.amazon.awssdk:auth` on the classpath.
20+
/// Use the `hardwood-aws-auth` module dependency to pull it in.
21+
public final class SdkCredentialsProviders {
22+
23+
private SdkCredentialsProviders() {
24+
}
25+
26+
/// Returns a provider backed by the full AWS default credential chain
27+
/// (env vars, `~/.aws/credentials`, EC2/ECS instance profile, SSO, web identity, etc.).
28+
public static S3CredentialsProvider defaultChain() {
29+
var sdk = DefaultCredentialsProvider.builder().build();
30+
return () -> toHardwood(sdk.resolveCredentials());
31+
}
32+
33+
/// Returns a provider for a specific named profile from `~/.aws/credentials`.
34+
///
35+
/// @param profileName the profile name
36+
public static S3CredentialsProvider fromProfile(String profileName) {
37+
var sdk = ProfileCredentialsProvider.builder().profileName(profileName).build();
38+
return () -> toHardwood(sdk.resolveCredentials());
39+
}
40+
41+
private static S3Credentials toHardwood(AwsCredentials sdkCreds) {
42+
if (sdkCreds instanceof AwsSessionCredentials session) {
43+
return new S3Credentials(session.accessKeyId(), session.secretAccessKey(), session.sessionToken());
44+
}
45+
return S3Credentials.of(sdkCreds.accessKeyId(), sdkCreds.secretAccessKey());
46+
}
47+
}

‎bom/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,11 @@
6161
<artifactId>hardwood-s3</artifactId>
6262
<version>${project.version}</version>
6363
</dependency>
64+
<dependency>
65+
<groupId>dev.hardwood</groupId>
66+
<artifactId>hardwood-aws-auth</artifactId>
67+
<version>${project.version}</version>
68+
</dependency>
6469
<dependency>
6570
<groupId>dev.hardwood</groupId>
6671
<artifactId>hardwood-avro</artifactId>

‎cli/pom.xml‎

Lines changed: 4 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@
1313
<properties>
1414
<executable-suffix />
1515
<dist.qualifier>early-access</dist.qualifier>
16-
<aws-sdk.version>2.42.17</aws-sdk.version>
1716
<quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
1817
<quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
1918
<quarkus.platform.version>3.27.2</quarkus.platform.version>
@@ -73,6 +72,10 @@
7372
<groupId>dev.hardwood</groupId>
7473
<artifactId>hardwood-s3</artifactId>
7574
</dependency>
75+
<dependency>
76+
<groupId>dev.hardwood</groupId>
77+
<artifactId>hardwood-aws-auth</artifactId>
78+
</dependency>
7679

7780
<dependency>
7881
<groupId>com.github.freva</groupId>
@@ -128,16 +131,6 @@
128131
<scope>test</scope>
129132
</dependency>
130133

131-
<!-- The AWS SDK's ChecksumProvider has hard bytecode references to CRT checksum
132-
classes (CRC64NVME, CRC32C, etc.) that GraalVM must resolve at build time.
133-
At runtime the CRT native library is not loaded, so the SDK falls back to
134-
pure-Java checksums. Only the class stubs are needed; the CRT itself is inert. -->
135-
<dependency>
136-
<groupId>software.amazon.awssdk.crt</groupId>
137-
<artifactId>aws-crt</artifactId>
138-
<version>0.43.5</version>
139-
</dependency>
140-
141134
<!-- netty-buffer is an optional dependency of brotli4j (declared optional in its POM).
142135
It must be explicitly provided so GraalVM native-image can resolve the
143136
ByteBufUtil reference in DirectDecompress at build time. -->

0 commit comments

Comments
 (0)