When using an S3-compatible Object Storage, be cautious when upgrading **SDK for Java 2.x** to version **2.30.0 or later**

TL;DR

S3 has the following Data Integrity Protections mechanism:
https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html

This mechanism is now automatically enabled starting from SDK for Java 2.x 2.30.0.
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-checksums.html

If the S3-compatible Object Storage you are using does not support this feature, requests to the Object Storage may result in errors.
(If you have proper tests in place, you should notice this, but it’s a good idea to be cautious when upgrading versions.)

If you encounter this issue, apply the following settings.

// sync clientの場合
S3Client
    .builder()
    // ...
    .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
    .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
    // ...
    .build()

// async clientの場合
S3AsyncClient
    .builder()
    // ...
    .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
    .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
    // ...
    .build()

Background

After upgrading software.amazon.awssdk:s3 to 2.30.0 or later, we started encountering the following error in the S3-compatible Object Storage used internally.

(Service: S3, Status Code: 400, Request ID: MASKED...)
software.amazon.awssdk.services.s3.model.S3Exception:  (Service: S3, Status Code: 400, Request ID: MASKED...)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:155)
...

Cause

S3 has the following Data Integrity Protections mechanism:
https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html

To briefly explain the mechanism:

• When uploading to S3, the SDK calculates a checksum of the content and uploads it along with the content.
  • The S3 server verifies the checksum of the transmitted content.
• When downloading content from S3, the checksum is also downloaded along with the content.
  • The SDK verifies the checksum of the received content.

(Previously, data integrity verification could be performed by specifying the Content-MD5 header at the time of upload. This seems to be a more advanced version of that. The ability to verify data integrity during downloads is a useful addition.)

This mechanism is now automatically enabled starting from 2.30.0.
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-checksums.html

However, since the S3-compatible Object Storage I was using did not support it, errors started occurring.

In my case, I quickly noticed the issue through tests, but since it was just a minor version change, I imagine many people might overlook it.

Support Status of Various S3-Compatible Object Storages

I wanted to properly investigate this but gave up!

Based on a rough check using o1 pro mode & Deep Search, MinIO supports it, but other storages do not.

Since this mechanism was introduced in February 2022, which is relatively recent (though it has been three years), support is likely still limited.
https://aws.amazon.com/jp/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/

Miscellaneous

The SDK for Java 1.x still does not support this feature, so this issue does not occur. However, since it is reaching EOL this year, those who have not yet migrated to the 2.x series should do so.
https://aws.amazon.com/jp/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/