it seems that using a forward-only reader, such as the Go `tar` reader, can be challenging when uploading large files to Amazon S3 using the AWS SDK for Go.
The key points are:
1. Memory Management Issues: The AWS SDK for Go's S3 uploader allocates and fills chunk buffers for the entire file before uploading them. This can cause the process to run out of memory when uploading large files with a reader that doesn't support `Seek` or `ReadAt` functions, like the Go `tar` reader.
2. Intelligent Chunking: The GitHub issue #1104 suggests that the SDK could be improved to use more intelligent chunking when dealing with readers that don't support `Seek` and `ReadAt`. This would help manage memory usage better during large file uploads.
3. Streaming and Multipart Uploads: The AWS SDK for Go does support streaming and multipart uploads, which can help manage memory usage during large file uploads. However, these features may not work as well with a forward-only reader.
4. Alternative Solutions: The search results suggest using the Minio SDK or the Google CDK, which may have better support for streaming large files to S3. Additionally, the aws-s3-reader-seeker project provides a custom reader implementation that can handle large S3 objects more efficiently.
In summary, while it is possible to use a forward-only reader like the Go `tar` reader for uploading large files to S3 using the AWS SDK for Go, it may not be the most efficient or memory-friendly approach. Considering alternative solutions or using the SDK's streaming and multipart upload features may be a better option in such cases.
Citations:[1] https://github.com/aws/aws-sdk-go/issues/1104
[2] https://github.com/nikolaydubina/aws-s3-reader/blob/master/aws_s3_reader_seeker.go
[3] https://www.reddit.com/r/golang/comments/131bq42/need_to_stream_large_files_to_s3_using_go/
[4] https://www.linkedin.com/pulse/golang-uploading-files-aws-s3-bucket-using-pre-signed-tiago-melo
[5] https://pkg.go.dev/github.com/SnellerInc/sneller/aws/s3