arn.go batch.go bucket_region.go buffered_read_seeker.go default_read_seeker_write_to.go default_writer_read_from.go doc.go download.go pool.go read_seeker_write_to.go upload.go upload_input.go writer_read_from.go
Internal call graph â¹ Internal call graph â¾In the call graph viewer below, each node is a function belonging to this package and its children are the functions it calls—perhaps dynamically.
The root nodes are the entry points of the package: functions that may be called from outside the package. There may be non-exported or anonymous functions among them if they are called dynamically from another package.
Click a node to visit that function's source code. From there you can visit its callers by clicking its declaring func token.
Functions may be omitted if they were determined to be unreachable in the particular programs or tests that were analyzed.
const (
DefaultBatchSize = 100
)
const DefaultDownloadConcurrency = 5
DefaultDownloadConcurrency is the default number of goroutines to spin up when using Download().
const DefaultDownloadPartSize = 1024 * 1024 * 5
DefaultDownloadPartSize is the default range of bytes to get at a time when using Download().
const DefaultUploadConcurrency = 5
DefaultUploadConcurrency is the default number of goroutines to spin up when using Upload().
const DefaultUploadPartSize = MinUploadPartSize
DefaultUploadPartSize is the default part size to buffer chunks of a payload into.
const (
ErrDeleteBatchFailCode = "DeleteBatchError"
)
const MaxUploadParts = 10000
MaxUploadParts is the maximum allowed number of parts in a multi-part upload on Amazon S3.
const MinUploadPartSize int64 = 1024 * 1024 * 5
MinUploadPartSize is the minimum allowed part size when uploading a part to Amazon S3.
func GetBucketRegion ¶func GetBucketRegion(ctx aws.Context, c client.ConfigProvider, bucket, regionHint string, opts ...request.Option) (string, error)
GetBucketRegion will attempt to get the region for a bucket using the regionHint to determine which AWS partition to perform the query on.
The request will not be signed, and will not use your AWS credentials.
A "NotFound" error code will be returned if the bucket does not exist in the AWS partition the regionHint belongs to. If the regionHint parameter is an empty string GetBucketRegion will fallback to the ConfigProvider's region config. If the regionHint is empty, and the ConfigProvider does not have a region value, an error will be returned..
For example to get the region of a bucket which exists in "eu-central-1" you could provide a region hint of "us-west-2".
sess := session.Must(session.NewSession())
bucket := "my-bucket"
region, err := s3manager.GetBucketRegion(ctx, sess, bucket, "us-west-2")
if err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == "NotFound" {
fmt.Fprintf(os.Stderr, "unable to find bucket %s's region not found\n", bucket)
}
return err
}
fmt.Printf("Bucket %s is in %s region\n", bucket, region)
By default the request will be made to the Amazon S3 endpoint using the Path style addressing.
s3.us-west-2.amazonaws.com/bucketname
This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false).
region, err := s3manager.GetBucketRegion(ctx, sess, "bucketname", "us-west-2", func(r *request.Request) {
r.S3ForcePathStyle = aws.Bool(false)
})
To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client.
sess, err := session.NewSession(&aws.Config{
Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
})
region, err := s3manager.GetBucketRegion(context.Background(), sess, "bucketname", "")
func GetBucketRegionWithClient ¶
func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string, opts ...request.Option) (string, error)
GetBucketRegionWithClient is the same as GetBucketRegion with the exception that it takes a S3 service client instead of a Session. The regionHint is derived from the region the S3 service client was created in.
By default the request will be made to the Amazon S3 endpoint using the Path style addressing.
s3.us-west-2.amazonaws.com/bucketname
This is not compatible with Amazon S3's FIPS endpoints. To override this behavior to use Virtual Host style addressing, provide a functional option that will set the Request's Config.S3ForcePathStyle to aws.Bool(false).
region, err := s3manager.GetBucketRegionWithClient(ctx, client, "bucketname", func(r *request.Request) {
r.S3ForcePathStyle = aws.Bool(false)
})
To configure the GetBucketRegion to make a request via the Amazon S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the utility is called with. The hint region will be ignored if an endpoint URL is configured on the session or client.
region, err := s3manager.GetBucketRegionWithClient(context.Background(),
s3.New(sess, &aws.Config{
Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"),
}),
"bucketname")
See GetBucketRegion for more information.
func NewBatchError ¶func NewBatchError(code, message string, err []Error) awserr.Error
NewBatchError will return a BatchError that satisfies the awserr.Error interface.
func WithDownloaderRequestOptions ¶func WithDownloaderRequestOptions(opts ...request.Option) func(*Downloader)
WithDownloaderRequestOptions appends to the Downloader's API request options.
func WithUploaderRequestOptions ¶func WithUploaderRequestOptions(opts ...request.Option) func(*Uploader)
WithUploaderRequestOptions appends to the Uploader's API request options.
type BatchDelete ¶type BatchDelete struct {
Client s3iface.S3API
BatchSize int
}
BatchDelete will use the s3 package's service client to perform a batch delete.
func NewBatchDelete ¶func NewBatchDelete(c client.ConfigProvider, options ...func(*BatchDelete)) *BatchDelete
NewBatchDelete will return a new delete client that can delete a batched amount of objects.
Example:
batcher := s3manager.NewBatchDelete(sess, size)
objects := []BatchDeleteObject{
{
Object: &s3.DeleteObjectInput {
Key: aws.String("key"),
Bucket: aws.String("bucket"),
},
},
}
if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
Objects: objects,
}); err != nil {
return err
}
func NewBatchDeleteWithClient ¶
func NewBatchDeleteWithClient(client s3iface.S3API, options ...func(*BatchDelete)) *BatchDelete
NewBatchDeleteWithClient will return a new delete client that can delete a batched amount of objects.
Example:
batcher := s3manager.NewBatchDeleteWithClient(client, size)
objects := []BatchDeleteObject{
{
Object: &s3.DeleteObjectInput {
Key: aws.String("key"),
Bucket: aws.String("bucket"),
},
},
}
if err := batcher.Delete(aws.BackgroundContext(), &s3manager.DeleteObjectsIterator{
Objects: objects,
}); err != nil {
return err
}
func (*BatchDelete) Delete ¶
func (d *BatchDelete) Delete(ctx aws.Context, iter BatchDeleteIterator) error
Delete will use the iterator to queue up objects that need to be deleted. Once the batch size is met, this will call the deleteBatch function.
type BatchDeleteIterator ¶type BatchDeleteIterator interface {
Next() bool
Err() error
DeleteObject() BatchDeleteObject
}
BatchDeleteIterator is an interface that uses the scanner pattern to iterate through what needs to be deleted.
func NewDeleteListIterator ¶func NewDeleteListIterator(svc s3iface.S3API, input *s3.ListObjectsInput, opts ...func(*DeleteListIterator)) BatchDeleteIterator
NewDeleteListIterator will return a new DeleteListIterator.
type BatchDeleteObject ¶type BatchDeleteObject struct {
Object *s3.DeleteObjectInput
After func() error
}
BatchDeleteObject is a wrapper object for calling the batch delete operation.
type BatchDownloadIterator ¶type BatchDownloadIterator interface {
Next() bool
Err() error
DownloadObject() BatchDownloadObject
}
BatchDownloadIterator is an interface that uses the scanner pattern to iterate through a series of objects to be downloaded.
type BatchDownloadObject ¶type BatchDownloadObject struct {
Object *s3.GetObjectInput
Writer io.WriterAt
After func() error
}
BatchDownloadObject contains all necessary information to run a batch operation once.
type BatchError ¶type BatchError struct {
Errors Errors
}
BatchError will contain the key and bucket of the object that failed to either upload or download.
func (*BatchError) Code ¶func (err *BatchError) Code() string
Code will return the code associated with the batch error.
func (*BatchError) Error ¶func (err *BatchError) Error() stringfunc (*BatchError) Message ¶
func (err *BatchError) Message() string
Message will return the message associated with the batch error.
func (*BatchError) OrigErr ¶func (err *BatchError) OrigErr() error
OrigErr will return the original error. Which, in this case, will always be nil for batched operations.
type BatchUploadIterator ¶type BatchUploadIterator interface {
Next() bool
Err() error
UploadObject() BatchUploadObject
}
BatchUploadIterator is an interface that uses the scanner pattern to iterate through what needs to be uploaded.
type BatchUploadObject ¶type BatchUploadObject struct {
Object *UploadInput
After func() error
}
BatchUploadObject contains all necessary information to run a batch operation once.
type BufferedReadSeeker ¶type BufferedReadSeeker struct {
}
BufferedReadSeeker is buffered io.ReadSeeker
func NewBufferedReadSeeker ¶func NewBufferedReadSeeker(r io.ReadSeeker, b []byte) *BufferedReadSeeker
NewBufferedReadSeeker returns a new BufferedReadSeeker if len(b) == 0 then the buffer will be initialized to 64 KiB.
func (*BufferedReadSeeker) Read ¶func (b *BufferedReadSeeker) Read(p []byte) (n int, err error)
Read will read up len(p) bytes into p and will return the number of bytes read and any error that occurred. If the len(p) > the buffer size then a single read request will be issued to the underlying io.ReadSeeker for len(p) bytes. A Read request will at most perform a single Read to the underlying io.ReadSeeker, and may return < len(p) if serviced from the buffer.
func (*BufferedReadSeeker) ReadAt ¶func (b *BufferedReadSeeker) ReadAt(p []byte, off int64) (int, error)
ReadAt will read up to len(p) bytes at the given file offset. This will result in the buffer being cleared.
func (*BufferedReadSeeker) Seek ¶func (b *BufferedReadSeeker) Seek(offset int64, whence int) (int64, error)
Seek will position then underlying io.ReadSeeker to the given offset and will clear the buffer.
type BufferedReadSeekerWriteTo ¶type BufferedReadSeekerWriteTo struct {
*BufferedReadSeeker
}
BufferedReadSeekerWriteTo wraps a BufferedReadSeeker with an io.WriteAt implementation.
func (*BufferedReadSeekerWriteTo) WriteTo ¶func (b *BufferedReadSeekerWriteTo) WriteTo(writer io.Writer) (int64, error)
WriteTo writes to the given io.Writer from BufferedReadSeeker until there's no more data to write or an error occurs. Returns the number of bytes written and any error encountered during the write.
type BufferedReadSeekerWriteToPool ¶type BufferedReadSeekerWriteToPool struct {
}
BufferedReadSeekerWriteToPool uses a sync.Pool to create and reuse []byte slices for buffering parts in memory
func NewBufferedReadSeekerWriteToPool ¶func NewBufferedReadSeekerWriteToPool(size int) *BufferedReadSeekerWriteToPool
NewBufferedReadSeekerWriteToPool will return a new BufferedReadSeekerWriteToPool that will create a pool of reusable buffers . If size is less then < 64 KiB then the buffer will default to 64 KiB. Reason: io.Copy from writers or readers that don't support io.WriteTo or io.ReadFrom respectively will default to copying 32 KiB.
func (*BufferedReadSeekerWriteToPool) GetWriteTo ¶func (p *BufferedReadSeekerWriteToPool) GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func())
GetWriteTo will wrap the provided io.ReadSeeker with a BufferedReadSeekerWriteTo. The provided cleanup must be called after operations have been completed on the returned io.ReadSeekerWriteTo in order to signal the return of resources to the pool.
type DeleteListIterator ¶type DeleteListIterator struct {
Bucket *string
Paginator request.Pagination
}
DeleteListIterator is an alternative iterator for the BatchDelete client. This will iterate through a list of objects and delete the objects.
Example:
iter := &s3manager.DeleteListIterator{
Client: svc,
Input: &s3.ListObjectsInput{
Bucket: aws.String("bucket"),
MaxKeys: aws.Int64(5),
},
Paginator: request.Pagination{
NewRequest: func() (*request.Request, error) {
var inCpy *ListObjectsInput
if input != nil {
tmp := *input
inCpy = &tmp
}
req, _ := c.ListObjectsRequest(inCpy)
return req, nil
},
},
}
batcher := s3manager.NewBatchDeleteWithClient(svc)
if err := batcher.Delete(aws.BackgroundContext(), iter); err != nil {
return err
}
func (*DeleteListIterator) DeleteObject ¶
func (iter *DeleteListIterator) DeleteObject() BatchDeleteObject
DeleteObject will return the current object to be deleted.
func (*DeleteListIterator) Err ¶func (iter *DeleteListIterator) Err() error
Err will return the last known error from Next.
func (*DeleteListIterator) Next ¶func (iter *DeleteListIterator) Next() bool
Next will use the S3API client to iterate through a list of objects.
type DeleteObjectsIterator ¶type DeleteObjectsIterator struct {
Objects []BatchDeleteObject
}
DeleteObjectsIterator is an interface that uses the scanner pattern to iterate through a series of objects to be deleted.
func (*DeleteObjectsIterator) DeleteObject ¶func (iter *DeleteObjectsIterator) DeleteObject() BatchDeleteObject
DeleteObject will return the BatchDeleteObject at the current batched index.
func (*DeleteObjectsIterator) Err ¶func (iter *DeleteObjectsIterator) Err() error
Err will return an error. Since this is just used to satisfy the BatchDeleteIterator interface this will only return nil.
func (*DeleteObjectsIterator) Next ¶func (iter *DeleteObjectsIterator) Next() bool
Next will increment the default iterator's index and ensure that there is another object to iterator to.
type DownloadObjectsIterator ¶type DownloadObjectsIterator struct {
Objects []BatchDownloadObject
}
DownloadObjectsIterator implements the BatchDownloadIterator interface and allows for batched download of objects.
func (*DownloadObjectsIterator) DownloadObject ¶func (batcher *DownloadObjectsIterator) DownloadObject() BatchDownloadObject
DownloadObject will return the BatchDownloadObject at the current batched index.
func (*DownloadObjectsIterator) Err ¶func (batcher *DownloadObjectsIterator) Err() error
Err will return an error. Since this is just used to satisfy the BatchDeleteIterator interface this will only return nil.
func (*DownloadObjectsIterator) Next ¶func (batcher *DownloadObjectsIterator) Next() bool
Next will increment the default iterator's index and ensure that there is another object to iterator to.
type Downloader ¶type Downloader struct {
PartSize int64
Concurrency int
S3 s3iface.S3API
RequestOptions []request.Option
BufferProvider WriterReadFromProvider
}
The Downloader structure that calls Download(). It is safe to call Download() on this structure for multiple objects and across concurrent goroutines. Mutating the Downloader's properties is not safe to be done concurrently.
func NewDownloader ¶func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downloader
NewDownloader creates a new Downloader instance to downloads objects from S3 in concurrent chunks. Pass in additional functional options to customize the downloader behavior. Requires a client.ConfigProvider in order to create a S3 service client. The session.Session satisfies the client.ConfigProvider interface.
Example:
// The session the S3 Downloader will use
sess := session.Must(session.NewSession())
// Create a downloader with the session and default options
downloader := s3manager.NewDownloader(sess)
// Create a downloader with the session and custom options
downloader := s3manager.NewDownloader(sess, func(d *s3manager.Downloader) {
d.PartSize = 64 * 1024 * 1024 // 64MB per part
})
func NewDownloaderWithClient ¶
func NewDownloaderWithClient(svc s3iface.S3API, options ...func(*Downloader)) *Downloader
NewDownloaderWithClient creates a new Downloader instance to downloads objects from S3 in concurrent chunks. Pass in additional functional options to customize the downloader behavior. Requires a S3 service client to make S3 API calls.
Example:
// The session the S3 Downloader will use
sess := session.Must(session.NewSession())
// The S3 client the S3 Downloader will use
s3Svc := s3.New(sess)
// Create a downloader with the s3 client and default options
downloader := s3manager.NewDownloaderWithClient(s3Svc)
// Create a downloader with the s3 client and custom options
downloader := s3manager.NewDownloaderWithClient(s3Svc, func(d *s3manager.Downloader) {
d.PartSize = 64 * 1024 * 1024 // 64MB per part
})
func (Downloader) Download ¶
func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)
Download downloads an object in S3 and writes the payload into w using concurrent GET requests. The n int64 returned is the size of the object downloaded in bytes.
Additional functional options can be provided to configure the individual download. These options are copies of the Downloader instance Download is called from. Modifying the options will not impact the original Downloader instance.
It is safe to call this method concurrently across goroutines.
The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer.
Specifying a Downloader.Concurrency of 1 will cause the Downloader to download the parts from S3 sequentially.
If the GetObjectInput's Range value is provided that will cause the downloader to perform a single GetObjectInput request for that object's range. This will caused the part size, and concurrency configurations to be ignored.
func (Downloader) DownloadWithContext ¶func (d Downloader) DownloadWithContext(ctx aws.Context, w io.WriterAt, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)
DownloadWithContext downloads an object in S3 and writes the payload into w using concurrent GET requests. The n int64 returned is the size of the object downloaded in bytes.
DownloadWithContext is the same as Download with the additional support for Context input parameters. The Context must not be nil. A nil Context will cause a panic. Use the Context to add deadlining, timeouts, etc. The DownloadWithContext may create sub-contexts for individual underlying requests.
Additional functional options can be provided to configure the individual download. These options are copies of the Downloader instance Download is called from. Modifying the options will not impact the original Downloader instance. Use the WithDownloaderRequestOptions helper function to pass in request options that will be applied to all API operations made with this downloader.
The w io.WriterAt can be satisfied by an os.File to do multipart concurrent downloads, or in memory []byte wrapper using aws.WriteAtBuffer.
Specifying a Downloader.Concurrency of 1 will cause the Downloader to download the parts from S3 sequentially.
It is safe to call this method concurrently across goroutines.
If the GetObjectInput's Range value is provided that will cause the downloader to perform a single GetObjectInput request for that object's range. This will caused the part size, and concurrency configurations to be ignored.
func (Downloader) DownloadWithIterator ¶func (d Downloader) DownloadWithIterator(ctx aws.Context, iter BatchDownloadIterator, opts ...func(*Downloader)) error
DownloadWithIterator will download a batched amount of objects in S3 and writes them to the io.WriterAt specificed in the iterator.
Example:
svc := s3manager.NewDownloader(session)
fooFile, err := os.Open("/tmp/foo.file")
if err != nil {
return err
}
barFile, err := os.Open("/tmp/bar.file")
if err != nil {
return err
}
objects := []s3manager.BatchDownloadObject {
{
Object: &s3.GetObjectInput {
Bucket: aws.String("bucket"),
Key: aws.String("foo"),
},
Writer: fooFile,
},
{
Object: &s3.GetObjectInput {
Bucket: aws.String("bucket"),
Key: aws.String("bar"),
},
Writer: barFile,
},
}
iter := &s3manager.DownloadObjectsIterator{Objects: objects}
if err := svc.DownloadWithIterator(aws.BackgroundContext(), iter); err != nil {
return err
}
type Error ¶
type Error struct {
OrigErr error
Bucket *string
Key *string
}
Error will contain the original error, bucket, and key of the operation that failed during batch operations.
func (*Error) Error ¶func (err *Error) Error() stringtype Errors ¶
type Errors []Error
Errors is a typed alias for a slice of errors to satisfy the error interface.
func (Errors) Error ¶func (errs Errors) Error() stringtype MultiUploadFailure ¶
type MultiUploadFailure interface {
awserr.Error
UploadID() string
}
A MultiUploadFailure wraps a failed S3 multipart upload. An error returned will satisfy this interface when a multi part upload failed to upload all chucks to S3. In the case of a failure the UploadID is needed to operate on the chunks, if any, which were uploaded.
Example:
u := s3manager.NewUploader(opts)
output, err := u.upload(input)
if err != nil {
if multierr, ok := err.(s3manager.MultiUploadFailure); ok {
// Process error and its associated uploadID
fmt.Println("Error:", multierr.Code(), multierr.Message(), multierr.UploadID())
} else {
// Process error generically
fmt.Println("Error:", err.Error())
}
}
type PooledBufferedReadFromProvider ¶
type PooledBufferedReadFromProvider struct {
}
PooledBufferedReadFromProvider is a WriterReadFromProvider that uses a sync.Pool to manage allocation and reuse of *bufio.Writer structures.
func NewPooledBufferedWriterReadFromProvider ¶func NewPooledBufferedWriterReadFromProvider(size int) *PooledBufferedReadFromProvider
NewPooledBufferedWriterReadFromProvider returns a new PooledBufferedReadFromProvider Size is used to control the size of the underlying *bufio.Writer created for calls to GetReadFrom.
func (*PooledBufferedReadFromProvider) GetReadFrom ¶func (p *PooledBufferedReadFromProvider) GetReadFrom(writer io.Writer) (r WriterReadFrom, cleanup func())
GetReadFrom takes an io.Writer and wraps it with a type which satisfies the WriterReadFrom interface/ Additionally a cleanup function is provided which must be called after usage of the WriterReadFrom has been completed in order to allow the reuse of the *bufio.Writer
type ReadSeekerWriteTo ¶type ReadSeekerWriteTo interface {
io.ReadSeeker
io.WriterTo
}
ReadSeekerWriteTo defines an interface implementing io.WriteTo and io.ReadSeeker
type ReadSeekerWriteToProvider ¶type ReadSeekerWriteToProvider interface {
GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func())
}
ReadSeekerWriteToProvider provides an implementation of io.WriteTo for an io.ReadSeeker
type UploadInput ¶type UploadInput struct {
ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"`
Body io.Reader
Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"`
BucketKeyEnabled *bool `location:"header" locationName:"x-amz-server-side-encryption-bucket-key-enabled" type:"boolean"`
CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"`
ChecksumAlgorithm *string `location:"header" locationName:"x-amz-sdk-checksum-algorithm" type:"string" enum:"ChecksumAlgorithm"`
ChecksumCRC32 *string `location:"header" locationName:"x-amz-checksum-crc32" type:"string"`
ChecksumCRC32C *string `location:"header" locationName:"x-amz-checksum-crc32c" type:"string"`
ChecksumSHA1 *string `location:"header" locationName:"x-amz-checksum-sha1" type:"string"`
ChecksumSHA256 *string `location:"header" locationName:"x-amz-checksum-sha256" type:"string"`
ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"`
ContentEncoding *string `location:"header" locationName:"Content-Encoding" type:"string"`
ContentLanguage *string `location:"header" locationName:"Content-Language" type:"string"`
ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"`
ContentType *string `location:"header" locationName:"Content-Type" type:"string"`
ExpectedBucketOwner *string `location:"header" locationName:"x-amz-expected-bucket-owner" type:"string"`
Expires *time.Time `location:"header" locationName:"Expires" type:"timestamp"`
GrantFullControl *string `location:"header" locationName:"x-amz-grant-full-control" type:"string"`
GrantRead *string `location:"header" locationName:"x-amz-grant-read" type:"string"`
GrantReadACP *string `location:"header" locationName:"x-amz-grant-read-acp" type:"string"`
GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"`
Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"`
Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"`
ObjectLockLegalHoldStatus *string `location:"header" locationName:"x-amz-object-lock-legal-hold" type:"string" enum:"ObjectLockLegalHoldStatus"`
ObjectLockMode *string `location:"header" locationName:"x-amz-object-lock-mode" type:"string" enum:"ObjectLockMode"`
ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"`
RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"`
SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"`
SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"`
SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"`
SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"`
SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"`
ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"`
StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"`
Tagging *string `location:"header" locationName:"x-amz-tagging" type:"string"`
WebsiteRedirectLocation *string `location:"header" locationName:"x-amz-website-redirect-location" type:"string"`
}
UploadInput provides the input parameters for uploading a stream or buffer to an object in an Amazon S3 bucket. This type is similar to the s3 package's PutObjectInput with the exception that the Body member is an io.Reader instead of an io.ReadSeeker.
The ContentMD5 member for pre-computed MD5 checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, the ContentMD5 will be used.
The Checksum members for pre-computed checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, will include the checksum member in the request.
type UploadObjectsIterator ¶type UploadObjectsIterator struct {
Objects []BatchUploadObject
}
UploadObjectsIterator implements the BatchUploadIterator interface and allows for batched upload of objects.
func (*UploadObjectsIterator) Err ¶func (batcher *UploadObjectsIterator) Err() error
Err will return an error. Since this is just used to satisfy the BatchUploadIterator interface this will only return nil.
func (*UploadObjectsIterator) Next ¶func (batcher *UploadObjectsIterator) Next() bool
Next will increment the default iterator's index and ensure that there is another object to iterator to.
func (*UploadObjectsIterator) UploadObject ¶func (batcher *UploadObjectsIterator) UploadObject() BatchUploadObject
UploadObject will return the BatchUploadObject at the current batched index.
type UploadOutput ¶type UploadOutput struct {
Location string
VersionID *string
UploadID string
ETag *string
}
UploadOutput represents a response from the Upload() call.
type Uploader ¶type Uploader struct {
PartSize int64
Concurrency int
LeavePartsOnError bool
MaxUploadParts int
S3 s3iface.S3API
RequestOptions []request.Option
BufferProvider ReadSeekerWriteToProvider
}
The Uploader structure that calls Upload(). It is safe to call Upload() on this structure for multiple objects and across concurrent goroutines. Mutating the Uploader's properties is not safe to be done concurrently.
The ContentMD5 member for pre-computed MD5 checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, the ContentMD5 will be used.
The Checksum members for pre-computed checksums will be ignored for multipart uploads. Objects that will be uploaded in a single part, will include the checksum member in the request.
func NewUploader ¶func NewUploader(c client.ConfigProvider, options ...func(*Uploader)) *Uploader
NewUploader creates a new Uploader instance to upload objects to S3. Pass In additional functional options to customize the uploader's behavior. Requires a client.ConfigProvider in order to create a S3 service client. The session.Session satisfies the client.ConfigProvider interface.
Example:
// The session the S3 Uploader will use
sess := session.Must(session.NewSession())
// Create an uploader with the session and default options
uploader := s3manager.NewUploader(sess)
// Create an uploader with the session and custom options
uploader := s3manager.NewUploader(session, func(u *s3manager.Uploader) {
u.PartSize = 64 * 1024 * 1024 // 64MB per part
})
â¹ Example (OverrideReadSeekerProvider)
â¾ Example (OverrideReadSeekerProvider)
ExampleNewUploader_overrideReadSeekerProvider gives an example on a custom ReadSeekerWriteToProvider can be provided to Uploader to define how parts will be buffered in memory.
Code:
sess := session.Must(session.NewSession())
uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.BufferProvider = s3manager.NewBufferedReadSeekerWriteToPool(25 * 1024 * 1024)
})
_, err := uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String("examplebucket"),
Key: aws.String("largeobject"),
Body: bytes.NewReader([]byte("large_multi_part_upload")),
})
if err != nil {
fmt.Println(err.Error())
}
â¹ Example (OverrideTransport)
â¾ Example (OverrideTransport)
ExampleNewUploader_overrideTransport gives an example on how to override the default HTTP transport. This can be used to tune timeouts such as response headers, or write / read buffer usage when writing or reading respectively from the net/http transport.
Code:
tr := &http.Transport{
ResponseHeaderTimeout: 1 * time.Second,
}
sess := session.Must(session.NewSession(&aws.Config{
HTTPClient: &http.Client{Transport: tr},
}))
uploader := s3manager.NewUploader(sess)
_, err := uploader.Upload(&s3manager.UploadInput{
Bucket: aws.String("examplebucket"),
Key: aws.String("largeobject"),
Body: bytes.NewReader([]byte("large_multi_part_upload")),
})
if err != nil {
fmt.Println(err.Error())
}
func NewUploaderWithClient ¶
func NewUploaderWithClient(svc s3iface.S3API, options ...func(*Uploader)) *Uploader
NewUploaderWithClient creates a new Uploader instance to upload objects to S3. Pass in additional functional options to customize the uploader's behavior. Requires a S3 service client to make S3 API calls.
Example:
// The session the S3 Uploader will use
sess := session.Must(session.NewSession())
// S3 service client the Upload manager will use.
s3Svc := s3.New(sess)
// Create an uploader with S3 client and default options
uploader := s3manager.NewUploaderWithClient(s3Svc)
// Create an uploader with S3 client and custom options
uploader := s3manager.NewUploaderWithClient(s3Svc, func(u *s3manager.Uploader) {
u.PartSize = 64 * 1024 * 1024 // 64MB per part
})
func (Uploader) Upload ¶
func (u Uploader) Upload(input *UploadInput, options ...func(*Uploader)) (*UploadOutput, error)
Upload uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines. You can configure the buffer size and concurrency through the Uploader's parameters.
Additional functional options can be provided to configure the individual upload. These options are copies of the Uploader instance Upload is called from. Modifying the options will not impact the original Uploader instance.
Use the WithUploaderRequestOptions helper function to pass in request options that will be applied to all API operations made with this uploader.
It is safe to call this method concurrently across goroutines.
Example:
// Upload input parameters
upParams := &s3manager.UploadInput{
Bucket: &bucketName,
Key: &keyName,
Body: file,
}
// Perform an upload.
result, err := uploader.Upload(upParams)
// Perform upload with options different than the those in the Uploader.
result, err := uploader.Upload(upParams, func(u *s3manager.Uploader) {
u.PartSize = 10 * 1024 * 1024 // 10MB part size
u.LeavePartsOnError = true // Don't delete the parts if the upload fails.
})
func (Uploader) UploadWithContext ¶
func (u Uploader) UploadWithContext(ctx aws.Context, input *UploadInput, opts ...func(*Uploader)) (*UploadOutput, error)
UploadWithContext uploads an object to S3, intelligently buffering large files into smaller chunks and sending them in parallel across multiple goroutines. You can configure the buffer size and concurrency through the Uploader's parameters.
UploadWithContext is the same as Upload with the additional support for Context input parameters. The Context must not be nil. A nil Context will cause a panic. Use the context to add deadlining, timeouts, etc. The UploadWithContext may create sub-contexts for individual underlying requests.
Additional functional options can be provided to configure the individual upload. These options are copies of the Uploader instance Upload is called from. Modifying the options will not impact the original Uploader instance.
Use the WithUploaderRequestOptions helper function to pass in request options that will be applied to all API operations made with this uploader.
It is safe to call this method concurrently across goroutines.
func (Uploader) UploadWithIterator ¶func (u Uploader) UploadWithIterator(ctx aws.Context, iter BatchUploadIterator, opts ...func(*Uploader)) error
UploadWithIterator will upload a batched amount of objects to S3. This operation uses the iterator pattern to know which object to upload next. Since this is an interface this allows for custom defined functionality.
Example:
svc:= s3manager.NewUploader(sess)
objects := []BatchUploadObject{
{
Object: &s3manager.UploadInput {
Key: aws.String("key"),
Bucket: aws.String("bucket"),
},
},
}
iter := &s3manager.UploadObjectsIterator{Objects: objects}
if err := svc.UploadWithIterator(aws.BackgroundContext(), iter); err != nil {
return err
}
type WriterReadFrom ¶
type WriterReadFrom interface {
io.Writer
io.ReaderFrom
}
WriterReadFrom defines an interface implementing io.Writer and io.ReaderFrom
type WriterReadFromProvider ¶type WriterReadFromProvider interface {
GetReadFrom(writer io.Writer) (w WriterReadFrom, cleanup func())
}
WriterReadFromProvider provides an implementation of io.ReadFrom for the given io.Writer
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4