Package swift provides an easy to use interface to Swift / Openstack Object Storage / Rackspace Cloud Files
Standard Usage ¶Most of the work is done through the Container*() and Object*() methods.
All methods are safe to use concurrently in multiple go routines.
Object Versioning ¶As defined by http://docs.openstack.org/api/openstack-object-storage/1.0/content/Object_Versioning-e1e3230.html#d6e983 one can create a container which allows for version control of files. The suggested method is to create a version container for holding all non-current files, and a current container for holding the latest version that the file points to. The container and objects inside it can be used in the standard manner, however, pushing a file multiple times will result in it being copied to the version container and the new file put in it's place. If the current file is deleted, the previous file in the version container will replace it. This means that if a file is updated 5 times, it must be deleted 5 times to be completely removed from the system.
Rackspace Sub Module ¶This module specifically allows the enabling/disabling of Rackspace Cloud File CDN management on a container. This is specific to the Rackspace API and not Swift/Openstack, therefore it has been placed in a submodule. One can easily create a RsConnection and use it like the standard Connection to access and manipulate containers and objects.
const ( DefaultUserAgent = "goswift/1.0" DefaultRetries = 3 TimeFormat = "2006-01-02T15:04:05" UploadTar = "tar" UploadTarGzip = "tar.gz" UploadTarBzip2 = "tar.bz2" )View Source
var (
NotModified = newError(304, "Not Modified")
BadRequest = newError(400, "Bad Request")
AuthorizationFailed = newError(401, "Authorization Failed")
ContainerNotFound = newError(404, "Container Not Found")
ContainerNotEmpty = newError(409, "Container Not Empty")
ObjectNotFound = newError(404, "Object Not Found")
ObjectCorrupted = newError(422, "Object Corrupted")
TimeoutError = newError(408, "Timeout when reading or writing data")
Forbidden = newError(403, "Operation forbidden")
TooLargeObject = newError(413, "Too Large Object")
RateLimit = newError(498, "Rate Limit")
TooManyRequests = newError(429, "TooManyRequests")
ContainerErrorMap = errorMap{
400: BadRequest,
403: Forbidden,
404: ContainerNotFound,
409: ContainerNotEmpty,
498: RateLimit,
}
)
NotLargeObject is returned if an operation is performed on an object which isn't large.
SLONotSupported is returned as an error when Static Large Objects are not supported.
FloatStringToTime converts a floating point number string to a time.Time
The string is floating point number of seconds since the epoch (Unix time). The number should be in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z"
Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.
If an error is returned then time will be returned as the zero time.
TimeToFloatString converts a time.Time object to a floating point string
The string is floating point number of seconds since the epoch (Unix time). The number is in fixed point format (not exponential), eg "1354040105.123456789" which represents the time "2012-11-27T18:15:05.123456789Z". Trailing zeros will be dropped from the output.
Some care is taken to preserve all the accuracy in the time.Time (which wouldn't happen with a naive conversion through float64) so a round trip conversion won't change the data.
Account contains information about this account.
Auth defines the operations needed to authenticate with swift
This encapsulates the different authentication schemes in use
BulkDeleteResult stores results of BulkDelete().
Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where the object was to be deleted, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".
BulkUploadResult stores results of BulkUpload().
Individual errors may (or may not) be returned by Errors. Errors is a map whose keys are a full path of where an object was to be created, and whose values are Error objects. A full path of object looks like "/API_VERSION/USER_ACCOUNT/CONTAINER/OBJECT_PATH".
Connection holds the details of the connection to the swift server.
You need to provide UserName, ApiKey and AuthUrl when you create a connection then call Authenticate on it.
The auth version in use will be detected from the AuthURL - you can override this with the AuthVersion parameter.
If using v2 auth you can also set Region in the Connection structure. If you don't set Region you will get the default region which may not be what you want.
For reference some common AuthUrls looks like this:
Rackspace US https://auth.api.rackspacecloud.com/v1.0 Rackspace UK https://lon.auth.api.rackspacecloud.com/v1.0 Rackspace v2 https://identity.api.rackspacecloud.com/v2.0 Memset Memstore UK https://auth.storage.memset.com/v1.0 Memstore v2 https://auth.storage.memset.com/v2.0
When using Google Appengine you must provide the Connection with an appengine-specific Transport:
import (
"appengine/urlfetch"
"fmt"
"github.com/ncw/swift/v2"
)
func handler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
tr := urlfetch.Transport{Context: ctx}
c := swift.Connection{
UserName: "user",
ApiKey: "key",
AuthUrl: "auth_url",
Transport: tr,
}
_ := c.Authenticate()
containers, _ := c.ContainerNames(nil)
fmt.Fprintf(w, "containers: %q", containers)
}
If you don't supply a Transport, one is made which relies on http.ProxyFromEnvironment (http://golang.org/pkg/net/http/#ProxyFromEnvironment). This means that the connection will respect the HTTP proxy specified by the environment variables $HTTP_PROXY and $NO_PROXY.
ctx := context.Background()
// Create a v1 auth connection
c := &swift.Connection{
// This should be your username
UserName: "user",
// This should be your api key
ApiKey: "key",
// This should be a v1 auth url, eg
// Rackspace US https://auth.api.rackspacecloud.com/v1.0
// Rackspace UK https://lon.auth.api.rackspacecloud.com/v1.0
// Memset Memstore UK https://auth.storage.memset.com/v1.0
AuthUrl: "auth_url",
}
// Authenticate
err := c.Authenticate(ctx)
if err != nil {
panic(err)
}
// List all the containers
containers, err := c.ContainerNames(ctx, nil)
if err != nil {
panic(err)
}
fmt.Println(containers)
// etc...
// ------ or alternatively create a v2 connection ------
// Create a v2 auth connection
c = &swift.Connection{
// This is the sub user for the storage - eg "admin"
UserName: "user",
// This should be your api key
ApiKey: "key",
// This should be a version2 auth url, eg
// Rackspace v2 https://identity.api.rackspacecloud.com/v2.0
// Memset Memstore v2 https://auth.storage.memset.com/v2.0
AuthUrl: "v2_auth_url",
// Region to use - default is use first region if unset
Region: "LON",
// Name of the tenant - this is likely your username
Tenant: "jim",
}
// as above...
Account returns info about the account in an Account struct.
AccountUpdate adds, replaces or remove account metadata.
Add or update keys by mentioning them in the Headers.
Remove keys by setting them to an empty string.
ApplyEnvironment reads environment variables and applies them to the Connection structure. It won't overwrite any parameters which are already set in the Connection struct.
To make a new Connection object entirely from the environment you would do:
c := new(Connection)
err := c.ApplyEnvironment()
if err != nil { log.Fatal(err) }
The naming of these variables follows the official Openstack naming scheme so it should be compatible with OpenStack rc files.
For v1 authentication (obsolete)
ST_AUTH - Auth URL ST_USER - UserName for api ST_KEY - Key for api access
For v2 authentication
OS_AUTH_URL - Auth URL OS_USERNAME - UserName for api OS_PASSWORD - Key for api access OS_TENANT_NAME - Name of the tenant OS_TENANT_ID - Id of the tenant OS_REGION_NAME - Region to use - default is use first region
For v3 authentication
OS_AUTH_URL - Auth URL OS_USERNAME - UserName for api OS_USER_ID - User Id OS_PASSWORD - Key for api access OS_APPLICATION_CREDENTIAL_ID - Application Credential ID OS_APPLICATION_CREDENTIAL_NAME - Application Credential Name OS_APPLICATION_CREDENTIAL_SECRET - Application Credential Secret OS_USER_DOMAIN_NAME - User's domain name OS_USER_DOMAIN_ID - User's domain Id OS_PROJECT_NAME - Name of the project OS_PROJECT_DOMAIN_NAME - Name of the tenant's domain, only needed if it differs from the user domain OS_PROJECT_DOMAIN_ID - Id of the tenant's domain, only needed if it differs the from user domain OS_TRUST_ID - If of the trust OS_REGION_NAME - Region to use - default is use first region
Other
OS_ENDPOINT_TYPE - Endpoint type public, internal or admin ST_AUTH_VERSION - Choose auth version - 1, 2 or 3 or leave at 0 for autodetect
For manual authentication
OS_STORAGE_URL - storage URL from alternate authentication OS_AUTH_TOKEN - Auth Token from alternate authentication
Library specific
GOSWIFT_RETRIES - Retries on error (default is 3) GOSWIFT_USER_AGENT - HTTP User agent (default goswift/1.0) GOSWIFT_CONNECT_TIMEOUT - Connect channel timeout with unit, eg "10s", "100ms" (default "10s") GOSWIFT_TIMEOUT - Data channel timeout with unit, eg "10s", "100ms" (default "60s") GOSWIFT_INTERNAL - Set this to "true" to use the the internal network (obsolete - use OS_ENDPOINT_TYPE)
Authenticate connects to the Swift server.
If you don't call it before calling one of the connection methods then it will be called for you on the first access.
Authenticated returns a boolean to show if the current connection is authenticated.
Doesn't actually check the credentials against the server.
Call runs a remote command on the targetUrl, returns a response, headers and possible error.
operation is GET, HEAD etc container is the name of a container Any other parameters (if not None) are added to the targetUrl
Returns a response or an error. If response is returned then the resp.Body must be read completely and resp.Body.Close() must be called on it, unless noResponse is set in which case the body will be closed in this function
If "Content-Length" is set in p.Headers it will be used - this can be used to override the default chunked transfer encoding for uploads.
This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired
This method is exported so extensions can call it.
Container returns info about a single container including any metadata in the headers.
ContainerCreate creates a container.
If you don't want to add Headers just pass in nil
No error is returned if it already exists but the metadata if any will be updated.
ContainerDelete deletes a container.
May return ContainerDoesNotExist or ContainerNotEmpty
ContainerNames returns a slice of names of containers in this account.
ContainerNamesAll is like ContainerNames but it returns all the Containers
It calls ContainerNames multiple times using the Marker parameter ¶It has a default Limit parameter but you may pass in your own
ContainerUpdate adds, replaces or removes container metadata.
Add or update keys by mentioning them in the Metadata.
Remove keys by setting them to an empty string.
Container metadata can only be read with Container() not with Containers().
Containers returns a slice of structures with full information as described in Container.
ContainersAll is like Containers but it returns all the Containers
It calls Containers multiple times using the Marker parameter ¶It has a default Limit parameter but you may pass in your own
DynamicLargeObjectCreate creates or truncates an existing dynamic large object returning a writeable object. This sets opts.Flags to an appropriate value before calling DynamicLargeObjectCreateFile
DynamicLargeObjectCreateFile creates a dynamic large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passes to the largeObjectCreate method.
DynamicLargeObjectDelete deletes a dynamic large object and all of its segments.
DynamicLargeObjectMove moves a dynamic large object from srcContainer, srcObjectName to dstContainer, dstObjectName
GetStorageUrl returns Swift storage URL.
LargeObjectDelete deletes the large object named by container, path
LargeObjectGetSegments returns all the segments that compose an object If the object is a Dynamic Large Object (DLO), it just returns the objects that have the prefix as indicated by the manifest. If the object is a Static Large Object (SLO), it retrieves the JSON content of the manifest and return all the segments of it.
Object returns info about a single object including any metadata in the header.
May return ObjectNotFound.
Use headers.ObjectMetadata() to read the metadata in the Headers.
ObjectCopy does a server side copy of an object to a new position
All metadata is preserved. If metadata is set in the headers then it overrides the old metadata on the copied object.
The destination container must exist before the copy.
You can use this to copy an object to itself - this is the only way to update the content type of an object.
ObjectCreate creates or updates the object in the container. It returns an io.WriteCloser you should write the contents to. You MUST call Close() on it and you MUST check the error return from Close().
If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted on Close()
If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted on Close() if it is incorrect.
If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".
If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension
ObjectDelete deletes the object.
May return ObjectNotFound if the object isn't found
ObjectGet gets the object into the io.Writer contents.
Returns the headers of the response.
If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.
headers["Content-Type"] will give the content type if desired.
ObjectGetBytes returns an object as a []byte.
This is a simplified interface which checks the MD5
ObjectGetString returns an object as a string.
This is a simplified interface which checks the MD5
ObjectMove does a server side move of an object to a new position
This is a convenience method which calls ObjectCopy then ObjectDelete ¶All metadata is preserved.
The destination container must exist before the copy.
ObjectNames returns a slice of names of objects in a given container.
ObjectNamesAll is like ObjectNames but it returns all the Objects
It calls ObjectNames multiple times using the Marker parameter. Marker is reset unless KeepMarker is set
It has a default Limit parameter but you may pass in your own
ObjectOpen returns an ObjectOpenFile for reading the contents of the object. This satisfies the io.ReadCloser and the io.Seeker interfaces.
You must call Close() on contents when finished ¶Returns the headers of the response.
If checkHash is true then it will calculate the md5sum of the file as it is being received and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted. It will also check the length returned. No checking will be done if you don't read all the contents.
Note that objects with X-Object-Manifest or X-Static-Large-Object set won't ever have their md5sum's checked as the md5sum reported on the object is actually the md5sum of the md5sums of the parts. This isn't very helpful to detect a corrupted download as the size of the parts aren't known without doing more operations. If you want to ensure integrity of an object with a manifest then you will need to download everything in the manifest separately.
headers["Content-Type"] will give the content type if desired.
ObjectPut creates or updates the path in the container from contents. contents should be an open io.Reader which will have all its contents read.
This is a low level interface.
If checkHash is True then it will calculate the MD5 Hash of the file as it is being uploaded and check it against that returned from the server. If it is wrong then it will return ObjectCorrupted.
If you know the MD5 hash of the object ahead of time then set the Hash parameter and it will be sent to the server (as an Etag header) and the server will check the MD5 itself after the upload, and this will return ObjectCorrupted if it is incorrect.
If you don't want any error protection (not recommended) then set checkHash to false and Hash to "".
If contentType is set it will be used, otherwise one will be guessed from objectName using mime.TypeByExtension
ObjectPutBytes creates an object from a []byte in a container.
This is a simplified interface which checks the MD5.
ObjectPutString creates an object from a string in a container.
This is a simplified interface which checks the MD5
ObjectTempUrl returns a temporary URL for an object
ObjectUpdate adds, replaces or removes object metadata.
Add or Update keys by mentioning them in the Metadata. Use Metadata.ObjectHeaders and Headers.ObjectMetadata to convert your Metadata to and from normal HTTP headers.
This removes all metadata previously added to the object and replaces it with that passed in so to delete keys, just don't mention them the headers you pass in.
Object metadata can only be read with Object() not with Objects().
This can also be used to set headers not already assigned such as X-Delete-At or X-Delete-After for expiring objects.
You cannot use this to change any of the object's other headers such as Content-Type, ETag, etc.
Refer to copying an object when you need to update metadata or other headers such as Content-Type or CORS headers.
May return ObjectNotFound.
ObjectUpdateContentType updates the content type of an object
This is a convenience method which calls ObjectCopy ¶All other metadata is preserved.
Objects returns a slice of Object with information about each object in the container.
If Delimiter is set in the opts then PseudoDirectory may be set, with ContentType 'application/directory'. These are not real objects but represent directories of objects which haven't had an object created for them.
ObjectsAll is like Objects but it returns an unlimited number of Objects in a slice
It calls Objects multiple times using the Marker parameter
ObjectsWalk is uses to iterate through all the objects in chunks as returned by Objects or ObjectNames using the Marker and Limit parameters in the ObjectsOpts.
Pass in a closure `walkFn` which calls Objects or ObjectNames with the *ObjectsOpts passed to it and does something with the results.
Errors will be returned from this function ¶It has a default Limit parameter but you may pass in your own
c, rollback := makeConnection(nil)
defer rollback()
objects := make([]string, 0)
err := c.ObjectsWalk(context.Background(), container, nil, func(ctx context.Context, opts *swift.ObjectsOpts) (interface{}, error) {
newObjects, err := c.ObjectNames(ctx, container, opts)
if err == nil {
objects = append(objects, newObjects...)
}
return newObjects, err
})
fmt.Println("Found all the objects", objects, err)
Discover Swift configuration by doing a request against /info
StaticLargeObjectCreate creates or truncates an existing static large object returning a writeable object. This sets opts.Flags to an appropriate value before calling StaticLargeObjectCreateFile
StaticLargeObjectCreateFile creates a static large object returning an object which satisfies io.Writer, io.Seeker, io.Closer and io.ReaderFrom. The flags are as passed to the largeObjectCreate method.
StaticLargeObjectDelete deletes a static large object and all of its segments.
StaticLargeObjectMove moves a static large object from srcContainer, srcObjectName to dstContainer, dstObjectName
UnAuthenticate removes the authentication from the Connection.
VersionContainerCreate is a helper method for creating and enabling version controlled containers.
It builds the current object container, the non-current object version container, and enables versioning.
If the server doesn't support versioning then it will return Forbidden however it will have created both the containers at that point.
c, rollback := makeConnection(nil)
defer rollback()
// Use the helper method to create the current and versions container.
if err := c.VersionContainerCreate(context.Background(), "cds", "cd-versions"); err != nil {
fmt.Print(err.Error())
}
VersionDisable disables versioning on the current container.
c, rollback := makeConnection(nil)
defer rollback()
// Disable versioning on a container. Note that this does not delete the versioning container.
err := c.VersionDisable(context.Background(), "movies")
if err != nil {
panic(err)
}
VersionEnable enables versioning on the current container with version as the tracking container.
May return Forbidden if this isn't supported by the server
ctx := context.Background()
c, rollback := makeConnection(nil)
defer rollback()
// Build the containers manually and enable them.
if err := c.ContainerCreate(ctx, "movie-versions", nil); err != nil {
fmt.Print(err.Error())
}
if err := c.ContainerCreate(ctx, "movies", nil); err != nil {
fmt.Print(err.Error())
}
if err := c.VersionEnable(ctx, "movies", "movie-versions"); err != nil {
fmt.Print(err.Error())
}
// Access the primary container as usual with ObjectCreate(), ObjectPut(), etc.
// etc...
VersionObjectList returns a list of older versions of the object.
Objects are returned in the format <length><object_name>/<timestamp>
Container contains information about a container
ContainersOpts is options for Containers() and ContainerNames()
type CustomEndpointAuthenticator interface {
StorageUrlForEndpoint(endpointType EndpointType) string
}
type DynamicLargeObjectCreateFile struct {
}
DynamicLargeObjectCreateFile represents an open static large object
Close satisfies the io.Closer interface
Seek sets the offset for the next write operation
func (file *DynamicLargeObjectCreateFile) Size() int64
func (file *DynamicLargeObjectCreateFile) Write(buf []byte) (int, error)
type Error struct {
StatusCode int
Text string
}
Error - all errors generated by this package are of this type. Other error may be passed on from library functions though.
Error satisfy the error interface.
type Expireser interface {
Expires() time.Time
}
Expireser is an optional interface to read the expiration time of the token
Headers stores HTTP headers (can only have one of each header like Swift).
AccountMetadata converts Headers from account to a Metadata.
The keys in the Metadata will be converted to lower case.
ContainerMetadata converts Headers from container to a Metadata.
The keys in the Metadata will be converted to lower case.
Metadata gets the Metadata starting with the metaPrefix out of the Headers.
The keys in the Metadata will be converted to lower case
ObjectMetadata converts Headers from object to a Metadata.
The keys in the Metadata will be converted to lower case.
LargeObjectOpts describes how a large object should be created
Metadata stores account, container or object metadata.
AccountHeaders converts the Metadata for the account.
ContainerHeaders converts the Metadata for the container.
GetModTime reads a modification time (mtime) from a Metadata object
This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.
If an error is returned then time will be returned as the zero time.
Headers convert the Metadata starting with the metaPrefix into a Headers.
The keys in the Metadata will be converted from lower case to http Canonical (see http.CanonicalHeaderKey).
ObjectHeaders converts the Metadata for the object.
SetModTime writes an modification time (mtime) to a Metadata object
This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time (as read using os.Stat) for an object. It is stored using the key 'mtime', which for example when written to an object will be 'X-Object-Meta-Mtime'.
type Object struct {
Name string `json:"name"`
ContentType string `json:"content_type"`
Bytes int64 `json:"bytes"`
ServerLastModified string `json:"last_modified"`
LastModified time.Time
Hash string `json:"hash"`
SLOHash string `json:"slo_etag"`
PseudoDirectory bool
SubDir string `json:"subdir"`
ObjectType ObjectType
}
Object contains information about an object
type ObjectCreateFile struct {
}
ObjectCreateFile represents a swift object open for writing
Close the object and checks the md5sum if it was required.
Also returns any other errors from the server (eg container not found) so it is very important to check the errors on this method.
CloseWithError closes the object, aborting the upload.
Headers returns the response headers from the created object if the upload has been completed. The Close() method must be called on an ObjectCreateFile before this method.
Write bytes to the object - see io.Writer
type ObjectOpenFile struct {
}
ObjectOpenFile represents a swift object open for reading
Close the object and checks the length and md5sum if it was required and all the object was read
Length gets the objects content length either from a cached copy or from the server.
Read bytes from the object - see io.Reader
Seek sets the offset for the next Read to offset, interpreted according to whence: 0 means relative to the origin of the file, 1 means relative to the current offset, and 2 means relative to the end. Seek returns the new offset and an Error, if any.
Seek uses HTTP Range headers which, if the file pointer is moved, will involve reopening the HTTP connection.
Note that you can't seek to the end of a file or beyond; HTTP Range requests don't support the file pointer being outside the data, unlike os.File
Seek(0, 1) will return the current file pointer.
ObjectType is the type of the swift object, regular, static large, or dynamic large.
const ( RegularObjectType ObjectType = iota StaticLargeObjectType DynamicLargeObjectType )
Values that ObjectType can take
ObjectOpts is options for Objects() and ObjectNames()
A closure defined by the caller to iterate through all objects
Call Objects or ObjectNames from here with the context.Context and *ObjectOpts passed in
Do whatever is required with the results then return them
RequestOpts contains parameters for Connection.storage.
type StaticLargeObjectCreateFile struct {
}
StaticLargeObjectCreateFile represents an open static large object
Seek sets the offset for the next write operation
func (file *StaticLargeObjectCreateFile) Size() int64
func (file *StaticLargeObjectCreateFile) Write(buf []byte) (int, error)
type SwiftInfo map[string]interface{}
SwiftInfo contains the JSON object returned by Swift when the /info route is queried. The object contains, among others, the Swift version, the enabled middlewares and their configuration
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4