A general file client to access files in different backends.
The client loads a file or text in a specified backend from its path and returns it as a binary or text file. There are two ways to choose a backend, the name of backend and the prefix of path. Although both of them can be used to choose a storage backend, backend has a higher priority that is if they are all set, the storage backend will be chosen by the backend argument. If they are all None, the disk backend will be chosen. Note that It can also register other backend accessor with a given name, prefixes, and backend class. In addition, We use the singleton pattern to avoid repeated object creation. If the arguments are the same, the same object will be returned.
Examples
>>> # only set backend >>> file_client = FileClient(backend='petrel') >>> # only set prefix >>> file_client = FileClient(prefix='s3') >>> # set both backend and prefix but use backend to choose client >>> file_client = FileClient(backend='petrel', prefix='s3') >>> # if the arguments are the same, the same object is returned >>> file_client1 = FileClient(backend='petrel') >>> file_client1 is file_client True
The backend object.
Check whether a file path exists.
Read data from a given filepath with ‘rb’ mode.
Note
There are two types of return values for get, one is bytes and the other is memoryview. The advantage of using memoryview is that you can avoid copying, and if you want to convert it to bytes, you can use .tobytes().
filepath (str or Path) – Path to read data.
Expected bytes object or a memory view of the bytes object.
Download data from filepath and write the data to local path.
get_local_path is decorated by contxtlib.contextmanager(). It can be called with with statement, and when exists from the with statement, the temporary path will be released.
Note
If the filepath is a local path, just return itself.
Warning
get_local_path is an experimental interface that may change in the future.
filepath (str or Path) – Path to be read data.
Examples
>>> file_client = FileClient(prefix='s3')
>>> with file_client.get_local_path('s3://bucket/abc.jpg') as path:
... # do something here
Read data from a given filepath with ‘r’ mode.
Infer a suitable file client based on the URI and arguments.
Examples
>>> uri = 's3://path/of/your/file'
>>> file_client = FileClient.infer_client(uri=uri)
>>> file_client_args = {'backend': 'petrel'}
>>> file_client = FileClient.infer_client(file_client_args)
Instantiated FileClient object.
Check whether a file path is a directory.
Check whether a file path is a file.
Concatenate all file paths.
Join one or more filepath components intelligently. The return value is the concatenation of filepath and any members of *filepaths.
Scan a directory to find the interested directories or files in arbitrary order.
dir_path (str | Path) – Path of the directory.
list_dir (bool) – List the directories. Defaults to True.
list_file (bool) – List the path of files. Defaults to True.
suffix (str or tuple[str], optional) – File suffix that we are interested in. Defaults to None.
recursive (bool) – If set to True, recursively scan the directory. Defaults to False.
Iterable[str] – A relative path to dir_path.
Parse the prefix of a uri.
uri (str | Path) – Uri to be parsed that contains the file prefix.
str | None
Examples
>>> FileClient.parse_uri_prefix('s3://path/of/your/file')
's3'
Write data to a given filepath with ‘wb’ mode.
Note
put should create a directory if the directory of filepath does not exist.
Write data to a given filepath with ‘w’ mode.
Note
put_text should create a directory if the directory of filepath does not exist.
Register a backend to FileClient.
This method can be used as a normal class method or a decorator.
class NewBackend(BaseStorageBackend):
def get(self, filepath):
return filepath
def get_text(self, filepath):
return filepath
FileClient.register_backend('new', NewBackend)
or
@FileClient.register_backend('new')
class NewBackend(BaseStorageBackend):
def get(self, filepath):
return filepath
def get_text(self, filepath):
return filepath
name (str) – The name of the registered backend.
backend (class, optional) – The backend class to be registered, which must be a subclass of BaseStorageBackend. When this method is used as a decorator, backend is None. Defaults to None.
force (bool, optional) – Whether to override the backend if the name has already been registered. Defaults to False.
prefixes (str or list[str] or tuple[str], optional) – The prefixes of the registered storage backend. Defaults to None. New in version 1.3.15.
Remove a file.
filepath (str, Path) – Path to be removed.
None
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4