airfs: A Python library for cloud and remote file Systems¶
airfs brings standard Python I/O to various storage (like cloud objects storage, remote file-systems, …) by providing:
- Abstract classes of Cloud objects with the complete
- Features equivalent to the standard library (
shutil) for seamlessly managing storage objects and local files.
These functions are source agnostic and always provide the same interface for all files from storage or local file systems.
Buffered storage objects also support the following features:
- Buffered asynchronous writing of any object size.
- Buffered asynchronous preloading in reading mode.
- Write or read lock depending on memory usage limitation.
- Maximization of bandwidth using parallels connections.
Example of code:
import airfs # Open an object on AWS S3 as text for reading with airfs.open('s3://my_bucket/my_object.txt', 'rt') as file: text = file.read() # Open an object on AWS S3 as binary for writing with airfs.open('s3://my_bucket/my_object.bin', 'wb') as file: file.write(b'binary_data') # Copy file from the local file system to OpenStack Swift airfs.copy( 'my_file', 'https://objects.my_cloud.com/v1/12345678912345/my_container/my_object' ) # Get size of a file over internet airfs.getsize('https://www.example.org/any_object') >>> 956
airfs is compatible with the following storage services:
- Alibaba Cloud OSS
- Amazon Web Services S3 (and compatible)
- GitHub (Read Only)
- Microsoft Azure Blobs Storage
- Microsoft Azure Files Storage
- OpenStack Swift / Object Store
airfs can also access any publicly accessible file via HTTP/HTTPS (Read only).
All storage are not real file systems and may have the following limitations ( Depending on the selected storage):
- Storage objects may not be seekable in write mode.
- Storage objects may be written entirely at once.
- Storage objects may not be locked when accessed.
- Storage object attributes available may be more limited.
- Some file-system features like symbolic links may ne be present
- Getting Started
- airfs API