Sidecar is the Hadoop – compatible caching (read/write) file system. It was specifically designed to support faster read/write access to remote cloud storage systems.
file– local file system (for testing only)hdfs– Hadoop Distributed File System.s3a– AWS S3adl– Azure Data Lake File System Gen 1abfs– ABFS Azure Data Lake Gen 2abfss– Secure ABFSS Azure Data Lake Gen 2wasb– WASB native Azure File Systemwasbs– Secure WASB native Azure FSgs– Google Cloud Storageoss– Chineese Aliyun OSS (Alibaba Cloud)swift– Open Stack Swift Object Store
You will need to modify Hadoop configuration file to use a caching version of a remote file system.
fs.file.impl=com.carrotdata.sidecar.fs.file.FileSidecarCachingFileSystem
fs.hdfs.impl=com.carrotdata.sidecar.hdfs.SidecarDistributedFileSystem
fs.s3a.impl=com.carrotdata.sidecar.s3a.SidecarS3AFileSystem
fs.adl.impl=com.carrotdata.sidecar.adl.SidecarAdlFileSystem
fs.abfs.impl=com.carrotdata.sidecar.abfs.SidecarAzureBlobFileSystem
fs.abfss.impl=com.carrotdata.sidecar.abfs.SecureSidecarAzureBlobFileSystem
fs.wasb.impl=com.carrotdata.sidecar.wasb.SidecarNativeAzureBlobFileSystem
fs.wasbs.impl=com.carrotdata.sidecar.wasb.SecureSidecarNativeAzureBlobFileSystem
fs.gs.impl=com.carrotdata.sidecar.gcs.SidecarGoogleHadoopFileSystem
fs.oss.impl=com.carrotdata.sidecar.oss.SidecarAliyunOSSFileSystem
fs.oss.impl=com.carrotdata.sidecar.oss.SidecarAliyunOSSFileSystem
fs.swift.impl=com.carrotdata.sidecar.swift.SidecarSwiftNativeFileSystem
https://github.com/VladRodionov/sidecar
Leave a Reply