Sidecar is the Hadoop – compatible caching (read/write) file system. It was specifically designed to support faster read/write access to remote cloud storage systems.
file
– local file system (for testing only)hdfs
– Hadoop Distributed File System.s3a
– AWS S3adl
– Azure Data Lake File System Gen 1abfs
– ABFS Azure Data Lake Gen 2abfss
– Secure ABFSS Azure Data Lake Gen 2wasb
– WASB native Azure File Systemwasbs
– Secure WASB native Azure FSgs
– Google Cloud Storageoss
– Chineese Aliyun OSS (Alibaba Cloud)swift
– Open Stack Swift Object Store
You will need to modify Hadoop configuration file to use a caching version of a remote file system.
fs.file.impl=com.carrotdata.sidecar.fs.file.FileSidecarCachingFileSystem
fs.hdfs.impl=com.carrotdata.sidecar.hdfs.SidecarDistributedFileSystem
fs.s3a.impl=com.carrotdata.sidecar.s3a.SidecarS3AFileSystem
fs.adl.impl=com.carrotdata.sidecar.adl.SidecarAdlFileSystem
fs.abfs.impl=com.carrotdata.sidecar.abfs.SidecarAzureBlobFileSystem
fs.abfss.impl=com.carrotdata.sidecar.abfs.SecureSidecarAzureBlobFileSystem
fs.wasb.impl=com.carrotdata.sidecar.wasb.SidecarNativeAzureBlobFileSystem
fs.wasbs.impl=com.carrotdata.sidecar.wasb.SecureSidecarNativeAzureBlobFileSystem
fs.gs.impl=com.carrotdata.sidecar.gcs.SidecarGoogleHadoopFileSystem
fs.oss.impl=com.carrotdata.sidecar.oss.SidecarAliyunOSSFileSystem
fs.oss.impl=com.carrotdata.sidecar.oss.SidecarAliyunOSSFileSystem
fs.swift.impl=com.carrotdata.sidecar.swift.SidecarSwiftNativeFileSystem