HDFS integration with Openstack Swift

The overall model in HDFS data flow with Openstack Swift integration.

The works of HDFS and Openstack Swift have been done, refer to  HADOOP-8545. It introduces an new concept in big data storage. And benefit from this marriage,
Hadoop cluster would be more agile and easier to be managed.
However, there are still some debates about this new model. One of the most controversial issues are the data
locality and performance, and there is an report to address this issue, refer to
AMPLab
analysis. The second one is the security and how to manage user's privacy. To date, the new model doesn't have
any mechanism of permission control, refer to
this warning.

The user specifies the connection URI, such as

hdfs://namenode:8020

or

swift://store.example.org

The HDFS client would initialize the corresponding implements for the class FileSystem.
The class FileSystem defines the interfaces of file operations.

Next we can drill down and take "file read" as an example to explain the data flow.

Anatomy of a File Read in case of the Swift backend

Before any operation of HDFS, the application should initialize the class FileSystem by following, for example in java

String uri = "swift://storage.example.org";
FileSystem fs = FileSystem(uri, new Configuration());

Because the scheme is specified as "swift" by user, the instance of FileSystem would be SwiftNativeFileSystem.