Enable S3 Listing and Wild Card Optimization¶
Listing files in the S3 location can be a slow process that can be optimized using a configuration option. Similarly, listing directories that contains wildcards can be also slow that can be made faster using a configuration option.
Enable S3 Listing Optimization¶
As part of the split computation, the Hive must list all files in the table’s S3 location. Hadoop jobs must list all
files in an S3 location. The implementation in Apache Hadoop for listing files in S3 is very slow. Optimizations have
been incorporated to speed this up by setting
fs.s3.inputpathprocessor=true by default for Hive queries and Hadoop jobs.
Enable WildCard Optimization¶
There are two forms of wildcard character: asterisk (
*) that matches zero or more arbitrary characters and
question mark (
?) that matches exactly one arbitrary character.
While listing directories that contain an asterisk, for example, s3://my-bucket/my-dir/
*/file.csv, the process
may become slow. To speed up the listing process, set
mapred.job.natives3filesystem.globstatus.use to true.