Refresh Table (AWS)

Amazon S3 directories often back the Hive partitions and only running the recover partitions command creates Hive partitions. The Refresh Table template can create Hive partitions.

It is important to understand the concept of Stability Interval in the context of the Refresh Table template. After directories in Amazon S3 buckets are created, upload data into them. Do not create a Hive partition corresponding to such S3 directories until the process of populating the directory is complete. The Stability Interval defines the number of minutes that must elapse since the last modification to a directory before the Hive partition corresponding to it is created. See Composing a Refresh Table Query (AWS) for more information.

As an example, consider a process where data is loaded into an Amazon S3 directory at midnight every day. The process of uploading the logs takes about an hour. So, you can configure the Refresh Table template with a Stability Interval of 2 hours (120 minutes) to ensure that the process of uploading the data is complete before the Hive partitions are created.