Spark¶

New Feature¶

SPAR-3979: Dynamic Partition Pruning: Using Dynamic Filtering values, Dynamic Partition Pruning selects the specific partitions within the table that need to be read at runtime. This improves job performance for queries in which the join condition is on the partitioned column, by significantly reducing the amount of data read and processed. Dynamic Partition Pruning is available in Spark 2.4.3 and later versions. Gradual Rollout.

SPAR-4125: Fixes a problem that could cause the loss of a cluster node to be incorrectly reported as ExecutorLostDueToOOM.