Visualising Using Python Plotting Libraries
You can visualize Python on the Spark driver by using the display(<dataframe-name>)
function.
The following Python libraries are supported:
plotly
matplotlib
seaborn
altair
pygal
leather
Note
The display()
function is supported only on PySpark kernels.
Using plotly
import plotly.express as px
data_canada = px.data.gapminder().query("country == 'Canada'")
fig = px.bar(data_canada, x='year', y='pop')
display(fig)
The following image shows the visualization of the plotly plot.

Using matplotlib
import pandas as pd
import matplotlib.pyplot as plt
plt.switch_backend('agg')
sdf = spark.sql("select * from default_qubole_airline_origin_destination limit 10")
data = sdf.toPandas()
data['distance'] = pd.to_numeric(data['distance'], errors='coerce')
data.plot(kind='bar', x='dest', y='distance', color='blue')
display(plt)
The following image shows the visualization of the matplotlib plot.

Using seaborn
import numpy as np
import matplotlib.pyplot as plt
plt.switch_backend('agg')
import seaborn as sns
print(sns)
data = np.random.normal(0, 1, 3)
plt.figure(figsize=(9, 2))
sns.boxplot(x=data);
display(plt)
The following image shows the visualization of the seaborn plot.

Using altair
import altair as alt
import pandas as pd
source = pd.DataFrame({
'a': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'],
'b': [28, 55, 43, 91, 81, 53, 19, 87, 52]
})
plt = alt.Chart(source).mark_bar().encode(
x='a',
y='b'
)
The following image shows the visualization of the altair plot.

Using pygal
import pygal
bar_chart = pygal.Bar()
bar_chart.add('Fibonacci', [0, 1, 1, 2, 3, 5, 8])
display(bar_chart)
The following image shows the visualization of the pygal plot.

Using leather
import random
import leather
dot_data = [(random.randint(0, 250), random.randint(0, 250)) for i in range(100)]
def colorizer(d):
return 'rgb(%i, %i, %i)' % (d.x, d.y, 150)
chart = leather.Chart('Colorized dots')
chart.add_dots(dot_data, fill_color=colorizer)
display(chart)
The following image shows the visualization of the leather plot.

For other plot types, refer to the PlotExamplesPySpark.ipynb in the Example Notebooks of the Jupyter notebooks.