Converting Plotly charts into images in parallel

We use Plotly charts extensively in the company I work for. They make it easy to create interactive graphics that look good. The Python experience via the Plotly Express library is great and the bar to get started is low.

We have two main use-cases for Plotly graphs:

• For our interactive dashboards with Plotly Dash. The integration of Plotly charts into Dash is obviously great.
• For our PDF reports where we convert the chart into an image before rendering the PDF.

For a typical PDF report, we use 5-20 figures to show the evolution of a particular metric over time, the distribution of some value over a number of categories, or the comparison of different categories next to each other.

To create our PDF reports, we use a combination of Weasyprint, Jinja, and Plotly charts. To render a report as a PDF, we first have to render all graphs as images.

Rendering graphs with Kaleido

To do so, we use the great Kaleido package. It uses a Chrome browser to render the graph and save it as an image. The API is straight forward to use.

from kaleido.scopes.plotly import PlotlyScope

scope = PlotlyScope()
img_bytes = scope.transform(
    figure=figure, format="png", width=1000, height=1000, scale=4,
)

This renders the figure in figure as an image with 1000px height and width and a rendering scale of 4 (i.e. the image actually has dimensions 4000px x 4000px). The higher the scale, the more DPI the final image has, the better it looks, and the larger the final PDF is.

Rendering a lot of graphs

Rendering graphs takes a little bit of time and if you render a lot of them (10-20), it will be a significant share of your program's runtime. To speed up our PDF rendering pipeline, we deployed the following solution.

Internally, Kaleido just outsources the problem of rendering the graph as an image to an included Chrome webbrowser. This means, for Python itself rendering this image is basically waiting for I/O.

To speed up this particular process and since we just wait for I/O, we can use multithreading.

Creating a random graph

Let's start by creating a random figure, like so:

import pandas as pd
import numpy as np
import plotly.graph_objects as go

def get_random_figure() -> go.Figure:
    n_bars = 50
    dates = pd.date_range(start="2021-01-01", end="2021-12-31", freq="M")

    figure = go.Figure()
    for i in range(n_bars):
        values = np.random.rand(len(dates))
        figure.add_trace(go.Bar(x=dates, y=values, name=f"Label {i+1}"))

    figure.update_layout(
        dict(
            barmode="group",
            legend=dict(orientation="h", yanchor="top", xanchor="left"),
        )
    )
    figure.update_layout(yaxis=dict(tickformat=".0%"), xaxis=dict(showgrid=False))
    return figure

Now, turning a figure into an image can be done using the code from above:

from kaleido.scopes.plotly import PlotlyScope
import plotly.graph_objects as go

def figure_to_bytes(figure: go.Figure) -> bytes:
    scope = PlotlyScope()
    return scope.transform(figure=figure, format="png", width=1000, height=1000, scale=4)

And finally we also define for later:

def transform_random_figure() -> bytes:
    return figure_to_bytes(get_random_figure())

Running the image transformation in threads

You may or may not know that due to the GIL (global interpreter lock) in Python, only one thread can execute Python code at the same time. Since the transformation of the graph to an image is not Python code, we can make use of threads to start the transformation of a lot of graphs at the same time and then collect the results.

For that, we define a helper class:

from threading import Thread

# Make the join() method return the value returned by the target function.
class ThreadWithReturnValue(Thread):
    def __init__(self, group=None, target=None, name=None, args=(), kwargs={}, Verbose=None):
        Thread.__init__(self, group, target, name, args, kwargs)
        self._return = None

    def run(self):
        if self._target is not None:
            self._return = self._target(*self._args, **self._kwargs)

    def join(self, *args):
        Thread.join(self, *args)
        return self._return

This class will help us retrieve the result of the transformation (i.e. the bytes of the image).

The next thing we have to do is follow the standard pattern for working with threads in Python:

1. Start the threads you want to start with the start() method.
2. Using the join() method to wait for the thread to return the result.

Our threads should each call transform_random_figure() and then return the bytes. We start 10 threads in this case.

n_threads = 10
threads = [ThreadWithReturnValue(target=transform_random_figure) for _ in range(n_threads)]
for thread in threads:
    thread.start()

The start() method will also call the run() method of the thread that starts the actual logic (i.e. executes the given function, which in our case means transform_random_figure()).

To collect the results, we use the join() method of the threads and write the result into files.

results = [thread.join() for thread in threads]
for i, res in enumerate(results):
    with open(f"result{i}.png", "wb") as f:
        f.write(res)

How it works

The main idea here is to, whenever we want to transform a graph into an image, we start a thread and this thread will wait for the graph to be finished in the background.

Once we put together the whole report, we call join() on all threads and retrieve the images for all graphs and then put them into the report.

This way, we can already generate the whole report without graphs and save time by not waiting for every graph on its own to be transformed.

Summary

In summary, if you want to transform multiple Plotly charts into images, use the multithreading module in the Python standard library to speed up your conversion process.

You can do so very easily just by moving the transform() call into a thread and then waiting for all threads to finish.

Appendix: The Code

import logging
from threading import Thread

import numpy as np
import pandas as pd
import plotly.graph_objects as go
from kaleido.scopes.plotly import PlotlyScope

logging.basicConfig(format="%(asctime)s %(message)s", level=logging.DEBUG)
logger = logging.getLogger()


def get_random_figure() -> go.Figure:
    n_bars = 50
    dates = pd.date_range(start="2021-01-01", end="2021-12-31", freq="M")
    figure = go.Figure()

    for i in range(n_bars):
        values = np.random.rand(len(dates))
        figure.add_trace(go.Bar(x=dates, y=values, name=f"Label {i+1}"))

    figure.update_layout(
        dict(
            barmode="group",
            legend=dict(orientation="h", yanchor="top", xanchor="left"),
        )
    )
    figure.update_layout(yaxis=dict(tickformat=".0%"), xaxis=dict(showgrid=False))
    return figure


def figure_to_bytes(figure: go.Figure) -> bytes:
    scope = PlotlyScope()
    logger.info("Transforming the graph %s.", id(figure))
    img_bytes = scope.transform(
        figure=figure,
        format="png",
        width=1000,
        height=1000,
        scale=4,
    )
    logger.info("Done transforming the graph %s.", id(figure))
    return img_bytes


def transform_random_figure() -> bytes:
    return figure_to_bytes(get_random_figure())


# Make the join() method return the value returned by the target function.
class ThreadWithReturnValue(Thread):

    def __init__(self, group=None, target=None, name=None, args=(), kwargs={}, Verbose=None):
        Thread.__init__(self, group, target, name, args, kwargs)
        self._return = None

    def run(self):
        if self._target is not None:
            self._return = self._target(*self._args, **self._kwargs)

    def join(self, *args):
        Thread.join(self, *args)
        return self._return


n_threads = 5
threads = [ThreadWithReturnValue(target=transform_random_figure) for _ in range(n_threads)]
for thread in threads:
    thread.start()

results = [thread.join() for thread in threads]
for i, res in enumerate(results):
    with open(f"result{i}.png", "wb") as f:
        f.write(res)