Update Plotly charts with cron jobs and Python

All Python code for this article can be copied from this IPython notebook or this Python script.

Creating Python-powered dashboards or reports for your company? Consider Plotly Enterprise – fully on-premise, customizable instances of Plotly’s cloud.

Cron jobs are tasks scheduled to run periodically on a computer or server. They’re easy to set up in Ubuntu and can be tied to a Python script such that the cron job runs it automatically.

Cron jobs are perfect for collecting data from a MySQL database or API and updating a graph or dashboard of graphs with the newest data. This article shows you how in four easy steps.

If you have questions or get stuck, tweet to @plotlygraphs or email feedback[at]plot[dot]ly.

STEP 1: python script to graph weather data

Let’s collect temperature data for Montréal and San Francisco from the Weather Underground API and create a d3.js graph of the result with Plotly’s Python client.

The full code for this script can be found and copied from this IPython notebook or this Python script.

STEP 2: Run PYTHON SCRIPT every 5 min. with a cron job

Note for Mac and Windows users: Mac operating systems support cron jobs. To edit the crontab scheduler on a Mac, type env EDITOR=nano crontab -e in the Terminal. Windows does not support cron jobs. The equivalent in Windows is the Windows Task Scheduler. Various options for getting started with Ubuntu if you are a Mac or Windows user are described at the beginning of this article.

Open your Ubuntu terminal and type crontab -e . You may be asked to choose an editor to edit your crontab. We recommend the editor “nano” because it is the easiest to navigate for beginners.

In your crontab file, press the down arrow key until you are at the bottom of the file. Then, add this line:

Make sure that you change the path and file name so that it matches the name and location of your Python script. My Python script is called temperature.py and is saved in /home/ubuntu/ on my Ubuntu server.

The >/dev/null 2>&1 is optional. It tells cron to not send any output of your script to your server’s mail inbox. If your script has print statements, you’ll need this line. Alternatively, you can install a mail inbox for you server with:

The five numbers and stars (” 0,30 * * * * “) correspond to minute, hour, day, month, and weekday, respectively. The above command tells cron to run temperature.py every 30 minutes, every day of the year. If you instead wanted to run your Python script only once per day, at 4:22am, you would use this syntax:

Google searching is a great friend for more complex cron job syntax (ie, “run my Python script every other day and every Sunday at 10:00pm”). Here is the official Ubuntu documentation for cron jobs.

Press control-x to exit the crontab file. If you’re asked whether you want to save, make sure that you type “y”. Your graph should now update every 30 minutes.

STEP 3: troubleshooting

There are a few gotchas that can mess up a cron job. Here are a few things to try.

a. Check the permissions of your Python Script

Make sure that the permissions of your Python script are set to “755”. This ensures that cron has the permissions to execute your script. You can set permissions with the command chmod:

b. Change the cron interval to every minute

When you are troubleshooting, it can be helpful to set your cron job to a more frequent interval, to be sure that it is working. Here the cron syntax, for example, to run the Python script every minute:

C. Check the /var/log/syslog log

Cron errors in Ubuntu are recorded in /var/log/syslog – You can watch this log file in realtime with the tail command:

To read the full log, try the cat command:

D. Use Python exceptions and email yourself the error

Sometimes, your cron job will work perfectly for a while, then stop working or miss a cron job cycle. This may be due to an error in the Python script that was not occurring before – for example, a database or API from which the script was reading has changed. To catch these errors when they occur, we recommend wrapping your Python script in a try-except statement and sending yourself an email with the exception as the email body. The smptlib library is an easy way to send yourself emails from your server in Python.

E. #!/usr/bin/env python

Make sure that #!/usr/bin/env python is the first line at the top of your Python script. Cron won’t be able to run your Python script if this is not the first line.

STEP 4: embed chart in a website or dashboard

When signed in, your graph will appear with your files in the Plotly workspace. Here are instructions for embedding the graph as an iframe in an external website or HTML report.

The chart below is updated every 30 minutes by a cron job running this Python script, all on an AWS Ubuntu server.

Note that the time corresponds to the server’s time and may not match your local time.

Also note that there a few “skips” in the data where the Python script failed for some reason – likely a failed API call. See the troubleshooting tips above for investigating these errors when they occur.

Questions? Suggestions? Tweet to @plotlygraphs, use StackOverflow, or email feedback[at]plot[dot]ly.