Stefan Pfenninger simulating PV and wind power plants

Together with Iain Staffell, I published two papers, available today, using reanalysis and satellite data to simulate PV and wind power plants across Europe (references at the bottom of this post). Alongside these two publications, we built a large database of measured power output data to validate the simulations and perform bias corrections. This now allows us to accurately model hypothetical European wind and PV power output over several decades, investigating issues like the impact of increasing PV deployment on net power demand:

Duck curve of Germany

Duck curve of Germany, showing reported demand and PV production data for two days in 2014 (black lines) and the range of net demand from 30 years of simulated PV power production for 1.5 times (red) and 3 times (blue) the 2014 PV capacity. Source: Pfenninger and Staffell (2016).

There’s an Imperial College London story on both papers and an ETH Zürich news story on the wind paper.

To make the simulations available for other researchers, the platform I created and developed together with Iain Staffell alongside these two papers allows users to run simulations of the hourly power output from wind and solar power plants located anywhere in the world: Screenshot screenshot

Data from are free to use under the Creative Commons BY-NC license. The papers are both open access and fully available online. We hope to reduce parallel efforts in creating and validating such simulations, and are looking forward to seeing other people make use of our data!

Paper references

Stefan Pfenninger and Iain Staffell (2016). Long-term patterns of European PV output using 30 years of validated hourly reanalysis and satellite data. Energy 114, pp. 1251-1265. doi: 10.1016/

Iain Staffell and Stefan Pfenninger (2016). Using Bias-Corrected Reanalysis to Simulate Current and Future Wind Power Output. Energy 114, pp. 1224-1239. doi: 10.1016/

Tags: python

Calliope, an open-source energy systems modeling framework

Energy system models are used by analysts in academia, government and industry. They allow researchers to build internally coherent simulations and scenarios of the extraction, conversion, transportation and use of energy, either today or in the future, at scales ranging from cities to entire countries or continents. These models are particularly important because they can help with understanding and planning the transition from a fossil-fuel dominated energy system to one primarily consisting of clean and renewable energy.

Calliope Logo

For about a year, I’ve been working on Calliope, a framework to develop energy system models using a modern and open source Python-based toolchain (the name derives from the idea of “multi-scale energy systems modeling” which leads to MUSES and thus Calliope).

Basically, I started to work on Calliope because I wanted something that ticks all these boxes:

  • Capable of using data with arbitrary (to some degree) resolution in space and time in an intelligent way
  • Able to define a number of model parameters to iterate over, generate a range of models to do so, and deploy these to a computing cluster, then process the results in a unified manner
  • Allow the specification of energy models in a text-based, modular, hierarchical data format (based on YAML and CSV files)
  • Written in a modern high-level language (Python)
  • Modular and easily extensible
  • Use a permissive Open Source license (Apache 2.0)

The goal of Calliope is not to compete with established large modeling systems such as TIMES. Rather, it is to provide a framework that enables the rapid construction of (small or large) models built to answer specific research questions. Furthermore, by combining its model specification format and the open code base, its aim is also to contribute to more open sharing of data and code in the energy modeling community.

It has its own website at

It’s still work in progress, with significant further functionality to be added in future releases. Currently, I’m the only developer, but contributions from others for whom this might be a useful tool are very welcome. I’m using Calliope for several ongoing research projects and it is undergoing continued development, so more material including academic papers will become available over the coming months. For now, there is a brief tutorial in the documentation that explains some of the things that Calliope does.

Comments, criticism and contributions are very welcome!

Tags: python

IPython notebook extensions to ease day-to-day work

Update 26 October 2015: The ipython-extensions repository has been updated with instructions for Jupyter/IPython 4.0, but for now, the CodeMirror customizations do not seem to work.

Update 4 March 2015: As of IPython 3.0, it seems that code in custom.js (such as the examples below) should be attached to the app_initialized.NotebookApp event (the ipython-extensions repository is updated accordingly):

$([]).on("app_initialized.NotebookApp", function () {
  // your code here

Since I use the IPython notebook for a lot of explorative data analysis, I wanted some extra functionality to make day-to-day notebook work just a little bit more pleasant. I use several of Min RK’s excellent notebook extensions, particularly the floating table of contents and the Gist button.

This post describes some additional extensions I created to do things I need in day-to-day work, as well as how to leverage the customization possibilities of the CodeMirror editor to add vertical rulers to code cells. The notebook extensions described here are available on GitHub.


Sometimes larger analysis tasks take more than a couple of seconds to compute, and I want to alt-tab to do something else while I wait. The IPython-notify extension ensures that you don’t forget what you were originally doing, by displaying a notification once the kernel becomes idle again after a set minimum busy time:

Notification screenshot

When the plugin is activated, a button to request notification permissions is shown in the toolbar. After notification permissions have been granted, this button is replaced by a dropdown menu with five choices: Disabled, 0, 5, 10 and 30.

To activate notifications, select a minimum kernel busy time required to trigger a notification (e.g. if selecting 5, a notification will only be shown if the kernel was busy for more than 5 seconds). The selection is saved in the notebook’s metadata and restored when the notebook is re-opened.

Theme switching

As described in a previous post, depending on ambient lighting and time of day I prefer seeing my code in light text on a dark background, so I created a simple extension to switch between two themes.

The default CSS included only changes code cells (this has the added effect that it’s easier to tell code cells apart from markdown or raw cells):

Dark IPython Notebook

The CSS files can easily be exchanged for a different ones, for example a Base16 theme.

Customizing the CodeMirror editor

IPython notebooks use the CodeMirror editor, which includes a wide range of addons. IPython also allows the user to add additional javascript and CSS via the custom.js and custom.css files in the static/custom directory inside the IPython profile (likely ~/.ipython/profile_default/static/custom on Linux or OS X).

I want to display some rulers to indicate line lengths I shouldn’t exceed to conform with PEP8. Thanks to CodeMirror’s flexible configuration system, this can be achieved by adding the following code to custom.js, making use of the rulers addon:


var clsname = "ipynb_ruler";
var rulers = [{column: 79, className: clsname},
              {column: 99, className: clsname}];

IPython.Cell.options_default.cm_config.rulers = rulers;

By adding the class ipynb_ruler to custom.css, the look of the rulers can be tweaked, for example to use a lighter color:

.ipynb_ruler {
    border-color: #DFDFDF;

Unfortunately, there is a bug in older CodeMirror versions that in some browsers causes extra scrollbars when enabling rulers. For the time being, this can be fixed by placing a fixed rulers.js into ~/.ipython/profile_default/static/components/codemirror/addon/display (adjust as needed if your IPython profile directory is different). I backported a fix into the rulers.js shipped with IPython 2.2, available here.

Can concentrating solar power (CSP) deliver reliable electricity?

We just published a paper in Nature Climate Change where we examine whether concentrating solar power (CSP) can provide the reliability we’re used to from conventional (fossil or nuclear) power plants. In other words, under what conditions can CSP plants provide baseload power, i.e. an essentially stable output of power over time similar to nuclear plants, and dispatchable power, i.e. power output that can be quickly called upon if needed to balance demand?

Gemasolar Plant

Gemasolar power plant (Source: Torresol Energy)

That CSP plants can do this has frequently been assumed, but we didn’t find any detailed examination of how well they could do so, or at what cost, in different parts of the world. We used an optimization model with high-resolution weather data to investigate the question.

Our model results suggest that getting a completely dispatchable or baseload capable fleet of CSP plants can be achieved within a cost range comparable to existing technologies, given some caveats like the need for coordination between individual plants and reductions in component costs (which we would expect to see if the technology is deployed at scale). The cost of doing so varies across the different world regions we investigated (the Southwestern United States, India, South Africa, and the Mediterranean/Middle East/North African region). In India and the United States, the cost of providing high availability CSP plants may be prohibitive, while in South Africa and the Mediterranean basin it is within the range of other technologies’ costs.

The full paper is available on the Nature Climate Change website.

Tags: python

IPython extension to toggle dark code cells

Update 25 August 2014: As described in this newer post, this is now part of the ipython-extensions repository.

Sometimes it’s more pleasant to read light text on a dark background, particularly when dealing with code. However, depending on ambient lighting, it can be necessary to switch back to dark text on a light background on short notice.

To make this possible, I created a simple extension for the IPython notebook (version 2.0 and above). It adds a button to the toolbar that toggles between a light (default) and dark display mode for code cells in the notebook, with an optional keyboard shortcut.

This turns

Light IPython Notebook


Dark IPython Notebook

Details and installation instructions are in the GitHub repository.