Jan-Philipp Eisenbach has worked with machine learning at Volue from the very start. When he joined the Volue Insight team three years ago, he found hundreds of models, some of them very simple statistical models.

But the market transformation has pushed the team to build more automated models. Today, trading happens on interconnected marketplaces that run 24-7.

“The Volue Insight team takes raw weather data and transforms it into energy. The intermittency of renewable power and the focus on short-term markets has made short-term forecasting more challenging – and crucial,” says Jan-Philipp Eisenbach.

In addition, the markets are getting faster.

“Three years ago, we didn’t have the intraday forecast. We had the spot forecast. On the spot market, you trade once per day. But like the stock market, intraday can be traded every minute of the day.”

Volue Insight’s market analysts see a staggering 1 800 000 000 daily datapoints to the Volue API. Weather-driven fundamentals and market forecasts are in equally high demand.

In addition, because of Europe’s many interconnectors, the European market is becoming one. In 2022, seven more countries will join the integrated European market coupling. This calls for models that are pan-European.

This increases complexity big time. It is data-intensive and requires more power to calculate and forecast than individual zones.

Switching from statistical to machine learning models

For Jan-Philipp Eisenbach, Till Ilic and Felix Hofmann (the team behind Volue's short-term models), switching from classical statistical models to modern machine learning techniques was a gradual process.

The team found that only by using a high degree of automation and streamlining the data processing pipelines, they were able to keep up with the increasing amount of data and the market becoming faster.

“For the intraday market, we have a forecast every five minutes. The more data we got, the more difficult it became for a human to grasp it on their own. To replicate this, we would need someone putting numbers in the system non-stop.”

The machines can process the data more easily.

“We simply cannot forecast intraday without machine learning. That’s the only way we can be quicker than the market. If we are very slow, our forecast is redundant,” says Jan-Philipp Eisenbach.

Data quality is key. It’s important to have the correct data with no errors.

“We extract the data from different sources and present it in an understandable way. For this complex task, we need code. We have to make this code stable, performant, reliable and flexible so that we can easily add new features and trust the results.”

In addition to speed and data, the team takes into account complexity.

“The spot market, in particular, is very complex with many special rules and regulations. We have to take the limitations of the grid into account as well. For this, we have an optimisation engine running behind. Our spot model is very cool.”

For Jan-Philipp, the biggest challenge is that things are in flux.

“You can build a model that works very well but as soon as the regulations or something else changes because of the transformation going on in the market, it becomes more difficult to learn from the past and use that knowledge to forecast the future.”

In 2021, the energy markets had to deal with “the highest power prices we’ve ever seen” on many occasions.

“New data combinations can result in unexpected model outputs. It is difficult to forecast if a scenario is very different from what has happened in the past.”

The team has to make sure that the models learn what they are expected to learn. Also, in machine learning, it’s imperative that analysts understand the models they build.

“If a model fails, we need to know why. We need to be able to fix it. Often, the problem can be traced back to poor data quality or limitations in the algorithm. If we know what is wrong, we can come up with a solution.”

Domain knowledge is also very important in this context. If incorporated in the model it can reduce overfitting.

“Fortunately, we now have much better computers and we can create much more complex models. And the community is building better and better tools to build better models.”

For Jan-Philipp Eisenbach, the future of power market analysis is machine learning.

Jan-Philipp Eisenbach has a background in mathematics. He has specialised in data science. He has experience in gathering data and analysing it, making forecasts with machine learning.