by Rosella (Qian-Ze) Zhu
figures by MacKenzie Mauger

Have you ever meticulously planned a holiday around some pleasant weather, according to the weather forecasts, only to find yourself caught in relentless rain throughout your vacation? The challenge of predicting weather, with its inherent unpredictability, extends beyond mere inconvenience; it’s crucial for ensuring safety in scenarios like driving in heavy rain, avoiding areas prone to wildfires, or preparing for strong winds and floods. Imagine a world where weather forecasts are as precise and personalized as the navigation app on your smartphone, and deciding whether to carry a raincoat or planning safe travel routes isn’t a morning dilemma clouded by ambiguous forecasts. This vision isn’t a distant dream– it’s rapidly becoming our reality thanks to the revolutionary impact of artificial intelligence (AI) and machine learning (ML) on meteorology, helping scientists better tackle and conquer the complexities of weather prediction.

AI, with its remarkable ability to sift through immense datasets to uncover complicated patterns, heralds a new era in meteorology. Major technology companies like Google Research, Google Deepmind, and Huawei have recently demonstrated the ability of ML-based models to outperform the traditional gold-standard methods in weather predictions, while requiring only a fraction of computational resources. From providing farmers with precise agricultural forecasts to predicting the path of deadly cyclones, AI and ML are transforming how we interact with and understand the weather (Figure 1). In this article, we’ll explore the transformative role of AI and ML in weather forecasting, delving into the underlying science, the potentially revolutionary improvement and potential applications they bring, as well as the challenges that lie ahead in our quest to predict the unpredictable.

Figure 1. The evolution of weather forecasting techniques over time. Starting with basic instruments such as balloons for atmospheric sampling, mercury thermometers for temperature measurement, and barometers for pressure readings, the techniques evolved to the use of sophisticated computer models to simulate and predict weather patterns. Further advancements incorporate AI and ML methods, indicating the ongoing trend towards more precise and data-driven weather forecasting methods.

The Traditional Way of Weather Prediction

For decades, meteorologists have relied on complex physical equations of the atmosphere to generate forecasts. In principle, if meteorologists know the current weather conditions, such as wind, pressure, and precipitation, they can use these pieces to generate a complete weather forecast; however, these equations are often inadequate, struggling to capture the intricate and ever-changing nature of our planet’s weather systems. Small mistakes in measuring the current weather or in the calculations can lead to big errors in the forecast.

Despite these great difficulties, scientists have significantly improved these traditional physics-based models by refining equations and reducing error. However, these improvements are costly, requiring expensive computational power and expert knowledge. Furthermore, these models won’t work in areas where accurate equations and robust numerical methods are not available, as in heat wave prediction or short-term rainfall forecasting.

Another main challenge with these models is managing the conflict between different timescales. Typically, people are interested in weather forecasts ranging from several hours to a few days ahead for everyday use. On the other hand, some people – in particular, environmentalists and scientists – often care about climate patterns that develop over decades. To resolve these multiple timescales effectively requires coming up with different underlying equations, or specially tailored algorithms and substantial computational resources.

Learning from the Sky: AI Joins the Forecast

AI models, specifically ML, are a type of technology that allows computers to learn from data and improve over time, much like how humans learn from experience. At its core, ML involves feeding a computer system plenty of data, which could be anything from pictures to numbers to text. This system then looks for patterns and rules within the data, usually with a function informing it of how good its current prediction aligns with the ground truth (Figure 2). Therefore, by using ML weather models, which are trained on massive amounts of weather data, we can capitalize on the rich history of weather patterns and learn to predict the future weather with greater accuracy and efficiency. Note that these models don’t just work with the data– they learn from it, identifying patterns and trends that human forecasters or traditional models might overlook.

Figure 2. Illustration of the workflow of ML-based weather forecasting models as described in the main text.

The integration of AI and ML into meteorology marks a paradigm shift. A few months ago, studies from both Google Deepmind and Huawei demonstrated the power of AI-based models that are particularly adept at grasping complex phenomena like long-term climate predictions and extreme weather events, which traditional methods often struggle to predict accurately. More recently, Google Research and Google Deepmind announced their most up-to-date hybrid model, which combines the time-tested physics-based methods and the learning-focused techniques of ML. 

Specifically, this hybrid model has two main parts. The first part, the trunk, is called the dynamical core, which is similar to the traditional method and is built on a set of physics-based equations. This first part divides the Earth’s surface into a giant 3D grid and calculates wind, heat, and humidity for each grid point. This helps in understanding how different points interact with each other weather-wise.

The second part, called the learned physics module, fills in as the branches on the trunk. It deals with more nuanced weather aspects that the trunk can’t fully capture, like how clouds form, where and when it might rain, and how sunlight interacts with the Earth. In particular, it uses ML models to take in data about the current weather and predict what changes might happen in the next moment. 

In a nutshell, the hybrid model works by combining these two parts together in a step-by-step process. It starts with the current weather, uses the dynamical core to make some initial predictions, and then the learned physics module refines these predictions by adding more details. This process repeats iteratively, advancing the model to make predictions for the desired future timeframe.

Figure 3. An illustrated comparison between traditional vs ML-based models.

While the traditional numerical methods spend a lot of time and computational power on the physical details, the hybrid model streamlines this, making the whole process quicker and more efficient with its AI-driven learned physics module (Figure 3). Therefore, the hybrid model excels in improving short-term weather forecasts while also providing reliable and consistent predictions for the long term, ranging from 1-15 days to climate forecasts spanning a decade (Figure 4).

Figure 4. Prediction of the traditional (red) and AI-driven (green) forecasting models compared with the ground truth (black), in terms of cyclone tracking.

Outlook

While AI has made impressive strides in weather forecasting, challenges remain. Ensuring the accuracy of AI predictions, especially in predicting rare or extreme events, and integrating these predictions into effective public warning systems, are ongoing tasks. However, the potential for further advancements is vast. With continuous advancements in AI technology and an ever-growing pool of data, it is promising that we will one day be able to forecast weather with even greater precision, finer resolution, and over longer time horizons.

As we stand at the cusp of a new era of the warming world, AI-driven weather forecasting is not just a technological marvel, but a necessity in our rapidly changing climate. The integration of AI and traditional methods is transforming the chaotic tapestry of weather into a clearer picture, promising a future where we can navigate our lives with a clearer understanding of the skies above.


Rosella (Qian-Ze) Zhu is a PhD student in the Applied Physics Program at Harvard University.

MacKenzie Mauger is a Ph.D. student in the Biological and Biomedical Sciences program at Harvard Medical School, where she is studying cell type-specific gene repression.

Cover image by WikiImages from pixabay.

For more information:

  • For a detailed description of the hybrid model discussed here, check out the original paper.
  • Check out the most recent (at the time of this writing) probabilistic weather forecasting model in this paper by Google Deepmind.
  • For a collection of large weather models, check out this github repository.

One thought on “A Sky Full of Data: Weather forecasting in the age of AI

  1. With current developments simultaneously taking place, I firmly believe that potential advances will extend far beyond accurate weather forecasting. In fact it extends so far that it may change the way we store and transmit our precious digital data.

    Radiative transfer can be well-described the transport of energy by electromagnetic waves through a gas. Via remodeling electromagnetic data in the form of pulsar like radiative spherical emitters, it has been – at least partially – shown that it is quite possible to transmit data through ionosphere at unprecedentedly rapid and dense rate.

    The ionosphere contains a high proportion of free electrons which influence radio propagation. High Frequency (HF) radio waves hitting the free electrons in the ionosphere cause them to vibrate and re-radiate the energy back down at the same frequency, effectively bouncing the radio wave back towards the Earth. Nowadays it also seems to be the case that those stimulated electron clusters; in the right form of EMP exposure may store data with impressively preserved digital & structural integrity.

    Back in 1970s , initial exclusive EMP experiments were mainly tested in order to permanently stagnate and render enemy vessels, armored units and electronical equipment ineffective and useless but it was also surprisingly discovered that a specific form of EMP pulses in the right altitude literally and “protectively” freezes transmitting data; stabilizing the digital mold of a given transmittance.

    Above mentioned discovery is not without downsides and self-inflicted hindrances. The amount and magnitude of EMP generation required along with the sophistication caliber and cost of the necessary equipment prevents this novelly discovered method to be rigorously tested and further investigated; at least for now. But technical breakthroughs always have a similar technical fate: once it shines bright enough, the innovative ambition follows.

Leave a Reply

Your email address will not be published. Required fields are marked *