Wave Forecast Verification

Jump to: Overview and justification| Review of validation data | Data dissemination policies | Requirements for data formats I Participating Centres I WMO Lead Centre for Wave Forecast Verification

Updated on 11 December 2023

Please report broken links to mmo@wmo.int

One of the most important activities of the ETWS-ETWCH, in support of Met-Ocean Information and Maritime Safety Services, in particular, was the Operational Wave Forecast Verification Project. A routine inter-comparison of wave model forecast verification data was first established in 1995 to provide a mechanism for benchmarking and assuring the quality of wave forecast model products that contribute to applications, such as safety of life at sea, ship routing, and, in general, the Global Maritime Distress and Safety System GMDSS. A Technical Report was produced on this activity in early 2006.[Bidlot, J.R. and Holt, M.W., 2006: Verification of Operational Global and Regional Wave Forecasting Systems against Measurements from Moored buoys, JCOMM Technical Report No. 30]

As of 2012, the project included 17 centres, many running global wave forecast systems, with different wave models, different wind forcing, and different model configurations and the goal was to continue to add new participants, including regional participants (where appropriate), and to expand the scope of the intercomparison as feasible.

It is recognized that centres engaged in wave forecasting benefit from this activity in the same way as weather centres benefit from the exchange of forecast verification scores.

Overview and justification

A routine inter-comparison of wave model forecast verification data was first established in 1995 to provide a mechanism for benchmarking and assuring the quality of wave forecast model products that contribute to applications, such as safety of life at sea, ship routing, and, in general, the Global Maritime Distress and Safety System GMDSS.

This original inter-comparison was developed around the exchange of model forecast data at an agreed list of moored buoy sites at which instrumented observations of significant wave height, wave period and wind speed are available over the WMO GTS. Five centres (the ECMWF, Met Office (United Kingdom), FNMOC (USA), NCEP (USA) and the Canadian Meteorological Centre) routinely running global wave forecast models contributed to the original exchange. The initial results were presented during the WAVES97 meeting (Bidlot et al., 1998)

The exchange was subsequently expanded with the addition of data from the Météo-France system in 2001. A paper discussing results from the exchange was published in 2002 [Bidlot, J.R, Holmes, D.J., Wittmann, P.A., Lalbeharry, R., Chen, H.S., 2002: Intercomparison of the Performance of Operational Ocean Wave Forecasting Systems with Buoy Data. Weather and Forecasting 17, pp. 287-310]. The Expert Team on Wind Waves and Storm Surges, during its first meeting (ETWS-I, Halifax, Canada, June 2003) noted the value of the exchange and endorsed the further expansion of the scheme to include other wave forecast systems.

The exchange expanded from the six centres participating in 2001 (ECMWF, Met Office (United Kingdom), FNMOC (USA), NCEP (USA), Canadian Meteorological Centre and Météo-France) to include Deutsche Wetter Dienst (DWD), the Bureau of Meteorology (BOM), Service Hydrographique et Océanographique de la Marine (SHOM) and Japan Meteorological Agency (JMA). All ten centres actively contributed data on a routine basis.

The mechanism for the data exchange was similar to the one set up for the original exchange. On a monthly basis, each centre provided a file of model data collocated with the buoy locations in an agreed format to the ECMWF, where the data were collated for subsequent access. The collated datasets were then processed to provide statistics for each centre at each buoy. Observation data were also collated at the ECMWF, and were quality controlled, with wind speeds adjusted to 10m height. A report on the intercomparison was submitted following the SCG-I meeting and was published in 2006 [Bidlot, J.R. and Holt, M.W., 2006: Verification of Operational Global and Regional Wave Forecasting Systems against Measurements from Moored buoys, JCOMM Technical Report No. 30]. This report provides a description of the project activity and includes a full technical specification of the data exchange process.

The project was again discussed during the 10^th International Workshop on Wave Hindcasting and Forecasting, in Hawaii in November 2007. A small summary of the project was produced as part of the workshop proceeding.

Review of validation data

The Wave Forecast Validation Project has originally focused on validation with buoy data only. However, other routine observations of waves are made and are available routinely. In evaluating the present project, the ETWS deemed it desirable to expand the project to include other observation sources. The initial focus is on providing database entries for the data so that the data can be archived. These data can then be used for both validations, and for the development of appropriate validation techniques.

The following alternate data sources and corresponding requirements for wave model result archiving have been identified:

Spectral wave observations from in-situ platforms, requiring wave model spectra to be archived at selected locations and times

Spectral wave observations from remotely sensed sources, for instance SAR data, requiring wave model spectra to be archived at selected locations and times.

Observations of wave heights and wind speeds from space-borne altimeters, requiring collocated wave model wave heights and wind speeds along altimeter tracks in both space and time.

It has also been observed that wave data over the ocean is notoriously sparse. This has two important implications for wave model validation.

There is systematically insufficient data to provide spatial wavefield analyses without a major impact on the first-guess wave fields used in such analyses. It is therefore misleading to validate wave models with analyzed fields, since the features of such fields are dominated by the underlying wave model, and not by the observations.

Even without the availability of objective analyses, much can be learned from a comparison of wave fields from different wave models. It is therefore useful to add the archiving of selected wavefields to project, to facilitate spatial wavefield intercomparisons between wave models.

Finally, it should be noted that the equations used in wave models represent a strongly forced and damped system. Unlike models for predicting atmospheric and oceanic circulation, waves do not pose an initial value problem, and initial conditions and data assimilation are not critical for wave models. In fact, the basic behaviour and quality of a wave model can be assessed solely by hindcasts studies and is most clearly assessed by validating wave model results obtained without any form of wave data assimilation. Hence, it would be beneficial for centres that utilize data assimilation in their forecasts system to also provide hindcast model results obtained without data assimilation. To facilitate analysis of such model data and/or alternate model runs at centres, the original archive data format needs to be expanded to identify alternate model runs at participating centres.

Data dissemination policies

Some of the additional data to be considered in this project are proprietary. Also, some model data are not considered to be freely available to the general public by the centres involved. To assure that data dissemination policies do not hamper the project, raw data from the archives will be available to team members only. Derived products such as summary graphics will be made available through this website subject to the consent of members of the group.

Requirements for data formats

Data formats are central to any intercomparison work involving several centres, hence specification and implementation of appropriate data formats is key to the success of the project. The formats employed need to have the following characteristics:

Backward compatibility: the centres involved in the exchange are operational centres, and the flexibility for updating the operational systems is often less than that for research systems. Timescales for all centres to conform to new requirements can therefore be long. In consequence, backwards compatibility of data formats is essential to avoid disruption to the exchange during the introduction of changes.
Simplicity: format specifications should be designed to be as simple as possible whilst ensuring that the content is adequate for the purposes of the exchange. Simple formats are generally easier to implement and reduce the risk of discrepancies in the interpretation of the format specification between the participating centres.
Flexibility: formats should be designed to be extendable allowing the future expansion of the exchange. This may include the potential to add additional parameters, additional meta-data, and increased data volumes.
Precisely defined and strictly adhered to: to ensure that the work involved in setting up routine, automated processing of the data the formats need to be defined precisely, and participants need to ensure that their data conforms to the definition. Even small deviations from the definition can generate additional work within the subsequent processing of the data.

Specification of the data formats for the existing exchange and the proposed extensions are given in Annex 1 of the JCOMM Technical Report No. 30.

Participating Centres (as of May 2012)

Outputs from all participating partners (fully operational forecasting centres) were compared to buoy and platform data as broadcasted to the meteorological community via the Global Telecommunication System (GTS). The different data sets from participants in the JCOMM Wave Forecast Verification were subsequently merged and made available to all participating partners for further evaluation. Some examples, in graphical and tabular forms, can be found in the Monthly Reports on intercomparison of operational wave forecasting system processed at ECMWF:

BoM: Bureau of Meteorology (Australia)
CNR-AM: Consiglio Nazionale delle Ricerche (Italy)
DMI: Danmarks Meteorologiske Institut (Denmark)
DWD: Deutscher Wetterdienst (Meteorological Service of Germany)
ECMWF: European Centre for Medium-range Weather Forecasts
FNMOC: Fleet Numerical Meteorology and Oceanography Center (USA)
JMA: Japan Meteorological Agency
KMA: Korea Meteorological Administration
METNO: Norwegian Meteorological Institute
MSC: Meteorological Service of Canada
Meteo-France
NCEP: National Centers for Environmental Prediction (USA)
NIWA: National Institute of Water & Atmospheric (New Zealand)
Puerto del Estado (Spain)
SHN-SM: Department of Meteorology of the Naval Hydrographic Service (Argentina)
SHOM: Service Hydrographique et Océanographique de la Marine (France)
UK MetOffice

WMO Lead Centre for Wave Forecast Verification

In 2016, the World Meteorological Organization (WMO) Commission for Basic Systems recommended that ECMWF become the Lead Centre for Wave Forecast Verification (LC‑WFV). With more than 20 years of experience in wave forecast verification and wave model intercomparison (see ECMWF Newsletter No. 150, winter 2016/17), ECMWF was ideally placed to formally take on this role. Three years later, the LC‑WFV has reached a stage where most centres contributing to the original intercomparison are providing data to the new system, and where verification results are published regularly on the LC-WFV web page at https://confluence.ecmwf.int/display/WLW. The role of the Lead Centre enables ECMWF to immediately identify weaknesses in its wave forecasts compared to others, which helps to inform further improvements to the wave model. Model intercomparison is based on the exchange of forecast fields rather than scores, making it more sustainable in the longer term and providing the necessary flexibility for introducing new scores and observation datasets in the future.

Return to WMO Marine homepage