WIS2 Downloader

What is the WIS2 Downloader?

The WIS2 Downloader is a Python tool used for downloading real-time data from the WIS2 network.

In particular, it starts a Flask server that allows users to view current subscriptions, add new subscriptions, and remove existing ones.

This tool is highly configurable, with settings for:

Global Broker URL and port
Login details
Transport protocol
Download directory for all incoming data
Topics to subscribe and their corresponding sub-directories for downloading
Number of download threads
Flask server IP and port
Log file directory

How Can I Get Started?

To get started with the WIS2 downloader tool, there are three simple steps:

Installation
Configuration
Running

with an optional fourth step of modifying on-going subscriptions through the Flask app.

Installation

Open the terminal on your computer and execute the following command:

pip install https://github.com/wmo-im/wis2-downloader/archive/main.zip

If you encounter an error with pip not being recognized, please ensure Python is installed and retry the command.

After installation, verify the installation by running the help command:

wis2downloader -h

You should see the help message for the app configuration.

Configuration

Now that we have the tool installed, we need to to create a file that stores our initial subscription configuration. Create a JSON file for the initial subscription configuration with the following structure:

{
    "broker_url": "replace with url of the global broker e.g. globalbroker.meteo.fr",
    "broker_port": "replace with the port to use on the global broker e.g. 443",
    "username": "username to use on the global broker default everyone",
    "password": "password to use on the global broker default everyone",
    "protocol": "transport protocol to use either tcp or websockets",
    "topics": {
        "initial topic 1": "associated sub-directory", ...
    },
    "download_dir": "default base download directory",
    "flask_host": "127.0.0.1",
    "flask_port": 8080,
    "download_workers": 1,
    "save_logs": false,
    "log_dir": "default base directory for logs to be saved"
}

Save this file and note its directory.

Note: The associated sub-directory for each topic is the folder structure inside the base download directory to which the data of this topic is saved to. For example, writing “$TOPIC” will set this sub-directory to that of the topic hierarchy being downloaded.

Running

Start the downloader using the command:

wis2downloader --config <path to config file>

You should see a series of informational messages indicating the operational status.

To view your active subscriptions, open your browser of choice and paste the following URL into the search bar:

http://flask_host:flask_port/list

where the flask host and flask port were specified in the configuration file.

Optional: Adding or Removing Topics

You can add and remove topics without stopping the current subscription. This is done by opening your browser of choice and pasting a URL into the search bar.

Adding a Topic

To add a topic, we use http://flask_host:flask_port/add?. After the /add? part, we write the topic (/add?topic=…) and optionally the associated sub-directory (…&target=…). For example:

http://localhost:8080/add?topic=cache/a/wis2/%2B/data/core/weather/%23&target=example_folder

Note: If the associated sub-directory is not specified, it will default to that of the topic hierarchy.

Removing a Topic

To remove a topic, we similarly use http://flask_host:flask_port/delete?. After the /remove? part, we write the topic to remove (/remove?topic=…). For example:

http://localhost:8080/remove?topic=cache/a/wis2/%2B/data/core/weather/%23

Note: If the topic to add/remove contains any special characters (+, #), they must be URL encoded in order to work (+ is written as %2B, # is written as %23).

More information on the WIS2 Downloader can be found here.