API Reference#

SwarmRequest#

class viresclient.SwarmRequest(url=None, token=None, config=None, logging_level='NO_LOGGING')[source]#

Bases: viresclient._client.ClientRequest

Handles the requests to and downloads from the server.

Examples

Retrieve data:

from viresclient import SwarmRequest
# Set up connection with server
request = SwarmRequest("https://vires.services/ows")
# Set collection to use
request.set_collection("SW_OPER_MAGA_LR_1B")
# Set mix of products to fetch:
#  measurements (variables from the given collection)
#  models (magnetic model predictions at spacecraft sampling points)
#  auxiliaries (variables available with any collection)
request.set_products(
    measurements=["F", "B_NEC"],
    models=["CHAOS-Core"],
    auxiliaries=["QDLat", "QDLon"],
    sampling_step="PT10S"
)
# Fetch data from a given time interval
data = request.get_between(
    start_time="2014-01-01T00:00",
    end_time="2014-01-01T01:00"
)
# Load the data as an xarray.Dataset
ds = data.as_xarray()

Check what data are available:

request.available_collections(details=False)
request.available_measurements("MAG")
request.available_auxiliaries()
request.available_models(details=False)
Parameters

  • url (str) –

  • token (str) –

  • config (str or ClientConfig) –

  • logging_level (str) –

add_filter(filter_)[source]#

Add an arbitrary data filter.

Parameters

filter (str) – string defining the filter, as shown below

Filter grammar:

filter: predicate
predicate:
     variable == literal |
     variable != literal |
     variable < number |
     variable > number |
     variable <= number |
     variable >= number |
     variable & unsigned-integer == unsigned-integer |
     variable & unsigned-integer != unsigned-integer |
     (predicate AND predicate [AND predicate ...]) |
     (predicate OR predicate [OR predicate ...]) |
     NOT predicate
literal: boolean | integer | float | string
number: integer | float
variable: identifier | identifier[index]
index: integer[, integer ...]

Both single- and double quoted strings are allowed.
NaN values are matched by the ==/!= operators, i.e., the predicates
are internally converted to a proper "IS NaN" or "IS NOT NaN"
comparison.

Examples

“Flags & 128 == 0”

Match records with Flag bit 7 set to 0.

“Elevation >= 15”

Match values with values greater than or equal to 15.

“(Label == “D” OR Label == “N” OR Label = “X”)”

Match records with Label set to D, N or X.

“(Type != 1 AND Type != 34) NOT (Type == 1 OR Type == 34)”

Exclude records with Type set to 1 or 34.

“(Vector[2] <= -0.1 OR Vector[2] >= 0.5)”

Match records with Vector[2] values outside of the (-0.1, 0.5) range.

applied_filters()[source]#

Print currently applied filters.

available_auxiliaries()[source]#

Returns a list of the available auxiliary parameters.

available_collections(groupname=None, details=True)[source]#

Show details of available collections.

Parameters

  • groupname (str) – one of: (“MAG”, “EFI”, etc.)

  • details (bool) – If True then print a nice output. If False then return a dict of available collections.

available_measurements(collection=None)[source]#

Returns a list of the available measurements for the chosen collection.

Parameters

collection (str) – one of: (“MAG”, “EFI”, “IBI”, “TEC”, “FAC”, “EEF”)

available_models(param=None, details=True, nice_output=True)[source]#

Show details of avalable models.

If details is True, return a dictionary of model names and details. If nice_output is True, the dictionary is printed nicely. If details is False, return a list of model names. If param is set, filter to only return entries including this

Note

F = Fast-Track Products
C = Comprehensive Inversion
D = Dedicated Chain
MCO = Core / main
MLI = Lithosphere
MMA = Magnetosphere
MIO = Ionosphere
Parameters

  • param (str) – one of “F C D MCO MLI MMA MIO”

  • details (bool) – True for a dict of details, False for a brief list

  • nice_output (bool) – If True, just print the dict nicely

available_observatories(collection, start_time=None, end_time=None, details=False, verbose=True)[source]#

Get list of available observatories from server.

Search availability by collection, one of:

"SW_OPER_AUX_OBSH2_"
"SW_OPER_AUX_OBSM2_"
"SW_OPER_AUX_OBSS2_"

Examples

from viresclient import SwarmRequest
request = SwarmRequest()
# For a list of observatories available:
request.available_observatories("SW_OPER_AUX_OBSM2_")
# For a DataFrame also containing availability start and end times:
request.available_observatories("SW_OPER_AUX_OBSM2_", details=True)
# For available observatories during a given time period:
request.available_observatories(
    "SW_OPER_AUX_OBSM2_", "2013-01-01", "2013-02-01"
)
Parameters

  • collection (str) – OBS collection name, e.g. “SW_OPER_AUX_OBSM2_”

  • start_time (datetime / ISO_8601 string) –

  • end_time (datetime / ISO_8601 string) –

  • details (bool) – returns DataFrame if True

  • verbose (bool) – Notify with special data terms

Returns

IAGA codes (and start/end times)

Return type

list or DataFrame

available_times(collection, start_time=None, end_time=None)#

Returns temporal availability for a given collection

Parameters

  • (str) – collection name

  • start_time (datetime / ISO_8601 string) –

  • end_time (datetime / ISO_8601 string) –

Returns

DataFrame

clear_filters()[source]#

Remove all applied filters.

clear_range_filter()#

Remove all applied filters.

get_between(start_time=None, end_time=None, filetype='cdf', asynchronous=True, show_progress=True, show_progress_chunks=True, leave_intermediate_progress_bars=True, nrecords_limit=None, tmpdir=None)#

Make the server request and download the data.

Parameters

  • start_time (datetime / ISO_8601 string) –

  • end_time (datetime / ISO_8601 string) –

  • filetype (str) – one of (‘csv’, ‘cdf’)

  • asynchronous (bool) – True for asynchronous processing, False for synchronous

  • show_progress (bool) – Set to False to remove progress bars

  • show_progress_chunks (bool) – Set to False to remove progress bar for chunks

  • leave_intermediate_progress_bars (bool) – Set to False to clean up the individual progress bars left when making chunked requests

  • nrecords_limit (int) – Override the default limit per request (e.g. nrecords_limit=3456000)

  • tmpdir (str) – Override the default temporary file directory

Return type

ReturnedData

get_conjunctions(start_time=None, end_time=None, threshold=1.0, spacecraft1='A', spacecraft2='B', mission1='Swarm', mission2='Swarm')[source]#

Get times of the spacecraft conjunctions.

Currently available for the following spacecraft pairs:

  • Swarm-A/Swarm-B

Parameters

  • start_time (datetime / ISO_8601 string) – optional start time

  • end_time (datetime / ISO_8601 string) – optional end time

  • threshold (float) – optional maximum allowed angular separation in degrees; by default set to 1; allowed values are [0, 180]

  • spacecraft1 – identifier of the first spacecraft, default to ‘A’

  • spacecraft2 – identifier of the second spacecraft, default to ‘B’

  • mission1 (str) – mission of the first spacecraft, defaults to ‘Swarm’

  • mission2 (str) – mission of the first spacecraft, defaults to ‘Swarm’

Return type

ReturnedData

get_model_info(models=None, custom_model=None, original_response=False)[source]#

Get model info from server.

Handles the same models input as .set_products(), and returns a dict like:

{‘IGRF12’: { ‘expression’: ‘IGRF12(max_degree=13,min_degree=0)’, ‘validity’: {‘start’: ‘1900-01-01T00:00:00Z’, ‘end’: ‘2020-01-01T00:00:00Z’ }, …}

If original_response=True, return the list of dicts like:

{‘expression’: ‘MCO_SHA_2C(max_degree=16,min_degree=0)’, ‘name’: ‘MCO_SHA_2C’, ‘validity’: {‘start’: ‘2013-11-30T14:38:24Z’, ‘end’: ‘2018-01-01T00:00:00Z’}}, …

Parameters

  • models (list/dict) – as with set_products

  • custom_model (str) – as with set_products

  • original_response (bool) –

Returns

dict or list

get_orbit_number(spacecraft, input_time, mission='Swarm')[source]#

Translate a time to an orbit number.

Parameters

  • spacecraft (str) – Swarm: one of (‘A’,’B’,’C’) or (“Alpha”, “Bravo”, “Charlie”) GRACE: one of (‘1’,’2’) GRACE-FO: one of (‘1’,’2’) CryoSat-2: None

  • input_time (datetime) – a point in time

  • mission (str) – one of (‘Swarm’, ‘GRACE’, ‘GRACE-FO’, ‘CryoSat-2’)

Returns

The current orbit number at the input_time

Return type

int

get_times_for_orbits(start_orbit, end_orbit, mission='Swarm', spacecraft=None)[source]#

Translate a pair of orbit numbers to a time interval.

Parameters

  • start_orbit (int) – a starting orbit number

  • end_orbit (int) – a later orbit number

  • spacecraft (str) – Swarm: one of (‘A’,’B’,’C’) or (“Alpha”, “Bravo”, “Charlie”) GRACE: one of (‘1’,’2’) GRACE-FO: one of (‘1’,’2’) CryoSat-2: None

  • mission (str) – one of (‘Swarm’, ‘GRACE’, ‘GRACE-FO’, ‘CryoSat-2’)

Returns

(start_time, end_time) The start time of the start_orbit and the ending time of the end_orbit. (Based on ascending nodes of the orbits)

Return type

tuple (datetime)

list_jobs()#

Return job information from the server.

Returns

dict

set_bitmask_filter(parameter, selection=0, mask=- 1, negate=False)[source]#

Set a bitmask filter to apply.

Filters data for parameter & mask == selection & mask, or parameter & mask != selection & mask if negated.

Note

See SwarmRequest.add_filter() for arbitrary filters.

Parameters

  • parameter (str) –

  • mask (integer) –

  • selection (integer) –

Examples

request.set_bitmask_filter("Flags_F", 0, 1)

to set “Flags_F & 1 == 0” (i.e. bit 1 is set to 0)

set_choice_filter(parameter, *values, negate=False)[source]#

Set a choice filter to apply.

Filters data for parameter in values, or parameter not in values if negated.

Note

See SwarmRequest.add_filter() for arbitrary filters.

Parameters

  • parameter (str) –

  • values (float or integer or string) –

Examples

request.set_choice_filter("Flags_F", 0, 1)

to set “(Flags_F == 0 OR Flags_F == 1)”

request.set_choice_filter("Flags_F", 0, 1, negate=True)

to set “(Flags_F != 0 AND Flags_F != 1)”

set_collection(*args, verbose=True)[source]#

Set the collection(s) to use.

Parameters

  • (str) – one or several from .available_collections()

  • verbose (bool) – Notify if special data terms

set_products(measurements=None, models=None, custom_model=None, auxiliaries=None, residuals=False, sampling_step=None, ignore_cached_models=False)[source]#

Set the combination of products to retrieve.

If residuals=True then just get the measurement-model residuals, otherwise get both measurement and model values.

Parameters

  • measurements (list(str)) – from .available_measurements(collection_key)

  • models (list(str)/dict) – from .available_models() or defineable with custom expressions

  • custom_model (str) – path to a custom model in .shc format

  • auxiliaries (list(str)) – from .available_auxiliaries()

  • residuals (bool) – True if only returning measurement-model residual

  • sampling_step (str) – ISO_8601 duration, e.g. 10 seconds: PT10S, 1 minute: PT1M

  • ignore_cached_models (bool) – True if cached models should be ignored and calculated on-the-fly

set_range_filter(parameter, minimum=None, maximum=None, negate=False)[source]#

Set a range filter to apply.

Filters data for minimum ≤ parameter ≤ maximum, or parameter < minimum OR parameter > maximum if negated.

Note

Parameters

  • parameter (str) –

  • minimum (float or integer) –

  • maximum (float or integer) –

Examples

request.set_range_filter("Latitude", 0, 90)

to set “Latitude >= 0 AND Latitude <= 90”

request.set_range_filter("Latitude", 0, 90, negate=True)

to set “(Latitude < 0 OR Latitude > 90)”

AeolusRequest#

class viresclient.AeolusRequest(url=None, token=None, config=None, logging_level='NO_LOGGING')[source]#

Bases: viresclient._client.ClientRequest

Handles the requests to and downloads from the server.

Parameters

  • url (str) –

  • username (str) –

  • password (str) –

  • token (str) –

  • config (str or ClientConfig) –

  • logging_level (str) –

available_collections(collection=None, field_type=None, like=None, details=True)[source]#
available_times(collection, start_time=None, end_time=None)#

Returns temporal availability for a given collection

Parameters

  • (str) – collection name

  • start_time (datetime / ISO_8601 string) –

  • end_time (datetime / ISO_8601 string) –

Returns

DataFrame

clear_range_filter()[source]#

Remove all applied filters.

get_between(start_time=None, end_time=None, filetype='cdf', asynchronous=True, show_progress=True, show_progress_chunks=True, leave_intermediate_progress_bars=True, nrecords_limit=None, tmpdir=None)#

Make the server request and download the data.

Parameters

  • start_time (datetime / ISO_8601 string) –

  • end_time (datetime / ISO_8601 string) –

  • filetype (str) – one of (‘csv’, ‘cdf’)

  • asynchronous (bool) – True for asynchronous processing, False for synchronous

  • show_progress (bool) – Set to False to remove progress bars

  • show_progress_chunks (bool) – Set to False to remove progress bar for chunks

  • leave_intermediate_progress_bars (bool) – Set to False to clean up the individual progress bars left when making chunked requests

  • nrecords_limit (int) – Override the default limit per request (e.g. nrecords_limit=3456000)

  • tmpdir (str) – Override the default temporary file directory

Return type

ReturnedData

get_from_file(path=None, filetype='nc')[source]#

Get VirES ReturnedData object from file path

Allows loading of locally saved netCDF file (e.g. using to_file method) providing access to data manipulation methods such as as_xarray

Parameters

  • path (str) –

  • filetype (str) –

list_jobs()#

Return job information from the server.

Returns

dict

print_available_collections(collection=None, field_type=None, regex=None, details=True, path=False)[source]#
set_bbox(bbox=None)[source]#

Set a bounding box to apply as filter.

Note

Dictionary argument has to contain n, e, s, w keys for north, east, south and west values as EPSG 4326 coordinates

Parameters

bbox (dict) –

set_collection(collection)[source]#
set_fields(observation_fields=None, measurement_fields=None, ica_fields=None, sca_fields=None, mca_fields=None, mie_profile_fields=None, rayleigh_profile_fields=None, rayleigh_wind_fields=None, mie_wind_fields=None, rayleigh_grouping_fields=None, mie_grouping_fields=None, group_fields=None, fields=None)[source]#
set_range_filter(parameter=None, minimum=None, maximum=None)[source]#

Set a filter to apply.

Filters data for minimum ≤ parameter ≤ maximum

Note

Apply multiple filters with successive calls to set_range_filter()

Parameters

  • parameter (str) –

  • minimum (float) –

  • maximum (float) –

set_variables(aux_type=None, fields=None, dsd_info=False)[source]#

ReturnedData#

class viresclient.ReturnedData(filetype=None, N=1, tmpdir=None)[source]#

Bases: object

Flexible object for working with data returned from the server

Holds a list of ReturnedDataFile objects under self.contents

Example usage:

...
data = request.get_between(..., ...)
data.sources
data.data_filters
data.magnetic_models
data.as_xarray()
data.as_xarray_dict()
data.as_dataframe(expand=True)
data.to_file()
as_dataframe(expand=False)[source]#

Convert the data to a pandas DataFrame.

If expand is True, expand some columns, e.g.:

B_NEC -> B_NEC_N, B_NEC_E, B_NEC_C

B_VFM -> B_VFM_i, B_VFM_j, B_VFM_k

Parameters

expand (bool) –

Returns

pandas.DataFrame

as_xarray(reshape=False)[source]#

Convert the data to an xarray Dataset.

Parameters

reshape (bool) – Reshape to a convenient higher dimensional form

Returns

xarray.Dataset

as_xarray_dict()[source]#

Convert the data to a dict containing an xarray per group.

Returns

dict of xarray.Dataset

property contents#

List of ReturnedDataFile objects

property data_filters#

Get list of filters applied.

property filetype#

Filetype string

property magnetic_models#

Get list of magnetic models used.

property sources#

Get list of source product identifiers.

to_file(path, overwrite=False)[source]#

Saves the data to the specified file, when data is only in one file.

Only write to file if it does not yet exist, or if overwrite=True. Currently handles CSV and CDF formats.

Note

This is currently only implemented for smaller data when the request has not been split into multiple requests - the limit is the equivalent of 50 days of 1Hz measurements. In these situations, you can still load the data as pandas/xarray objects (the contents of each file is automatically concatenated) and save them as a different file type. Or use .to_files() to save the split data directly.

Parameters

  • path (str) – path to the file to save as

  • overwrite (bool) – Will overwrite existing file if True

to_files(paths, overwrite=False)[source]#

Saves the data to the specified files.

Only write to file if it does not yet exist, or if overwrite=True. Currently handles CSV and CDF formats.

Parameters

  • paths (list of str) – paths to the files to save as

  • overwrite (bool) – Will overwrite existing file if True

class viresclient.ReturnedDataFile(filetype=None, tmpdir=None)[source]#

Bases: object

For handling individual files returned from the server.

Holds the data returned from the server and the data type. Data is held in a NamedTemporaryFile, which is automatically closed and destroyed when it goes out of scope. Provides output to different file types and data objects.

as_dataframe(expand=False)[source]#

Convert the data to a pandas DataFrame.

Returns

pandas.DataFrame

as_xarray(group=None, reshape=False)[source]#

Convert the data to an xarray Dataset.

Note

Does not support csv

Only supports scalar and 3D vectors (currently)

Returns

xarray.Dataset

as_xarray_dict()[source]#

Convert the data to an xarray Dataset.

Note

Only supports netCDF format

Returns

dict of xarray.Dataset

property data_filters#
property filetype#

Filetype is one of (“csv”, “cdf”, “nc”)

property magnetic_models#
open_cdf()[source]#

Returns the opened file as cdflib.CDF

property sources#
to_file(path, overwrite=False)[source]#

Saves the data to the specified file.

Only write to file if it does not yet exist, or if overwrite=True. Currently handles CSV and CDF formats.

Parameters

  • path (str) – path to the file to save as

  • overwrite (bool) – Will overwrite existing file if True

to_netcdf(path, overwrite=False)[source]#

Saves the data as a netCDF4 file (this is compatible with HDF5)

Extension should be .nc

ClientConfig#

class viresclient.ClientConfig(path=None)[source]#

Bases: object

Client configuration.

Example usage:

cc = ClientConfig()      # use default configuration file
cc = ClientConfig("./viresconf.ini")  # use custom configuration file

print(cc.path)           # print path
print(cc)                # print the whole configuration

cc.default_url = "https://foo.bar/ows"  # set default server

# access to credentials configuration ...
cc.set_site_config("https://foo2.bar/ows", token="...")

cc.save()    # save configuration
property default_url#

Get default URL or None if not set.

get_site_config(url)[source]#

Get configuration for the given URL.

init(env_var_name='VIRES_ACCESS_CONFIG')[source]#

Initialize client configuration.

property path#

Get path of the configuration file.

save()[source]#

Save the configuration file.

set_site_config(url, **options)[source]#

Set configuration for the given URL.

set_token#

viresclient.set_token(url='https://vires.services/ows', token=None, set_default=False)[source]#

Set the access token for a given URL, using user input.

Get an access token at https://vires.services/accounts/tokens/

See https://viresclient.readthedocs.io/en/latest/config_details.html

This will create a configuration file if not already present, and input a token configuration for a given URL, replacing the current token. It sets the given URL as the default if one is not already set. It uses getpass to hide the token from view.

Example usage:

set_token()
# user prompted for input of token, for https://vires.services/ows

set_token(url="https://vires.services/ows")
# user prompted for input of token, for given url

set_token(url="https://vires.services/ows", token="...")
# set a given url and token (no prompting)

DataUpload#

class viresclient.DataUpload(url, token, **kwargs)[source]#

Bases: object

VirES for Swarm data upload API proxy.

Example usage:

from viresclient import ClientConfig, DataUpload

du = DataUpload("https://vires.services", token="...")

cc = ClientConfig()
url = cc.default_url
du = DataUpload(url, **cc.get_site_config(url))

# upload file
info = du.post("example.csv")
print(info)

# get information about the uploaded files
info = du.get()
print(info)

# remove any uploaded files
du.clear()

# check if the upload is valid and get list of missing mandatory parameters
info = du.post("example.cdf")
is_valid = info.get('is_valid', True)
missing_fields = info.get('missing_fields', {}).keys()
print(is_valid, missing_fields)

# get constant parameters
id = info['identifier']
parameters = du.get_constant_parameters(id)
print(parameters)

# set new constant parameters
parameters = du.set_constant_parameters(id, {'Radius': 7000000, 'Latitude': 24.0})
print(parameters)

# clear all constant parameters
parameters = du.set_constant_parameters(id, {}, replace=True)
print(parameters)

For more information about the supported file format see the file format specification

exception Error[source]#

Bases: Exception

Data upload error exception.

args#
with_traceback()#

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

PATH_OWS = '/ows'#
PATH_UPLOAD = '/custom_data/'#
clear()[source]#

Remove all uploaded items.

delete(identifier)[source]#

REST/API DELETE request. Delete item of the given identifier.

get(identifier=None)[source]#

REST/API GET If an identifier provided, get info about the uploaded item. If no identifier provided, list all uploaded items.

classmethod get_api_url(url)[source]#

Translate WPS URL path to the upload REST/API URL path.

get_constant_parameters(identifier)[source]#

Get dictionary of the currently set constant parameters.

classmethod get_ows_url(url)[source]#

Translate REST/API URL path to the upload WPS URL path.

property ids#

Get list of identifiers.

patch(identifier, data)[source]#

REST/API PATCH Update metadata of the uploaded dataset.

post(file, filename=None)[source]#

HTTP POST multipart/form-data Upload file to the server and get info about the uploaded file.

set_constant_parameters(identifier, parameters, replace=False)[source]#

Set constant parameters form from give key value dictionary. Set replace to True if you prefer to replace the already set parameters rather then update them.