qumin.utils package

Submodules

qumin.utils.metadata module

class qumin.utils.metadata.Metadata(path=None, cfg=None, rundir_path=None)

Bases: object

Metadata manager for Qumin scripts. Wrapper around the Frictionless Package class.

Basic usage :

1. Register Metadata manager;

2. Get an absolute path to the metadata folder;

3. Write to that path;

4. After writing a file, register it and set metadata (description, custom dict);

5. Export the JSON descriptor.

The Metadata class can easily be used in scripts that reuse Qumin results. In that case, one must pass a value to runtime_path, if hydra is not set (which is very likely if you write a simple script).

Examples

import omegaconf
cfg = omegaconf.dictconfig.DictConfig({data="myparalex/package.json"})
md = Metadata(cfg=cfg, path="myprevious_run/metadata.json")
name = 'path/myfile.txt'
filename = md.get_path(name)
# Open an IO stream and write to ``filename``.
md.register_file(name, description="My nice file", custom={"property": "value"})
md.save_metadata(path)

Variables:

• start (datetime) – timestamp at the beginning of the run.

• prefix (Path) – normalized prefix for the output files

• cfg (OmegaConf) – all arguments passed to the python script

• paralex (frictionless.Package) – a frictionless Package representing a dataset.

__init__(path=None, cfg=None, rundir_path=None)

Parameters:

• cfg (OmegaConf.dictconfig.DictConfig) – arguments passed to the script.

• path (str) – Path to a Frictionless descriptor of a previous run to be imported.

• rundir_path (str) – Directory that should be used to export the results. Useful only if path is None and hydra is not used.

get_paradigm_conf(cfg)

Load paradigm creation keywords from previous run.

A few security checks are performed to ensure the user didn’t pass contradictory arguments. If this is the case, a warning is thrown and old arguments are kept.

Under some conditions, arguments can be overwritten (e.g. cells list).

Parameters:

cfg (OmegaConf.dictconfig.DictConfig) – Arguments passed to the current run. These arguments might override arguments from the previous run, under specific conditions.

get_paradigms(md, **kwargs)

Creates paradigms with a stable config strategy.

Parameters:

• md (qumin.utils.metadata.Metadata) – Metadata handler of the current run.

• kwargs (dict) – Additional keyword arguments are passed to qumin.representations.paradigms.Paradigms.

get_path(rel_path)

Return an absolute path to a file and create parent directories.

Parameters:

rel_path (str) – relative path to the file or folder.

Returns:

absolute path to the file or folder.

Return type:

pathlib.Path

get_pattern_conf()

Load pattern creation keywords from previous run. No security checks: all relevant arguments have already been tested when loading the paradigms.

get_patterns(paradigms, **kwargs)

Creates patterns with a stable config strategy.

Parameters:

• paradigms (qumin.representations.paradigms.Paradigms) – Paradigms representation.

• kwargs (dict) – Additional keyword arguments are passed patterns.from_file().

get_resource_path(resource)

Return the full path to a resource

Parameters:

resource (str) – A resource name

Returns:

a path to the resource.

Return type:

pathlib.Path

get_table_path(table_name)

Return the path to a dataset table

register_file(rel_path, name=None, custom=None, override=False, **kwargs)

Add a file as a frictionless resource.

Parameters:

• rel_path (str or pathlib.Path) – the relative path to the file.

• name (str) – name of the resource. By default, this will be the name of the file without the extension.

• override (bool) – Whether to override an existing resource.

• custom (dict) – Custom properties to save.

• **kwargs (dict) – Optional keyword arguments passed to Resource, e.g. description.

save_metadata()

Save the metadata as a JSON file.

qumin.utils.metadata.diff(msg, old, new)

qumin.utils.metadata.get_recursively(cfg, key)

Module contents

qumin.utils.adjust_cpus(n)

qumin.utils.memory_check(df, factor, max_gb=2, force=False)

Checks memory usage for a dataframe and warn if it exceeds a certain limit.

Parameters:

• df (pandas.DataFrame) – dataframe to test

• factor (int) – multiplication factor for the test.

• max_gb (float) – Threshold for memory warning.

• force (bool) – whether to allow overpassing the limit. Defaults to False.

qumin.utils.metadata module

class qumin.utils.metadata.Metadata(path=None, cfg=None, rundir_path=None)

Bases: object

Metadata manager for Qumin scripts. Wrapper around the Frictionless Package class.

Basic usage :

1. Register Metadata manager;

2. Get an absolute path to the metadata folder;

3. Write to that path;

4. After writing a file, register it and set metadata (description, custom dict);

5. Export the JSON descriptor.

The Metadata class can easily be used in scripts that reuse Qumin results. In that case, one must pass a value to runtime_path, if hydra is not set (which is very likely if you write a simple script).

Examples

import omegaconf
cfg = omegaconf.dictconfig.DictConfig({data="myparalex/package.json"})
md = Metadata(cfg=cfg, path="myprevious_run/metadata.json")
name = 'path/myfile.txt'
filename = md.get_path(name)
# Open an IO stream and write to ``filename``.
md.register_file(name, description="My nice file", custom={"property": "value"})
md.save_metadata(path)

Variables:

• start (datetime) – timestamp at the beginning of the run.

• prefix (Path) – normalized prefix for the output files

• cfg (OmegaConf) – all arguments passed to the python script

• paralex (frictionless.Package) – a frictionless Package representing a dataset.

__init__(path=None, cfg=None, rundir_path=None)

Parameters:

• cfg (OmegaConf.dictconfig.DictConfig) – arguments passed to the script.

• path (str) – Path to a Frictionless descriptor of a previous run to be imported.

• rundir_path (str) – Directory that should be used to export the results. Useful only if path is None and hydra is not used.

get_paradigm_conf(cfg)

Load paradigm creation keywords from previous run.

A few security checks are performed to ensure the user didn’t pass contradictory arguments. If this is the case, a warning is thrown and old arguments are kept.

Under some conditions, arguments can be overwritten (e.g. cells list).

Parameters:

cfg (OmegaConf.dictconfig.DictConfig) – Arguments passed to the current run. These arguments might override arguments from the previous run, under specific conditions.

get_paradigms(md, **kwargs)

Creates paradigms with a stable config strategy.

Parameters:

• md (qumin.utils.metadata.Metadata) – Metadata handler of the current run.

• kwargs (dict) – Additional keyword arguments are passed to qumin.representations.paradigms.Paradigms.

get_path(rel_path)

Return an absolute path to a file and create parent directories.

Parameters:

rel_path (str) – relative path to the file or folder.

Returns:

absolute path to the file or folder.

Return type:

pathlib.Path

get_pattern_conf()

Load pattern creation keywords from previous run. No security checks: all relevant arguments have already been tested when loading the paradigms.

get_patterns(paradigms, **kwargs)

Creates patterns with a stable config strategy.

Parameters:

• paradigms (qumin.representations.paradigms.Paradigms) – Paradigms representation.

• kwargs (dict) – Additional keyword arguments are passed patterns.from_file().

get_resource_path(resource)

Return the full path to a resource

Parameters:

resource (str) – A resource name

Returns:

a path to the resource.

Return type:

pathlib.Path

get_table_path(table_name)

Return the path to a dataset table

register_file(rel_path, name=None, custom=None, override=False, **kwargs)

Add a file as a frictionless resource.

Parameters:

• rel_path (str or pathlib.Path) – the relative path to the file.

• name (str) – name of the resource. By default, this will be the name of the file without the extension.

• override (bool) – Whether to override an existing resource.

• custom (dict) – Custom properties to save.

• **kwargs (dict) – Optional keyword arguments passed to Resource, e.g. description.

save_metadata()

Save the metadata as a JSON file.

qumin.utils.metadata.diff(msg, old, new)

qumin.utils.metadata.get_recursively(cfg, key)