Data Management

DataDim(value[, names, module, qualname, ...])	Enum for dimensionality representation of data
DataSource(value[, names, module, qualname, ...])	Enum for source of data
DataDistribution(value[, names, module, ...])	Enum for distribution of data
Axis(*args, **kwargs)	Object holding info and data about physical axis of some data
DataBase(*args, **kwargs)	Base object to store homogeneous data and metadata generated by pymodaq's objects.
DataRaw(*args, **kwargs)	Specialized DataWithAxes set with source as 'raw'.
DataCalculated(*args, **kwargs)	Specialized DataWithAxes set with source as 'calculated'.
DataFromRoi(*args, **kwargs)	Specialized DataWithAxes set with source as 'calculated'.
DataToExport(*args, **kwargs)	Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal...

Axes

Created the 28/10/2022

@author: Sebastien Weber

class pymodaq_data.data.Axis(*args, **kwargs)

Object holding info and data about physical axis of some data

In case the axis’s data is linear, store the info as a scale and offset else store the data

Parameters:

• label (str) – The label of the axis, for instance ‘time’ for a temporal axis

• units (str) – The units of the data in the object, for instance ‘s’ for seconds

• data (ndarray) – A 1D ndarray holding the data of the axis

• index (int) – an integer representing the index of the Data object this axis is related to

• scaling (float) – The scaling to apply to a linspace version in order to obtain the proper scaling

• offset (float) – The offset to apply to a linspace/scaled version in order to obtain the proper axis

• size (int) – The size of the axis array (to be specified if data is None)

• spread_order (int) – An integer needed in the case where data has a spread DataDistribution. It refers to the index along the data’s spread_index dimension

Examples

>>> axis = Axis('myaxis', units='seconds', data=np.array([1,2,3,4,5]), index=0)

create_linear_data(nsteps: int)

replace the axis data with a linear version using scaling and offset

static deserialize(bytes_str) → Tuple[Axis, bytes]

Convert bytes into an Axis object

Convert the first bytes into an Axis reading first information about the Axis

Returns:

• Axis (the decoded Axis)

• bytes (the remaining bytes string if any)

find_index(threshold: float) → int

find the index of the threshold value within the axis

flip()

flip the direction of the axis

force_units(units: str)

Change immediately the units to whatever else. Use this with care!

get_data() → ndarray

Convenience method to obtain the axis data (usually None because scaling and offset are used)

get_data_at(indexes: int | Iterable | slice) → ndarray

Get data at specified indexes

Parameters:

indexes –

get_quantity() → Quantity

Convenience method to obtain the numerical data as a quantity array

get_scale_offset_from_data(data: ndarray = None)

Get the scaling and offset from the axis’s data

If data is not None, extract the scaling and offset

Parameters:

data (ndarray) –

static serialize(axis: Axis)

Convert an Axis object into a bytes message together with the info to convert it back

Parameters:

axis (Axis) –

Returns:

bytes

Return type:

the total bytes message to serialize the Axis

Notes

The bytes sequence is constructed as:

• serialize the axis label

• serialize the axis units

• serialize the axis array

• serialize the axis

• serialize the axis spread_order

property data

get/set the data of Axis

Type:

np.ndarray

property index: int

get/set the index this axis corresponds to in a DataWithAxis object

Type:

int

property label: str

get/set the label of this axis

Type:

str

property size: int

get/set the size/length of the 1D ndarray

Type:

int

property units: str

get/set the units for this axis without conversion (equivalent to force_units)

Type:

str

DataObjects

Created the 28/10/2022

@author: Sebastien Weber

class pymodaq_data.data.DataBase(*args, **kwargs)

Base object to store homogeneous data and metadata generated by pymodaq’s objects.

To be inherited for real data

Parameters:

• name (str) – the identifier of these data

• source (DataSource or str) – Enum specifying if data are raw or processed (for instance from roi)

• dim (DataDim or str) – The identifier of the data type

• distribution (DataDistribution or str) – The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points

• data (list of ndarray or Quantities) – The data the object is storing. In case of Quantities, the object units attribute will be forced to the unit of this quantity, ignoring the units argument.

• labels (list of str) – The labels of the data nd-arrays

• origin (str) – An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.

• units (str) – A unit string identifier as specified in the UnitRegistry of the pint module

• kwargs (named parameters) – All other parameters are stored dynamically using the name/value pair. The name of these extra parameters are added into the extra_attributes attribute

name

the identifier of these data

Type:

str

source

Enum specifying if data are raw or processed (for instance from roi)

Type:

DataSource or str

dim

The identifier of the data type

Type:

DataDim or str

distribution

The distribution type of the data: uniform if distributed on a regular grid or spread if on specific unordered points

Type:

DataDistribution or str

data

The data the object is storing

Type:

list of ndarray

labels

The labels of the data nd-arrays

Type:

list of str

origin

An identifier of the element where the data originated, for instance the DAQ_Viewer’s name. Used when appending DataToExport in DAQ_Scan to disintricate from which origin data comes from when scanning multiple detectors.

Type:

str

shape

The shape of the underlying data

Type:

Tuple[int]

size

The size of the ndarrays stored in the object

Type:

int

length

The number of ndarrays stored in the object

Type:

int

extra_attributes

list of string giving identifiers of the attributes added dynamically at the initialization (for instance to save extra metadata using the DataSaverLoader

Type:

List[str]

See also

DataWithAxes, DataFromPlugins, DataRaw, DataSaverLoader

Examples

>>> import numpy as np
>>> from pymodaq.utils.data import DataBase, DataSource, DataDim, DataDistribution
>>> data = DataBase('mydata', source=DataSource['raw'], dim=DataDim['Data1D'],     distribution=DataDistribution.uniform, data=[np.array([1.,2.,3.]), np.array([4.,5.,6.])],    labels=['channel1', 'channel2'], origin='docutils code')
>>> data.dim
<DataDim.Data1D: 1>
>>> data.source
<DataSource.raw: 0>
>>> data.shape
(3,)
>>> data.length
2
>>> data.size
3

abs()

Take the absolute value of itself

angle()

Take the phase value of itself

append(data: DataWithAxes)

Append data content if the underlying arrays have the same shape and compatible units

as_dte(name: str = 'mydte') → DataToExport

Convenience method to wrap the DataWithAxes object into a DataToExport

average(other: DataBase, weight: int) → DataBase

Compute the weighted average between self and other DataBase

Parameters:

• other_data (DataBase) –

• weight (int) – The weight the ‘other’ holds with respect to self

Returns:

DataBase

Return type:

the averaged DataBase object

fliplr()

Reverse the order of elements along axis 1 (left/right)

flipud()

Reverse the order of elements along axis 0 (up/down)

force_units(units: str)

Change immediately the units to whatever else. Use this with care!

get_data_index(index: int = 0) → ndarray

Get the data by its index in the list, same as self[index]

get_dim_from_data(data: List[ndarray])

Get the dimensionality DataDim from data

get_full_name() → str

Get the data ful name including the origin attribute into the returned value

Returns:

str

Return type:

the name of the ataWithAxes data constructed as : origin/name

Examples

d0 = DataBase(name=’datafromdet0’, origin=’det0’)

imag()

Take the imaginary part of itself

pop(index: int) → DataBase

Returns a copy of self but with data taken at the specified index

real()

Take the real part of itself

set_dim(dim: DataDim | str)

Addhoc modification of dim independantly of the real data shape, should be used with extra care

stack_as_array(axis=0, dtype=None) → ndarray

Stack all data arrays in a single numpy array

Parameters:

• axis (int) – The new stack axis index, default 0

• dtype (str or np.dtype) – the dtype of the stacked array

Return type:

np.ndarray

See also

np.stack()

to_dB() → DataBase

Get a new data object in decibels

new in 4.3.0

to_dict()

Get the data arrays into dictionary whose keys are the labels

units_as(units: str, inplace=True, context: str = None, **context_kwargs) → DataBase

Set the object units to the new one (if possible)

Parameters:

• units (str) – The new unit to convert the data to

• inplace (bool) – default True. If True replace the data’s arrays by array in the new units If False, return a new data object

• context (str) – See pint documentation

unwrap()

unwrap the underlying array (should be angles otherwise meaningless)

value(units: str = None) → float

Returns the underlying float value (of the first elt in the data list) if this data holds only a float otherwise returns a mean of the underlying data

Parameters:

units (str) – if unit is compatible with self.units, convert the data to these new units before getting the value

values(units: str = None) → List[float]

Returns the underlying float value (for each data array in the data list) if this data holds only a float otherwise returns a mean of the underlying data

property data: List[ndarray]

get/set (and check) the data the object is storing

Type:

List[np.ndarray]

property dim

the enum representing the dimensionality of the stored data

Type:

DataDim

property distribution

the enum representing the distribution of the stored data

Type:

DataDistribution

property length

The length of data. This is the length of the list containing the nd-arrays

property quantities: list[pint.Quantity]

Get the arrays as pint quantities (with units)

property shape

The shape of the nd-arrays

property size

The size of the nd-arrays

property source

the enum representing the source of the data

Type:

DataSource

class pymodaq_data.data.DataCalculated(*args, **kwargs)

Specialized DataWithAxes set with source as ‘calculated’. To be used for processed/calculated data

class pymodaq_data.data.DataFromRoi(*args, **kwargs)

Specialized DataWithAxes set with source as ‘calculated’. To be used for processed data from region of interest

class pymodaq_data.data.DataRaw(*args, **kwargs)

Specialized DataWithAxes set with source as ‘raw’. To be used for raw data

class pymodaq.utils.data.DataActuator(*args, **kwargs)

Specialized DataWithAxes set with source as ‘raw’. To be used for raw data generated by actuator plugins

class pymodaq.utils.data.DataFromPlugins(*args, **kwargs)

Specialized DataWithAxes set with source as ‘raw’. To be used for raw data generated by Detector plugins

It introduces by default to extra attributes, do_plot and do_save. Their presence can be checked in the extra_attributes list.

Parameters:

• do_plot (bool) – If True the underlying data will be plotted in the DAQViewer

• do_save (bool) – If True the underlying data will be saved

do_plot

If True the underlying data will be plotted in the DAQViewer

Type:

bool

do_save

If True the underlying data will be saved

Type:

bool

Data Characteristics

Created the 28/10/2022

@author: Sebastien Weber

class pymodaq_data.data.DataDim(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Enum for dimensionality representation of data

class pymodaq_data.data.DataDistribution(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Enum for distribution of data

class pymodaq_data.data.DataSource(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Enum for source of data

Union of Data

When exporting multiple set of Data objects, one should use a DataToExport

Created the 28/10/2022

@author: Sebastien Weber

class pymodaq_data.data.DataToExport(*args, **kwargs)

Object to store all raw and calculated DataWithAxes data for later exporting, saving, sending signal…

Includes methods to retrieve data from dim, source… Stored data have a unique identifier their name. If some data is appended with an existing name, it will replace the existing data. So if you want to append data that has the same name

Parameters:

• name (str) – The identifier of the exporting object

• data (list of DataWithAxes) – All the raw and calculated data to be exported

name

timestamp

data

affect_name_to_origin_if_none()

Affect self.name to all DataWithAxes children’s attribute origin if this origin is not defined

average(other: DataToExport, weight: int) → DataToExport

Compute the weighted average between self and other DataToExport and attributes it to self

Parameters:

• other (DataToExport) –

• weight (int) – The weight the ‘other_data’ holds with respect to self

classmethod deserialize(bytes_str) → Tuple[DataToExport, bytes]

Convert bytes into a DataToExport object

Convert the first bytes into a DataToExport reading first information about the underlying data

Returns:

• DataToExport (the decoded DataToExport)

• bytes (the remaining bytes if any)

get_data_from_Naxes(Naxes: int, deepcopy: bool = False) → DataToExport

Get the data matching the given number of axes

Parameters:

Naxes (int) – Number of axes in the DataWithAxes objects

Returns:

DataToExport

Return type:

filtered with data matching the number of axes

get_data_from_attribute(attribute: str, attribute_value: Any, deepcopy=False) → DataToExport

Get the data matching a given attribute value

Returns:

DataToExport

Return type:

filtered with data matching the attribute presence and value

get_data_from_dim(dim: DataDim, deepcopy=False) → DataToExport

Get the data matching the given DataDim

Returns:

DataToExport

Return type:

filtered with data matching the dimensionality

get_data_from_dims(dims: List[DataDim], deepcopy=False) → DataToExport

Get the data matching the given DataDim

Returns:

DataToExport

Return type:

filtered with data matching the dimensionality

get_data_from_full_name(full_name: str, deepcopy=False) → DataWithAxes

Get the DataWithAxes with matching full name

get_data_from_missing_attribute(attribute: str, deepcopy=False) → DataToExport

Get the data matching a given attribute value

Parameters:

• attribute (str) – a string of a possible attribute

• deepcopy (bool) – if True the returned DataToExport will contain deepcopies of the DataWithAxes

Returns:

DataToExport

Return type:

filtered with data missing the given attribute

get_data_from_name(name: str) → DataWithAxes

Get the data matching the given name

get_data_from_name_origin(name: str, origin: str = '') → DataWithAxes

Get the data matching the given name and the given origin

get_data_from_sig_axes(Naxes: int, deepcopy: bool = False) → DataToExport

Get the data matching the given number of signal axes

Parameters:

Naxes (int) – Number of signal axes in the DataWithAxes objects

Returns:

DataToExport

Return type:

filtered with data matching the number of signal axes

get_data_from_source(source: DataSource, deepcopy=False) → DataToExport

Get the data matching the given DataSource

Returns:

DataToExport

Return type:

filtered with data matching the dimensionality

get_data_with_naxes_lower_than(n_axes=2, deepcopy: bool = False) → DataToExport

Get the data with n axes lower than the given number

Parameters:

Naxes (int) – Number of axes in the DataWithAxes objects

Returns:

DataToExport

Return type:

filtered with data matching the number of axes

get_full_names(dim: DataDim = None)

Get the ful names including the origin attribute into the returned value, eventually filtered by dim

Parameters:

dim (DataDim or str) –

Returns:

list of str

Return type:

the names of the (filtered) DataWithAxes data constructed as : origin/name

Examples

d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)

get_names(dim: DataDim = None) → List[str]

Get the names of the stored DataWithAxes, eventually filtered by dim

Parameters:

dim (DataDim or str) –

Returns:

list of str

Return type:

the names of the (filtered) DataWithAxes data

get_origins(dim: DataDim = None)

Get the origins of the underlying data into the returned value, eventually filtered by dim

Parameters:

dim (DataDim or str) –

Returns:

list of str

Return type:

the origins of the (filtered) DataWithAxes data

Examples

d0 = DataWithAxes(name=’datafromdet0’, origin=’det0’)

index(data: DataWithAxes)

Here use a comparison to assert data is equal to one element in the list

But the __eq__ method is not checking the name while it is the main issue for elt finding Hence here I’m doing both checks

index_from_name_origin(name: str, origin: str = '') → List[DataWithAxes]

Get the index of a given DataWithAxes within the list of data

merge_as_dwa(dim: str | DataDim, name: str = None) → DataRaw

attempt to merge filtered dwa into one

Only possible if all filtered dwa and underlying data have same shape

Parameters:

• dim (DataDim or str) – will only try to merge dwa having this dimensionality

• name (str) – The new name of the returned dwa

plot(plotter_backend: str = 'matplotlib', *args, **kwargs)

Call a plotter factory and its plot method over the actual data

pop(index: int) → DataWithAxes

return and remove the DataWithAxes referred by its index

Parameters:

index (int) – index as returned by self.index_from_name_origin

See also

index_from_name_origin

remove(dwa: DataWithAxes)

Use the DataWithAxes object comparison __eq__ to retrieve the elt to remove

Parameters:

dwa (DataWithAxes) – The da to remove from the list

static serialize(dte: DataToExport)

Convert a DataToExport into a bytes string

Parameters:

dte (DataToExport) –

Returns:

bytes

Return type:

the total bytes message to serialize the DataToExport

Notes

The bytes sequence is constructed as:

• serialize the string type: ‘DataToExport’

• serialize the timestamp: float

• serialize the name

• serialize the list of DataWithAxes

property data: List[DataWithAxes]

get the data contained in the object

Type:

List[DataWithAxes]