Temporary Storage
A temporary storage that uses GenericIO to save data |
|
A temporary storage that uses numpy.savez to save data |
References
GenericIOStore
- class haccytrees.utils.datastores.GenericIOStore(partition, box_size, temporary_path=None)[source]
A temporary storage that uses GenericIO to save data
The data has to be a Structure-of-Arrays (SoA) (i.e. a python dictionary str->`np.ndarray`), with each array having the same length.
This class behaves like a python dictionary of SoA`s. If `temporary_path is set, the arrays associated to a key will be stored in GenericIO files, otherwise, they will be kept in memory.
- Parameters:
partition (
mpipartition.Partition) – A Partition instance defining the MPI layoutbox_size (float) – the physical size of the volume
temporary_path (str) – The base filesystem path where temporary data is stored. If None, the data will be kept in memory.
Examples
>>> # Creating a partition >>> partition = mpipartition.Partition() >>> # Creating a store >>> store = GenericIOStore(partition, 1.0, './tmp') >>> data = {x: np.random.uniform(10) for x in 'xyz'} >>> store['pos_0'] = data >>> del data >>> # Do some memory-expensive stuff until you need the data again... >>> data_0 = store['pos_0'] >>> # Cleanup >>> store.remove('pos_0')
- __getitem__(key)[source]
Retrieve a Structure-of-Arrays from the store
- Parameters:
key (str) – The storage key of the SoA
- Returns:
The SoA associated with the key. A python dictionary of type {str: np.ndarray}
- Return type:
Mapping[str, np.ndarray]
- __setitem__(key, data)[source]
Adding an SoA to the storage
- Parameters:
key (str) – The storage key, will be appended to the temporary_path and therefore has to be a valid if used in a filesystem path.
data (Mapping[str, ndarray]) – The SoA to be added to the store. A dictionary with types {str: np.ndarray}, where the numpy array have to have the same shape and dim=1.
- Return type:
None
- get_field(key, field)[source]
Retrieve specific arrays in a Structure-of-Arrays from the store
- Parameters:
key (str) – The storage key of the SoA
field (Sequence[str]) – The keys of the specific arrays that are to be returned. Can be a str or a list of str.
- Returns:
The SoA associated with the key, with only the fields specified. A python dictionary of type {str: np.ndarray}.
- Return type:
Mapping[str, np.ndarray]
NumpyStore
- class haccytrees.utils.datastores.NumpyStore(partition, temporary_path=None)[source]
A temporary storage that uses numpy.savez to save data
The data has to be a dictionary of arrays (i.e. str->`np.ndarray`), arrays can have variable lengths and dimensions. If temporary_path is set, the arrays associated to a key will be stored in .npz files (one per MPI rank), otherwise, they will be kept in memory.
- Parameters:
partition (
mpipartition.Partition) – A Partition instance defining the MPI layouttemporary_path (Optional[str]) – The base filesystem path where temporary data is stored. If None, the data will be kept in memory.
Examples
>>> # Creating a partition >>> partition = mpipartition.Partition() >>> # Creating a store >>> store = NumpyStore(partition, './tmp') >>> data = {x: np.random.uniform(10) for x in 'xyz'} >>> store['pos_0'] = data >>> # Do some memory-expensive stuff until you need the data again... >>> data_0 = store['pos_0'] >>> # Cleanup >>> store.remove('pos_0')
- __getitem__(key)[source]
Retrieve a dictionary of arrays from the store
- Parameters:
key (str) – The storage key
- Returns:
The data associated with the key. A python dictionary of type {str: np.ndarray}
- Return type:
Mapping[str, np.ndarray]
- __setitem__(key, data)[source]
Adding a dictionary of arrays to the storage
- Parameters:
key (str) – The storage key, will be appended to the temporary_path and therefore has to be a valid if used in a filesystem path.
data (Mapping[str, np.ndarray]) – The data to be added to the store. A dictionary with types {str: np.ndarray}, where the numpy array can have variable shape and dimensions.
- Return type:
None