Datamix library to register, automate and replay operations on maps and geospatial data

Data processing to produce maps often involves many operations and intermediate stages. These operations are sometimes repetitive and can be very tedious for the operator, thus need automation. The first classic operations consist in the production of basic maps representing various data. It usually involves large amounts of data that can be measurements at some points or on some geometric contours.

Automating the sequence of steps allows for a double gain in efficiency. The gain lies in the time saved by the operator carrying out the treatments. This also concerns the reduction of human errors that inevitably occur when repeating monotonous tasks.

The Datamix.workflow Python library is designed to define, chain, and replay workflow steps efficiently by processing input and output data streams through Python functions. It enables automated execution on large datasets and facilitates stochastic realizations, such as Monte Carlo simulations, to generate probabilistic results.

The Datamix.workflow package, developed in the Tellus project, is equipped with a User Interface for accessibility, allowing users to define each workflow step by executing pre-listed Python methods. Data transformation and chaining are managed through user-named variables, which can represent file names or uncertain parameters tied to stochastic laws for Monte Carlo simulations. This functionality enables random evaluations during realization calculations, with output variables stored in a designated directory for analysis.

The methods available in the datamix.maps package for creating maps can be extended and made available in the workflow management tool. The available stochastic laws must be completed and enriched (uniform and normal laws are available) The use of the datamix.workflow package is not limited to map creation methods. The principle can be applied to other types of data processing.