How is it possible to do the following Step in new version(use of pandas_concat_inputs)?:

                                    transformer=GroupbyAggregationsFeatures(AGGREGATION_RECIPIES),
                                    input_steps=[df_step],
                                    input_data=['input'],
                                    adapter=Adapter({
                                        'X': ([('input', 'X'),
                                               (df_step.name, 'X')],
                                              pandas_concat_inputs)
                                    }),
                                    cache_dirpath=config.env.cache_dirpath)

Pull Request template

Doc contributions

Contributing.html FAQ.html intro.html testdoc.html

tested by running in docs/

>>> (Steppy) sphinx-apidoc -o generated/ -d 4 -fMa ../steppy
 >>> (Steppy) clear;make clean;make html

Regards Bruce

core contributors to the minerva.ml

I have the following structure of my steps. The problem is that many steps are called more than once and it makes the process of training very slow. Is it possible somehow to simplify it? more precisely, how to optimize this part? I would like to compute input_missing just once

I do not really get the difference between these two things. Both of them cache the result of execution in the disc. Is it a good idea to add cache_output to all the Steps to avoid any executions twice? In some of your examples, you use both cache and persist at the same time, I think it is a good idea to use one of it...

Fixes https://github.com/neptune-ml/steppy/issues/39

This PR adds an optional id field to data dictionary. When cache_output is set to True, theid field is appended to step.nameto distinguish between output caches produced by different data dictionaries.

For example:

data_train = {
    'id': 'data_train'
    'input': {
        'features': np.array([
            [1, 6],
            [2, 5],
            [3, 4]
        ]),
        'labels': np.array([2, 5, 3]),
    }
}
step = Step(
    name='test_cache_output_with_key',
    transformer=IdentityOperation(),
    input_data=['input'],
    experiment_directory='/exp_dir',
    cache_output=True
)
step.fit_transform(data_train)

This will produce a output cache file at /exp_dir/cache/test_cache_output_with_key__data_train.

This is my idea for simplifying adapter syntax. The benefit is that importing the extractor E from the adapter module is no longer needed. On the other hand, the rules for deciding if something is an atomic recipe or part of a larger recipe or even a constant get more complicated.

Problem: Currently User must from steppy.adapter import Adapter, E in order to use adapters.

Refactor so that:

• Use does not have to import E
• add Example to docstrings

Refactor is comprehensive, so that:

• correct the code
• correct tests
• correct docstrings

Look here: https://github.com/minerva-ml/gradus/blob/dev/steps/pytorch/callbacks.py#L266 If self.epoch_id is equal to 0, then loss_sum is equal to self.best_score and model is not saved. I think it should be fixed, because sometimes we want to have model after first epoch saved.

Current syntax for adapters has some peculiarities. Consider the following example.

        step = Step(
            name='ensembler',
            transformer=Dummy(),
            input_data=['input_1'],
            adapter={'X': [('input_1', 'features')]},
            cache_dirpath='.cache'
        )

This step basically extracts one element of the input. It seems redundant to write brackets and parentheses. Doing adapter={'X': ('input_1', 'features')}, should be sufficient.

Moreover, to my suprise adapter={'X': [('input_1', 'features'), ('input_2', 'extra_features')]}, is incorrect, and currently leads to ValueError: too many values to unpack (expected 2)

My suggestions to make the syntax consistent are:

1. adapter={'X': ('input_1', 'features')} should map X to extracted features.
2. adapter={'X': [...]} should map X to a list of extracted objects (specified by elements of the list). In particular adapter={'X': [('input_1', 'features')]} should map X to a one-element list with extracted features.
3. adapter={'X': ([...], func)} should extract appropriate objects and put them on the list, then func should be called on that list, and X should map to the result of that call.

Pull Request template

Doc contributions

This represents 0.01, where we/you were at 0.0? As you should be able to see I was able to use 95% of what was there previously. redid index.rst redid conf.py added directory docs.nbdocs

needs more work . about days worth. before pushing out to read the docs.

i found the docstrings very strong.

i not very strongly suggest step-toolkit and steppy-examples be merged into one project.

I see you use goggle-docstring-style. i will switch from numpy-style.

Regards Bruce

Started. intend on first pass to fill with my (naive/embarassing) discoveries and really good (i.e. incredibly stupid) questions and enlightening answers from gaggle.

Hi! Let's assume I'm training a huge network for a lot of epochs and it saves checkpoints in checkpoints folder. I suggest to prepare a possibility to run transform on a pipeline, when transformer is not in experiment_dir/transformers, but a checkpoint is available in checkpoints folder. What do you think?

@kamil-kaczmarek, @jakubczakon I know it is a bunch of different ideas and suggestions clustered in one issue. Let me know which of those are compatible with the current roadmap. (I am happy to contribute/collaborate on some.)

• default data folder (e.g. ./.steppy/step_name/) or to be configurable if needed; overriding only when strictly necessary
• no input_data; it complicates things for no obvious reason!
• names optional, automatically generated from class names + number
• more explicit job structure (steps = Sequence([step1, step2])); vide Keras API
• adapters as inheriting from BaseTrainers,step = Rename({'a': 'aaa', 'b': 'bbb'}), vide rename in Pandas
• how to separate persist-data vs persist-parameters? (e.g. for image preprocessing, it may be time-saving to save once processed images)
• built-in data tests (e.g. len(X) == len(Y)), in def test
• built-in test if persist->load is correct (i.e. loaded data is the same as saved)

Is it necessary to divide executions inside my class to be separate Thread or just divide them between Steps? For example, I can to fit KNN, PCA in one class method and parallel them or create two separate classes for them...

Hi, I have a proposal: let's make it possible to dump adapted input of a step to disk. It's very handy when you are working on a 5th or 10th step in a pipeline that has 2,3 or more input steps. Now you have to set flag load_saved_output=True on each of the input steps to be able to work on your beloved step. If you could just set load_saved_input=True (adapted or not adapted, I think it's worth discussion) on the step you are currently working on, it would be much easier. What do you think?