Hey,

Thanks for your awesome repo.

I have a question if you don't mind could you please give me an example on how to change RandomForestClassifier and ExtraTreesClassifier in the CascadeForestClassifier?

Hey,

This is an initial implementation, however I'm not sure it will work I see that we will get the following error:

AttributeError: 'CascadeForestClassifier' object has no attribute 'estimators_'

What do you think @xuyxu ?

When I was using this package, I experienced the following problem. According to my observation, there is still a lot of available memory. Thus, what's the problem?

  File "deepforest/tree/_tree.pyx", line 123, in deepforest.tree._tree.DepthFirstTreeBuilder.build
  File "deepforest/tree/_tree.pyx", line 256, in deepforest.tree._tree.DepthFirstTreeBuilder.build
  File "deepforest/tree/_tree.pyx", line 480, in deepforest.tree._tree.Tree._resize_node_c
  File "deepforest/tree/_utils.pyx", line 34, in deepforest.tree._utils.safe_realloc
MemoryError: could not allocate 0 bytes

Now, the fit method only support np.array for input. However, most ml algorithms with scikit-learn-Compatible API (i.e. XGBoost , NGBoost) support DataFrame object or List or numpy array of predictors (n x p) in numeric format using sklearn.utils.check_array . This RP is to make Deep-Forest consistent with other algorithms so that it will be more easy to be used in other integrated machine learning frameworks (i.e. PyCaret).

Describe the bug CascadeForestRegressor somehow cannot be inserted into a DataFrame

To Reproduce

import pandas as pd
from deepforest import CascadeForestRegressor
from ngboost import NGBRegressor

ngr = NGBRegressor()  # ngboost regressor for example. xgb, lgb should also be no problem.
cfr = CascadeForestRegressor()
df= pd.DataFrame()

# somehow OK
df.insert(0, "ngr", [ngr])
# somehow error
df.insert(0, "cf", [cforest])

Expected behavior No error

Additional context

ValueError                                Traceback (most recent call last)
<ipython-input-32-ab0139d10254> in <module>
----> 1 df.insert(0, "cf", [cforest])

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/pandas/core/frame.py in insert(self, loc, column, value, allow_duplicates)
   3760             )
   3761         self._ensure_valid_index(value)
-> 3762         value = self._sanitize_column(column, value, broadcast=False)
   3763         self._mgr.insert(loc, column, value, allow_duplicates=allow_duplicates)
   3764 

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/pandas/core/frame.py in _sanitize_column(self, key, value, broadcast)
   3900             if not isinstance(value, (np.ndarray, Index)):
   3901                 if isinstance(value, list) and len(value) > 0:
-> 3902                     value = maybe_convert_platform(value)
   3903                 else:
   3904                     value = com.asarray_tuplesafe(value)

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/pandas/core/dtypes/cast.py in maybe_convert_platform(values)
    110     """ try to do platform conversion, allow ndarray or list here """
    111     if isinstance(values, (list, tuple, range)):
--> 112         values = construct_1d_object_array_from_listlike(values)
    113     if getattr(values, "dtype", None) == np.object_:
    114         if hasattr(values, "_values"):

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/pandas/core/dtypes/cast.py in construct_1d_object_array_from_listlike(values)
   1636     # making a 1D array that contains list-likes is a bit tricky:
   1637     result = np.empty(len(values), dtype="object")
-> 1638     result[:] = values
   1639     return result
   1640 

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/deepforest/cascade.py in __getitem__(self, index)
    518 
    519     def __getitem__(self, index):
--> 520         return self._get_layer(index)
    521 
    522     def _get_n_output(self, y):

/mnt/hdd2/lvhao/miniconda3/envs/pycaret/lib/python3.7/site-packages/deepforest/cascade.py in _get_layer(self, layer_idx)
    561             logger.debug("self.n_layers_ = "+ str(self.n_layers_))
    562             logger.debug("layer_idx = "+ str(layer_idx))
--> 563             raise ValueError(msg.format(self.n_layers_ - 1, layer_idx))
    564 
    565         layer_key = "layer_{}".format(layer_idx)

ValueError: The layer index should be in the range [0, 1], but got 2 instead.

This bug can be simpliy fixed if we change if not 0 <= layer_idx < self.n_layers_: to if not 0 <= layer_idx <= self.n_layers_:, but I still don't know the cause of this error and whether this fix is corret.

Hi All,

As I expressed previously in another post, I want to express my gratitude for your amazing research. I am delighted you found time in your library to deal with custom estimators. However, I am having difficulty with the following:

Assume I develop the following implementation (using imblearn) and obtain an AUROC score of 0.62:

model = BalancedRandomForestClassifier(random_state=global_seed_random_state,
                                       class_weight="balanced_subsample",
                                       n_jobs=-1,
                                       replacement=True,
                                       )

model.fit(X_train, y_train)
y_pred = model.predict(X_test)
show_output(model, X_test, y_test, y_pred)


Classification_report:
              precision    recall  f1-score   support

           0       0.97      0.61      0.75       499
           1       0.08      0.63      0.14        27

    accuracy                           0.61       526
   macro avg       0.52      0.62      0.45       526
weighted avg       0.92      0.61      0.72       526

ROC AUC Score:
0.6204260372597047

According to the reviews I've been reading regarding your original paper, if we have good results with RF and other similar classifiers, it is worthwhile to attempt Deep Forest and as base learner the one that worked well. However, I attempted to use the custom estimators via the following implementation:

model = CascadeForestClassifier(
    random_state=global_seed_random_state,
)

main_estimators = [BalancedRandomForestClassifier(
    class_weight="balanced_subsample",
    n_jobs=-1,
    replacement=True,
    random_state=global_seed_random_state,
) for _ in range(2)]


diverse_estimators = [BalancedRandomForestClassifier(
    class_weight="balanced_subsample",
    n_jobs=-1,
    replacement=True,
    random_state=global_seed_random_state,
) for _ in range(2)]

estimators = main_estimators + diverse_estimators

# layer
model.set_estimator(estimators)

The findings, however, are 10% less impressive, with an AUROC of 0.555. Note: Above, diverse estimators appears because I attempted to add ExtraTrees or XGBoost instead of a second set of BalancedRandomForestClassifiers. Could you please attempt to direct me in the proper direction? What did I do incorrectly? From your perspective, what type of diversified classifier should I use? Note 2: An AUROC above of 0.6 is quite promising for my current application.

Thank you very much for your help in advanced. Great day,

Hi maintainer,

I am wondering is that possible to cascade random survival forest (maybe a sksurv model) instead of RF in your deep forest model? As in #48, it seems that the supported model types are classification and regression. (or did I miss some parts of those tutorial docs?)

Thanks.

resolves #29 #26

Steps

• [x] Implement K-Fold wrapper for base estimators
• [x] Implement customized cascade layer
• [x] Implement set_estimator and set_predictor for the model
• [x] Add unit tests
• [x] Add backward compatibility
• [x] Add documentation and working examples

Code Snippet

from deepforest import CascadeForestClassifier

model = CascadeForestClassifier()

# New Steps
estimator_1, estimator_2 = your_estimator(), your_estimator()
model.set_estimator(estimator=[estimator_1, estimator_2],  # a list of your base estimators
                    n_splits=5,  # the number of folds
                    oob_approx=False,  # whether to use out-of-bag approximation
                    random_state=None)  # random state used for base estimators

model.set_predictor(predictor=your_predictor)  # an instantiated object of your predictor

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Resolved issue #13

This is a very naive label encoder implemented with sklearn.preprocessing.LabelEncoder

• [x] single output (1-D) partial mode
• [x] single output (1-D) full mode
• [x] unit test

ERROR: Could not find a version that satisfies the requirement deep-forest (from versions: none) ERROR: No matching distribution found for deep-forest

system: mac python version: 3.8.5 pip version: 20.2.4

调用数据集训练出现错误： File "deepforest/_cutils.pyx", line 59, in deepforest._cutils._map_to_bins File "deepforest/_cutils.pyx", line 76, in deepforest._cutils._map_to_bins ValueError: Buffer dtype mismatch, expected 'const X_DTYPE_C' but got 'long'

i was trying to install the package using wsl2. But the terminal raises an error:

ERROR: Could not find a version that satisfies the requirement deep-forest (from versions: none) ERROR: No matching distribution found for deep-forest

i don't find any related articles or even stackoverflow post to solve this, please help me.

How to apply shap model to DF model to interpret features? I tried to apply it directly, but suggested that the model was not in the SHAP package. https://github.com/slundberg/shap. I suggest that the author improve the interpretability of the DF model,thanks.

Got this erroe with importing

ImportError Traceback (most recent call last) Input In [59], in <cell line: 24>() 22 import time 23 import io ---> 24 from deepforest import CascadeForestRegressor 25 import joblib 26 from sklearn.utils.fixes import joblib

File ~\anaconda3\lib\site-packages\deepforest_init_.py:1, in ----> 1 from .cascade import CascadeForestClassifier, CascadeForestRegressor 2 from .forest import RandomForestClassifier, RandomForestRegressor 3 from .forest import ExtraTreesClassifier, ExtraTreesRegressor

File ~\anaconda3\lib\site-packages\deepforest\cascade.py:17, in 15 from . import _utils 16 from . import _io ---> 17 from ._layer import ( 18 ClassificationCascadeLayer, 19 RegressionCascadeLayer, 20 CustomCascadeLayer, 21 ) 22 from ._binner import Binner 25 def _get_predictor_kwargs(predictor_kwargs, **kwargs) -> dict:

File ~\anaconda3\lib\site-packages\deepforest_layer.py:17, in 14 from sklearn.base import BaseEstimator, ClassifierMixin, RegressorMixin 16 from . import _utils ---> 17 from ._estimator import Estimator 18 from .utils.kfoldwrapper import KFoldWrapper 21 def _build_estimator( 22 X, 23 y, (...) 32 sample_weight=None, 33 ):

File ~\anaconda3\lib\site-packages\deepforest_estimator.py:7, in 4 all = ["Estimator"] 6 import numpy as np ----> 7 from .forest import ( 8 RandomForestClassifier, 9 ExtraTreesClassifier, 10 RandomForestRegressor, 11 ExtraTreesRegressor, 12 ) 13 from sklearn.ensemble import ( 14 RandomForestClassifier as sklearn_RandomForestClassifier, 15 ExtraTreesClassifier as sklearn_ExtraTreesClassifier, 16 RandomForestRegressor as sklearn_RandomForestRegressor, 17 ExtraTreesRegressor as sklearn_ExtraTreesRegressor, 18 ) 21 def make_classifier_estimator( 22 name, 23 criterion, (...) 30 ): 31 # RandomForestClassifier

File ~\anaconda3\lib\site-packages\deepforest\forest.py:34, in 32 from sklearn.utils import check_random_state, compute_sample_weight 33 from sklearn.exceptions import DataConversionWarning ---> 34 from sklearn.utils.fixes import _joblib_parallel_args 35 from sklearn.utils.validation import check_is_fitted, _check_sample_weight 36 from sklearn.utils.validation import _deprecate_positional_args

ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (C:\Users\Mohammad\anaconda3\lib\site-packages\sklearn\utils\fixes.py)

scikit-learn was upgraded joblib was upgraded still got error

Describe the bug cannot correctly clone CascadeForestClassifier/CascadeForestRegressor object with sklearn.base.clone when using customized stimators

To Reproduce

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.base import clone
from deepforest import CascadeForestRegressor
import xgboost as xgb
import lightgbm as lgb

X, y = load_boston(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
model = CascadeForestRegressor(random_state=1)

# set estimator
n_estimators = 4  # the number of base estimators per cascade layer
estimators = [lgb.LGBMRegressor(random_state=i)  for i in range(n_estimators)]
model.set_estimator(estimators)

# set predictor 
predictor = xgb.XGBRegressor()
model.set_predictor(predictor)

# clone model
model_new = clone(model)

# try to fit
model.fit(X_train, y_train)

Expected behavior No error

Additional context

~/miniconda3/envs/pycaret/lib/python3.8/site-packages/deep_forest-0.1.5-py3.8-linux-x86_64.egg/deepforest/cascade.py in fit(self, X, y, sample_weight)
   1004                 if not hasattr(self, "predictor_"):
   1005                     msg = "Missing predictor after calling `set_predictor`"
-> 1006                     raise RuntimeError(msg)
   1007 
   1008             binner_ = Binner(

RuntimeError: Missing predictor after calling `set_predictor`

This bug occours because when the model is cloned, if the model has customized predictor or estimators, predictor='custom' will be cloned, while self.predictor_ / self.dummy_estimators will not be correctly cloned, which introduced the bug described above.

I think this bug can be easily fixed by putting the predictor and the list of estimators into the parameter of CascadeForestClassifier/CascadeForestRegressor, just like the way of those meta estimators (e.g. ngboost), but maybe the corresponding APIs will have to be changed.

For example, the API parameters could be:

model = CascadeForestRegressor(
    estimators=[lgb.LGBMRegressor(random_state=i) for i in range(n_estimators)],
    predictor=xgb.XGBRegressor(),
)

Got error with code: from sklearn.datasets import load_digits from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score

from deepforest import CascadeForestClassifier

model = CascadeForestClassifier(random_state=1) model.fit(X_train, y_train)

TypeError Traceback (most recent call last) in 6 7 model = CascadeForestClassifier(random_state=1) ----> 8 model.fit(X_train, y_train.values.ravel())

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/deepforest/cascade.py in fit(self, X, y, sample_weight) 1395 y = self._encode_class_labels(y) 1396 -> 1397 super().fit(X, y, sample_weight) 1398 1399 def predict_proba(self, X):

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/deepforest/cascade.py in fit(self, X, y, sample_weight) 754 755 # Bin the training data --> 756 X_train_ = self.bin_data(binner, X, is_training_data=True) 757 X_train_ = self.buffer_.cache_data(0, X_train_, is_training_data=True) 758

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/deepforest/cascade.py in _bin_data(self, binner, X, is_training_data) 665 tic = time.time() 666 if is_training_data: --> 667 X_binned = binner.fit_transform(X) 668 else: 669 X_binned = binner.transform(X)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/base.py in fit_transform(self, X, y, **fit_params) 697 if y is None: 698 # fit method of arity 1 (unsupervised transformation) --> 699 return self.fit(X, **fit_params).transform(X) 700 else: 701 # fit method of arity 2 (supervised transformation)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/deepforest/_binner.py in fit(self, X) 128 self.validate_params() 129 --> 130 self.bin_thresholds = _find_binning_thresholds( 131 X, 132 self.n_bins - 1,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/deepforest/_binner.py in _find_binning_thresholds(X, n_bins, bin_subsample, bin_type, random_state) 75 if n_samples > bin_subsample: 76 subset = rng.choice(np.arange(n_samples), bin_subsample, replace=False) ---> 77 X = X.take(subset, axis=0) 78 79 binning_thresholds = []

TypeError: take() got an unexpected keyword argument 'axis'

Dataset is loaded with vaex, is this a problem particular for vaex?