對于一個研究專案，我正在使用各種機器學習演算法分析相關性。因此，我運行以下代碼（為演示而簡化）：
# Make a custom scorer for pearson's r (from scipy)
scorer = lambda regressor, X, y: pearsonr(regressor.predict(X), y)[0]
# Create a progress bar
progress_bar = tqdm(14400)
# Initialize a dataframe to store scores
df = pd.DataFrame(columns=["data", "pipeline", "r"])
# Loop over datasets
for data in datasets: #288 datasets
X_train = data.X_train
X_test = data.X_test
y_train = data.y_train
y_test = data.y_test
# Loop over pipelines
for pipeline in pipelines: #50 pipelines
scores = cross_val_score(pipeline, X_train, y_train, cv=int(len(X_train)/3), scoring=scorer)
r = scores.mean()
# Create a new row to save data
df.loc[(df.last_valid_index() or 0) 1] = {"data": data.name, "pipeline": pipeline, "r": r}
progress_bar.update(1)
progress_bar.close()
X_train 是一個形狀為 (20, 34) 的 pandas 資料框
X_test 是一個形狀為 (9, 34) 的 pandas 資料框
y_train 是長度為 20 的熊貓系列
y_test 是一個長度為 9 的 pandas 系列
管道的一個例子是：
Pipeline(steps=[('scaler', StandardScaler()),
('poly', PolynomialFeatures(degree=9)),
('regressor', LinearRegression())])
但是，經過大約 8700 次迭代（總共），我得到以下 MemoryError：
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-9ff48105b8ff> in <module>
40 y = targets[label]
41 #Finally, we can test the correlation
---> 42 scores = cross_val_score(regressor, X_train, y.loc[train_indices], cv=int(len(X_train)/3), scoring=lambda regressor, X, y: pearsonr(regressor.predict(X), y)[0]) #Three samples per test set, as that seems like the logical minimum for Pearson
43 r = scores.mean()
44 # print(f"{regressor} was able to predict {label} based on the {band} band of the {network} network with a Pearson's r of {r} of the data that could be explained.\n")
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in cross_val_score(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, error_score)
513 scorer = check_scoring(estimator, scoring=scoring)
514
--> 515 cv_results = cross_validate(
516 estimator=estimator,
517 X=X,
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in cross_validate(estimator, X, y, groups, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch, return_train_score, return_estimator, error_score)
283 )
284
--> 285 _warn_or_raise_about_fit_failures(results, error_score)
286
287 # For callabe scoring, the return type is only know after calling. If the
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py in _warn_or_raise_about_fit_failures(results, error_score)
365 f"Below are more details about the failures:\n{fit_errors_summary}"
366 )
--> 367 raise ValueError(all_fits_failed_message)
368
369 else:
ValueError:
All the 6 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.
Below are more details about the failures:
--------------------------------------------------------------------------------
2 fits failed with the following error:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 686, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\pipeline.py", line 382, in fit
self._final_estimator.fit(Xt, y, **fit_params_last_step)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\_base.py", line 692, in fit
X, y, X_offset, y_offset, X_scale = _preprocess_data(
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\_base.py", line 262, in _preprocess_data
X = check_array(X, copy=copy, accept_sparse=["csr", "csc"], dtype=FLOAT_DTYPES)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 925, in check_array
array = np.array(array, dtype=dtype, order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 41.8 GiB for an array with shape (16, 350343565) and data type float64
--------------------------------------------------------------------------------
4 fits failed with the following error:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\model_selection\_validation.py", line 686, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\pipeline.py", line 382, in fit
self._final_estimator.fit(Xt, y, **fit_params_last_step)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\_base.py", line 692, in fit
X, y, X_offset, y_offset, X_scale = _preprocess_data(
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\_base.py", line 262, in _preprocess_data
X = check_array(X, copy=copy, accept_sparse=["csr", "csc"], dtype=FLOAT_DTYPES)
File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 925, in check_array
array = np.array(array, dtype=dtype, order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 44.4 GiB for an array with shape (17, 350343565) and data type float64
我能做些什么來防止這個錯誤，它是如何產生的？我嘗試在我記憶中的管道上使用 sklearn 的克隆函式，然后呼叫 fit，但我得到了同樣的錯誤。但是，當我創建一個新管道（仍在同一個會話中）并對其呼叫 fit 時，它確實有效。
uj5u.com熱心網友回復：
問題是你正在做的巨大的基礎擴展。為 34 個特征添加 9 次多項式特征會產生 52,451,256 個特征。即使您只有少量樣本，也難怪您的記憶體不足。
只需看看第二學位PolynomialFeatures為您提供的 4 個功能：
>>> import numpy as np
>>> from sklearn.preprocessing import PolynomialFeatures
>>> from sklearn.pipeline import make_pipeline
>>> arr = np.random.random(size=(10, 4))
>>> poly = PolynomialFeatures(degree=2).fit(arr)
>>> poly.get_feature_names()
這導致：
['1',
'x0',
'x1',
'x2',
'x3',
'x0^2',
'x0 x1',
'x0 x2',
'x0 x3',
'x1^2',
'x1 x2',
'x1 x3',
'x2^2',
'x2 x3',
'x3^2']
如果您在 20 個資料實體上使用 52 個特征，您很可能會進入過度擬合領域。即使是資料上的 2 次多項式也會為您提供 630 個特征，這實在是太多了。我會使用檢查（例如配對圖）、特征重要性，也許還有 PCA 來降低維度，然后放棄基礎擴展，直到你知道事情的發展方向。
uj5u.com熱心網友回復：
MemoryError 意味著 Python 解釋器用完了 RAM 和交換空間來分配新的記憶體。通常解決方案包括 1) 使用較小的資料集 2) 獲得具有更多 RAM 的計算機。3）檢查您的代碼不會泄漏記憶體。
轉載請註明出處，本文鏈接：https://www.uj5u.com/houduan/487714.html