我sklearn.naive_bayes.MultinomialNB在大資料上用于命名物體識別時遇到了 MemoryError ，其中train.shape = (416330, 97896).
data_train = pd.read_csv(path[0] "train_SENTENCED.tsv", encoding="utf-8", sep='\t', quoting=csv.QUOTE_NONE)
data_test = pd.read_csv(path[0] "test_SENTENCED.tsv", encoding="utf-8", sep='\t', quoting=csv.QUOTE_NONE)
print('TRAIN_DATA:\n', data_train.tail(5))
# FIT TRANSFORM
X_TRAIN = data_train.drop('Tag', axis=1)
X_TEST = data_test.drop('Tag', axis=1)
v = DictVectorizer(sparse=False)
X_train = v.fit_transform(X_TRAIN.to_dict('records'))
X_test = v.transform(X_TEST.to_dict('records'))
y_train = data_train.Tag.values
y_test = data_test.Tag.values
classes = np.unique(y_train)
classes = classes.tolist()
nb = MultinomialNB(alpha=0.01)
nb.partial_fit(X_train, y_train, classes)
new_classes = classes.copy()
new_classes.pop()
predictions = nb.predict(X_test)
錯誤輸出如下：
Traceback (most recent call last):
File "naive_bayes_classifier/main.py", line 107, in <module>
X_train = v.fit_transform(X_TRAIN.to_dict('records'))
File "lib/python3.8/site-packages/sklearn/feature_extraction/_dict_vectorizer.py", line 313, in fit_transform
return self._transform(X, fitting=True)
File "lib/python3.8/site-packages/sklearn/feature_extraction/_dict_vectorizer.py", line 282, in _transform
result_matrix = result_matrix.toarray()
File "lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1031, in toarray
out = self._process_toarray_args(order, out)
File "lib/python3.8/site-packages/scipy/sparse/base.py", line 1202, in _process_toarray_args
return np.zeros(self.shape, dtype=self.dtype, order=order)
numpy.core._exceptions.MemoryError: Unable to allocate 304. GiB for an array with shape (416330, 97896) and data type float64
雖然，我將這樣的引數設定為DictVectorizer(sparse=False, dtype=np.short)，代碼然后為nb.partial(X_train, y_train, classes)行回傳錯誤。
如何防止此記憶體錯誤？有沒有合適的方法解決？我想過拆分訓練集，但是由于向量適合相應的資料集，這會是正確的解決方案嗎？
uj5u.com熱心網友回復：
可悲的是，問題是使用 sklearn 解決這個問題確實需要大量記憶體，因為它很復雜。
.partial_fit()是一個很好的方法，但我建議將您的資料分成更小的部分，然后在它們上部分擬合分類器。嘗試將您的資料集拆分成小塊，看看是否有效。如果您仍然遇到相同的錯誤，請嘗試更小的位
轉載請註明出處，本文鏈接：https://www.uj5u.com/shujuku/391879.html
上一篇：在Pandas中堆疊列