pandas、spark計算相關性系數速度對比

相關性計算有三種演算法：pearson、spearman，kenall，

在pandas庫中，對一個Dataframe，可以直接計算這三個演算法的相關系數correlation，方法為：data.corr()

底層是依賴scipy庫的演算法，

為了提升計算速度，使用spark平臺來加速執行，

比較了pandas，spark并發scipy演算法，spark mllib庫的計算速度，

總體來說，spark mllib速度最快，其次是spark并發，pandas速度最慢，

corr執行速度測驗結果

時間單位：秒

說明：spearmanr 演算法在spark scipy組合下執行速度較慢，需要再對比分析，感覺存在問題的，

三種演算法腳本如下：

pandas 腳本

import numpy as np
import pandas as pd
import time

C = 1000
N = 3600

data = https://www.cnblogs.com/StitchSun/p/pd.DataFrame(np.random.randn(C * N).reshape(C, -1))


print("============================ {}".format(data.shape))
print("start pandas corr ---{} ".format(time.time()))
start = time.time()
# {'pearson', 'kendall', 'spearman'}
res = data.corr(method='pearson')
end_1 = time.time()

res = data.corr(method='spearman')
end_2 = time.time()

res = data.corr(method='kendall')
end_3 = time.time()

print("pandas pearson count {} total cost : {}".format(len(res), end_1 - start))
print("pandas spearman count {} total cost : {}".format(len(res), end_2 - end_1))
print("pandas kendall count {} total cost : {}".format(len(res), end_3 - end_2))

spark scipy腳本

from pyspark import SparkContext
sc = SparkContext()
import numpy as np
import pandas as pd
from scipy import stats as st
import time

# t1 = st.kendalltau(x, y)
# t2 = st.spearmanr(x, y)
# t3 = st.pearsonr(x, y)

C = 1000
N = 3600

data = https://www.cnblogs.com/StitchSun/p/pd.DataFrame(np.random.randn(C * N).reshape(C, -1))


def pearsonr(n):
    x = data.iloc[:, n]
    res = [st.pearsonr(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


def spearmanr(n):
    x = data.iloc[:, n]
    res = [st.spearmanr(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


def kendalltau(n):
    x = data.iloc[:, n]
    res = [st.kendalltau(x, data.iloc[:, i])[0] for i in range(data.shape[1])]
    return res


start = time.time()
res = sc.parallelize(np.arange(N)).map(lambda x: pearsonr(x)).collect()
# res = sc.parallelize(np.arange(N)).map(lambda x: spearmanr(x)).collect()
# res = sc.parallelize(np.arange(N)).map(lambda x: kendalltau(x)).collect()
end = time.time()

print("pearsonr count {} total cost : {}".format(len(res), end - start))
print("spearmanr count {} total cost : {}".format(len(res), end - start))
print("kendalltau count {} total cost : {}".format(len(res), end - start))


# 純python演算法
s = time.time()
res = [st.spearmanr(data.iloc[:, i], data.iloc[:, j])[0] for i in range(N) for j in range(N)]
end = time.time()
print(end-s)

start = time.time()
dd = sc.parallelize(res).map(lambda x: st.spearmanr(data.iloc[:, x[0]], data.iloc[:, x[1]])).collect()
end = time.time()
print(end-start)

start = time.time()
dd = sc.parallelize(res).map(lambda x: st.kendalltau(data.iloc[:, x[0]], data.iloc[:, x[1]])).collect()
end = time.time()
print(end-start)

spark mllib腳本

from pyspark import SparkContext
sc = SparkContext()
from pyspark.mllib.stat import Statistics
import time
import numpy as np

L = 1000
N = 3600
t = [np.random.randn(N) for i in range(L)]

data = https://www.cnblogs.com/StitchSun/p/sc.parallelize(t)

start = time.time()
res = Statistics.corr(data, method="pearson")  # spearman  pearson
end = time.time()
print("pearson : ", end-start)


start = time.time()
res = Statistics.corr(data, method="spearman")  # spearman  pearson
end = time.time()
print("spearman: ", end-start)

標籤：Python

上一篇：02爬取豆瓣最受歡迎的250部電影

下一篇：反編譯微信小程式