我有一個如下示例中所示的文本檔案，我想將其轉換為 csv 檔案（當前使用 Pandas）。
挑戰在于我事先不知道鍵（列標題是什么）以及它們的順序。
最后的列順序并不重要。
示例檔案：
name: john| dob: 10-06-1960| address: 4853 Radio Park Drive
name: jane| dob: 07-10-1973| address: 1537 Timbercrest Road| mobile: 706-289-6746
name: liam| dob: 12-08-1986| address: 4853 498 Fairmont Avenue| telephone number: 706-687-5021
name: chris| dob: 09-12-1965| address: 485 Green Avenue| state: California| Telephone Number: 510-855-5213
期望的輸出：
Name | dob | address | mobile | telephone number | state |
----- ------------ -------------------------- -------------- ------------------ ------------
john | 10-06-1960 | 4853 Radio Park Drive | | | |
jane | 07-10-1973 | 1537 Timbercrest Road | 706-289-6746 | | |
liam | 12-08-1986 | 4853 498 Fairmont Avenue | | 706-687-5021 | |
chris| 09-12-1965 | 485 Green Avenue | | 510-855-5213 | California |
我的代碼：
import pandas as pd
df = pd.DataFrame()
file = open('D:\sample.log', 'r')
lines = file.readlines()
for line in lines:
pairs = line.split('|')
my_dict = {}
for pair in pairs:
key = pair.split(': ')[0].strip()
value = pair.split(': ')[1].strip()
my_dict[key] = value
df.append(my_dict, ignore_index=True)
這種追加方式非常慢。我怎樣才能讓它更快。
還是有更好的解決方案（例如通過 json 字串）？
uj5u.com熱心網友回復：
TL;博士：
pd.DataFrame.from_records(
dict(field.split(': ') for field in line.split('|'))
for line in lines
)
長版
假設您已經將資料拆分為行，那么您需要將它們處理成記錄，例如：
{' address': '4853 Radio Park Drive', ' dob': '10-06-1960', 'name': 'john'}
每行需要拆分為欄位：
>>> line = 'name: john| dob: 10-06-1960| address: 4853 Radio Park Drive'
>>> line.split('|')
['name: john', ' dob: 10-06-1960', ' address: 4853 Radio Park Drive']
然后需要將每個欄位拆分為列名和值本身：
>>> field = 'name: John'
>>> field.split(': ')
['name', 'john']
對行中的每個欄位執行此操作后，您最終會得到以下串列：
>>> [field.split(': ') for field in line.split('|')]
[['name', 'john'],
[' dob', '10-06-1960'],
[' address', '4853 Radio Park Drive']]
使用此串列初始化的字典會從答案的開頭為您提供記錄。
由于您有很多行，因此您需要生成許多記錄，但最好是懶惰地生成這些記錄，換句話說，使用生成器：
>>> (dict(field.split(': ') for field in line.split('|')) for line in s.split('\n'))
<generator object <genexpr> at 0x7f0d06bf8dd0>
Rather than producing you a whole list of records, the generator gives you one at a time when you iterate over it. This way you can start forming your dataframe without having to wait for all the records to be processed.
There is a special syntax in Python called generator comprehension that let's you define generators to be passed as an argument to functions and constructors.
Putting it all together, we construct a dataframe using the appropriate constructor (from_records) and the generator defined above:
pd.DataFrame.from_records(
dict(field.split(': ') for field in line.split('|'))
for line in lines
)
This produces the following output:
name dob address mobile telephone number state Telephone Number
0 john 10-06-1960 4853 Radio Park Drive NaN NaN NaN NaN
1 jane 07-10-1973 1537 Timbercrest Road 706-289-6746 NaN NaN NaN
2 liam 12-08-1986 4853 498 Fairmont Avenue NaN 706-687-5021 NaN NaN
3 chris 09-12-1965 485 Green Avenue NaN NaN California 510-855-5213
As a bonus, you can speed this up further by reading the file lazily too. Define a custom generator for reading lines:
def lines(path):
with open(path) as file:
while line := file.readline():
yield line.rstrip()
請注意，這只適用于 Python 3.8 。否則，您需要這樣做，而不是使用海象運算子：
def lines(path):
with open(path) as file:
while True:
line = file.readline()
if line:
yield line.rstrip()
else:
return
uj5u.com熱心網友回復：
我認為這是完成這項任務的最快方法之一，因為它利用了 pandas 庫的內置多處理（用 c/c 撰寫），它比遍歷行要快得多。
首先將整個文本讀入一個變數。接著，
import pandas as pd
data = '''name: john| dob: 10-06-1960| address: 4853 Radio Park Drive
name: jane| dob: 07-10-1973| address: 1537 Timbercrest Road| mobile: 706-289-6746
name: liam| dob: 12-08-1986| address: 4853 498 Fairmont Avenue| telephone number: 706-687-5021
name: chris| dob: 09-12-1965| address: 485 Green Avenue| state: California| Telephone Number: 510-855-5213'''
def get_dict(line_elems):
line_dict = {}
for elem in line_elems:
k, v = elem.split(':')
line_dict[k]=v
return line_dict
df = pd.DataFrame({'lines':data.split('\n')})
df['line_list'] = df['lines'].apply(lambda x:x.split('|'))
df['line_dict'] = df['line_list'].apply(get_dict)
dict_list = df['line_dict'].tolist()
final_df = pd.DataFrame.from_dict(dict_list)
final_df
如果你愿意，我可以解釋代碼，但讓我知道它與其他代碼相比的性能。
要從檔案中讀取整個文本，您可以使用
with open('my_text_file',mode='r') as file:
data= file.read()
注意：
- 如果您想要快速的性能，我認為您應該避免迭代線路，無論您使用哪種方法。如果你在 python 方面足夠先進，你可以嘗試使用 Python Multiprocessing。但是我上面的代碼被剪掉了，避免了所有這些麻煩。
轉載請註明出處，本文鏈接：https://www.uj5u.com/yidong/425822.html