我一直試圖從一些維基百科轉儲中提取文本。我需要從完整歷史英文維基百科轉儲中的 id、title、ns、timestamp、username、ip 和 text 標簽中獲取文本。
我從https://www.heatonresearch.com/2017/03/03/python-basic-wikipedia-parsing.html閱讀并修改了代碼。
我能夠撰寫以下代碼：
import xml.etree.ElementTree as etree
import codecs
import csv
import time
import os
PATH_WIKI_XML = '/home/wikipedia'
FILENAME_WIKI = 'enwiki-latest-pages-meta-history1.xml-p24706p25444'
FILENAME_ARTICLES = 'articles.csv'
FILENAME_TEMPLATE = 'articles_template.csv'
ENCODING = "utf-8"
def hms_string(sec_elapsed):
h = int(sec_elapsed / (60 * 60))
m = int((sec_elapsed % (60 * 60)) / 60)
s = sec_elapsed % 60
return "{}:{:>02}:{:>05.2f}".format(h, m, s)
def strip_tag_name(t):
t = elem.tag
idx = k = t.rfind("}")
if idx != -1:
t = t[idx 1:]
return t
pathWikiXML = os.path.join(PATH_WIKI_XML, FILENAME_WIKI)
pathArticles = os.path.join(PATH_WIKI_XML, FILENAME_ARTICLES)
pathTemplateRedirect = os.path.join(PATH_WIKI_XML, FILENAME_TEMPLATE)
totalCount = 0
articleCount = 0
templateCount = 0
title = None
timestamp= None
username= None
ip= None
text=None
start_time = time.time()
with codecs.open(pathArticles, "w", ENCODING) as articlesFH, \
codecs.open(pathTemplateRedirect, "w", ENCODING) as templateFH:
articlesWriter = csv.writer(articlesFH, quoting=csv.QUOTE_MINIMAL)
templateWriter = csv.writer(templateFH, quoting=csv.QUOTE_MINIMAL)
articlesWriter.writerow(['id', 'title', 'timestamp','username','ip','text'])
templateWriter.writerow(['id', 'title'])
for event, elem in etree.iterparse(pathWikiXML, events=('start', 'end')):
tname = strip_tag_name(elem.tag)
if event == 'start':
if tname == 'page':
title = ''
id = -1
inrevision = False
incontributor= False
ns = 0
elif tname == 'revision':
# Do not pick up on revision id's
inrevision = True
elif tname == 'contributor':
incontributor = True
else:
if tname == 'title':
title = elem.text
elif tname == 'id' and not inrevision and not incontributor:
id = int(elem.text)
elif tname == 'ns':
ns = int(elem.text)
elif tname == 'timestamp':
timestamp = int(elem.text)
elif tname == 'username':
username = elem.text
elif tname == 'ip':
ip = elem.text
elif tname == 'text':
text = elem.text
elif tname == 'page':
totalCount = 1
if ns == 10:
templateCount = 1
templateWriter.writerow([id, title])
elif len(title) > 0:
articleCount = 1
articlesWriter.writerow(['id', 'title', 'timestamp','username','ip','text'])
# if totalCount > 100000:
# break
if totalCount > 1 and (totalCount % 100000) == 0:
print("{:,}".format(totalCount))
elem.clear()
elapsed_time = time.time() - start_time
print("Total pages: {:,}".format(totalCount))
print("Template pages: {:,}".format(templateCount))
print("Article pages: {:,}".format(articleCount))
print("Elapsed time: {}".format(hms_string(elapsed_time)))
但是，生成的 csv 檔案只有 id 和標題。其他列是空的，我猜是因為時間戳標記嵌套在修訂標記中，但我的事件僅處理頁面標記的開始和結束。用戶名和 IP 也嵌套在貢獻者標簽中，如下面的示例 XML 檔案所示。有人可以告訴我如何解決這個問題嗎？我應該在事件中回圈事件以從所需的嵌套標簽中提取文本嗎？有沒有可以在這里為我作業的代碼？所需的輸出是具有所需標簽作為標題和行中的文本的檔案。請記住，一個頁面可以有多個修訂版本，并且每個修訂版本都有不同的元資料。我想從所需標簽中獲取所有文本以及頁面內的元資料，在我擁有該頁面中的所有內容后，繼續下一頁。謝謝。
<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.10/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.10/
http://www.mediawiki.org/xml/export-0.10.xsd" version="0.10" xml:lang="sco">
<siteinfo>
<sitename>Wikipedia</sitename>
<dbname>scowiki</dbname>
<base>http://sco.wikipedia.org/wiki/Main_Page</base>
<generator>MediaWiki 1.25wmf12</generator>
<case>first-letter</case>
<namespaces>
<namespace key="-2" case="first-letter">Media</namespace>
<namespace key="-1" case="first-letter">Special</namespace>
<namespace key="0" case="first-letter" />
<namespace key="1" case="first-letter">Talk</namespace>
<namespace key="2" case="first-letter">User</namespace>
<namespace key="3" case="first-letter">User talk</namespace>
<namespace key="4" case="first-letter">Wikipedia</namespace>
<namespace key="5" case="first-letter">Wikipedia talk</namespace>
<namespace key="6" case="first-letter">File</namespace>
<namespace key="7" case="first-letter">File talk</namespace>
<namespace key="8" case="first-letter">MediaWiki</namespace>
<namespace key="9" case="first-letter">MediaWiki talk</namespace>
<namespace key="10" case="first-letter">Template</namespace>
<namespace key="11" case="first-letter">Template talk</namespace>
<namespace key="12" case="first-letter">Help</namespace>
<namespace key="13" case="first-letter">Help talk</namespace>
<namespace key="14" case="first-letter">Category</namespace>
<namespace key="15" case="first-letter">Category talk</namespace>
<namespace key="100" case="first-letter">Portal</namespace>
<namespace key="101" case="first-letter">Portal talk</namespace>
<namespace key="828" case="first-letter">Module</namespace>
<namespace key="829" case="first-letter">Module talk</namespace>
</namespaces>
</siteinfo>
<page>
<title>Inglis leid</title>
<ns>0</ns>
<id>2</id>
<revision>
<id>7</id>
<timestamp>2005-06-22T10:17:05Z</timestamp>
<contributor>
<ip>24.251.198.251</ip>
</contributor>
<model>wikitext</model>
<format>text/x-wiki</format>
<text xml:space="preserve">Tha '''Inglis''' (English) leid is a west [[Gairmanic leid]] at cam frae Ingland an thats forebear wis [[auld Inglis]]. Tha name "English" cams frae tha pairt o [[Gairmanie]] caw'd "Angeln". Inglis is tha waruld's seicont maist widelie spaken first leid, an his aboot 340 million hameborn speikers waruldwide.
[[en:English language]]</text>
<sha1>6m5yxiaalrm6te7e3x3fiw1aq7wk9ir</sha1>
</revision>
</page>
</mediawiki>
uj5u.com熱心網友回復：
將嘗試的腳本簡化到您需要的最低限度，例如洗掉計時。這里的程序iterparse通常用于非常大的 XML 檔案，以逐個標簽迭代地決議標簽位于檔案中的任何位置，例如根、父、子、后代等。
因此，逐個標簽清理邏輯標簽，然后在最后需要的標簽上，使用當前分配的變數將行寫入 csv，這些變數在每個<page>標簽上重置。
pathWikiXML = "Input.xml"
pathWikiCSV = "Output.csv"
def strip_tag_name(t):
return t.split("}")[1] if "}" in t else t
with codecs.open(pathWikiCSV, "w", "utf-8") as f:
cw = csv.writer(f)
cw.writerow(['id', 'title', 'timestamp','username','ip','text'])
for event, elem in etree.iterparse(pathWikiXML, events=('start', 'end')):
tname = strip_tag_name(elem.tag)
if event == 'start':
if tname == 'page':
title = None
timestamp = None
username = None
ip = None
text = None
elif tname == 'title':
title = elem.text
elif tname == 'id':
id = int(elem.text)
elif tname == 'ns':
ns = int(elem.text)
elif tname == 'timestamp':
timestamp = elem.text
elif tname == 'username':
username = elem.text
elif tname == 'ip':
ip = elem.text
elif tname == 'text':
text = elem.text
cw.writerow([id, title, timestamp, username, ip, text])
CSV 輸出
| ID | 標題 | 時間戳 | 用戶名 | ip | 文本 |
|---|---|---|---|---|---|
| 7 | 英格利斯雷德 | 2005-06-22T10:17:05Z | 24.251.198.251 | “Tha '''Inglis'''（英文）... |
uj5u.com熱心網友回復：
您發布的示例代碼有點復雜。就像@Parfait 推薦的那樣，嘗試減少問題，并獲得一些基本的作業，并對其作業原理有一個基本的了解。
我認為iterparsewith events 的方法非常復雜，而且一開始就令人困惑。
還有另一種在 XML 中搜索元素的方法。它稱為XPath，它允許您按關系搜索：“在此元素下找到此元素”。一旦你有了一個element，你所要做的就是呼叫它的text屬性；沒有剝離命名空間。哦！不過，您確實需要考慮命名空間，但這很容易。
#!/usr/bin/env python3
import csv
import xml.etree.ElementTree as ET
# Set up the namespace that needs to be a part of every XPath query
ns_dict = {'xmlns': 'http://www.mediawiki.org/xml/export-0.10/'}
# Open a file and parse it
root = ET.parse('sample.xml')
# Get the page element, as a Python object; you must use 'xmlns' (with a colon) and
# pass `ns_dict`, from earlier, `.//xmlns` means "find, from here (at root), the page element at any depth (with the predefined xmlns namespace)"
page = root.find('.//xmlns:page', ns_dict)
# With the page "element", find its direct children (always using `xmlns:` and passing `ns_dict`)
# `./xmlns:<element>` means "find, from here (at page), the element that's only 1 level (directly) under page"
pg_title = page.find('./xmlns:title', ns_dict)
pg_ns = page.find('./xmlns:ns', ns_dict)
pg_id = page.find('./xmlns:id', ns_dict)
# Get the revision element
revision = page.find('./xmlns:revision', ns_dict)
rev_id = revision.find('./xmlns:id', ns_dict)
rev_ts = revision.find('./xmlns:timestamp', ns_dict)
# Find ip under contributor
contrib_ip = revision.find('./xmlns:contributor/xmlns:ip', ns_dict)
print('page title:', pg_title.text)
print('page id:', pg_id.text)
print('rev id:', rev_id.text)
print('rev timestamp:', rev_ts.text)
print('contributor ip:', contrib_ip.text)
# From here, write out to a CSV
with open('out.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerow(['Page title', 'Page id', 'Rev id', 'Rev timestamp', 'Contributor ip'])
writer.writerow([pg_title.text, pg_id.text, rev_id.text, rev_ts.text, contrib_ip.text])
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/378875.html