Beautifulsoup抓取.text并自動拆分它們

這里是一個完全的 BeautifulSoup 新手。我需要從某個 URL 中抓取發布選項卡的內容（在下面的代碼示例中列出）。我需要抓取出版物并將它們拆分為“作者”、“標題”和“期刊”，然后我可以將它們轉換為 pandas DataFrame。我嘗試使用以下代碼抓取內容：

page_url = 'https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory'

def get_soup(url):
    r = requests.get('http://localhost:8050/render.html', params={'url': url, 'wait': 3})
    soup = BeautifulSoup(r.text, 'html.parser')

url_soup = get_soup(page_url)

publis = url_soup.find_all('div',{'class':'tab-content pad'})

for item in publis:
    publi = {
        'items': item.find('div',{'id':'tabs-publications'}).text.split('\n\n')
    }

publication_df = pd.DataFrame(publi)
new_publication_df = publication_df['items'].str.split(':', 1, expand=True)
new_publication_df[[1,2]] = new_publication_df[1].str.split(']',expand=True)

這會回傳我需要的內容，但它確實很容易出現拼寫錯誤（例如，有一份出版物使用“J”而不是“]”）。BeautifulSoup有什么辦法可以自動將文本分成三列？

uj5u.com熱心網友回復：

認為您應該使用可用的結構而不是拆分字串。

選擇所有<em>包含標題的，它的previous_sibling，包含作者的，還有它的next_sibling，包含期刊的。

例子

import requests
from bs4 import BeautifulSoup


r = requests.get('https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory')
soup = BeautifulSoup(r.text)
data = []
for e in soup.select('#tabs-publications em'):
    data.append({
        'author':e.previous.get_text(strip=True)[:-1],
        'title':e.get_text(strip=True),
        'journal':e.next_sibling.get_text(strip=True)
    })

data

輸出資料

[{'author': 'Hantsoo Liisa, Kornfield Sara, Anguera Montserrat C, Epperson C Neill',
  'title': 'Inflammation: A Proposed Intermediary Between\xa0Maternal Stress and Offspring Neuropsychiatric Risk. [PMID30314641]',
  'journal': 'Biological psychiatry 85(2): 97-106, Jan 2019.'},
 {'author': 'Sierra Isabel, Anguera Montserrat C',
  'title': 'Enjoy the silence: X-chromosome inactivation diversity in somatic cells.[PMID31108425]',
  'journal': 'Current opinion in genetics & development 55: 26-31, May 2019.'},
 {'author': 'Syrett Camille M, Anguera Montserrat C',
  'title': 'When the balance is broken: X-linked gene dosage from two X chromosomes and female-biased autoimmunity. [PMID31125996]',
  'journal': 'Journal of leukocyte biology May 2019.'},
 {'author': 'Kotzin Jonathan J, Iseka Fany, Wright Jasmine, Basavappa Megha G, Clark Megan L, Ali Mohammed-Alkhatim, Abdel-Hakeem Mohamed S, Robertson Tanner F, Mowel Walter K, Joannas Leonel, Neal Vanessa D, Spencer Sean P, Syrett Camille M, Anguera Montserrat C, Williams Adam, Wherry E John, Henao-Mejia Jorge',
  'title': 'The long noncoding RNA regulates CD8 T cells in response to viral infection.[PMID31138702]',
  'journal': 'Proceedings of the National Academy of Sciences of the United States of America 116(24): 11916-11925, Jun 2019.'},
 {'author': 'Syrett Camille M, Paneru Bam, Sandoval-Heglund Donavon, Wang Jianle, Banerjee Sarmistha, Sindhava Vishal, Behrens Edward M, Atchison Michael, Anguera Montserrat C',
  'title': 'Altered X-chromosome inactivation in T cells may promote sex-biased autoimmune diseases. [PMID30944248',
  'journal': 'JCI insight 4(7), Apr 2019.'},...]

輸出資料框

pd.DataFrame(data)

...

標籤：Python 熊猫 数据框 网页抓取 美丽的汤

上一篇：在Excel的電源查詢中從網頁獲取href

下一篇：使用python從GitHub爬取和下載Readme.md檔案