NLTK自然語言處理庫-有解無憂

　　自然語言處理，通常簡稱為NLP，是人工智能的一個分支，處理使用自然語言的計算機與人之間的互動，NLP的最終目標是以有價值的方式閱讀，解讀，理解和理解人類語言，大多數NLP技術都依靠機器學習來從人類語言中獲取含義，

Siri的作業流程：

聽
懂
思考
組織語言
回答

其中分別涉及的技術：

語音識別
自然語言處理 - 語意分析
邏輯分析 - 結合業務場景與背景關系
自然語言處理 - 分析結果生成自然語言文本
語音合成

自然語言處理的常用處理程序：

　　先針對訓練文本進行分詞處理（詞干提取、原型提取），統計詞頻，通過詞頻-逆檔案頻率演算法獲得該詞對樣本語意的貢獻，根據每個詞的貢獻力度，構建有監督分類模型，把測驗樣本交給模型處理，得到測驗樣本的語意類別，

本篇文章主要通過講解NLTK(語言語言處理庫)——來給大家講解自然語言處理NLP

nltk.download()下載資料集

jieba中文分詞庫

文本分詞

分詞處理相關API：

import nltk.tokenize as tk

sent_list = tk.sent_tokenize(text)          # 把樣本按句子進行拆分  sent_list:句子串列
word_list = tk.word_tokenize(text)          # 把樣本按單詞進行拆分  word_list:單詞串列

# 把樣本按單詞進行拆分 punctTokenizer：分詞器物件
punctTokenizer = tk.WordPunctTokenizer()    
word_list = punctTokenizer.tokenize(text)

案例：

import nltk.tokenize as tk

doc = "Are you curious about tokenization? " \
      "Let's see how it works! " \
      "We need to analyze a couple of sentences " \
      "with punctuations to see it in action."
print(doc)
tokens = tk.sent_tokenize(doc)  # 句子分詞
for i, token in enumerate(tokens):
    print("%2d" % (i + 1), token)
# 1 Are you curious about tokenization?
# 2 Let's see how it works!
# 3 We need to analyze a couple of sentences with punctuations to see it in action.

tokens = tk.word_tokenize(doc)  # 單詞分詞
for i, token in enumerate(tokens):
    print("%2d" % (i + 1), token)
# 1 Are
# 2 you
# 3 curious
# 4 about
# ...
# 28 action
# 29 .
tokenizer = tk.WordPunctTokenizer()  # 單詞分詞
tokens = tokenizer.tokenize(doc)
for i, token in enumerate(tokens):
    print("%2d" % (i + 1), token)
# 1 Are
# 2 you
# 3 curious
# ...
# 27 it
# 28 in
# 29 action
# 30 .

詞干提取

文本樣本中的單詞的詞性與時態對于語意分析并無太大影響，所以需要對單詞進行 詞干提取，

詞干提取相關API：

import nltk.stem.porter as pt
import nltk.stem.lancaster as lc
import nltk.stem.snowball as sb

stemmer = pt.PorterStemmer() # 波特詞干提取器，偏寬松
stemmer = lc.LancasterStemmer() # 朗卡斯特詞干提取器，偏嚴格

# 思諾博詞干提取器，偏中庸
stemmer = sb.SnowballStemmer('english') 
r = stemmer.stem('playing')     # 提取單詞playing的詞干

案例：

import nltk.stem.porter as pt
import nltk.stem.lancaster as lc
import nltk.stem.snowball as sb

words = ['table', 'probably', 'wolves', 'playing', 'is',
         'dog', 'the', 'beaches', 'grounded', 'dreamt', 'envision']
pt_stemmer = pt.PorterStemmer()         # 波特詞干提取器，偏寬松
lc_stemmer = lc.LancasterStemmer()      # 朗卡斯特詞干提取器，偏嚴格
sb_stemmer = sb.SnowballStemmer('english')  # 思諾博詞干提取器，偏中庸
for word in words:
    pt_stem = pt_stemmer.stem(word)
    lc_stem = lc_stemmer.stem(word)
    sb_stem = sb_stemmer.stem(word)
    print('%8s %8s %8s %8s' % (word, pt_stem, lc_stem, sb_stem))
#    table     tabl     tabl     tabl
# probably  probabl     prob  probabl
#   wolves     wolv     wolv     wolv
#  playing     play     play     play
#       is       is       is       is
#      dog      dog      dog      dog
#      the      the      the      the
#  beaches    beach    beach    beach
# grounded   ground   ground   ground
#   dreamt   dreamt   dreamt   dreamt
# envision    envis    envid    envis

詞性還原

　　詞性還原與詞干提取的作用類似，詞性還原更利于人工二次處理，因為有些詞干并非正確的單詞，人工閱讀更麻煩，詞性還原可以把名詞復數形式恢復為單數形式，動詞分詞形式恢復為原型形式，

詞性還原相關API：

import nltk.stem as ns
# 獲取詞性還原器物件
lemmatizer = ns.WordNetLemmatizer()
?
n_lemma = lemmatizer.lemmatize(word, pos='n')   # 把單詞word按照名詞進行還原
v_lemma = lemmatizer.lemmatize(word, pos='v')   # 把單詞word按照動詞進行還原

案例：

import nltk.stem as ns
words = ['table', 'probably', 'wolves', 'playing',
         'is', 'dog', 'the', 'beaches', 'grounded',
         'dreamt', 'envision']
lemmatizer = ns.WordNetLemmatizer()
for word in words:
    n_lemma = lemmatizer.lemmatize(word, pos='n')   # 名詞 詞性還原
    v_lemma = lemmatizer.lemmatize(word, pos='v')   # 動詞 詞性還原
    print('%8s %8s %8s' % (word, n_lemma, v_lemma))
#    table    table    table
# probably probably probably
#   wolves     wolf   wolves
#  playing  playing     play
#       is       is       be
#      dog      dog      dog
#      the      the      the
#  beaches    beach    beach
# grounded grounded   ground
#   dreamt   dreamt    dream
# envision envision envision

詞袋模型

　　一句話的語意很大程度取決于某個單詞出現的次數，詞袋模型以每一個句子作為一個樣本，用特征名和特證值構建的數學模型稱為“詞袋模型”

特證名：句子中所有可能出現的單詞
特證值：單詞在句子中出現的次數

The brown dog is running. The black dog is in the black room. Running in the room is forbidden.

1 The brown dog is running

2 The black dog is in the black room

3 Running in the room is forbidden

the	brown	dog	is	running	black	in	room	forbidden
1	1	1	1	1	0	0	0	0
2	0	1	1	0	2	1	1	0
1	0	0	1	1	0	1	1	1

詞袋模型化相關API：

import sklearn.feature_extraction.text as ft

cv = ft.CountVectorizer()           # 構建詞袋模型

bow = cv.fit_transform(sentences)   # 訓練模型
print(bow.toarray())                # 獲取單詞出現的次數
words = cv.get_feature_names()      # 獲取所有特征名

案例：

import nltk.tokenize as tk
import sklearn.feature_extraction.text as ft

doc = 'The brown dog is running. ' \
      'The black dog is in the black room. ' \
      'Running in the room is forbidden.'

# 對doc按照句子進行拆分
sents = tk.sent_tokenize(doc)

cv = ft.CountVectorizer()           # 構建詞袋模型
bow = cv.fit_transform(sents)       # 訓練詞袋模型
print(cv.get_feature_names())       # 獲取所有特征名
# ['black', 'brown', 'dog', 'forbidden', 'in', 'is', 'room', 'running', 'the']
print(bow.toarray())
# [[0 1 1 0 0 1 0 1 1]
#  [2 0 1 0 1 1 1 0 2]
#  [0 0 0 1 1 1 1 1 1]]

詞頻（TF）

$$詞頻=\frac{單詞在句中出現的次數}{句子的總詞數}$$

　　詞頻：一個單詞在一個句子中出現的頻率，詞頻相比單詞的出現次數可以更加客觀的評估單詞對一句話的語意的貢獻度，詞頻越高，對語意的貢獻度越大，對詞袋矩陣歸一化即可得到詞頻，

案例：對詞袋矩陣進行歸一化

import nltk.tokenize as tk
import sklearn.feature_extraction.text as ft
import sklearn.preprocessing as sp
doc = 'The brown dog is running. The black dog is in the black room. ' \
      'Running in the room is forbidden.'

sentences = tk.sent_tokenize(doc)       # 通過句子分詞

cv = ft.CountVectorizer()
bow = cv.fit_transform(sentences)
print(bow.toarray())        # 詞 出現的次數
words = cv.get_feature_names()
print(words)                # 詞 特征名
tf = sp.normalize(bow, norm='l1')
print(tf)                   # 詞頻

# [[0.    0.2   0.2     0.          0.        0.2          0.          0.2         0.2]
#  [0.25  0.    0.125   0.          0.125     0.125        0.125       0.         0.25 ]
#  [0.    0.    0.      0.16666667  0.16666667 0.16666667  0.16666667 0.16666667 0.16666667]]

檔案頻率（DF）

$$檔案頻率=\frac{含有某個單詞的檔案樣本數}{總檔案樣本數}$$

　　DF越低，代表當前單詞對語意的貢獻越高

逆檔案頻率（IDF）

$$逆檔案頻率=\frac{總樣本數}{(含有某個單詞的樣本數+1)}$$

　　IDF越高，代表當前單詞對語意的貢獻越高

詞頻-逆檔案頻率(TF-IDF)

詞頻矩陣中的每一個元素乘以相應單詞的逆檔案頻率，其值越大說明該詞對樣本語意的貢獻越大，根據每個詞的貢獻力度，構建學習模型，

獲取詞頻逆檔案頻率（TF-IDF）矩陣相關API：

# 獲取詞袋模型
cv = ft.CountVectorizer()
bow = cv.fit_transform(sentences).toarray()
# 獲取TF-IDF模型訓練器
tt = ft.TfidfTransformer()
tfidf = tt.fit_transform(bow).toarray()

案例：獲取TF_IDF矩陣：

import nltk.tokenize as tk
import sklearn.feature_extraction.text as ft
import numpy as np

doc = 'The brown dog is running. ' \
      'The black dog is in the black room. ' \
      'Running in the room is forbidden.'

# 對doc按照句子進行拆分
sents = tk.sent_tokenize(doc)

# 構建詞袋模型
cv = ft.CountVectorizer()
bow = cv.fit_transform(sents)

# TFIDF
tt = ft.TfidfTransformer()          # 獲取TF-IDF模型訓練器
tfidf = tt.fit_transform(bow)       # 訓練
print(np.round(tfidf.toarray(), 2))       # 精確到小數點后兩位
# [[0.   0.59 0.45 0.   0.   0.35 0.   0.45 0.35]
#  [0.73 0.   0.28 0.   0.28 0.22 0.28 0.   0.43]
#  [0.   0.   0.   0.54 0.41 0.32 0.41 0.41 0.32]]

文本分類(主題識別)

使用給定的文本資料集進行主題識別訓練，自定義測驗集測驗模型準確性，

import numpy as np
import sklearn.datasets as sd
import sklearn.feature_extraction.text as ft
import sklearn.naive_bayes as nb

train = sd.load_files('../machine_learning_date/20news',
                      encoding='latin1', shuffle=True,
                      random_state=7)
# train.data: 2968個樣本，每個樣本都是一篇郵件檔案
print(np.array(train.data).shape)       # (2968,)

# train.target: 2968個樣本，每個樣本都是檔案對應的類別
print(np.array(train.target).shape)     # (2968,)
print(train.target_names)
# ['misc.forsale', 'rec.motorcycles', 'rec.sport.baseball', 'sci.crypt', 'sci.space']

cv = ft.CountVectorizer()           # 詞袋模型
tt = ft.TfidfTransformer()          # 獲取TF-IDF模型訓練器

bow = cv.fit_transform(train.data)  # 訓練詞袋模型
tfidf = tt.fit_transform(bow)       # 訓練TF-IDF模型訓練器
print(tfidf.shape)              # (2968, 40605)

model = nb.MultinomialNB()      # 創建樸素貝葉斯模型
model.fit(tfidf, train.target)  # 訓練樸素貝葉斯模型

# 自定義測驗集進行測驗
test_data =https://www.cnblogs.com/LXP-Never/p/ [
    'The curveballs of right handed pitchers tend to curve to the left',
    'Caesar cipher is an ancient form of encryption',
    'This two-wheeler is really good on slippery roads']
# 怎么訓練的，就必須怎么預測
bow = cv.transform(test_data)
tfidf = tt.transform(bow)
pred_y = model.predict(tfidf)

for sent, index in zip(test_data, pred_y):
    print(sent, '->', train.target_names[index])
# The curveballs of right handed pitchers tend to curve to the left -> rec.sport.baseball
# Caesar cipher is an ancient form of encryption -> sci.crypt
# This two-wheeler is really good on slippery roads -> rec.motorcycles

性別識別

使用nltk提供的分類器對語料庫中英文男名與女名文本進行性別劃分訓練，最終進行性別驗證，

nltk提供的語料庫及分類方法相關API：

import nltk.corpus as nc
import nltk.classify as cf
?
# 讀取語料庫中names檔案夾里的male.txt檔案，并且進行分詞
male_names = nc.names.words('male.txt')
?
'''
train_data的格式不再是樣本矩陣，nltk要求的資料格式如下：
[ ({'age': 15, 'score1': 95, 'score2': 95}, 'good'),
  ({'age': 15, 'score1': 45, 'score2': 55}, 'bad') ]
'''
# 基于樸素貝葉斯分類器訓練測驗資料 
model = cf.NaiveBayesClassifier.train(train_data)
# 使用測驗資料計算分類器精確度得分（測驗資料格式與訓練資料一致）
ac = cf.accuracy(model, test_data)
# 對具體的某個樣本進行類別劃分
feature = {'age': 15, 'score1': 95, 'score2': 95}
gender = model.classify(feature)

案例：

import random
import nltk.corpus as nc
import nltk.classify as cf
male_names = nc.names.words('male.txt')
female_names = nc.names.words('female.txt')

data = []
for male_name in male_names:
    feature = {'feature': male_name[-2:].lower()}   # 取名字后面兩個字母
    data.append((feature, 'male'))
for female_name in female_names:
    feature = {'feature': female_name[-2:].lower()}
    data.append((feature, 'female'))
random.seed(7)
random.shuffle(data)
train_data = data[:int(len(data) / 2)]      # 用資料集的前一半作為 訓練資料
test_data = https://www.cnblogs.com/LXP-Never/p/data[int(len(data) / 2):]       # 用資料集的后一半作為 測驗訊據
model = cf.NaiveBayesClassifier.train(train_data)       # 樸素貝葉斯分類器
ac = cf.accuracy(model, test_data)

names, genders = ['Leonardo', 'Amy', 'Sam', 'Tom', 'Katherine', 'Taylor', 'Susanne'], []
for name in names:
    feature = {'feature': name[-2:].lower()}
    gender = model.classify(feature)
    genders.append(gender)
for name, gender in zip(names, genders):
    print(name, '->', gender)
# Leonardo -> male
# Amy -> female
# Sam -> male
# Tom -> male
# Katherine -> female
# Taylor -> male
# Susanne -> female

nltk分類器

　　nltk提供了樸素貝葉斯分類器方便的處理自然語言相關的分類問題，并且可以自動處理詞袋，完成TFIDF矩陣的整理，完成模型訓練，最終實作類別預測，使用方法如下：

import nltk.classify as cf
import nltk.classify.util as cu
'''
train_data的格式不再是樣本矩陣，nltk要求的資料格式如下：
[ ({'How': 1, 'are': 1, 'you': 1}, 'ask'),
  ({'fine': 1, 'Thanks': 2}, 'answer') ]
'''
# 基于樸素貝葉斯分類器訓練測驗資料 
model = cf.NaiveBayesClassifier.train(train_data)
ac = cu.accuracy(model, test_data)
print(ac)
pred = model.classify(test_data)

情感分析

分析語料庫中movie_reviews檔案，通過正面及負面評價進行自然語言訓練，實作情感分析，

import nltk.corpus as nc
import nltk.classify as cf
import nltk.classify.util as cu

# 存盤所有的正向樣本  
# pdata: [({單詞:true}, 'pos'),(),()...]
pdata =https://www.cnblogs.com/LXP-Never/p/ []
# pos檔案夾中的每個檔案的路徑
fileids = nc.movie_reviews.fileids('pos')
# print(len(fileids))
# 整理所有正面評論單詞，存入pdata串列
for fileid in fileids:
    sample = {}
    # words: 把當前檔案分詞處理
    words = nc.movie_reviews.words(fileid)
    for word in words:
        sample[word] = True
    pdata.append((sample, 'POSITIVE'))
# 整理所有反向樣本，存入ndata串列
ndata =https://www.cnblogs.com/LXP-Never/p/ []
fileids = nc.movie_reviews.fileids('neg')
for fileid in fileids:
    sample = {}
    words = nc.movie_reviews.words(fileid)
    for word in words:
        sample[word] = True
    ndata.append((sample, 'NEGATIVE'))

# 拆分測驗集與訓練集數量（80%作為訓練集）
pnumb, nnumb = int(0.8 * len(pdata)), int(0.8 * len(ndata))
train_data = pdata[:pnumb] + ndata[:nnumb]
test_data = pdata[pnumb:] + ndata[nnumb:]
# 基于樸素貝葉斯分類器訓練測驗資料 
model = cf.NaiveBayesClassifier.train(train_data)
ac = cu.accuracy(model, test_data)
print(ac)

# 模擬業務場景
reviews = [
    'It is an amazing movie.',
    'This is a dull movie. I would never recommend it to anyone.',
    'The cinematography is pretty great in this movie.',
    'The direction was terrible and the story was all over the place.']
for review in reviews:
    sample = {}
    words = review.split()
    for word in words:
        sample[word] = True
    pcls = model.classify(sample)
    print(review, '->', pcls)

主題抽取

經過分詞、單詞清洗、詞干提取后，基于TF-IDF演算法可以抽取一段文本中的核心主題詞匯，從而判斷出當前文本的主題，屬于無監督學習，gensim模塊提供了主題抽取的常用工具，

主題抽取相關API：

import gensim.models.ldamodel as gm
import gensim.corpora as gc
?
# 把lines_tokens中出現的單詞都存入gc提供的詞典物件，對每一個單詞做編碼，
line_tokens = ['hello', 'world', ...]
dic = gc.Dictionary(line_tokens)
# 通過字典構建詞袋
bow = dic.doc2bow(line_tokens) 
?
# 構建LDA模型
# bow: 詞袋
# num_topics: 分類數
# id2word: 詞典
# passes: 每個主題保留的最大主題詞個數
model = gm.LdaModel(bow, num_topics=n_topics, id2word=dic, passes=25)
# 輸出每個類別中對類別貢獻最大的4個主題詞
topics = model.print_topics(num_topics=n_topics, num_words=4)

案例：

import nltk.tokenize as tk
import nltk.corpus as nc
import nltk.stem.snowball as sb
import gensim.models.ldamodel as gm
import gensim.corpora as gc
doc = []
with open('../machine_learning_date/topic.txt', 'r') as f:
    for line in f.readlines():
        doc.append(line[:-1])
tokenizer = tk.WordPunctTokenizer() 
stopwords = nc.stopwords.words('english')
signs = [',', '.', '!']
stemmer = sb.SnowballStemmer('english')
lines_tokens = []
for line in doc:
    tokens = tokenizer.tokenize(line.lower())
    line_tokens = []
    for token in tokens:
        if token not in stopwords and token not in signs:
            token = stemmer.stem(token)
            line_tokens.append(token)
    lines_tokens.append(line_tokens)
# 把lines_tokens中出現的單詞都存入gc提供的詞典物件，對每一個單詞做編碼，
dic = gc.Dictionary(lines_tokens)
# 遍歷每一行，構建詞袋串列
bow = []
for line_tokens in lines_tokens:
    row = dic.doc2bow(line_tokens)
    bow.append(row)
n_topics = 2
# 通過詞袋、分類數、詞典、每個主題保留的最大主題詞個數構建LDA模型
model = gm.LdaModel(bow, num_topics=n_topics, id2word=dic, passes=25)
# 輸出每個類別中對類別貢獻最大的4個主題詞
topics = model.print_topics(num_topics=n_topics, num_words=4)
for label, words in topics:
    print(label, '->', words)
# 0 -> 0.022*"cryptographi" + 0.022*"use" + 0.022*"need" + 0.013*"cryptograph"
# 1 -> 0.046*"spaghetti" + 0.021*"made" + 0.021*"italian" + 0.015*"19th"

轉載請註明出處，本文鏈接：https://www.uj5u.com/qita/65608.html

標籤：其他

上一篇：最小二乘法

下一篇：機器學習-樸素貝葉斯