• 创建词汇表
  将文档中的新词集合添加到词汇表
  
  创建词汇表

  Python

  def createVocabList(dataSet): vocabSet = set([]) for document in dataSet: # 创建两个集合的并集 vocabSet = vocabSet | set(document) return list(vocabSet)

  1 2 3 4 5 6 7 8 9

  def createVocabList(dataSet):       vocabSet = set([])       for document in dataSet:         # 创建两个集合的并集         vocabSet = vocabSet | set(document)       return list(vocabSet)

• 词集模型(set-of-words model)
  通过词汇表将文档转换为文档向量, 向量的每个元素表示词汇表中的单词在输入文档中是否出现
  
  词集模型

  Python

  def setOfWords2Vec(vocabList, inputSet): # 创建一个和词汇表等长, 所含元素都为0的向量 returnVec = [0] * len(vocabList) # 遍历文档中所有单词 for word in inputSet: # 如果出现了词汇表中的单词 if word in vocabList: # 将输出的文档向量中的对应值设为1 returnVec[vocabList.index(word)] = 1 else: print("the word: %s is not in my Vocabulary!" % word) return returnVec

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  def setOfWords2Vec(vocabList, inputSet):       # 创建一个和词汇表等长, 所含元素都为0的向量     returnVec = [0] * len(vocabList)       # 遍历文档中所有单词     for word in inputSet:     # 如果出现了词汇表中的单词     if word in vocabList:         # 将输出的文档向量中的对应值设为1         returnVec[vocabList.index(word)] = 1     else:         print("the word: %s is not in my Vocabulary!" % word)       return returnVec

• 词袋模型(bag-of-words model)
  文档向量的每个函数表示词汇表中的单词在文档中出现的次数
  
  词袋模型

  Python

  def bagOfWords2VecMN(vocabList, inputSet): returnVec = [0] * len(vocabList) for word in inputSet: if word in vocabList: returnVec[vocabList.index(word)] += 1 return returnVec

  1 2 3 4 5 6

  def bagOfWords2VecMN(vocabList, inputSet):     returnVec = [0] * len(vocabList)     for word in inputSet:         if word in vocabList:             returnVec[vocabList.index(word)] += 1     return returnVec

• 使用正则表达式切分文本
  捕获所有单词, 去掉少于两个字母的字符串, 并将所有字符串转换为小写
  
  使用正则表达式切分文本

  Python

  def textParse(bigString): import re listOfTokens = re.split('\w+', bigString) return [tok.lower() for tok in listOfTokens if len(tok) > 2]

  1 2 3 4

  def textParse(bigString):     import re     listOfTokens = re.split('\w+', bigString)     return [tok.lower() for tok in listOfTokens if len(tok) > 2]

p.s. 书中使用的正则表达式为 r'\W*', 运行时出现警告 split() requires a non-empty pattern match. 官方文档如下:

Note: split() doesn’t currently split a string on an empty pattern match. For example:

Even though ‘x*’ also matches 0 ‘x’ before ‘a’, between ‘b’ and ‘c’, and after ‘c’, currently these matches are ignored. The correct behavior (i.e. splitting on empty matches too and returning [”, ‘a’, ‘b’, ‘c’, ”]) will be implemented in future versions of Python, but since this is a backward incompatible change, a FutureWarning will be raised in the meanwhile.
Patterns that can only match empty strings currently never split the string. Since this doesn’t match the expected behavior, a ValueError will be raised starting from Python 3.5:

训练函数和分类函数

• 训练函数
  训练函数

  Python

  def trainNB0(trainMatrix, trainCategory): numTrainDocs = len(trainMatrix) numWords = len(trainMatrix[0]) # 任意文档属于侮辱性文档的概率 pAbusive = sum(trainCategory)/float(numTrainDocs) # 初始化概率 p0Num = ones(numWords) p1Num = ones(numWords) p0Denom = 2.0 p1Denom = 2.0 # 遍历训练集中所有文档 for i in range(numTrainDocs): if trainCategory[i] == 1: # 向量相加 p1Num += trainMatrix[i] p1Denom += sum(trainMatrix[i]) else: p0Num += trainMatrix[i] p0Denom += sum(trainMatrix[i]) # 对每个元素做除法, 并取自然对数避免下溢出 p1Vect = log(p1Num/p1Denom) p0Vect = log(p0Num/p1Denom) return p0Vect, p1Vect, pAbusive

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

  def trainNB0(trainMatrix, trainCategory):     numTrainDocs = len(trainMatrix)     numWords = len(trainMatrix[0])     # 任意文档属于侮辱性文档的概率     pAbusive = sum(trainCategory)/float(numTrainDocs)       # 初始化概率     p0Num = ones(numWords)     p1Num = ones(numWords)     p0Denom = 2.0     p1Denom = 2.0       # 遍历训练集中所有文档     for i in range(numTrainDocs):         if trainCategory[i] == 1:         # 向量相加         p1Num += trainMatrix[i]         p1Denom += sum(trainMatrix[i])     else:         p0Num += trainMatrix[i]         p0Denom += sum(trainMatrix[i])       # 对每个元素做除法, 并取自然对数避免下溢出     p1Vect = log(p1Num/p1Denom)     p0Vect = log(p0Num/p1Denom)       return p0Vect, p1Vect, pAbusive

• 分类函数
  分类函数

  Python

  def classifyNB(vec2Classify, p0Vec, p1Vec, pClass1): p1 = sum(vec2Classify * p1Vec) + log(pClass1) p0 = sum(vec2Classify * p0Vec) + log(1.0 - pClass1) if p1 > p0: return 1 else: return 0

  1 2 3 4 5 6 7

  def classifyNB(vec2Classify, p0Vec, p1Vec, pClass1):     p1 = sum(vec2Classify * p1Vec) + log(pClass1)     p0 = sum(vec2Classify * p0Vec) + log(1.0 - pClass1)     if p1 > p0:         return 1     else:         return 0

垃圾邮件测试函数

p.s. 官方提供的测试数据中有非法字符