Extracting words from a long text and creating statistics on them. What’s wrong?

Question

We have the book "Pride and Prejudice by Jane Austen" of the project Gutenberg:

http://www.gutenberg.org/ebooks/1342

The goal is to extract all the words of the text, creating statistics such as: frequency of each word, total characters in the text, average word size, average sentence size and a "top 10" of the longest words.

Looking at the text, I found that many words contain strange characters like:

"the,"

Requiring initial removal of these characters.

Like I tried to do:

# -*- coding: UTF-8 -*-
from string import punctuation
from collections import Counter
with open("1342-0.txt",encoding='utf8') as f:
    texto = f.read()

words = texto.split()
n_words =[]
for word in words:
    for p in punctuation:
        if p in word:
            n_word = word.replace(p,"")
            n_words.append(n_word)
        n_words.append(word)

"1342-0.txt" is the book in question. The above code tries to delete unwanted characters but does not work. What is wrong? Any better idea?

Answer 1

An initial idea is to do the split not only by spaces, but by any character that is not part of a word:

from collections import Counter
import re

r = re.compile(r'\W+')
c = Counter()
with open("1342-0.txt", encoding='utf8') as f:
    for linha in f:
        for word in r.split(linha):
            c.update([word])

print(c)

The shortcut \W is "all that nay for letter, number, or character _" - and how in the text it has "words" as _she_, That’s considered a different word than she. I also consider that the numbers (like 1) are "words", which are also accounted for.

As I find the words, I update the Counter using method update (if the key does not exist, it is created with the value 1, and if it exists, add 1 to its value - in the end we have the total count of each word).

Another detail is that read() loads the entire contents of the file into memory at once. Depending on the size of the file, this may be a problem. Already the above code reads one line at a time (and I am assuming that there is no case of a word starting on one line and ending on another - although in this case, use read and split would also not consider that it is the same word).

If you don’t want to include the _ as part of a word, simply change the regex to:

r = re.compile(r'[\W_]+')

---

The problem is that there are also words with hyphenation, such as over-scrupulous. The above code considers two different words ("over" and "scrupulous"). If you want them to be one word, you have to change a little:

from collections import Counter
import re

r = re.compile(r'\b\w+(?:-\w+)*\b')
c = Counter()
with open("1342-0.txt", encoding='utf8') as f:
    for linha in f:
        for word in r.findall(linha):
            c.update([word])

print(c)

Now I use \w+ (one or more characters that form a word), and I place a section containing a hyphen and \w+ (and this whole section can repeat itself zero or more times). So I get words with one or more hyphens too.

If you don’t want to include the _ as part of a word, use:

r = re.compile(r'\b[^\W_]+(?:-[^\W_]+)*\b')

---

It is worth remembering that string.punctuation only consider the characters !"#$%&'()*+,-./:;<=>?@[]^_{|}`. If you have any other character in the text other than letter, number or _, he will not be removed.

An example is the character “ (present in the text), which nay is the same thing as " (are different quotes, the first is the "LEFT DOUBLE QUOTATION MARK" and the second is "QUOTATION MARK", and if you use punctuation, will only remove the second).

Answer 2

To remove punctuation characters from a python text just one line:

from string import punctuation

texto = '''It is a truth universally acknowledged, that a single man in
      possession of a good fortune, must be in want of a wife.

      However little known the feelings or views of such a man may be
      on his first entering a neighbourhood, this truth is so well
      fixed in the minds of the surrounding families, that he is
      considered the rightful property of some one or other of their
      daughters.      
'''

#Remove os pontuadores
print(texto.translate(str.maketrans('', '', punctuation)))

Resulting:

  It is a truth universally acknowledged that a single man in
  possession of a good fortune must be in want of a wife

  However little known the feelings or views of such a man may be
  on his first entering a neighbourhood this truth is so well
  fixed in the minds of the surrounding families that he is
  considered the rightful property of some one or other of their
  daughters     

Running on Repl.it: https://repl.it/repls/DarkcyanPointlessChord

Example 2: https://repl.it/repls/ResponsibleVariableTheories

The logic is the following, the method str.translate() returns a copy of the string in which each character was mapped through the conversion table specified by the method str.maketrans().