Demo program from the Python workshop at UGA 11/2/2010. Try running it with the sample text: python wordcount.py gettysburg.txt

gettysburg.txt

Four score and seven years ago our fathers brought forth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal.

Now we are engaged in a great civil war, testing whether that nation, or any
nation so conceived and so dedicated, can long endure. We are met on a great
battle-field of that war. We have come to dedicate a portion of that field, as a
final resting place for those who here gave their lives that that nation might
live. It is altogether fitting and proper that we should do this.

But, in a larger sense, we can not dedicate -- we can not consecrate -- we can
not hallow -- this ground. The brave men, living and dead, who struggled here,
have consecrated it, far above our poor power to add or detract. The world will
little note, nor long remember what we say here, but it can never forget what
they did here. It is for us the living, rather, to be dedicated here to the
unfinished work which they who fought here have thus far so nobly advanced. It
is rather for us to be here dedicated to the great task remaining before us --
that from these honored dead we take increased devotion to that cause for which
they gave the last full measure of devotion -- that we here highly resolve that
these dead shall not have died in vain -- that this nation, under God, shall
have a new birth of freedom -- and that government of the people, by the people,
for the people, shall not perish from the earth.

wordcount.py

#!/usr/bin/env python

"""Count the words in a text file or stream, and print a table of word counts
sorted by frequency.

If no argument is given, reads from standard input.

Usage:
    wordcount.py [filename]
"""

import sys
import collections


def count_words(infile):
    """Count the occurrences of each word in infile.

    Input: File handle open for reading (containing text)
    Output: dict of {string: integer} words and counts

    Example:
        >>> import StringIO
        >>> infile = StringIO("yes no yes no maybe")
        >>> count_words(infile)
        {'maybe': 1, 'yes': 2, 'no': 2}
    """
    word_counts = collections.defaultdict(int)
    for line in infile:
        for word in line.split():
            # Ignore case and adjacent punctuation
            word = word.strip(',.;:?!-()[]{}/|\\~@#$%^&*+="\'`').lower()
            word_counts[word] += 1
    if '' in word_counts:
        del word_counts['']  # From isolated punctuation
    return word_counts


def print_dictionary(dct):
    """Print the keys and values in dct, sorted by decreasing value.

    Input: dict with string keys and any-type values
    Output: None
    """
    # Calculate the width of the left column (for printing words)
    max_width = max(len(key) for key in dct)
    # Sort the word-count pairs by decreasing count
    word_count_pairs = dct.items()
    word_count_pairs.sort(key=lambda kv: kv[1], reverse=True)
    # Print pairs as a two-column table, separated by a tab
    for key, value in word_count_pairs:
        print key.rjust(max_width), '\t', value


if __name__ == '__main__':
    if len(sys.argv) == 1:
        # Text stream from a pipe
        infile = sys.stdin
    elif len(sys.argv) == 2:
        # Read text from the given file
        infile = open(sys.argv[1], 'r')
    else:
        # Too many arguments! Print usage & quit
        sys.exit(__doc__)

    # Now, do everything
    word_counts = count_words(infile)
    print_dictionary(word_counts)