all Python 3astterm

tokenize

module

A lexical scanner for Python source code: splits program text into tokens, comments included.

Syntax

import tokenize

Use tokenize when the source text itself matters: highlighting, counting comments, automated edits that keep formatting intact. If you need the program's structure rather than a stream of tokens, use ast — it builds a tree but drops comments. tokenize.tokenize() reads bytes and detects the encoding itself, while generate_tokens() takes strings; mixing them up is a common mistake.

Examples

import io
import tokenize
src = "x = 1 + 2  # sum\n"
for tok in tokenize.generate_tokens(io.StringIO(src).readline):
    if tok.string.strip():
        print(tokenize.tok_name[tok.type], tok.string)  # → NAME x / OP = / NUMBER 1 / OP + / NUMBER 2 / COMMENT # sum

Common errors

Open in the full glossarydocs.python.orgReport an inaccuracy