Segment

Turn raw text into ordered units: characters, tokens, or sentences.

lexograph.segment.units.segment(text, unit='sentences', *, punkt=False)

Segment text into ordered units of the requested kind.

This is the single entry point to the segmentation step of the spine.

Parameters:

Name	Type	Description	Default
text	str	The source text.	required
unit	UnitKind	"chars", "tokens", or "sentences".	'sentences'
punkt	bool	Use NLTK Punkt for sentence splitting (ignored for other kinds).	False

Returns:

Type	Description
list[Unit]	The ordered list of units.

Raises:

Type	Description
ValueError	If unit is not one of the three recognised kinds.

Contract

• The returned list preserves source order.
• "chars" keeps every character; "tokens" and "sentences" drop pure-whitespace units.

Examples:

>>> segment("One. Two.", unit="sentences")
['One.', 'Two.']
>>> segment("One two", unit="tokens")
['One', 'two']
>>> len(segment("abc", unit="chars"))
3

Source code in lexograph/segment/units.py

def segment(
    text: str, unit: UnitKind = "sentences", *, punkt: bool = False
) -> list[Unit]:
    """Segment ``text`` into ordered units of the requested kind.

    This is the single entry point to the segmentation step of the spine.

    Args:
        text: The source text.
        unit: ``"chars"``, ``"tokens"``, or ``"sentences"``.
        punkt: Use NLTK Punkt for sentence splitting (ignored for other kinds).

    Returns:
        The ordered list of units.

    Raises:
        ValueError: If ``unit`` is not one of the three recognised kinds.

    Contract:
        - The returned list preserves source order.
        - ``"chars"`` keeps every character; ``"tokens"`` and ``"sentences"``
          drop pure-whitespace units.

    Examples:
        >>> segment("One. Two.", unit="sentences")
        ['One.', 'Two.']
        >>> segment("One two", unit="tokens")
        ['One', 'two']
        >>> len(segment("abc", unit="chars"))
        3
    """
    if unit == "chars":
        return characters(text)
    if unit == "tokens":
        return tokens(text)
    if unit == "sentences":
        return sentences(text, punkt=punkt)
    msg = f"unit must be 'chars', 'tokens', or 'sentences', got {unit!r}"
    raise ValueError(msg)

lexograph.segment.units.sentences(text, *, punkt=False)

Split text into sentences, in order.

Parameters:

Name	Type	Description	Default
text	str	The source text.	required
punkt	bool	If True, use NLTK's Punkt sentence tokenizer (downloading the model on first use). If False (the default), use the bundled offline regex splitter, which guards common abbreviations.	False

Returns:

Type	Description
list[Unit]	The ordered list of sentences, each stripped of surrounding whitespace.
list[Unit]	Empty or whitespace-only sentences are dropped.

Examples:

>>> sentences("Mr. Bennet replied that he had not. He said no more.")
['Mr. Bennet replied that he had not.', 'He said no more.']

Source code in lexograph/segment/units.py

def sentences(text: str, *, punkt: bool = False) -> list[Unit]:
    """Split ``text`` into sentences, in order.

    Args:
        text: The source text.
        punkt: If ``True``, use NLTK's Punkt sentence tokenizer (downloading the
            model on first use). If ``False`` (the default), use the bundled
            offline regex splitter, which guards common abbreviations.

    Returns:
        The ordered list of sentences, each stripped of surrounding whitespace.
        Empty or whitespace-only sentences are dropped.

    Examples:
        >>> sentences("Mr. Bennet replied that he had not. He said no more.")
        ['Mr. Bennet replied that he had not.', 'He said no more.']
    """
    if punkt:
        return _punkt_sentences(text)
    return _regex_sentences(text)

lexograph.segment.units.tokens(text)

Return the word tokens of text in order.

A token is a run of word characters with optional internal apostrophes or hyphens; punctuation and whitespace are dropped.

Parameters:

Name	Type	Description	Default
text	str	The source text.	required

Returns:

Type	Description
list[Unit]	The ordered list of word tokens.

Examples:

>>> tokens("It's a good-humoured day.")
["It's", 'a', 'good-humoured', 'day']

Source code in lexograph/segment/units.py

def tokens(text: str) -> list[Unit]:
    """Return the word tokens of ``text`` in order.

    A token is a run of word characters with optional internal apostrophes or
    hyphens; punctuation and whitespace are dropped.

    Args:
        text: The source text.

    Returns:
        The ordered list of word tokens.

    Examples:
        >>> tokens("It's a good-humoured day.")
        ["It's", 'a', 'good-humoured', 'day']
    """
    return _TOKEN_RE.findall(text)

lexograph.segment.units.characters(text)

Return every character of text in order.

Parameters:

Name	Type	Description	Default
text	str	The source text.	required

Returns:

Type	Description
list[Unit]	One single-character string per character, including whitespace and
list[Unit]	punctuation (the punctuation-spiral preset filters these itself).

Examples:

>>> characters("Hi!")
['H', 'i', '!']

Source code in lexograph/segment/units.py

def characters(text: str) -> list[Unit]:
    """Return every character of ``text`` in order.

    Args:
        text: The source text.

    Returns:
        One single-character string per character, including whitespace and
        punctuation (the punctuation-spiral preset filters these itself).

    Examples:
        >>> characters("Hi!")
        ['H', 'i', '!']
    """
    return list(text)