Built-in scalars

Dependency-free per-unit channel sources — no analysis stack required.

lexograph.scalars.lengths(units)

Return the character length of each unit.

Parameters:

Name	Type	Description	Default
units	Sequence[str]	The segmented units.	required

Returns:

Type	Description
FloatArray	A float array of per-unit character counts.

Examples:

>>> lengths(["hi", "there"]).tolist()
[2.0, 5.0]

Source code in lexograph/scalars.py

def lengths(units: Sequence[str]) -> FloatArray:
    """Return the character length of each unit.

    Args:
        units: The segmented units.

    Returns:
        A float array of per-unit character counts.

    Examples:
        >>> lengths(["hi", "there"]).tolist()
        [2.0, 5.0]
    """
    return np.asarray([len(u) for u in units], dtype=float)

lexograph.scalars.positions(n)

Return 0, 1, ..., n - 1 as a position channel.

Parameters:

Name	Type	Description	Default
n	int	The number of units.	required

Returns:

Type	Description
FloatArray	A float array of positions in sequence order.

Examples:

>>> positions(3).tolist()
[0.0, 1.0, 2.0]

Source code in lexograph/scalars.py

def positions(n: int) -> FloatArray:
    """Return ``0, 1, ..., n - 1`` as a position channel.

    Args:
        n: The number of units.

    Returns:
        A float array of positions in sequence order.

    Examples:
        >>> positions(3).tolist()
        [0.0, 1.0, 2.0]
    """
    return np.arange(n, dtype=float)

lexograph.scalars.frequencies(units, *, ignore_case=True)

Return how many times each unit's value occurs in the sequence.

Parameters:

Name	Type	Description	Default
units	Sequence[str]	The segmented units (typically tokens).	required
ignore_case	bool	Count case-insensitively.	True

Returns:

Type	Description
FloatArray	A float array the same length as units; entry i is the total
FloatArray	number of occurrences of units[i].

Examples:

>>> frequencies(["the", "cat", "the"]).tolist()
[2.0, 1.0, 2.0]

Source code in lexograph/scalars.py

def frequencies(units: Sequence[str], *, ignore_case: bool = True) -> FloatArray:
    """Return how many times each unit's value occurs in the sequence.

    Args:
        units: The segmented units (typically tokens).
        ignore_case: Count case-insensitively.

    Returns:
        A float array the same length as ``units``; entry ``i`` is the total
        number of occurrences of ``units[i]``.

    Examples:
        >>> frequencies(["the", "cat", "the"]).tolist()
        [2.0, 1.0, 2.0]
    """
    keys = [u.lower() for u in units] if ignore_case else list(units)
    counts = Counter(keys)
    return np.asarray([counts[k] for k in keys], dtype=float)