Skip to content

Commit 0cb45b0

Browse files
ChrisW09claude
andcommitted
fix(onehot): encode categories before onehot_from_ordinal
``categorical_method="onehot_from_ordinal"`` appended only ``OneHotFromOrdinalTransformer``, which requires input that is already ordinal encoded. The Preprocessor hands it raw column values, so any string column died inside ``np.max(X, axis=0).astype(int)`` with a bare ValueError: invalid literal for int() with base 10: 'c' that says nothing about the chosen method. Since ``_detect_column_types`` routes every non-numeric column to the categorical side, this was the common case. Insert ``ContinuousOrdinalTransformer`` ahead of it so the method does what the Preprocessor docstring already claimed -- integer codes, then one-hot. Because that encoder numbers categories from 1 and reserves 0 for unseen values, a column with k categories expands to k + 1 columns and categories unseen at fit time land in the reserved column instead of producing an all-zero row. Docs reconciled: the Preprocessor docstring, the user-guide table and the class itself previously gave three different accounts of this method. Closes #17 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 51c3043 commit 0cb45b0

4 files changed

Lines changed: 55 additions & 4 deletions

File tree

docs/user_guide/preprocessing.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ preprocessor = Preprocessor(
7777
| --------------------- | ------------------------------ | ----- |
7878
| `int` | `ContinuousOrdinalTransformer` | Integer/ordinal encoding (default) |
7979
| `one-hot` | `OneHotEncoder` | One-hot encoding |
80-
| `onehot_from_ordinal` | `OneHotFromOrdinalTransformer` | One-hot from pre-encoded ordinals |
80+
| `onehot_from_ordinal` | `ContinuousOrdinalTransformer` -> `OneHotFromOrdinalTransformer` | Integer codes, then one-hot (reserves column `0` for unseen categories) |
8181
| `pretrained` | `LanguageEmbeddingTransformer` | Pretrained language embeddings |
8282
| `custombin` | `CustomBinTransformer` | Binning of categorical codes |
8383
| `none` | `NoTransformer` | Pass-through |

pretab/pipeline/categorical.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,11 @@ def get_categorical_transformer_steps(
4949
bin_kwargs.setdefault("output_dim", output_dim)
5050
steps.append(("custombin", CustomBinTransformer(**bin_kwargs)))
5151
elif method == "onehot_from_ordinal":
52+
# ``OneHotFromOrdinalTransformer`` expects integer codes; the Preprocessor
53+
# hands it raw column values, which are usually strings. Encode first so
54+
# the method matches its documented "integer codes then one-hot"
55+
# behaviour instead of dying in ``np.max(...).astype(int)``.
56+
steps.append(("continuous_ordinal", ContinuousOrdinalTransformer()))
5257
steps.append(("onehot_from_ordinal", OneHotFromOrdinalTransformer()))
5358
else:
5459
raise invalid_param_error(

pretab/preprocessor.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,11 @@ class Preprocessor(TransformerMixin, BaseEstimator):
5252
categorical_method : str, default="int"
5353
Preprocessing strategy applied to every categorical column unless overridden per feature.
5454
Choices: ``"int"`` (contiguous integer codes), ``"one-hot"`` (dummy columns),
55-
``"onehot_from_ordinal"`` (integer codes then one-hot), ``"pretrained"`` (sentence-transformer
56-
language embeddings), and ``"custombin"`` (discretized bin codes). Pass ``None`` (resolved to
57-
``"none"``) to leave categorical columns unchanged.
55+
``"onehot_from_ordinal"`` (contiguous integer codes, then one-hot; reserves the first
56+
column for categories unseen at fit time, so a column with ``k`` categories expands to
57+
``k + 1`` columns), ``"pretrained"`` (sentence-transformer language embeddings), and
58+
``"custombin"`` (discretized bin codes). Pass ``None`` (resolved to ``"none"``) to leave
59+
categorical columns unchanged.
5860
feature_preprocessing : dict, optional
5961
Mapping of individual column names to a method, overriding the global ``numerical_method`` /
6062
``categorical_method`` for those columns only, e.g.

tests/test_categorical_pipeline.py

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
1+
from typing import cast
2+
13
import numpy as np
4+
import pandas as pd
25
import pytest
36
from sklearn.pipeline import Pipeline
47

8+
from pretab import Preprocessor
59
from pretab.pipeline import get_categorical_transformer_steps
610

711

@@ -28,3 +32,43 @@ def test_one_hot_handle_unknown_override():
2832
with pytest.raises(ValueError):
2933
pipe.transform(np.array([["C"]]))
3034

35+
36+
37+
# --------------------------------------------------------------------------- #
38+
# ``onehot_from_ordinal`` must accept raw (string) categoricals.
39+
#
40+
# The pipeline appended only ``OneHotFromOrdinalTransformer``, which requires
41+
# already-ordinal input, so ``np.max(X, axis=0).astype(int)`` died with a bare
42+
# ``ValueError: invalid literal for int() with base 10: 'c'``.
43+
# --------------------------------------------------------------------------- #
44+
def test_onehot_from_ordinal_encodes_string_categories():
45+
frame = pd.DataFrame({"c": ["a", "b", "c"] * 30})
46+
pre = Preprocessor(categorical_method="onehot_from_ordinal", numerical_method="none")
47+
48+
out = cast("np.ndarray", pre.fit_transform(frame, return_array=True))
49+
50+
# 3 categories plus the reserved column 0 for unseen values.
51+
assert out.shape == (90, 4)
52+
np.testing.assert_array_equal(out[:3], np.eye(4)[[1, 2, 3]])
53+
54+
55+
def test_onehot_from_ordinal_pipeline_encodes_before_one_hot():
56+
steps = [name for name, _ in get_categorical_transformer_steps("onehot_from_ordinal")]
57+
assert steps.index("continuous_ordinal") < steps.index("onehot_from_ordinal")
58+
59+
60+
def test_onehot_from_ordinal_sends_unseen_categories_to_the_reserved_column():
61+
frame = pd.DataFrame({"c": ["a", "b", "c"] * 30})
62+
pre = Preprocessor(categorical_method="onehot_from_ordinal", numerical_method="none").fit(frame)
63+
64+
out = cast("np.ndarray", pre.transform(pd.DataFrame({"c": ["a", "ZZZ", "c"]}), return_array=True))
65+
66+
np.testing.assert_array_equal(out[1], [1.0, 0.0, 0.0, 0.0])
67+
68+
69+
def test_onehot_from_ordinal_feature_names_match_width():
70+
frame = pd.DataFrame({"c": ["a", "b", "c"] * 30})
71+
pre = Preprocessor(categorical_method="onehot_from_ordinal", numerical_method="none").fit(frame)
72+
73+
transformed = cast("np.ndarray", pre.transform(frame, return_array=True))
74+
assert len(pre.get_feature_names_out()) == transformed.shape[1]

0 commit comments

Comments
 (0)