Skip to content

Pipeline Typing

typing

DownstreamConnectable = Union['Step', 'GlobalStep'] module-attribute

Alias for the Step types that can be connected as downstream steps.

UpstreamConnectableSteps = TypeVar('UpstreamConnectableSteps', bound=Union['Step', 'GlobalStep', 'GeneratorStep']) module-attribute

Type for the Step types that can be connected as upstream steps.

DownstreamConnectableSteps = TypeVar('DownstreamConnectableSteps', bound=DownstreamConnectable, covariant=True) module-attribute

Type for the Step types that can be connected as downstream steps.

PipelineRuntimeParametersInfo = Dict[str, Union[List['RuntimeParameterInfo'], Dict[str, 'RuntimeParameterInfo']]] module-attribute

Alias for the information of the runtime parameters of a Pipeline.

InputDataset = Union['Dataset', 'pd.DataFrame', List[Dict[str, str]]] module-attribute

Alias for the types we can process as input dataset.

LoadGroups = Union[List[List[Any]], Literal['sequential_step_execution']] module-attribute

Alias for the types that can be used as load groups.

  • if List[List[Any]], it's a list containing lists of steps that have to be loaded in isolation.
  • if "sequential_step_execution", each step will be loaded in a different stage i.e. only one step will be executed at a time.

StepLoadStatus

Bases: TypedDict

Dict containing information about if one step was loaded/unloaded or if it's load failed

Source code in src/distilabel/pipeline/typing.py 
50
51
52
53
54
55
class StepLoadStatus(TypedDict):
    """Dict containing information about if one step was loaded/unloaded or if it's load
    failed"""

    name: str
    status: Literal["loaded", "unloaded", "load_failed"]