Skip to content

APIGenExecutionChecker

Executes the generated function calls.

This step checks if a given answer from a model as generated by APIGenGenerator can be executed against the given library (given by libpath, which is a string pointing to a python .py file with functions).

Attributes

  • libpath: The path to the library where we will retrieve the functions. It can also point to a folder with the functions. In this case, the folder layout should be a folder with .py files, each containing a single function, the name of the function being the same as the filename.

  • check_is_dangerous: Bool to exclude some potentially dangerous functions, it contains some heuristics found while testing. This functions can run subprocesses, deal with the OS, or have other potentially dangerous operations. Defaults to True.

Input & Output Columns

graph TD
    subgraph Dataset
        subgraph Columns
            ICOL0[answers]
        end
        subgraph New columns
            OCOL0[keep_row_after_execution_check]
            OCOL1[execution_result]
        end
    end

    subgraph APIGenExecutionChecker
        StepInput[Input Columns: answers]
        StepOutput[Output Columns: keep_row_after_execution_check, execution_result]
    end

    ICOL0 --> StepInput
    StepOutput --> OCOL0
    StepOutput --> OCOL1
    StepInput --> StepOutput

Inputs

  • answers (str): List with arguments to be passed to the function, dumped as a string from a list of dictionaries. Should be loaded using json.loads.

Outputs

  • keep_row_after_execution_check (bool): Whether the function should be kept or not.

  • execution_result (str): The result from executing the function.

Examples

Execute a function from a given library with the answer from an LLM

from distilabel.steps.tasks import APIGenExecutionChecker

# For the libpath you can use as an example the file at the tests folder:
# ../distilabel/tests/unit/steps/tasks/apigen/_sample_module.py
task = APIGenExecutionChecker(
    libpath="../distilabel/tests/unit/steps/tasks/apigen/_sample_module.py",
)
task.load()

res = next(
    task.process(
        [
            {
                "answers": [
                    {
                        "arguments": {
                            "initial_velocity": 0.2,
                            "acceleration": 0.1,
                            "time": 0.5,
                        },
                        "name": "final_velocity",
                    }
                ],
            }
        ]
    )
)
res
#[{'answers': [{'arguments': {'initial_velocity': 0.2, 'acceleration': 0.1, 'time': 0.5}, 'name': 'final_velocity'}], 'keep_row_after_execution_check': True, 'execution_result': ['0.25']}]

References