Skip to content

PairRM

Rank the candidates based on the input using the LLM model.

Note

This step differs to other tasks as there is a single implementation of this model currently, and we will use a specific LLM.

Attributes

  • model: The model to use for the ranking. Defaults to "llm-blender/PairRM".

  • instructions: The instructions to use for the model. Defaults to None.

Input & Output Columns

Inputs

  • inputs (List[Dict[str, Any]]): The input text or conversation to rank the candidates for.

  • candidates (List[Dict[str, Any]]): The candidates to rank.

Outputs

  • ranks (List[int]): The ranks of the candidates based on the input.

  • ranked_candidates (List[Dict[str, Any]]): The candidates ranked based on the input.

  • model_name (str): The model name used to rank the candidate responses. Defaults to "llm-blender/PairRM".

Examples

Rank LLM candidates

from distilabel.steps.tasks import PairRM

# Consider this as a placeholder for your actual LLM.
pair_rm = PairRM()

pair_rm.load()

result = next(
    scorer.process(
        [
            {"input": "Hello, how are you?", "candidates": ["fine", "good", "bad"]},
        ]
    )
)
# result
# [
#     {
#         'input': 'Hello, how are you?',
#         'candidates': ['fine', 'good', 'bad'],
#         'ranks': [2, 1, 3],
#         'ranked_candidates': ['good', 'fine', 'bad'],
#         'model_name': 'llm-blender/PairRM',
#     }
# ]

References