🧑‍🏫Evaluation

Traditional MLOps involves validating ML models on a hold-out validation set with a performance metric to gauge the model's effectiveness. But when it comes to LLMs, determining the quality of a response becomes less straightforward. At present, organizations seem to be employing A/B testing strategies for their models.

To aid in LLM evaluation, tools such as HoneyHive and HumanLoop have been developed. These platforms provide a means for gauging the performance of LLMs, helping developers to ensure that the models are producing suitable responses and functioning as expected.