Latency

Another concern respondents mentioned in the LLM in production survey was latency. The completion length of an LLM significantly affects latency. Although latency concerns have to be considered in MLOps as well, they are much more prominent in LLMOps because this is a big issue for the experimentation velocity during development and the user experience in production.