Different loss functions used to train LLMs produce very distinct flavors of misalignment: imitative learning causes 'seven deadly sins' misalignment, human approval causes sycophancy, automatic verifiers cause literal genie behavior, and LLM judges cause trickster misalignment

for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment.

https://www.alignmentforum.org/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-of-llm-misalignment