In brief
- Alignment is an overarching goal, not the name of a single training method.
- Classical RLHF uses human comparisons, a learned reward model and reinforcement learning.
- Supervised fine-tuning, RLHF and human approval during live operation serve different purposes.
- RLHF guarantees neither truth nor complete safety and does not replace company-specific controls.
Alignment is a goal, not a single method
During pretraining, a language model is primarily optimised to generate likely continuations. Reliably following instructions, refusing inappropriate requests or observing the rules of a particular company does not emerge automatically. Alignment attempts to bring observable behaviour closer to such expectations.
There is no single universal measure of a fully aligned model. People may judge instructions, risks and values differently. It therefore matters whose requirements a system was aligned to and for which use its behaviour has actually been tested.
- Model alignment
- Training and evaluation shape general behaviour such as instruction following, handling uncertainty and appropriate refusals.
- Application alignment
- System instructions, approved sources, tools, permissions and guardrails adapt a general model to a specific use case.
- Operational alignment
- Ownership, evals, monitoring, escalation and human decisions keep the application within its intended limits during everyday use.
Pretraining, fine-tuning, RLHF and human review serve different purposes
Some of these methods build on one another, but they should not be treated as equivalent. A classical RLHF process often starts with a model that has already been pretrained and further optimised using demonstrations. A person reviewing a live business response is not performing model training.
| Method | Learning or review signal | Primary purpose | Does not replace |
|---|---|---|---|
| Pretraining | Very large text and data collections, usually self-supervised learning | Develop general language and pattern capabilities | Reliable instruction following and deployment rules |
| Supervised fine-tuning | Curated inputs paired with target responses | Learn tasks, formats or desired responses from demonstrations | Current knowledge, permissions and ongoing quality control |
| Reinforcement learning from human feedback (RLHF) | Human comparisons or ratings of model responses | Optimise behaviour towards the evaluated preferences | Factual accuracy, universal values and use-case accountability |
| Human review in operation | A responsible person assesses a specific real case | Approve, correct or escalate before an outcome takes effect | Does not automatically change the model and is not RLHF |
How classical RLHF works
RLHF stands for reinforcement learning from human feedback. Rather than expressing every quality of a desired response as a mathematical rule, people indicate which of several responses they prefer. A classical process first uses those comparisons to train a reward model and then uses that model to optimise the language model.
- Step 1
Define the desired behaviour
Guidelines and examples describe what should count as helpful, appropriately uncertain or unacceptable.
- Step 2
Generate response alternatives
The model produces several possible responses to the same prompts.
- Step 3
Ask people to compare them
Trained raters score or rank alternatives according to a defined rubric.
- Step 4
Model the preferences
In classical RLHF, a reward model learns to predict which responses raters would prefer.
- Step 5
Optimise the language model
Reinforcement learning changes the model policy so responses with a higher predicted reward become more likely.
- Step 6
Evaluate independently
Separate cases examine improvements, side effects, excessive refusals and new failure modes.
How RLHF relates to reinforcement learning
Reinforcement learning is a broad area of machine learning. A system tries actions and learns from rewards which strategy performs better over time. The reward may come from a simulation, a game score or a technically defined objective.
In RLHF, the objective signal is instead derived indirectly from human preferences. For language models, people typically compare complete responses rather than evaluating each token. The learned signal is then used for reinforcement learning.
- RL is the broader field
- Reinforcement learning does not require human ratings. An environment can provide rewards automatically.
- RLHF is a particular application
- Human comparisons help translate qualities that are difficult to specify, such as helpfulness or appropriate behaviour, into an optimisation signal.
- Not all preference training is RLHF
- Methods such as direct preference optimisation learn directly from preference data without a separate reward model and reinforcement learning. Feedback supplied by AI systems rather than people is often called RLAIF.
Where model alignment ends and company responsibility begins
A provider can train a general model for helpfulness and broad safety requirements. This does not teach it the binding processes or risk thresholds of an individual company. Productive use therefore requires evaluation of the entire system, not only the description of its base model.
- Run independent evals with representative and critical company cases
- Technically limit approved sources, tools, actions and permissions
- Require responsible people to approve binding decisions and exceptions
- Retest model and provider updates before rollout
- Do not turn feedback into training data without review and data governance
- Preferences are not objective
- Ratings depend on instructions, expertise, language, culture and the composition of the rater group. Minority views or rare cases can be underrepresented.
- Rewards are proxies
- A reward model represents human expectations imperfectly. Excessive optimisation can satisfy the measurement signal without reliably achieving the underlying goal.
- Desired behaviour is not the same as truth
- A persuasively phrased response can still be factually wrong. RLHF is neither a knowledge base nor proof that information is current.
- New situations remain difficult
- Behaviour that works on training examples can fail for other languages, specialist domains, attacks or unusual inputs.
Example: a customer service assistant with explicit limits
A company uses a language model that its provider has already aligned for helpful responses using demonstrations and preference data. This is not sufficient for the company's own service process: the assistant may use only approved product information, must not make binding contractual commitments and cannot issue goodwill payments on its own. Such cases are handed to employees. A fixed eval set checks source use, appropriate refusals and escalation after each model update. General model alignment, application-specific rules and human review complement one another rather than replace one another.
What to remember
Alignment is a layered process. RLHF can improve general model behaviour, but company-specific rules, independent evals, technical limits and human accountability are still required for controlled business use.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed