Free, gamified roadmaps for LLM engineering: an Inference Engineering path (KV caches, CUDA kernels, production vLLM serving) and a Model Training path (pretraining on a budget, scaling laws, SFT/DPO/GRPO) — 183 tasks with auto-verified mi…
#dpo
46 items
Show HN: Free Inference Engineer and Model Training Roadmap (inferquest.org via hn) SFT + DPO on open-sourced SLMs (www.reddit.com) Hey folks, this is for those who appreciate experimentation on open-sourced AI models. We fine-tuned open-sourced SMLs (3B and 7B parameters) with SFT + DPO against commercial models like GPT-5.4, Gemini 3.1 Pro, Claude Opus 4.6, Google Do…
Free hands-on lab: build a ReAct agent 3 ways (create_agent, raw LangGraph with tool-call budget, NVIDIA NAT YAML) (www.reddit.com) Probe-Detected Grokking in Multi-Probe DPO (openinterp.org via hn) Probe-Detected Grokking in Multi-Probe DPO Orthogonal Learning Beyond Task-Specific Detectors in Qwen3.6-27B Probe-Detected Grokking in Multi-Probe DPO: Orthogonal Learning Beyond Task-Specific Detectors Abstract We report a phase-transiti…
Show HN: Minimal LLM Post-Training Experiments on an 8GB GPU (SFT, DPO, GRPO) (github.com via hn) Minimal LLM Post-Training on an 8GB GPU: Understanding KL, SFT, DPO, GRPO and DeepSeek-Style Reasoning with Open-Source Frameworks Using open-source training frameworks (HuggingFace TRL) and minimal, reproducible experiments to see — one b…
From Academic Research to a Frontier LLM: A Case Study in DPO [video] (www.youtube.com via hn) About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
Karpathy's autoresearch, 50 DPO experiments, 300 human judges (huggingface.co via hn) When does autoresearch need a human? Autonomous research agents are everywhere in AI research workflows now.
I tried a selective training method for hallucination — beats DPO and SFT with ~10% data (www.reddit.com) github link : genji970/hallucination-mitigation-via-contrastive-sampling-method: Selective contrastive post-training for hallucination mitigation in LLMs — improves factuality with ~10% data. ## Experimental Results ### (a) DPO vs.
DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimization (arxiv.org) Automated depression screening from clinical interviews requires attribution of utterances to the clinician or patient. We evaluate two datasets: DAIC-WOZ, where participant-only recordings require re-synthesizing both sides for controlled…
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data (arxiv.org) Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly.
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation (arxiv.org) Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions o…
You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs (arxiv.org) Ask a language model to respond "very excitedly," and its output is typically only mildly more energetic. We quantify this effect.
Beyond Single-Negative Preference: Multi-Negative DPO for LLM-Centric Historical Entity Linking (arxiv.org) Large language models (LLMs) have recently shown promise for historical entity linking, but preference optimization for this task is often formulated with only one negative candidate per training instance. This discards information from th…
Inducing Emergent Misalignment from Reward Hacks with Iterative DPO (arxiv.org) Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and counterme…
Steering Geometry: Validating Human Value Geometry in LLM Steering Space (arxiv.org) As large language models (LLMs) are increasingly deployed in alignment-sensitive contexts, activation steering has emerged as a lightweight, inference-time alternative to fine-tuning methods (e.g., RLHF, DPO) for behavioral control. Howeve…
Inference-Time Nash Alignment (arxiv.org) Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models.
Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO (arxiv.org) Language models can ignore prompt evidence when it conflicts with memorized knowledge. Post-training can make models follow such evidence more reliably, but it is unclear whether these gains require new machinery or strengthen machinery al…
LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO (arxiv.org) Multimodal search agents answer visual questions by interleaving image understanding, web retrieval, tool use, and evidence synthesis. Strong systems exist, but in two expensive regimes: proprietary frontier models such as GPT-5 and Gemini…
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization (arxiv.org) Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause h…
Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration (arxiv.org) Arabic large language models must refuse harmful prompts without over-refusing benign or sensitive prompts, yet a single refusal rate hides this trade-off. We evaluate it using benign refusal B and harmful-prompt refusal H, where H measure…
Disentangling Optimization Scale from Preference Scale in DPO (arxiv.org) Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient $\beta$ commonly interpreted as controlling the KL constraint to a reference policy. We show that $\bet…
Token-weighted Direct Preference Optimization with Attention (arxiv.org) Direct Preference Optimization (DPO) aligns Large Language Models with human preferences without the need for a separate reward model. However, DPO treats all tokens in responses equally, neglecting the differing importance of individual t…
VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models (arxiv.org) How precisely can we tell a language model how to feel? Most work on emotional generation answers with a discrete label - happy, angry, sad - which cannot express a target like "mildly downcast but calm." We instead specify the desired aff…
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design (arxiv.org) Cyclic peptides are emerging as promising molecular scaffolds in drug discovery due to their high binding affinity and structural stability. However, extending generative models from linear to cyclic peptide design remains challenging, as…
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment (arxiv.org) Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored.
Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training (arxiv.org) Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a re…
Cat-DPO: Category-Adaptive Safety Alignment (arxiv.org) Aligning large language models with human preferences must balance two competing goals: responding helpfully to legitimate requests and reliably refusing harmful ones. Most preference-based safety alignment methods collapse safety into a s…
Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs (arxiv.org) Achieving effective self-correction, where models verify and correct their own mistakes, remains a fundamental challenge for large language models (LLMs). In this work, we propose Self-Fix Step-DPO (SFS-DPO), a reinforcement learning based…
Agent Safety Should Be a Runtime Contract (arxiv.org) The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages…
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization (arxiv.org) Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to th…
Learning When to Reason for Text-to-SQL via SFT and DPO (arxiv.org) Recent Text-to-SQL methods rely heavily on reasoning-centric paradigms such as Chain-of-Thought (CoT), achieving substantial gains on complex benchmarks at the cost of high inference-time overhead. However, a large fraction of real-world q…
TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue (arxiv.org) The sycophancy of large language models can increase the safety risk in intervention dialogue for autistic children. Supervised fine-tuning can somewhat reduce sycophancy, but relying solely on positive examples is often insufficient to id…
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization (arxiv.org) Direct Preference Optimization (DPO), the efficient alternative to PPO-based RLHF, falls short on knowledge-intensive generation: standard preference signals from human annotators or LLM judges exhibit a systematic verbosity bias that rewa…
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories (arxiv.org) Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns t…
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs (arxiv.org) Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over disprefe…
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming (arxiv.org) RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertai…
Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contexts (arxiv.org) Long-context dialogue systems suffer from state inertia, where models over-attend to history and fail to adapt to evolving intents. We demonstrate that standard alignment methods like DPO and even recent long-context optimization technique…
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable) (arxiv.org) Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimization (DPO) cleverly bypasses reward modeling by making an ex…
Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines (arxiv.org) Industrial LLM teams often ship behavior updates by repeatedly DPO-training a base model on sequences of related preference-data campaigns. The dominant failure mode in this regime is not always classical catastrophic forgetting: a pipelin…
Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study (arxiv.org) We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computatio…
AMD R9700 vs GB10 (www.reddit.com via reddit) I have a budget of 5K, and want to buy some gpus my requirement is 48gb+ vram, because I finetune small language model, perform DPO, in general tinkering/ development is my usecase. if you where in my shoe which among these would you get,…
Fine-tuned Qwen2.5-7B to 96% of Claude Haiku on a domain-specific task using ~$3 of API calls and zero human labelers (www.reddit.comhttps) Built a decision-reasoning engine (Orlog) and wanted to fine-tune a local model for it instead of paying per-call forever. The method (DV-DPO): Run a 3-voice council on each question, produce a synthesis Cross-examine: losing voices challe…
DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment (arxiv.org) P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization (arxiv.org) Ultra-ml-intern: huggingface/ml-intern's workflow as a Claude Code plugin (www.reddit.com) huggingface/ml-intern is HF's autonomous ML engineer — reads papers, audits datasets, ships SFT/DPO/LoRA/GRPO runs to HF Jobs. it's a standalone python harness with its own agent loop calling the Claude API.
Fine-tune Llama 2 with DPO (huggingface.co)