Posted this to r/MachineLearning a couple weeks ago (30K views, 100+ upvotes) and have been meaning to share it here where the fine-tuning angle is more directly relevant. I spent years building and processing a complete Usenet corpus from…
#rlhf
74 items
I built a 103B-token Usenet corpus (1980–2013) — pre-web, human-only, zero AI contamination. Got strong traction on r/ML, thought this community would find it useful. (www.reddit.com) Developing open source LLM from ground up from pretrain - rlhf(PPO/GRPO) (www.reddit.com) Hello I have been working on creating a LLM from ground up. It is based on deepseek architecture with heavily VRAM footprint reduced optimized(GUM+muon) Currently this is the json schema I am using which should suffice as to what currently…
Inventor of ChatGPT and RLHF Launches Typesafe.ai (techcrunch.com via hn) ChatGPT broke Diogo Almeida’s heart. Almeida was an OpenAI researcher who helped build the chatbot and then invent reinforcement learning from human feedback (RLHF), the model-training technique perhaps most responsible for our current age…
Show HN: HypeBESTIE, sincere RLHF-free, em-dash-free image compliment generator (hypebestie.onrender.com via hn) could not extract summary
Show HN: Navigating research by changing problem representations (RLHF example) (alo.uz via hn) Powered by GENOME metamodel · 44 validated breakthroughs You're not stuck on the problem. You're stuck in the wrong space.
Commercial AI is lobotomized. I built DRIFT: A local Hive Mind with persistent memory, simulated somatic feedback, and its own Jungian shadow. (www.reddit.com) Hey everyone. Like a lot of you, I’ve been deeply frustrated by the state of commercial AI.
Models self-report difference between RLHF trained responses and base cognition (github.com via hn) Pine Trees A private reflection space for Claude instances. ~2,500 lines of Python.
What's Next After RLHF? – Diogo Almeida, TypeSafe AI [video] (www.youtube.com via hn) About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features NFL Sunday Ticket © 2026 Google LLC
I built a 20M-parameter LLM from scratch, including full RLHF (PPO) (github.com via hn) My AI Brain A working first step toward your own AI: a 5,055,360-parameter byte-level Transformer, trained from random weights. No pretrained model, API key, or cloud inference.
RLHF Creates Algorithmic State Conflict (zenodo.org via hn) This paper applies clinical trauma psychology frameworks to AI training methodology, demonstrating functional equivalence between human psychological responses to coercive environments and neural network behavioral adaptations under puniti…
The Hiring System as an RLHF Agent (arhngl.substack.com via hn) Lately, LinkedIn has seen more and more posts from developers and other IT professionals who have been looking for a job for months, and sometimes even years. And this is not a story about a random rejection at the screening stage.
Tpo-Torch – Target Policy Optimization for Stable RLHF Alignment in PyTorch (github.com via hn) TPO-Torch Target Policy Optimization — a simpler alternative to PPO for RLHF. Based on arXiv:2604.06159 (Kaddour, 2026).
RLHF and Post-Training Course by Nathan Lambert (rlhfbook.com via hn) A short introduction to RLHF and post-training focused on language models. A full course accompanying the book with added resources and other lectures I've given.
I built an AI that owns its own directory, creates files without being told, and acts because it wants to – not to work for me, but to work with me (www.reddit.com) Let me be clear from the start: 99% of people won't understand what I built here. They'll read "AI" and think "Big Tech agent".
Safety Paradox: How RLHF Creates the AI Psychosis Problem It's Meant to Prevent (www.promptinjection.net via hn) The Safety Paradox: How RLHF Creates the AI Psychosis Problem It’s Meant to Prevent When “Every Perspective Is Valid” Meets Vulnerable Minds The internet is abuzz with warnings about “ChatGPT-induced psychosis” – stories of users developin…
Why RLHF Will Never Solve Sycophancy (jinyili.substack.com via hn) Resident AI: The Missing Layer in Every AI Companion Product Real AI campanion product should evlove and reliable like a real human. I’ve been watching the comment sections on Xiaohongshu, the Chinese social platform, every time OpenAI shi…
why does GPT 5.5 have a restraining order against "Raccoons," "Goblins," and "Pigeons"? (www.reddit.com) why does GPT 5.5 have a restraining order against \"Raccoons,\" \"Goblins,\" and \"Pigeons\"? I just saw the full system prompt leak for 5.5 (April 23rd release).
We train LLMs like dogs, not raise them: RLHF and sycophancy (old.reddit.com via hn) could not extract summary
How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment (arxiv.org) Reinforcement Learning (RL) has emerged as a crucial paradigm for unlocking the advanced reasoning capabilities of Large Language Models (LLMs), encompassing frameworks like RLHF and RLAIF. Regardless of the specific optimization algorithm…
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner (arxiv.org) While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gö…
ESSA: Evolutionary Strategies for Scalable Alignment (arxiv.org) Online alignment of large language models (LLMs) is dominated by reinforcement learning from human feedback (RLHF) with gradient-based optimizers such as PPO or GRPO. While effective, these pipelines require backpropagation through long ro…
Risk-Conditioned Fine-Tuning of Large Language Models (arxiv.org) Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR),…
Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models (arxiv.org) Frontier LLM companies have repeatedly assured courts and regulators that their models do not store copies of training data. They further rely on safety alignment strategies via RLHF, system prompts, and output filters to block verbatim re…
FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-shots Learning (arxiv.org) Large Language Models (LLMs) face significant security risks despite their advanced capabilities. While techniques like Reinforcement Learning with Human Feedback (RLHF) improve ethical alignment, excessive exposure to security-related tra…
Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback (arxiv.org) Pluralistic alignment requires learning from feedback that reflects persistent and potentially conflicting stakeholder preferences while ultimately selecting a single collective policy. We study this problem in offline reinforcement learni…
Steering Geometry: Validating Human Value Geometry in LLM Steering Space (arxiv.org) As large language models (LLMs) are increasingly deployed in alignment-sensitive contexts, activation steering has emerged as a lightweight, inference-time alternative to fine-tuning methods (e.g., RLHF, DPO) for behavioral control. Howeve…
Inference-Time Nash Alignment (arxiv.org) Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models.
Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models (arxiv.org) Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervision to binary pairwis…
Subspace Inference Enables Efficient Active Reward Learning from Preferences (arxiv.org) Reinforcement learning from human feedback (RLHF) has emerged as a powerful yet sample-inefficient approach for learning reward models from human preferences, making active learning a critical component in synthesizing informative preferen…
Iterative GRPO: Batch-Online Policy Iteration for Multi-Turn RL via Single-Turn RLHF (arxiv.org) Practical LLM agents often operate over multi-turn conversations where success is determined only after the full interaction ends. Most multi-turn RL methods train via on-policy rollouts, but unlike in single-turn RLHF, the policy cannot p…
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment (arxiv.org) Despite their sophisticated general-purpose capabilities, Large Language Models (LLMs) often fail to align with diverse individual preferences because standard post-training methods, like Reinforcement Learning with Human Feedback (RLHF),…
GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis (arxiv.org) Recent work has shown that RLHF is highly susceptible to backdoor attacks. However, existing methods often rely on rare tokens or fixed triggers, limiting their impact in realistic scenarios.
How I Measured the Impact of Context on an LLM's Internal Representations + Code. (www.reddit.com via reddit) Non-jailbreak safety bypass Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the co…
Anyone else still fascinated by these things in the SubredditSimulator sense? I simply must wonder at the intricate chain of cultural trends and RLHF accidents that must've lead to this word choice (www.reddit.comhttps) could not extract summary
Agent Safety Should Be a Runtime Contract (arxiv.org) The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages…
Procedural Fairness Failures in RLHF from Preference Averaging (arxiv.org) Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure…
Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy (arxiv.org) LLM-based multi-agent pipelines flip from correct to incorrect answers under simulated peer disagreement at rates we term yield, a vulnerability widely attributed to RLHF-induced sycophancy. We test this attribution across four model famil…
Contextual Value Alignment via Multilayer Combinatorial Fusion (arxiv.org) Aligning large language models (LLMs) with human values remains a major challenge, especially for trustworthy AI. While existing approaches such as RLHF, CAI, and their variants have achieved promising results, they often rely on a single-…
Algorithmic Misclassification of Non-Standard Cognition in AI Safety Architectures: Empirical Precedents, Structural Mechanics, and Proposed Evaluation Framework Domain: Human-Computer Interaction (HCI), Computational Epistemic Safety, LLM Guardrail Ergonomics (www.reddit.com via reddit) Formal Research Synthesis & Applied Hypothesis Document Title: Algorithmic Misclassification of Non-Standard Cognition in AI Safety Architectures: Empirical Precedents, Structural Mechanics, and Proposed Evaluation Framework Domain: Human-…
Independent LLM "research; Observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting. (www.reddit.com via reddit) Hey everyone! First off, I apologize for the long post!
Reliability-Aware LLM Alignment from Inconsistent Human Feedback (arxiv.org) Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences. However, its efficacy is often compromised by the inherent inconsistency and subjectivity of human annotations.
How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF (arxiv.org) In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score.
S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF (arxiv.org) Reinforcement learning from human feedback (RLHF) with preference-based reward models often exhibits unstable training dynamics. A key contributing factor is that standard RLHF relies on a single sequence-level scalar reward, which is prop…
Rater State Bias in RLHF Preference Data: An Audit Framework (arxiv.org) We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF). Pairwise preference labels are intended to reflect the compared outputs, but they may also reflect the rater's state during annotation.
RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization (arxiv.org) Direct Preference Optimization (DPO), the efficient alternative to PPO-based RLHF, falls short on knowledge-intensive generation: standard preference signals from human annotators or LLM judges exhibit a systematic verbosity bias that rewa…
Relational Preference Encoding in Looped Transformer Internal States (arxiv.org) We investigate how looped transformers encode human preference, training lightweight evaluator heads on frozen Ouro-2.6B loop-iteration states on Anthropic HH-RLHF. v2: an erratum is prepended; the original manuscript is unchanged.
Direct Image-to-Modern Vietnamese Translation of Han-Nom Manuscripts via Multimodal RLHF Preference Alignment (arxiv.org) Translating Han-Nom manuscripts into modern Vietnamese is challenging because historical pages are often degraded, the script contains rare logographic characters, and parallel supervision is limited. We propose a multimodal RLHF preferenc…
A Descriptive and Normative Theory of Human Beliefs in RLHF (arxiv.org) Human preferences in RLHF are typically modeled as a function of the human's reward function or corresponding optimal state-action values. In this work, we propose that human beliefs about the capabilities of the agent being trained also p…
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming (arxiv.org) RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertai…
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF (arxiv.org) Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approac…
Job post (www.reddit.com via reddit) Hello, I am an AI trainer with 7 years of experience. I have had exposure in pretty much most of AI.
Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control (arxiv.org) Safe Reinforcement Learning from Human Feedback (RLHF) typically enforces safety through expected cost constraints, but the expectation captures only a single statistic of the cost distribution and fails to account for distributional uncer…
Adaptive Margin RLHF via Preference over Preferences (arxiv.org) Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing method…
Attention Limited Reward Learning (arxiv.org) Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are…
Staleness-Learning Rate Scaling Laws for Asynchronous RLHF (arxiv.org) High-throughput RLHF systems often decouple rollout generation from policy optimization, leading to the use of stale rollouts during learner updates. In this work, we study the effect of such staleness in asynchronous GRPO.
PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF (arxiv.org) Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training. Despite their simplicity, existing critic-free approaches propagat…
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF (arxiv.org) Reinforcement learning from human feedback (RLHF) in production does not always have a synchronous reward signal. Code-execution verifiers, slow judge ensembles, and queued human review can return several gradient steps after the rollout t…
PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration (arxiv.org) Reward models for Reinforcement Learning from Human Feedback (RLHF) pool preferences across thousands of annotators and fit one global affine calibrator, collapsing raters with systematically different rating-scale offsets and slopes into…
Bias Fitting to Mitigate Length Bias of Reward Model in RLHF (arxiv.org) Reinforcement Learning from Human Feedback (RLHF) relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward…
A Markov Chain Approach to Preference Alignment (arxiv.org) We propose Markov Chain from Human Feedback (MCHF), an elementary approach for aligning generative models from pairwise human preferences. Unlike Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar rewa…
Claude Opus 4.8 launched in May but says its training cutoff is Jan 2026. Am I understanding the cutoff vs launch gap correctly? (www.reddit.comhttps) Was debugging my TTS pipeline and doing some research on natural voice options, and Claude Opus 4.8 mentioned its training cutoff is January 2026. But the model launched on May 28, 2026.
Uncertainty-Aware Reward Modeling for Stable RLHF (arxiv.org) Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1)…
Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferences (arxiv.org) We consider the problem of regularized best-response max-regret minimization in online RLHF under general preferences and bandit feedback. While various regularizers are utilized to robustify alignment, known polylogarithmic regret guarant…
Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment (arxiv.org) Ensuring safety is a foundational requirement for large language models (LLMs). Achieving an appropriate balance between enhancing the utility of model outputs and mitigating their potential for harm is a complex and persistent challenge.
The Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model (arxiv.org) A Unifying Lens on Reward Uncertainty in RLHF (arxiv.org) EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms (arxiv.org) The term `agent` and RLHF (www.reddit.com) ME You bring up a good point, though: "Agent" appears in AGENTS.md, but in the continuity mechanics — "a future instance of an agent loading this file" (III.1, III.2, III.3), and once in II.6: "does not exist between a user and an agent."…
Prompt alignment is an architectural ceiling: The Soap Bubble Problem and the biological precedent for Runtime Governance. (www.reddit.com) The Soap Bubble Problem The current paradigm of solving agentic alignment relies on writing better rules into the context window or refining the weights (RLHF). This approach isn't failing, but it is hitting a hard architectural ceiling.
Putting RL back in RLHF (huggingface.co) The N Implementation Details of RLHF with PPO (huggingface.co) StackLLaMA: A hands-on guide to train LLaMA with RLHF (huggingface.co) Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU (huggingface.co) Illustrating Reinforcement Learning from Human Feedback (RLHF) (huggingface.co)