Inkling Open weights, ready to tinker An efficient open model for text, images, and audio, available to fine-tune. Model at a glance 975B total, 41B active Parameters Mixture of Experts Architecture 1M tokens Context window 64K / 256K on T…
#mixture-of-experts
41 items
Murati's Thinking Machines Releases Open-Weights 975B Parameter LLM (thinkingmachines.ai via hn) Abliterlitics: Benchmarks and Tensor Comparison for Heretic, Abliterlix, Huiui, HauhauCS for GLM 4.7 Flash (www.reddit.com) This is a follow up to the previous benchmark and tensor analysis of abliteration techniques across the Qwen model family. Same approach, same toolkit, new model family.
XiaomiMiMo MiMo-V2.5 (not pro) - Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters (www.reddit.com) https://huggingface.co/XiaomiMiMo/MiMo-V2.5 Interesting because unlike its bigger brother it can be run on "more human" configurations
Ask HN: HotPin – lossless 120B MoE inference on 24GB RAM (CPU, 50 loc) (news.ycombinator.com) I'm a mechatronics designer with a background in control systems, robotics, PCB design, and embedded hardware. I design physical systems: motors, sensors, microcontrollers, and real-time control loops.
Running Qwen 3.6 Locally on a Mac Mini M4 with 16GB RAM (maloyan.xyz via hn) Running Qwen 3.6 Locally on a Mac Mini M4 with 16GB RAM Two days ago Qwen open-sourced Qwen 3.6-35B-A3B — a 35-billion parameter Mixture of Experts model that only activates 3 billion parameters per token. It's Apache 2.0 licensed, ships w…
EMO: Pretraining mixture of experts for emergent modularity (allenai.org via hn) Today we're releasing EMO, a new mixture-of-experts (MoE) model pretrained end-to-end so that modular structure emerges directly from the data without relying on human-defined priors. EMO lets you use a small subset of its experts – just 1…
Empirical Research on Moe Safety Mechanisms (dealign.ai via hn) Independent research into the safety architecture of large-scale Mixture of Experts reasoning models. 200+ controlled experiments.
Mixture of Experts (Moe): How Transformers Scale Without Activating Everything (chizkidd.github.io via hn) Mixture of Experts (MoE): How Transformers Scale Without Activating Everything. - Mixture of Experts (MoE) is one of the main techniques used to scale modern language models without making every token pay the full computational cost of the…
The cut in the Mixture of Experts compute graph (idlemachines.co.uk via hn) Mixture of Experts looks like it's one of those few changes you can make to the architecture of a model that comes almost for free: many more parameters, barely any more compute. The forward pass is just a router, a softmax and a top-k.
Stratum: System-Hardware Co-Design with 3D-Stackable DRAM for Efficient Moe (dl.acm.org via hn) Abstract Abstract As Large Language Models (LLMs) continue to evolve, Mixture of Experts (MoE) architecture has emerged as a prevailing design for achieving state-of-the-art performance across a wide range of tasks. MoE models use sparse g…
Zyphra releases the ZAYA1-8B MoE model optimized for intelligence density (huggingface.co via hn) ZAYA1-8B ZAYA1-8B is a small mixture of experts language model with 760M active parameters and 8.4B total parameters trained end-to-end by Zyphra. ZAYA1-8B sets a new standard of intelligence efficiency for its parameter count through a co…
Multi-Modal Tumor Survival Prediction via Graph-Guided Mixture of Experts (arxiv.org) Large Language Models (LLMs) have displayed impressive capabilities in handling tasks that require few demonstration examples, making them effective few-shot learners. Despite their potential, LLMs face challenges when it comes to addressi…
Structure Aware Neural Architecture Search for Mixture of Experts (arxiv.org) Neural Architecture Search (NAS) has so far rarely been applied to Mixture-of-Experts (MoE) models, and existing MoE designs leave the alignment between experts and the structure of the data to emerge on its own. We propose an architecture…
N-gram vs Experts explained (www.reddit.com via reddit) Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary.
A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning (arxiv.org) Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks through an input-dependent gating mechanism. Existing interpretable MoE approaches, such as Mixture…
TielCoder's 22 GB 4-bit quant matches Opus4.6 medium on recent real life coding issues, surpassing KAT-Coder and Nail as strongest and fastest MoE picks. (www.reddit.comhttps) Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal.
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion (arxiv.org) Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost of training and evaluating models. Efficient Pareto front a…
TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts (arxiv.org) Multimodal time series forecasting has garnered significant attention for its potential to provide more accurate predictions than traditional single-modality models by leveraging rich information inherent in other modalities. However, due…
Mixture of experts architectures for machine learning interatomic potentials (arxiv.org) Machine Learning Interatomic Potentials (MLIPs) enable accurate large-scale atomistic simulations, yet improving their expressive capacity efficiently remains challenging. Here we systematically investigate Mixture-of-Experts (MoE) and Mix…
C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning (arxiv.org) Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs. However, real-world data often suffer from missing modalities due to transmission errors or user behavior, seve…
Incast-Free MoE Rate-Based Scheduling (arxiv.org) Mixture of Experts (MoE) architectures have become key to large language models; however, their typical round-robin (RR) scheduling introduces significant bottlenecks. In this paper, we demonstrate that RR causes a previously-undiscovered…
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models (arxiv.org) Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severe…
Varying-coefficient mixture of experts model for dynamic heterogeneous populations: application to mouse cortical development (arxiv.org) As cells differentiate, gene-gene associations may change. Because the composition of cell subtypes also shifts with development, it is challenging to establish whether those changes reflect real changes in gene regulation within one or mo…
ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts (arxiv.org) Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs). However, MoE inference on conventional hardware is constrained by three fundamental bottlenecks.
Image classification via a quantum-inspired strategy involving a mixture of experts (arxiv.org) Pattern recognition problems arise in a variety of physical image processing situations, and convolutional neural networks are a popular scheme for the required feature extraction and classification tasks. The classical networks use diffus…
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities (arxiv.org) In this paper, we address the problem of multimodal federated learning with missing modality. Existing methods utilize an additional public dataset or perform naive feature synthesis that is based solely on the available modality.
MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts (arxiv.org) Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale…
Multi-scale Mixture of World Models for Embodied Agents in Evolving Environments (arxiv.org) Embodied agents operating in the real world require multi-scale reasoning and knowledge adaptation as conditions change. We identify two challenges in applying Mixture of Experts (MoE) to this setting: routing lacks an explicit notion of s…
A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution (arxiv.org) Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning (arxiv.org) Federated Learning (FL) emerged as a promising distributed machine learning paradigm. However, extending FL to the class incremental learning scenarios introduces unique challenges: 1) Capacity conflict and catastrophic forgetting from the…
Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach (arxiv.org) As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consiste…
Multi-Modal Spatio-Temporal Graph Neural Network with Mixture of Experts for Soil Organic Carbon Prediction (arxiv.org) Top-soil organic carbon (SOC) prediction is fundamental to agricultural sustainability, land use policy and fertilization planning. Existing approaches face two limitations: they pair hand-crafted covariates with classical ML or single-mod…
Generalizing GNNs with Tokenized Mixture of Experts (arxiv.org) Deployed graph neural networks (GNNs) are frozen at deployment yet must fit clean data, generalize under distribution shifts, and remain stable to perturbations. We show that static inference induces a fundamental tradeoff: improving stabi…
Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training (arxiv.org) Multivariate time-series data often exhibit complex temporal dependencies, irregular sampling, and heterogeneous dynamics across multiple time scales, making accurate sequence modeling particularly challenging. Traditional recurrent neural…
Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling (arxiv.org) CoRe-MoE: Contrastive Reweighted Mixture of Experts for Multi-Terrain Humanoid Locomotion with Gait Adaptation (arxiv.org) Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps (arxiv.org) FAME: Forecastability-Aware Mixture of Experts for Heterogeneous Time Series Forecasting (arxiv.org) Large-scale retail and industrial forecasting systems contain many heterogeneous time series whose lifecycle, sparsity, volatility, seasonality, spectral patterns, and contextual sensitivity differ substantially. A single forecasting model…
I built a PyTorch MoE/MoD training framework with custom CUDA kernels [Apache 2.0] (www.reddit.com via reddit) PyTorch framework for training transformer LLMs with MoE and MoD architecture support, custom CUDA kernels, and DeepSpeed integration. Key things it does: - Custom CUDA kernels for RMSNorm, RoPE, SwiGLU, MoE routing.
Mixture of Experts (MoEs) in Transformers (huggingface.co) Welcome Mixtral - a SOTA Mixture of Experts on Hugging Face (huggingface.co)