Mixture of Experts (MoE)

Technical Overview of Mixture of Experts (MoE)

Mixture of Experts (MoE) is a sparse transformer architecture that replaces dense feed-forward network (FFN) layers with multiple independent “expert” networks.

Dense models process every token using all parameters, which increases computational requirements. MoE addresses this by using a gating network (router) to dynamically assign each token to a subset of experts. This increases overall model capacity while keeping the computational cost per token relatively constant.

Key Architecture & Implementation

The MoE architecture consists of two primary components:

1. Sparse MoE Layers:
   • A set of independent feed-forward networks (e.g., 8 or 16 experts per layer).
2. Gating Router:
   • A small parameterized network that calculates a probability distribution over the experts for each token.
   • Top-K Routing: Most models activate only the top-1 or top-2 experts per token ($K=1$ or $K=2$), keeping other expert networks inactive.

Input Token ---> [ Gating Router ] ---\
                   | (Top-2)         |
                   v                 v
             [ Expert 1 ]      [ Expert 4 ]  (Others remain inactive)

Core Parameters

• Active Parameters: The number of parameters used to process a single token (e.g., 12.9B active parameters in a model with 47B total parameters).
• Load Balancing: The router must distribute tokens evenly across experts to prevent hardware bottlenecks.

Real-world Applications

• Used in models like Mixtral 8x7B, Mixtral 8x22B, and GPT-4 to achieve high performance with lower compute budgets.