What is Mixture of Experts (MoE) and what are its advantages and disadvantages?

Mixture of Experts (MoE) is a neural network architecture that consists of multiple specialized models (experts) and a routing mechanism (gate) that selects which experts to activate for processing a specific input. The idea is for each expert to be trained on its part of the task, while the gate dynamically distributes computations, allowing models to scale without proportional increases in computational costs.

Pros:

• Efficiency: Very large models can be used by activating only a subset of experts for each request, reducing computational load.
• Specialization: Experts can specialize in different aspects of data, improving quality.
• Scalability: New experts can be added easily without retraining the entire model.

Cons:

• Training complexity: It is challenging to train the gate effectively so that it correctly distributes requests.
• Uneven load: Some experts may be overloaded, while others are idle.
• Generalization issues: If the gate makes mistakes, the quality may deteriorate.

An example of a simplified pseudocode scheme:

experts = [Expert1(), Expert2(), Expert3()]
gate = GateNetwork()

input = get_input()
weights = gate(input)  # probabilities for each expert
output = sum(w * expert(input) for w, expert in zip(weights, experts))

In modern LLMs, MoE helps create very large models with fewer resource costs.