What are A2C and A3C?

A2C (Advantage Actor-Critic) and A3C (Asynchronous Advantage Actor-Critic) are reinforcement learning algorithms belonging to the Actor-Critic family.

• Actor-Critic: consists of two components — the actor, which chooses actions, and the critic, which evaluates the quality of the chosen actions (usually through a value function).

• A2C — is a synchronous version where multiple agents (or threads) collect experience in parallel, but model updates occur synchronously, averaging gradients.

• A3C — is an asynchronous version where multiple agents learn in parallel and update a shared model independently, which speeds up learning and improves exploration of the state space.

Both algorithms use the advantage function to reduce gradient variance and improve training stability.

An example of a simplified A2C/A3C structure:

# pseudocode
for each agent in parallel:
    while not done:
        action = actor(state)
        next_state, reward = env.step(action)
        advantage = reward + gamma * critic(next_state) - critic(state)
        update actor using advantage
        update critic to minimize TD error
        state = next_state