The landscape of artificial intelligence (AI) in strategic games has witnessed groundbreaking achievements, with AI conquering complexities in games like chess and Go. However, a new milestone has been achieved with Cicero, an AI that exhibits human-level performance in the multifaceted board game Diplomacy, a realm that involves not just strategy, but the nuances of negotiation and human interaction.
Diplomacy, unlike traditional two-player zero-sum games, demands a blend of cooperation and competition, making it a challenging domain for AI. Players in Diplomacy engage in private negotiations, forming alliances and plotting betrayals, requiring a deep understanding of human behavior and strategy.
Cicero represents a leap in AI development, integrating a sophisticated language model with strategic reasoning. It understands and infers players’ intentions, crafting its strategies accordingly. In an anonymous online league, Cicero distinguished itself, ranking in the top 10% of players, a testament to its advanced capabilities.
Developing Cicero involved overcoming significant challenges inherent in multi-agent environments. The AI needed to balance competitive tactics with cooperative strategies and emulate human-like negotiation skills, a complex task far removed from the straightforward objectives of games like chess.
In the context of AI for Diplomacy, grounding refers to the AI's ability to base its decisions and communications on the current context or reality of the game. It's about linking the AI's dialogue and strategy to the actual state of the game environment, which is crucial for realistic and effective gameplay.
Cicero excels in grounding by integrating its strategic planning with context-aware dialogue generation. The AI analyzes the current game state, past moves, and player interactions to generate contextually relevant strategies and communications. This allows Cicero to engage in negotiations and form alliances that are reflective of the actual game dynamics, making its gameplay more realistic and effective.
Cicero's planning engine is at the heart of its strategic decision-making. It is responsible for modeling how other players are likely to act based on the current state of the game and the ongoing conversations. This engine combines elements of strategic reasoning with reinforcement learning, tailored specifically for the multi-faceted environment of Diplomacy.
The core algorithm driving Cicero’s planning engine is based on a combination of reinforcement learning techniques and the piKL (policy iterative Kullback-Leibler) algorithm. The piKL algorithm is instrumental in predicting players’ actions, which are crucial for Cicero to plan its own moves. This predictive modeling involves understanding the likelihood of various actions in the game and choosing an optimal action for Cicero based on these predictions. The planning process is dynamic, recalculating strategies as the game progresses and new information from player interactions is received.
Cicero’s strategic reasoning module plays a pivotal role in generating intents for dialogue and deciding the final actions for each turn. It functions by predicting other players' policies – essentially, a probability distribution over their possible actions based on the current state of the board and shared dialogue. Cicero then determines its own policy that best responds to these predicted policies.
In modeling human players, Cicero initially uses a technique known as behavioral cloning, which involves learning from human gameplay data to predict human-like policies. However, this method can be unreliable if there is deception in the dialogue or if plans change after the dialogue.
The piKL algorithm is a sophisticated method that Cicero uses to address the shortcomings of behavioral cloning. It’s an iterative algorithm that predicts player policies by balancing two objectives:
pi).pi) and a baseline or anchor policy (ti), determined by behavioral cloning.The piKL algorithm operates under a modified utility function, defined as:
Ui(pi) = ui(pi, p−i) − λDKL(pi∥ti)
Here, p−i represents the policies of all players other than player i, and ui(pi, p−i) is the expected value of policy pi given other players’ policies. The term λDKL(pi∥ti) represents the divergence from the anchor policy, penalizing deviations from human-like behavior.
pDt_i(ai) = ti(ai) * exp(Qt-1_i(ai) / λ)
This formula adjusts the probability of choosing action ai for player i at iteration t. It combines the anchor policy probability ti(ai) with an exponential term that favors actions with higher utility, as assessed in the previous iteration.
pt = ((t-1)/t) * pt-1 + (1/t) * pDt
This formula updates the policy prediction at iteration t, gradually incorporating the new temporary policy pDt into the overall policy prediction.
A key feature of piKL is its ability to strike a balance between optimizing game strategies and maintaining behavior that resonates with human norms. For example, in a scenario where Cicero must decide between forming an alliance or pursuing a competitive strategy, piKL evaluates these options not just for their immediate strategic value but also for their alignment with typical human gameplay.
Behavioral cloning in piKL serves as a benchmark for human-like behavior, guiding Cicero's strategies to stay within the realm of human norms. This integration allows Cicero to learn from human gameplay patterns, ensuring its strategies are relatable and effective.
Training the piKL algorithm involved a comprehensive analysis of human gameplay data, which provided insights into diverse strategic approaches. One challenge in this process was ensuring that Cicero’s strategies remained adaptable and did not overfit to specific gameplay patterns.
During gameplay, Cicero uses piKL to select intent actions for the current turn, balancing honesty and strategic coordination. For future turns, intents are based on human-imitation models, aligning Cicero's actions with typical human strategies.
The anchor policy in Cicero’s AI serves as a baseline for human-like behavior in the game of Diplomacy. It guides the AI's strategies to ensure they are consistent with typical human gameplay, helping Cicero to make decisions that are not only strategically sound but also relatable and understandable to human players.
The anchor policy is trained using behavioral cloning, a technique where the AI learns to mimic human behavior by analyzing and replicating patterns found in human gameplay data. This involves processing large datasets from actual Diplomacy games, allowing Cicero to observe and learn various strategies, moves, and negotiation styles employed by human players. The trained anchor policy then acts as a reference point, against which Cicero’s strategic decisions are compared and aligned.
The dialogue engine in Cicero is what allows it to communicate with human players effectively. It’s not just about generating text; the engine needs to produce dialogue that is relevant to the game's context, strategically beneficial, and aligned with Cicero’s current plans and intents.
Cicero’s dialogue engine is built upon a neural generative language model, trained specifically for the game of Diplomacy. This model was further refined to be controllable through a set of intents, representing planned actions for Cicero and its speaking partners. The training involved a dataset derived from human gameplay, ensuring that the dialogue generated is grounded in both the history of the game and the current state.
The dialogue engine operates by generating messages conditioned on the strategic plans formulated by the planning engine. This ensures coherence between Cicero’s strategic intents and its communicative actions. The integration of the dialogue engine with the planning engine is crucial, as it allows Cicero to negotiate, form alliances, and even deceive, just as human players do.
By understanding and elaborating on these two critical components of Cicero - the strategic planning engine and the dialogue engine - we gain deeper insights into the sophisticated AI mechanisms that enable Cicero to perform at human level in the complex game of Diplomacy.
In the sophisticated environment of the game Diplomacy, Cicero's AI utilizes a multi-model approach for generating and training intents, crucial for its strategic dialogue and decision-making.
The combination of these models allows Cicero to master the art of Diplomacy, skillfully blending human-like negotiation with astute strategic gameplay. The AI's ability to generate contextually relevant and strategically aligned intents sets it apart in the realm of advanced AI systems, showcasing a significant leap in AI's capabilities in handling complex, multi-agent environments.
CICERO: An AI agent that negotiates, persuades, and cooperates with peopleLink to blog
Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Link to paper Link to github
Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning. Link to paper
Modeling Strong and Human-Like Gameplay with KL-Regularized Search.Link to paper
No-Press Diplomacy from Scratch. Link to paper
Strategic Reasoning with Language Models. Link to paper
Diplomacy Cicero and Diplodocus. Link to github
No, CICERO has not "mastered" Diplomacy. Link to YouTube
Meta AI | Human-level Play in Diplomacy Through Language Models & Reasoning. Link to YouTube
Created 2023-12-19T10:50:33-08:00, updated 2024-02-06T05:28:51-08:00 · History · Edit