AI for the board game Diplomacy

The agents cooperate better through communication and negotiation, and agreeing to the broken promises that help keep them honest

Successful communication and cooperation was very important to help societies in history. Closed tables of table games can be a sandbox for modeling and investigating interaction and communication – and we can learn a lot of playing. In our last research, I published today in Nature Communications, we explain how artificial agents can use communication for better cooperation in the pastula games diplomacy, a vibrant field in artificial intelligence research (AI), known for its focus on building the alliance.

Diplomacy is a challenge because it has simple rules, but the high complexity arising due to the strong interdependence between the players and its huge work space. To help solve this challenge, we have designed negotiating algorithms that allow agents to communicate and agree on joint plans, and enable them to overcome agents who lack this ability.

Cooperation is a particular challenge when we cannot rely on our peers to do what they promise. We use diplomacy as a sand fund to explore what is happening when agents deviate from their previous agreements. Our research shows the risks that appear when complex agents are able to distort their intentions or mislead others regarding their future plans, which leads to another big question: What are the conditions that enhance reliable communication and teamwork?

We explain that the sanctions strategy on their peers who break the contracts greatly reduces the advantages that they can obtain by giving up their obligations, thus enhancing the most honest communication.

What is diplomacy and why is it important?

Games such as chess, poker, Go and many video games have always been fertile floor of artificial intelligence research. Diplomacy is a negotiating game of seven players and the formation of an alliance, which is played on an old map of Europe that divides it into provinces, where each player controls multiple units (diplomatic rules). In the standard version of the game, called the press diplomacy, each turn includes a negotiating stage, after which all players reveal their chosen movements at the same time.

The heart of diplomacy is the stage of negotiation, where the players try to agree on their following moves. For example, one unit may support another unit, allowing it to overcome resistance by other units, as shown here:

The calculations of diplomacy have been discussed since the 1980s, many of which have been explored on a simpler copy of the game called No-Press diplomacy, as strategic communication is not allowed between the players. The researchers also suggested computer -friendly negotiation protocols, which are sometimes called “restricted pressure”.

What did we study?

We use diplomacy as proof of negotiation in the real world, and providing ways for artificial intelligence agents to coordinate their movements. We take diplomatic agents other than communicating and increasing them to play diplomacy with communication by giving them a protocol to negotiate contracts for a joint work plan. We call these strengthened agents as the basic negotiations, and they are obligated to their agreements.

We consider a protocol: the mutual proposal protocol and the proposed protocol, which was discussed in detail in the full paper. Our agents apply algorithms that define useful deals for both parties by simulating how the game evolves in different decades. We use Nash’s solution from the theory of the game as an initial basis to determine high -quality agreements. The game may be revealed in several ways according to the actions of the players, so the Monte-Carlo simulation agents use to find out what may happen at the next turn.

Our experiences show that our negotiating mechanism allows primary negotiators to greatly outperform the basic factors other than communication.

Agents breaking agreements

In diplomacy, the agreements that were made during negotiation are not binding (“cheap talk”). But what happens when the agents who agree to a contract at one of the turns about it in the next next? In many real places, people agree to act in a certain way, but they fail to meet their obligations later. We used diplomacy to study how to give up our obligations to erode trust and cooperation, and to determine the conditions that enhance honest cooperation.

Therefore, we consider deviation agents, who overcome the main and honest negotiators by deviating from the agreed contracts. Simple deviations simply “forget” agreed to a contract and move as they wish. Conditional deviations are more advanced, and improving their actions on the assumption that other players who accepted a contract will act accordingly.

We show that simple and police deviations greatly outperform the main negotiators, and the conditional deviations by an overwhelming majority.

Fansly agents to be honest

Next, we deal with the problem of deviation using defensive agents, which negatively respond to deviations. We verify bilateral negotiators, who simply cut contacts with agents who break with them. But the suspicion is a slight reaction, so we also develop penal agents, who do not receive betrayal lightly, but instead, we appreciate their goals to try to reduce the value of deviation – discount with a grudge! We explain that both types of defensive factors reduce the advantage of deviation, especially penalties.

Finally, we offer the deviations learned, who adapt and improve their behavior against sanctions agents on multiple games, trying to make the defenses above less effective. The deviation that is used will only break the contract when the immediate gains from the deviation are high enough and the ability of the other agent to take revenge is low enough. In practice, educated deviations sometimes break the contracts late in the game, thus achieving a slight advantage over sanctions agents. However, these sanctions push the deviation to honor more than 99.7 % of their contracts.

We also study the potential learning dynamics of penalties and deviation: what happens when penaltors deviate from contracts, and the potential incentive to stop the punishment when this behavior is expensive. These problems can gradually wear cooperation, so additional mechanisms such as repetition of interaction through multiple games or the use of confidence and reputation systems may be needed.

Our paper leaves many questions open to future research: Is it possible to design more sophisticated protocols to encourage the most honest behavior? How can one combine communication technologies and complete information? Finally, what other mechanisms can deter the agreements? Building a fair, transparent and confidential artificial intelligence systems is a very important topic, and it is a major part of the Deepmind mission. The study of these questions in sand boxes, such as diplomacy, helps us to understand the tensions between cooperation and competition that may exist in the real world better. In the end, we believe that addressing these challenges allows us to better understand how to develop artificial intelligence systems in line with the values ​​and priorities of society.

Read our full paper here.