In mid-1980s, I was working on my Ph. D. dissertation at the University of Texas at Austin in the Department of Computer Sciences while refereeing soccer. My dissertation topic in Artificial Intelligence (AI) was “Identifying Visual Objects using General World Knowledge”. At the same time frame, I was one of the top officials of South Texas.
I never reached the pinnacle of either area. Neither did I receive a doctoral degree nor became a FIFA referee. Even though I served on the panel of FIFA Referee Instructors – as the only one on the panel without a FIFA Badge – for seven years.
It never crossed my mind that those two areas – Soccer Refereeing and AI – would cross each other in 2020’s. The use of technology in soccer refereeing started with use of Goal Line Technology. In July 2012, the International Football Association Board (IFAB) officially approved the use of goal line technology, amending the Laws of the Game (LOTG) to permit (but not require) its use. Since it is an expensive technology, it is only used at high level competitions.
The use of video review came into the game a bit later. Later than most team sports. The use of video assistant referees (VARs) in soccer was included in the LOTG in the 2018-19 edition.
In this article, we will investigate the extent of which soccer refereeing could be automated, like a car without a driver. Driverless cars are on the streets, how long will it take to have soccer games without referees? Do we really want every aspect of soccer refereeing to be automated? Do we want to automate all the decisions or just critical decisions like the VAR system? These are valid and sometimes philosophical questions.
There are two types of decisions in a soccer game: Objective and subjective decisions. Objective decisions are decisions like whether the ball is in play or not, whether a player is an offside position and whether team A or B should take the throw-in. The subjective decisions are decisions like whether a foul is careless (just a foul), reckless (yellow card) or using excessive force (red card), whether a foul in the penalty area by the defending team warrants a penalty kick and whether a player in an offside position is interfering with an opponent. Clearly, according to the LOTG,”Decisions will be made to the best of the referee’s ability according to the Laws of the Game and the ‘spirit of the game’ and will be based on the opinion of the referee, who has the discretion to take appropriate action within the framework of the Laws of the Game.”
Subjective decisions are clearly based on the “opinion” of the referee and hence is the hardest to automate.
The logical extension of the Goal Line Technology (GLT) will be the use of technology to decide whether a ball is in or out of play. This will be quite an expensive technology like the GLT, but it is clearly doable and does not require tools of AI. The key question here is whether it is financially feasible to implement this expensive technology for all in and out decisions when ball in and out critical questions can be addressed by the VAR system, which is dependent on a human being.
FIFA decided to use AI in deciding whether a player is in an offside position or not during the 2022 World Cup. This is another objective decision which would help the VAR/AVAR in reaching a faster decision during the process. AI is used to identify the parts of the body which leads to a decision whether a player is an offside position or not. LOTG says: “A player is in an offside position if: • any part of the head, body or feet is in the opponents’ half (excluding the halfway line) and • any part of the head, body or feet is nearer to the opponents’ goal line Than both the ball and the second-last opponent, the hands and arms of all players, including the goalkeepers, are not considered. For the purposes of determining offside, the upper boundary of the arm is in line with the bottom of the armpit.Without this technology, VAR or AVAR tries to manually identify the part of the body that might cause the player to be offside/onside. This is a time-consuming process and the use of AI-based technology expedites the process.
The next step would be to use technology to decide who played the ball last before the ball went out of bounds. Clearly, a similar AI based technology could be used to identify who played/touched the ball last. Since the VAR system is used for match critical incidents, this is not in the domain of VAR applications. But a corner kick missed, or a corner kick awarded instead of a goal kick might have some impact on the outcome of the game. So, the guardians of IFAB might consider using AI based to technology for corner kick/goal kick decisions to help the referee on the field. Since AI technology would expedite the decision, it should not steal any time off from the game clock. I do not see any value in the use of technology for deciding who should take the throw-ins.
Another use of AI for objective decisions would be to indicate whether a foul by the defending team was committed inside the penalty area or not. The system must find the point of contact and its projection onto the field of play.
All the above AI based technologies that have been developed and those I propose to be developed are to help the VAR system.
Subjective decisions are based on the opinion of the referee. In other team sports, they avoid subjective decisions from the domain of video replay, but soccer has included some subjective decisions that are critical match incidents in the domain of the VAR protocol. Where can AI or technology be used in the subjective decisions? One of the most controversial applications of the VAR system is what is called the On the Field Review (OFR). When should a VAR call the referee for an OFR? I believe an AI based system can help VARs to make more standard calls for an OFR while expediting the process.
Before going any further, one must understand the cognitive processes that the referee carries out while making a decision. It could be summarized as perception, decision, and execution. The critical part is the decision process. During this process, the referee uses and accesses Long Term Memory (LTM) elements to reach a decision. LTM is composed of two different memory types. Explicit and implicit memories. The explicit memory has semantic and episodic memories associated with it. Explicit memories are those that are consciously remembered. On the other hand, implicit memories are those that we remember unconsciously and are expressed in our behavior in some way. Most of our procedural memories fall into this category.
Let us relate these concepts to decision-making in refereeing. The letter of LOTG is stored in the semantic memory whereas the referees past experiences as well as hundreds of instructional clips watched over time are stored in the episodic memory. Leagues, Federations and Confederations as well as FIFA have hundreds of clips regarding various incidents with the “correct” decisions set. Using both semantic and episodic memory, the referee makes a decision and the execution of the decision — pointing to the penalty spot, raising a card etc. — is stored in the procedural memory. In the first phase, based on his/her visual perception, the referee accesses both his/her semantic and episodic memories to make a decision. Usually, the referee has no problem accessing the semantic memory, for example committing a pushing foul by a defender in his/her penalty area results in a penalty kick. If he/she does have a problem accessing the semantic memory, the referee might be committing a cardinal sin of misapplying the LOTG, which is extremely rare. The problem is classifying and accessing the episodic memory. This is where I believe using an AI-based system VAR can help the referee so that a critical match incident is not misinterpreted.
Let us say a there is contact between the hand of a defender and the ball in the penalty area. A correct decision can only be made using both semantic and episodic memory. The referee has to choose from his/her episodic memory the most appropriate incident for this contact and follow the advice of the governing body regarding this incident. This is the difficult part in the process. There might be some episodes that the referee might have a hard time to access effectively and identify the closest one.
An AI-based system can analyze the incident through tens of instructional clips regarding handball and various decisions by the same referee to come up with a probability of a decision. The system can tell the VAR that this is X% a penalty kick based on clips and similar decisions by the referee. Similar decisions of the referee are important since those are manifestations of how he/she interprets the Law. The final decision should also reflect the style of the referee. This will also end potential criticism of the referee as being inconsistent.
The governing body can choose a minimal/maximal percentage for the VAR to intervene with an OFR. For example, if the system says that the contact between the hand and the ball should have been a penalty kick by a probability of 85% or higher then the VAR should direct the referee for an OFR.
VAR can also help the system by identifying the type of foul or infraction. Once that is done, the AI-based system will give the VAR a percentage in a matter of seconds.
One might ask the question: How about an incident that does not fit onto any of the instructional clips or past experiences of the referee? That is a very valid question.
Let is look at what the LOTG say about this: LOTG 2022-23 Page 11 “The Laws cannot deal with every possible situation, so where there is no direct provision in the Laws, The IFAB expects the referee to make a decision within the ‘spirit’ of the game and the Laws – this often involves asking the question, ‘what would football want/expect?’ “
This is where technology in the near future – next 20 years – cannot help soccer refereeing. This will require a homo sapiens referee who can analyze, think, and deduce the solution from his/her knowledge of the spirit of the LOTG and answer the question “what would football want/expect”. Unfortunately, in the age of an educational system – both in soccer refereeing and elsewhere – that does not value deep analysis, reasoning and answering the question “why” that seems to be a futile expectation.
Humans did create driver-less cars, but creating a referee-less soccer game does not seem to be feasible even though AI and technology can help us to alleviate some errors in critical match incidents.
Ahmet Guvener (email@example.com) is a Partner with The Game Planners, LLC and the former Secretary General and Chief Soccer Officer of the Turkish FA. He was also the Head of Refereeing for the Turkish FA. He served as a panel member for the FIFA Panel of Referee Instructors and UEFA Referee Convention. He now lives and works as a soccer consultant in Georgetown, TX.