The iBorderCtrl system unifies different interdisciplinary modules and converges into an overall system and two phases to speed up the procedure of crossing the borders especially for bona fide travellers.
The two phase procedure
At pre-registration step, travellers self-report in the comfort of their own home or other convenient place through the Traveller User Application (TUA), an on-line system that collects all relevant data helping them ensure they are fulfilling their obligations and allowing for pre-checks to take place in advance. TUA engages the travellers before they reach the Border Crossing Point and empowers them to speed-up, lower the cost and improve the accuracy of border control. The traveller User Application is a web as well as a mobile application that allows the traveller to register and inform the authorities about a potential trip they intend to make. The TUA includes the traveller user interface, which is the presentation layer (visualisation) of the application in a user friendly and intuitive format for the traveller. Through this application, travellers are able to enter and update their personal information, upload travel related documents (such as VISA, passport) and travel information (means of transport, purpose of trip, etc.), see expected traffic statistics regarding the selected country and Border Control Point (BCP) and undergo an avatar interview. The application backend stores the information provided by the traveller to the iBorderCtrl database and provides simplified access to the stored data to the other iBorderCtrl related modules that perform checks and produce risk scores. At the end of a new trip registration and upon completion of all required preregistration steps, the TUA generates a QR code that is also sent to the traveller via email; when the traveller displays the QR code to the Border Guard at the BCP all information collected during the preregistration phase is retrieved in order to speed up the border check process and facilitate the decision of the border guard.
For the pilot tests, the TUA relied on the informed consent for users. In a possible operational scenario, which is however not within the scope of iBorderCtrl as a research project, an implementation could follow the principle of voluntariness or, if a respective statutory legal basis was enacted, be used as a standard system for border crossing procedures. At the same time, it will be important to ensure that persons, especially if inexperienced in using computers, are able to interact exclusively with human border agents if they prefer; in this regard, the TUA would serve as an optional, potentially time-saving tool for many travellers (akin to the automatic e-passport gates that people may use to reduce queuing time at airports, albeit in this case available before they begin their journey).
At the border crossing stage iBorderCtrl provides key technology to the border guards both integrated to existing static installations, as well as a portable hardware platform that empowers -through technology- the border guard. At this stage all information of the traveller gathered during the pre-registration phase is now available to the Border Guard through the Border Guard User Application (BGUA) that brings all analytic results from each technology together to identify risks to the border guard that support her/him in both an overall evaluation of the traveller, as well as highlighting specific potential issues the agent should focus on. All documents and required information needed to cross the border are re-evaluated at the border crossing in their original hard form, however as the pre-check has already been performed virtually, the real-world checks are for the most part limited to validating that indeed the originals contain the same information as what was collected at pre-registration. In addition, biometric checks (fingerprints, palm vein and face matching) will take place for identity verification of the traveller. Furthermore, in cases where the traveller wishes to cross the border with a vehicle, an additional check may be performed to detect any hidden humans inside the vehicle. Here an electronic tool developed by the Project allows this without the need physically to search the luggage or dismantle the car.
As the above shows, the iBorderCtrl project relies on a multitude of technical innovations that would enable a more robust and efficient land border control platform. The key objective was to employ existing and proven technologies as well as novel ones in a way to increase the accuracy (sensitivity- ability of border agents to identify problematic crossings that should be halted, and specificity – their ability to identify valid crossings) and efficiency (throughput while reducing the average cost to travellers (in terms of time and stress) of border checks.
The main modules that compose the overall solution are the following:
- the Biometrics Module incorporating fingerprints and palm vein technologies (BIO module), for the biometric identity validation of the traveller. The BIO module is used at the Border Crossing and two different scanners are used respectively. In this regard, the project experimented with a variety of biometric features to also validate the efficacy of other procedures than fingerprint scanners, which have downsides with regard to hygiene, scanning times and efficacy in difficult conditions (e.g. cold hands). In both cases the aim is to be able to then compare the fingerprints and/or palm vein images to relevant information stored in databases (legacy systems in the case of fingerprints and a baseline database for palm vein images) in order to assist the Border Guards to validate the traveller’s identity. In the context of the project pilot-testing, only simulated data was used. However, if implemented in practice, the intention (subject to the necessary legal safeguards), would be to link to relevant border and/or law-enforcement agency databases.
- the Face Matching Tool (FMT), is the tool for performing facial recognition for iBorderCtrl during both pre-registration and border crossing phases. The module is prepared to receive an image or video and create a biometric patron of the subject detected to verify his/her identity.
- the Document Authenticity Analytics Tool (DAAT) is used both at the pre-registration and the border crossing phases. DAAT utilises information provided by two actors: a) the traveller during the pre-registration and b) the border guard at the real border check. The traveller uploads his/her travel documents (passport, visa, ID, residence permit) using the Traveller User Application (TUA). When the traveller reaches the border crossing point, the Border Guard uses the document scanner integrated to the Portable Unit to scan the travel documents. The security features of travel documents (passport, visa) are examined by DAAT against fraud characteristics.
- the Hidden Human Detection Tool (HHD) supports the Border Guard in detecting any hidden people inside various vehicles (i.e. passengers attempting illegal border crossing). The functionality provided allows detection of humans hidden within vehicles such as cars or closed compartments (containers carried on trucks or train wagons). The HHD module consists of two types of sensing subsystems: an electromagnetic / radar sensor and an acoustic sensor. The radar sensor is meant for all non-metallic spaces and compartments where people can hide; it is better suited for ordinary vehicles (i.e. cars or vans), where it mostly facilitates the Border Guard since it is smaller and easier to use. On the other hand, the acoustic sensor is meant mainly for metallic cargo containers, which are usually large, closed compartments. The HHD sensors were specifically selected to be small and light weighted. The HHD allows for a minimally-intrusive search of vehicles or containers.
- the External Legacy and Social interfaces system (ELSI), is used to crosscheck the traveller’s information from legacy systems, such as SIS II, providing the necessary interoperability interfaces.
- The Risk Based Assessment Tool (RBAT), is designed to cover the necessity of calculating and managing traveller related risks in the Border check procedure in an easy and effective way. RBAT is a robust, user friendly and flexible tool to support the decision-making process of the Border Authorities. RBAT enables a common, harmonised model for risk management including risk of fraud and any other risk which appears to threaten the Authorities’ objectives. RBAT also identifies cases that deserve further investigation, facilitating in this way better resource allocation for the Border Managers and Agents and ideally reducing the number of false positives (i.e. persons the agent wrongly believes not to be bona fide travellers). These risks are key to the performance of the system as they declutter the information provided to the agent by compressing all data into meaningful actionable risk scores that help the agent at the border target any follow up checks and questioning to the traveller.
- the Integrated Border Control Analytics Tool (BCAT) is a dedicated tool for deploying advanced analytics methodologies to the Border Managers. This can be used to provide prognostics as demonstrated for risk and traffic, or it can be used to utilize past travels to identify outliers in behaviour. BCAT analysis can reveal otherwise undetected illicit activities or patterns of behaviour that are apparently associated to illicit activities. At the same time, BCAT evaluates the performance of each iBorderCtrl system and its effectiveness. BCAT also discovers key patterns in the data associated with either false accept or false rejects of travellers, which can be used for better decision making at border control. BCAT also provides analysis on traffic data, so that it can provide the traffic history and the expected traffic for certain dates.
- The Automatic Deception Detection System (ADDS) performs, controls and assesses the pre-registration interview by sequencing a series of questions posed to travellers by an Avatar. ADDS quantifies the probability of deceit in interviews by analysing interviewees’ non-verbal micro-gestures. This, coupled with an avatar, moves this novel approach to deception detection to the pre-registration phase resulting in its deployment without an impact on the time spent at the border crossing by the traveller. The avatar also allows for consistent and controllable stimuli across interviews, in terms of both verbal and non-verbal communication from the avatar agent to the traveller; this can be adjusted to take appropriate account of the gender and first-language of the traveller, reducing variability compared to human interviewers and potentially improving the accuracy of the system. At the same time, it is important to emphasize that, while ADDS may reveal statistical likelihoods of deceptive behaviour, each case would require further checking by a human agent to determine if deception is individually present. This is also a legal requirement under the EU General Data Protection Regulation and law enforcement directive (prohibition on automatic decision-making). ADDS is based on previously developed technology, in particular the so-called Silent Talker . The iBorderCtrl project has adopted this technology but is also well aware of the scientific controversy around its efficacy. A scientific foundation is achieved when a research starts from a position of ignorance and follows the scientific method to dispel that ignorance. As AI scientists, one research question of particular importance to us is “Are there non-verbal behavioural (NVB) indicators of deception” The experiments carried out in iBorderCtrl collect data to add to the evidence needed to support or refute the relevant hypotheses. This tool has undergone the following research-methodological steps:
The work started with interviewing psychologists and reviewing the psychology literature to find a pool of candidate features to which machine learning could be applied to answer the research questions. Thus, there is not an explicit model. There is however, an overall conceptual model assuming there are drivers of non-verbal behaviour that create inconsistencies with truthful NVB (detectable through machine learning) when an interviewee is being deceptive. These include (but are not limited to) arousal (including “stress” and “duping delight”), cognitive load and behaviour control.
The so far collected evidence supports the hypothesis that NVB can be used to detect deception at levels that are statistically significantly greater than chance. Early evidence was published in  and . Further publications are forthcoming after peer review. Initial iBorderCtrl work was presented in . Substantial work has been done in the interim to produce a scalable system that can undergo larger scale trials through web and cloud technologies, supporting the present work.
It is also important to point out that the target for the Silent Talker component of ADDS is to move to Technology Readiness Level 7 during the programme. TRL7 means “system prototype demonstration in operational environment”. Regarding the expected accuracy - it would be unrealistic to expect 100% accuracy from any AI-based deception detection technology, no matter how mature. Consequently, iBorderCtrl is designed with this in mind. (As a comparator, it may be noted that human border guards also do not achieve 100% accuracy in their decisions as to which persons should undergo a thorough check.)
iBorderCtrl is a two-stage process with many components that address various aspects of the border control procedures, and each provides its own risk estimation for the traveller. These are combined by a Risk-Based Analysis Tool which synthesises a single risk score from a weighted combination of components. Therefore, deception detection is a single component of an overall robust system. Moreover, the score itself is an indicator of risk that is then communicated to a human guard (human-in-the-loop principle), in charge of deciding whether to pass the traveller or require a second-line interview. Thus, the Consortium does not accept that, if implemented, the overall system of which ADDS is a part would lead (in the words of certain of its critics) to “an implementation of a pseudoscientific border control”. At the same time, it acknowledges that – besides implementing all required legal and ethical safeguards - further research in the field of AI will be required to further enhance the system, i.e. making it more sensitive to cultural differences and characteristics, prevent misinterpretations stemming e.g. from feeling uncomfortable with human-machine interaction, avoiding bias in algorithms and datasets, etc. To this end, larger datasets to train the system, and further enhance its accuracy and reliability, would be required.
More generally, the Consortium is mindful of concerns about the role of artificial intelligence-based systems and their role in society:
There is no reason to believe that stress is an effective single indicator of deception (in fact there is no indicator that there are any effective single indicators of deception). In fact, this is a fundamental assumption of the design of the Silent Talker architecture. Silent Talker is not designed to measure stress or any other single indicator.
iBorderCtrl is a human in the loop system and the Border Guard will use his/her experience in making the final decision. The Consortium is aware though that border guards might be unduly influenced by the system, and that training and sensitisation, as well as ethical oversight would be crucial to reduce risks for fundamental rights to an acceptable level, balanced with the advantages such a system might offer.
The project is well aware of the legal -in particular data protection- and ethical issues that might arise in the context of the developed component and in particular ADDS. Such issues were addressed in a dedicated Work Package of the project , in which they were handled in close consultation with an ethics advisor.
During the lifespan of the project, the pilot tests of the iBorderCtrl system including ADDS, were encapsulated (i.e. isolated off from the real-world environment) so as not result to any actual legal effects on the project research participants. To achieve this, volunteers were invited to participate in a simulation of a border check using the iBorderCtrl system. Any data processing in the iBorderCtrl test pilots was based on informed consent. In the first phase of the project, ethical principles and legal safeguards relating to human-machine interaction, privacy, personal data protection and informed consent, etc., were analysed. The data, and their use, were reviewed at each stage of the project’s development and then deleted at its conclusion.
 Rothwell, J., Bandar, Z., O'Shea, J. and McLean, D., 2006. Silent talker: a new computer‐based system for the analysis of facial cues to deception. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 20(6), pp.757-777.
 Rothwell, J., Bandar, Z., O’Shea, J. and McLean, D., 2007. Charting the behavioural state of a person using a backpropagation neural network. Neural Computing and Applications, 16(4-5), pp.327-339.
 O'Shea, J. Crockett, K. Khan, Kindynis, P. Antoniades, A. (2018) Intelligent Deception Detection through Machine Based Interviewing, IEEE International Joint conference on Artificial Neural Networks (IJCNN), DOI: 10.1109/IJCNN.2018.8489392
 Crockett, K. Stoklas, J. O’Shea, J. Krügel, T. Khan, W. Reconciling Adapted Psychological Profiling with the New European Data Protection Legislation (2020), Computational Intelligence, Eds: Sabourin, C. Mereio, J. Barranco, N. Madani, K. Warwick, K. Springer, in-press