Processing Device for Liveness Determination Based on a Multi-Branch Network

This invention relates to the field of face processing, and more particularly, to face liveness determination, which aims at determining whether a face is real/live, from a target person at a point of capture, or spoof/fake.

More and more applications and services now rely on a prior step of face recognition to ensure that a user is authorized to use a resource such as a service/application.

This is the case for example in the domain of telecommunications, when unlocking a phone for example, when accessing to a physical place, for example in airports, in biometric systems for identity verification or in digital identification, for example for banking services.

In the above-mentioned services, face recognition is becoming one of the preferred biometric modality. Once the face of a user is captured, image frames are generally processed and compared with stored images labelled with user identities, so as to infer the identity of a user.

To access services/applications of the genuine person, attackers may implement Presentation Attacks, noted PAs hereafter. PAs may consist in presenting face spoof to the recognition system.

PAs may be static such as printing a face on paper, or wearing a 2D or 3D mask, or may be dynamic such as replay attacks replaying a video of the genuine person on digital device such as a Smartphone.

To counteract these PAS, liveness determination methods have been developed and are applied prior to face recognition, to ensure that the genuine person that is subjected to recognition is real and live, and not spoof.

Some liveness determination methods or systems are called active, where they require the target person to perform a series of one or several instructions, requesting some type of cooperative behavior, such as blinking of the eyes, mouth/lip movement and/or turning of the head.

However, active methods/systems have the drawback to be obtrusive, which is a problem in security applications where a target person needs to be recognized without noticing it.

In contrast, passive liveness determination do not require any cooperation from the target person, so are unobtrusive, and processes images or frames that are acquired by a capture device, such as a camera.

Some prior art solutions are based on machine learning principles to train models to detect whether the target person is live or spoof.

Some of them rely on Convolutional Neural Networks, CNNs, receiving a single input middle resolution image. Such CNNs can be trained using supervised learning, to obtain a global binary classification using a cross entropy loss function and based on live and spoof labels, or in a pixel-wise manner.

However, such a CNN taking as input a single frame to not take a temporal information into account, although this information may be relevant to achieve an accurate prediction.

Other methods make use of temporal information from a sequence of frames, by relying on a three-dimensional CNN, 3DCNN, or a Long Short Term Memory CNN, CNN-LSTM, used in combination of a cross entropy loss function. However, these methods require large computational resources as they need to process a large number of frames. Alternatively, they use a reduced number of frames in the sequence, but these frames may not be representative as they may contain highly correlated or redundant information.

Moreover, 3DCNN or CNN-LSTM are typically difficult to train. In the end, 3DCNN and CNN-LSTM are not more efficient than CNN receiving a single input middle resolution image.

Therefore, there is a need for a passive method of facial liveness detection that achieve high accuracy, in particular upon reception of a sequence of frames, and that is able to cope with different types of PAs.

1 13 15 The present invention provides a solution for all the aforementioned problems by a method for determining liveness of a target person according to claim, a processing device according to claim, and a system according to claim. In dependent claims, preferred embodiments of the invention are defined.

obtaining a series of N frames from a sequence of frames acquired by a frame capture device, N being equal to or greater than 2, at least some of the frames representing the target person; determining a first partial score for the series of N frames; determining a second partial score for the series of N frames; determining a fusion score based on at least the first partial score and the second partial score, said fusion score being representative of liveness of the target person;wherein the first partial score is determined based on first features among local spatial features of the N frames, on global spatial features of the N frames, or on temporal features of the N frames, and wherein the second partial score is determined based on second features among local spatial features of the N frames, on global spatial features of the N frames, or on temporal features of the N frames, the first features and the second features being different. processing the series of N frames by a multi-branch convolutional neural network, comprising N branches, wherein the processing comprises: In a first inventive aspect, the invention provides a method for determining liveness of a target person comprising:

Therefore, the fusion score is determined based on at least two different sets of features, which enables to enhance the scope of PA attacks that the method is able to efficiently detect, while using a single CNN with multiple branches as input, which reduces the latency of the liveness determination.

According to some embodiments, the method may further comprise: determining a third partial score for the series of N frames, the third partial score may be determined based on third features among local spatial features of the N frames, on global spatial features of the N frames, or on temporal features of the N frames, the third features being different from the first features and from the second features.

Therefore, the fusion score is determined based on three different sets of features, which enables to enlarge the scope of PA attacks that the method is able to efficiently detect.

In complement, the first partial score may be determined based on local spatial features, the second partial score may be determined based on global spatial features and the third partial score may be determined based on temporal features.

Therefore, the fusion score is determined based on three different sets of features, which enables to enlarge the scope of PA attacks that the method is able to efficiently detect.

According to some embodiments, the fusion score may be obtained based on a linear function applied at least to the first and second partial scores, the linear function may comprise a weighted sum, respective weights being attributed at least to the first and second partial scores.

Therefore, the fusion score is determined based on three sets of features of different types, which enables to enlarge the scope of PA attacks that the method is able to efficiently detect.

In complement, the method may further comprise adjusting the respective weights based on external information.

This allows to dynamically give more importance to one or two of the local, global and temporal features of the frames depending on the external information. The accuracy of the fusion score is therefore increased.

According to some embodiments, the method may further comprise transmitting the fusion score to a classification module.

The fusion score can therefore be used by the classification module to take a decision as whether the access to a resource is granted or not. This allows to externalize this decision to the classification module, which can therefore set the required level of certainty based on requirements that are specific to the classification module.

According to some embodiments, the method may further comprise transmitting at least one of the first and second partial scores to an external entity.

The external entity can therefore use the partial score or the partial scores for other applications than granting an access to a resource.

According to some embodiments, the method may further comprise a prior step of obtaining the multi-branch convolutional neural network by machine learning by machine learning, based on a training set of data, each data of the set comprising a series of N frames and at least a binary information indicating whether a target person in the series of N frames is live or spoof.

Such a supervised learning enables to increase the accuracy associated with the fusion score.

In complement, the multi-branch convolutional may comprise at least a first processing part comprising at least one layer and a second processing part comprising at least one layer, the first processing part being trained by supervised learning to determine the first partial score, and the second processing part being trained by supervised learning to determine the second partial score, the set of data may comprise, for each series of N frames, at least one supervisory signal dedicated to the first processing part and at least one supervisory signal dedicated to the second processing part.

130 Therefore, each of the processing parts of the MBNmay be obtained by separate machine learning procedures, which increases the accuracy of the partial scores and of the fusion score.

According to some embodiments, the method may further comprise receiving a sequence of M frames from the frame capture device, M being greater than N, determining relationships between the M frames and selecting N frames among the M frames based on the determined relationships.

This enables to select relevant frames, which reduces the complexity of the calculations performed by the multi-branch network, as the number of frames can be limited, allows processing frames of higher quality, and therefore enables to increase the accuracy of the fusion score.

for each frame of the sequence, extracting features from the frame; determining, for each pair of frames in the sequence, a distance based on the features of the frames of the pair; for each frame in the sequence, determining an average distance based on the respective distances between the frame and the other frames of the sequence; computing a distribution of the average distances of the frames of the sequence; based on at least one statistic of the distribution, selecting N frames among the M frames. In complement, determining relationships between the M frames may comprise:

This allows to select N frames that are representative of the face of the target person. For example, this allows to avoid processing redundant frames, which reduces the complexity of the calculations performed by the multi-branch network.

In complement, the at least one statistic may comprise an average of the average distances and a standard deviation of the distribution of average distances.

Therefore, N relevant frames can be selected with a selection procedure that involves low complexity calculations.

a receiving interface for receiving a sequence of frames from a frame capture device, wherein the processing device is configured for obtaining a series of N frames from the received sequence of frames, N being equal to or greater than 2, at least some of the frames representing a target person; determining a first partial score for the series of N frames; determining a second partial score for the series of N frames; determining a fusion score based on at least the first partial score and the second partial score, said fusion score being representative of liveness of the target person;wherein the first partial score is determined based on first features among local spatial features of the N frames, on global spatial features of the N frames, or on temporal features of the N frames, and wherein the second partial score is determined based on second features among local spatial features of the N frames, on global spatial features of the N frames, or on temporal features of the N frames, the first features and the second features being different. a processor configured for processing the series of N frames by a multi-branch convolutional neural network, comprising N branches, the processor being configured for: In a second inventive aspect, the invention provides a processing device comprising:

In a third inventive aspect, the invention provides a system for determining liveness of a target person comprising a processing device according the second aspect of the invention, a frame capture device and a classification module arranged to receive at least the fusion score from the processing device.

Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealised or overly formal sense unless expressly so defined herein.

In this specification, the term “comprises” and its derivations (such as “comprising”, etc.) should not be understood in an excluding sense, that is, these terms should not be interpreted as excluding the possibility that what is described and defined may include further elements, steps, etc.

All the features described in this specification (including the claims, description and drawings) and/or all the steps of the described method can be combined in any combination, with the exception of combinations of such mutually exclusive features and/or steps.

1 FIG. 100 shows a systemfor liveness determination of a target person according to some embodiments of the invention.

110 111 110 110 110 The system comprises a frame capture devicesuch as a camera or a video camera, arranged to acquire a sequenceof M frames, which can be static frames or video frames. No restriction is attached to the technology associated with the frame capture device, which is preferably arranged to acquire digital video frames. To this end, the frame capture devicecomprises at least some optics and a sensor, and may comprise an internal memory and some processing capabilities to pre-process the frames that are acquired. For example, the cameramay be a single channel Red Green Blue camera, able to acquire RGB frames.

110 110 The frame capture deviceis arranged to acquire frames representing a scene in a field of view of the frame capture device, the field of view being static or dynamic. The quality of the frames that are acquired depends on the sensitivity and the format of the sensor, the ambient luminosity, velocity of the target person, settings of the camera, such as aperture, exposure time and focal length.

100 The scene that is represented in at least some of the frames of the sequence, represents at least a face of a target person. The systemaims at determining whether the face of the target person is live or spoof, for example to grant access to some resources.

110 The frame capture moduleis installed at an end user site, which can be at a building entrance, in front of a door, at a security gate of an airport or any other physical location.

111 110 No restriction is attached to the number M of frames in the sequenceor the time period between each frame. For example, at least 24 frames per second may be acquired when the frame capture deviceis a video camera. The sequence may have a duration of one or several seconds for example.

111 170 131 111 110 The sequenceis communicated to a processing devicearranged to determine at least a fusion scorebased on the sequenceof M frames received from the camera.

170 170 170 4 FIG. No restriction is attached to the processing device, which can be any device comprising processing capabilities and a non-transitory memory. The processing devicemay for example be incorporated into a server, desktop computer, smart tablet, smart phone, mobile internet device, personal digital assistant, wearable device, image capture device or any combination thereof. The processing devicecomprises fixed-functionality hardware logic, configurable logic, logic instructions, etc, or any combination thereof, as it will be better understood from the description ofhereafter.

110 170 111 In addition, no restriction is attached to a communication link between the cameraand the processing device, on which the sequenceof M frames is transmitted. The communication link can for example be a wireless link, or a wired link. For example, wired protocols may include RS-232, RS-422, RS-485, I2C, SPI, IEEE 802.3 and TCP/IP. Wireless protocols may include IEEE 802.11a/b/g/n, Bluetooth, Bluetooth Low Energy (BLE), FeliCa, Zigbee, GSM, LTE, 3G, 4G, 5G, RFID and NFC.

110 110 170 120 1 FIG. According to some embodiments, the sequence of M frames is not directly issued by the frame captured devicebut rather output by a face detection module and/or by a quality filtering module, which are not shown on. Alternatively, the sequence of M frames is received from the frame capture device, and the face detection module and/or the quality filtering module may be integrated in the processing device, upstream from the optional selection moduledescribed hereafter.

110 The face detection module may be configured to detect a face of a target person in each of the frames acquired by the frame capture module. The face detection module may be able to identify a zone where the face of the target person is detected or may crop the frame so that it is centred and zoomed in the detected face. Methods for detecting a face in a frame are well known and are not further described in the present description.

110 The quality filtering module may be able to discard or reject frames acquired by the frame capture devicethat do not provide a minimum level of quality. The rejected frames are not integrated in the sequence of M frames.

111 110 Therefore, the sequenceof M frames preferably comprise frames of good quality focused on the target person's face. The frame capture devicemay therefore provide more than M frames, and the M frames of the sequence are the ones that comprise the face of the target person and that satisfy the quality criteria set by the quality filtering module.

170 120 111 170 111 2 FIG. a. The processing deviceoptionally comprises a selection moduleof the system, arranged to select a series of N frames among the M frames of the sequence, N being less than or equal to M. The processing moduleis configured to obtain N frames representative of the face of the target person based on the sequenceof M frames. For example, the N frames can be selected based on relationships between the frames, as it will be better understood from the description of

170 130 130 120 130 130 The processing devicefurther comprises a multi-branch convolutional neuronal network, MBN, wherein the number of branches correspond to the number of inputs of the MBN. The number of inputs is equal to N. Therefore, the selection modulecan be built based on the number of branches of the MBNso as to transmit the right amount of frames to the MBN. The N frames can be statistically determined to be the most representative of the face of the target person, so as to increase the accuracy of the fusion score determined by the processing module.

120 111 130 130 The selection moduleis optional. In variant, the sequenceof M frames is directly transmitted to the MBN. In that case, M is equal to N so that the sequence of M frames comprises a number of frames corresponding to the number of branches of the MBN.

130 121 1 121 2 121 3 2 2 The MBNtherefore receives a series of N frames.,.,., N being equal to or greater than. In what follows, an example where N is equal tois considered, for illustrative purposes only. The invention also applies to values of N that are strictly greater than 2.

130 3 FIG. CNNs are artificial neural networks that comprise filters or kernels organized in different layers, comprising an input layer, one or several hidden layers, and an output layer. A MBN comprises several input layers in parallel arranged for receiving several frames at the same time. The MBNtherefore comprises N input layers, as it will be detailed when describinghereafter.

130 determine a first partial score for the series of N frames received as input; determine a second partial score for the series of N frames received as input; determine a fusion score based at least on the first partial score and on the second partial score, the fusion score being representative of liveness of the target person; The MBNis arranged to:

The first partial score is determined based on first features among local spatial features determined based on the N frames, on global spatial features determined based on the N frames, or on temporal features determined based on the N frames, and the second partial score is determined based on second features among local spatial features determined based on the N frames, on global spatial features determined based on the N frames, or on temporal features determined based on the N frames, the first features and the second features being different.

130 130 To this end, the MBNcomprises a first processing part configured to determine the first features and to determine the first partial score based on the first features. The MBN also comprises a second processing part configured to determine the second features and to determine the second partial score based on the second features. An output part of the MBNis configured to determine the fusion score based at least on the first partial score and the second partial score.

170 the first partial score may be based on local spatial features and the second partial score may be based on global spatial features; or the first partial score may be based on local spatial features and the second partial score may be based on temporal features; or the first partial score may be based on global spatial features and the second partial score may be based on temporal features. Therefore, the fusion score is determined based on at least two different sets of features, which may be of different types, which enables to enhance the scope of PA attacks that the processing deviceis able to efficiently detect. Any combination of two sets of features may be used according to the invention:

No restriction is attached to the local spatial features, which may be a depth map or local skin texture feature. No restriction is also attached to the global spatial features, which may be a light reflection feature and/or a skin texture feature and/or a Moire pattern. No restriction is attached to the temporal features which may be a movement feature, such as the movement of a specific pixel or zone, for example a nose movement, eye movement or any face part movement, or screen flickering.

130 130 3 FIG. In some embodiments of the invention, the MBNis further arranged to determine a third partial score. Therefore, in these embodiments, a partial score is determined for each feature among the global spatial features, the temporal features and the local spatial features. These embodiments are illustrated in the MBNdescribed with reference to, which is given for illustrative purpose.

The fusion score is representative of liveness of the target person. For example, the fusion score may be a probability that the target person is live. Alternatively, the fusion score may be a probability that the target person is spoof. Still alternatively, the fusion score may be a vector with two components, comprising a probability that the target person is live and a probability that the target person is spoof.

170 131 150 150 131 150 The processing deviceis able to transmit the fusion scoreto a classifying module. The classifying moduleis configured to determine whether the target person is live or spoof based at least on the fusion score. The classifying modulemay thus grant or refuse access to a resource based on the determination made.

170 132 150 131 132 1 FIG. In addition, the processing devicemay transmit at least one of the partial scoresto an external entity. The external entity may be the classifying module, as shown on. The external entity can therefore use the at least one partial score to determine whether the target person is live or spoof based at least on the fusion scoreand based on the at least one partial score. Alternatively, the external entity may further process the at least one partial score for any other purpose.

170 130 The processing devicemay also transmit all the determined partial scores to the external entity, such as the first and second partial scores, and optionally the third partial score when the MBNis configured to determine a third partial score.

170 150 No restriction is attached to the communication link between the processing deviceand the classifying module. The communication link can for example be a wireless link, or a wired link. For example, wired protocols may include RS-232, RS-422, RS-485, I2C, SPI, IEEE 802.3 and TCP/IP. Wireless protocols may include IEEE 802.11a/b/g/n, Bluetooth, Bluetooth Low Energy (BLE), FeliCa, Zigbee, GSM, LTE, 3G, 4G, 5G, RFID and NFC.

150 170 Alternatively, the classifying moduleis integrated in the processing device.

2 a FIG. 1 FIG. 120 100 shows the steps of a selection procedure that is optionally carried out by the selection moduleof the systemshown on.

110 130 120 The selection procedure allows to select a series of N frames among the sequence of M frames acquired by the camera, that are representative of the face of the target person. No restriction is attached to the criterion or criteria used to assess the representativeness of the N frames. For example, one criterion may be to ensure that the MBNinput contains informative frames that are different from each other, rather than highly correlated frames, which carry redundant information. To this end, frame similarities and relationships may be determined, and the N frames may be determined based on the relationships between the frames. The relationships may be comparisons or distances between some frame features. Once the relationships are determined, the selection modulemay be ordered from the most common to the least common, and the selected N frames may be determined based on such ordering.

An example of a selection procedure is given hereafter, for illustrative purposes only.

201 111 110 The selection procedure starts with a stepof receiving the sequenceof M frames from the frame capture device.

202 111 i At step, at least some features are extracted from each frame among the M frames of the sequence. No restriction is attached to the extracted feature or features, which may be luminance values, chrominance values, blurriness values, texture features extracted from Local Binary Pattern, LBP, or Fourier transform, or any combination thereof. A matrix of features can therefore be obtained for each frame of index i, i varying between 1 and M. Each matrix of features F1may have a dimension of k1*k2, k1 and k2 being integer equal to or greater than 1.

203 i i i At an optional step, each matrix of features F1may be transformed into a reduced matrix of features F2, by applying a predetermined reduction function to the matrix of features F1. The reduced matrix of features has a dimension of k3*k4, where k3 is less than or equal to k1 and k4 is less than or equal to k2. No restriction is attached to the reduction function, which may comprise average functions or interpolation functions, to replace a group of values by a unique value determined by the values from the group.

204 203 i,j i j i j i i,j At step, for each frame of index i, distances Dof frame i with every frame of index j, j varying between 1 and M excluding i, are determined based on a distance function applied to F2and F2. Alternatively, when stepis not implemented, the distance function is applied to F1and F1. The average distance ADis then determined for each frame of index i, by averaging the distances D, j varying between 1 and M, excluding i.

205 i i At step, the distribution of the averages ADis computed and statistics of the distribution are determined. For example, the average A of the average distances ADis computed. In complement, optionally, standard deviation SD is computed.

i i i 2 b FIG. The distribution of the averages ADis shown on. The abscissas represent the values of the averages AD, whereas the ordinates show the number of averages ADhaving a given value, or being comprised in a given range of values.

i 2 b FIG. A The average A of the averages ADis shown as well as the standard deviation SD. In the specific example shown on, the averageis approximately equal to 32 whereas SD is equal to 6.

206 205 At step, N frames are selected among the M frames based on the statistics determined at step.

i1 i2 For example, when N is equal to 2, the frame of index i1 for which the average distance ADis the closest from A−SD, or A−2*SD is selected, and the frame i2 for which the average distance ADis the closest from A+SD or A+2*SD, is selected. This allows to select a frame i1 which is close from many other frames and a frame i2 which is very different from many other frames including frame i2. Therefore, it is ensured that the pair of frames that is selected is diversified and carry more information than a random selection or a selection of frames that are close to each other and that therefore carry redundant information.

2 b FIG. Referring to the example of, selecting the frames that have the average distances respectively the closest from A−2*SD and A+2*SD, consists in selecting the frames for which the average distances are the closest from 32−12=20 and the closest from 32+12=44.

i1 i2 i3 For example, when N is equal to 3, the frame of index i1 for which the average distance ADis the closest from A−SD, or A−2*SD is selected, the frame i2 for which the average distance ADis the closest from A+SD or A+2*SD, is selected, and the frame i3 for which the average distance ADis the closest from A is selected. This allows to obtain three different frames that carry different information, and therefore allows to select frames that have a high representativeness of the face of the target person, instead of redundant frames.

130 130 Selecting a subset of frames also allows providing frames of high quality, without inducing an increased complexity in the calculations performed by the MBN. The accuracy of the determination of liveness is therefore increased compared to submitting more frames of medium quality to the MBN.

205 These examples are given for illustrative purposes and other rules of selection may be used to select the N frames based on the statistics computed at step. For example, the N frames may be randomly selected.

3 FIG. 130 100 shows a structure of the MBNof the systemaccording to some embodiments of the invention.

130 130 2 120 110 3 FIG. The MBNofcomprises two branches, for illustrative purpose only. The MBNis therefore configured to receive a series of N frames, N being equal to. As previously explained, the N frames are received from the selection moduleor directly from the camera.

121 1 301 1 121 2 301 2 301 1 301 2 A first frame.is received by a first backbone CNN.and the second frame.is received by a second backbone CNN.. The first and second backbone CNN.and.are identical.

121 1 121 2 121 1 110 121 2 120 The first frame.is temporally prior to the second frame.. This means that the first frame.has a first index in the sequence of M frames acquired by the camerathat is lower than a second index of the second frame.. When the selection procedure is applied by the selection module, the first and second indices may not be consecutive.

301 1 301 2 130 t 1 The backbone CNN.and.are configured to calculate a first spatial feature map xbased on the first frame and a second spatial feature map xbased on the second frame. The first and second spatial maps have a given dimension h*w*d, with h and w depending on the resolution of the first and second frames and d being predetermined based on the structure of the MBN. For example, d can be equal to 2048. Therefore, the backbone CNN is a feature extractor.

301 1 301 2 301 1 301 2 301 1 301 2 130 No restriction is attached to the number of layers of the backbone CNN.and.which may comprise several convolution and pooling layers, and to the features that are extracted by the backbone CNN.and.. For example, the backbone CNN.and.can be derived from a well-known ResNet-50 network, from which some upper layers are replaced by new upper layers, that form the input layers of the MBN. Alternatively, the backbone CNN can be derived from a MobileNet network, from an EfficientNet network, etc.

t t 302 1 302 1 The first spatial feature map xis transmitted to a first depth map layer., which is configured to determine a first depth map based on the first spatial feature map x. The first depth map layer.may be a regression layer.

t+1 t+1 302 2 302 2 302 1 The second spatial feature map xis transmitted to a second depth map layer., which is configured to determine a second depth map based on the second spatial feature map x, the second depth map layer.being identical to the first depth map layer..

130 303 304 The MBNfurther comprises a concatenation layerthat is configured to concatenate the first depth map and the second depth map, the concatenated depth map being transmitted to a first linear function layer.

304 The first linear function layeris configured to determine a first partial score based on the concatenated depth map, the first partial score being a local spatial score, as it is based on local spatial features (the concatenated depth map).

302 1 302 2 304 The depth map layers.and.and the first linear functioncan be obtained by machine learning, using respective loss functions and supervisory signals.

130 302 1 302 2 302 1 302 2 For example, when series of N frames from a training data set are submitted to the MBN, a first loss function may evaluate the distance between the output of each of the depth map layers.and.and dedicated first supervisory signals. The first loss function may be a Mean Square Error, MSE, function for example. The depth map layers.and.are therefore constructed by supervised learning.

130 304 305 1 305 2 305 1 305 2 t t+1 t t+1 Also, when series of N frames from a training data set are submitted to the MBN, a second loss function may evaluate the distance between the output of the first linear functionand a dedicated second supervisory signal. The second loss function may be a Cross Entropy, CE, function for example The first spatial feature map xmay be transmitted to a first pooling layer.and the second spatial feature map xmay be transmitted to a second pooling layer.. The first and second pooling layers.and.may be averaging pooling layer, and are arranged to reduce the dimensions of the first and second spatial feature maps xand x.

305 1 305 2 305 1 305 2 305 1 305 2 The first and second pooling layers.and.may also be obtained by machine learning, using a dedicated loss function and may be self-supervised by comparing the respective outputs of the first and second pooling layers.and.using the loss function. This loss function may be a MSE function, also called L2 loss, for example and the first and second pooling layers.and.may be trained to control the relationship between the pooled features of the first and second frames. Alternatively, the loss function may be a Least Absolute Deviations, LAD, function also called L1 loss, or it can be other loss such as cosine embedding loss (CEL) that uses cosine similarity. Still alternatively, the loss function may be a combination of L1, L2 and CEL loss.

305 1 305 2 306 The outputs of the first and second pooling layer.and.are also transmitted to a concatenation layer, that is configured to obtain a concatenated output.

307 308 The concatenated output may be transmitted to a second linear function layerand to a fusion embedding modulecomprising several layers.

307 The second linear function layeris configured to determine a second partial score based on global spatial features, such as skin texture features or light reflection features for example. The second partial score is therefore a global spatial score.

307 307 The second linear function layermay also be obtained by machine learning, using a dedicated third loss function and a dedicated third supervisory signal. For example, the third loss function may be used to compare the output of the second linear function layerwith the third supervisory signal. The third loss function may be the CE function.

308 121 1 121 2 308 308 The fusion embedding moduleis arranged to determine temporal features based on the concatenated output, using at least one non-linear function. In particular, temporal features may indicate temporal changes between the first and second frames.and.. The fusion embedding modulemay comprise a fully connected layer followed by a non-linear activation layer, such as ReLU for example), and with another fully connected layer. Alternatively, the fusion embedding modulemay comprise a layer determining the difference between vectors corresponding to first and second frames, followed by non-linear functions to obtain the temporal features.

309 The temporal features are transmitted to a third linear function layer, which is configured to determine a third partial score based on the temporal features, the third partial score being a temporal score.

308 309 308 309 The fusion embedding layerand the third linear function layercan be obtained by machine learning using dedicated loss functions and supervisory signals. For example, a fourth loss function may be used to compare the output of the fusion embedding layerwith a fourth supervisory signal. The fourth loss function may be a contrastive loss function for example. A fifth loss function may be used to compare the output of the third linear function layerwith a fifth supervisory signal. The fifth loss function may be the CE function.

130 310 311 The MBNfurther comprises a concatenation layerthat is arranged to concatenate the first partial score, the second partial score and the third partial score, and to transmit a concatenated score to a fourth linear function.

311 131 310 The fourth linear functionis configured to determine the global scorebased on the concatenated score output by the concatenation layer.

311 The fourth linear functionmay for example be a weighted sum of the first, second and third partial scores. According to some embodiments of the invention, the weights of the first, second and third partial scores may be adjustable based on external information. This allows to dynamically give more importance to one or two of the local, global and temporal features of the frames depending on the external information. No restriction is attached to the external information, which can be which can be a quality level or resolution level of the acquired frames, which can be lighting or illumination features, motion features for example.

311 311 The fourth linear functioncan also be obtained by machine learning, in particular by supervised learning, using a sixth loss function and a dedicated sixth supervisory signal. The sixth loss function may be used to compare the output of the fourth linear function layerwith the sixth supervisory signal. The sixth loss function may be the CE function.

301 1 301 2 130 The backbone CNN.and.together form an input part of the MBN.

302 1 302 2 304 130 The depth map layers.and., the concatenation layer and the first linear functiontogether form a first processing part of the MBN, configured to determine a first partial score based on local spatial features of the frames.

305 1 305 2 306 307 130 The pooling layers.,., the concatenation layerand the second linear function layertogether form a second processing part of the MBNconfigured to determine a second partial score based on global spatial features of the frames.

305 1 305 2 306 308 309 130 The pooling layers.,., the concatenation layer, the fusion embedding layerand the third linear function layertogether form a third processing part of the MBNconfigured to determine a third partial score based on temporal features of the frames.

310 311 130 131 The concatenation layerand the fourth linear function layertogether form an output part of the MBN, arranged to output the fusion scorebased on the partial scores received from the first, second and third processing parts.

130 As already explained, according to the invention, the MBNmay comprise only two of the first, second and third processing parts.

130 311 130 The MBNtherefore enables to provide a fusion score that is representative of liveness of the target person, based on several types of features, which allows detecting many different types of PAs. In addition, the fourth linear functionmay be adjustable based on external information, which increases the accuracy of the fusion score. Also, each of the processing parts and the output part of the MBNmay be obtained by separate machine learning procedures, which increases the accuracy of each of these parts, and therefore the accuracy of the partial scores and of the fusion score.

4 FIG. 170 100 shows a detailed structure of a processing deviceof the systemaccording to some embodiments of the invention.

170 401 202 The processing devicecomprises a processorand a memory.

401 The processormay comprise one or multiple microprocessors, a Central Processing Unit (CPU), on a single Integrated Circuit (IC) or several IC chips.

402 401 402 No restriction is attached to the memory, which may be any non-transient, tangible form of memory. For example, it can comprise ROM, EEPROM and/or flash memory. The processormay be programmable and may be configured to execute instructions that are stored in its internal memory or to execute instructions that are stored in the memory.

402 130 402 120 1 FIG. 3 FIG. 1 FIG. The memorycan indeed store instructions corresponding to the MBNas described onand/or. In addition, the memorymay store instructions for implementing the selection moduleas described in reference to.

401 130 120 Alternatively, the processormay comprise fixed-functionality hardware logic or configurable logic, configured to implement the MBN, and optionally the selection module.

170 403 404 403 111 110 404 150 The processing devicefurther comprises a receiving interfaceand a transmitting interface. The receiving interfaceis configured to receive the sequenceof M frames from the cameraand the transmitting interfaceis configured for transmitting the fusion score, and optionally one or several of the partial scores, to the classification module, or to any other external entity.

403 404 203 204 170 No restriction is attached to the interfacesand, which may use any wired or wireless communication protocol. For example, wired protocols may include RS-232, RS-422, RS-485, I2C, SPI, IEEE 802.3 and TCP/IP. Wireless protocols may include IEEE 802.11a/b/g/n, Bluetooth, Bluetooth Low Energy (BLE), FeliCa, Zigbee, GSM, LTE, 3G, 4G, 5G, RFID and NFC. The interfacesandmay comprise hardware (an Ethernet port, a wireless radio), software (drivers, firmware application) or a combination thereof to enable communications to and from the processing device.

170 130 120 Alternatively, the processing devicemay comprise a first processor dedicated to implement the MBNand another second processor dedicated to implement the selection procedure of the selection module.

5 FIG. shows the steps of a method for determining liveness of a target person according to some embodiments of the invention.

1 FIG. 170 The method can be carried out by the system illustrated and described above referring to, in particular by the processing device.

501 130 130 A prior phase comprises a stepof training the MBNby machine learning. To this end, a training data set is obtained, comprising several hundreds or thousands of series of N frames representing a live or spoof target person. Preferably, and as explained above, each series of N frames of the training data set is also associated with several supervisory signals, so as to enable to train different layers of the MBNseparately.

510 511 516 The method further comprises a current phasecomprising stepsto.

511 170 111 110 111 170 170 111 511 At step, the processing devicereceives the sequenceof M frames from the camera. As explained above, the sequenceof M frames is preferably processed before being received by the processing device, so as to focus on the face of the target person and/or so as to cancel frames that do not satisfy a predetermined quality criterion. Alternatively, the quality filtering and/or face cropping is processed by the processing deviceupon reception of the sequenceat step.

512 120 111 512 211 216 2 a FIG. At optional step, the selection moduleselects a series of N frames among the M frames of the sequence, by implementing the selection procedure as previously described, in particular with reference to. Stepcan therefore correspond to stepstothat have been previously described.

513 130 301 1 301 2 513 At step, features are extracted from the N frames of the series by the input part of the MBN, the input part comprising the backbone CNN.and.as described above for example. N spatial feature maps are therefore obtained at the end of step.

514 1 130 At step., a first partial score is determined by a first processing part of the MBN, based on first features among local spatial features, global spatial features and temporal features.

514 2 At step., a second partial score is determined by a second processing part of the MBN, based on second features among local spatial features, global spatial features and temporal features, the second features being different from the first features.

514 4 At an optional step., a third partial score is determined by a third processing part of the MBN, based on third features among local spatial features, global spatial features and temporal features, the third features being different from the first and second features.

515 At step, a fusion score is computed by an output part of the MBN based at least on the first and second partial scores, and optionally on the third score also.

516 150 At step, at least the fusion score can be transmitted to an external entity such as the classification module. Optionally, at least one of the partial scores may also be transmitted to an external entity.

The example embodiments are described in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.