Page 37 - IJEEE-2022-Vol18-ISSUE-1

Shakir & Al-Azza
reduce the visual artifacts which result from discontinuities.
The advantage of variable length units is that there are fewer
concatenation boundaries, but the search is mostly more
difficult. Moreover, the output animation is generally
constrained to the talker and the environment of the original
recording. Methods relied on parametric statistical models
comprise switching linear dynamical systems [59], shared
Gaussian process latent variable models [60], artificial neural
networks [61], and hidden Markov models [62–64]. One of
the noteworthy early work, Voice Puppetry [65], suggested
an HMM-based talking face synthesis driven by speech
signal. Xie et al. [66] suggested coupled HMMs (cHMMs) to
model auditory-visual asynchrony. Choi et al. [67] and
Terissi et. al [68] utilised HMM inversion (HMMI) to infer
the visual parameters from speech signal. Zhang et al. [69]
utilised a DNN to map speech features into HMM states, then
further maps to synthesised faces.
Deep Learning is a recent direction in artificial
intelligence and machine learning research. Lately, new deep
learning frameworks are being born, outperforming state-of-
the-art machine learning approaches. A few DNN-based
methods have been investigated. Suwajanakorn et al. [70]
designed an LSTM network to synthesis photo-realistic
talking head videos of a target identity directly from speech
feature. This system needs a number of hours of face videos
of the target identity, which limits its application practically. classified into geometric and physics-based. Linear
Chung et al. [71] introduced an encoder-decoder
convolutional neural network (CNN) model to synthesis
talking face video from speech feature and a single face
image of the target identity. In this work, the reduction from
a number of hours of face videos to a single face image to
learn the target identity is a great improvement. The main
limitation of end-to-end synthesis is the lack of freedom for
further manipulation of the synthesised face video. For
instance, within a synthesised video, one might need to vary
the gestures, facial expressions, and lighting conditions, all
of which could be independent of the content of the speech.
These end-to-end frameworks could not provide such
manipulations unless these factors could be taken as extra
inputs. However, that would significantly increase the
amount and diversity of data required for training the
systems. For such manipulations a modular design which
splits the generation of key parameters and the fine details of
synthesised face images is more flexible. Pham et al. [72]
adopted a modular design, speech features firstly mapped to
3D deformable shape and rotation parameters utilising an
LSTM framework, and then a 3D animated face synthesised
in realtime from the predicted parameters. This approach is
improved in [73] by substituting speech features with raw
waveforms as the input and substituting the LSTM
framework with a convolutional architecture. Chen et al. [74]
first transferred the auditory to face and then synthesised
video frames conditioned on the landmarks. Song et al. [75]
suggested a conditional recurrent adversarial network that
integrated auditory and image features in recurrent units.
But, the head pose generated by these 2D-based methods is
almost fixed during talking. This drawback is caused by the
defect inherent in 2D-based methods, since it is difficult to
only use 2D information alone for naturally modelling the
change of pose. They introduce 3D geometry information
| 33
into the proposed system to simultaneously model
personalised head pose, expression and lip synchronisation.
The current overview of generative adversarial network
GANs [76] has shifted the motivation of the machine
learning group to generative modelling. GANs contain two
challenging networks: generative and discriminative
networks. The generator’s target is to generate realistic
samples while the discriminator’s target is to discriminate
between the real and produced samples. This competition
leads the generator to produce robustly realistic samples.
Vougioukas [77–79] proposed an end-to-end model using
temporal GANs for speech-driven facial animation, capable
of generating a video of a talking head from an audio signal.
Guo [80] introduce a GAN-based, end-to-end TTS training
algorithm, which propose the generated sequence to GAN
training to avoid exposure bias in autoregressive decoder.
The suggested algorithm improves both output quality and
generalisation of the model.
III. METHODS COMPARISON AND EVALUATION
Facial blendshapes are the general option for realistic
face animation in the film industry. They have driven
animated characters in Hollywood movies and attracted
many research attentions. Facial blendshapes can be
interpolation plays a principal role in geometric face
blendshapes. Linear interpolation-based face blendshapes
are general meanwhile they have the advantages of
expressiveness, simplicity and interpretability. Despite of
this, the following limitations have been identified in [81].
Facial blendshapes might be considered as samples from a
hypothesized manifold of face expressions. Producing a new
face shape needs enough target face shapes to sample the
manifold and describe local linear interpolation functions.
Generating sufficient target face shapes is typically an
iterative and effort intensive procedure. Physics-based face
blendshapes are to augment physics to facial blendshapes
which has a prospect to tackle the above issue. Particularly,
when physics-based simulations are combined with data-
driven methods, truthful face animation can be generated.
The weaknesses with physically-based methods
produced by their complexity. Acquiring physics-based rigs
configured could be a very hard and boring task for
performers. There has been effort achieved to automate some
of the creation of these rigs for physics based facial
animation [81], however there is still a noteworthy amount
of effort that has to be achieved by hand to make the rigs look
accurate and not fall into the uncanny valley. As well as the
complexity causing the rigs to be hard to set up, these
complex rigs need enormous computations to compute the
animations. These large computations make physically-
based animations inappropriate for real-time applications.
Physics-based solutions to face animation have given the
entertainment industry great animations that are becoming
closer to being anatomically truthful, however the great
amount of effort to get these rigs completed will keep the
method from being utilised more widely.
MPEG-4 is could be seen as a formalisation of FACS,
however what makes it different method is that, unlike