Selected Publications of Hyewon Seo categorized by theme

# * Learning-based Modeling of Human Shapes & Animations (2003-2005, 2019-)

 DSNet: Dynamic skin deformation prediction by Recurrent Neural Network Seo H., Zou K. and Cordier F.   Skin dynamics contributes to the enriched realism of human body models in rendered scenes. Traditional methods rely on physics-based simulations to accurately reproduce the dynamic behavior of soft tissues. Due to the model complexity and thus the heavy computation, however, they do not directly offer practical solutions to domains where real-time performance is desirable.... We present a learning-based method for dynamic skin deformation. At the core of our work is a recurrent neural network that learns to predict the nonlinear, dynamics-dependent shape change over time from pre-existing mesh deformation sequence data. Our network also learns to predict the variation of skin dynamics across different individuals with varying body shapes. After training the network delivers realistic, high-quality skin dynamics that is specific to a person in a real-time course. We obtain results that significantly saves the computational time, while maintaining comparable prediction quality compared to state-of-the-art results.   To be presented at CGI 2021. Slides to be presented and the accompanying Video Generating 3D Facial Expressions with Neural Networks Seo H. and Luo G.   Learning based methods have proved effective at high-quality image synthesis tasks, such as content-preserving image rendering with different style, and the generation of new images depicting learned objects. Some of the properties that make neural networks suitable for such tasks, for example robustness to the input’s low-level feature, and the ability to access contextual information, are also desirable in 3D shape domain. However, recent successful learning techniques, e.g. deep neural networks, do not easily generalize to 3D shapes, since the regular grid assumption with 2D images does not have a straightforward equivalent in the common shape representation in 3D. Here, we concentrate on the 3D facial shape and expression generation task, an important problem in computer graphics and other application domains, where existing data-driven approaches mostly rely on direct shape capture or shape transfer. At the core of our approach are a generative neural network and with proper shape representations…   In a Springer book “Intelligent Scene Modelling and Human Computer Interaction”. An Example-Based Approach to Human Body Manipulation, Seo H. and Magnenat-Thalmann N.,   We discuss a set of techniques based on examples for generating realistic, controllable human whole-body models. Users are assisted in automatically generating a new model or modifying an existing one by controlling the parameters provided. Our approach is based on examples and consists of three major parts. First, each example from the 3D range scanner is preprocessed so that the topology of all examples is identical. Second, the system that we call the modeling synthesizer learns from these examples the correlation between the parameters and the body geometry. After this learning process the synthesizer is devoted to the generation of appropriate shape and proportion of the body geometry through interpolation. Finally, we demonstrate our modifier synthesizer for more subtle manipulations of example models, using high-level parameters such as fat percentage. On any synthesized model, the underlying bone and skin structure is properly adjusted, so that the model remains completely animatable using the joint animation. By allowing automatic modification from a set of parameters, our approach may eventually lead to the automatic generation of a variety of population models.   Graphical Models (Academic Press) 2004. Synthesizing Animatable Body Models with Parameterized Shape Modifications Seo H., Cordier F. and Magnenat-Thalmann N.,   Based on an existing modeller that can generate realistic and controllable whole-body models, we introduce our modifier synthesizer for obtaining higher level of manipulations of body models by using parameters such as fat percentage and hip-to-waist ratio. Users are assisted in automatically modifying an existing model by controlling the parameters provided. On any synthesized model, the underlying bone and skin structure is properly adjusted, so that the model remains completely animatable using the underlying skeleton. Based on statistical analysis of data models, we demonstrate the use of body attributes as parameters in controlling the shape modification of the body models while maintaining the distinctiveness of the individual as much as possible.   ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2003. An Automatic Modeling of Human Bodies from Sizing Parameters Seo H., and Magnenat-Thalmann N.,   In this paper, we present an automatic, runtime modeler for modeling realistic, animatable human bodies. A user can generate a new model or modify an existing one simply by inputting a number of sizing parameters. We approach the problem by forming deformation functions that are devoted to the generation of appropriate shape and proportion of the body geometry by taking the parameters as input. Starting from a number of 3D scanned data of human body models as examples, we derive these functions by using radial basis interpolation. A prerequisite of such formulation is to have correspondence among example models in the database. We obtain the correspondence by fitting a template onto each scanned data. Throughout the paper, body geometry is considered to have two distinct entities, namely rigid and elastic component of the deformation. The rigid deformation is represented by the corresponding joint parameters, which will determine the linear approximation of the physique. The elastic deformation is essentially vertex displacements, which, when added to the rigid deformation, depicts the detail shape of the body. ...   ACM SIGGRAPH Symposium on Interactive 3D Graphics, 2003

# * Matching, similarity, segmentation (2013-)

 Luo G., Deng Z., Jin X., Zhao X.,Zeng W., Xie  W. and Seo H.   With the recent advances of data acquisition techniques, the compression of various 3D mesh animation data has become an important topic in computer graphics community. In this paper, we present a new spatio-temporal segmentation-based approach for the compression of 3D mesh animations...     ACM Trans. Multimedia Computing, Communications, and Applications, 2020. Extended version of the long paper presented at ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2019. Lavoué G., Cordier F., Seo H., and Larabi  C.  Understanding the attentional behavior of the human visual system when visualizing a rendered 3D shape is of great importance for many computer graphics applications. Eye tracking remains the only solution to explore this complex cognitive mechanism. Unfortunately, despite the large number of studies dedicated to images and videos, only a few eye tracking experiments have been conducted using 3D shapes. Thus, potential factors that may influence the human gaze in the specific setting of 3D, are still to be understood. In this work, we conduct two eye-tracking experiments involving 3D shapes, with both static and time-varying camera positions…   Computer Graphics Forum (Proc. Eurographics), 2018. Seo H. and Cordier F.,   This paper presents a new technique which makes use of deformation and motion properties between animated meshes for finding their spatial correspondences. Given a pair of animated meshes exhibiting a semantically similar motion, we compute a sparse set of feature points on each mesh and compute spatial correspondences among them so that points with similar motion behavior are put in correspondence. At the core of our technique is our new, dynamic feature descriptor named AnimHOG, which encodes local deformation characteristics. AnimHOG is ob-tained by computing the gradient of a scalar field inside the spatiotemporal neighborhood of a point of interest, where the scalar values are obtained from the deformation characteristic associated with each vertex and at each frame. The final matching has been formulated as a discreet optimization problem …   Computer Graphics Forum (Proc. Pacific Graphics), 2016. Luo G., Lei G. and Seo H.,   The detection of abrupt changes in a sequence is a main strategy for extracting key-frames. In this paper, we present a joint entropy based key-frame extraction method for 3D animations. We start by applying an existing deformation-based feature descriptor to measure the degree of deformation of each triangle within each frame, from which we compute the statistical joint probability distribution of triangles’ deformation between neighboring frames. To avoid noises, we modify to compute joint probability distribution between two consecutive subsequences of frames with a fixed length, with which we further compute joint entropy between the two subsequences. Finally, we extract key-frames by taking the local maximal from the joint entropy curve of a given 3D animation...   The Visual Computer (Springer) 2016. Extension of a short paper presented at Computer Graphics International (CGI), 2015. Luo G., Cordier F., and Seo H.   Although there have been a large body of works on computing the similarity of static shapes, similarity judgments on deforming meshes are not studied well. In this study, we investigate a similarity measurement method for comparing two deforming meshes. Based on the degree of deformation, we first binarily label each triangle within each frame as either 'deformed' or 'rigid', then merge the 'deformed' triangles in both spatial and temporal domains for the segmentation. The segmentation results are encoded in a form of evolving graph, with an aim of obtaining a compact representation of the motion of the mesh. Finally, we formulate the similarity measurement as a sequence matching problem. ...   The Visual Computer (Springer) 2015. Extension of an article presented at EG Workshop on 3DOR 2014 ("Similarity of Deforming Meshes Based on Spatio-temporal Segmentation") Mykhalchuk V., Seo H., and Cordier F.   Although automatic feature detection has been a long-sought subject by researchers in computer graphics and computer vision, feature extraction on deforming models remains as relatively unexplored area. In this paper, we develop a new method for automatic detection of spatio-temporal feature points on animated meshes. Our algorithm consists of three main parts. We first define local deformation characteristics, based on strain and curvature values computed for each point at each frame. Next, we construct multi-resolution space-time Gaussians and Difference-of-Gaussian (DoG) pyramids on the deformation characteristics representing the input animated mesh, where each level contains 3D smoothed and subsampled representation of the previous level. Finally, we estimate locations and scales of spatio-temporal feature points by using a scale-normalized differential operator. A new, precise approximation of spatio-temporal scale-normalized Laplacian has been introduced, based on the space-time DoG....   The Visual Computer (Springer) 2015. Extension/revision of an article presented at Computer Graphics International 2014 ("AniM-DoG: A Spatio-Temporal Feature Point Detector for Animated Mesh"). Luo G., Seo H., and Cordier F.,   The 31st Computer Graphics International (CGI), 2014. Mykhalchuk V., Cordier F., Seo H., We present an efficient and robust algorithm for the landmark transfer on 3D meshes that are approximately isometric. Given one or more custom landmarks placed by the user on a source mesh, our method efficiently computes corresponding landmarks on a family of target meshes. The technique is useful when a user is interested in characterization and reuse of application-specific landmarks on meshes of similar shape (for example, meshes coming from the same class of objects). Consequently, across a set of multiple meshes consistency is assured among landmarks, regardless of landmark geometric distinctiveness. The main advantage of our method over existing approaches is its low computation time. Differently from existing non-rigid registration techniques, our method detects and uses a minimum number of geometric features that are necessary to accurately locate the user-defined landmarks and avoids performing unnecessary full registration. In addition, unlike previous techniques that assume strict consistency with respect to geodesic distances, we adopt histograms of geodesic distance to define feature point coordinates, in order to handle the deviation of isometric deformation. ...   Computers & Graphics (Elsevier), 2013 (Vol. 37, Num. 5, pp. 539–552) Luo G., Cordier F., Seo H., Computer Animation and Virtual Worlds (Wiley), also presented at Computer Animation and Social Agents, 2013.

# * Sketch-Based Shape Reconstruction (2007-)

 Descrptive: Interactive 3D Shape Modeling from a Single Descriptive Sketch, Bobenrieth C., Cordier F., Habibi A., and Seo H. In this paper, we present a sketch-based modeler that reconstructs a 3D shape by combining a single descriptive sketch and minimal user intervention. The user provides a single 2D drawing in the form of a descriptive sketch, where solid curves describe the visible silhouette, and dashed curves the hidden outline. The curves are partitioned into a set of closed curves in a semi-automatic manner, each of which is consolidated into a closed surface element by solving a constrained optimization problem. The final 3D shape is generated by assembling these surface elements. The algorithmic reconstruction is complemented by allowing users to optionally guide the shape computation or correct any inaccuracy. This is done by successively specifying different kinds of local constraints on sparsely selected points in rotated views, such as adjustment of volume thickness along the projection line, or curvature discontinuity.…   Computer Aided Design 2020. Slides presented by Cédric Bobenrieth at journées du Groupe de Travail en Modélisation Géométrique 2020 (in French). Reconstructing Flowers from Sketches, Bobenrieth C., Seo H., Cordier F., Habibi A. As the symbol of beauty, floral objects have been one of the most popular subjects of artistic drawing. However, designing 3D floral models is generally time- and resource-consuming, because of their structural and geometrical complexity. In this paper, we address the problem of reconstructing floral objects from sketch input. The user draws a relatively clean sketch of a flower and a few additional guide markings from an arbitrary view to rapidly create quality geometric models of flowers. Our system offers a novel modeling scheme compared to several existing flower modelers accepting sketch as input, where the user is required to work with different views, providing step-by-step sketch input. …   Computer Graphics Forum (Pacific Graphics) 2018. Slides presented by Cédric Bobenrieth. Reconstruction of Helices from their orthogonal projection, Cordier F., Seo H., Melkemi M. We describe a method for modeling helices from planar curves. Given a polygonal curve in the (x,y$x,y$) plane, the method computes a helix such that its orthogonal projection onto the (x,y$x,y$) plane fits the polygonal curve. The helix curve is first sampled and the transformation matrix that best aligns points of the sampled helix to those of the polygonal curve is calculated. This transformation matrix is then used to estimate the parameters of the helix whose projection fits the polygonal curve. Computer Aided Geometric Design (Elsevier) 2016. Inferring mirror symmetric 3D shapes from sketches, Cordier F., Seo H., Melkemi M., N. S. Sapidis,   We describe a system for taking a 2D sketch of a mirror-symmetric 3D shape and lifting the curves to 3D, inferring the symmetry relationship from the original hand-drawn curves. The system takes as input a hand-drawn sketch and generates a set of 3D curves such that their orthogonal projection matches the input sketch. The main contribution is a method which is able to identify the symmetry relationship among the hand-drawn curves even in the presence of ambiguity in the sketch.   Computer-Aided Design (Elsevier) 2013, also presented at ACM Symposium on Solid and Physical Modeling 2012. Sketching of Mirror-symmetric Shapes Cordier F., Seo H., Park J., Noh J.,   This paper presents a system to create mirror-symmetric surfaces from free-form sketches. The system takes as input a hand-drawn sketch and generates a surface whose silhouette approximately matches the input sketch. The input sketch typically consists of a set of curves connected at their endpoints, forming T-junctions and cusps. Our system is able to identify the skewed mirror and translational symmetry between the hand-drawn curves and uses this information to reconstruct the occluded parts of the surface and its 3D shape.   IEEE Trans. on Visualization and Computer Graphics, 2011. Free-Form Sketching of Self-Occluding Objects, Cordier F. and Seo H.,   Sketching interfaces to 3D object modeling facilitate 3D object reconstruction from a 2D drawing provided by a designer. Igarashi presented the Teddy silhouette-based sketching system, which has a simple, intuitive interface. Followup research has mainly focused on the representation issue for the resulting 3D objects, as given in recent work: variational implicit surfaces2 and other forms of implicit surfaces.In this article, we address a different issue, the extension of the modeling domain. In particular, we consider the modeling of self-occluding objects (or multiple objects possibly occluding each other)…   IEEE Computer Graphics and Applications, 2007.

# * Texture Mapping (2010-)

 Constrained Texture Mapping using Image Warping, Seo H. and Cordier F.,   We introduce in this paper a new method for smooth foldover-free warping of images. It allows users to specify the constraints in two different ways: positional constraints to constrain the position of points in the image and gradient constraints to constrain the orientation and scaling of some parts of the image. We then show how our method is used for texture mapping with hard constraints. We start with an unconstrained planar embedding of the target mesh calculated with conventional methods. In order to obtain a mapping that satisfies the user-defined constraints, we use our warping method to align the features of the texture image with those of the unconstrained embedding. Compared to previous work, our method generates a smoother texture mapping and offers higher level of control for defining the constraints.   Computer Graphics Forum (Blackwell Publishers now Wiley) 2010.

# * Character Animation, Games, Virtual Reality Systems (2000-2010)

 3D Billiards Game with Haptic Interface Seo H., Bang J.-W., Park J.-M., Jeon S.-H.   Intuitive human-computer interaction has been an active research area with the participation of both academic and industrial groups. It is an important technical and psychological element in computer games, digital product design, cybersports, etc. In this paper, we present our work on a 3D billiards game featuring a haptic interface. Most of the existing games available today concentrate on the development of sophisticated physics engines to obtain realistic motion of billiards balls or of sound engines for realistic sound effect. Our focus in this work is on the improved realism on the hitting action of the game with the aid of haptic device. We do so by computing accurate feedback force on the haptic handle, considering the exact point of contact and physically faithful collision response. Also presented is the physically based modeling of the game environment, including the collision interaction among billiards balls, and between the table and billiards balls…   Computer Animation and Virtual Worlds (John Wiley & Sons) 2010. Automatic Transition Detection of Segmented Motion Clips Using PCA-based GMM Method Wang Y., Seo H., and Jeon S.-H., In this work, we record a dancer’s rhythmic movement with background music. The captured motion sequences are then segmented into dozens of motion clips, to construct a motion database consisting of sets of labeled motion clips. Since the dance motion has been synchronized with background music, the segmentation is rather straightforward – we segment the motion sequence with regular frame distance, or more precisely, at every major beat of the music. Many of these motion clips contain short and rapid transition from one main dancing motion to another, which causes unnatural, awkward movements when they are connected in different orders than the original sequence. In this paper, we describe our approach for automatically detecting the transition parts in the segmented motion clips....   Int’l Conference on Cyberworlds 2008. Motion Control of a Dancing Character with Music   Kim G.-W., Wang Y., and Seo H., pp. 930-936, Proc. IEEE/ACIS 6th Int’l Conf. on Computer & Information Science 2007 (11-13 July), Melbourne, Australia. Simulating Virtual Humans in Networked Virtual Environments,   Pandzic I., Babski C., Capin T., Lee W., Magnenat-Thalmann N., Musse S. R., Moccozet L., Seo H. and Thalmann D.,   In the past decade Networked Virtual Environments have been an increasingly active area of research, with first commercial systems emerging recently. Graphical and behavioral representation of users within such systems is a particularly important issue that has lagged in development behind other issues like network architectures, space structuring etc. In this paper we expose the importance of using Virtual Humans within these systems and provide a brief overview of several Virtual Humans technologies used in particular for simulation of crowds. As the main technical contribution the paper presents the integration of these technologies with the COVEN-DIVE platform, the extension of the DIVE system developed within the COVEN project. In conjunction with this, we present our contributions through the COVEN project to the MPEG-4 standard concerning the representation of Virtual Humans.   Presence (MIT Press), 2001. VPARK -- A Windows NT Software platform for a Virtual Networked Amusement Park Seo H., Joslin C., Berner U., Magnenat-Thalmann N., Jovovič M., Esmerado J., Thalmann D. and Palmer I.,   Computer Graphics International 2000 (IEEE Computer Society Press). LoD Management on Animating Face Models Seo H., and Magnenat-Thalmann N.,   In this paper, we present our work on a level of detail(LoD) technique for human-like face models in virtual environments. Conventional LoD techniques have been adapted to allow facial animation on simplified geometric models. This includes optimization on both geometry, and animation parameters. Simplified models are generated in a region-based manner in consideration with the mobility of each region. The animation process is decomposed into two sub-processes and each step is optimized. In the MPA(Minimum Perceptible Action) level optimization, a hierarchical structure was devised for the multi-level animation model. The deformation level is simplified by reducing the number of control points.  During runtime, the level of animation is selected in combination with the viewpoint information the level of geometry.   IEEE Virtual Reality 2000.

# * Shape Signal Processing

 Fourier-Based Modeling of Topologically Complex Bone Data Using Various Alternatives of 3D Scalar Fields Piao Y., Lim I.-S., and Seo H.   This article presents a new approach for Fourier-based modeling of bone anatomies for compression and smoothing. By treating the bone surface as a level set of a 3D scalar field, we can model topologically complex models such as bones. In particular, we experiment with five different alternatives, and prove that 3D scalar field which allows monotonically continuity around the boundary can be a good choice for volumetric description of the surface. This allows avoiding Gibbs phenomena which previous volume-based methods have suffered from, returning better results in compression and smoothing than other scalar fields. We demonstrate the efficacy of the proposed method by showing results with various bone data.   IEEE Int’l Conf. on Acoustics, Speech and Signal Processing 2009.