Providing user guidance to use and train a generative adversarial network

The present invention relates to a computer program product, system, and method for providing user guidance to use and train a generative adversarial network.

Generative Adversarial Networks (GAN) are trained to generate realistic images. A style GAN (StyleGAN) is trained to adjust the image via style mixing or a style vector to control the image output to incorporate the style mixing. In a GAN, a discriminator, comprising a machine learning model trained to recognize real images, such as photographs, in a particular domain, classifies output from a generator machine learning model, seeded with random input, as real or fake images. If the generator output is classified as fake, that feedback of the fake result for the output is fed back into the generator using backpropagation to train the generator not to produce that output classified as fake. The generator continually produces output that is classified by the discriminator until a point is reached where the discriminator classifies the generator output as real. For instance, the discriminator may be trained to classify human faces as real or fake from a training set of real human photographs. The generator is trained with the feedback from the discriminator on whether it is outputting realistic facial photographs until a point is reached where the discriminator classifies the generator output facial images as real. In this way, the generator and discriminator in a GAN compete against each other.

A conditional GAN provides additional input to the discriminator and generator, such as a class label, to provide a classification of the data being generated. This allows a condition to be used to control the generator to output a specific type of image and to train the discriminator to classify as real or fake an image of a specific type.

There is a need in the art for improved techniques to train and use a GAN in various use case scenarios.

Provided are a computer program product, system, and method for providing user guidance to use and train a generative adversarial network. A discriminator, implementing machine learning, is trained to output a high probability from input comprising an input image comprising a real image representing a desirable design item superimposed on a body in a pose represented in the input image, a context of the input image and a pose of the body represented in the input image. The desirable design item comprises a design item determined to be desirable to a target audience. A generator, implementing machine learning, generates an output image based on an input context of a design item to generate, an input pose of a body on which the design item will be superimposed in the output image, and random noise. The discriminator outputs a probability the output image, from the generator, represents a real image having a desirable design item from input comprising the output image, the input context, and the input pose. The generator is trained to output the output image for the input pose and the input context with the probability outputted by the discriminator.

Described embodiments provide improvements to computer technology for a Generated Adversarial Network (GAN) by allowing the user to provide guidance to control the GAN to produce images that represent design items in a user selected pose and context. Further, described embodiments provide techniques for gathering images determined to have a high desirability to a target audience and use that feedback on the design items represented in the images to train the GAN to generate realistic images having desirable design items to a target audience. Described embodiments provide improvements to computer technology to obtain feedback scores from images generated by the GAN to use to further train the GAN discriminator to recognize images representing desirable design items. The discriminator output is then used, in turn, to train the GAN generator to output realistic images representing desirable design items.

illustrates an embodiment of an image generation and feedback systemhaving a conditional generator adversarial network (cGAN)including a generator, implementing a machine learning model, that is trained to receive user inputcomprising a contextand a poseof a body, and random noisefrom a random noise generator, to generate an imagewith a generated design item superimposed over a body in the pose. The generated imagemay be stored in an image file, as shown in, including an image identifier (ID), such as a name of the image; a context, such as context; a pose, such as pose; and image contentcomprising the imagegenerated by the generator. The embodiment ofis advantageous for computational processing operations because the image filehas metadata of the contextand poseof the generated imagethat may later be used to form a training set, shown in, to train the generatorand discriminatoras part of a conditional GAN, or cGAN.

The generated design items represented in the imagemay comprise articles of clothing superimposed over the body in the poseor accessories adorning the person in the pose, such as clothes, shoes, handbags, hats, head coverings, jewelry, accoutrements, etc. The contextmay comprise one or more classifications of a contextof the image for the generatorto generate, including, without limitation, an activity in which the body in the poseis engaged, a style of design items to be superimposed over the body in the pose, an environment in which the design items are to be worn, e.g., formal wear, exercise, casual, beach ware, demographics of a person that would utilize the design items, such as age, region, etc., a cultural milieu in which the design items would be presented (e.g., rancher, youth culture, retirement, hipster, urban, rural, beach life, high society, etc.). The posemay comprise a mask showing positions of body keypoints that represent a pose on which the design items will be superimposed.

The cGAN cincludes a discriminator, implementing a machine learning model, that is trained to classify received input images, such as images representing a body in a pose with design items superimposed over the body, as real images of desirable design items. The discriminatoroutputs a probability/confidence level the input imageis a real image of design items superimposed or worn on a body in a pose and the represented design items are desirable to the target market of the design items. The discriminatormay be trained with a labeled data set of real images of people in poses, adorned with design items that have been determined through market research to be desirable to target markets, to output a high probability for real images with desirable design items to the target market and output a low probability for fake images or real images with design items determined to be undesirable or of low interest to the target market. The discriminatormay be trained by the developer of the GANbefore distributing to users and may also be trained during operations while deployed in a user systemon data sets including images generated by the generatorand feedback on the images, such as feedback gathered during marketing research.

A material model, comprising a machine learning model, may be trained to receive as input the imagesincluding their context and pose, and generate information on materialsthat may be most suitable to manufacture the design items based on the context. The material modelmay receive as input the image fileincluding the generated image, context, and poseto generate the information on materialsto use to manufacture the design items represented in the generated image.

The output imagefrom the generatormay further be sent to a feedback systemwhich may assemble multiple imagesinto an image portfoliocomprising one or more imagesof design items superimposed over bodies in the user provided poseand context. The portfoliomay be transmitted over a network, such as the Internet, to post on a social network, including a metaverse or online virtual reality community, to be presented to a target market that would most likely be interested in the design items represented in the imagesin the portfolio. Viewers of the portfolioon the social networkmay provide feedback, such as a feedback score, indicating the extent to which the users of the social networkapprove or desire the design items represented in the images, or are uninterested or disapprove of the design items represented in the images. The feedback systemmay generate a training setcomprising the imagefor which feedback is provided, the contextand poseused to generate the image, and the feedback scoreto provide to the discriminatorthat trains the generator. In this way, the feedback systemengages in product market research, by generating product research images of prospective design items, generated by the cGAN, to post on a social networkto receive crowdsourced feedback on the desirability of the product research images.

is an embodiment of a training setthat includes an image IDof an image that will be used to train the discriminator; contextof the image; a poseof a body represented in an image; image content; and a probability (confidence level) or favorability scoreat which the generatorand discriminatorshould generate the image.

illustrates an embodiment of the training process implemented in the cGANwith the generatorand discriminatordescribed with respect to. A training manager, used to train the generator, generates, for training purposes, a poseand contextand random input, from a random generator, to seed the generatorto produce a training image. The generatorgenerates a training imagethat is included in a training image fileand sent to the discriminator. The discriminatorprocesses the training imageand training contextand training pose, from which the training imagewas generated, and outputs a probabilityindicating the extent to which the training imageis real, such as from a photo, and likely desirable to a target audience or is fake or real and not likely desirable to a target audience. If the training imagesare classified as fake or real and not desirable, then that is a generator lossbecause the generatorwas unable to generate an image that would trick the discriminatorinto classifying the training imageas real and desirable. This generator losswould trigger the generatorto retrain and adjust its weights and parameters to output the training imagewith contextand posewith a low probability/confidence level.

The probabilitymay comprise a confidence level indicating a degree of confidence, such as a percentage, in the prediction of whether the input image is real and has design items desirable to a prospective market. Discriminatormay be trained to output a probability, i.e., confidence level, for real and desirable images that is set to a relatively high value such as 80%. A confidence level, confidence interval or confidence score may comprise a number between 0 and 1, or other numerical range or fixed number of levels (e.g., high, medium or low), that represents the likelihood that the output of the discriminatorcorrectly predicts an image as real and having desirable design items. The confidence level comprises a margin of error in the cost function of the probabilityand a ground truth probability or feedback scoreassigned to the images in the training image dataset, such that if the feedback scoreis high, e.g., 95%, indicating high desirability of a real design item, then the discriminatoris trained to produce the probabilitywith a margin of error within 5%.

If the training imageis classified as real, then that is a discriminator lossbecause the discriminatorwas unable to discern the imageas fake from the generator. This discriminator lossmay trigger the discriminatorto retrain and adjust its weights and parameters to output the training imagewith a low confidence level or probability, which would indicate it is fake.

Further, the discriminatormay be trained in a user deployed cGANor by the vendor on a training data setincluding historical training setsof images determined to be real and represent desirable items. The training data setmay further comprise feedback training setsgenerated by the feedback systemthat indicates the result of marketing research by posting the imagesfrom the generatoron a social networkwebsite. Data set training images, from the data set, along with an accompanying contextand poseof the data set imageare inputted into the discriminatorto produce a training probabilityfor the data set and a discriminator loss, if any, that is backpropagated through the discriminatorto retrain the discriminatorto output the training data set imagewith a probability provided with the training set,in the data set.

illustrates an alternative embodiment of the cGANinas cGANwhere the pose and context are not included in the training data settraining sets. This would often be the case because obtaining labeled data sets may be difficult. In such case, where the pose and context are not provided with the training data setsin data set, cGANincludes a pose estimation model, implementing machine learning, to process the imageand output an estimated poseof a body represented in the image. The cGANfurther includes a context model, implementing machine learning, to process the imageand output an estimated context. The estimated pose, estimated context, and the imageare then inputted into the discriminatorto output a probabilitythe image is real and desirable, such as if the input imageshave desirable design items, with a high probability value, as described above with respect to.

In certain embodiments, the cGANmay comprise a style based GAN, such as StyleGAN, which offers control over the style of the generated image. The discriminatormay comprise a classification neural network. In certain embodiments, many of the described components, such as the generator, discriminator, material model, pose estimation model, and context modelmay use machine learning and deep learning algorithms, such as decision tree learning, association rule learning, neural networks, inductive programming logic, support vector machines, Bayesian network, recurrent neural networks (RNN), Feedforward neural networks, Convolutional Neural Networks (CNN), Deep Convolutional Neural Networks (DCNNs), Generative Adversarial Network (GAN), Conditional GAN (cGAN), etc. For artificial neural network program implementations, the neural network may be trained using backward propagation to adjust weights and biases at nodes in a hidden layer to produce their output based on the received inputs. In backward propagation used to train a neural network machine learning module, biases at nodes in the hidden layer are adjusted accordingly to produce the expected output having specified confidence levels based on the input parameters.

Backward propagation may comprise an algorithm for supervised and semi-supervised learning of artificial neural networks using gradient descent. Given an artificial neural network and an error function, the method may use gradient descent to find the parameters (coefficients) for the nodes in a neural network or function that minimizes a cost function measuring the difference or error between actual and predicted values for different parameters. The parameters are continually adjusted during gradient descent to minimize the error.

In backward propagation used to train a neural network machine learning module, margins of error are determined based on a difference of the calculated predictions and user rankings of the output. Biases (parameters) at nodes in the hidden layer are adjusted accordingly to minimize the margin of error of the error function.

In an alternative embodiment, other techniques may be used to train the components, such as an unsupervised machine learning module, or machine learning implemented in methods other than neural networks, such as multivariable linear regression models.

The networkmay comprise a network such as a Storage Area Network (SAN), Local Area Network (LAN), Intranet, the Internet, wireless network, broadband network, satellite network, etc.

The arrows shown inbetween the components and objects represent a data flow between the components.

Generally, program modules, such as the program components,,,,,,,,, among others, may comprise routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The program components and hardware devices of the systemand GANs,ofmay be implemented in one or more computer systems, where if they are implemented in multiple computer systems, then the computer systems may communicate over a network.

The program components,,,,,,,,, among others, may be accessed by a processor from memory to execute. Alternatively, some or all of the program components,,,,,,,,, among others, may be implemented in separate hardware devices, such as Application Specific Integrated Circuit (ASIC) hardware devices and accelerator engines.

The functions described as performed by the program,,,,,,,,, among others, may be implemented as program code in fewer program modules than shown or implemented as program code throughout a greater number of program modules than shown.

The image generation and feedback systemmay comprise a server or other type of computing device, such as a laptop, desktop computer, tablet, smartphone, wearable computer, etc.

illustrates an embodiment of operations performed by components in the cGAN,to train the discriminator. Upon initiating (at block) a discriminatortraining phase to train the discriminator on training setsin a training data set,, for each image in the training data set,, the cGAN,determines (at block) the context.and pose.of the image.. In the embodiment of, the contextand posemay be included with metadata for the training image. In the embodiment of, the pose estimation modeland context modelmay process the training imageand output an estimated poseand estimated context, respectively.

The training image,, context,and pose,are inputted (at block) into the discriminatorto output a probability,reflecting that the image is real and likely desirable to a target market. If (at block) the probability exceeds a high threshold, e.g., 90%, then control ends as the discriminatorhas accurately predicted the training set image as real and representing desirable items. If the training image in the training data set,was produced with a low probability, indicating it was fake or real and not desirable, then the discriminatorwould be trained to output a low probability,. If (at block) the probability does not exceed the high threshold, for real/desirable images, or low for fake or real/desirable images, then the cGAN,may use (at block) backpropagation to retrain the discriminatorto output a high probability value, e.g., 0.9, for input comprising a real image representing desirable design items.

With the embodiment of, an improved cGAN training technique is provided to train a discriminatorto recognize images based on conditional information, such as context and pose, that are real and represent desirable design items based on feedback obtained by a feedback system.

illustrates an embodiment of operations performed by components in the cGAN, as described with respect to, to train the generator. Upon initiating (at block) operations to train the generator, the training managergenerates (at block) inputs of random noise, context, and poseto input to the generatorto generate an output image. The training managerinputs (at block) the output image, from the generator, along with the context, and poseinput, into the discriminatorto generate the probabilityindicating the imageas real and representing desirable design items or one of fake and real representing undesirable design items. For imagesfrom the generatorclassified as fake or real representing undesirable design items, resulting in generator loss, the training managertrains (at block) the generator, using backpropagation, to adjust the parameters to minimize the probability the generator outputs the image, which is fake or real representing undesirable design items, from the previously inputted poseand context, and noise. For imagesfrom the generatorthe discriminatorclassifies as real and representing desirable design items, the training managertrains (at block) the generatorto adjust its parameters to maximize the probability the generator outputs the imagethe discriminator classifies as real and representing desirable design items from the previously inputted poseand context, and noise.

With the embodiment of, the cGANtrains the generatorto output real images representing desirable design articles based on the poseand contextof an imageto generate. Further, generator lossfrom the discriminatoris used to adjust the generatoroutput when it does not produce imagesthat the discriminator classifies as fake or real images representing desirable design items. This provides improvements to computer technology for generating images having design items that will be desirable to a target market.

illustrates an embodiment of operations performed by the cGANand feedback systemto operate the generator, as shown in, in a user mode to generate images representing design items to present to a prospective market to obtain feedback. Upon operating (at block) the generatorin user mode, the cGANreceives user inputof a selected contextand poseto incorporate into the imageto generate. The user selected contextand posemay be expressed in a semantic or pictorial format in human understandable form. The human understandable poseand contextmay be translated to a computer readable vector or mask. The cGANinputs (at block) the user selected contextand pose, which may have been translated to a vector or format suitable for processing by the generator, and noiseinto the generatorto output the imagerepresenting design items, e.g., clothing, on the body in the pose. The feedback systemmay generate (at block) a catalogof one or more generated imagesfor the same or different poses and contexts. The catalogmay be in digital format.

The feedback systemmay post (at block) the catalogon a social network account of the user of the systemwith feedback prompts to allow people viewing the portfolio images on the social networkto provide a feedback scoreof the design items represented in the images. The feedback scorefor a particular image indicates the extent the user desired or was positively disposed to the design item represented in the image. Upon receiving (at block) feedback scoresfor the images in the portfolio, the feedback systemgenerates (at block) training sets, where each training setincludes the context, pose, image content, and an aggregated feedback score, from the different received feedback scoresfor the image, in fields,,, and, respectively, of the training set. The feedback systemforwards (at block) the generated training setto the discriminator, shown in, to train the discriminator to classify the image with a probability derived from the aggregated feedback score.

With the embodiment of, the systemmay be used to initiate a marketing research campaign for the user to obtain feedback on design items, e.g., clothing and accessories, represented in the images, generated by the generator, to determine design items that are desirable to specific marketing segments, based on the contexts of the images, to then select for manufacturing and sale. Described embodiments integrate a cGANto generate images representing desirable design items based on user selected context and pose and post the images on a social network site to obtain crowdsourced feedback from the social network. This crowdsourced feedback may be used to further train the cGANto output realistic images representing desirable design items.

illustrates an embodiment of operations performed by the cGANofto retrain the generatorand discriminatorbased on the crowdsourced feedback scoresin the training sets. Upon the cGANreceiving (at block) training setsfrom the feedback systemincluding crowdsourced aggregated feedback scoresfrom the social network, the cGANmay convert (at block) the crowdsourced aggregated feedback scoresin the training setsto crowdsourced probabilities. The cGANperforms the operations of(at block) to use backpropagation to retrain the discriminatorto output the probabilities, converted from the crowdsourced feedback scores, from the image, context, and posein the crowdsourced training set. The cGANperforms the operations of(at block) to retrain the generatorusing the discriminator, retrained at stepbased on the crowdsourced probabilities, to improve the ability of the generatorto output realistic images having desirable design items.

With the embodiment of, the discriminatoris retrained to output the images with the crowdsourced probabilities from the crowdsourced feedback scoresfrom the social network. The discriminatoris trained to output a low probability/feedback score indicating the design items are not desirable to the target market or high indicating the design articles are desirable. Further, generatoris retrained to output images representing desirable design items as determined from crowdsourced feedback scoresfrom a social network.

In the described embodiments, the design items comprised clothing, accessories, shoes, jewelry, and other articles a person wears. In alternative embodiments, the design items may comprise design items a person wants to purchase unrelated to clothing, such as artwork, automobiles, furniture, smartphones, non-fungible tokens (NFTs), computers, etc., and the context and pose may be appropriate to the design item.

In further embodiments, a video and image analysis module may be used to identify different poses in video and images the user may select for input to the generator. In further embodiments, a pose estimation model may analyze videos to create a series of poses with stick figures to use as the input poseto the generator. Further embodiments may identify stick figure poses to classify based on different types of activities performed by different types of users. Still further, the stick figure poses may be classified based on demographic, location, and the purpose of usage. The extracted information on poses may then be provided as input to the GAN to enable the GAN to create different types of design items having different styles and designs.

In further embodiments, the training data setmay be gathered from images of existing fashion item stocks, and their usage, and sentiments on the fashion items from social network sites. Once the generator generates simulated fashion items, then the feedback systemwill present assemble the generated fashion items appear, with different poses and activities, to create a fashion photography portfolio for different types of fashion items represented in the generatorgenerated images. The imagesof GAN generated fashion items with different poses are posed on the social network sites to gather public sentiments, and to allow further adjustment of design items represented in further generated images before a final design is created, or sent to the material model for manufacturing specifications. This allows crowdsourcing to train the cGAN generatorby having participants of a social network, such as a metaverse, review the fashion portfolio in order to garner feedback from the participants about the fashion items, and the same will be used to retrain the discriminatorto further refine the generator to refine the generated fashion items.

In further embodiments, for the material model, a pose estimation model may analyze the poses and the activities in different images to identify types of forces applied on different fashion items while performing different activities and to identify different applied forces on different sides of the fashion item during usage, as part of determining the materialsto use to manufacture the design items represented in the images.

In further embodiments, the generatormay receive additional conditional inputs, including pre-defined rules, to generate the imagefor different types of fashion items. The pre-defined rules may comprise different types of guideline, such as color, material used, size, dimension related information etc.

The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

Various aspects of the present disclosure are described by narrative text, flowcharts, block diagrams of computer systems and/or block diagrams of the machine logic included in computer program product (CPP) embodiments. With respect to any flowcharts, depending upon the technology involved, the operations can be performed in a different order than what is shown in a given flowchart. For example, again depending upon the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, concurrently, or in a manner at least partially overlapping in time.

A computer program product embodiment (“CPP embodiment” or “CPP”) is a term used in the present disclosure to describe any set of one, or more, storage media (also called “mediums”) collectively included in a set of one, or more, storage devices that collectively include machine readable code corresponding to instructions and/or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device that can retain and store instructions for use by a computer processor. Without limitation, the computer readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these mediums include: diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device (such as punch cards or pits/lands formed in a major surface of a disc) or any suitable combination of the foregoing. A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals communicated through a wire, and/or other transmission media. As will be understood by those of skill in the art, data is typically moved at some occasional points in time during normal operations of a storage device, such as during access, de-fragmentation or garbage collection, but this does not render the storage device as transitory because the data is not transitory while it is stored.

illustrates computing environmentproviding an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods to generate realistic images representing desirable design items superimposed on a body in a pose in block, including the feedback system, cGAN, and material modeldescribed above with respect to. In addition to block, computing environmentincludes, for example, computer, wide area network (WAN), end user device (EUD), remote server, public cloud, and private cloud. In this embodiment, computerincludes processor set(including processing circuitryand cache), communication fabric, volatile memory, persistent storage(including operating systemand blockas identified above), peripheral device set(including user interface (UI) device set, storage, and Internet of Things (IoT) sensor set), and network module. Remote serverincludes remote database. Public cloudincludes gateway, cloud orchestration module, host physical machine set, virtual machine set, and container set.

COMPUTERmay take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer, mainframe computer, quantum computer or any other form of computer or mobile device now known or to be developed in the future that is capable of running a program, accessing a network or querying a database, such as remote database. As is well understood in the art of computer technology, and depending upon the technology, performance of a computer-implemented method may be distributed among multiple computers and/or between multiple locations. On the other hand, in this presentation of computing environment, detailed discussion is focused on a single computer, specifically computer, to keep the presentation as simple as possible. Computermay be located in a cloud, even though it is not shown in a cloud in. On the other hand, computeris not required to be in a cloud except to any extent as may be affirmatively indicated.

PROCESSOR SETincludes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitrymay be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitrymay implement multiple processor threads and/or multiple processor cores. Cacheis memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located “off chip.” In some computing environments, processor setmay be designed for working with qubits and performing quantum computing.

Computer readable program instructions are typically loaded onto computerto cause a series of operational steps to be performed by processor setof computerand thereby effect a computer-implemented method, such that the instructions thus executed will instantiate the methods specified in flowcharts and/or narrative descriptions of computer-implemented methods included in this document (collectively referred to as “the inventive methods”). These computer readable program instructions are stored in various types of computer readable storage media, such as cacheand the other storage media discussed below. The program instructions, and associated data, are accessed by processor setto control and direct performance of the inventive methods. In computing environment, at least some of the instructions for performing the inventive methods may be stored in blockin persistent storage.

COMMUNICATION FABRICis the signal conduction path that allows the various components of computerto communicate with each other. Typically, this fabric is made of switches and electrically conductive paths, such as the switches and electrically conductive paths that make up buses, bridges, physical input/output ports and the like. Other types of signal communication paths may be used, such as fiber optic communication paths and/or wireless communication paths.