Visual display systems and method for manipulating images of a real scene using augmented reality

This application is a continuation of U.S. patent application Ser. No. 17/338,837, filed Jun. 4, 2021, which is a continuation of U.S. patent application Ser. No. 16/185,269, filed Nov. 9, 2018, which is a continuation of U.S. patent application Ser. No. 16/007,284, filed Jun. 13, 2018, which is a continuation of U.S. patent application Ser. No. 16/000,861, filed Jun. 5, 2018. The content of the foregoing applications is incorporated herein in its entirety by reference.

This disclosure relates generally to the field of augmented reality. More specifically, and without limitation, this disclosure relates to systems and methods for manipulating images of a real scene using augmented reality.

Current techniques for authentication and verification of users suffer from multiple drawbacks. For example, in a typical authentication or verification method, a user registers for an account and provides registration information required for authentication. This registration information typically includes a username and password. As many passwords can be easily guessed, circumvented, or cracked with current technologies, users are typically required to enter a complex password consisting of a combination of different letters, numbers, and/or symbols. However, complex passwords still suffer from technical problems and create security vulnerabilities. Additionally, such passwords can be difficult for users to remember, creating greater security risks when users reuse passwords across multiple accounts or store their passwords in easily accessible locations or documents. Moreover, the typical user interfaces for authentication and verification are standardized and thus easy to manipulate, presenting further security risks. For example, common phishing scams include sending a user an email containing a link to a legitimate-looking login page, where the user is tricked into entering sensitive account information. A need therefore exists for systems and methods of providing improved authentication and verification for users in a secure, user-friendly interface.

Augmented reality technology allows virtual imagery to be mixed with a real-world physical environment. In other words, computer-generated information may be superimposed on images of real-world elements of a physical environment to provide interactive and enhanced user environment. Augmented reality techniques are typically performed in real time and in semantic context with environmental elements. Immersive perceptual information is sometimes combined with supplemental information. For example, the augmented reality view may include information related to a physical environment presented to the user, via a display.

Commercial applications for augmented reality technology are primarily for informational purposes. For example, augmented reality technology may be used in sports broadcasts to superimpose live scores over a live video feed of a sporting event. While it is known to use augmented reality tools to enrich user experience, at present, there is no available method to integrate augmented reality technology into user authentication methods.

The disclosed system and methods for presenting augmented reality content to a user address the existing problems set forth above, as well as other deficiencies in the prior art, and are directed to an improved visual display system for authenticating users by manipulating images of a real scene using augmented reality technology.

Embodiments of the present disclosure use visual display systems and methods to manipulate images of a real scene using augmented reality. The visual display systems enable secure registration and authentication of users via augmented reality technology. In this manner, the disclosed embodiments can provide a marked improvement over the processes known in the prior art.

One embodiment of the present disclosure is directed to a visual display system for manipulating images of a real scene using augmented reality. The system includes at least one processor in communication with a first mobile device, and a storage medium storing instructions that, when executed, configure the at least one processor to perform operations. The operations may include receiving a request from the first mobile device to access an account of a user, receiving a first image from an image sensor of the first mobile device, the first image depicting a real scene, receiving a selection of a first virtual object, receiving an augmented reality image comprising the first virtual object overlaid on the first image, comparing the augmented reality image to one or more stored augmented reality images, authenticating the user based on the comparison, and authorizing access to the user account based on the authentication.

Another embodiment of the present disclosure is directed to a computer-implemented method for manipulating images of a real scene using augmented reality. The method comprises receiving a request from a first mobile device to access a user account, receiving a first image from an image sensor of the first mobile device, the first image depicting a real scene, receiving a selection of a virtual object, receiving an augmented reality image comprising the virtual object overlaid on the first image, comparing the augmented reality image to one or more stored augmented reality images, authenticating the user based on the comparison; and authorizing access to the user account based on the authentication.

A further embodiment of the present disclosure is directed to a non-transitory computer-readable medium. The computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to operate a visual display system for manipulating images of a real scene using augmented reality including receiving a request from a first mobile device to access a user account, receiving a first image from an image sensor of the first mobile device, the first image depicting a real scene, receiving a selection of a virtual object, receiving an augmented reality image comprising the virtual object overlaid on the first image, comparing the augmented reality image to one or more stored augmented reality images, authenticating the user based on the comparison, and authorizing access to the user account based on the authentication.

In some embodiments, the visual display system is configured to perform further operations including receiving a request to register the user using augmented reality, receiving a registration image from an image sensor of a second mobile device, the registration image depicting a real scene, receiving a selection of a second virtual object, generating an augmented reality registration image comprising the second virtual object overlaid on the registration image, and storing the augmented reality registration image in the user account.

In some embodiments, the visual display system is configured to perform further operations including determining a location of the second mobile device, and embedding location information into the registration image based on a determined location of the second mobile device. In further aspects, the visual display system is configured to perform further operations including determining a location of the first mobile device, and embedding location information into the first image based on a determined location of the first mobile device.

In some embodiments, the visual display system is configured to perform further operations including verifying that the first image was received from the image sensor of the first mobile device. The verification may include at least one of detecting a file format of the first image, detecting usage of the image sensor, detecting movement of the first mobile device, or determining a location of the first mobile device. In one aspect, the file format of the first image includes video data and still image data.

In some embodiments, the visual display system is configured to perform further operations including calculating a comparison metric associated with the comparison of the augmented reality image and one or more augmented reality registration images stored in a database, and matching the augmented reality image to an augmented reality registration image based on the comparison metric. In some aspects, the comparison metric comprises a metric generated by a machine learning processor.

In some embodiments, the visual display system is configured to perform further operations including calculating a first comparison metric associated with a comparison of the augmented reality image to one or more stored augmented reality registration images, and calculating a second comparison metric associated with a comparison of the first image with one or more stored registration images.

In some embodiments, the visual display system is configured to perform further operations including determining a confidence score representing a level of confidence that the augmented reality image matches the augmented reality registration image, and authenticating the user when the determined confidence score is above a first threshold. In some aspects, the user is authenticated only when the determined confidence score is at least equal to the first threshold. In yet another aspect, a verification challenge is performed when the determined confidence score is below the first threshold and above a second threshold. The verification challenge may include a one-time passcode transmitted to the user via the first mobile device. The user may be authenticated when the system receives a one-time passcode matching the transmitted one-time passcode. The one-time passcode may include a passcode valid only within a predetermined amount of time.

In some embodiments, the visual display system is configured to perform further operations including determining a confidence score based on the comparison metric. The comparison metric may include a percentage match between the augmented reality image and the augmented reality registration image. In some embodiments, the comparison metric may comprise a first comparison metric associated with a comparison of the augmented reality image to one or more stored augmented reality registration images, and a second comparison metric associated with a comparison of the first image with one or more stored registration images. In some embodiments, the confidence score is based on the first and/or the second comparison metric.

In some aspects, the confidence score is determined based on a comparison of a factor associated with the augmented reality image and the augmented reality registration image. The factor may include one or more of embedded location information, a time stamp, a date stamp, lighting conditions, or image metadata.

In some embodiments, the visual display system is configured to perform further operations including displaying the registration image on a display of the second mobile device, overlaying a grid on the displayed registration image, and overlaying the second virtual object within a user-selected position on the grid.

In some embodiments, the visual display system is configured to perform further operations including receiving a user identifier associated with the user account, retrieving an augmented reality registration image stored in the user account, and transmitting a translucent version of the augmented reality registration image for display on the first mobile device. The translucent version of the image may provide guidance for obtaining the first image.

Additional objects and advantages of the present disclosure will be set forth in part in the following detailed description, and in part will be obvious from the description, or may be learned by practice of the present disclosure. The objects and advantages of the present disclosure will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the disclosed embodiments.

The disclosed embodiments relate to systems and methods for authentication by manipulating images of a real scene using augmented reality. Reference will now be made in detail to exemplary embodiments and aspects of the present disclosure, examples of which are illustrated in the accompanying drawings. While numerous specific details are set forth in order to provide a thorough understanding of the disclosed example embodiments, it would be understood by those skilled in the art that the principles of the example embodiments may be practiced without every specific detail. Unless explicitly stated, the example methods and processes described herein are not constrained to a particular order or sequence, or constrained to a particular system configuration. Additionally, some of the described embodiments or elements thereof can occur or be performed simultaneously, at the same point in time, or concurrently. Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings.

Embodiments of the present disclosure may be implemented using a general-purpose computer. Alternatively, a special-purpose computer may be built consistent with embodiments of the present disclosure using suitable circuit elements.

As used herein, the term “image data” or “image” may refer to digital information representing an image and stored in an appropriate format. Appropriate formats include, for example, static image formats (e.g., bitmap, Graphics Interchange Format (“GIF”), Portable Network Graphics format (“PNG”), Joint Photographic Experts Group (“JPEG”)) or dynamic formats (e.g., animated GIF, MPEG-4, Flash Video (“FLV”), Windows Media Video (“WMV”)). The term “image data” or “image” may also refer to digital information representing an image and stored in an iOS “live” format, comprising a combination of a static image file (e.g., JPEG) and a video file (e.g., HD video or quicktime movie file).

1 FIG. 1 FIG. 100 100 100 105 110 120 130 140 150 160 100 180 190 100 170 100 170 is a block diagram of an exemplary system. Systemmay be used to register and authenticate a user by manipulating images of a real scene using augmented reality, consistent with disclosed embodiments. Systemmay include a server systemwhich may include a registration module, an authentication module, an image-processing module, a location determination system, a processor, and a memory. Systemmay additionally include a databaseand a mobile device. In some embodiments, as shown in, each component of systemmay be connected to a network. However, in other embodiments components of systemmay be connected directly with each other, without network.

150 160 Processormay include one or more processors, microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, servers, or any combination thereof. Memorymay include one or more storage devices configured to store data which may be read by a processor, computer, or like device.

160 160 Memorymay include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of computer-readable medium or computer-readable storage devices. For example, memorymay include random-access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM), ROM, magnetic or optical storage medium, flash memory devices, electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals).

180 105 110 120 130 140 160 190 190 180 180 140 4 FIG. 5 FIG. Databasemay include one or more computing devices configured to store and/or provide image data to one or more of server system, registration module, authentication module, image-processing module, location determination system, memory, and mobile device. In some aspects, such image data can represent images obtained from mobile device, registration images, virtual images depicting virtual objects, augmented reality images, and stored information about the images, such as metadata. In some aspects, databasemay include image data needed for the registration process described below in connection with, and/or the authentication process described below in connection with. In some aspects, databasemay include location information obtained from location determination system. A “module” may be a device implemented in hardware, software, firmware, or any combination thereof.

180 105 Databasemay include one or more computing devices configured to store and/or provide account information relating to a user (e.g., a user account of server system). In some aspects, such account information may include user profile information, such as a user identifier, username, account number, login credentials, device identifier, address, passcode, or other such information. In some aspects, such account information may include personal information such as demographic (e.g., age), financial (e.g., income), marital status, and/or prior behavior information (e.g. purchase record).

180 180 Databasemay include, for example, one or more Oracle™ databases, Sybase™ databases, or other relational databases or non-relational databases, such as Hadoop™ sequence files, HBase™, or Cassandra™. Database(s)may include computing components (e.g., database management system, database server, etc.) configured to receive and process requests for data stored in memory devices of the database(s) and to provide data from the database(s).

180 180 105 110 120 130 160 190 While databaseis shown separately, in some embodiments databasemay be included in or otherwise related to one or more of server system, registration module, authentication module, image-processing module, memory, and mobile device.

180 190 180 105 110 120 130 180 180 3 FIG. Databasemay be configured to collect, store, and/or maintain image data from mobile device. Databasecan be configured to provide such data to the server system, registration module, authentication module, and image-processing module. Databasemay collect image data from a variety of other sources, including, for example, online resources. Databaseis further described below in connection with.

190 190 190 190 190 190 190 190 190 170 190 4 5 FIGS.and Mobile devicemay include one or more computing devices configured to perform operations consistent with disclosed embodiments. For example, mobile devicemay include at least one of a laptop, a tablet, a smart phone, a gaming device, a wearable computing device, or other type of computing device. Mobile devicemay include one or more processors configured to execute software instructions stored in memory, such as memory included in mobile device. Mobile devicemay include software that when executed by a processor performs Internet-related communication and content display processes. For instance, mobile devicemay execute browser software that generates and displays interfaces including content on a display device included in, or connected to, mobile device. Mobile devicemay execute applications that allow mobile deviceto communicate with components over network, and generate and display content in interfaces via a display device included in mobile device. The display device may be configured to display images described in.

190 190 105 180 190 190 190 The disclosed embodiments are not limited to any particular configuration of mobile device. For instance, mobile devicecan be a device that stores and executes mobile applications that interact with server systemand/or databaseto perform aspects of the disclosed embodiments. In certain embodiments, mobile devicemay be configured to execute software instructions relating to location services, such as GPS locations. For example, mobile devicemay be configured to determine a geographic location and provide location data and time stamp data corresponding to the location data. In yet other embodiments, mobile devicemay capture video and/or images.

110 110 190 180 110 120 130 140 190 160 180 190 110 190 Registration modulemay include one or more computing components configured to register a user for augmented reality authentication. Registration modulemay be configured to receive and process images from mobile deviceand/or database. Consistent with disclosed embodiments, these images may include registration images depicting a real scene, virtual images depicting virtual objects, and/or augmented reality images. In some embodiments, registration modulecan be configured to communicate with authentication module, image-processing module, location determination system, and mobile deviceto carry out the registration steps. Consistent with disclosed embodiments, an augmented reality registration image is stored in memory, such as memory, database, or mobile device. In some embodiments, registration modulemay be implemented on mobile device.

120 110 160 180 190 120 190 120 120 120 110 130 140 190 120 190 Authentication modulemay include one or more computing components configured to perform operations consistent with authenticating a user. In some embodiments, a user registers for augmented reality authentication through registration module. During registration, user registration information may be generated and stored in memory, database, or mobile device, for example. The user registration information may include the registration image and/or augmented reality registration image. To authenticate the user using authenticated reality authentication, a registered user can provide image data to the authentication modulevia mobile device. Consistent with disclosed embodiments, the image data may include data representing real scenes, virtual images, augmented reality images, and/or data related to images, such as meta-data. Authentication modulecan be configured to authenticate the user based on a comparison of the image data with previously-stored registration image data. In some embodiments, authentication modulecan grant access to the user account based on the authentication. In some embodiments, authentication modulecan be configured to communicate with registration module, image-processing module, location determination system, and mobile deviceto carry out the authentication steps. In some embodiments, authentication modulemay be implemented on mobile device.

130 130 110 120 140 180 190 Image-processing modulemay include one or more computing components configured to collect and process images. Image-processing modulemay be configured to communicate with registration module, authentication module, location determination system, database, and mobile deviceto enable registration and/or authentication using augmented reality.

130 110 120 140 150 160 180 190 130 130 Consistent with disclosed embodiments, image-processing modulemay be configured to perform an image verification procedure on image data. For example, the image verification procedure may receive data representing images, metadata, and other information from registration module, authentication module, location determination system, processor, memory, database, and mobile deviceto verify that image data was received from an image sensor of a mobile device. In some embodiments, image-processing moduleverifies the image data based on image metadata, such as file format and embedded location information. Image-processing modulemay also verify the image data based on detected usage of a mobile device, such as usage of an image sensor or movement of the mobile device.

130 130 110 120 140 160 180 190 130 Consistent with disclosed embodiments, image-processing modulemay be configured to perform image matching between information representing two or more images. For example, image-processing modulemay receive the image data from registration module, authentication module, location determination system, memory, database, or mobile device. Image-processing modulemay extract image features from received image data to calculate a comparison metric. In some aspects, the comparison metric is calculated using identification models. The identification models may include convolutional neural networks that determine attributes in an image based on features extracted from the image. In various aspects, identification models may include statistical algorithms to determine a similarity between images. For example, identification models may include regression models that estimate the relationships among input and output variables. In some aspects, identification models may additionally sort elements of a dataset using one or more classifiers to determine the probability of a specific outcome. Statistical identification models may be parametric, non-parametric, and/or semi-parametric models. A convolutional neural network model can be configured to process an image into a collection of features. The convolutional neural network can comprise an input layer, one or more middle layers, and one or more output layers. Image data can be applied to the input layer. In some embodiments, the input layer can comprise multiple matrices (e.g., a matrix for each of the red, green, and blue pixel values in an RGB image). In some embodiments, the input layer can comprise a single matrix (e.g., a single two-dimensional matrix of pixel brightness values). In some aspects, each middle layer and the output layer can be a deterministic function of the values of the preceding layer. The convolutional neural network can include one or more convolutional layers. Each convolutional layer can be configured to convolve one or more spatial filters with the convolutional layer input to generate a convolutional layer output tensor. In some embodiments, the convolutional neural network can also include pooling layers and fully connected layers according to methods known in the art. Identification models may also include Random Forests composed of a combination of decision tree predictors. Such decision trees may comprise a data structure mapping observations about something, in the “branch” of the tree, to conclusions about that thing's target value, in the “leaves” of the tree. Each tree may depend on the values of a random vector sampled independently and with the same distribution for all trees in the forest. Identification models may additionally or alternatively include classification and regression trees, or other types of models known to those skilled in the art.

130 130 130 110 120 140 150 160 180 190 Consistent with disclosed embodiments, image-processing modulemay be configured to calculate a confidence score representing a confidence level that two or more images match. For example, image-processing modulemay calculate the confidence score based on one or more factors, such as a calculated comparison metric between two or more images. In some aspects, image-processing modulemay communicate with registration module, authentication module, location determination system, processor, memory, database, or mobile deviceto collect and process image data, such as location information, time stamp information, date stamp information, lighting conditions, and/or image metadata to calculate the confidence score.

130 120 120 120 Consistent with disclosed embodiments, image-processing modulecan be configured to provide the calculated confidence score to authentication module. Authentication modulemay determine whether the confidence score meets a predetermined threshold before authenticating the user. In some aspects, authentication modulemay perform a verification challenge if the confidence score is within a predetermined range. For example, the verification challenge must be performed successfully to authenticate the user.

130 110 120 130 130 190 In some embodiments, image-processing modulemay be configured to receive image data from registration moduleor authentication moduleto generate data representing an augmented reality image. For example, image-processing modulemay receive data representing one or more virtual images to superimpose on image data depicting a real scene. In some embodiments, image-processing modulemay be implemented on mobile device.

1 FIG. 110 120 130 110 120 130 shows registration module, authentication module, and image-processing moduleas different components. However, registration module, authentication module, and image-processing modulemay be implemented in the same computing system.

140 190 140 190 190 140 170 140 140 Location determination systemmay include one or more computing components configured to perform operations consistent with determining a location of a mobile device. In some embodiments, location determination systemmay transmit a request to mobile devicefor location information. In response to the request, mobile devicemay transmit location information such as geographic coordinates or identifiers to the location determination systemvia network. The location determination systemmay map the geographic coordinates or identifiers to a specific geographic location. The location determination systemmay include satellite-based geolocation systems, e.g., GPS, Glonass, Galileo, etc.; long-range terrestrial geolocation systems, e.g., LORAN; short-range network based geolocation systems, e.g., network-based cellular e911; short-range proximity location sensing, e.g., 802.11 access point identification, line-of-sight location identification, e.g., IRdA or other visible, sonic or invisible electromagnetic waves which are confined by barriers; RFID based location detectors; and the like.

140 110 120 130 190 140 110 120 130 190 140 190 In some embodiments, location determination systemcan be configured to communicate with registration module, authentication module, image-processing module, and mobile device. For example, location determination modulemay receive image data from registration module, authentication module, image-processing module, and/or mobile device, and embed location information into the image data. In some embodiments, location determination systemmay be implemented on mobile device.

170 100 170 100 100 170 100 Networkmay be any type of network configured to provide communications between components of system. For example, networkmay be any type of network (including infrastructure) that provides communications, exchanges information, and/or facilitates the exchange of information, such as the Internet, a Local Area Network, near field communication (NFC), optical code scanner, or other suitable connection(s) that enables the sending and receiving of information between the components of system. In some embodiments, one or more components of systemcan communicate through network. In various embodiments, one or more components of systemmay communicate directly through one or more dedicated communication links.

100 It is to be understood that the configuration and boundaries of the functional building blocks of systemhave been defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments.

2 FIG. 190 190 202 204 206 210 190 190 190 is a block diagram of mobile device, consistent with disclosed embodiments. In one embodiment, mobile devicemay include processor, location sensor, input/output (I/O) system, and memory. In some embodiments, mobile devicemay take the form of a mobile computing device (e.g., a wearable device, smartphone, tablets, laptop, or similar device), a desktop (or workstation or similar device), or a server. Alternatively, mobile devicemay be configured as a particular apparatus, embedded system, dedicated circuit, and the like based on the storage, execution, and/or implementation of the software instructions that perform operations consistent with the disclosed embodiments. According to some embodiments, mobile devicemay be configured to provide a web browser or similar computing application capable of accessing web sites, consistent with disclosed embodiments.

202 Processormay include one or more known processing devices, such as mobile device microprocessors manufactured by Intel™, NVIDIA™, or various processors from other manufacturers. As would be appreciated by one of skill in the art, the disclosed embodiments are not limited to a particular processor type.

210 210 212 202 190 190 190 210 212 210 216 212 212 110 120 130 Memorymay include one or more storage devices configured to store instructions for performing operations related to disclosed embodiments. For example, memorymay be configured with one or more software components, such as program, that when executed by processor, can cause mobile deviceto perform operations consistent with the disclosed embodiments. The disclosed embodiments are not limited to separate programs or computers configured to perform dedicated tasks. For example, mobile devicecan be configured to perform the disclosed functions of mobile deviceby one or more programs stored in memory(e.g., program). In some embodiments, memorycan be configured to store dataused by one or more programs. In some embodiments, programmay include one or more of registration module, authentication module, or image processing module.

210 214 202 214 105 190 In certain embodiments, memorymay store an augmented reality applicationthat may be executed by processor(s)to perform one or more augmented reality processes consistent with disclosed embodiments. In certain aspects, augmented reality application, or another software component, may be configured to request identification from server systemor determine the location of mobile device.

204 190 204 190 190 204 100 170 204 204 Location sensormay include sensor(s) and/or system(s) capable of determining a location of mobile device, such as a Global Positioning System (GPS) receiver, Bluetooth transceiver, or WiFi transceiver. When location sensorincludes multiple sensor(s) and/or system(s), mobile devicecan be configured to determine geographic coordinates or identifiers, such as latitude or longitude, based on data provided from multiple sensor(s) and/or system(s). Mobile devicecan be configured to send the geographic coordinates or identifiers determined by location sensorto other components of systemvia, for example, network. Location sensormay also include motion sensors for detecting movement of the mobile device. For example, location sensormay include a proximity sensor, accelerometer, gyroscope, magnetometer, and/or ambient light sensor.

206 190 190 100 206 206 206 190 206 190 206 105 I/O systemmay include one or more devices configured to allow data to be received and/or transmitted by mobile deviceand to allow mobile deviceto communicate with other machines and devices, such as other components of system. For example, I/O systemmay include a screen for providing information to the user. VO systemmay also include components for NFC communication. I/O systemmay also include one or more digital and/or analog devices that allow a user to interact with mobile devicesuch as a touch-sensitive area, buttons, or microphones. I/O systemmay also include one or more accelerometers to detect the orientation and inertia of mobile device. I/O systemmay also include other components known in the art for interacting with server system.

190 220 220 100 170 In some embodiments, mobile devicemay include an image sensor(e.g., a digital camera). Image sensorcan be configured to generate data representing still images or video and send it to other components of systemvia, for example, network.

190 The components of mobile devicemay be implemented in hardware, software, or a combination of both hardware and software, as will be apparent to those skilled in the art.

3 FIG. 180 180 302 304 310 312 314 180 is a block diagram of an exemplary database, in accordance with disclosed embodiments. Databasemay include a communication device, one or more database processors, and database memoryincluding one or more database programsand data. In some embodiments, databasemay be hosted on one or more computing devices (e.g., desktops, workstations, servers, and computing clusters).

302 100 105 110 120 130 140 190 302 110 120 130 Communication devicemay be configured to communicate with one or more components of system, such as server system, registration module, authentication module, image-processing module, location determination system, and/or mobile device. In particular, communication devicemay be configured to provide to registration module, authentication module, image-processing moduleimage data that may be used to register and/or authenticate a user.

180 180 180 The components of databasemay be implemented in hardware, software, or a combination of both hardware and software, as will be apparent to those skilled in the art. For example, although one or more components of databasemay be implemented as computer processing instruction modules, all or a portion of the functionality of databasemay be implemented instead in dedicated electronics hardware.

314 190 314 314 Datamay be data associated with image data obtained from mobile device, registration images, virtual images depicting virtual objects, augmented reality images, and stored information about the images, such as metadata. Datamay include, for example, account information relating to a user such as a user identifier, username, account number, login credentials, device identifier, address, passcode, or other user profile information. Datamay include personal information such as demographic (e.g., age), financial (e.g., income), marital status, and/or prior behavior information (e.g. purchase record).

4 FIG. 400 400 100 400 190 400 105 400 105 190 is a flowchart of an exemplary methodfor registering a user for augmented reality authentication, consistent with embodiments of the present disclosure. Methodcan be performed using system. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device.

402 100 110 190 170 190 105 At step, systemreceives a request from a user to register for augmented reality authentication. In some embodiments, the request is received by registration modulefrom mobile deviceover network. For example, the user may interact with a user interface on the mobile deviceto transmit the request for registration. The user may have a registered account with server system. In some aspects, the user may register for one or more augmented reality authentications for one or more registered accounts.

404 100 190 110 190 220 190 140 130 160 180 190 6 FIG. 7 FIG. At step, systemreceives registration image data depicting a real scene from a mobile device. In some embodiments, registration moduleprovides instructions to a user interface on the mobile device. For example, the user interface may instruct the user to capture an image using an image sensorof mobile device. The image depicts a real scene such as a bedroom or office or interior of an automobile. In the example embodiments, a real scene can include any scene for which significant details remain substantially constant, such that the particular scene remains recognizable in subsequent images. In some embodiments, a registration image corresponds to a scene that is familiar to and frequented by the user. In some aspects, data representing the captured image is provided to location determination system, as discussed below with respect to. In some aspects, the captured image data is provided to image-processing moduleto perform an image verification procedure, as discussed below with respect to. The registration image data may be stored in memory, database, and/or mobile device.

406 100 110 190 180 160 190 220 216 190 404 At step, systemreceives a selection of one or more virtual images (or virtual objects) to overlay on the registration image. In some embodiments, registration moduleprovides instructions to a user interface on the mobile device. For example, the user interface may instruct the user to select one or more virtual images from a list of virtual images. In some aspects, the virtual image may depict a virtual object such as a telephone, teddy bear, geometric shapes, random objects, things or any other virtual object. The virtual images may be acquired from database, memory, and/or mobile device. For example, data representing the virtual images may be acquired from an image sensoror dataof mobile device. The user interface may then display the registration image from step, and instruct the user to select a location on the registration image on which to overlay the selected one or more virtual images. For example, the user may select a first location on the registration image to overlay a first virtual image, and select a second location on the registration image to overlay a second virtual image.

408 100 406 130 130 140 6 FIG. At step, the systemgenerates an augmented reality registration image. In some aspects, the augmented reality registration image comprises the selected one or more virtual images overlaid on the registration image at the locations selected at step. In some embodiments, the image-processing modulereceives data representing the registration image, the selected one or more virtual images, and the selected one or more locations on the registration image on which to overlay the selected one or more virtual images, and generates the augmented reality registration image. For example, the image-processing moduleoverlays the virtual image on the registration image at the selected location, and generates the augmented reality registration image. In some embodiments, data representing the augmented reality registration image is sent to location determination system, as discussed in further detail below with respect to. The augmented reality registration image may include image data, such as geolocation information, time stamp information, date stamp information, lighting conditions, unique device identifier (either mobile, laptop, desktop, or any IoT device) from which the image was captured and/or image metadata.

410 100 160 180 190 At step, systemstores the augmented reality registration image. For example, the augmented reality registration image may be stored in memory, database, and/or mobile device. In some aspects, the augmented reality registration image is stored in an account associated with the user.

400 In some embodiments, the user may repeat the steps in methodto register for multiple augmented reality authentications. For example, the system may generate and store multiple augmented reality registration images for one or more accounts associated with the user. This enables the system to generate augmented reality registration images for the user based on different registration images depicting real scenes. For example, the user may select their bedroom at their place of residence for a first registration image to generate a first augmented reality registration image. The user may further select their office at their place of business for a second registration image to generate a second augmented reality registration image, for added flexibility.

5 FIG. 500 100 500 190 500 105 500 105 190 500 502 105 190 105 170 402 105 is a flowchart of an exemplary method for authenticating a user using augmented reality, consistent with embodiments of the present disclosure. Methodcan be performed using system. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device. According to method, at step, server systemcan be configured to receive a request for authentication using augmented reality authentication. For example, the request may be received from (or at) a mobile deviceassociated with a user and relayed to server systemvia network. In some embodiments, the request is received from the same or a different mobile device as discussed at stepduring the registration for augmented reality authentication. In some embodiments, the user authentication request is for access to a user account (e.g., a user account of server system).

504 100 190 220 210 212 214 190 105 170 120 130 402 120 190 220 190 404 140 130 160 180 190 6 FIG. 7 FIG. At step, systemis configured to receive a first image depicting a real scene. For example, the first image may be received at mobile deviceassociated with the user from sensorand stored in memoryfor processing by programor application. In some embodiments, the first image is transmitted from mobile deviceto server systemvia networkfor processing by authentication moduleor image processing module. In some embodiments, the first image is received at or from the same or a different mobile device as discussed at stepduring the registration for augmented reality authentication. Authentication modulemay provide instructions to a user interface on the mobile device. For example, the user interface may instruct the user to capture an image using an image sensorof mobile device. The user may be instructed to capture substantially the same image as that taken in stepduring registration. In some aspects, the captured image is provided to location determination system, as discussed below with respect to. In some aspects, the captured image is provided to image-processing moduleto perform an image verification procedure, as discussed below with respect to. The registration image may be stored in memory, database, and/or mobile device.

120 190 502 502 210 190 120 190 In some embodiments, authentication modulemay transmit instructions to the user interface of mobile deviceto aid the user with the authentication process. For example, the instructions may provide the user interface with a visual aid. In some embodiments, the user is prompted to enter a user identifier associated with the user account after step. For example, the user may be prompted to enter a username, token, passcode, mobile device identifier, or other identifier. In some embodiments, the request at stepincludes the user identifier, mobile device identifier, or a determined location of the mobile device. For example, the user identifier, mobile device identifier, or determined location of the mobile device may be stored in memoryand/or transmitted automatically from mobile device. Authentication module, after receiving the identifier, may then retrieve registration image or augmented reality registration image stored in an account associated with the identifier. For example, the retrieved registration image or augmented reality registration image may be displayed on the user interface of mobile device.

120 120 190 120 In some embodiments, authentication modulemay compare the determined location of the mobile device with location information embedded in the registration image or augmented reality registration image stored in an account associated with the identifier. For example, authentication modulemay retrieve the registration image or augmented reality registration image for display on mobile devicewhen the determined location is within a threshold distance of a location determined from the location information embedded in the registration image or augmented reality registration image. In another example, authentication modulemay retrieve the registration image or augmented reality registration image when the determined location is within a geofence area associated with the location information embedded in the registration image or augmented reality registration image.

220 190 In some embodiments, the displayed registration image or augmented reality registration image is a translucent version of the original image. For example, if the location of the device matches the geolocation of the original image, the user may use a translucent version of the original image as a guide for capturing substantially the same image of the real scene as depicted in the registration image or the augmented reality registration image. For example, user may be prompted to capture the same image of the real scene using sensorof mobile device.

100 190 190 120 190 120 130 In some embodiments, systemis configured to preprocess the received first image before displaying the retrieved registration image or augmented reality registration image on the user interface of mobile device. For example, the first image may be preprocessed to identify some features within the image for comparison with similar features identified in the registration image or augmented reality registration image. For example, the first image may be preprocessed to identify significant features or landmarks within the image, such as a desk, a window, or a bed. If a comparison with the retrieved registration image or augmented reality registration image meets a threshold similarity value, i.e., a threshold match, with respect to the identified significant features or landmarks, the retrieved registration image or augmented reality registration image is displayed on the user interface of mobile device. In some embodiments, authentication modulemay retrieve the registration image or augmented reality registration image for display on mobile devicewhen there is a threshold match with respect to the identified significant features or landmarks, as described above, and when a location of the mobile device is within a threshold distance of a location determined from the location information embedded in the registration image or augmented reality registration image. In some embodiments, authentication moduleor image processing modulemay be configured to preprocess the first image.

100 220 190 500 190 In some embodiments, systemis configured to preprocess an image depicting a real scene obtained from sensorin real-time, before capturing the first image. For example, preprocessing of the image may be conducted in real-time on the mobile device, to identify significant features or landmarks within the image as described above. In some embodiments, preprocessing of the image must yield a threshold match with respect to the identified significant features or landmarks within a predetermined amount of time. For example, if the predetermined amount of time expires before yielding a match, methodmay end without authenticating the user. In some embodiments, the registration image or augmented reality registration image is displayed on the mobile devicefor a predetermined amount of time to assist the user.

506 100 190 210 212 214 190 105 170 120 130 506 406 120 190 180 160 190 220 216 190 504 At step, systemis configured to receive a selection of one or more virtual images to overlay onto the first image. In some embodiments, mobile devicereceives the selections of virtual images and store the selections in memoryfor processing by programor application. In some embodiments, mobile devicetransmits the selections of virtual images to server systemvia networkfor processing by authentication moduleor image processing. The selection of virtual images in stepmay proceed similarly to the selection of virtual images in step. In some embodiments, authentication moduleprovides instructions to a user interface on the mobile device. For example, the user interface may instruct the user to select one or more virtual images from a list of virtual images. In some aspects, the virtual image may depict a virtual object such as a telephone, teddy bear, geometric shapes, random objects, things or any other virtual object. The virtual images may be acquired from database, memory, and/or mobile device. For example, the virtual images may be acquired from an image sensoror dataof mobile device. In some embodiments, the list of virtual images includes a set of new virtual objects that have not been previously selected by the user. The user interface may then display the first image from step, and instruct the user to select a location on the first image on which to overlay the selected one or more virtual images. For example, the user may select a first location on the first image to overlay a first virtual object, and select a second location on the first image to overlay a second virtual object.

508 100 212 214 190 210 190 105 170 120 130 508 408 506 At step, systemgenerates an augmented reality image. In some embodiments, programor applicationof mobile devicegenerates the augmented reality image and stores the image in memory. In some embodiments, mobile devicetransmits the first image and the selections of virtual images to server systemvia network, as described above, to generate the augmented reality image. For example, authentication moduleor image processingmay generate the augmented reality image. In some embodiments, stepmay proceed similarly to the generation of augmented reality image in step. In some aspects, the augmented reality image comprises the selected one or more virtual images overlaid on the first image at the locations selected at step.

130 130 140 6 FIG. In some embodiments, image-processing modulereceives the first image, the selected one or more virtual images, and the selected one or more locations on the first image on which to overlay the selected one or more virtual images, and generates the augmented reality image. For example, image-processing moduleoverlays the virtual image on the first image at the selected location, and generates the augmented reality image. In some embodiments, the augmented reality image, and/or the first image, is sent to location determination system, as discussed in further detail below with respect to. The augmented reality image may include image data, such as location information, time stamp information, date stamp information, lighting conditions, and/or image metadata.

510 105 130 504 506 508 400 190 8 FIG. 8 FIG. At step, server systemcalculates a comparison metric, as described in further detail below with respect to. For example, image-processing modulemay receive the first image from step, the selection of one or more virtual images of step, and/or the augmented reality image generated in step, and calculate a comparison metric reflecting a percentage match between two or more of the images, as discussed in further detail with respect to. In some embodiments, the comparison metric reflects a percentage match between the augmented reality image and the one or more stored augmented reality images. In some embodiments, the stored augmented reality images include one or more augmented reality registration images generated during method. In some embodiments, comparison metric is calculated at mobile device.

512 105 130 190 10 FIG. 10 FIG. At step, server systemcalculates a confidence score, as described in further detail below with respect to. In some embodiments, image-processing modulecalculates a confidence score based on the comparison metric and/or additional parameters, as discussed in further detail with respect to. For example, the confidence score may be calculated based on location, time stamp, and/or date stamp information relating to the images, lighting conditions, image metadata, and/or device information. In some embodiments, the confidence score is calculated at mobile device.

514 105 512 120 9 FIG. At step, server systemauthenticates the user based on the confidence score calculated at step. For example, authentication modulemay authenticate the user based on a comparison of the confidence score and a number of predetermined thresholds, as discussed in further detail with respect tobelow.

500 506 510 105 504 506 506 406 512 190 10 FIG. In some embodiments, methodmay proceed directly from stepto stepto calculate the comparison metric. For example, server systemmay receive the first image at stepand the virtual images at step, and calculate the comparison metric. The comparison metric may reflect a percentage match between the first image and the registration image, a percentage match between the virtual images received at stepand the virtual images received during registration at step, or a percentage match between a combination of the images. Here, the method continues to stepto calculate the confidence score based on the comparison metric. For example, the confidence score may be calculated based on the comparison metric and location information associated with the images and mobile device, as discussed in further detail below with respect to.

516 105 105 105 190 At step, server systemauthorizes access based on the authentication. For example, server systemmay authorize access to an account associated with the user (e.g., a user account of server system) based on the authentication. In some embodiments, the example authentication techniques may be configured to enable access to data on mobile device.

6 FIG. 600 140 105 190 600 190 600 105 600 105 190 is a flowchart of an exemplary method for embedding location information in images, consistent with embodiments of the present disclosure. In some embodiments, methodis implemented by location determination systemon server systemor on mobile device. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device.

602 190 204 At step, the location of mobile deviceis determined (e.g., using location sensors).

604 190 220 110 120 130 160 180 220 408 508 At step, an image is received from mobile device. For example, the image may be received from image sensor. In some embodiments, the image is received from registration module, authentication module, image-processing module, memory, or database. The image may depict a real scene captured by image sensor, a virtual image, or an augmented reality image generated at stepsand, as disclosed above.

606 602 600 602 604 140 110 120 130 160 180 190 At step, location information determined from stepis embedded into the image. For example, geolocation data may be embedded in the image as metadata. In some embodiments, methodincludes only stepsand. For example, location determination systemmay embed the location information in an image stored registration module, authentication module, image-processing module, memory, database, or mobile device.

7 FIG. 700 100 700 130 190 700 190 700 105 700 105 190 is a flowchart of an exemplary method for verifying images, consistent with embodiments of the present disclosure. Methodcan be performed using system. For example, methodmay be implemented by image-processing moduleto verify that an image is authentic and originated from the image sensor of mobile device. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device.

702 190 190 220 404 504 190 700 190 220 At step, an image is received from (or accessed by) mobile device. For example, the image may be captured at mobile deviceby image sensor. In some embodiments, the image may depict a real scene, such as the image received at stepsand. In some embodiments, the image is an augmented reality registration image comprising one or more virtual images overlaid onto an image received from the mobile device. For example, the image verification methodcan verify that the augmented reality registration image includes an image captured at the mobile deviceby image sensor.

704 130 220 190 706 714 706 130 706 220 708 130 220 214 202 220 190 220 710 130 190 204 220 712 130 190 140 190 130 714 130 6 FIG. The image verification procedure is then performed at step. Image-processing modulemay verify that the image was received from an image sensorof mobile deviceby performing one or more of steps-. For example, at step, image-processing modulemay detect a file format of the image at step. For example, sensormay capture an image and generate image data using a specific file format. In some aspects, the file format may include a “live” image file, comprising a video file and an image file. As another example, at step, image-processing modulemay detect usage of image sensor. For example, augmented reality applicationor processormay transmit a confirmation that image sensorwas used when transmitting the image from mobile devicefor verification. As another example, metadata embedded in the image, such as a time stamp at the time of image creation, may be compared to a time the image sensorwas used to capture the image. As another example, at step, image-processing modulemay verify the image by detecting movement of the mobile device(e.g. using location sensor) at the time the image was captured by sensor. As yet another example, at step, image-processing modulemay determine a location of mobile deviceand compare the determined location to geolocation metadata embedded in the image at the time of capture, as disclosed above with respect to. For example, location determination systemmay determine the location of the mobile deviceand transmit the determined location to image-processing moduleto perform the image verification procedure. As another example, at step, image-processing modulemay verify the image by detecting a unique device identifier for the user device and comparing the detected unique device identifier to device identifier information embedded in the image at the time of capture.

716 130 220 190 706 714 400 410 500 At step, image-processing modulemay verify that the image was received from image sensorof mobile deviceusing one or more of steps-. In some embodiments, methodrequires image verification before storing the augmented reality image in step. In some embodiments, methodrequires image verification before authenticating a user.

8 FIG. 800 100 800 190 800 105 800 105 190 130 130 is a flowchart of an exemplary method for calculating comparison metrics between images, consistent with embodiments of the present disclosure. Methodcan be performed using system. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device. In some embodiments, the comparison metric is calculated by image-processing moduleusing machine learning techniques. In some embodiments, the comparison metric is calculated by image-processing moduleusing a convolutional neural network.

802 130 804 160 180 190 804 130 806 802 130 804 160 180 190 804 130 806 100 100 105 190 105 a a a b b b After starting in step, image-processing modulereceives a first image in step. For example, first image may be received from memory, such as memory, database, or mobile device. After step, image-processing modulecan extract machine-learning features from the image using a convolutional neural network in step. Similarly, after starting in step, image-processing modulereceives a second image in step. For example, second image may be received from memory, such as memory, database, or mobile device. After step, image-processing modulecan extract machine-learning features from the second image using a convolutional neural network in step. In some aspects, systemcan be configured to extract machine-learning features from the first and second images using different components of system. For example, server systemcan extract the machine-learning features of the second image, and mobile devicecan extract the machine-learning features of the first image. The extracted features may include unique features relating to the images, such as a landmark or object depicted within the image. In some embodiments, server systemcan determine the coordinates of the extracted features within the image.

400 500 130 808 130 808 130 808 130 808 5 FIG. 5 FIG. 5 FIG. 5 FIG. In some embodiments, first and second images may include registration images depicting a real scene, virtual images depicting virtual objects, and/or augmented reality images generated during methodsand/or. For example, image-processing modulemay extract machine learning features from the registration image and the augmented reality image (for example, the augmented reality image generated as described in) to calculate comparison metric. As another example, image-processing modulemay extract machine learning features from the registration image and the base first image (for example, the first image as described in) to calculate comparison metric. As another example, image-processing modulemay extract machine learning features from the augmented reality registration image and the augmented reality image (for example, the augmented reality image generated as described in) to calculate comparison metric. As another example, image-processing modulemay extract machine learning features from augmented reality registration image and the augmented reality image (for example, the augmented reality image generated as described in) to calculate comparison metric.

806 806 130 808 130 806 806 a b a b 5 FIG. After stepsand, image-processing modulemay calculate a comparison metric in step. In some embodiments, the calculated comparison metric may reflect a percentage match between the first and second images. This calculation can involve inputting the machine-learning features extracted from the first and second image data into a classifier, such as another neural network. In additional embodiments, image-processing modulecan be configured to calculate a metric based on the extracted machine-learning features (e.g., an inner product of the machine learning features extracted from the first image data and the machine learning features extracted from the second image). A value of the metric can indicate the degree of similarity between the images. In some instances values of the metric can increase with increasing similarity. In some embodiments, the calculated comparison metric may reflect a degree of similarity between the virtual object's position within the augmented reality registration image and the virtual object's position within the augmented reality image (for example, the augmented reality image generated as described in). For example, the coordinates of the extracted features within the image determined at steps,may be used to calculate the comparison metric.

130 130 10 FIG. In some embodiments, image-processing modulemay calculate first comparison metric associated with a comparison of the augmented reality image to one or more stored augmented reality registration images. The image-processing modulemay calculate a second comparison metric associated with a comparison of the first image with one or more stored registration images. In some embodiments, the augmented reality image is matched to the augmented reality registration image based on the first and second comparison metrics, as further discussed below with respect to.

9 FIG. 900 100 900 190 900 105 900 105 190 is a flowchart of an exemplary method for using a confidence score to authenticate users, consistent with embodiments of the present disclosure. Methodcan be performed using system. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device.

902 105 404 504 406 506 408 508 130 10 FIG. At step, server systemcalculates a confidence score representing a level of confidence that a first image matches a second image. In some embodiments, the first and second images include the images depicting a real scene received at stepsand, the virtual images received at stepsand, and/or the augmented reality images disclosed in stepsand. In some embodiments, the confidence score is calculated by image-processing module, as disclosed in further detail below with respect to.

904 105 904 914 120 120 906 120 908 120 914 120 910 190 120 120 906 At step, server systemcompares the calculated confidence score to a first predetermined threshold. In some embodiments, steps-are performed by authentication module. For example, authentication modulemay determine that the confidence score is greater than the first threshold and proceed to authenticate the user at step. If the authentication moduledetermines that the confidence score is less than the first threshold, it may compare the confidence score to a second predetermined threshold at step. If the confidence score is determined to be less than the second threshold, authentication moduleproceeds to stepand does not authenticate the user. In some embodiments, if the confidence score is determined to be less than the second threshold, authentication moduleproceeds to stepand performs a verification challenge. For example, the verification challenge may include an over-the-phone challenge. In some embodiments, the over-the-phone challenge includes transmitting a token or passcode to the mobile deviceassociated with the user. The user is then prompted through a user interface to enter the transmitted token or passcode. Authentication modulethen compares the transmitted and received tokens or passcode to determine if there is a match. The authenticated modulethen proceeds to stepto authenticate the user when there is match. In some embodiments, the token or passcode is a one-time token or passcode that expires after a predetermined amount of time.

10 FIG. 1000 130 105 190 1000 190 1000 105 1000 105 190 is a flowchart of an exemplary method for calculating a confidence score, consistent with embodiments of the present disclosure. In some embodiments, methodis implemented by image-processing moduleon server systemor on mobile device. In some embodiments, one or more steps of methodcan be performed by mobile device. In various embodiments, one or more steps of methodcan be performed by server system. In certain embodiments, methodcan be performed entirely by server system, or entirely by mobile device.

1000 1002 404 504 406 506 408 508 130 1018 1004 1006 1008 1010 1012 1014 1016 Methodbegins at stepto calculate a confidence score representing a level of confidence that a first image matches a second image. In some embodiments, the first and second images include the images depicting a real scene received at stepsand, the virtual images received at stepsand, and/or the augmented reality images disclosed in stepsand. In some embodiments, image-processing modulemay calculate the confidence score at stepby comparing one or more factors associated with the first and second images. For example, the one or more factors may include comparison metric, location information, time stamp information, date stamp information, lighting conditions, image metadata, and device information. In some embodiments, the factors are weighted to calculate the confidence score.

1004 130 1004 1004 1004 8 FIG. 8 FIG. In some embodiments, comparison metricmay be obtained from image-processing moduleas disclosed above with respect to. For example, the comparison metricmay include a percentage match between the first and second images. In some embodiments, comparison metricmay comprise one or more comparison metrics, as discussed above with respect to. For example, comparison metricmay comprise a first comparison metric associated with a comparison of the augmented reality image to one or more stored augmented reality registration images, and may also comprise a second comparison metric associated with a comparison of the first image with one or more stored registration images.

1006 140 130 130 1016 1016 130 1008 1010 130 1012 1014 1018 130 1004 1016 1020 140 1006 6 FIG. 7 FIG. In disclosed aspects, location informationincludes location information obtained from location determination systemand embedded in the first and second images, as disclosed inabove. For example, image-processing modulemay compare geolocation metadata embedded in the first and second images. In some embodiments, image-processing modulemay compare device informationassociated with a device from which the first and second images were captured to ensure that the devices used to perform the augmented reality authentication are the same devices. For example, such device informationmay include a unique device identifier, a make, and/or a model of the device. Image-processing modulemay compare time stamp informationor date stamp informationembedded in the first and second images to verify the images are authentic. Image-processing modulemay compare lighting conditionsor other image metadata, such as date of creation, author of image, and other technical information describing the image. At step, image-processing modulecalculates the confidence score by weighing one or more of the factors-and ends at step. In some embodiments, the image verification procedure disclosed inmay be used as a factor for calculating the confidence score. In some embodiments, the location of the device (e.g., determined by location determination system) may be compared to location informationembedded in the first or second images and used as a factor for calculating the confidence score.

11 FIG. 4 FIG. 5 FIG. 190 190 220 1102 220 190 1104 1102 1102 105 190 is an illustration of a mobile device capturing an image of a real-world environment, consistent with embodiments of the present disclosure. As disclosed above, the mobile devicemay be a mobile device, a tablet, a laptop, a cellular device, a personal computer, a smartphone (e.g., Apple iPhone, Blackberry, Android-based phones, etc.), and/or a wearable smart device such as a smart watch. The mobile devicemay include an image sensorsuch as a camera. A user of the mobile device may be prompted through a graphical user interface to capture an image of a real sceneusing image sensorof mobile device. The captured image may be displayed on a user interfaceof the mobile device display. For example, the user may capture a real sceneof a real-world environment such as the user's office. The captured image of the real scenemay be stored or processed by server systemor mobile deviceas the registration image ofor first image of.

12 13 FIGS.- 190 220 190 190 220 are exemplary illustrations of a user interface on a mobile device, consistent with embodiments of the present disclosure. In some embodiments, the exemplary graphical user interfaces may be displayed on mobile deviceand may take the image from image sensorand modify it to include elements displayed in a user interface on mobile device. In some embodiments, the exemplary graphical user interfaces may be displayed when a user opens an application running on mobile device, activating the image sensor.

1202 1102 1202 1102 1202 1102 1302 1304 1102 1304 1304 1304 1102 105 190 190 105 Graphical user interfaceshows an interface with a grid outline to assist the user in capturing real scene. The graphical user interfacemay prompt the user to capture an image, such as real scene. The interfacemay further prompt the user to modify the captured image, for example, by prompting the user to select one or more virtual images to overlay on the captured real sceneto generate an augmented reality image. For example, the user may be prompted through user interfaceto select a virtual image, such as virtual objectdepicting a telephone, to overlay on the captured real scene. The user may select a location on the grid of user interfaceon which to overlay the selected virtual object. In some embodiments, the user selects the location on the grid to overlay the virtual image before selecting the virtual image from a list of available virtual images. For example, the user selects a location on the grid, prompting the user interfaceto provide a dialogue box prompting the user to select a virtual image from a list of available virtual images. Once the user selects the desired virtual image, it is overlayed on the captured real sceneat the selected location on the grid. In some embodiments, the user selects the desired virtual image first, and then selects the location on the grid on which to overlay the virtual image. In some embodiments, the user selects more than one virtual image to overlay on the captured image. In some embodiments, the steps of capturing the real scene, selecting the virtual image, and overlaying the virtual image onto the real scene to generate the augmented reality image are performed entirely by server systemor by mobile device. In some embodiments, the images are stored temporarily on mobile devicefor processing, and then transmitted to server systemfor storage and/or further processing.

The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to precise forms or embodiments disclosed. Modifications and adaptations of the embodiments will be apparent from consideration of the specification and practice of the disclosed embodiments. For example, the described implementations include hardware and software, but systems and methods consistent with the present disclosure can be implemented with hardware alone. In addition, while certain components have been described as being coupled to one another, such components may be integrated with one another or distributed in any suitable fashion.

Moreover, while illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and/or alterations based on the present disclosure. The elements in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application, which examples are to be construed as nonexclusive. Further, the steps of the disclosed methods can be modified in any manner, including reordering steps and/or inserting or deleting steps.

Instructions or operational steps stored by a computer-readable medium may be in the form of computer programs, program modules, or codes. As described herein, computer programs, program modules, and code based on the written description of this specification, such as those used by the controller, are readily within the purview of a software developer. The computer programs, program modules, or code can be created using a variety of programming techniques. For example, they can be designed in or by means of Java, C, C++, assembly language, or any such programming languages. One or more of such programs, modules, or code can be integrated into a device system or existing communications software. The programs, modules, or code can also be implemented or replicated as firmware or circuit logic.

The features and advantages of the disclosure are apparent from the detailed specification, and thus, it is intended that the appended claims cover all systems and methods falling within the true spirit and scope of the disclosure. As used herein, the indefinite articles “a” and “an” mean “one or more.” Similarly, the use of a plural term does not necessarily denote a plurality unless it is unambiguous in the given context. Words such as “and” or “or” mean “and/or” unless specifically directed otherwise. Further, since numerous modifications and variations will readily occur from studying the present disclosure, it is not desired to limit the disclosure to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope of the disclosure.

Other embodiments will be apparent from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered as example only, with a true scope and spirit of the disclosed embodiments being indicated by the following claims.