System and Method for Multimodal Healthcare Data Compression, Storage, and Harmonization

Cloud computing systems process immense amounts of data each day, with estimates suggesting that the amount can reach 1 billion gigabytes. For example, healthcare systems handle enormous amounts of data daily, from different sources, in multiple formats and modalities which amount to about 30% of all the world's data. Managing the storage and access of multimodal data (i.e., data from multiple modalities and source systems) is challenging in many data storage solutions because of the unique properties of each data modality. For instance, when breaking down data silos in the healthcare space, organizations can make better decisions and advance their research to pave the way for new medical discoveries and precision medicine. Additionally, having access to harmonized image and clinical data about a patient helps organizations improve their treatments with more customized solutions that account for multimodal patterns. Further, unified access to multimodal data allows organizations to deploy artificial intelligence (AI) models to predict prognosis and diagnoses of disease.

In the context of the healthcare space, the challenge in creating a harmonized data estate stems from the complexity of healthcare data. The volume of data required for robust AI analytics is immense, computationally heavy, and managing this data requires sophisticated data governance, security, and privacy.

Like reference symbols in the various drawings indicate like elements.

Implementations of the present disclosure enable a cloud-scale, multimodal data platform that manages the rapid growth of heterogeneous medical data (e.g., clinical and image data) and is flexible enough to incorporate new data types and standards as they become available. For example, the multimodal data management process compresses multimodal data (e.g., first modality data and second modality data) in their respective formats in a first datastore of a data lake. Certain data modalities, like image data or video data, can require significant amounts of storage space. Aggregating different data modalities in a single datastore using conventional approaches requires siloed datastores for different data modalities and combining subsequently. These approaches are vulnerable to multimodal data integrity and compatibility issues as separate data modalities may be subject to different types of compression or truncation that result in poor connections between data of one modality and data of another modality (e.g., a patient's electronic health record (EHR) and medical imaging systems). Accordingly, the present disclosure provides a process for systematic, multi-layered processing of multimodal data that compresses each data modality while enhancing the structure and combinability of the multimodal data.

Each modality data is compressed into delta tables of a first predefined schema. The compressed modality data is transformed into normalized data in the form of delta tables of a second predefined schema. This normalized modality data is stored in a second datastore of the data lake. The normalized modality data is converted to enhanced modality data by converting the delta tables of the second predefined schema to delta tables of a third predefined schema. The enhanced modality data is stored in a third datastore of the data lake. The normalized modality data in the second predefined schema and the enhanced modality data in the third predefined schema allows multimodal data to be associated with data of other modalities and processed using a multimodal data operation. For example, a multimodal query can be performed on the normalized modality data and/or the aggregated enhanced modality data. In another example, a machine learning model can process multimodal data from the combination of the normalized modality data and/or the enhanced modality data.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the description, the drawings, and the claims.

1 2 FIGS.- 10 100 102 104 106 108 Referring to, multimodal data management processcompressesfirst modality data in a first processing datastore by: compressing the first modality data into a first delta table of a first schema and storing the first delta table of the first schema in the first processing datastore. The first modality data is transformedinto normalized first modality data by: transforming the first delta table of the first schema into a first delta table of a second schema and storing the second delta table of the second schema in a second processing datastore. The second modality data is compressedin the first processing datastore by: extracting a metadata tag from the second modality data into a second delta table of the first schema and storing the second delta table of the first schema in the first processing datastore. The second modality data is transformedinto normalized second modality data by: transforming the second delta table of the first schema into a second delta table of the second schema and storing the second delta table of the second schema in the second processing datastore. The normalized first modality data is associatedwith the normalized second modality data in the second processing datastore.

10 100 10 In some implementations, multimodal data management processcompressesfirst modality data in a first processing datastore. Modality data is data of a particular form or type. Examples of modality data include text data, image data, audio data, video data. In some implementations, modality data is defined as data having a distinct type for formatting. In some implementations, modality data includes a set of data used in a particular field (e.g., clinical data, sensor data, financial data, legal data, etc.). In one example, first modality data (i.e., one of a set of multiple modalities of data) is clinical data. Clinical data includes medical information (e.g., patient demographics, medical history information, laboratory test results, clinical trial data, medication information, etc.) associated with an individual or group of individuals. As will be discussed in greater detail below, upon ingestion into a data lake, first modality data may include various formats of clinical data (e.g., different types of text files or formats for electronic health records (EHRs)). Accordingly, multimodal data management processprocesses first modality data for storage in a multi-layered processing datastore.

100 110 10 100 In some implementations, compressingthe first modality data includes compressingthe first modality data at a first compression level. For example, compressing the first modality data includes encoding the first modality data from its initial state and size into a compressed data type or format associated with the first modality. In one example, first modality data is clinical data and includes an electronic health record associated with a particular person. In this example, multimodal data management processcompressesthe first modality data using a data compression algorithm. Examples of data compression algorithms include Huffman coding, Run-Length Encoding (RLE), Discrete Cosine Transform (DCT).

10 100 10 In some implementations, multimodal data management processcompressesthe first modality data by compressing the first modality data into a first delta table of a first schema. In one example, multimodal data management processcompresses the first modality data into a delta table. A delta table is a data structure used in data lakes and is designed to support Atomicity, Consistency, Isolation, and Durability (ACID) transactions, ensuring data integrity and reliability during concurrent operations. Additionally, delta tables enforce schema constraints (e.g., types of data for each entry of the schema). In one example, the delta table is a Parquet file. A Parquet file is a columnar storage file format optimized for use with data processing frameworks and is designed to efficiently store and process large datasets, providing both performance and storage benefits.

10 In some implementations, a schema is an organization of data within tables of the first schema, including the data types, constraints, and relationships between different data elements. For example, the schema defines the columns in each table and their respective data types (e.g., integer, string, date). This ensures that data is stored in each delta table in a consistent format. In one example, the first schema is a staging table schema. For example, staging tables in the first datastore include a predefined number of columns and are designed to hold each data modality and format in a single table. In another example, the first schema is the Parquet table schema that defines the structure (i.e., names, types and value types for fields) within Parquet files. As will be discussed in greater detail below, the staged processing of multimodal data management processallows multimodal data to be compressed and normalized in steps across different schemas. A common schema shared by both data modalities enable multimodal data operations to be performed on the combination of first modality data and second modality data.

10 110 In some implementations, the compression of first modality data is defined by a data compression level. The compression level is defined by the type or modality of the data being compressed. In one example with clinical data as the first modality data, multimodal data management processcompressesthe first modality data at a compression level of up to 75%. However, it will be appreciated that other compression levels may be used and are dependent upon the compression algorithm and the modality of the data. In some implementations, the compression level is configurable by a user (e.g., using a user interface and/or by selecting a particular compression level or compression algorithm).

10 200 10 202 204 200 206 202 208 2 FIG. In some implementations, multimodal data management processcompresses the first modality data into a first delta table of a first schema and stores the first delta table of the first schema in a first processing datastore. Referring also toand in some implementations, a data lake is a centralized repository designed to store, process, and secure large volumes of structured, semi-structured, and unstructured data. Data lakeallows data to be stored in various forms until it is processed for analysis, providing flexibility and scalability. In some implementations, multimodal data management processprocesses first modality data (e.g., first modality data) for storage in a first processing datastore (e.g., first processing datastore) within data lakeusing a first compression process (e.g., first compression process) to generate compressed first modality data′ and a delta table of a first predefined schema (e.g., delta table).

204 200 204 206 In one example, first processing datastoreis a first lakehouse of data lake. In some implementations, a lakehouse is a subset of a data lake. In some implementations, first processing datastorestores first modality data in its raw form (following compression by first compression process). In this example and as will be discussed in greater detail below, each processing datastore stores modality data in different stages of processing (e.g., raw, normalized, and enhanced).

10 102 202 208 10 102 202 204 210 210 202 204 10 102 202 In some implementations, multimodal data management processtransformsthe first modality data into normalized first modality data by transforming the first delta table of the first schema into a first delta table of a second schema and storing the first delta table of the second schema in a second processing datastore. For example, first modality datastored in first processing datastore includes a compressed version of the initial first modality data and a delta table in the first schema (e.g., delta table). Multimodal data management processtransformsfirst modality datafrom first processing datastoreto normalized first modality data (e.g., normalized first modality data). In some implementations, normalized first modality datais a representation of first modality datathat adheres to a second predefined schema. For example, first modality data stored in first processing datastoremay be defined in a first schema that lacks the mapping to other data modalities. Accordingly, multimodal data management processtransformsfirst modality datainto normalized first modality data by transforming first modality data into delta table(s) of a second, predefined schema.

202 10 In one example, the second schema concerns clinical data with a key-value pair, where each key describes what a corresponding value represents (e.g., PatientID(integer), VisitDate(date), Diagnosis(string), Treatment(string), DoctorID(integer)). In another example, the second schema concerns e-commerce data with a key-value pair, where each key describes what a corresponding value represents (e.g., OrderID(integer), CustomerID(integer), ProductID(integer), OrderDate(date), Quantity(integer), TotalPrice(decimal)). In this example, the second schema may be the Fast Healthcare Interoperability Resources® (FHIR) schema. The FHIR schema is a standard developed by Health Level Seven International (HL7) for the electronic exchange of healthcare information and is designed to facilitate interoperability between healthcare systems. The FHIR schema provides a common data language that enables data analysts, data scientists, and developers to collaborate and build data-driven solutions that improve patient outcomes and business performance. This schema supports data from across different healthcare business domains such as clinical, administrative, financial, and social. Accordingly, the transformation of modality data to delta tables of the FHIR schema captures data defined by the FHIR standard by making FHIR resources available as tables and columns within the second processing datastore. In one example, by flattening the FHIR information from first modality datainto delta tables, multimodal data management processenables the use of tools like T-SQL® and Spark SQL® to explore and analyze the multimodal data.

10 102 200 212 10 102 202 210 10 202 202 210 10 202 202 210 In some implementations, multimodal data management processtransformsthe first modality data into normalized first modality data by storing the first delta table of the second schema in a second processing datastore. As discussed above and in some implementations, the second processing datastore is a second lakehouse of the data lake (e.g., data lake). In one example, second processing datastorestores first modality data (e.g., first delta table of the second schema) in a normalized form relative to the first schema. In one example, multimodal data management processprocesses a predefined mapping of first modality data to corresponding delta table entries of the first schema to transformfirst modality datato normalized first modality data. In another example, multimodal data management processprocesses first modality datausing a trained machine learning model to map delta table entries in delta tables of the first schema from first modality datato corresponding entries in delta table(s) of the second schema for normalized first modality data. In another example, multimodal data management processprocesses first modality datausing a configurable template to map delta table entries in delta tables of the first schema from first modality datato corresponding entries in delta table(s) of the second schema for normalized first modality data.

10 112 210 210 10 112 210 212 214 214 210 10 112 210 In some implementations, multimodal data management processconvertsthe normalized first modality data into enhanced first modality data by converting the first delta table of the second schema into a first delta table of a third schema and storing the first delta table of the third schema in a third processing datastore. For example, normalized first modality datastored in second processing datastore is a normalized version of the compressed first modality data. However, normalized first modality datamay be defined in a second schema that is unable to aggregate data from other data modalities. Accordingly, multimodal data management processconvertsnormalized first modality datafrom second processing datastoreto enhanced first modality data (e.g., enhanced first modality data). In some implementations, enhanced first modality datais a representation of normalized first modality datathat adheres to a third schema by mapping key-value pairs across data modalities. For example, multimodal data management processconvertsnormalized first modality datainto enhanced first modality data by converting the delta table(s) of the second schema to a delta table of the third schema.

214 10 112 210 214 10 210 210 214 10 210 210 214 In some implementations, the third schema is Observational Medical Outcomes Partnership (OMOP). OMOP is a schema designed for standardizing observational health data for research and analytics. The OMOP schema provides a common data model (CDM) that allows users to conduct large-scale observational studies and comparative effectiveness research. As discussed above and in one example, normalized first modality data defines a delta table(s) of the second schema (e.g., FHIR). In this example, the FHIR schema is resource-based, with each resource representing a specific healthcare concept (e.g., patient, observation, medication). By contrast, OMOP is a comprehensive data model that is structured to support complex queries and analyses, making it suitable for research purposes. Accordingly, enhanced modality data (e.g., enhanced first modality data) includes modality data that is converted from a normalized data to delta tables of a third schema that also map to other modalities. In one example, multimodal data management processprocesses a predefined mapping of delta table entries of the second schema to corresponding delta table entries of the third schema to convertnormalized first modality datato enhanced first modality data. In another example, multimodal data management processprocesses normalized first modality datausing a trained machine learning model to map delta table entries in delta tables of the second schema from normalized first modality datato corresponding entries in delta table(s) of the third schema for enhanced first modality data. In another example, multimodal data management processprocesses normalized first modality datausing a configurable template to map delta table entries in delta tables of the second schema from normalized first modality datato corresponding entries in delta table(s) of the third schema for enhanced first modality data.

10 112 214 200 216 In some implementations, multimodal data management processconvertsthe normalized first modality data into enhanced first modality data by storing the first delta table of the third schema in a third processing datastore. As discussed above and in some implementations, the third processing datastore (e.g., third processing data store) is a third lakehouse of the data lake (e.g., data lake). In one example, third processing datastorestores enhanced first modality data in an enhanced form relative to the second schema (e.g., first delta table of the third schema). As discussed above and in some implementations, the third schema allows multimodal data (e.g., first multimodal data and second multimodal data) to be aggregated and processed in a multimodal data operation.

10 104 10 218 202 In some implementations, multimodal data management processcompressessecond modality data in the first processing datastore. For example, in addition to a first modality data, multimodal data management processprocesses second modality data (e.g., second modality data) that is data of a different modality than first modality data. In one example, the second modality data is image data. Compared to clinical data, for example, image data is typically stored in different file formats than clinical data. Additionally, image data generally consumes more storage space within a datastore than clinical data. In some implementations, image data is generated and stored separately than clinical data. For example, medical images produced by medical imaging machines (e.g., an X-ray machine or a magnetic resonance imaging (MRI) machine) are typically managed by the respective machine vendors or by the software operating the particular imaging machine. Accordingly, associating first modality data (e.g., clinical data) to related second modality data (e.g., image data) may not be possible from a comparison of each modality data separately in its raw format.

104 114 10 114 218 10 116 10 116 10 114 10 106 10 202 204 200 220 218 222 2 FIG. In some implementations, compressingthe second modality data includes compressingthe second modality data at a second compression level. As discussed above, the compression level for the second modality data is defined by the type or modality of the data being compressed. Continuing with the example of second modality data being image data, multimodal data management processcompressessecond modality dataat a second compression level for image data. In one example, multimodal data management processcompressesimage data by compressing all related image data in a folder or group-based compression. In another example, multimodal data management processcompressesimage data in a lossless manner as PNG or GIF format. In another example, multimodal data management processcompressesimage data in a lossy manner as JPEG format. However, it will be appreciated that other compression levels may be used when compressing second modality data. In some implementations, multimodal data management processcompressessecond modality data into a Parquet file. Referring again to, multimodal data management processprocesses second modality data (e.g., second modality data) for storage in a first processing datastore (e.g., first processing datastore) within data lakeusing a second compression process (e.g., second compression process) to store compressed second modality data′ and delta tables of the first schema (e.g., delta table).

10 104 10 104 222 204 In some implementations, multimodal data management processcompressesthe second modality data in the first processing datastore by extracting a metadata tag from the second modality data into a second delta table of the first schema and storing the second delta table of the first schema in the second processing datastore. Continuing with the above example, second modality data includes image data within the healthcare space. In this example and in some implementations, the image data is generated in an initial data format (e.g., Digital Imaging and Communications in Medicine (DICOM)). DICOM is a standard for transmitting, storing, and sharing medical images and related information. The DICOM standard facilitates the integration of medical imaging devices, such as scanners, servers, workstations, and network hardware, from multiple manufacturers. In some implementations, the DICOM standard or schema defines a plurality of tags (e.g., over 5,000 tags) that include standardized tags (e.g., “patientID”, “numberOfStudyRelatedSeries”, “instanceNumber”, etc.) and vendor-specific tags. In this example, multimodal data management processcompressesthe DICOM data of second modality data into delta tables of the first schema (e.g., staging tables or Parquet files) and stores the delta table (e.g., delta table) in first processing datastore.

10 106 10 106 218 224 106 218 224 10 218 224 10 218 224 In some implementations, multimodal data management processtransformsthe second modality data into normalized second modality data by transforming the second delta table of the first schema into a second delta table of the second schema and storing the second delta table of the second schema in the second processing datastore. For example, multimodal data management processtransformssecond modality datainto normalized second modality data (e.g., normalized second modality data) by processing a predefined mapping of delta table entries of the first schema to corresponding delta table entries of the second schema to transformsecond modality datato normalized second modality data. In another example, multimodal data management processprocesses second modality datausing a trained machine learning model to map metadata tags to corresponding entries in delta table(s) of the first schema for normalized second modality data. In another example, multimodal data management processprocesses second modality datausing an extensible configuration to map metadata tags to corresponding entries in delta table(s) of the first schema for normalized second modality data.

10 108 108 210 224 210 224 210 224 210 224 212 10 224 210 108 210 224 212 210 224 210 224 210 224 226 2 FIG. In some implementations, multimodal data management processassociatesthe normalized first modality data with the normalized second modality data in the second processing datastore. Associatingnormalized first modality datawith normalized second modality dataincludes linking particular portions of normalized first modality datawith corresponding portions of normalized second modality data. For example, suppose normalized first modality dataincludes EHR data for a particular person and normalized second modality dataincludes medical imaging dataset referenced by the EHR data. In this example as normalized first modality dataand normalized second modality dataare within second processing datastoreand are “normalized” to be in the same schema, multimodal data management processassociates the relevant medical imaging dataset of normalized second modality datawith the EHR data of normalized first modality data. In some implementations, associatingnormalized first modality dataand normalized second modality dataincludes generating an entry in a table within second processing datastoreindicating the association of normalized first modality dataand normalized second modality data. Referring again to, an association between normalized first modality dataand normalized second modality datais represented using the line between normalized first modality dataand normalized second modality data(e.g., association).

10 116 10 228 228 212 10 In some implementations, multimodal data management processstoreseach metadata tag of a plurality of metadata tags for the image data in a metadata store within the second processing datastore. For example, when processing image data as the second modality data, multimodal data management processretains every metadata tag of the second modality data in a metadata store (e.g., metadata store). In some implementations, metadata storeis a data structure within the second processing datastore (e.g., second processing datastore). In this manner, multimodal data management processmaintains each and every metadata tag from the second modality data as it is processed (e.g., normalized and enhanced).

118 218 204 10 118 230 232 204 218 230 204 200 10 204 In some implementations, transforming the second modality data into the normalized second modality data includes generatinga reference to each pixel of the image data stored in the first processing datastore without copying the pixel in the second processing datastore. For example, when transforming second modality datainto normalized second modality data, instead of copying each pixel of the image data stored in first processing datastore, multimodal data management processgeneratesa reference (e.g., pixel reference) to the pixel (e.g., pixel) stored in first processing datastore. In this manner, as second modality datais normalized and enhanced, pixels are not copied through each processing datastore but a reference to each pixel (e.g., pixel reference) in first processing datastoreis propagated through data lake. Accordingly, multimodal data management processreduces the amount of storage space required by image data of second modality data using references back to the pixel stored in first processing datastore.

10 120 224 224 202 10 120 224 212 234 234 224 10 120 224 In some implementations, multimodal data management processconvertsthe normalized second modality data into enhanced second modality data by converting the second delta table of the first schema into a second delta table of the second schema and storing the second delta table of the second schema in the third processing datastore. As discussed above, normalized second modality datastored in second processing datastore is a normalized version of the compressed second modality data. However, normalized second modality datamay be defined in the first schema that lacks the ability to be aggregated with data of other data modalities (e.g., first modality data). Accordingly, multimodal data management processconvertsnormalized second modality datafrom second processing datastoreto enhanced second modality data (e.g., enhanced second modality data). In some implementations, enhanced second modality datais a representation of second modality datathat adheres to the third schema by mapping key-value pairs across data modalities. For example, multimodal data management processconvertsnormalized second modality datainto enhanced second modality data by converting the delta table(s) of the second schema to a delta table of the third schema.

234 10 120 224 234 10 224 224 234 10 224 224 234 As discussed above and in some implementations, the third schema is OMOP which allows second modality data to be enhanced in OMOP's comprehensive data model structured to support complex queries and analyses. Accordingly, enhanced second modality data (e.g., enhanced second modality data) includes second modality data that is converted from a normalized data to delta table(s) of a third schema that also map to other modalities. In one example, multimodal data management processprocesses a predefined mapping of delta table entries of the second schema to corresponding delta table entries of the third schema to convertnormalized second modality datato enhanced second modality data. In another example, multimodal data management processprocesses normalized second modality datausing a trained machine learning model to map delta table entries in delta tables of the second schema from normalized second modality datato corresponding entries in delta table(s) of the third schema for enhanced second modality data. In another example, multimodal data management processprocesses normalized second modality datausing an extensible configuration to map delta table entries in delta tables of the second schema from normalized second modality datato corresponding entries in delta table(s) of the third schema for enhanced second modality data.

110 122 10 230 232 212 216 10 122 230 216 232 204 224 234 In some implementations, convertingthe normalized second modality data into enhanced second modality data includes maintainingthe reference to each pixel of the image data stored in the first processing datastore without copying the pixel in the third processing datastore. For example and as discussed above, multimodal data management processgenerates pixel referenceto avoid copying pixelin each of second processing datastoreand third processing datastore. Accordingly, multimodal data management processmaintainspixel referencein third processing datastoreto pixelstored in first processing datastoreduring the conversion of normalized second modality datainto enhanced second modality data.

10 124 214 234 10 124 214 234 10 124 236 228 216 214 234 124 214 234 238 10 214 234 238 10 In some implementations, multimodal data management processprocessesa multimodal data operation on a combination of the enhanced first modality data and the enhanced second modality data using the association of the enhanced first modality data and the enhanced second modality data in the third processing datastore. For example, with enhanced first modality dataand enhanced second modality dataand the association therebetween, multimodal data management processprocessesa multimodal data operation on the combination of enhanced first modality dataand enhanced second modality data. In one example, multimodal data management processprocessesa multimodal query (e.g., multimodal query) from a user or computing device. Multimodal queryis a query executed on third processing datastorethat retrieves enhanced first modality dataand/or enhanced second modality data. In some implementations, processingthe multimodal data operation includes processing enhanced first modality dataand enhanced second modality datausing a machine learning model (e.g., machine learning model). In this example, multimodal data management processprovides the association between enhanced first modality dataand enhanced second modality datato establish connections between inputs of machine learning model. Accordingly, multimodal data management processallows multimodal data storage and access by processing multimodal data through multiple stages of processing to compress, normalize, and enhance the multimodal data.

3 FIG. 10 300 302 300 Referring to, a multimodal data management processis shown to reside on and is executed by cloud system, which is connected to network(e.g., the Internet or a local area network). Examples of cloud systeminclude: a Network Attached Storage (NAS) system, a Storage Area Network (SAN), a personal computer with a memory system, a server computer with a memory system, and a cloud-based device with a memory system. A SAN includes one or more of a personal computer, a server computer, a series of server computers, a minicomputer, a mainframe computer, a RAID device, and a NAS system.

300 The various components of cloud systemexecute one or more operating systems, examples of which include: Microsoft® Windows®; Mac® OS X®; Red Hat® Linux®, Windows® Mobile, Chrome OS, Blackberry OS, Fire OS, or a custom operating system (Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other countries or both; Mac and OS X are registered trademarks of Apple Inc. in the United States, other countries or both; Red Hat is a registered trademark of Red Hat Corporation in the United States, other countries or both; and Linux is a registered trademark of Linus Torvalds in the United States, other countries or both).

10 304 300 300 304 10 300 The instruction sets and subroutines of multimodal data management process, which are stored on storage deviceincluded within cloud system, are executed by one or more processors (not shown) and one or more memory architectures (not shown) included within cloud system. Storage devicemay include: a hard disk drive; an optical drive; a RAID device; a random-access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices. Additionally or alternatively, some portions of the instruction sets and subroutines of multimodal data management processare stored on storage devices (and/or executed by processors and memory architectures) that are external to cloud system.

302 306 In some implementations, networkis connected to one or more secondary networks (e.g., network), examples of which include: a local area network; a wide area network; or an intranet.

308 310 312 314 316 300 308 300 300 Various input/output (IO) requests (e.g., IO request) are sent from client applications,,,to cloud system. Examples of IO requestinclude data write requests (e.g., a request that content be written to cloud system) and data read requests (e.g., a request that content be read from cloud system).

310 312 314 316 318 320 322 324 326 328 330 332 326 328 330 332 318 320 322 324 326 328 330 332 326 328 330 332 326 328 330 332 The instruction sets and subroutines of client applications,,,, which may be stored on storage devices,,,(respectively) coupled to client electronic devices,,,(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices,,,(respectively). Storage devices,,,may include: hard disk drives; tape drives; optical drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices. Examples of client electronic devices,,,include personal computer, laptop computer, smartphone, laptop computer, a server (not shown), a data-enabled, and a dedicated network device (not shown). Client electronic devices,,,each execute an operating system.

334 336 338 340 300 302 306 300 302 306 342 Users,,,may access cloud systemdirectly through networkor through secondary network. Further, cloud systemmay be connected to networkthrough secondary network, as illustrated with link line.

302 306 326 302 332 306 328 302 344 328 346 302 346 344 328 346 330 302 348 330 350 302 The various client electronic devices may be directly or indirectly coupled to network(or network). For example, personal computeris shown directly coupled to networkvia a hardwired network connection. Further, laptop computeris shown directly coupled to networkvia a hardwired network connection. Laptop computeris shown wirelessly coupled to networkvia wireless communication channelestablished between laptop computerand wireless access point (e.g., WAP), which is shown directly coupled to network. WAPmay be, for example, an IEEE 802.11a, 802.11b, 802.11g, 802.11n, Wi-Fi®, and/or Bluetooth® device that is capable of establishing a wireless communication channelbetween laptop computerand WAP. Smartphoneis shown wirelessly coupled to networkvia wireless communication channelestablished between smartphoneand cellular network/bridge, which is shown directly coupled to network.

As will be appreciated by one skilled in the art, the present disclosure may be embodied as a method, a system, or a computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present disclosure may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium.

Any suitable computer usable or computer readable medium may be used. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium may include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. The computer-usable or computer-readable medium may also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to the Internet, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the present disclosure may be written in an object-oriented programming language. However, the computer program code for carrying out operations of the present disclosure may also be written in conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through a local area network/a wide area network/the Internet.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer/special purpose computer/other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that may direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures may illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, not at all, or in any combination with any other flowcharts depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiment was chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

A number of implementations have been described. Having thus described the disclosure of the present application in detail and by reference to embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure defined in the appended claims.