Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method of detecting a data organization issue in a storage system including a backup server computer coupled to storage devices and executing a backup process, comprising: collecting input/output (I/O) traces as dynamic traces for data transfers to and from the storage devices in a virtualized file system that creates virtual devices that emulate physical storage and presents data as files of a local virtual disk, for address locations that move due to virtualization that maps space between a back-end LUN (logical unit number) not presented to a host computer, and a front-end LUN that is presented to the host computer; collecting file system traces as static traces for file and volume layout information of the data storage; tracking, in a splitter component of the backup server computer, write operations captured in the dynamic traces to the storage devices organized into primary and recovery storage locations; storing each write operation as a snapshot backup on the storage devices; storing, by a metadata collector, metadata for each write operation in a metadata repository; processing, by an analytics component, the dynamic and static traces to determine an effect of fragmentation for various workloads; analyzing, in an analytics component processing the dynamic and static traces, both a static layout of the file system and layout conditions leading to fragmentation, by analyzing the metadata to determine an amount of address space fragmentation manifested as non-contiguous storage of file data by the backup process by calculating a fragmentation value through a formula combining factors comprising a sequentiality of addresses within the virtual file system for file system objects spread among the different virtual disks, a number of backend input/output (I/O) operations; cache efficiency of metadata and data caches, and performance of the storage devices, to detect data organization issues in the virtual file system using metadata analytics and operational parameters to identify one or more indicators of the fragmentation; applying a plurality of rules to calculate the fragmentation value on the basis of factors comprising a relative amount of address sequentiality, number of backend metadata operations, I/O density, randomness of I/O patterns, cache miss ratio, I/O operation disk spread, and fragmentation in the virtual file system and storage on flash memory; expressing the calculated fragmentation value as one of a subjective relative characterization or a number that can be compared against a defined scale to facilitate determining if the amount of address space fragmentation presents a system performance issue; and suggesting remediation steps for a workload to reduce the amount of address space fragmentation across the virtual disks, to thereby improve space utilization by the backup process.
This invention relates to detecting and addressing data organization issues, particularly fragmentation, in virtualized storage systems. The system includes a backup server connected to storage devices, executing a backup process in a virtualized file system that emulates physical storage as virtual disks. The virtualization maps space between back-end LUNs (not visible to host computers) and front-end LUNs (visible to hosts), causing address locations to shift. The method collects dynamic I/O traces for data transfers and static file system traces for file and volume layout information. A splitter component tracks write operations to primary and recovery storage locations, storing them as snapshots. Metadata for each write operation is stored in a repository. An analytics component processes these traces to assess fragmentation effects on workloads by analyzing both static file system layouts and dynamic conditions leading to fragmentation. The analytics component calculates a fragmentation value using factors like address sequentiality, backend I/O operations, cache efficiency, and storage performance. This value is expressed as a relative characterization or a numerical score compared to a defined scale to identify performance issues. The system applies rules to evaluate fragmentation based on address sequentiality, metadata operations, I/O density, randomness, cache misses, disk spread, and flash memory fragmentation. It then suggests remediation steps to reduce fragmentation and improve space utilization in the backup process.
2. The method of claim 1 further comprising comparing the fragmentation value as the number against the defined scale of threshold values to determine whether the address space fragmentation is above an acceptable level, and wherein the subjective relative characterization comprises high, low, moderate, and negligible fragmentation.
This invention relates to monitoring and assessing memory address space fragmentation in computing systems. The method involves analyzing memory allocation patterns to quantify fragmentation, then categorizing it into subjective levels such as high, low, moderate, or negligible. The process begins by tracking memory allocation requests and deallocations to identify free address space blocks. A fragmentation value is calculated based on the distribution and size of these blocks. This value is then compared against predefined threshold values to determine if fragmentation exceeds acceptable limits. The thresholds define boundaries for the subjective categories, allowing system administrators to quickly assess fragmentation severity. The method helps optimize memory usage by identifying when fragmentation may degrade performance, enabling proactive measures like defragmentation or memory reallocation. The subjective characterization provides a human-readable assessment, making it easier to prioritize fragmentation management tasks. The technique is particularly useful in systems where memory efficiency is critical, such as embedded devices or high-performance computing environments. By quantifying fragmentation and providing clear categorization, the method supports better decision-making for memory management.
3. The method of claim 2 wherein the data is organized as files, blocks, or objects on the storage system.
The invention relates to data organization and management in storage systems, addressing the challenge of efficiently structuring and accessing data to optimize performance and resource utilization. The method involves organizing data in a storage system using different formats, including files, blocks, or objects, to enhance flexibility and adaptability. Files are structured as hierarchical entities with metadata and content, blocks are fixed or variable-sized units of data, and objects are self-contained data units with associated metadata. The method ensures compatibility with various storage architectures, allowing seamless integration with different storage systems. By supporting multiple data organization schemes, the invention enables efficient data retrieval, storage, and management, improving overall system performance and scalability. The approach is particularly useful in environments requiring diverse data handling capabilities, such as cloud storage, distributed systems, and enterprise data centers. The method optimizes resource allocation and reduces overhead by dynamically selecting the appropriate data organization format based on specific use cases and system requirements. This adaptability enhances data accessibility and ensures efficient utilization of storage resources.
4. The method of claim 3 further comprising, if the fragmentation value is above a minimum threshold value, implementing one or more of the suggested remediation steps to reduce the address space fragmentation.
This invention relates to managing address space fragmentation in computing systems, particularly in environments where memory allocation and deallocation operations lead to inefficient use of available memory. The problem addressed is the degradation of system performance due to fragmented address spaces, which can cause delays in memory allocation and increased overhead in memory management. The method involves monitoring the fragmentation level of an address space, where fragmentation is quantified by a fragmentation value. If this value exceeds a predefined minimum threshold, the system automatically implements one or more remediation steps to reduce fragmentation. These steps may include consolidating free memory blocks, relocating active memory regions, or adjusting allocation strategies to minimize future fragmentation. The system may also analyze historical fragmentation data to predict and preemptively mitigate fragmentation before it impacts performance. The method ensures that memory resources are used efficiently, reducing the likelihood of allocation failures and improving overall system responsiveness. By dynamically responding to fragmentation levels, the system maintains optimal memory utilization without requiring manual intervention. This approach is particularly useful in high-performance computing environments where memory efficiency is critical.
5. The method of claim 3 wherein the storage system for the local virtual disk includes one or more virtual machines.
A storage system for a local virtual disk is configured to manage data storage for virtual machines (VMs). The system includes a virtual disk that is accessible by one or more VMs, where the virtual disk is stored on a physical storage device. The storage system dynamically allocates storage space for the virtual disk based on the storage requirements of the VMs, ensuring efficient use of available storage capacity. The system also monitors the storage usage of the VMs and adjusts the allocation of storage space as needed to prevent resource contention or performance degradation. Additionally, the storage system may implement data redundancy mechanisms, such as replication or snapshots, to enhance data reliability and availability. The virtual disk can be shared among multiple VMs, allowing for centralized management of storage resources while maintaining isolation between the VMs. This approach optimizes storage utilization, simplifies administration, and improves the scalability of virtualized environments. The system may also support features like thin provisioning, where storage space is allocated on-demand rather than pre-allocating fixed amounts, further improving efficiency. The storage system ensures that the virtual disk remains accessible even if individual VMs are migrated or reconfigured, maintaining seamless operation in dynamic virtualized environments.
6. The method of claim 5 wherein the virtual file system comprises one of a Unix File System 64 (UFS64) and a Common Block File System (CBFS).
A virtual file system is used to manage and organize data in a computing environment, particularly in systems where traditional file systems may not be directly applicable, such as embedded systems or virtualized environments. The challenge is to provide a reliable and efficient file system that can handle large files and complex data structures while maintaining compatibility with existing software and hardware. This invention describes a method for implementing a virtual file system that supports either a Unix File System 64 (UFS64) or a Common Block File System (CBFS). UFS64 is an extended version of the traditional Unix File System designed to handle large files and volumes, while CBFS is a block-based file system optimized for embedded systems and virtual environments. The method ensures that the virtual file system can dynamically switch between these two file system types based on system requirements or user preferences, providing flexibility in deployment. The virtual file system is configured to manage file operations such as reading, writing, and metadata handling, while abstracting the underlying storage medium. This allows applications to interact with the file system without needing to know the specific implementation details. The system also includes mechanisms for error handling, data integrity checks, and performance optimization, ensuring reliable and efficient data management. By supporting both UFS64 and CBFS, the invention provides a versatile solution for different computing environments, from large-scale servers to resource-constrained embedded devices.
7. The method of claim 5 wherein the address space fragmentation within the virtual file system is caused by non-sequential storage of files or file data by reassignment of address locations by the virtualization mechanisms.
This invention relates to managing address space fragmentation in a virtual file system. The problem addressed is the inefficiency caused by non-sequential storage of files or file data, where virtualization mechanisms reassign address locations, leading to fragmented address spaces that degrade performance. The method involves detecting and mitigating fragmentation within the virtual file system. It identifies fragmented address spaces resulting from non-sequential storage or reassignment of address locations by virtualization mechanisms. Once detected, the method reorganizes the address space to reduce fragmentation, improving storage efficiency and performance. This may include relocating files or file data to contiguous address locations, consolidating scattered storage blocks, or adjusting virtualization mappings to minimize fragmentation. The solution is particularly useful in virtualized environments where dynamic address reassignment is common, ensuring that the virtual file system operates efficiently despite frequent changes in storage allocation. By proactively managing fragmentation, the method prevents performance degradation and maintains optimal resource utilization. The approach can be applied to various virtual file systems and storage architectures, enhancing reliability and speed in data access operations.
8. The method of claim 4 wherein the remediation steps comprise at least one of: sequential addressing or re-tiering of the data in the data storage system, increasing a cache size, re-distributing storage workload across multiple spindles of the storage devices, using of faster storage devices; and updating file system pointers to memory blocks after defragmentation occurs.
This invention relates to methods for optimizing data storage systems to improve performance and efficiency. The problem addressed is the degradation of storage system performance over time due to factors such as data fragmentation, uneven workload distribution, and suboptimal storage configurations. The invention provides a method for remediating these issues by implementing various optimization techniques. The method involves performing remediation steps to address performance bottlenecks in a data storage system. These steps include sequentially reorganizing or re-tiering data to improve access patterns, increasing cache size to reduce latency, redistributing storage workloads across multiple storage device spindles to balance I/O operations, utilizing faster storage devices for critical data, and updating file system pointers to memory blocks after defragmentation to ensure data integrity and efficient access. The remediation steps are applied dynamically based on system monitoring to maintain optimal performance. This approach ensures that the storage system remains efficient and responsive by proactively addressing common performance degradation factors.
9. The method of claim 1 wherein the formula comprises a product of the factors, the method further comprising applying a rule to each of the factors to determine a factor value for a respective factor of the formula.
This invention relates to a method for evaluating a formula composed of multiple factors, particularly in computational or analytical systems where formulas are dynamically assessed. The problem addressed is the need to efficiently compute and adjust factor values within a formula to achieve accurate or optimized results. The method involves breaking down the formula into its constituent factors and applying specific rules to each factor to determine its value. These rules may include mathematical operations, conditional logic, or other computational processes tailored to the nature of each factor. By systematically applying these rules, the method ensures that each factor contributes appropriately to the overall formula evaluation. The approach allows for flexibility in adjusting individual factors without requiring a complete reformulation of the entire formula, making it useful in applications such as financial modeling, scientific calculations, or algorithmic decision-making where dynamic adjustments are necessary. The method may also include validating the computed factor values to ensure they meet predefined criteria or constraints, further enhancing the reliability of the formula's output.
10. The method of claim 1 , further comprising: performing an I/O analytics process on the collected I/O traces to provide first organization layout advisories; and performing a file system analytics process on the collected file system traces to provide second organization layout advisories.
This invention relates to data storage systems and methods for optimizing data organization and layout within storage devices. The problem addressed is inefficient data storage and retrieval, which can lead to performance bottlenecks, increased latency, and wasted storage capacity. The invention provides a solution by analyzing input/output (I/O) and file system traces to generate advisories for improving data organization. The method involves collecting I/O traces, which capture read and write operations, and file system traces, which track file access patterns and metadata. These traces are analyzed to identify inefficiencies, such as fragmented data, suboptimal file placement, or inefficient storage utilization. The I/O analytics process evaluates the collected I/O traces to determine how data is accessed and suggests layout changes to reduce latency and improve performance. The file system analytics process examines the file system traces to identify opportunities for reorganizing files and directories to enhance storage efficiency and accessibility. By combining insights from both I/O and file system analytics, the invention generates comprehensive advisories that guide users or automated systems in restructuring data storage layouts. This results in faster data retrieval, reduced fragmentation, and better utilization of storage resources. The solution is particularly useful in large-scale storage environments where performance and efficiency are critical.
11. A system for detecting data organization issues in a storage system utilizing a file system including one or more virtual machines, comprising: an input/output (I/O) metadata capture component collecting input/output (I/O) traces as dynamic traces for data transfers to and from the storage devices in a virtualized file system that creates virtual devices that emulate physical storage and presents data as files of a local virtual disk, for address locations that move due to virtualization that maps space between a back-end LUN (logical unit number) not presented to a host computer, and a front-end LUN that is presented to the host computer; a file system metadata capture component collecting file system traces as static traces for file and volume layout information of the data storage; an I/O analytics component performing an I/O analytics process on the collected I/O traces to provide first organization layout advisories; a file system analytics component performing a file system analytics process on the collected file system traces to provide second organization layout advisories; a splitter component tracking write operations captured in the dynamic traces to storage devices organized into primary and recovery storage locations; a backup process storing each write operation as a snapshot backup on the storage devices; a metadata collector storing metadata for each write operation in a metadata repository; and an analytics component holistically analyzing both a static layout of the file system and layout conditions leading to fragmentation, in an analytics component processing the dynamic and static traces to determine an effect of fragmentation for various workloads, by analyzing the metadata to determine an amount of address space fragmentation manifested as non-contiguous storage of file data by the backup process by calculating a fragmentation value through a formula combining factors comprising a sequentiality of addresses within the virtual file system for file system objects spread among the different virtual disks, a number of backend input/output (I/O) operations; cache efficiency of metadata and data caches, and performance of the storage devices, to detect data organization issues in the virtual file system using metadata analytics and operational parameters to identify one or more indicators of fragmentation, by applying a plurality of rules to calculate the fragmentation value on the basis of factors comprising a relative amount of address sequentiality, number of backend metadata operations, I/O density, randomness of I/O patterns, cache miss ratio, I/O operation disk spread, and fragmentation in the virtual file system and storage on flash memory and expressing the calculated fragmentation value as one of a subjective relative characterization or a number that can be compared against a defined scale to facilitate determining if the amount of address space fragmentation presents a system performance issue, and suggesting remediation steps for a workload to reduce the amount of address space fragmentation across the virtual disks, to thereby improve space utilization by the backup process.
This system detects and addresses data organization issues in virtualized storage systems, particularly fragmentation in file systems used by virtual machines. The system monitors both dynamic I/O operations and static file system metadata to identify fragmentation patterns that degrade performance. It captures I/O traces from data transfers between virtualized storage devices, which emulate physical storage, and collects file system traces for file and volume layout information. An analytics component processes these traces to assess fragmentation by evaluating factors like address sequentiality, backend I/O operations, cache efficiency, and storage performance. The system calculates a fragmentation value using metrics such as I/O density, randomness, cache misses, and disk spread, then compares this value against predefined thresholds to determine if fragmentation is causing performance issues. It also tracks write operations to primary and recovery storage locations, storing snapshots and metadata for analysis. The system provides remediation suggestions to optimize storage utilization and reduce fragmentation across virtual disks, improving overall system efficiency. The approach combines metadata analytics and operational parameters to holistically assess and mitigate fragmentation in virtualized environments.
12. The system of claim 11 wherein the data is organized as files, blocks, or objects on the storage system, and wherein the subjective relative characterization comprises high, low, moderate, and negligible fragmentation.
The system is designed for managing data storage efficiency by assessing and categorizing data fragmentation levels. Data is stored as files, blocks, or objects on a storage system, and the system evaluates the degree of fragmentation subjectively, classifying it into four distinct levels: high, low, moderate, and negligible. Fragmentation refers to the scattering of data across non-contiguous storage locations, which can degrade performance by increasing access times and reducing throughput. The system identifies and quantifies fragmentation to optimize storage operations, such as defragmentation, data placement, or allocation strategies. By categorizing fragmentation into these levels, the system enables targeted interventions, such as prioritizing defragmentation for highly fragmented data or maintaining negligible fragmentation for critical performance-sensitive data. The subjective classification allows for flexible adaptation to different storage environments and performance requirements, ensuring efficient use of storage resources while minimizing performance degradation. The system may also integrate with existing storage management tools to provide actionable insights for administrators.
13. The system of claim 12 wherein the collected file system traces include at least one of the following: file metadata map, data I/O pattern, metadata data I/O pattern, cache effectiveness, slice distribution, and snapshot state.
This invention relates to a system for analyzing file system traces to optimize storage performance. The system collects detailed traces from a file system, including various types of data that help assess and improve storage efficiency. The collected traces include a file metadata map, which tracks metadata attributes such as file names, sizes, and timestamps. The system also captures data I/O patterns, which record how data is read from and written to storage, as well as metadata I/O patterns, which track metadata access operations. Additionally, the system evaluates cache effectiveness by measuring how well the cache reduces storage access latency. Slice distribution data is collected to analyze how data is distributed across storage slices, and snapshot state information is gathered to monitor the state of file system snapshots. These collected traces are then processed to identify inefficiencies, optimize storage operations, and enhance overall system performance. The system may also include a user interface for visualizing the collected data and generating reports. The analysis helps administrators make informed decisions about storage management, such as adjusting caching strategies, redistributing data, or optimizing snapshot operations.
14. The system of claim 13 wherein the file system comprises one of a Unix File System 64 (UFS64) and a Common Block File System (CBFS).
This invention relates to a data storage system designed to enhance file system compatibility and performance. The system addresses the challenge of managing large-scale data storage across different file system formats, particularly in environments requiring seamless integration between Unix-based and block-based storage solutions. The system includes a file system that supports either a Unix File System 64 (UFS64) or a Common Block File System (CBFS). UFS64 is a 64-bit extension of the traditional Unix File System, enabling support for large files and high-capacity storage. CBFS is a block-based file system optimized for efficient data block management, often used in embedded or specialized storage applications. The system dynamically selects or switches between these file systems based on operational requirements, ensuring compatibility with diverse storage hardware and software environments. Additionally, the system incorporates a storage controller that manages data access and allocation across the file system. The controller optimizes performance by dynamically adjusting file system parameters, such as block size and caching strategies, to match the characteristics of the underlying storage medium. This adaptability improves data throughput and reduces latency, particularly in mixed workload environments. The invention also includes error detection and correction mechanisms to maintain data integrity. These mechanisms monitor file system operations for inconsistencies and apply corrective actions, such as block remapping or checksum validation, to prevent data corruption. The system further supports multi-user access, allowing concurrent read and write operations while ensuring data consistency through locking and synchronization protocols. Overall, the system
15. The system of claim 14 further comprising an I/O metadata repository storing the metadata traces, wherein the repository implements a Hadoop cluster system.
A system for managing input/output (I/O) operations in a computing environment addresses the challenge of efficiently tracking and analyzing metadata associated with I/O operations. The system includes a metadata collection module that captures metadata traces from I/O operations, such as timestamps, data sizes, and storage locations. These traces are stored in an I/O metadata repository, which is implemented using a Hadoop cluster system. The Hadoop cluster provides scalable storage and processing capabilities, allowing the system to handle large volumes of metadata efficiently. The repository enables querying and analysis of the metadata traces to optimize I/O performance, detect anomalies, and improve data management. The system may also include a metadata processing module that processes the stored metadata to generate insights, such as performance trends or usage patterns. By leveraging the distributed architecture of Hadoop, the system ensures high availability and fault tolerance for the metadata repository. This approach enhances the ability to monitor and manage I/O operations in large-scale computing environments.
16. A computer program product comprising a non-transitory computer usable medium having machine readable code embodied therein for detecting a data organization issue in a storage system including a backup server computer coupled to storage devices and executing a backup process by: collecting input/output (I/O) traces as dynamic traces for data transfers to and from the storage devices in a virtualized file system that creates virtual devices that emulate physical storage and presents data as files of a local virtual disk, for address locations that move due to virtualization that maps space between a back-end LUN (logical unit number) not presented to a host computer, and a front-end LUN that is presented to the host computer; collecting file system traces as static traces for file and volume layout information of the data storage; tracking, in a splitter component of the backup server computer, write operations captured in the dynamic traces to the storage devices organized into primary and recovery storage locations; storing each write operation as a snapshot backup on the storage devices; storing, by a metadata collector, metadata for each write operation in a metadata repository; processing, by an analytics component, the dynamic and static traces to determine an effect of fragmentation for various workloads; analyzing, in an analytics component processing the dynamic and static traces, both a static layout of the file system and layout conditions leading to fragmentation, by analyzing the metadata to determine an amount of address space fragmentation manifested as non-contiguous storage of file data by the backup process by calculating a fragmentation value through a formula combining factors comprising a sequentiality of addresses within the virtual file system for file system objects spread among the different virtual disks, a number of backend input/output (I/O) operations; cache efficiency of metadata and data caches, and performance of the storage devices, to detect data organization issues in the virtual file system using metadata analytics and operational parameters to identify one or more indicators of fragmentation; applying a plurality of rules to calculate the fragmentation value on the basis of factors comprising a relative amount of address sequentiality, number of backend metadata operations, I/O density, randomness of I/O patterns, cache miss ratio, I/O operation disk spread, and fragmentation in the virtual file system and storage on flash memory; expressing the calculated fragmentation value as one of a subjective relative characterization or a number that can be compared against a defined scale to facilitate determining if the amount of address space fragmentation presents a system performance issue; and suggesting remediation steps for a workload to reduce the amount of address space fragmentation across the virtual disks, to thereby improve space utilization by the backup process.
This invention relates to detecting and addressing data organization issues, particularly fragmentation, in virtualized storage systems. The system includes a backup server connected to storage devices, managing data transfers through a virtualized file system that emulates physical storage as virtual disks. The virtualization maps space between back-end LUNs (not visible to hosts) and front-end LUNs (visible to hosts), causing address locations to shift. The system collects dynamic I/O traces for data transfers and static file system traces for file and volume layout information. A splitter component tracks write operations, storing them as snapshots in primary and recovery storage. Metadata for each write is stored in a repository. An analytics component processes these traces to assess fragmentation effects on workloads by analyzing both static file system layouts and dynamic conditions leading to fragmentation. Fragmentation is quantified using a formula that combines factors like address sequentiality, backend I/O operations, cache efficiency, and storage device performance. The system evaluates fragmentation by measuring address space fragmentation, I/O density, randomness, cache misses, and disk spread. The calculated fragmentation value is expressed as a relative characterization or a numerical score, compared against a defined scale to identify performance issues. The system then suggests remediation steps to reduce fragmentation and improve space utilization in the backup process.
Unknown
September 15, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.