Modern datacenters operate at extraordinary scale. Large cloud platforms deploy fleets comprising millions of processors, each integrating increasingly complex components. As scale and complexity grow, so does the exposure to processor faults. While many faults manifest as crashes or exceptions, others are far more subtle to identify. Silent data corruptions (SDCs), also known as silent data errors26 or silent errors,13 are violations of data integrity that occur without immediate, observable, and explicit indications. Because of their silent nature, a non-negligible delay exists between the occurrence of an SDC and its detection. This delayed detection allows corrupted data to propagate across applications, potentially triggering cascading failures and even large-scale service catastrophes.13 The combination of detection difficulty and systemic risk makes SDCs a critical concern for modern datacenter services.
Some SDCs originate from CPU misbehavior; here, we refer to them as CPU SDCs. Historically, CPU SDCs have been largely attributed to external sources, such as irradiation or environmental noise.2,31,34 Under such assumptions, SDCs are believed to have extremely low prevalence in datacenters and are rarely observed in production environments, consequently drawing limited attention from datacenter service vendors.13
However, production evidence from modern datacenters challenges these longstanding assumptions. Recent works indicate that CPU SDCs do occur whose root causes are internal processor faults rather than external sources. Specifically, these works fall into three main categories: in-depth case studies of individual incidents,11,35 collaborations with large-scale industrial partners that provide incident data from production environments,20,38 and synthetic incident generation via hardware simulation.24,33 These works also challenge the conventional narrative: The behavior, manifestation patterns, and impact of CPU SDCs may be more complicated than previously believed, highlighting the need to clearly understand the differences between SDCs caused by internal processor faults and those triggered by external sources.
In this article, we systematically revisit CPU SDCs from a production datacenter perspective. We specifically target CPU SDCs observed in production environments, where processor faults serve as the root cause, and compare them with those under traditional assumptions (i.e., CPU SDCs with external sources as root causes) to characterize the deficiencies of current fault models and strategies. We explore the presence of CPU SDCs, investigate their patterns, and evaluate existing strategies against them. Based on the insights from our investigation and analysis, we also provide potential future research directions to promote the reliability of modern datacenter services.
CPU SDCs in the Wild
In this section, we investigate the presence of CPU SDCs across different system layers, to answer the question of why CPU SDCs have become a significant issue in modern datacenters.
Root causes of CPU SDCs. Processor faults are defects and bugs in hardware components and/or chip designs. Many types of processor faults have the potential to generate CPU SDCs. As depicted in Figure 1, these faults can be divided into two categories: device faults and in-field faults. Device faults occur before the use of processors, such as design bugs and manufacturing defects.36,39 In-field faults occur after users begin using processors, such as transistor weakness and wearing out.30
A representative root cause of CPU SDCs is timing marginalities, where hardware defects prevent semiconductor circuits from reaching steady states within the configured clock rate.6,10 Processors with issues of timing marginalities can function correctly under a sufficient power supply and intermittently manifest wrong results under a low voltage supply. The testing of timing marginalities presents a significant challenge due to the massive corner cases that must be covered.
Both device faults and in-field faults are inevitable in production environments. First, considering the cost associated with the comprehensive high-volume manufacturing (HVM) testing, defects and bugs related to device faults cannot be completely eliminated in production chips. Second, in modern datacenters, in-field faults occur throughout the whole working lifespan of processors,39 leaving CPU SDCs caused by in-field faults continuing to affect online services.14,20 Moreover, the increasing complexity and advances in semiconductor lithography further intensify both the processor vulnerability and the difficulty in chip testing. This trend suggests that CPU SDCs are likely to pose greater challenges in the future.
Prevalence of CPU SDCs. Measuring the prevalence of faulty processors that can generate SDCs is necessary to evaluate the influence of CPU SDCs in datacenter services. Several datacenters have reported their measurements on faulty processors that have the potential to generate SDCs: Google and Meta have indicated that the failure rate of SDC-related processor faults is a few occurrences in every several thousand processors.14,20 Alibaba Cloud offers more precise data, indicating that this failure rate is 3.61‱, based on measurement in over one million production processors.38 These reports suggest that CPU SDCs are relatively infrequent events, particularly when compared to other types of errors commonly observed in datacenters.18 Consequently, CPU SDCs were rarely detected in the past, mainly due to the smaller system scales in earlier years. With the scale-up trends in recent years, however, CPU SDCs have been more frequently observed.
As a comparison, the prevalence in the traditional assumption is extremely low: CPU SDCs caused by external sources are expected to occur once in about 1,000 years in a production datacenter.31 The error prevalence significantly impacts the design of related strategies, determining their practical applicability. Due to their extremely rare prevalence, CPU SDCs caused by external sources have received limited attention and are often overlooked in datacenters.37 In contrast, CPU SDCs caused by processor faults are more prevalent and manifest as a general problem in datacenter services, especially in large-scale systems.
Prevalence across processor components. Studies have shown that CPU SDCs tend to originate from specific processor components.7,8,26 It is possible that these components are indeed more complex and consequently more vulnerable. For example, floating-point computations are believed to be more complex than integer computations in many operations, and recent research confirms that processor components related to floating-point computations are indeed more vulnerable to CPU SDCs.38 It is also possible that some components tend to generate errors that have explicit signals; consequently, they are not vulnerable to SDCs, as confirmed by recent studies.7
Severity of CPU SDCs. In Table 1, we list several cases where CPU SDCs have impacted real-world services. The service failures caused by CPU SDCs exhibit computational impacts analogous to those caused by software bugs. However, their distinct root causes necessitate different strategies for detection, diagnosis, and handling. Here are a few representative examples; the others exhibit similar behaviors:
In case MT1, a mathematical function produces incorrect results on a faulty processor in the fleet. This function is used to calculate the decompressed file size in a database application and incorrect results lead to data loss.
In case ALI1, the checksum calculation is affected. This calculation is widely used in end-to-end checks; incorrect results abort numerous requests on the faulty processor, causing service quality to fall below the guarantee.
In case GGL1, some processors are prone to CPU SDCs when executing specific instructions in a newly updated library. Consequently, a series of SDCs occur when the analysis application is updated to use the newest library, leading to incorrect responses to users.
Table 1.
Part of production SDC cases. Note that in some CPU/GPU hybrid services, vendors report that they face both CPU and GPU SDCs, without a rigorous distinction between the sources in their cases.
| Case ID | Service Vendor | Root Cause | Service Application | Influence |
|---|---|---|---|---|
| MT114 | Meta | CPU | Database | Data loss |
| MT212 | Meta | CPU | KV Store | Data loss |
| ALI138 | Alibaba Cloud | CPU | Storage system | Performance degradation |
| ALI238 | Alibaba Cloud | CPU | Metadata management | Massive assertion failures |
| ALI340 | Alibaba Cloud | CPU | Block store | Performance degradation |
| GGL120 | CPU | Data-analysis pipeline | Wrong results | |
| GGL216 | CPU/GPU | Large language model (LLM) | Impacted training | |
| DS11 | DeepSeek-AI | CPU/GPU | DL framework | Degraded model performance |
From these cases, we observe that the scale of the deployment is a crucial factor when assessing the impact of CPU SDCs, as all of the cases listed in Table 1 are deployed on a large scale. For example, in case ALI1, the impacted service is deployed to over a million processors.
Moreover, we observe that impacts are prevalent across popular applications, including various storage applications (e.g., cases MT1 and ALI1) and data-analysis applications (e.g., case GGL1). This observation can be attributed to two reasons: First, with the high performance requirements and continuous development, these applications utilize some complex processor features (e.g., vector units) to improve performance. However, these complex processor features may also be vulnerable,7 making these applications susceptible to CPU SDCs. Second, these applications usually require large-scale deployment, which also leads to their susceptibility to CPU SDCs.
Furthermore, we observe that CPU SDCs usually compromise the basic functionality of the service, highlighting the severity of CPU SDCs. For example, in case MT1, data stored in the database is lost. The data loss violates the fundamental guarantee of storage services, leading to QoS violations. We classify the conditions under which CPU SDCs compromise the basic service functionality into two categories. In the first category, both systems and applications lack effective detection strategies against CPU SDCs, such as cases MT1 and GGL1. Due to the absence of effective detection, erroneous results propagate across operations, resulting in failures. By the time users notice the issue, the effects may exceed the failure recovery capability of the system, as reported in case MT2. In the second category, systems or applications do identify the occurrence of CPU SDCs, such as cases ALI1 and ALI2. However, due to the limited understanding of CPU SDCs, they are unable to respond appropriately to these errors, resulting in significant resource waste during the error handling process. For example, in case ALI1, the affected application attributes the root cause of detected SDCs to disk faults rather than to processor faults. Consequently, the faulty processor continues to generate CPU SDCs, leading to the waste of considerable resources on error handling. It is worth noting that the second condition rarely occurs in traditional assumptions, where the root causes (e.g., noise and irradiation) are usually assumed to be transient.2
Patterns of CPU SDCs
It is crucial to understand patterns of CPU SDCs to develop better fault models and strategies. In this section, we introduce patterns observed in production environments, compare them with patterns under traditional assumptions, and investigate the implications.
Patterns of corrupted data. Bitflips in CPU SDCs tend to occur at specific positions. For example, in floating-point numbers, bitflips predominantly occur in the fractional part.39 One architectural reason for this pattern lies in timing marginalities, which predominantly impact the least significant bits.6 Another reason for this pattern is the variation in hardware vulnerability across different bit positions. Specifically, the computational logic associated with the fractional part is more complex than the logic in other parts, making it more vulnerable to bitflips. Additionally, research observes that within the data cache, the least significant bits exhibit a higher likelihood of manifesting CPU SDCs, compared with the most significant bits.33
As a comparison, under traditional assumptions, each bit position has an equal probability of flipping.15 The differences in bitflip distribution between practical observations and traditional assumptions hold significant implications, particularly concerning the value of corrupted data. For example, for floating-point numbers, CPU SDCs will incur minor precision losses. As measured by Alibaba Cloud, 97.5% of the precision losses in double-precision floating-point numbers (float64) are below 0.01% (as shown in Figure 2). A case study by Google also reports that one of its CPU SDCs can only incur 0.0000003% precision loss.35 Certain computational methods, such as the Jacobi method, can tolerate these minor precision losses through several iterations.15 Conversely, under traditional assumptions, CPU SDCs can lead to significant accuracy degradation, as many bitflips occur in the exponential part.
In practical observations, the incidence of multiple bitflip events is non-trivial. For example, in extended double precision floating-point numbers, dual bitflips occur in approximately 20% of cases, and about 8% of cases have even more flipped bits.38 This prevalence contrasts sharply with traditional assumptions, where multiple bitflip events are relatively rare.31
Corrupted data also exhibits patterns in relation to inputs. First, in some CPU SDCs, certain input prerequisites, such as value constraints (e.g., a minimum value threshold) and bit constraints (e.g., a specific bit must be set to 1), are necessary to trigger the SDC. Second, a single processor fault often produces bitflips at consistent positions across varying inputs. These patterns do not exist in traditional assumptions, highlighting the need to refine existing fault models for CPU SDCs.
Patterns of SDC occurrence. A faulty processor may occasionally behave correctly even under conditions that can produce SDCs. To evaluate the reproducibility of CPU SDCs, researchers use the metric of occurrence frequency, quantified as the number of observed errors per minute. We observe that patterns exist in SDC occurrence, which also distinguish practically observed CPU SDCs from those traditionally assumed.
Significant occurrence variance. The reproducibility of CPU SDCs exhibits significant variability, with the occurrence frequency ranging from as low as 0.01 times per minute to several hundred times per minute.39 This can be attributed to the variations in processor faults, such as the degree of aging.25
Production observations20,38 suggest that CPU SDCs may differ from transient errors or permanent errors, which are used in some assumptions.7,32 However, due to the significant variance of occurrence frequency observed in production environments, CPU SDCs face the challenges of both transient and permanent errors: Less reproducible CPU SDCs tend to mimic transient errors, where the poor reproducibility makes identifying the root cause a challenge. Highly reproducible CPU SDCs tend to mimic permanent errors, where the faulty processor may need to be promptly identified and isolated to prevent the continual emergence of new CPU SDCs.
Triggering conditions. In addition to processor faults and workloads, the occurrence of CPU SDCs is influenced by various triggering conditions. These triggering conditions include environmental factors like temperature and humidity, as well as system configurations such as frequency and voltage.20,36
Temperature is a typical triggering condition, with its effects attributable to hardware phenomena like bias temperature instability (BTI) and hot carrier injection (HCI), both of which cause the performance and reliability of circuits to deteriorate as temperature rises. Figure 3 shows the occurrence frequency variance with processor temperature in a CPU SDC sample, from which we observe that a minimum temperature threshold is required for CPU SDC occurrence, and the occurrence frequency grows exponentially with increased temperatures.
Strategies against CPU SDCs
In this section, we evaluate strategies against CPU SDCs, organizing them into seven types and analyzing their strengths and limitations in consideration of practically observed patterns, with a brief summary in Table 2. We also introduce strategies used in existing production environments, highlighting the need for further research on strategy improvement.
Table 2.
Existing strategies against CPU SDCs.
| Type | Universality | Effectiveness | Overhead |
|---|---|---|---|
| Testing | ✓ | ✓ | ✕ |
| Control | – | ✓ | ✕ |
| Monitor | ✓ | ✕ | ✓ |
| Redundancy | ✓ | ✓ | ✕ |
| Coding | ✕ | ✕ | ✓ |
| Prediction | ✕ | ✕ | – |
| ABFT | ✕ | ✓ | ✓ |
Proactive strategies for mitigating CPU SDCs. Proactive strategies are performed before faulty processors manifest SDCs. These strategies can mitigate issues associated with CPU SDCs by reducing their occurrence, but they are ineffective in addressing already manifested CPU SDCs.
Testing. It is practical to reduce the incidence of faulty processors in the fleet by testing their functions. Various toolchains have been developed for identifying processor faults that could cause SDC issues.14,24,26,30,35 In a typical testing process, multiple carefully crafted test cases are executed on the targeted processor, and processor faults can be identified by comparing execution results with expected results.
Recent advances in testing strategies have expanded coverage across a wide range of SDC conditions. For example, a recent work employs a hardware-model-in-the-loop methodology to generate functional test programs, significantly enhancing SDC coverage across diverse micro-architectures and fault types.24 However, given the inevitability of processor faults in production environments (see “Root causes of CPU SDCs”), online or periodical testing in production environments becomes necessary. Such testing competes with online services for computational resources. Additionally, comprehensive identification of all CPU SDCs requires long testing durations and extreme environmental conditions.38 As a result, conducting comprehensive in-production testing incurs unbearable overhead.
Control. Since specific triggering conditions are required for the manifestation of CPU SDCs, keeping system parameters below their triggering thresholds can mitigate CPU SDC issues. However, the control mechanism needs to co-execute with online services, potentially introducing performance fluctuations to online services. The overhead of such control becomes especially unbearable for CPU SDCs that are easily triggered. A recent work suggests that controlling triggering conditions is best used as a complement to proactive SDC testing, especially for CPU SDCs that are difficult to identify by testing.38
Monitoring. Some system events, such as core dumps, can serve as the side channels of CPU SDCs.14,39 These events can be monitored and evaluated to identify SDCs with low operational overhead. However, their detection performance is limited, as these side channels are not direct indicators of SDC occurrence. Consequently, in practical deployments, these monitoring strategies also serve as a complement to other detection techniques.14
Passive strategies for tolerating CPU SDCs. Passive strategies are designed to address CPU SDCs that have already occurred. These strategies allow systems to detect such errors and tolerate their effects.
Redundancy. Redundancy strategies execute identical computational processes on separate hardware resources, comparing results from two redundant processes to detect SDCs or voting results from a greater number of redundant processes to correct incorrect results.9,19,27 Redundancy strategies can be applied in various layers. For example, dual-core lockstep (DCLS) enables hardware redundancy,9 and software redundancy can be applied to protect operations or reconstruct corrupted data.
Redundancy strategies provide stable detection and correction of CPU SDCs. However, the critical weakness of these strategies is the significant overhead: Given the infrequent occurrence of CPU SDC events, the excessive resource consumption for redundancy is generally unsustainable.
Coding. Data integrity can be ensured via checksums or parities, which are encoded from the data.22,23,40 These coding strategies have been widely used for detecting and/or correcting corruptions that affect data storage and transfer. Error correcting codes (ECCs) can be used to protect caches and registers.22 End-to-end checks can protect data transfer by using checksum algorithms, such as cyclic redundancy check (CRC). Erasure codes (ECs) facilitate the reconstruction of corrupted data through the use of parity blocks.23
The biggest concern with coding strategies is their ineffectiveness while facing CPU SDCs. ECCs used in processors are usually designed for tolerating a single flipped bit,22 but multiple-flipped-bit events occupy a non-negligible proportion in practical observations (see “Patterns of corrupted data”). Other coding strategies often rely on vulnerable components to accelerate algorithms like CRC and EC,23 leading to potential strategy ineffectiveness. Consequently, it is possible that coding strategies produce wrong results, like case ALI1 in Table 1. Additionally, coding strategies have limited universality in addressing CPU SDCs, since they cannot be used to protect computational components in the processor.
Prediction. Prediction strategies3,13 use machine learning (ML) techniques to predict a range for the expected result. The occurrence of SDCs is identified when the actual result of the computational process falls outside this range. These ML techniques leverage certain computational properties, such as the smoothness characteristic of iterative high-performance computing (HPC) applications.
Prediction strategies exhibit two limitations. First, precision losses observed in CPU SDCs are relatively minor (see “Patterns of corrupted data”), making it challenging to predict a narrow result range.28 Additionally, computational properties that such prediction relies on are limited in certain HPC scenarios, such as auto-regressive models and linear/quadratic curve fitting.13 Consequently, existing prediction strategies have limited application.
Algorithm-based fault tolerance. Algorithm-based fault tolerance (ABFT) is an effective strategy for stable detection of CPU SDCs.4,5,21 The core concept of ABFT strategies is to introduce information that has some relation to the data (e.g., linear dependence), and maintain this information during the computation process on the data. When an SDC occurs, ABFT strategies can detect the error by checking the relationship between the information and the outcome data. ABFT strategies incur relatively low overhead, but the practical usage of ABFT strategies is limited due to the “non-universality” issue: Each ABFT strategy is designed for specific algorithms, such as ordinary differential equations and bitonic sort, and systems still need additional strategies for other affected operations.13
Strategies used in practice. In current production environments, testing strategies are the most widely deployed among the aforementioned approaches. Many well-known datacenters, such as those operated by Google, Meta, and Alibaba Cloud, have deployed testing against CPU SDCs, with the scale of millions of processors and years of operational experience.14,35,38 This preference is due to the diversity of applications requiring protection in datacenter services, particularly in scenarios like platform-as-a-service (PaaS) and infrastructure-as-a-service (IaaS), where applications are varied and program internals are opaque. Testing strategies offer flexible and universal protection in such scenarios. Another advantage lies in accelerated diagnosis. When a datacenter service encounters a CPU SDC, software engineers need weeks of debugging and hardware-engineer cooperation to distinguish it from other possible causes (e.g., software bugs). The use of carefully crafted testing toolchains, which achieve relatively stable reproduction of CPU SDCs and give clear information for root-cause analysis, significantly shortens this diagnostic process. Furthermore, testing strategies also require no code modifications, significantly reducing engineering overhead.
Testing strategies, however, cannot always provide sufficient detection accuracy, especially for applications with high reliability requirements.38 In such scenarios, additional passive strategies are required to detect CPU SDCs. But existing strategies still struggle with universality, effectiveness, and/or overhead. Coding, prediction, and ABFT strategies only protect a subset of operations in certain applications, leaving numerous others still susceptible to CPU SDCs. For example, while storage applications widely adopt coding strategies to protect data-path integrity, many CPU SDCs occur outside the data path, as evidenced by real-world incidents (see Table 1). Consequently, these storage applications remain vulnerable to undetected CPU SDCs. Additionally, the high overhead of redundancy strategies restricts their applicability. For example, HDFS selectively deploys redundancy strategies for a subset of erasure coding operations, leaving many other operations unprotected.19
As a result, there remain numerous operations that cannot be effectively protected in modern production environments, indicating that current reliability requirements have not been achieved. In current datacenter services, the incidence of CPU SDCs reported by users is comparable to, or even surpasses, the incidence of CPU SDCs reported automatically.20 This discrepancy indicates the need for enhanced strategies against CPU SDCs.
Future Directions against CPU SDCs
Further research is urgently needed to promote the reliability of modern datacenter services. In this section, we provide potential future research directions from multiple perspectives.
Hardware design. As discussed earlier, the increasing complexity of electronic structures and the growing intricacy of lithographic processes indicate that CPUs may become more vulnerable in the future. Therefore, enhancing hardware reliability is imperative to decrease the incidence of CPU-induced errors. We suggest that research on hardware reliability improvements should focus on certain processor features rather than the entire processor, such as applying redundancy to particular instructions and optimizing the operating conditions of specific processor components. These targeted strategies may prove to be both effective and efficient, as CPU SDCs tend to occur in specific components. On the other hand, current methodologies for identifying vulnerable components remain inadequate, and require future research. For example, some recent works assume that component vulnerability related to CPU SDCs is mainly determined by chip area.33 However, other works show that even relatively small components (e.g., FPU) can exhibit significant vulnerability, likely due to architectural complexity rather than physical size.7,39
Another critical research direction involves investigating SDCs caused by hardware devices with architectural and manufacturing similarities to CPUs, such as tensor processing units (TPUs), graphics processing units (GPUs), and data processing units (DPUs). These processing units are increasingly prevalent in modern datacenters, and their growing deployment scale indicates they are also exposed to SDC risks. For example, recent research reports that a production fleet experienced GPU SDCs during LLM training, motivating systematic SDC testing on GPUs.29 Given the architectural kinship to the CPU, SDCs caused by these processing units may exhibit similar patterns to CPU SDCs. But workloads on these processing units, such as parallelized tensor computations and high-throughput data processing, are different from those on CPUs, which leads to divergent SDC implications compared to those in CPU scenarios. However, the precise patterns and impacts of SDCs caused by these processing units remain underexplored, necessitating further research to characterize the fault model and develop mitigating strategies. This line of research is challenging due to the low prevalence of SDCs. Research on CPU SDCs sheds light on this challenge: Industry plays an important role in advancing SDC research, as large-scale deployments generate a sufficient volume of real-world SDC incidents that are essential for academic studies.17
Software development. As analyzed earlier, existing strategies are inadequate to address CPU SDC issues. Considering the significant challenges of reducing strategy overhead, we suggest leveraging the universality of the strategy as a trade-off to achieve this. Unlike traditional assumptions, where any operations in the service may be affected, practical observations show that only the operations that extensively utilize vulnerable processor features are susceptible to CPU SDCs.38 We recommend that system engineers deploy testing strategies for basic protection, and deploy specific protection for operations that both exhibit susceptibility to CPU SDCs and have high reliability requirements. High-overhead strategies can then be selectively applied to these specific operations instead of all operations in the service, thereby reducing the overall overhead. For example, recent research has explored selective redundancy for several cloud applications against CPU SDCs and demonstrated cost-effective runtime performance.27 We further suggest engineers and researchers devote increased attention to developing novel ABFT strategies for susceptible operations not covered by existing ABFT strategies.
Datacenter maintenance. Many services, such as PaaS and IaaS, lack details of upper software applications, leading them to mitigate CPU SDCs during the maintenance phase. However, existing maintenance practices remain inefficient due to the lack of concentration on triggering conditions, often resulting in counterintuitive events in services, which require significant engineering effort to resolve. For example, in the Google fleet, a CPU SDC case exhibits a higher occurrence frequency when the processor is configured at a lower frequency state,20 where lower voltage supply becomes the triggering condition.36 Similarly, an incident at Alibaba Cloud reveals that the updated testing toolchain fails to achieve better detection and even has a decreased CPU SDC coverage, because the updated toolchain operates at a lower execution temperature, which diminishes the reproducibility of some CPU SDCs.38 We suggest that datacenter maintenance should monitor and manage these triggering conditions. For example, it would be beneficial to build the testing toolchain with the ability to configure and control processor temperature in testing and develop test cases that effectively reach corner working conditions.
Conclusion
In this article, we revisit CPU SDCs in modern datacenters from a systematic, deployment-driven perspective, contrasting real-world observations with longstanding assumptions. Based on our investigation, we outline several future research directions for promoting the reliability of modern datacenter services. We believe this work can enhance the understanding of CPU SDCs, thereby promoting the development of more effective strategies to address related challenges.
Acknowledgments
We thank all reviewers for their insightful comments and helpful suggestions. This work was supported by the National Natural Science Foundation of China under Grant 62025203.