A Hadoop job kept saying “running” after its Resource Manager restarted, but the work never finished. The app master, the process coordinating that job, received a reply telling it to reboot. One code path had a TODO and a log line where reboot handling should have been. The path regular jobs used did not check the reboot flag. MAPREDUCE-3186 records the report and the later fix. An engineer worried about having to clean up app masters across a cluster of 4,000 nodes; that comment describes a conditional concern, not a verified 4,000-node production outage.
Ding Yuan and colleagues used this case in a broader study. They examined 198 randomly sampled, user-reported failures in Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis. Their analysis followed reports, developer discussions, logs, code, and patches; they manually reproduced 73 cases. These are documented issue-tracker failures from five distributed data systems, not a survey of every software outage.
Read the paper
The USENIX OSDI 2014 paper is linked from the paper card below. We link to the proceedings copy rather than hosting a duplicate PDF. Printed proceedings pages 249 to 262 contain the study, its twelve findings, and Aspirator's evaluation.
This paper is hosted by its original publisher.
Read on usenix.org
