GitHub Faces Performance Challenges Due to DNS Incident in October 2024
Peter Zhang Nov 15, 2024 09:43
GitHub experienced a significant performance issue in October 2024, affecting DNS infrastructure and impacting services like Copilot and Actions workflows.
In October 2024, GitHub experienced a notable incident that led to degraded performance across its services, according to GitHub. The issue was traced back to a DNS infrastructure failure following a database migration at one of the company's sites.
Incident Overview
The incident began on October 11 at 05:59 UTC and lasted for over 19 hours. The initial problem occurred when the site's DNS infrastructure failed to resolve lookups after a database migration. Efforts to recover the database resulted in cascading failures, further impacting DNS systems. Customers began experiencing issues around 17:31 UTC, with 4% of Copilot users facing degraded IDE code completions and 25% of Actions workflow users encountering delays exceeding five minutes. Additionally, all code search requests failed for approximately four hours.
Response and Resolution
Attempts to mitigate the issue by redirecting the affected DNS site to an alternative location were initially unsuccessful, as this strategy impaired connectivity from healthy sites back to the degraded one. At 20:52 UTC, GitHub's team implemented a remediation plan, deploying temporary DNS resolution capabilities to the affected site. DNS resolution began to recover at 21:46 UTC and was fully operational by 22:16 UTC. Remaining issues with code search were resolved by 01:11 UTC on October 12.
Future Preventative Measures
Following the incident, GitHub committed to strengthening its resiliency and automation processes to expedite the diagnosis and resolution of similar issues in the future. The company aims to improve infrastructure reliability to prevent such incidents from recurring.
For real-time updates on GitHub's service status, users are encouraged to visit the GitHub Status Page. Additionally, insights into ongoing projects and improvements can be found on the GitHub Engineering Blog.
Image source: Shutterstock