Bungie done did it again... (Destiny)
We are investigating the re-emergence of the issue causing missing currencies and materials after Hotfix 2.7.1.1 went live. Destiny 2 will remain offline, please stand by for further updates.
— Bungie Help (@BungieHelp) February 11, 2020
I’d just avoid D2 for a few hours after any hot fix at this point...
Although you may not even have a choice. According to Bungie, D2 will be offline until 10pm EST.
Back up!
(See what I did there...eh... anyway...)
Destiny 2 servers are back online. Players may experience login issues as services return to normal.
— Bungie Help (@BungieHelp) February 12, 2020
Players who observe issues should report to the Help Forum:https://t.co/cOAIctERNC
Due to the rollback:
— Bungie Help (@BungieHelp) February 12, 2020
💠 Any progress or purchase made between 8:30-10:30 AM PST will need to be redone
💠 Silver spent during that time will be restored
💠 Platform store-purchased Silver bundles will not be affected by the rollback
💠 Fractaline will be returned to inventories https://t.co/Rq4SVfPNJX
Destiny Companion features have been re-enabled on Web, Mobile, and Third-Party Apps.
— Bungie Help (@BungieHelp) February 12, 2020
Players should report any issues with these features to the Help forum: https://t.co/dXNlQalio5
A fascinating debriefing/retrospective from the DOC
The Bungie Community Team has posted a fascinating debriefing/retrospective from the Destiny Operations Center or Player Support Team regarding the recent game outages.
Bungie.net | Destiny 2 Outage and Rollback
I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)
Such a good deep dive into the issue
The Bungie Community Team has posted a fascinating debriefing/retrospective from the Destiny Operations Center or Player Support Team regarding the recent game outages.
Bungie.net | Destiny 2 Outage and Rollback
I don't like the unfortunate circumstances but I LOVE reading these debriefing/retrospective/breakdowns so much! (I heard a talk recently from an IBM X-Force Team member that tickled my brain in such ways)
I Thank those who were involved with the actions and fix, as well as those who pieced everything together and put it into a readable document (which is not as easy as you might think it is)
This is exactly why I roll my eyes whenever some armchair developer suggests something is an "easy" fix. At a certain scale developers are literally wrestling with alternate realities existing at the same time...
Such a good deep dive into the issue
Humorously I had this exact discussion with someone at work. And my point was that it’s not practical to test 1:1 against a production environment and that it arose because the use-case that caused it wasn’t in their test sample. Which this confirms. It is also a good reminder that it is a game made by people and people don’t trend to perfection and that is okay. Their response to all of this has been, imo, a good way to handle such an immediate recurrence.
+1
- No text -
TL;DR - Spaghetti
Disabled timestamps on some quests, which lowered the stack limit on items to 0.
Good writeup though, I dig the transparency.
Cached / stale servers / data..
... yeah, can't say I haven't been bit by that a few times.
And like what happened to them, you can miss it because you got (un)lucky and hit a fresh instance in test while old instances are still out there.
Such a good deep dive into the issue
Humorously I had this exact discussion with someone at work. And my point was that it’s not practical to test 1:1 against a production environment and that it arose because the use-case that caused it wasn’t in their test sample. Which this confirms. It is also a good reminder that it is a game made by people and people don’t trend to perfection and that is okay. Their response to all of this has been, imo, a good way to handle such an immediate recurrence.
Yep.
And it was still their first public data failure in 16 years of running games that consume and process persistent data.
The fact that both rollbacks were caused by the same bug and just sheer bad luck points that overall, they've been pretty good at what they do.
Solid Write-Up!
I'm not sure I understood the whole thing mind you, but I enjoyed the peek behind the curtain all the same. :)