I agree Nectum and great write up! If I may rebuttal as I have not read all the replies, here are my thoughts: - This is not an enterprise IT department such as in Microsoft, Oracle, etc. (relating to them mayube mainly due to being related to complex databases, and from what I think i know, bungie might be running a hombrew database, they make their own engines. - But this is an enterprise company that relies on IT and Devs to ensure their only product, an online service, stays online. - Microsoft 1809 patch deleting peoples data... with no way for microsoft to revert that change.. its happens lots so just making sure we are not on a bungie only bandwagon, happens everywhere to everyone. (well those are consumers on targeted non managed WaaS) - As I posted elsewhere, its not just testing, rollout and pull/rollback. Bungie already outlined all patch notes for every major flaw in the game being patched. A simple rollback and delay in patching, with game downtime being 1-2 hours instead of 12, was not a choice, they had to identify now, dev a patch for the patch, and re-release on a restored database cluster to prevent assumingly a week or more of entire community exploits - Next is enterprise IT and services... yes they are online only.. but its a game that we agree in the EULA that there are no guarantees, no up-time assurance, we dont pay for support services. We pay (those that do) for a game, in which half the community rips anal cavities for a DLC thats $5 more... Hard to rebute that they should offer such for an online game service when the community responds this way We also do not pay for a monthly or annual license. Destiny 1 was played heavily by the community yesterday, its an online service that people paid for, but once, not annually, not monthly... there is no guaranteed revenue for bungie to created assurances, SLA reimbursements etc. There is no other online hosted example in enterprise or from IT perspectives I think that can create a system in which guarantees are guaranteed from bungie.. right? now onto testing... I hate testing patches, updates, upgrades. (you can say i suck) but as THE decision maker for and IT department, I promote limited testing. But I also promise recovery from any and every incident. The "downtime" or "recovery time" spent in the past decade ACTUALLY recovering from little to no testing of database upgrades, patches, etc. and having it mess up and spending time to recover..... well its a fraction of human, technical, and financial resources when compared to the time and cost to vet everything fully in every aspect.... so why as the decision maker would i make the decision to hire 20% more labor resources, delay patches 25% longer, and spend 25% more on the testing environment? Another Enterprise example: Why would i delay March 2017 SMB patch by 90 days because that's our policy and need to test it in every foreseeable aspect possible before deployment, only to have our entire enterprise infrastructure ransomwared by wanna cry in May 2017? Another rebuttal.... outside of bungies lost revenue from downtime.... How is bungie, an enterprise cloud service hosting provider, negatively impacting any of its customers who do not pay in a recurring manner? not at all, bungies customers are not losing business, in fact they are probably doing more business lol. LAST SCENARIO: On to testing methodology. What if the flaw for currency was only happening to 10% of players and was due to one planetary material being over 1,200, while also having this shader in inventory, and this quest line unfinished... those conditions specifically were what cased the database to meltdown. - I know that the scenario can be tested... but thats one variable to test. - As enterprise IT can you understand this side? Then acknowledge "ok we offer a game service, that on this day during this time even if it goes bad will have minimal loss in revenue, and our clients no loss in revenue" - If you are THE decision maker of that team do you say "we should double our annual IT budget for extended testing"? - Or do you say "yea 50 hours of downtime per year, when 80% of it's experienced during uneventful weekly resets and non-DLC purchasing time periods and only costs us 0.0345% of what extended testing would cost is preferred and acceptable"? - UAT would have worked here... but does UAT work for bi-monthly patches? How many guardians do you know want to go play a test patch to see if the hardlight is any good before anyone else.. only to have that time spent earn you no progress on the actual game? And the team to design such. Ok so we have to setup a side system for testing approvals.. ok we got it same thing as our beta testing environment... ok but beta testing you lose all progress, unless its a new game no one wants to beta test the hardline shake reduction.. So now we need to create a patch testing instance that approved can join but still ear progress, but only limited progress.... like we are introducing new quests so we have to omit this from the full patch so they cant complete a quest before the community... ok we need to spend half a million per year more in payroll for another 5 employees to manage this test platform that allows bounty progress, promotes testing, but still hides new quests, perks etc, until released...... no that suck.... lets make them all sign NDAs.... no that doesnt work, thats too small a sample size. PTR.. lets just test a nightfall and crucible.... well players still have to access their account info unlike a beta test... they have to have some way to have the PTR and Live instances link as the core issue is a database bug from having a weird combination of this that and this when in this state will cause all your materials to be lost... if 5% of the community was impacted by yesterdays failed patch... they need at least 2-5% to participate to have a large enough sample to identify an issue? only 10% of the community plays weekly so... its hard Ok lets use machine learning and AI to simulate every calculation of guardians, inventories, and classes, and will be able to simulate all these insane test scenarios, that are view as unnecessary, to test for us? get ready to drop $milly on that tech... or maybe "hey we are bungie.. yea we can spend millions more on labor and test environments... but the AI affordability is around the corner... lets just frooping release a dang patch and recover from backup when needed...