Custom Upload Update delayed - Murphy's Law hit us hard
I am deeply sorry to announce, that we have to delay the custom assets update.
Once the new servers were up and running last Friday, we were set out to apply some minor adjustments, do some fine tuning and bring the update online bit by bit.
Everything started out fine, until we ran into our first major issue: The new server was suddenly serializing image requests instead of handling them asynchronous. In other words: If 1000 people are requesting an asset, the last one has to wait until all other images are delivered. We were instantly on that task and made good progress, when the second thing broke: The automated deployment scripts that used to work fine suddenly refused to work anymore and we were forced to deploy our updates manually. That is usually manageable, when you have a test state and a live state, but with our new three-phase-deployment and at least three developers working simultaneously on updates and bugfixes utter chaos was on the brink of breaking out.
Still hoping that we can get everything sorted out until Sunday, we reorganized us in two separated work groups: One to fix the server issues and one to continue with the update. But that turned out to be extremely time consuming: As soon as something on the servers needed to be changed, the update group had to pause and as soon as a new build was ready for testing, the server group had to delay their work and deploy the new build manually.
It was a back and forth that costed us the whole sleep-deprived weekend without knowing for sure, if everything works out as intended when we put it together today.
We had to decide: Put everything together and release it - more or less untested - hoping that it will work out fine, or admit defeat and ask for more time.
So instead of forcing an update that has a high chance of breaking things, just for the sake of being on schedule, I made the call and decided to do the latter:
We need more testing, before our server architecture is stable and we have regained some trust in our deployment scripts. Then, afterwards we will have the time and the means to properly test the new build in a constant testing-feedback-update-cycle until we can make sure, that the new custom asset features is working as intended.
I apologize sincerely for this delay and will keep you posted as soon as we have things sorted out.