Super Facinating! (Destiny)
Here's some notes I sorta wrote while watching. They explain the problems in a lot more detail and its very much worth watching if you have a bit of programming knowledge. My version below is greatly simiplified and probably a bit wrong but it gets the gist enought right... I hope...
In Halo, all the different assets (sounds, shaders, models, the concept of an enemy AI unit, etc) were all fairly separate. This meant you could swap out the sound an exploding box made with a different sound and quickly rebuild the game and test your result. Maybe that new sound file had to be compressed down into the right console specific format, but no big deal. Same with models, same with everything.
One of the problems with Halo though was you couldn't really do fast small patches. They basically had to ship the entire game as one big unit and the only time we really ever saw major patches was with the big DLC releases where the player was already expecting to download a significant amount of data (or load it off the multiplayer DLC disk they bought.)
For Destiny, Bungie intended to update things much more frequently than with Halo. It no longer made sense to make the player redownload the entire game just to add a mission or whatever. I'm hitting the limit of my understand a bit, but they at the same time were also trying to make it easier to develop Destiny and make rapid changes.
Through some bad assumptions and trying to be too clever they shot themselves in the foot... with a nuke. They didn't go super in depth, but it sounds like a part of this was caused by having to support four very different platforms all at the same time. For instance, the PS3's Cell processor had a main CPU and the smaller simpler high performance SPU sub processors that individually didn't have a lot of memory or resources. So the systems that translated Destiny from artwork and sounds and models into PS3 code had to be very clever about chopping things up so the pieces could each fit into a single SPU.
In Destiny development, unlike in Halo's, it ended up that objects became all linked and tangled together. The example they gave was a box might glow brighter as it took damage. So the box might reference a glow shader and might reference the sound it made when you punched it and so on those sounds and shaders had to know the box's hp value.
In Halo development you could swap out the box or the shader or the sound and since none of those pieces knew anything about the other it all kinda worked.
In Destiny making a change to any of those pieces meant the system that took all the assets and translated them into the proper format to run on the different consoles not only had to recompress the sound file, it had to load up the box's model, and shaders and and everything else that made the box work.
Worse, because of the way the levels are optimized as a whole to save space and memory, changing the sound a box makes when punched might mean that the tool that translated everything to the way it needed to be for the consoles would have to load in and rebuild an entire level area.
So the smallest of changes, like swapping out a sound, would mean the audio guy might need to wait two hours while an entire area like the Mothyards and every object and shader and sound and unit was loaded in from disk and reoptimized back to code the consoles could run.
To make matters worse, some of the systems and tools that were supposed to let them make changes without having to load in everything simply weren't ready in time.
Bungie has since rebuilt their systems significantly to improve build times. They basically found a way to split up the way objects and assets reference each other. Everything still has to reference everything in the end because that's how the game engine works, but their new system lets them isolate each piece so making a little change doesn't kick off a huge overnight rebuild.
So it sounds like swapping out that sound file only triggers a much much smaller rebuild. The tool that builds the game can now often dig down and just mess around with the sound file or the dimensions of a crate instead of having to load in and rebuild an entire level from scratch.
There were other big problems too. The tool that did the job of building the game... it would run out of memory part way through and die because the process it had to do, following the references from object to object in an ever expanding tree of complexity, got so big that the application was literally running out of memory space in Windows.
Ultimately, it sounds like a lot of the big problems have been fixed, or at least significantly worked around. They didn't say exactly, but it would seem that merely swapping out a sound or graphic will no longer trigger an multi-hour or overnight rebuild. What took five hours might now be accomplished in minutes.
We've all been wondering what happened to Destiny. Why it didn't live up to what we hoped it would be. It sound like this major development tool blunder sucked in a lot of programming talent to fix and was a extremely major issue for all of Destiny's development slowing down the workflow of everybody from the mission planners to the individual artists and sound designers. I can't even imagine how they managed to ship such a polished game as they did with these massive problems clogging their development pipeline.
Hearing that things have been made better gives me Hope for the Future that Destiny 2 will be much easier for Bungie to develop and they'll be able to get a lot more work done without super long rebuild times popping up for even the smallest change.
Complete thread:
- Asset Systems and Scalability -
Cody Miller,
2017-01-04, 17:36
- Cool. Saw this yesterday then forgot about it... Will watch. -
Ragashingo,
2017-01-04, 17:56
- TLDW: - Cody Miller, 2017-01-04, 18:49
- Super Facinating! -
Ragashingo,
2017-01-05, 03:31
- Super Facinating! -
Cody Miller,
2017-01-05, 13:34
- Super Facinating! -
Ragashingo,
2017-01-05, 13:56
- Super Facinating! -
CruelLEGACEY,
2017-01-05, 14:30
- Super Facinating! - Ragashingo, 2017-01-05, 15:51
- Super Facinating! -
CruelLEGACEY,
2017-01-05, 14:30
- Super Facinating! -
Ragashingo,
2017-01-05, 13:56
- Super Facinating! -
Cody Miller,
2017-01-05, 13:34
- One Other Thing... -
Ragashingo,
2017-01-05, 16:16
- One Other Thing... -
Claude Errera,
2017-01-05, 16:46
- One Other Thing... - CruelLEGACEY, 2017-01-05, 16:56
- I'm never telling you about the near screw ups - Durandal, 2017-01-05, 17:22
- One Other Thing... -
bluerunner,
2017-01-05, 17:25
- One Other Thing... - Claude Errera, 2017-01-05, 19:02
- Agreed 100% - marmot 1333, 2017-01-05, 17:37
- One Other Thing... - slycrel, 2017-01-05, 17:41
- One Other Thing... -
dogcow,
2017-01-05, 18:31
- One Other Thing... - Cody Miller, 2017-01-05, 19:25
- One Other Thing... -
Cody Miller,
2017-01-05, 19:27
- One Other Thing... -
Claude Errera,
2017-01-05, 19:45
- One Other Thing... -
CruelLEGACEY,
2017-01-05, 20:12
- One Other Thing... - Cody Miller, 2017-01-05, 22:46
- One Other Thing... - cheapLEY, 2017-01-05, 22:52
- One Other Thing... -
Cody Miller,
2017-01-06, 16:59
- One Other Thing... - Claude Errera, 2017-01-06, 17:18
- One Other Thing... -
Kermit,
2017-01-06, 18:21
- One Other Thing... -
Cody Miller,
2017-01-06, 18:38
- One Other Thing... -
Kermit,
2017-01-06, 19:23
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 19:28
- One Other Thing... - CruelLEGACEY, 2017-01-06, 19:32
- One Other Thing... -
Cody Miller,
2017-01-06, 20:34
- One Other Thing... - CruelLEGACEY, 2017-01-06, 20:54
- One Other Thing... -
Korny,
2017-01-07, 15:36
- One Other Thing... - Harmanimus, 2017-01-09, 16:54
- On a related note... -
CruelLEGACEY,
2017-01-07, 19:04
- I have seen it - Cody Miller, 2017-01-07, 20:10
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 19:28
- One Other Thing... -
Kermit,
2017-01-06, 19:23
- One Other Thing... -
Cody Miller,
2017-01-06, 18:38
- One Other Thing... -
CruelLEGACEY,
2017-01-05, 20:12
- One Other Thing... - CruelLEGACEY, 2017-01-05, 19:57
- One Other Thing... -
Claude Errera,
2017-01-05, 19:45
- Agreed. - Kermit, 2017-01-05, 22:22
- I take offense to your statements -
unoudid,
2017-01-05, 22:31
- I take offense to your statements - CruelLEGACEY, 2017-01-05, 23:08
- I take offense to your statements -
Oholiab,
2017-01-06, 00:24
- I take offense to your statements - unoudid, 2017-01-06, 01:13
- Tell me more about chairs pls. -
Funkmon,
2017-01-06, 04:49
- +1 - cheapLEY, 2017-01-06, 12:35
- Tell me more about chairs pls. - unoudid, 2017-01-06, 12:46
- Tell me more about chairs pls. -
CruelLEGACEY,
2017-01-06, 12:48
- If I must -
unoudid,
2017-01-06, 14:22
- Bonus points for hitting 2 birds with 1 stone XD -
CruelLEGACEY,
2017-01-06, 16:33
- A Stone you say? -
unoudid,
2017-01-06, 17:33
- Is there any way you can tie this to ScarJo? lol -
CruelLEGACEY,
2017-01-06, 17:51
- Well, this got weird - stabbim, 2017-01-06, 18:26
- Wasn't she tied to a chair in Avengers? -
Ragashingo,
2017-01-06, 18:38
- This research at work is getting sketchy - unoudid, 2017-01-06, 18:42
- The best I can do -
unoudid,
2017-01-06, 19:02
- You're setting a dangerous precident by humouring me ;p -
CruelLEGACEY,
2017-01-06, 19:23
- Time for a trip down memory Lane -
unoudid,
2017-01-06, 19:43
- Cranbrook also has a few museums on its land - Funkmon, 2017-01-07, 11:53
- chairs.bungie.org - someotherguy, 2017-01-08, 21:56
- Time for a trip down memory Lane -
unoudid,
2017-01-06, 19:43
- You're setting a dangerous precident by humouring me ;p -
CruelLEGACEY,
2017-01-06, 19:23
- Is there any way you can tie this to ScarJo? lol -
CruelLEGACEY,
2017-01-06, 17:51
- Birds? Le Corbusier aka The Crow -
unoudid,
2017-01-06, 17:50
- A much better image than Jobs -
unoudid,
2017-01-06, 20:08
- That chair looks like if someone said to me... -
Funkmon,
2017-01-07, 11:55
- That chair looks like if someone said to me... -
cheapLEY,
2017-01-07, 13:06
- +1 - CruelLEGACEY, 2017-01-07, 14:21
- That chair looks like if someone said to me... -
cheapLEY,
2017-01-07, 13:06
- That chair looks like if someone said to me... -
Funkmon,
2017-01-07, 11:55
- A much better image than Jobs -
unoudid,
2017-01-06, 20:08
- A Stone you say? -
unoudid,
2017-01-06, 17:33
- DBO: We know more about chairs than you do. -
Ragashingo,
2017-01-06, 19:32
- DBO: We know more about chairs than you do. - unoudid, 2017-01-06, 19:44
- If I must - BeardFade, 2017-01-06, 21:55
- Bonus points for hitting 2 birds with 1 stone XD -
CruelLEGACEY,
2017-01-06, 16:33
- If I must -
unoudid,
2017-01-06, 14:22
- I take offense to your statements -
Xenos,
2017-01-06, 15:20
- I take offense to your statements -
Claude Errera,
2017-01-06, 16:23
- I take offense to your statements - CyberKN, 2017-01-06, 16:28
- I take offense to your statements -
Claude Errera,
2017-01-06, 16:23
- RIP Mary Tyler Moore -
unoudid,
2017-01-26, 12:25
- RIP Mary Tyler Moore - Kermit, 2017-01-26, 17:05
- "Everyone I know Goes away in the end" -
Pyromancy,
2017-01-28, 08:18
- "Everyone I know Goes away in the end" -
unoudid,
2017-01-28, 16:05
- This is amazing -
Robot Chickens,
2017-01-30, 23:31
- This is amazing - unoudid, 2017-01-31, 00:01
- This is amazing -
Robot Chickens,
2017-01-30, 23:31
- "Everyone I know Goes away in the end" -
unoudid,
2017-01-28, 16:05
- Counterpoints in the dark -
narcogen,
2017-01-06, 02:01
- Counterpoints in the dark -
Claude Errera,
2017-01-06, 03:14
- FTFY - Kermit, 2017-01-06, 15:10
- Counterpoints in the dark -
Claude Errera,
2017-01-06, 03:14
- One Other Thing... -
Kahzgul,
2017-01-06, 05:05
- One Other Thing... - Harmanimus, 2017-01-06, 15:39
- One Other Thing... -
Ragashingo,
2017-01-06, 16:41
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 17:47
- One Other Thing... -
Ragashingo,
2017-01-06, 18:05
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 19:16
- One Other Thing... - Cody Miller, 2017-01-06, 21:31
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 19:16
- One Other Thing... - stabbim, 2017-01-06, 18:29
- One Other Thing... -
Cody Miller,
2017-01-06, 18:48
- One Other Thing... Wall Hugging Hippos. -
dogcow,
2017-01-09, 14:09
- +1 - Kermit, 2017-01-09, 14:34
- One Other Thing... Wall Hugging Hippos. -
dogcow,
2017-01-09, 14:09
- One Other Thing... -
Ragashingo,
2017-01-06, 18:05
- One Other Thing... -
Kahzgul,
2017-01-07, 05:53
- +1 million -
cheapLEY,
2017-01-07, 13:23
- Thanks buddy :) It's taken me awhile to figure out, too -
Kahzgul,
2017-01-07, 17:57
- For whatever it's worth . . . - cheapLEY, 2017-01-07, 21:11
- Thanks buddy :) It's taken me awhile to figure out, too -
Kahzgul,
2017-01-07, 17:57
- One Other Thing... -
Claude Errera,
2017-01-07, 18:40
- One Other Thing... -
Cody Miller,
2017-01-07, 20:05
- One Other Thing... -
Kermit,
2017-01-08, 00:18
- One Other Thing... -
Cody Miller,
2017-01-08, 01:59
- One Other Thing... -
Kermit,
2017-01-09, 14:50
- One Other Thing... - Cody Miller, 2017-01-09, 17:00
- One Other Thing... -
Kermit,
2017-01-09, 14:50
- One Other Thing... -
Cody Miller,
2017-01-08, 01:59
- One Other Thing... -
Kermit,
2017-01-08, 00:18
- One Other Thing... -
Kahzgul,
2017-01-07, 21:53
- Lots of people felt misled about Halo 2. -
Funkmon,
2017-01-08, 07:13
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 13:56
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 14:39
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 14:43
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 15:06
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 15:16
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 15:27
- Lots of people felt misled about Halo 2. -
Ragashingo,
2017-01-08, 16:05
- Lots of people felt misled about Halo 2. - cheapLEY, 2017-01-08, 16:51
- Lots of people felt misled about Halo 2. -
uberfoop,
2017-01-08, 18:15
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 18:27
- Lots of people felt misled about Halo 2. -
Cody Miller,
2017-01-08, 20:39
- Did we just full circle this thing? - ZackDark, 2017-01-08, 23:27
- Lots of people felt misled about Halo 2. -
Cody Miller,
2017-01-08, 20:39
- Lots of people felt misled about Halo 2. -
Ragashingo,
2017-01-08, 18:49
- Lots of people felt misled about Halo 2. - Kahzgul, 2017-01-09, 05:23
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 18:27
- Lots of people felt misled about Halo 2. -
Ragashingo,
2017-01-08, 16:05
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 15:27
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 15:16
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 15:06
- Lots of people felt misled about Halo 2. -
stabbim,
2017-01-09, 12:49
- Lots of people felt misled about Halo 2. -
DiscipleN2k,
2017-01-09, 15:42
- Lots of people felt misled about Halo 2. - uberfoop, 2017-01-09, 19:13
- Lots of people felt misled about Halo 2. -
Cody Miller,
2017-01-09, 20:03
- Nor their tails -
Durandal,
2017-01-09, 21:58
- Nor their tails - dogcow, 2017-01-09, 22:18
- Nor their tails -
Ragashingo,
2017-01-09, 22:58
- Nor their tails - Cody Miller, 2017-01-10, 00:23
- Nor their tails -
Durandal,
2017-01-09, 21:58
- Lots of people felt misled about Halo 2. - Kahzgul, 2017-01-10, 21:39
- Lots of people felt misled about Halo 2. -
DiscipleN2k,
2017-01-09, 15:42
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 14:43
- Lots of people felt misled about Halo 2. -
cheapLEY,
2017-01-08, 14:39
- Lots of people felt misled about Halo 2. -
Xenos,
2017-01-08, 13:56
- Lots of people felt misled about Halo 2. -
Funkmon,
2017-01-08, 07:13
- One Other Thing... -
Cody Miller,
2017-01-07, 20:05
- +1 million -
cheapLEY,
2017-01-07, 13:23
- One Other Thing... -
CruelLEGACEY,
2017-01-06, 17:47
- One Other Thing... -
Claude Errera,
2017-01-05, 16:46
- Asset Systems & Scalability, Thx Butcher&Bungie, Fascinating -
dogcow,
2017-01-05, 20:08
- +1000 - Claude Errera, 2017-01-05, 20:09
- Asset Systems & Scalability, Thx Butcher&Bungie, Fascinating - CruelLEGACEY, 2017-01-05, 20:22
- He always give quality talks - kidtsunami, 2017-01-05, 22:30
- Building Your Airplane While Flying - uberfoop, 2017-01-07, 08:04
- Cool. Saw this yesterday then forgot about it... Will watch. -
Ragashingo,
2017-01-04, 17:56