Thursday, October 2, 2008

Is Gamebryo Good Middleware?

Kyle Wilson writes nicely thought out articles on game development. He just wrote a piece, Defining Good Middleware, on Gamasutra (also on his site, Game Architect).

I’ve been asked how Gamebryo stacks up to his definition of good middleware. So, I’ll discuss that, plus throw in some additional thoughts. Read his article first.

How does Gamebryo stack up?

Good middleware lets you hook your own memory allocator.

Certainly. We ship an optimized pooled allocator for small objects, and a memory debugging/tracking/analysis system as well. Snapping on your own allocator is easy.

Getting this implemented in a code base is much easier to do early on! Major Kudos go to Shawn Kime who was the muscle man implementing this feature onto a game engine that had already shipped across a console generation and contained nearly 10,000 files.

Good middleware lets you hook your own I/O functions.
Yes, you can register your own functions for loading engine content. There are a few edge cases, e.g. on some platforms we use platform specific APIs sometimes to load certain file types.

Good middleware has extensible functionality.
We design for customer extensions. But, this is a significant task! Every feature must be considered for extensibility, and the right amount of flexibility selected. Some extensibility comes with a performance or memory cost. Most flexibility adds complexity to code.

Solid C++ design can be used in many areas, allowing customers to register custom derived objects with a factory system. The cost is polymorphism, and adding conditionals to code to handle unforeseen cases customers may introduce. Other areas can use template programming with policies, functors, visitors, etc.

Gamebryo 2.5 shipped with a massive overhaul of the geometry system, allowing customers to arbitrarily specify data formats, semantics, interleaving, and sharing. This offers significant benefits, but with a cost. Several fast paths must exist for code based on different formats of data. Some modules restrict the data layout to a known subset. Template programming is used to offer "perfect inner loops" at compile time, but increase the complexity of the code. And the exporting pipeline complexity is increased as client data format requests are normalized against known constraints of the runtime.

Extensibility can not be taken lightly, it is a significant design concern for every feature added to middleware.

Good middleware avoids symbol conflicts.
Gamebryo code has used the "Ni" prefix for a decade... legacy from the product name, "NetImmerse". Once you pick a method to encapsulate your symbols, you’ll find it time consuming to refactor, and a burden to customers to change as well. Big new features are coming for Gamebryo, though, and we are moving to using name spaces for those.

Good middleware is explicit about its thread safety.
Gamebryo has focused on concurrency for some time. Background loading is one of the most basic forms. We also demonstrate how to multithread other areas of the codebase, e.g. concurrent scene graph updates. And, we ship an extremely powerful stream processing solution, Floodgate, built to make high-performance cross-platform multithreading easy for our customers.

Good middleware fits into your data pipeline.
Generic and platform optimized assets are definitely supported. A data pipeline for a game is a complicated system however, and there’s more to it than just intermediate files and packaged level files. There is the need to format data precisely as needed for your game, and we offer rich configuration for data formatting. There’s also the need to manipulate, optimize, and augment data. For that, we offer a configurable plug-in based processing pipeline.

Good middleware is stable.
Gamebryo has a mature code base tended by a large team of engineers for a decade. It’s been battle tested in 200+ games. We ship only a couple of times per year, with a rigorous end cycle of automated and manual testing. And yet, it’s a massive code base, evolving to offer better performance and features, and exposing significant customer extensibility. There are still issues that come back to us, but that is why we have a support team.

The reality of the market is that customers prefer advanced features and performance over high stability. You want it all, but at the end of the day you have to choose where to put your time and effort. What does it take to have a 100% stable codebase? How much effort goes into flight control software stability? How much testing is the right amount for a game engine when there are opportunity costs?

Good middleware gives you source.
Absolutely. If you’re building a AAA game on a middleware engine there is no question, this is essential. That said, casual or low budget games can do amazing things with a binary only version.

This statement gathered quite a few comments on the original posts, a few quotes here:

"... you could equally argue that good middleware should be free (of course not), or that game developers should give the source of their game to game players so that they may fix bugs (of course not).
By providing source to their software, middleware manufacturers essentially give up their IP."
- Anonymous-1-Gamasutra
"... there are cases where it makes sense for a middleware company to release the source code. However, in those cases, the value of the middleware company is not in the source code."
- Anonymous-2-Gamasutra
As a consumer, I’d require source. Large games are too complicated and developed in too short a time to not be able to get into source. Additionally, final optimizations of games need to be able to optimize globally, across all source code.

As a middleware developer, giving away source and being exposed to thieves is difficult. People steal the source directly, indirectly, in whole, in part, in many ways. But, they do often face legal repercussions, and the rest is a cost of doing business and selling what people need. And, as pointed out in the comments, the real value of middleware is being able to use it, and that comes from the relationship with the company that supports it. Customers of Gamebryo get an excellent companion staff of engineers working with them to make their games succeed.

All that said, many have done great things with binary only copies of Gamebryo, for example check out this Coldwood Tech Demo.

There are other questions to ask about any piece of middleware: How much memory does it use? How much CPU time does it require? What’s the upgrade path for your current code and data? How does it interact with your other middleware? How good is the vendor’s support? How much does it cost?
Memory and CPU? Mostly that has to do with assets, and how customers configure them.

Upgrade path? Re-export your models, animations, particle systems from Max/Maya/XSI using the Gamebryo exporter. Some clean up will be required, and proprietary formats will take custom code to convert.

Interacting with other middleware is important! We specifically design for it in many sub-systems. E.g. Floodgate and how it shares resources on Playstation3. We also have a team of engineers assisting with and maintaining integrations with partner middleware companies.

Our support is widely acclaimed best in the business. I didn’t believe that statement until I heard it from current and previous customers over and over, studios that have used 2 or 3 of the major engines. I’ve never heard of a studio that used Gamebryo and anyone else who didn’t believe Gamebryo’s support was superior.

Gamebryo’s cost... ;) well... what are you developing? We have a portfolio of options. Talk to sales.

Motivation for Middleware

Kyle makes two strong points:

  1. Middleware provides you with more code than you could write yourself for a fraction of what it would cost you to try. No matter how clever you are, that’s how the economics of the situation work. ...

  2. Middleware offers structure. Middleware draws a line between the things that you have to worry about and the things you don’t. ... As games grow ever-larger and more complex it’s become incredibly valuable to be able to draw a line and say, The stuff on the other side of that line isn’t my responsibility, and I don’t have to worry about it.

I’m not going to rant (in this post) about why middleware makes sense. But, I am going to invite developers to do what they already do, better.

After shipping several console games, it became clear to me that the industry needed higher quality engine code, and that Middleware made sense. We’re hiring at Emergent, ramping up for a new wave of awesomeness, a new line of console hardware, and a better way to make games. If you’ve shipped games, drop me an email. Even if you just what to chat about what it is like to work on Middleware.

(Chrome engine picture used by flickr user lars hammar, Coldwood tech screen shot by Coldwood/Emergent)


  1. I would add too more very important features: integration with other middleware and tools.

    I would give Gamebryo a B for middleware integration... you guys do a good job of initial integration, but sometimes don't keep up with changes by the middleware companies.

    I would give Gamebryo a D for tools... your exporters from Max, Maya, etc are fine, but the rest of the tools have a lot of problems - most glaring is the non-standard camera that is difficult to use and even flips around as you change camera tools.

    As for all of the other items mentioned in the article, I would give you guys a B or better in all of those categories.

    Keep up the great work!

  2. Gamebyro doesn't provide a live readonly source branch. Getting updates or fixes are a crap shoot. We never know what you guys are working on so we have to write our own fixes. Animation code is sooo slow; code keeps rebuild matrixes. Lots of Fix Function and legacy code.

    I agree on the tools the Max exporters are good. The sceneviewer/animation viewer are lacking and need to combined.

    PS3 code barely functions. Xbox/PC code is very stable and works. Animation system being the problem area.

  3. Hello anonymous customer, you've got a range of points. Thanks for commenting. I moderate comments on the blog (after noticing spam getting through) but I've still posted your criticisms. I hope you've contacted support to discuss these issues, but let me respond publicly to some of these.

    Live source branch: No, we don't currently offer this. We offer major, and minor versions (e.g. 2.6) after robust internal testing. Incremental releases (e.g. 2.2.1) have limited changes, and selective testing. Patches are released frequently on an as needed basis, rolling up critical fixes and all bug fixes up until that time. Customers are delivered individual fixes for any priority items for them via support email.

    Roadmap, and knowing what we're working on: We have product managers who will respond to customers with our roadmaps. Before implementing a feature, we recommend customers write into support to see what Emergent may be doing with that feature, the time frame, and receive suggestions for how clients might best implement the feature.

    Animation code: Gamebryo 2.6 had a significant optimization pass, and we are happy to report improved memory and speed in our animation system. Regarding "code keeps rebuild matrixes", I'm not entirely sure what you mean there, it is easy to use the system in a non-optimal way. Again, contact support with your use case.

    Fixed function and legacy code: Yes, Gamebryo supports AAA games across PS3 and 360, down to much older fixed function PCs popular in Asia. Gamebryo has also been on the market over 10 years. We do regularly rearchitect and clean up legacy systems -- but that doesn't mean we gut everything. The most significant examples are the 2.0 release that required shader capable hardware which split from the 1.x version. Also, the 2.5 version that contains a complete geometry system rewrite. There are fast code paths that ignore older fixed function architectures, again, contact support with your particular use case.

    It's hard to offer a perfect product at a competitive price, but we do strive for that. In the end, I believe customers are receiving significant value for their dollar, and are saving years of redundant engineering time to have a solid starting point.