The Mali GPU, and ARM's 5-year strategy to dominate visual computing

ARM has established a position of dominance in mobile application processors, as the provider of CPU (central processing unit) core IP (intellectual property) to the likes of Qualcomm (Snapdragon), Texas Instruments (OMAP), Samsung (Hummingbird), NVIDIA (Tegra), and others. The company is now building on a strategy to do with GPUs (graphics processing units) what they have accomplished with CPUs. Tom Olson - Director of Graphics Research, and Jem Davies - ARM Fellow and VP of Technology in the Media Processing Division at ARM, spoke with the EE Daily News about the company's strategy to address the growing demand for high-performance visual computing.

ARM's venture into GPUs began five years ago, when they acquired Falanx - a 20-person startup based in Norway.  One year later, ARM formed the MPD (Media Processing Division) to address audio, video, and graphics as well as visual computing applications. ARM MPD has a diverse set of customers, including Cisco, and Netlogic in the networking space, ST Micro in STBs (set-top boxes), and mobile players that include ST-Ericsson, Samsung, Broadcom and LG. Samsung employs the Mali GPU in the forthcoming Galaxy S2, along with a dual-core Cortex A9 CPU.

Tom Olson says that the two critical factors that drive graphics in mobile devices are gaming and the UI (user interface). In the next five years ARM expects UI to move to gesture input and 3D interfaces (see Live from IDF10 - demo of gesture UI from GestureTek). Visual computing is not just about displaying images, says Olson, but also about taking images in and processing them, with much more complex use of images in the future. Emerging mobile applications such as AR (augmented reality), plenoptics - fixing out-of-focus images, and computational photography will drive requirements for more powerful GPUs.

Olson is also chair of the OpenGL® ES (embedded systems) committee. (OpenGL® ES is "a royalty-free, cross-platform API for full-function 2D and 3D graphics on embedded systems - including consoles, phones, appliances and vehicles").  The committee continues to work to develop standards to support the convergence of  graphics requirements, where smartphones now have the same capability as notebook PCs of just a few years ago.  Games continue to be the dominant category of mobile applications, and as handset screen resolutions have increased consumers expect their smartphone or tablet experience to be PC-like. The challenge, says Olson, is no longer how to integrate enough transistors to do the job, but how to manage the power dissipation within the energy budget that a battery can support.

Content remains king in the media industry, and the competition in Hollywood to produce the next big CG-driven (computer-graphics) blockbuster is very expensive. ARM cites the example of the latest "Pirates of the Caribbean" sequel, which cost $300 million to produce.  This has driven content owners and distributors towards multi-screen strategies, in order to leverage their investments beyond the cinema to home video and mobile devices.  At the same time, consumers are demanding the ability to place-shift their consumption of content anywhere they go, in the car, on smartphones and tablets. ARM wants to be the provider of a common scalable platform that will enable developers to deliver the rich HD (high-definition) media experience that consumers have come to expect across all CE (consumer electronics) devices.

Jem Davies says that the answer to meeting the challenge for scalable HD applications is the use of multi-core architectures.  SoC designers can utilize multiple ARM CPU and GPU cores to meet the peak requirements of an intensive video game, but turn off cores to save power for less intensive tasks such as reading email. Having all this fine-grained hardware control available can increase the complexity of the software required to manage it, but Olson says that ARM designed the Mali-400 MP, the first multicore GPU for mobile, to hide the power scaling functions from the application layer. The software driver, which you would typically run on your ARM CPU core, handles the distribution of tasks transparently. In the next-generation Mali-T604, ARM's GPU contains a hardware job manager that abstracts the power management task even further. Now, says Olson, the OS (operating system) can turn off a core in the middle of processing a graphics frame, and the job manager will handle the request without degrading graphic performance.

ARM sees themselves as ideally positioned to offer a complete optimized graphics sub-system,
integrating each of the four components necessary for high-performance visual computing.

Four components of a Graphics Sub-System
Designers and IP providers are increasingly taking a sub-system approach to SoC design. For graphics sub-systems, designers must optimize CPU-GPU interaction, and the associated memory bandwidth requirements, along with the interconnect fabric that ties the SoC together. As a provider of each of these SIP (silicon IP) components,  ARM's strategy is to leverage their combined strengths to make Mali cores as dominant in visual computing as the company's CPU cores are in the application processor market.

Davies and Olson see the ability to co-develop the GPU and CPU as absolutely necessary to visual computing subsystems. "You must be able to see the whole system", says Olson. Companies that only develop GPUs, will find that they are only addressing part of the problem, says Davies. One trend, that ARM sees developing more over the next few years, is the need to for tight integration of the CPU and GPU in order to work in tandem on the same computing task.

The GPU business is still in the early stage of development for ARM, but the company claims that Mali is already the SIP industry's most widely licensed GPU. ARM added 7 Mali licensees in Q1 of this year, for a total of 46, after adding 11 in all of 2010. Of the 46 Mali licensees, just 6 are in production and producing royalties for ARM, indicating that the business is just ramping up.