Android 4.4 (KitKat) design pattern in -Graphics subsystem

Recommended for you: Get network issues from WhatsUp Gold. Not end users.

The original address:

This paper simply from the perspective of design model to Kan Android4.4 (KitKat) Graphics system. You can see the Google in the KitKat to code or sorting, such as replacing SurfaceTexture like this first see what things, remove the ISurface like this play soy sauce definition, get rid of it is SurfaceHolder type skin white variable called surface. Since the concept of modified these obscure fate, my mother will not worry about I cannot read Android code. Of course, there are still a lot of legacy code, if you haven't seen the previous version of the words can be a little confused, but not hurt the important Essentials. Next, hark back to the subject. As an operating system, Android needs to take into account the flexibility, compatibility, usability, maintainability and other aspects, in order to achieve these requirements, it needs good design. Therefore, in the Android source code can be seen in many design patterns figure. Light is the Graphics system referred in this paper, on the use of such as Observer, Proxy, Singleton, Command, Decorator, Strategy, Adapter, Iterator and Simple Factory model. If you want to learn design patterns, I think Android source code is a good example of teaching materials. Of course, a lot of usage and the GoF of the classic example is not the same, but the idea is to learn value. The whole process as the key link, to get in with the corresponding design pattern. It covers the work principle of Android, and a glimpse of the design concept, shoot two hawks with one arrow. Here the intention is to top-down expansion, because Android source code is huge, very easy to get lost in the sea of code. The spirit of the principle of divide-and-conquer, this paper first introduces SurfaceFlinger's server processes, and for the App UI part of the application is left to say.

In order to make the analysis is not too abstract, first of all, let us first to find a running example as a starting point, makes the analysis more true to life. This election is/frameworks/native/services/surfaceflinger/tests/resize/resize.cpp. Select the test case for a number of reasons: first, it is a Native program, don't involve them in the Java layer that a proxy and Jni calls. Two, small but complete., it has the main application in Native layer. Three, no nonsense, simple but not simple, high-end fashion grade. Note that this case default compilation is wrong, but fortunately for the code farmers is not a big problem, redirect redirection is passed.

Below are a few lines of the core in this use case, we to have a look this simple task behind Android do what.

39    sp<SurfaceComposerClient> client = new SurfaceComposerClient();
41    sp<SurfaceControl> surfaceControl = client->createSurface(String8("resize"),
42            160, 240, PIXEL_FORMAT_RGB_565, 0);
44    sp<Surface> surface = surfaceControl->getSurface();
46    SurfaceComposerClient::openGlobalTransaction();
47    surfaceControl->setLayer(100000);
48    SurfaceComposerClient::closeGlobalTransaction();
50    ANativeWindow_Buffer outBuffer;
51    surface->lock(&outBuffer, NULL);
52    ssize_t bpr = outBuffer.stride * bytesPerPixel(outBuffer.format);
53    android_memset16((uint16_t*)outBuffer.bits, 0xF800, bpr*outBuffer.height);
54    surface->unlockAndPost();

Here probably the process is to first create a SurfaceComposerClient, Then it creates SurfaceControl, Get Surface, Then through the lock () assigned graphics buffer, Then the rendering of the things (here is the simple red) after a good painting, Using unlockAndPost (SurfaceFlinger) to go into the hardware buffer, It is painted to the screen. The end result is the screen should be able to see a small red block. Here is in no hurry to dig the code, to slow down and intuitive understanding of these concepts. SurfaceComposer can be understood as SurfaceFlinger's nickname, because SurfaceFlinger is mainly used to make each layer in Composition operation. But the SurfaceFlinger is in service, as an application to make it serve to Mr. into its corresponding client, also known as SurfaceComposerClient, because SurfaceComposerClient is to understand and deal with the protocol server. SurfaceComposerClient is located on the server proxy called Client, a remote procedure call applications through the former issue is through which to achieve. With it, the application can use it to apply for SurfaceFlinger Surface created for himself, the surface of this Surface is to draw the abstract. The removal of minor elements, below brief outline structure between these classes:

These structures drawn, beautiful and symmetry of the C/S architecture is evident on the computer. The middle interface layer, on both sides of the server and the client (i.e., applications can communicate through the interface). So both sides can do the design of the "interface oriented programming". As long as the interface unchanged, both sides realize how frustrating. As you can see, the application to the SurfaceFlinger will own content rendered to the screen is a client requests a service process. Where the application is the producer, provides a graphics buffer to rendering, SurfaceFlinger for the consumer, with the graphics buffer to merge rendering. From a structural point of view, you can see the object application end things are corresponding to the server. Note that the generated sequence of them is from the top to the bottom, namely for the application, with ComposerService, and SurfaceComposerClient, and Surface; for the server, followed by SurfaceFlinger, Client and Layer.

Here at least see two:Singleton design pattern and Proxy pattern. Of course, usage may and textbooks are not the same, but the idea is the same. ComposerService is used to abstract SurfaceFlinger, SurfaceFlinger global only one, so the ComposerService is the Singleton object. On the other hand, the application wants to let the server to do things, but the server is not in the same process, which requires the server to create local objects in the server agent (such as Client), which is Proxy mode. Proxy model application is very wide, can be used for remote object access (remote proxy), Virtualization (virtual proxy), access control (protection proxy) and smart pointers (smart reference proxy) etc. Here is remote proxy (of course there are factors of protection proxy). In addition smart reference proxy model examples such as smart pointer in Android.

General structure finished, below about the process step by step. First of all, sequence chart below to create SurfaceComposerClient:

No bright spot, with a pen. Before the play ended, below to get down to business, the application creates a Surface process is as follows:

The process is intuitive, details omitted. Note there are a few points:

A, client to server calls are carried out by Binder. If you don't know what Binder is also never mind, just take it as a system level oriented IPC mechanism. The above between the client and server many inter process procedure call is to use it to complete the.

Two, in reality, a call SurfaceFlinger createLayer () function which is not so directly, instead of using asynchronous transfer mode. The moment you.

Three, IGraphicBufferProducer, IGraphicBufferConsumer, BufferQueue, the SurfaceTextureLayer is come down in one continuous line. So the figure when the server creates a SurfaceTextureLayer object, to the client was transferred into the IGraphicBufferProducer is completely OK. There are many such usage oriented programming interface, its theoretical foundation is a substitution principle. It also reflects the interface segregation principle, the same objects on the client only exposed as producer and consumer interface, the interface exposed on the server, avoiding the interface pollution.

Then the above process uses which design pattern? I think the more obvious are the following:

Proxy model was used to lift a circular reference here and avoid reclaims the object. ConsumerBase to create a type of BufferQueue:: ConsumerListerner object listerner, Then the bread layer type for QueueBuffer:: ProxyConsumerListener proxy proxy object, It has a weak pointer pointing to listerner (due to weak reference does not affect recovery). On the one hand, and the agent is equivalent to the strong reference into a one-way strong reference, do so in order to avoid circular references between objects lead to recovery. On the other hand, if the strong pointer, the ConsumerBase constructor, once the function completes, the ConsumerListener (ConsumerBase) strong reference is not, then onLastStrongRef () might be called, thus recovering the object and use. We see the application of Proxy mode in a remote object access in, here we see another usage.

Observer model is used as a notification mechanism in the application when rendered submitted to the server. The first consumerConnect (BufferQueue) will be established by reference to the ConsumerBase object, On the member variable mConsumerListener, Then setFrameAvailableListener () to establish ConsumerBase object reference to Layer. Thus the following such a chain of Observer model:

Attention is the chain, and Chain of Responsiblity mode but nothing. Following the completion of registration, Later when the application finishes calls queueBuffer graphics buffers (when), The callback function onFrameAvailable BufferQueue will call the listener(), Then through the onFrameAvailable ConsumerBase calls the Layer callback function(), Finally, call signalLayerUpdate () SurfaceFlinger get notice. This enables SurfaceFlinger to concentrate on doing their thing, when there is a need to be informed of the nature. In fact, when be informed of new frames need to render time, SurfaceFlinger also not immediately stopped, but first do a lightweight, also is the corresponding message in the message queue for later processing. This makes all the applications and SurfaceFlinger can work asynchronously, ensure that the SurfaceFlinger performance is not affected. And it introduces the following Command model.

The Command mode is used to make SurfaceFlinger and other modules of asynchronous work. The Command model is often used to achieve undo operation or delay in processing, it is with the latter. Simply summarized is: when a message (such as INVALIDATE), to put it through the MessageQueue store (actually exist in Looper), then SurfaceFlinger thread will continue to search the message queue. If the queue is not empty and agreed to deal with the time has come, will remove the corresponding treatment. The specific process behind when it comes to SurfaceFlinger. Why bother it, mainly to ensure the stability and efficiency of SurfaceFlinger. The news can come in, but SurfaceFlinger can be processed asynchronously. This will ensure that the SurfaceFlinger does not disrupt the swing rhythm, so as to ensure that the user experience.

Create the different functions of the BufferQueue category is the use of like design pattern in Builder. When the BufferQueue is created, it has a default value, the client to make it into the BufferQueue you want, set parameters to a series of through its interface, like Wizard. Why would I want to do this, first understand the background knowledge. The way a graphics buffer from the application to the screen in two places used in the BufferQueue. An application for the good render graphics buffer passed to SurfaceFlinger, another to the synthesis of SurfaceFlinger good graphics buffer into the hardware graphics buffer.

The core data in BufferQueue is a GraphicBuffer queue. While the GraphicBuffer according to the use of different occasions from shared memory (i.e. Ashmem, because the memory to be shared in the application and the server process between two processes or hardware) from the graphics buffer (Framebuffer, SurfaceFlinger because it is rendered to the screen) distribution. In addition because of different uses, its type, size, and number in BufferQueue may be different. Although it is a class for different occasions, the origin is different, how to distinguish between what is tall, rich and handsome, what is the short poor contusion. Is very simple, when the BufferQueue is created, by Layer or FramebufferSurface to act as a director's role, make the corresponding BufferQueue. They are calling a series of functions (such as setConsumerUsageBits) and setDefaultMaxBufferCount (), () will build out of BufferQueue become applicable to their.

In addition, Adapter and Decorator in the code also often appear. From the objective aspect, the callee provides interfaces or function often can not meet the caller needs, if the interface does not meet with Adapter mode, if you want to add additional features to use Decorator mode. Structurally, Adapter is the Subclassing structure can also be Composition structure, while Decorator is generally Composition structure. In fact these two patterns are often mixed together with. In practice, I feel no need to care too much about what mode, can play a role. For example, SurfaceFlingerConsumer is the GLConsumer Wrapper, When the Layer SurfaceFlingerConsumer call interface, The corresponding implementation using GLConsumer (duplicate code in fact SurfaceFlingerConsumer and GLConsumer Implementation). I think it is used subclassing structure to achieve the kind of like Decorator model. Of course, this mode is not used very clear, just looking down mode, skip or examples.

Back to the main line of our test cases, Surface applications created the following lines, main function is to Z axis value painted the application layer; set so large, is also very cow can certainly see the place.

46    SurfaceComposerClient::openGlobalTransaction();
47    surfaceControl->setLayer(100000);
48    SurfaceComposerClient::closeGlobalTransaction();

As this change the screen or attributes of an application window action need to use openGlobalTransaction () and closeGlobalTransaction (package), so the middle operation change becomes a transaction. In the affairs of operating only in the local, only when the closeGlobalTransaction () is called together through Binder to SurfaceFlinger. This is mainly from another Singleton object of Compoesr to achieve. The property change transaction optimization of system resources, because these properties to change operation is often heavy, means a lot of things need to be calculated, so here these time-consuming operation played a package, to avoid duplication of work. Since this is not complex is not the focus, but also part of the process and the repeated, so skip directly into the climax - write a graphics buffer and the SurfaceFlinger synthesis of output to the screen. Let us have a look the following lines behind what they do:

50    ANativeWindow_Buffer outBuffer;
51    surface->lock(&outBuffer, NULL);
52    ssize_t bpr = outBuffer.stride * bytesPerPixel(outBuffer.format);
53    android_memset16((uint16_t*)outBuffer.bits, 0xF800, bpr*outBuffer.height);
54    surface->unlockAndPost();

The code first defines the graphics buffer, it is the application used to buffer the drawing. But there's only the meta information, put the drawing data where not allocated. Look at the following two line drawing buffer statements, their purpose is very simple, is to make the graphics buffer into a red. Note that the Surface type is PIXEL_FORMAT_RGB_565, so the red corresponds to 0xF800.

52    ssize_t bpr = outBuffer.stride * bytesPerPixel(outBuffer.format);
53    android_memset16((uint16_t*)outBuffer.bits, 0xF800, bpr*outBuffer.height);

Draw lock with buffer (unlockAndPost) and after () wrapped up, the two function is mainly used to the server for the graphics buffer, and then draw the graphics buffer well to SurfaceFlinger treatment. The whole process is as follows:

This application is the graphics buffer rendering itself good magnificent over to the SurfaceFlinger. One of the most important is to transfer the graphics buffer, the corresponding graphics buffer class for the GraphicBuffer. BufferQueue buffer queue it, can be regarded as an element for the GraphicBuffer queue. Under the background of knowledge spot, BufferQueue GraphicBuffer has the following several States, the relationship between conversion:

This part of the design patterns used mainly in the following:

Memento mode. From BufferQueue to GraphicBuffer is in the shared memory allocation, and the memory is represented with handle. We know that a process handle or address to another process is not legitimate. So why is Surface lock () function to get the GraphicBuffer directly when the local objects. Here is the introduction of Memento model. GraphicBuffer inherits the Flattenable class, the flatten () and unflatten function. When the GraphicBuffer is from the process of A via the Binder passed to the process B, first in the A in the process of its flatten () function is called, and then unflatten in the process of B () function is called. This would ensure effective still cross process after GraphicBuffer. See flatten () and unflatten (briefly) implementation. The server in the transmission of GraphicBuffer before calling flatten () function to existing structures GraphicBuffer basic information, Application received after the call to unflatten (end) will take out this structure after the call to GraphicBufferMapper:: registerBuffer () (it will then call the gralloc module in gralloc_register_buffer(), And then call gralloc_map(), Finally using MMAP ()) will be mapped distal GraphicBuffer is mapped into the local address space, Then in the BpGraphicBufferProducer:: requestBuffer () to construct a GraphicBuffer. So the application client and server in the GraphicBuffer is mapped to the same physical space, achieve a shared objective, but also for the application of this transformation process is completely transparent. Treatment to deal with GraphicBuffer QueueBufferOutput and QueueBufferInput are the same, passing between processes, the difference lies in the former two equivalent in another process a copy. Because they only contain a small amount of POD members, so the copy has little effect on the performance of.

Iterator model in the Android source code is also very scattered used. Such as: lock (Surface:), in order to make frontBuffer reuse (graphics rendering generally use double buffering, frontBuffer is used to display the output, backBuffer is used for mapping, so that both can be done at the same time do not influence each other. Most of the time in the next and previous frames than only a small is the need to re render, such as cutting the fruits many times in fruit around a region need to re render, this area is the dirty region), we need to know the dirty region. Dirty region information expressed in Region. And Region itself is composed of a plurality of rectangular, and as a client to access the Region of these rectangular, does not need to know their internal implementation. So you can write in the access to this dirty area:

Region::const_iterator head(reg.begin());
Region::const_iterator tail(reg.end());
while (head != tail) {


This client code is not dependent on the implementation of Region, whether Region is an array or a queue, or the upper, code unchanged, very beautiful. Similarly there are: queueBuffer (BufferQueue:) used in Fifo:: iterator.

Strategy mode is used to make the system can gracefully adapt to different HAL modules in the platform. Although here the part of the code is written in C, and the standard Strategy with slightly different, but the spirit is the same. For example, the gralloc module is platform dependent, in different platform has different implementation. As the client, such as GraphicBufferMapper, it does not want to rely on this change. So all the gralloc module platform provides uniform interface, and unified the interface object symbols corresponding to HAL_MODULE_INFO_SYM. So no matter what platform, the client should use dlopen () and dlsym () to load and query this structure can be realized by the corresponding interface (see hw_get_module loading process() ->hw_get_module_by_class() -> load()). Here refers to the gralloc_module_t interface, GraphicBufferMapper interface is used, the platform provides the gralloc must implement this interface. Similarly there are the hwcomposer module in HAL layer processing.

In addition, we also saw the design pattern mentioned again and again to appear, such as death notice is the Binder Observer mode based on GraphicBufferMapper, in addition to the gralloc module gralloc_module_t package can be viewed as the application of Adapter model.

To return to the main line, in front of the process is to call signalLayerUpdate (Layer) SurfaceFlinger on the end of the notice. The analysis of SurfaceFlinger process and subsequent processing for the application of the graphics buffer process. In order to make it look more complete, we start from the SurfaceFlinger initialization, and then went on to tell the story after the signalLayerUpdate (). Here begins with the creation of SurfaceFlinger. The original SurfaceFlinger two startup mode, initiated by the SystemServer thread, or was initiated by the init process. But Android 4.4 as the former removed, anyway I am not found. . . . The story is told from main_surfaceflinger.cpp.

The upper half is the initialization of SurfaceFlinger, the lower half part is SurfaceFlinger processing for VSync signal, which is a combined rendering process. Since the code in many branches, in order to explain simple, there were some assumptions: first assume platform supports hardware VSync signal, so as not to create a VSyncThread to simulate by software. Also assume that INVALIDATE_ON_VSYNC is 1, which is the INVALIDATE operation in the VSync signal to come. valueNote that the SurfaceFlinger threading model compared with the prior version had bigger change, the main reason is the virtualization introduces VSync. The thread is EventThread (vsyncSrc), EventThread (sfVsyncSrc), EventControlThread and DispSyncThread. This thing more fun, so let alone an article speaking().

The basic function of several important classes of roughly introduced:

EventControlThread is a simple other heinous thread, used to mark bit switch hardware VSync signal. But for the hair to separate into threads, it is expensive to switch hardware VSync signal?

RenderEngine is a EGL and GL function package. It may be that introduced EGL and GL function in other module looks too messy. The GL related stuff packaging up, only to show GL independent interface, so that the SurfaceFlinger module as the client does not need to understand the underlying details.

Looper is a generic class, will be responsible for event and message into a queue, and provide a polling interface processing in the event queue and message. Here is mainly used for processing TRANSACTION, INVALIDATE and REFRESH news and VSync events.

Main operating Looper MessageQueue. Because Looper is not specific to SurfaceFlinger, MessageQueue encapsulates some SurfaceFlinger specific information, which makes the EventThread and SurfaceFlinger modules can use it to direct the use of the Looper class.

SurfaceFlinger, DisplayDevice, The relationship between the FramebufferSurface and HWComposer comparison. SurfaceFlinger is the largest client. Abstract DisplayDevice display equipment, packaging for rendering Surface and HWComposer module, which makes SurfaceFlinger as long as possible to deal with it. Abstract FramebufferSurface SurfaceFlinger used to drawing Surface, the Surface corresponds to a BufferQueue for multi render buffer. Surface the difference between it and the previously mentioned end use is that it is based on hardware graphics buffer, not Ashmem. HWComposer package of two important modules of HAL, namely framebuffer and hwcomposer, the former corresponding hardware buffer, which corresponds to the hwcomposer hardware. Hwcomposer VSync control signal screen information, management and operation of the framebuffer. HWComposer is responsible for the main HAL module loading platform dependent, and to use the HAL module through the upper module. Note two easily confused concepts: HWComposer is a class, there will certainly be; hwcomposer is the hardware module, on some platforms may not available. The class is structured as follows:

SurfaceFlinger rendering work is mainly driven by the VSync signal. EventThread is responsible for issuing the virtual VSync signal (VSync signal by hardware virtualization get offset specified phase). The initialization, MessageQueue in the setEventThread () function to establish a connection with the EventThread, and then the communication between socket and EventThread (BitTube) handle registered into the Looper, also registered its own callback function. On the other hand, the SurfaceFlinger will pass Looper constantly polling the handle, the handle has no data. When receiving data on the handle, the callback function will call MessageQueue corresponding. After treatment, the last MessageQueue Handler received the message call to the corresponding processing function based on SurfaceFlinger. Thus, EventThread-> Looper-> MessageQueue-> SurfaceFlinger event message transfer process.

Here again see the familiar design pattern, If the Iterator mode is used to iterate through all the layers(HWComposer::LayerListIterator), Observer model for virtual VSync signal applied to the DispSyncThread virtual VSync event thread(DispSyncThread::EventListener), And the previously mentioned Command mode for delay in processing the message(postMessageAsync()).

In addition to these familiar figure, we can see some new faces. Such as: create (RenderEngine:) using a simple factory pattern, which according to the GLES version number to create the corresponding RenderEngine. In this way, as the RenderEngine user SurfaceFlinger and Layer became Strategy mode of the beneficiary, they don't care about RenderEngine implementations in various differences between different versions. And Mediator mode application in Service management. In the process of SurfaceFlinger, addService () function registers the SurfaceFlinger service to Service Manager. As the intermediary role that Service Manager communication between Client and Service, which makes the module structure of a network into a beautiful star structure. Of course, here as a service to look not to come out, so the other chapters tell.

As you can see, when VSync signal arrives, SurfaceFlinger is mainly to do with rendering and output works by processing the INVALIDATE and REFRESH messages. The core idea here is less able to deal with a little, so there are many dirty work area before rendering, such as processing behind those regions of the update can. It is of practical significance, on the one hand to cover the inter layer because, some invisible layer does not need to render. On the other hand, because of our application frame is generally only a small part of the variation, if each frame all estimators of people to see spit. Here is the call to this function:

handleMessageTransaction()The main treatment before the screen and the application window changes. Because the visible region of these changes are likely to change the layer, and then calculating the impact of the dirty region.

handleMessageInvalidate()The main call handlePageFlip () function. Here Page Flip refers to the figure of a buffer zone from the BufferQueue, as a "flip". The main function is to get the graphics buffer from the Layer corresponding to the BufferQueue data, and update the dirty region according to the content.

handleMessageRefresh()Is that the merge and render output. As important, this step more, a general framework as follows:

The first test case main part behind the story about so much. Space is limited, to omit many details. We can see, do so many things at the server, and for the application as long as the first lock (buffer), fill again, finally unlockAndPost (OK). This is also reflected in Facade mode.

In general, in the Android source code we can review to the basic principles of the application of design patterns: one is the only in the appropriate place. Many times we learn design patterns is to need not, precisely, not abuse. Two is to use the words do not need to rigidly adhere to the original or classical usage, in order to solve the problem for the purpose of proper free play.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download

Posted by Stephen at May 16, 2014 - 2:10 AM