Analysis of NOVA -- RabbitMQ analysis of the NOVA source code

Recommended for you: Get network issues from WhatsUp Gold. Not end users.

This article is some information from my experience in the process of reading NOVA source code, RabbitMQ official documents and online summary and become, also to facilitate subsequent to this part of the review.

    NOVA is the core module of the OpenStack system, life cycle management, is mainly responsible for the virtual machine instances of network management (first version), storage volume management (in previous versions), user management and other related cloud platform management function, similar to the Amazon EC2 and Rackspace Cloud Servers in ability.

    Message queue(Queue)With the database(Database)As two important component of the overall architecture in Nova two, through the system internal message transfer and information sharing between tasks, module, interface between the asynchronous deployment, in terms of system greatly simplifies the operation process and mode of complex task, is the core module of the OpenStack Nova system. The end user (DevOps, Developers and other OpenStack components) interact with OpenStack system by Nova API, the Nova daemon through message queue and databases to exchange information to perform API requests, completed the terminal users of cloud service request.

    Nova uses no sharing, flexible architecture based on message, means that Nova components are installed in different ways, each Nova-Service module can be installed separately on a single server, can also according to business needs will be multiple modules installed on multiple servers.


    OpenStack Nova system mainly uses RabbitMQ as information exchange center.

    RabbitMQ is a framework for processing message authentication, message transfer and message routing, it coordinates communication between applications, and minimizes the mutual awareness between applications or software module, the effective realization of decoupling.

    RabbitMQ is suitable for deployment in a flexible topology easily expanded scale system environment, Ensure the timeliness of news communication between different modules, different nodes, and different processes, Cluster HA security characteristic of RabbitMQ can realize the information hub system level backup, At the same time, a single node with message recovery ability, When the system process crashes or node down, Message queuing does not RabbitMQ is processing loss, The node reboot recoverable communication in time according to the state data of message queue and message data.

    RabbitMQ excellent ability in the functional, timeliness, reliability and safety aspects of SLA can effectively support the OpenStack cloud platform scale deployment, flexible expansion, flexible architecture and information security demand.


    AMQP is an open standard protocol of application layer, and the design of message oriented middleware, where RabbitMQ is an open source implementation of AMQP protocol, OpenStack Nova each software module realizes the information communication through AMQP protocol. Routing protocol communication network design and data of the AMQP protocol is very similar, can be summarized as connectionless communication system model based on the state of the. The difference is, data communication network is determined between the client and server communication link based on link state, while AMQP is decided between producers and consumers of news news link state based on the message queue. For AMQP, The forwarding path state information decision system message queue of communication, The link between the two ends of the link is not special and permanent, But according to the status and attributes of the message queue implementation information stored in the RabbitMQ server and forwarding, The IP data as the data communication network packet forwarding mechanism, All of the routers is the status of the communication link and the formation of the routing table based on, IP packet according to local storage and the sequential packet forwarding routing table, The two have different approaches but equally satisfactory results and wonderful in implementation mechanism.

    The goal of AMQP is to realize the information communication of end to end, it involves two basic concepts: AMQP realize the communication factor is what is and what AMQP realize the communication entities and mechanism.

    AMQP is a method of communication between an application message oriented, that is to say, "the message" is the basic factor to realize the communication of AMQP. AMQP has two core elements -- exchanger (Exchange) and queue (Queue) via message forwarding mechanism to realize the binding and information communication. Among them, the exchanger is created by the consumer applications, and can be shared with other applications service, its function and data communication routers in the network are very similar, after receiving the message routing table will be accurate and safe forwarded to the corresponding message queue by. A RabbitMQ server or consisting of multiple RabbitMQ servers cluster there can be multiple switches, each exchanger identified by a unique Exchange ID.

    Exchanger according to different application requirements, in the life cycle is flexible, mainly divided into three kinds: persistent exchanger, temporary exchanger and automatically deleted exchanger. Persistent exchanger remain in the RabbitMQ server, and not because of the system to restart or termination of the application and the elimination of the related data, long-term resident in the hard disk; temporary exchanger reside in memory, and disappeared with the system to shut down automatically delete exchanger; suspension of the host application and automatic die, can release the server resources.

    The queue is created by the consumer applications, mainly used for the realization of store and forward exchange messages sent, queue also have life cycle properties of flexible configuration, persistent storage, can realize the queue resides and automatically delete temporary.

    Can be seen from above, the message queue, and switches are the three key components of AMQP, the effectiveness of any one component can lead to interrupt the communication of information, therefore in view of the importance of the three key components, the system at the same time to create three component will hit the "Durable" label, indicating that the recovery in business function immediately system restart.

    Above introduces three key elements of AMQP, then between them is how to work?

    From the chart we can see that the sending application, switch to receive messages, by setting the forwarding table and binding rules to forward the message to match the message queue, the receiving application message queue and then the received message is forwarded to the corresponding. Data communication network formed by IP address routing table to achieve the IP packet forwarding, There is a similar communication mechanism in AMQP environment, Exchanger through AMQP message header (Header) in the routing key (Routing Key) binding rules and the formation of the (Binding) to realize the message forwarding, That is to say, ""That bind message queue connection switch and routing table. The news producer sent messages in the Routing Key exchanger forwarding judgment factors, AMQP is the "IP address", after the switch from Routing Key trigger to get the message routing, bound by rules will message to the corresponding queue, message consumers finally gets the message from the queue. AMQP defines three different types of switches: broadcast type exchanger (Fanout Exchange), direct exchanger (Direct Exchange) and topic type exchanger (Topic Exchange), the three switches achieve binding rules are different.

The application of RabbitMQ in 3.Nova

Implementation of 3.1RabbitMQ in Nova

    RabbitMQ is the information center of the OpenStackNova system, the Nova in each module through the RabbitMQ server to the RPC (remote procedure call) to realize the communication way, and loose coupling relationship formed between each module, scalability, security and performance aspects of advantages. By above knowable, there are three types of AMQP switches: Direct, Fanout and Topic, and message queue by message consumers according to the functions and business needs its own generated.

    First, three important concepts:


    Accept the message and the message is forwarded to a queue. Within each find your host, exchanger has a unique mapping name. The application can create, delete, use and share examples within his scope of authority. Exchanger can be permanent, temporary or deleted automatically. Persistent switches has been deleted on the Server side until he displayed; temporary exchanger stop work when the server is shut down automatically delete the exchanger; when no applications using it was deleted by the server.


    "Message queue ", it is a named buffer, it represents a group of consumer applications save message. These applications can create, use, sharing the message queue in the scope of authority within their. Similar to the exchanger, message queuing can also be permanent, temporary or deleted automatically. Stop working temporarily message queue is closed in the server queue; automatically deleted when no application using it by the server to automatically delete. Message queue message stored in memory, hard disk or combination of both. Message queuing message saved, and send messages to one or more client, especially the message queue will trace message acquisition, news of it must be acquired, so can prevent multiple clients at the same time consuming the same message, can also be used to load balance among multiple single queue a consumer.


    Can be understood as a kind of relationship between the exchanger and message queue, binding after the switch will know of messages sent to the queue, binding key called binding_key. In the procedure that we use:


    Exchange and Queue binding can be a many to many relationship, each sending Exchange messages will have a routing_key keyword, switch to send messages to a particular queue, the queue and the switch binding_key and routing_key message must be matched to OK.


    Three types of introduce of RabbitMQ exchanger:

    Broadcast type exchanger type(fanout)

    This type of exchanger does not analyze the received message in the Routing Key, the default forwarding the message to all bundled with the exchanger of the queue to go. Radio switch forwarding efficiency is the highest, but the low security, consumer applications can access this does not belong to their own message.

    Broadcast exchanger is one type of the most simple, like we literally to understand, it is all the received message is broadcast to all it knows the queue to go, no matter what the message key, the message will be routed to and the switch queues bound to.

    The way it works as shown below:

    In the procedure that a radio switch code:



    Direct exchange type(direct)

    This type of exchanger required to exactly match the Routing Key and BindingKey, such as message Routing Key = Cloud, then the message can only be forwarded to Binding Key = the Cloud message queue. Direct forwarding efficiency exchanger, security is better, but the lack of flexibility, the system configuration of large amount of.

    Relative radio exchanger, direct heat exchanger can bring us more flexibility. Routing algorithm for direct heat exchanger is very simple -- a message routing_key perfect matching a queue in binding_key, the routing of messages to the queue. Binding of keywords will queue and exchanger bound together. When the news of routing_key and multiple binding keyword matching message may be sent to a queue.

    We passed below to illustrate the direct heat exchanger working way:

    Such as: Q1, Q2 two queue is bound to a direct heat exchanger X, Q1 binding_key is "orange", Q2 has two binding, a binding_key is black, another binding_key is green. In this relationship, with a "orange" routing_key messages to the X exchanger will be routed to the cohort of X Q1, a "black" or "green" routing_key messages to the X exchanger will be routed to the Q2. While all the other messages will be lost.


    The theme of exchanger(Topic Exchange)

    The exchanger through message Routing Key and Binding Key pattern matching, forwarding the message to all the binding rules in the queue. Binding Key supports wildcards, where "*" matches a phrase, "#" matching multiple phrases (including zero). For example, Binding Key="*.Cloud.#"Forwarding Routing Key= "OpenStack.Cloud.GD.GZ", "OpenStack.Cloud.Beijing" and "OpenStack.Cloud" message, But can't match for Routing Key= "Cloud.GZ" message.

    The routing_key can use a similar form of regular expressions, but the special character is only a "*" and "#", "*" represents a word, "#" on behalf of 0 or more words. That sent the message if the routing_key defines a queue rules, it will be forwarded to the queue.


    In the Nova main application of Direct and Topic two kinds of exchanger, In the process of system initialization, each module based on the Direct exchanger for each system message automatic generation of multiple queue into the RabbitMQ server, according to the characteristics of Direct switch requirements, "Binding Key= MSG-ID" message queue can only store and forward Routing Key= "MSG-ID" message. At the same time, each module as news consumers based on automatic generation of Topic exchanger two queue into the RabbitMQ server.

    Each module to realize the communication between Nova based on AMQP message, but the real implementation process messages areThe mechanism of RPC. Nova implementation based on RabbitMQThe two call to RPC: RPC.CALL and RPC.CAST, wherein the RPC.CALL request and response based on RPC.CAST mode, only provide one-way request, Two kinds of RPC call mode has different application scenarios in Nova.

    Each Nova module in logic function can be divided into two types: Invoker and Worker, Among themThe main function of the Invoker module is sent to the system request messages in the message queue, Such as Nova-API and Nova-Scheduler; the Worker module of the system is obtained from the Invoker module to send messages in the queue request message and response message to the Invoker module recovery system, Such as Nova-Compute, Nova-Volume and Nova-Network. Invoker through RPC.CALL and the RPC.CAST two process sending system request message; Worker receiving messages from the message queue, and responding to RPC.CALL. Communication between different types of exchangers and queue in Invoker, Worker and RabbitMQ as shown in Fig.

    Nova can be logically divided into two exchange domain according to the communication between Invoker and Worker: Topic exchange domain and Direct domain swapping, Between two exchange areas are not strictly divided, in the information communication process is the relationship between the depth of embedding. Topic news producer Topic exchange in the domain (Nova-API or Nova-Scheduler) is connected with the Topic exchanger generation logic, through the PRC.CALL or RPC.CAST process will request messages to the Topic exchanger. Topic switch according to the system request message Routing Key were sent to different message queuing and forwarding, If the message Routing Key="NODE-TYPE.NODE-ID", It will be forwarded to the queue to point the message; if the message Routing Key="NODE-TYPE", It will be forwarded to the message queues. Topic news consumers to detect new information has entered the response queue, the application immediately receive messages from the queue and calls for the implementation of the system message requested. Each Worker has two Topic message consumer applications, The corresponding point-to-point message queue and message queues, Remote call link point-to-point message queue Topic message consumer application receives RPC.CALL requests, And after the execution of related computing tasks will result in system response message back to the Direct news consumers through the Direct exchanger; at the same time, link sharing remote invocation Topic message consumer application message queue only receives RPC.CAST requests to perform computational tasks related to, And no response message feedback. Therefore, Direct transform is not independent operation, remote call flow and results but constrained to the Topic exchange in the RPC.CALL domain, each RPC.CALL to activate a Direct message exchange operation, the message will generate a corresponding set of message queue and exchanger combined response for each system. Therefore, the OpenStack cloud platform scale, performance bottleneck of Direct switched domain for message handling large and the formation of the whole system.

3.2Nova RPC.CALL and RPC.CAST call flow

    As can be seen from the above, RPC.CALL is a two-way communication process, namely the Worker receiving system message producers generated request message, message consumers after treatment of the system corresponding results feedback to the Invoker program.

    For example, A user to NOVA-API by sending the external system would begin "virtual machine" demand, The NOVA-API as a news producer, The message is packaged for AMQP information to RPC.CALL through the Topic exchanger forwarding to queues on the news, At this time, Nova-Compute as news consumers, Receiving the information and the implementation of the corresponding virtual machine through the underlying virtualization software start-up process; after the successful launch of the user virtual machine, Nova-Compute as a news producer by Direct exchanger and response message queue "virtual machine start-up success" response message back to the Nova-API, At this time Nova-API as news consumers receive the message and notify the user virtual machine start-up success, A virtual machine complete startup RPC.CALL calls the end of the process. The specific process as shown:

    (1)Invoker generates a Topic message producers and a Direct message consumers. Among them, Topic news producer sending system request message to the Topic exchanger; Direct news consumers wait for the response message.
    (2)The Topic exchanger message according to the Routing Key message forwarding, Topic consumers receiving messages from the message queue, and pass to perform tasks related to Worker.

    (3)Worker according to the request message after the execution of tasks, assign a Direct producer, Direct news producer sends a response message to the Direct exchanger.

    (4)Direct switch according to the Routing Key response message is forwarded to the corresponding message queue, Direct consumers receive and pass it to the Invoker.

    RPC.CAST remote invocation process similar to RPC.CALL, but the lack of a system message response process. A Topic message producers sending system request message to the Topic switch, Topic switch will forward the message to a shared message queuing message according to the Routing Key, and shared by all Topic consumer message queue is receiving the system request message, and pass it to the response of Worker processing, the call flow as shown in Fig.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download

Posted by Gloria at January 08, 2014 - 4:19 AM