© Copyright IBM Corporation 2008, 2010
ALL RIGHTS RESERVED
The Really Small Message Broker is a small program that sends and receives data in the form of messages to and from applications and devices over TCP/IP network connections. Data from a range of sources, including other applications, other brokers, or sensors and meters that measure their physical environment, is published to the broker. The broker then sends a copy of that data to applications that have registered their interest in that data, by subscribing to it.
The Really Small Message Broker takes up only 50KB of storage space and can run in only 150KB or less of memory (actual figures vary), making it ideal for installing and running on small servers and embedded devices, such as on the Linksys NSLU2 or other low-powered servers.
A useful configuration of a broker is to install it on an internet-connected embedded device in a house, where it can obtain data about the house's electricity readings, for example, and publish them to another, external, broker. The external broker can collect the electricity readings from several houses and publish the data so that applications that produce visualizations of the data can subscribe to receive the data as it is published. You can see an example of such output at Current Cost electricity usage graphs.
You can try out the Really Small Message Broker by installing it on your computer, then sending test messages to the broker. The broker will then publish the messages and you can subscribe to the messages.
The Really Small Message Broker is a server implementation of the MQ Telemetry Transport (MQTT) protocol (version 3). Any client that implements this protocol properly can use this server for sending and receiving messages.
Two IBM products that also implement this protocol are WebSphere™ Message Broker and Lotus™ Expeditor micro broker. Really Small Message Broker can connect to both of these products using its bridge (see below), allowing messages to be forwarded to and from enterprise messaging systems as well as MQTT clients connected directly to it. Lotus Expeditor micro broker is a Java™ server which provides Java Messaging Service (JMS) connectivity as well as MQTT. It is available together with many other components as Lotus Expeditor.
For more information about MQTT, WebSphere Message Broker, and Lotus Expeditor micro broker see the MQTT.org website.
Alternatively, if several types of readings were obtained from each weather station (such as temperature, wind direction, pollen count), the topic names could be constructed as weather/london/temperature, weather/london/wind, weather/london/pollen, weather/southampton/temperature, weather/southampton/wind, and so on. Structuring the topic hierarchies by location is useful if the applications are likely to want to obtain all the weather data about a particular location.
A web site that displays the current weather conditions in London, could specifically subscribe to each of the three topics that relate to London. More flexibly, though, the web site could opt to receive any information about the weather in London by using a wildcard character to subscribe to all three topics (and any others that are subsequently added): weather/london/#
Alternatively, an application (another website, for instance) could opt to receive just the temperature readings for each of the three locations by using a different wildcard character: weather/+/temperature; or to receive any weather data from any location by using a wildcard character to specify a more general level in the topic hierarchy: weather/#
You can download and install the broker and explore the capabilities of publish/subscribe messaging,in particular on small or embedded devices.
Really Small Message Broker runs on the following platforms:
To install a broker:
C:\Program Files\broker
or /home/laura/broker
You are now ready to run the broker.
To start the broker on Windows:
To start the broker on Linux:
cd /home/laura/broker
./broker.ubuntu
nohup ./broker.ubuntu >> /dev/null &
The broker is now ready to publish messages.
To stop the broker cleanly, type Ctrl-C. On Linux, if you prefer, you can just kill the process of the running broker:
ps -ef | grep broker kill -term process_id_of_the_running_broker
When broker is used in an application integration environment, messages are sent to it by client applications; other client applications subscribe to receive the messages. The client applications can be written in any language, but must use the MQTT protocol to communicate with the broker.
Sample client applications in Java and in C are available to download from the MQTT.org website. To send and receive test messages with the broker, you will use the sample Java client application, known as WMQTT Utility.
stdoutsub sampleassuming that the broker is running on the same computer, listening on port 1883. If the computer and port are different to this, then use the following command:
stdoutsub sample --host 255.255.255.255 --port 1884using the appropriate IP address and port number. On Linux you may need to tell the OS to look in the current directory for the MQTT library, like so:
LD_LIBRARY_PATH=. stdoutsub sample --host 255.255.255.255 --port 1884
stdinpub sampleadding the host and port options if necessary, as shown in step 2.
java -jar wmqttSample.jarYou will use this instance of WMQTT Utility as a subscriber application to the messages you will send.
java -jar wmqttSample.jarYou will use this instance of WMQTT Utility as a publisher application to send messages to the broker.
The message that you entered is published to the broker on the testtopic topic. The broker then sends the message to the subscriber instance of WMQTT Utility which subscribed to the testtopic topic.
Retained messages are also known as last known good values. This name gives a hint of how they are intended to be used. If the value for some quantity (such as temperature) is published on a topic each time that value changes, it might be a long time between updates. When a program subscribes to that topic, it might have to wait a long time before receiving its first message. A retained message on that topic ensures that the client receives the last published value as soon as it subscribes. This value might be used, for example, to put an initial value on a visual display.
The message that you published to the testtopic topic, is not retained by the broker. When the broker received the message that you published, the broker simply sent the message to any connected applications that were already subscribed to the topic. Any applications that were not subscribed to the topic did not receive the message, even if they subsequently subscribed to the topic. This is because the message was not marked as retained by the publisher application.
If a publisher sets the retain
flag in a message, the
broker retains the message so that applications subsequently
subscribing to that topic
can receive the retained message immediately. The retained message is
kept by the broker until the next retained message published on that
topic is received by the broker.
You can try this out using WMQTT Utility:
By default, the Really Small Message Broker will not save any retained messages or
subscriptions when it stops, so both retained messsages for all topics and durable
subscriptions for all
clients will be lost. To preserve retained messages and durable subscriptions while a broker restarts,
you can specify the persistence
and persistence_location
settings in the configuration file.
The autosave
settings (autosave_interval and autosave_on_changes) specify that the
broker will save retained messages and durable subscriptions periodically while it is running.
This provides some protection against power failure or program crashes.
On Linux, sending a HUP signal (for example, kill -hup process_id_of_the_running_broker
) to the broker will cause it to save retained message and subscription state immediately if there are any changes since the last save.
The Really Small Message Broker, by default, starts up listening for MQTT connections on port 1883, and uses no persistence mechanism for storing any messages. This default behaviour can be changed by the use of a configuration file. You can also use the configuration file to control other attributes of the broker, and the configuration of the bridge. The parameters that you can set in the configuration files are described in Broker configuration parameters.
Enter each configuration parameter in the configuration file on a separate line, in the following format, where value can be one or more values, as appropriate for the parameter, separated by spaces:
parameter value
Lines starting with a #
are treated as comments, and are ignored by the broker.
When the broker starts, it looks for a configuration file called broker.cfg in the same directory as the broker (on Windows, for example, the same directory as the broker.exe file).
If you start the broker from the command line, you can specify the name and location of the configuration file as a parameter when you start the broker. For example, if the configuration file is called testbroker.cfg, enter the following command to start the broker:
./broker testbroker.cfg
As an example, a configuration file might look like this:
# sample configuration on port 1882 with retained persistence in /tmp port 1882 max_inflight_messages 50 max_queued_messages 200 persistence_location /tmp/ retained_persistence true
Often, some of the data that is being handled by a Really Small Message Broker is needed elsewhere, on another broker. Take the example of a petrol station. There might be a number of sensors, such as how much petrol is in the tanks, if the car wash is working, and so on, and also a number of applications that receive and send messages to keep the petrol station runnning.
Many of those messages are relevant only at the petrol station location. Other messages need to be sent to head office for information, alerting of problems, ordering new stock, and so on. Some data must also come from head office to the petrol station location, such as price changes and order inventory.
The Really Small Message Broker has a bridge which enables the broker to connect to another MQTT-capable broker, and exchange messages with it, in both directions. For more sophisticated architectures, such as peer-to-peer networks, the bridge allows multiple concurrent connections to several brokers.
When you run the Really Small Message Broker, you can specify
certain parameters by entering them in a configuration file called broker.cfg
, which is present in the same directory as the broker itself, and which is read when you start the broker.
To configure a bridge connection between two brokers
broker.cfg
in the same directory as the local broker.connection connection-name addresses IP_address:port topic parameter
There are several parameters you can set for a bridge connection but only the parameters listed above are required for a basic connection to work. The table in Broker configuration parameters lists the other possible parameters that you can set.
The topic parameter is used to specify what topics are bridged between the local Really Small Message Broker and the remote broker. The format is:
topic pattern direction local_prefix remote_prefix
By setting the local and remote prefixes, you can map a topic on the local broker to a different topic on the remote broker. This is most often used for inserting a local topic or topic tree into a contextually-appropriate place in a remote broker's topic tree.
For example, if you are publishing electricity meter readings on a broker inside your home, you might publish to the power topic. However, if you and many other people are also publishing their electricity readings to a remote broker so that the data can be graphed, or aggregated in some way, it is important that you differentiate your data from the other data. You can do this by mapping a remote prefix, such as meters/laura/, so on the remote broker, the local power topic appears as meters/laura/power.
The topic statement to enable this mapping is:
topic power out "" meters/laura/
Notice how the two quotation marks are used as a placeholder for the local_prefix
parameter.
The direction parameter specifies whether the topics are going from the local broker to a remote broker (out
), from a remote broker to the local broker (in
), or in both directions (both
). The default direction, if it is not specified, is out
.
The following pattern results in messages on a remote broker with topics matching remote/myprefix/# appearing on the local Really Small Message Broker as local/myprefix/#.
topic myprefix/# in local/ remote/
To remap a topic entirely, use "" for the topic pattern, which results in local topic a becoming remote topic b, and vice versa:
topic "" both a b
Be aware that it is quite easy to generate loops when bridging between two brokers. This can happen if you subscribe to the same topics on the remote broker as you are sending to the remote broker. This most often happens when you use the both setting for the direction in the topic. If you connect two Really Small Message Brokers together, special logic is included to prevent most looping situations, by recognising where messages came from, and not sending them back to the same broker. If you are connecting a Really Small Message Broker to a Lotus Expeditor micro broker, or to a WebSphere Message Broker, loops are not detected and you must be careful to avoid them.
By default, a connection will have the automatic start type. This means that it will try to be connected at all times. It will start soon after the broker starts, and whenever a connection fails it will try to restart after a short interval - about 20 seconds.
The other start types are:
Two reasons you might want the broker to listen on multiple ports are:
A configuration file which will limit external client connections on port 1883 to 1, and internal client connections on port 1884 to 100 looks like this:
listener 1883 max_connections 1 listener 1884 127.0.0.1 max_connections 100
A configuration file which will stop clients connected to ports on 1883 and 1884 from interfering by separating their topics:
listener 1883 mount_point 1883/ listener 1884 127.0.0.1 mount_point 1884/ listener 1885Clients connected to port 1885 will be able to access all topics and all messages.
Beginning with MQTT v3.1, a username and password can now be sent by the client at connect time. For older clients which do not support username and password, there is the capability of limiting connections to those clients whose clientid starts with one of a set of prefixes, with the clientid_prefixes configuration parameter.
By default, all clients are able to connect regardless of whether they provide
a username and password or not. This can be changed by setting the password_file
configuration parameter. This points at a password file that lists the valid usernames
in the following format:
username:password
As the passwords are stored in clear-text, care must be taken to protect access to the file using standard Operating System file permissions.
The allow_anonymous
configuration parameter can be used to control whether
clients that do not provide a username and password can still connect to the broker.
Once a client has successfully authenticated, by default, the broker does not restrict
what topics it is able to use. This can be changed by setting the acl_file
configuration parameter.
This points at a file that specifies what topics a user is allowed to use. The default behaviour
then becomes such that a client is not allowed to access a topic unless there is a rule in the ACL to permit it.
Each rule in the acl file identifies a topic that a user can access. This access can be read-only (subscribe),
write-only (publish) or full read-write. The file starts with a list of rules that apply to all users, including
anonymous users if allow_anonymous
is enabled, and then provides user-specific rules.
Rules use the format:
topic [read|write] topicstring
The read
/write
parameter is optional. If not specified, the rule allows full read
write access to the topic. An example acl file might look like:
topic home/public/# topic read meters/# user Fred topic write meters/fred topic home/fred/# user Barney topic write meters/barneytopic home/barney/#
This would result in the following set of rules applying:
home/public/#
topic treemeters/
topic, but can only write to their own.home/
topic treeRules that provide full or read access cannot use the '+
' wildcard. This does
not apply for write-only rules.
Predefined topic use the format:
id topic
Where:
Currently only supported placeholder is [ClientId]. It will be replaced by actual value of client Id. For example a message published by a client called "Sensorduino" sent to a pre-defined topic name sensor/[ClientId]/meter will be published on topic: sensor/ Sensorduino/meter.
Client specific configuration starts with:
clientId client_nameEach mapping following clientId client_name bellongs to the client only and precede broker wide configuration.
# BROKER WIDE configuration # Valid for every client unless client specific configuration found 1 configuration/clientId 2 configuration/type 3 configuration/uptime 4 sensor/meter 5 sensor/temperature 6 sensor/humidity # DYNAMIC PRED-DEFINED topic name # Currently supported placeholder is [ClientId]. It will be replaced # by actual value of client Id. For example a message published by a # client called "Sensorduino" sent to a pre-defined topic name "sensor/[ClientId]/meter" # will be published on topic: "sensor/ Sensorduino /meter". 7 sensor/[ClientId]/rssi # CLIENT SPECIFIC configuration # Client specific configuration starts with: # clientId client_name # Let's say Legacyduino is legacy device, its firmware settings are different clientId LegacyDuino 10 configuration/clientId 11 sensor/meter 12 sensor/temperature 13 sensor/humidity
The following table lists the parameters you can use in the
broker.cfg configuration file to configure the broker. Parameters and
values are case sensitive so, for example, if the possible values are true
or false
, do not specify True
or False
.
Parameter | Description | Default value |
---|---|---|
acl_file | Only applicable if password_file has been specified. The name of a file containing access control configuration. See Access Control List | (No access control is applied.) |
allow_anonymous | Only applicable if password_file has been specified. true means clients may connect without providing authentication information. false means clients must provide valid authentication information to connect. See Authentication of Clients | false |
autosave_on_changes | true means that the autosave interval indicates the number of changes. false means that the autosave interval indicates the number of seconds. | false |
autosave_interval | The length of the autosave interval either in seconds or the number of changes, depending on the autosave_on_changes setting. 0 means no autosave. This relates to the periodic writing to disk of retained messages in the broker. See Saving retained messages. | 1800 (30 minutes) |
bind_address | The local IP address to bind to for the default listener. Useful when a server has multiple network cards and you want to limit access to be only from one network. Specify 127.0.0.1 to restrict client connections to only those from the same machine as the broker. | (The broker allows connections from all network interfaces.) |
clientid_prefixes | A list of prefixes for client IDs that are allowed to connect to the broker. Any other connections are rejected. For example, test_ allows only clients with IDs such as test_1 and test_connection to connect. |
(Any client ID is accepted.) |
connection | The name of the bridge connection, which must be alphanumeric (for example, connection1). This parameter indicates the start of a bridge connection section in the configuration file. This name is combined with the server's hostname to give the clientID on the remote broker. | |
connection_messages | true means that the client connection and disconnection messages are logged. false means that they are not logged. | true |
ffdc_output | A string prefix that is used before the names of FFDC files. The prefix must include any trailing directory separator (/). | (Use the directory in which the broker is installed, or the persistence_location if defined.) The value off turns off FFDC writing altogether - not recommended as this will make problem determination difficult. |
log_level | The level of log output required. The levels, in order of increasing importance, are: config, detail, info, audit, warning, error, severe and fatal. Log messages are written to stdout and to the $SYS/broker/log topic. | info |
listener port [bind_address] | Creates a new listener with the specified port number and local bind address. This parameter indicates the start of a listener section in the configuration file. | (Listener allows connections from all network interfaces.) |
max_connections | If greater than 0, the maximum number of active clients which are allowed to be connected at one time to the default listener. | -1 (no limit) |
max_inflight_messages | The maximum number of persistent (QoS 1 or 2 *) outbound messages that can be in flight (being acknowledged or retried) at a time per client. | 10 |
max_log_entries | The number of log entries remembered for retrieval on request, either by the trace_dump command or for an FFDC. | 100 |
max_queued_messages | The maximum number of persistent (QoS 1 or 2 *) messages that can be queued for delivery to each client. Important: if the queue of messages for a client fills up, any subsequent messages for that client are discarded and are not delivered to that client. When the queue is able to accept messages again, normal message delivery resumes. | 100 |
max_trace_entries | The number of trace entries remembered for retrieval on request, either by the trace_dump command or for an FFDC. | 400 |
retry_interval | The number of seconds before broker will retry the sending of an unacknowledged QoS 1 or 2 message. | 20 |
password_file | The name of a file containing username/password authentication information. See Authentication of Clients | (No authentication is applied.) |
persistence retained_persistence |
true means that retained messages and durable subscriptions are saved when the broker is shut down and restored when the broker restarts. false means that the retained messages and subscriptions are not saved. See Saving retained messages and subscriptions. | false |
persistence_location | A string prefix that is used before the names of files that are used by Really Small Message Broker to store retained messages and durable subscriptions (if the value of the retained_persistence parameter is true). The prefix must include the trailing directory separator (/). |
(Use the directory in which the broker is installed.) |
port | The port number that the default listener uses to listen for MQTT client connections. | 1883 |
predefined_topics_file | The name of a file containing pre-defined topic configuration. See Pre-defined topics. | (No pre-defined topic configuration is applied.) |
trace_level | The level of trace taken and stored in an internal buffer. The levels are: minimum, medium, and maximum. | minimum |
trace_output | A destination to write trace entries as they occur. This will continue indefinitely until explicitly turned off, so beware of creating large files. Possible values are: off, stdout, stderr, protocol or a filename. The protocol setting will write an entry for every MQTT message sent to or received from a client to stdout. | off |
* QoS 0, 1, and 2 refer to Quality of Service, or persistence settings in MQTT, and indicate how hard the Really Small Message Broker will try to deliver the message. QoS0 is at most once, QoS1 is at least once, and QoS2 is exactly once. For full details, see the MQTT protocol specification.
Really Small Message Broker can support one or more simultaneous connections to other MQTT-capable message brokers. The configuration file contains sections that define each bridge connection. As an example, a configuration file with configuration settings for two bridge connections might look like this:
# bridge to a broker on localhost port 1884 connection local addresses 127.0.0.1:1884 topic # both local/ remote/ # bridge to one of two possible brokers connection remote addresses 1.2.3.4 5.6.7.8:1884 topic #
The following table lists the parameters that you can configure for the bridge. Any number of connections can be configured (in sections starting with the connection parameter), and the following parameters apply to each connection individually. Connection definitions must follow any broker-wide parameters in the configuration file.
Parameter | Description | Default value |
---|---|---|
address addresses |
A list of brokers to connect to, in the format ip_address:port.
This is a hunt list of brokers for the Really Small Message Broker to connect to. The usual mode of operation (round_robin false) is that the first address is the primary broker. The other brokers are fail-over brokers if the primary is not available, and are tried in turn until one accepts the bridge connection. If the connection fails over to one of the other brokers in the list, the primary broker is retried in the background until it is able to accept connections again, at which time the bridge connection drops and reconnects to the primary broker. In "round-robin" mode (round_robin true), the addresses list is traversed cyclically, connecting to the next broker in the list each time the bridge attempts a reconnection. |
|
clientid | Overrides the default setting of MQTT clientid used by the connection. The default setting prefixes the connection name with the hostname to help avoid name clashes when multiple connections to the same server are used from different machines. | hostname+'.'+connection_name |
cleansession | Overrides the default setting of clean session, an MQTT protocol setting. true means that all session state is cleaned up on connect and disconnect. Session state includes subscriptions and any queued messages. If the number of addresses is more than 1, then setting cleansession to false only makes sense if the servers at each address can continue interrupted MQTT sessions exactly where they left off. | true if number of addresses is greater than 1, else false |
idle_timeout | If the start_type is lazy, the number of seconds for which the connection is idle before it is automatically closed. | 60 |
keepalive_interval | The number of seconds between sending ping requests on a connection when there has been no other traffic. The minimum value that will be used is 5. | 60 |
notifications | true means that the bridge connection notifications are on. false means that the notifications are off. The notifications are retained messages that are published to a designated topic to indicate the status of each bridge connection at both ends of the bridge (on the local Really Small Message Broker, and on the remote broker). The message is either 1 (connected) or 0 (not connected), and can be checked at any time to determine the status of a bridge connection. The notification_topic parameter determines the topic to which notifications are published. | true |
notification_topic | The topic on which connection notification messages are published. If it is not set, a topic is constructed which includes the client ID of the bridge connection, which is a combination of the server host name and the name used in the connection parameter. If a topic is specified, all bridge notification messages are published to this topic; it is not possible to distinguish between the statuses of multiple bridge connections. | $SYS/broker/connection/{clientID}/state |
password | Sets the MQTT 3.1 password for connection to servers with authorization. | |
round_robin | true means that the list of addresses specfied in the addresses parameter is used in a "round robin" fashion (a circular list), rather than treating the first in the list as the primary, and the others as fail-over brokers. false means that the list of addresses is not used in a round robin fashion. | false |
start_type | automatic, lazy, manual or once. For full details, see Starting Bridge Connections. | automatic |
threshold | If the start_type is lazy, this is the number of messages waiting to be transmitted which will trigger the automatic starting of the connection. | 10 |
topic pattern direction local_prefix remote_prefix | Controls the set of topics which are bridged between a Really Small Message Broker and a remote broker. For full details, see Connecting two Really Small Message Brokers together. | |
try_private | true means that the Really Small Messsage Broker attempts to determine whether a broker that it is connecting to is also a Really Small Message Broker. false means that it will not check. Normally this parameter does not need to be changed because the broker automatically adjusts for connections to a remote broker, but if the bridge can not establish a connection with a particular type of remote broker, setting this parameter to false might solve the problem. | true |
username | Sets the MQTT 3.1 username for connection to servers with authorization. |
The following table lists the parameters that you can configure for a listener. Any number of listeners can be configured (in sections starting with the listener parameter), and the following parameters apply to each listener individually. Listener definitions must follow any broker-wide parameters in the configuration file.
Parameter | Description | Default value |
---|---|---|
max_connections | If greater than 0, the maximum number of active clients which are allowed to be connected at one time to the listener. | -1 (no limit) |
mount_point | A string which is prefixed to all topics used by clients connecting to this listener. This can be used to ensure clients on different listeners cannot interfere with each other. |
Commands can be issued to the broker by writing them to a special file. That file is "broker.upd" in the same directory as the persistence files, which by default is the working directory when the broker was started. The broker will read this file every five seconds if it exists. Once it has been read it is deleted, so that commands are obeyed only once.
The commands available are:
The following commands are related to problem determination.
The broker uses topics prefixed with $SYS/ to publish information about its internal state with retained messages. You must subscribe to a topic pattern starting with $SYS/ to receive these messages. For instance, subscribing to # does not give access to the $SYS topics. Subscribing to $SYS/# subscribes to all the system topics, but none of the non-system topics.
Topic name | Description |
---|---|
$SYS/broker/version | The version number of the broker. |
$SYS/broker/client count/connected | The number of clients that are currently connected. |
$SYS/broker/heap/current size | The current heap used, in bytes. |
$SYS/broker/heap/maximum size | The maximum amount of heap that has been used during the running of the broker. |
$SYS/broker/log/{severity}/{message_number} | Log messages, where severity is one of D, W, I or E, representing Debug, Informational, Warning or Error. Subscribe to $SYS/broker/log/# to get all log messages. |
$SYS/broker/uptime | The number of seconds since the broker was started. |
$SYS/messages/sent | The number of messages sent since the broker was started. |
$SYS/messages/received | The number of messages received since the broker was started. |
$SYS/broker/timestamp | The build timestamp of the broker. |
Some common problems and suggested solutions:
The system cannot execute the specified program
or The application has failed to start because its side-by-side configuration is incorrect
.
To debug interactions between the Really Small Message Broker and clients or other brokers, set the trace_output parameter to protocol in the broker.cfg file or with a command. The broker will then print a message describing each MQTT packet sent and received by the broker.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (™ or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.