GSoC: How integrates fedmsg with Kiskadee

If you want to know why we are using fedmsg [1] with Kiskadee [2] you can check this post, where I explain the reasons for such integration. Now we will cover the implementation of this integration and the current status of Kiskadee's architecture.

fedmsg-hub it's a daemon used to interact with the fedmsg bus, and with it we can receive and send messages from and to applications. If you have cloned Kiskadee repository, create the fedmsg-hub configurations files and run the daemon.

sudo mkdir -p /etc/fedmsg.d/
sudo cp util/ util/  /etc/fedmsg.d/
sudo cp util/ /etc/fedmsg.d/
pip install -e .
PYTHONPATH=`pwd` fedmsg-hub

If everything goes ok, fedmsg-hub will start, and will use our AnityaConsumer class to consume the fedmsg bus. The file will tell to fedmsg-hub a list of addresses from which fedmsg can send messages, in our case, this endpoint will point to a Anitya server, where new project releases are published. Basically, a fedmsg-hub consumer is a class that inherits from fedmsg.consumers.FedmsgConsumer and implements a consume method.

The fedmsg-hub daemon runs in a separate process of Kiskadee, so we need some mechanism to make the consumer send the bus messages to Kiskadee. To do this we are using ZeroMQ [3] as a pub/sub library, in a way that the consumer publishes the incoming messages, and the anitya plugin on Kiskadee consumes this messages.

Once the message arrives in Kiskadee, the default life cycle of a source will occur.

  • The Anitya plugin will queue the source.
  • the monitor component will dequeue the source.
    • If the source version not exists on the database, save it, and queue the source to the runner component.
    • If the source version exists on the database, do nothing.
  • the runner component will dequeue the source.
  • the runner component will run a analysis, and save the result on the database.

On this post we have a better description of the Kiskadee architecture. Each service that publishes on fedmsg have several topics, each on related to a specific event. Here you can check a list of topics where Anitya publish messages. Our consumer have only one responsibility: When a message arrives, publish it on ZeroMQ server, on the anitya topic. Kiskadee will be listen to this topic, and will receive the message. Let's take a look on the consumer code:

class AnityaConsumer(fedmsg.consumers.FedmsgConsumer):
"""Consumer used by fedmsg-hub to subscribe to fedmsg bus."""

topic = ''
config_key = 'anityaconsumer'
validate_signatures = False

def __init__(self, *args, **kw):
        """Anityaconsumer constructor."""
        super().__init__(*args, *kw)
        context = zmq.Context()
        self.socket = context.socket(zmq.PUB)

def consume(self, msg):
        """Consume events from fedmsg-hub."""
        self.socket.send_string("%s %s" % ("anitya", str(msg)))

ZeroMQ is also used by fedmsg, so the topic variable can cover more than one Anitya topic. If we want to receive all messages published by Anitya, we just need to use a more generic topic value:

topic = '*'