GSoC: The evolution of Kiskadee models.

The 0.1 release of Kiskadee [1] brought a minimal set of functionalities that permitted us to monitor some targets and analysis some packages, serving more as a proof of concept than as a real monitoring system. One of the limitations of the 0.1 release was that Kiskadee was saving all the analysis in a single field on the packages table. Every time that we need to generate a new analysis for a same package version, we appended the result on the analysis field. The next two code snippet was picked up from the Kiskadee source code.

reports = []
...
with kiskadee.helpers.chdir(path):
        kiskadee.logger.debug('ANALYSIS: Unpacked!')
        analyzers = plugin.analyzers()
        for analyzer in analyzers:
                kiskadee.logger.debug('ANALYSIS: running %s ...' % analyzer)
                analysis = kiskadee.analyzers.run(analyzer, path)
                firehose_report = kiskadee.converter.to_firehose(analysis,
                                                         analyzer)
                reports.append(str(firehose_report))
kiskadee.logger.debug('ANALYSIS: DONE running %s' % analyzer)
return reports
all_analyses = '\n'.join(reports)

Note that we generate a analysis for each plugin analyzer, and append it in the reports array. In other part of the runner code we generate, using a join function, a single string with all the analysis made. A bad implementation to a issue that should be solved by changing the Kiskadee models in order to support several analysis for a same package version, and that was what we have done with the PR #24.With this pull request, Kiskadee will be able to save different analysis from a same package version made by different analyzers. A lot of refactoring was made in the Runner component, specially because was difficult to implement new tests for it. The responsibility of save a analysis was removed from the analyze method, and moved to the _save_source_analysis.

def _save_source_analysis(source_to_analysis, analysis, analyzer, session):
if analysis is None:
        return None

source_name = source_to_analysis['name']
source_version = source_to_analysis['version']

package = (
        session.query(kiskadee.model.Package)
        .filter(kiskadee.model.Package.name == source_name).first()
)
version_id = package.versions[-1].id
_analysis = kiskadee.model.Analysis()
try:
        _analyzer = session.query(kiskadee.model.Analyzer).\
            filter(kiskadee.model.Analyzer.name == analyzer).first()
        _analysis.analyzer_id = _analyzer.id
        _analysis.version_id = version_id
        _analysis.raw = analysis
        session.add(_analysis)
except Exception as err:
        kiskadee.logger.debug(
        "The required analyzer was not registered on Kiskadee"
        )
        kiskadee.logger.debug(err)

With this pull request two new models was added to Kiskadee, analysis and analyzers. The Diagram 1 presents the current database structure of Kiskadee.

Diagram 1: Kiskadee database structure.

With the PR #24 we will fix some other issues:

  • Docker not run properly on the Jenkins VM. Issue #23
    • Docker and selinux are not good friends so we have to change the way that we were creating a container volume to run the analyzers.
  • CI integration. Issue #7
    • We now have a Jenkins instance running our tests for every push made to the pagure repository. In the future we want to add a continuous deploy to every code that be merged on master branch.
[1]https://pagure.io/kiskadee