GCHQ has open sourced one of its projects for creating mass-scale databases. Its Java project - called Gaffer - makes it "easy to store large-scale graphs in which the nodes and edges have statistics such as counts, histograms and sketches." Anyone can head over to Github - the code-sharing site - and take a look at Gaffer's code,
Gaffer was designed to meet the following requirements:
- Allow the creation of graphs with summarised properties within Accumulo with a very minimal amount of coding.
- Allow flexibility of statistics that describe the entities and edges.
- Allow easy addition of new types of nodes and edges.
- Allow quick retrieval of data on nodes of interest.
- Deal with data of different security levels - all data has a visibility, and this is used to restrict who can see data based on their authorizations.
- Support automatic age-off of data.
Since the Snowden revelations a few years ago, GCHQ's reputation has hit rock bottom amongst privacy advocates and large swathes of the public. This attempt to engage with outside developers could be a way to improve relations. Another ambition that the agency probably has is to rope in developers looking to start their careers or switch to a new job.
According to Gaffer's README on Github, Gaffer is no longer in active development. Instead, the agency has begun work on Gaffer2 which will be released to the public shortly. The motive for releasing version one was "so that the community could see where version 2 originated from, and so that we can perform testing to avoid significant performance regressions in version 2."
Anyone wishing to be involved with Gaffer2 will have to sign a contributor license agreement which is being worked on if they want GCHQ to accept pull requests to Gaffer2. A pull request lets you tell others about changes you've submitted (pushed) to the repository on Github. Once a pull request is sent, interested parties can review the set of changes, discuss potential modifications, and even push follow-up commits if necessary.