Fusion Matrix

Fusion Matrix is an out of the box solution for storage and retrieval of SDMX and EDI datasets. Point Fusion Matrix to the location of your structural repository, such as the Metadata Technology Registry, and import datasets. Matrix will automatically resolve the Data Structure, create any required database tables, and import the dataset. After data has been imported it can immediately be queried via the provided Web Services, allowing data retrieval in SDMX v1.0, v2.0, v2.1, EDI, and CSV of various formats.

Fusion Matrix has been tested for import and export performance.  There is no upper limit to the size of data file imported to the Matrix, or exported from the Matrix, this is due to the use of streaming technology.  By default any application that queries the Matrix that supports gzip encoding, will have the response retuned gzipped.  Gzip is a compression format supported by many client libraries, browsers, and server platforms (including Apache and Microsoft IIS). As SDMX data contains a lot of repetative information, gzip massively reduces the size of a data file that is transmitted over a network.

The tests were carried out by hosting the Matrix on a large Amazon cloud instance (8Gb available memory) running Ubuntu server 10.10.

Import Performance

The imports were performed by copying SDMX data files into the Matrix sweeper directory, and allowing the Matrix discover and import the file.

Each import test was carried out three times on an empty database, and the average time of each import was taken.  The Matrix manages its own memory when importing data in order to optimize the import time without exceeding the memory limits – therefore the import tests were performed against JVMs with different maximum memory allocations in order to determine how this impacted the import time.  The results of the import tests can be seen in the graph below.

As can be observed, Fusion Matrix performs significantly less well with 1Gb of Memory and best with 4gb of memory. However the performance difference between 2Gb and 4Gb is negligible. We would therefore reccomend a minimum of 2Gb Maximum memory to be allocated to the JVM in order to take advantage of the performance gains.

Query Performance

The queries were performed both locally on the server, to cut out network lag, over a network, and over a network with the compression support removed (to demonstrate gzip performance). Each query test was carried out three times, the maximum response time and minimum response time for each query is graphed below. Each query was retuning the response in 2.1 Structure Specific format. The response was not written to a file it was ignored, this was to eliminate any lag being introduced from the file system – however in later tests we discovered that writing the responses to a file had a negligible effect on performance.

The tests were performed on an Amazon cloud instance, running Ubuntu server 10.10 with 2Gb of allocated memory.

We found that when querying for larger datasets, anything over about 200k observations, we were getting an average response speed levelling out at about 85k observations per second – this is running the query locally on the server and does not take into account the network lag. The network lag always increased the response time of a query, with the largest query (returning over 1millions observations) taking around 11 seconds to be completed locally, and 23seconds over a network. With no compression enabled the largest query took 48seconds to complete, this is over double the response time of the same query with compression enabled.

We intend to publish written and video tutorials on our products.

To view the forum click here.

To view the online help click here.

Fusion Matrix runs within a standard Apache Tomcat instance (version 5 or later). For optimal performance it is recommended that the Tomcat instance is supplied with at least 2Gb of Memory.