Application Development
Data reporting and collection
If you collect or report statistical data then DDI, SDMX, and XBRL standards can play a key role in improving timeliness of data (through automation), accuracy of data (through automated validity checks based on metadata), and auditability of the data supply chain (through monitoring data reporting cycles).
Metadata Technology can help you integrate these standards into your application systems, and can help you to put into place the metadata systems that will help you to automate and to control the process. With our training and our tools we can help you to help your data providers to get up and going with quickly.
Data warehousing
SDMX standards can play a pivotal role in the building of the modern data warehouse, from acting simply as a pivot format for data loading and dissemination, to automating the data harvesting and loading operation based on registry technology.
The beginning and end phases are shown below. However, the move to the ultimate in automation can be gradual and controlled, with each step bringing business benefits and leading to the next step - so no investment is lost.
Data warehouses typically draw their data from many disparate sources, each requiring transformation into a common format for intermediary storage, where the data can be validated prior to loading into the warehouse. The data are extracted and "pushed" to the warehouse by the system responsible for the source data.
SDMX standards can be immediately useful in providing a common data structure outside of the warehouse, into which data are transformed by the source system, and from which various dissemination systems can transform to their formats.
A generic warehouse data structure can be built using Data Structure technology, which describes multi-dimensional data structures. Valuable metadata can be attached to the data.
With a metadata registry it is possible to automate the data input stage. The data sources inform the registry when new or revised data are available (again, Data Structure technology allows this to be done at varying degrees of granularity). The registry notifies the warehouse. The warehouse system queries the registry for details and formulates a query (SDMX-ML query) which is sent to the data source. The data source transforms the query into a database query, and returns the result to the warehouse in the common SDMX-ML format.
In this scenario the warehouse harvest the data as it becomes available and is always up to date. The registry allows the whole process to operate in an automated way.
Data portal
SDMX takes full advantage of the Internet for efficient "just in time" data reporting and dissemination. The data need not leave the database in which it is stored, except collection request. Compare this with traditional centralised reporting mechanisms and you can see that the savings are considerable.
Owners or producers of data need not send the data to a central repository for collation - they keep the data in their systems and register their existence in a registry. Web services query the registry for the data: there is sufficient metadata in the registry to enable the web service to formulate an SDMX query on all the databases that have relevant data. The databases process the SDMX query and return the data to the web service in the SDMX or DDI format.
The web service can therefore deliver data on demand or it can cache the data for frequent queries, and can be informed by the registry when the data have changed (e.g. a new month of data is now available).
This new paradigm of data availability is called "pull" technology, as opposed to the data reporting "push" mechanisms in common use today. By harnessing pull technology there are no time delays and the data are always up to date. The benefits of this can be considerable.
Top of page
|
