Department of Statistics Malaysia (DOSM)
The Department of Statistics Malaysia (DOSM) is the premier government agency that serves to collect data and information regarding economic and social aspects of the country. Under the jurisdiction of the Prime Minister’s Office, the department is tasked with collecting, analyzing and disseminating statistical reports that provide a snapshot of the national economic performance and social development.
The Trade Division of DOSM sought a more efficient way of cleansing data. Before Speedminer, data cleansing was done through custom written Cobol programming running on a mainframe system. This process was severely outdated and labour-intensive with much time wasted on conducting checks for errors. Even with a large group of employees and much time invested, errors were still produced. This previous process was also too rigid for new cleansing rules to be implemented, often presenting a scenario which required a whole revamp of the coding just to incorporate new changes. The process was also prone to bottlenecks whereby the sequential approach to cleansing multiple levels of trade data left a lot to be desired in terms of efficiency. This issue is even more pressing when extremely large volumes of data are considered.
The Speedminer Solution
A whole new approach was adopted through Speedminer. The Business Rule Engine allows for cleansing rules to be added into the system based on a structured manner, which ensured easier maintenance of data consistency, addition or modification of cleansing rules. Essentially, the result was that the entire flow of the process was more coherent and automated which would require less human intervention. Based on the OLAP model, Speedminer ensured greater operational effectiveness by processing dirty data and listing records that failed any cleansing rules. The records also go through multiple levels of cleansing which are performed concurrently. This generates previously unobtainable levels of efficiency, with the time frame needed for data cleansing drastically reduced.
The National Enterprise-Wide Statistical System (NEWSS) is an intelligence information technology solution that was designed to increase overall operational and management effectiveness, in line with the goal of delivering quality statistics. NEWSS faced the requirement of creating an integrated statistical framework for monitoring and managing census and surveys as well as producing statistical information based on the principles of timeliness and reliability. Such an ambitious task was plagued by problems, namely manual data transfers that caused undetectable data loss and data redundancy and an over-reliance on MS Excel for data analysis and reports production. Overall, report production was an unnecessarily tedious task that was a burden on resources. To make matters worse, without a functional overview of all integrated data, trend and pattern recognition from such a holistic view was not in reach.
- Catering for thousands of analyses and reports.
- Hassle-free and easy implementation
- Comprehensive data cleansing for different types of data
Speedminer empowered users to create and design their own reports based on ubiquitous access across multiple departments. All this was possible due to further integration in the process of data synchronization between departments. As a DW (Data Warehouse) and BPM (Business Process Management) tool, Speedminer contributed to making processes easier to conduct as well as reducing human error through powerful tracking and analysis features for the survey recording process. NEWSS is now enjoying great leaps in work productivity.
Examples of reports and analysis produced by the Speedminer System include the export, import and balance of trade report, the imported goods based on broad economic categories report, as well as main and selected imports reports just to name a few.
Speedminer for All
While originally only implemented in the Trade Division of DOSM, the astounding performance and results by the Speedminer System caught the attention of other departments. Speedminer is currently being used also by EKS (Small Medium Enterprise), Agriculture Census, Population Census, Data Cleansing, NEWSS (National Enterprise-wide Statistical Systems), Sabah and Sarawak and Finance sectors.