SimFiller. similarity-based missing values filling algorithm

Fateh Ur Rehman; Muhammad Abbas; Sajjad Murtaza; Wasi Haider Butt; Saad Rehman; Usman Qamar

doi:10.1109/ICDIM.2018.8846983

SimFiller. similarity-based missing values filling algorithm

Fateh Ur Rehman
, Muhammad Abbas
, Sajjad Murtaza
, Wasi Haider Butt
, Saad Rehman
, Usman Qamar

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

With the growth of heterogeneous data generation sources low-quality data volumes are expanding on a daily basis. This research proposed SimFiller: similarity-based missing (null) values filling algorithm, to enhance the quality of data for the data mining process. The proposed algorithm calculates the similarity of record pairs from the input data in such a way that at least one member of the pair has a non-null value for the attribute under consideration. After finding similar pairs, the algorithm fills the missing values by considering the pair having greatest similarity under the specified similarity threshold. The quality of resulted data is evaluated by analyzing the classification accuracy results for Audiology dataset. Five other missing values filling algorithms were selected and total six copies of filled Audiology dataset were created. All six copies of filled Audiology dataset were tested for their classification accuracy. Results show a huge boost in classification accuracy for the copy of the dataset filled with the proposed algorithm and indicate that the quality of the dataset is enhanced. The proposed algorithm can also be tested on other datasets for filling their missing (null) values and can also be extended to remove other inconsistencies from the datasets.

Original language	English
Title of host publication	2018 13th International Conference on Digital Information Management, ICDIM 2018
Editors	Ezendu Ariwa, Pit Pichappan, Pit Pichappan, Wael M El-Medany, Asif Naeem
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	77-81
Number of pages	5
ISBN (Electronic)	9781538652435
DOIs	https://doi.org/10.1109/ICDIM.2018.8846983
State	Published - Sep 2018
Externally published	Yes
Event	13th International Conference on Digital Information Management, ICDIM 2018 - Berlin, Germany Duration: 24 Sep 2018 → 26 Sep 2018

Publication series

Name	2018 13th International Conference on Digital Information Management, ICDIM 2018

Conference

Conference	13th International Conference on Digital Information Management, ICDIM 2018
Country/Territory	Germany
City	Berlin
Period	24/09/18 → 26/09/18

Bibliographical note

Publisher Copyright:
© 2018 IEEE.

Keywords

Data Cleansing
Data Preprocessing
Missing Values
Null Values
RM-RMVO (RapidMiner Replace Missing Values Operator)
SimFiller

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Information Systems and Management
Media Technology
Communication

Access to Document

10.1109/ICDIM.2018.8846983

Cite this

Rehman, F. U., Abbas, M., Murtaza, S., Butt, W. H., Rehman, S., & Qamar, U. (2018). SimFiller. similarity-based missing values filling algorithm. In E. Ariwa, P. Pichappan, P. Pichappan, W. M. El-Medany, & A. Naeem (Eds.), 2018 13th International Conference on Digital Information Management, ICDIM 2018 (pp. 77-81). Article 8846983 (2018 13th International Conference on Digital Information Management, ICDIM 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDIM.2018.8846983

Rehman, Fateh Ur ; Abbas, Muhammad ; Murtaza, Sajjad et al. / SimFiller. similarity-based missing values filling algorithm. 2018 13th International Conference on Digital Information Management, ICDIM 2018. editor / Ezendu Ariwa ; Pit Pichappan ; Pit Pichappan ; Wael M El-Medany ; Asif Naeem. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 77-81 (2018 13th International Conference on Digital Information Management, ICDIM 2018).

@inproceedings{d5844321ad3d4a398bc18db82312aae9,

title = "SimFiller. similarity-based missing values filling algorithm",

abstract = "With the growth of heterogeneous data generation sources low-quality data volumes are expanding on a daily basis. This research proposed SimFiller: similarity-based missing (null) values filling algorithm, to enhance the quality of data for the data mining process. The proposed algorithm calculates the similarity of record pairs from the input data in such a way that at least one member of the pair has a non-null value for the attribute under consideration. After finding similar pairs, the algorithm fills the missing values by considering the pair having greatest similarity under the specified similarity threshold. The quality of resulted data is evaluated by analyzing the classification accuracy results for Audiology dataset. Five other missing values filling algorithms were selected and total six copies of filled Audiology dataset were created. All six copies of filled Audiology dataset were tested for their classification accuracy. Results show a huge boost in classification accuracy for the copy of the dataset filled with the proposed algorithm and indicate that the quality of the dataset is enhanced. The proposed algorithm can also be tested on other datasets for filling their missing (null) values and can also be extended to remove other inconsistencies from the datasets.",

keywords = "Data Cleansing, Data Preprocessing, Missing Values, Null Values, RM-RMVO (RapidMiner Replace Missing Values Operator), SimFiller",

author = "Rehman, \{Fateh Ur\} and Muhammad Abbas and Sajjad Murtaza and Butt, \{Wasi Haider\} and Saad Rehman and Usman Qamar",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 13th International Conference on Digital Information Management, ICDIM 2018 ; Conference date: 24-09-2018 Through 26-09-2018",

year = "2018",

month = sep,

doi = "10.1109/ICDIM.2018.8846983",

language = "English",

series = "2018 13th International Conference on Digital Information Management, ICDIM 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "77--81",

editor = "Ezendu Ariwa and Pit Pichappan and Pit Pichappan and El-Medany, \{Wael M\} and Asif Naeem",

booktitle = "2018 13th International Conference on Digital Information Management, ICDIM 2018",

address = "United States",

}

Rehman, FU, Abbas, M, Murtaza, S, Butt, WH, Rehman, S & Qamar, U 2018, SimFiller. similarity-based missing values filling algorithm. in E Ariwa, P Pichappan, P Pichappan, WM El-Medany & A Naeem (eds), 2018 13th International Conference on Digital Information Management, ICDIM 2018., 8846983, 2018 13th International Conference on Digital Information Management, ICDIM 2018, Institute of Electrical and Electronics Engineers Inc., pp. 77-81, 13th International Conference on Digital Information Management, ICDIM 2018, Berlin, Germany, 24/09/18. https://doi.org/10.1109/ICDIM.2018.8846983

SimFiller. similarity-based missing values filling algorithm. / Rehman, Fateh Ur; Abbas, Muhammad; Murtaza, Sajjad et al.
2018 13th International Conference on Digital Information Management, ICDIM 2018. ed. / Ezendu Ariwa; Pit Pichappan; Pit Pichappan; Wael M El-Medany; Asif Naeem. Institute of Electrical and Electronics Engineers Inc., 2018. p. 77-81 8846983 (2018 13th International Conference on Digital Information Management, ICDIM 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SimFiller. similarity-based missing values filling algorithm

AU - Rehman, Fateh Ur

AU - Abbas, Muhammad

AU - Murtaza, Sajjad

AU - Butt, Wasi Haider

AU - Rehman, Saad

AU - Qamar, Usman

PY - 2018/9

Y1 - 2018/9

N2 - With the growth of heterogeneous data generation sources low-quality data volumes are expanding on a daily basis. This research proposed SimFiller: similarity-based missing (null) values filling algorithm, to enhance the quality of data for the data mining process. The proposed algorithm calculates the similarity of record pairs from the input data in such a way that at least one member of the pair has a non-null value for the attribute under consideration. After finding similar pairs, the algorithm fills the missing values by considering the pair having greatest similarity under the specified similarity threshold. The quality of resulted data is evaluated by analyzing the classification accuracy results for Audiology dataset. Five other missing values filling algorithms were selected and total six copies of filled Audiology dataset were created. All six copies of filled Audiology dataset were tested for their classification accuracy. Results show a huge boost in classification accuracy for the copy of the dataset filled with the proposed algorithm and indicate that the quality of the dataset is enhanced. The proposed algorithm can also be tested on other datasets for filling their missing (null) values and can also be extended to remove other inconsistencies from the datasets.

AB - With the growth of heterogeneous data generation sources low-quality data volumes are expanding on a daily basis. This research proposed SimFiller: similarity-based missing (null) values filling algorithm, to enhance the quality of data for the data mining process. The proposed algorithm calculates the similarity of record pairs from the input data in such a way that at least one member of the pair has a non-null value for the attribute under consideration. After finding similar pairs, the algorithm fills the missing values by considering the pair having greatest similarity under the specified similarity threshold. The quality of resulted data is evaluated by analyzing the classification accuracy results for Audiology dataset. Five other missing values filling algorithms were selected and total six copies of filled Audiology dataset were created. All six copies of filled Audiology dataset were tested for their classification accuracy. Results show a huge boost in classification accuracy for the copy of the dataset filled with the proposed algorithm and indicate that the quality of the dataset is enhanced. The proposed algorithm can also be tested on other datasets for filling their missing (null) values and can also be extended to remove other inconsistencies from the datasets.

KW - Data Cleansing

KW - Data Preprocessing

KW - Missing Values

KW - Null Values

KW - RM-RMVO (RapidMiner Replace Missing Values Operator)

KW - SimFiller

UR - https://www.scopus.com/pages/publications/85073521022

U2 - 10.1109/ICDIM.2018.8846983

DO - 10.1109/ICDIM.2018.8846983

M3 - Conference contribution

AN - SCOPUS:85073521022

T3 - 2018 13th International Conference on Digital Information Management, ICDIM 2018

SP - 77

EP - 81

BT - 2018 13th International Conference on Digital Information Management, ICDIM 2018

A2 - Ariwa, Ezendu

A2 - Pichappan, Pit

A2 - El-Medany, Wael M

A2 - Naeem, Asif

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 13th International Conference on Digital Information Management, ICDIM 2018

Y2 - 24 September 2018 through 26 September 2018

ER -

Rehman FU, Abbas M, Murtaza S, Butt WH, Rehman S, Qamar U. SimFiller. similarity-based missing values filling algorithm. In Ariwa E, Pichappan P, Pichappan P, El-Medany WM, Naeem A, editors, 2018 13th International Conference on Digital Information Management, ICDIM 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 77-81. 8846983. (2018 13th International Conference on Digital Information Management, ICDIM 2018). doi: 10.1109/ICDIM.2018.8846983

SimFiller. similarity-based missing values filling algorithm

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this