FASTER: A hybrid algorithm for feature selection and record reduction in rare frequent itemset

The amount of data that has to be analysed and processed to assist decision making has significantly increased in recent years. These datasets may contain potentially useful, but as yet undiscovered, information and knowledge. This high dimensionality of datasets leads to the phenomenon known as the curse of dimensionality. When faced with difficulties resulting from the high dimension of a space, the ideal approach is to decrease this dimension, without losing relevant information in the data. The use of Rough-Set theory to achieve feature selection is one approach that has proven successful. However, most approaches carry out reduction in only one dimension i.e in the number of attributes. In this investigation a new algorithm is proposed which allows for record reduction as well as attribute reduction. FASTER (FeAture SelecTion using Entropy and Rough sets) is a hybrid pre-processor algorithm which utilizes entropy and rough-sets to carry out record reduction and feature (attribute) selection respectively. FASTER produced an attribute reduction of 30% with a speed improvement of 2.6 times when used as preprocessor for two different rare itemset algorithms.