We address the problem of credit scoring as a classification and feature subset selection problem. Based on the current framework of sophisticated feature selection methods, we identify features that contain the most relevant information to distinguish good loan payers from bad loan payers. The feature selection methods are validated on several real world datasets with different types of classifiers. We show the advantages following from using the sub-space approach to classification. We discuss many practical issues related to the applicability of feature selection methods. We show and discuss some difficulties that use to be insufficiently emphasised in standard feature selection literature.