Screening Vital Few Variables and Development of Logistic Regression Model on a Large Data Set

Lim, Yong-B.; Cho, J.; Um, Kyung-A; Lee, Sun-Ah

Journal of Korean Society for Quality Management > Volume 34(2); 2006 > Article

Journal of Korean Society for Quality Management 2006;34(2): 129-.

대용량 자료에서 핵심적인 소수의 변수들의 선별과 로지스틱 회귀 모형의 전개

임용빈, 조재연, 엄경아, 이선아

이화여대 통계학과

Screening Vital Few Variables and Development of Logistic Regression Model on a Large Data Set

Yong-B. Lim, J. Cho, Kyung-A Um, Sun-Ah Lee

Department of Statistics, Ewha Womans

ABSTRACT

In the advance of computer technology, it is possible to keep all the related informations for monitoring equipments in control and huge amount of real time manufacturing data in a data base. Thus, the statistical analysis of large data sets with hundreds of thousands observations and hundred of independent variables whose some of values are missing at many observations is needed even though it is a formidable computational task. A tree structured approach to classification is capable of screening important independent variables and their interactions. In a Six Sigma project handling large amount of manufacturing data, one of the goals is to screen vital few variables among trivial many variables. In this paper we have reviewed and summarized CART, C4.5 and CHAID algorithms and proposed a simple method of screening vital few variables by selecting common variables screened by all the three algorithms. Also how to develop a logistics regression model on a large data set is discussed and illustrated through a large finance data set collected by a credit bureau for th purpose of predicting the bankruptcy of the company.

Key Words: Classification Tree;Screening Vital few Variables;Logistic Regression Model;