»çÀÌÆ®¸Ê  |  Contact Us
 
Ȩ > SAS Tech & Tip > SAS Enterprise Miner
[EM] ±×·¡µð¾ðÆ® ºÎ½ºÆÃ(Gradient Boosting) ÀÌ·Ð 2017.12.02
±èÈ¿¹ü 1866 0
http://www.mysas.co.kr/SAS_tiptech/j_eminer.asp?b_no=7553&gotopage=1&con=subject&keyword=&cmd=content&bd_no=29&gubun=



À̹ø½Ã°£¿¡´Â ±×·¡µð¾ðÆ® ºÎ½ºÆÃ(Gradient Boosting)¿¡ ´ëÇÏ¿© ´Ù·ïº¸°Ú½À´Ï´Ù.





introduction


±×·¡µð¾ðÆ® ºÎ½ºÆÃÀ̶õ Gradient Desent ¿Í BoostingÀÇ ÇÕ¼º¾î·Î ½±°Ô ¸»ÇØ Boosting¿¡ Gradient Descent¸¦

Á¢¸ñ½ÃŲ °­·ÂÇÑ ¸Ó½Å·¯´×±â¹ýÀÔ´Ï´Ù.


¿ì¼± BoostingÀ̶õ ¾Õ¼­ ¿¬ÀçµÈ ±Û¿¡¼­ ¼Ò°³ µÈ ¹Ù¿Í °°ÀÌ ´Ü¼øÇÑ ¸ðµ¨µéÀ» °áÇÕÇÏ¿© ´Ü°èÀûÀ¸·Î ÇнÀÇÔÀ¸·Î½á

 ÀÌÀü¸ðµ¨ÀÇ ¾àÁ¡À» Á¡Á¡ º¸¿ÏÇØ °¡´Â ¸ðµ¨ÀÔ´Ï´Ù.

 




Gradient Descent


Gradient Descent´Â ´Ü¾î ±×´ë·Î ÇØ¼®ÇÏ¸é ¡®°æ»ç Çϰ­¡¯ À̶õ ÀǹÌÀÔ´Ï´Ù. ¸» ±×´ë·Î ÇÔ¼öÀÇ ±â¿ï±â¸¦ ±¸ÇÏ¿©

±â¿ï±â°¡ ³·Àº ÂÊÀ¸·Î À̵¿½ÃŰ¸é¼­ ±Ø°ª¿¡ À̸¦ ¶§±îÁö ¹Ýº¹ÇÏ´Â ¹æ¹ýÀÔ´Ï´Ù.

À§ÀÇ ±×¸²Àº  ¸¦ ¾ò´Â ¸ð½ÀÀ»

 ³ªÅ¸³»°í ÀÖ½À´Ï´Ù.

Gradient DescentÀ» ¼ö½ÄÀûÀ¸·Î ¼³¸íÇÏÀÚ¸é ÇÔ¼ö f(x){\displaystyle f(\mathbf {x} )}¿¡ ´ëÇÏ¿© °¡ ÁÖ¾î Á³À» ¶§  Àº ´ÙÀ½°ú °°ÀÌ ³ªÅ¸³¾ ¼ö

 ÀÖ½À´Ï´Ù.

Àº À̵¿ÇÒ °Å¸®¸¦ Á¶ÀýÇÏ´Â ¸Å°³º¯¼öÀÔ´Ï´Ù.  Àº wolfe Á¶°ÇÀ» ¸¸Á·ÇÏ´Â ¶óÀΰ˻öÀ» ÅëÇØ ±¸Çϰųª ´ÙÀ½°ú

°°Àº Barzilai-Borwein ¹æ¹ýÀ» ÅëÇØ ±¸ÇÒ ¼ö ÀÖ½À´Ï´Ù.

 

±×·¡µð¾ðÆ® ºÎ½ºÆÃÀº ¾àÇÑ ¸ðµ¨µéÀ» ´Ü°èÀûÀ¸·Î ºÎ½ºÆÃÇÏ´Â °úÁ¤¿¡¼­ ÀÌÀü ¸ðµ¨ÀÇ ¿À·ù¸¦ ¼Õ½ÇÇÔ¼ö·Î ³ªÅ¸³»°í

ÀÌ ¼Õ½ÇÇÔ¼ö¸¦ ÃÖ¼ÒÈ­ÇÏ´Â ¹æ¹ýÀ¸·Î Gradient Descent¸¦ »ç¿ëÇÏ´Â ºÐ¼®±â¹ýÀÔ´Ï´Ù.

 





algorithm

 

 

±×·¡µð¾ðÆ® ºÎ½ºÆÃÀº y¸¦ ¿¹ÃøÇÏ´Â ºÒ¿ÏÀüÇÑ ¸ðµ¨ÀÌ ÀÖ´Ù´Â °¡Á¤À» ÇÏ¿´À» ¶§, ÃßÁ¤·® h¸¦ Ãß°¡ÇÏ¿© ¿¹Ãø¼º´ÉÀ» ¿Ã¸®°íÀÚ ÇÏ´Â ¸ðµ¨ÀÔ´Ï´Ù.  

ÀÌ·± ÇÔ¼ö½ÄÀ» ÃßÁ¤·®h(x)¿¡ ´ëÇÏ¿© ´Ù½Ã Á¤¸®Çϸé

À§¿Í °°Àº ½ÄÀÌ ³ª¿À´Âµ¥ ±×·¡µð¾ðÆ® ºÎ½ºÆÃÀÇ ÁÖ ¸ñÀûÀº ÀÌ h(x)¸¦ ÃßÁ¤ÇÏ´Â °ÍÀÔ´Ï´Ù. ±×¸®°í h(x)¸¦ ÃßÁ¤ÇÏ´Â °úÁ¤¿¡¼­ Gradient descent¸¦ »ç¿ëÇÏ°Ô µË´Ï´Ù.

ºÐ·ù³ª ¼øÀ§¹®Á¦¿¡¼­ ¼Õ½ÇÇÔ¼ö¿¡ ´ëÇÑ ÀϹÝÀûÀÎ ¾ÆÀ̵ð¾î´Â ÁÖ¾îÁø ¸ðµ¨¿¡ ´ëÇÑ ÀÜÂ÷ y-F(x)°¡ ¿ÀÂ÷Á¦°ö ¼Õ½ÇÇÔ¼ö  ÀÇ À½ÀÇ ±â¿ï±â¶ó´Â Á¡¿¡¼­ ³ª¿Ô½À´Ï´Ù.

µû¶ó¼­ ±×·¡µð¾ðÆ® ºÎ½ºÆÃÀÇ ¸ñÀûÀº ƯÁ¤ÇÑ ¼Õ½ÇÇÔ¼öÀÇ ±â´ë°ªÀ» ÃÖ¼ÒÈ­ÇÏ´Â ÇÔ¼ö F(x) ´ëÇÑ ±Ù»çÄ¡ À» ã´Â °ÍÀÔ´Ï´Ù.

±×·¡µð¾ðÆ® ºÎ½ºÆÃÀº ½ÇÁ¦°ª y¸¦ °¡Á¤ÇÏ°í Æ¯Á¤ ´Ü°è·Î ºÎÅÍÀÇ ÇÔ¼ö(x) ÀÇ °¡Áß ÇÕÀ» ÇÏ´Â ¹æ½ÄÀ¸·Î yÀÇ ±Ù»çÄ¡¸¦ ÃßÁ¤ÇÏ°Ô µË´Ï´Ù.

ERM(°æÇèÀû À§Çè ÃÖ¼ÒÈ­)¿øÄ¢¿¡ µû¶ó ÇнÀ¼Â¿¡¼­ ¼Õ½ÇÇÔ¼öÀÇ Æò±Õ°ªÀ» ÃÖ¼ÒÈ­ÇÏ´Â  ¸¦

±¸ÇÏ´Â ¹æ¹ýÀÔ´Ï´Ù. »ó¼öÇÔ¼ö·Î ±¸¼ºµÈ ¸ðµ¨ (x) ¿¡¼­ ½ÃÀÛÇÏ¿© Á¡ÁøÀûÀ¸·Î ¸ðµ¨ÀÌ °¡Áߵ˴ϴÙ.

ÇÏÁö¸¸ ÀÓÀÇÀÇ ¼Õ½ÇÇÔ¼ö L¿¡ ´ëÇÑ ÃÖ°íÀÇ ÇÔ¼öh¸¦ °í¸£´Â °ÍÀº °è»êÀûÀ¸·Î ºÒ°¡´ÉÇÕ´Ï´Ù. µû¶ó¼­ ÇÔ¼ö¸¦ ´Ü¼øÈ­ ½ÃŰ°Ô µË´Ï´Ù. À̶§ Gradient descent¸¦ Àû¿ëÇÏ°Ô µË´Ï´Ù. ¿¬¼ÓÇüÀÇ °æ¿ì, ¾Æ·¡¿Í °°Àº ¹æÁ¤½Ä¿¡ µû¶ó ¸ðµ¨À» ¾÷µ¥ÀÌÆ®ÇÕ´Ï´Ù.

±×¸®°í ÀÌ»êÇüÀÇ °æ¿ì, ¼Õ½ÇÇÔ¼öLÀÇ ±â¿ï±â¿¡ °¡Àå °¡±î¿î Èĺ¸ÇÔ¼öh¸¦ ¼±ÅÃÇÏ°í °è¼ö Àº line searchÀ» ÅëÇØ ±¸ÇÕ´Ï´Ù. ºñ·Ï ÀÌ·± Á¢±Ù¹ýÀÌ ¿Ïº®ÇÏ°Ô Á¤È®ÇÏÁö´Â ¾ÊÁö¸¸ ÃæºÐÈ÷ ¼³µæ·ÂÀÖ´Â ±Ù»ç°ªÀ» ¾òÀ» ¼ö ÀÖ½À´Ï´Ù.


 


loss function


´ëÇ¥ÀûÀÎ ¼Õ½Ç ÇÔ¼ö´Â Square loss :  °¡ ÀÖ°í Square loss ´Â ¼öÇÐÀûÀ¸·Î ´Ù·ç±â´Â

½±Áö¸¸ ÀÌ»óÄ¡¿¡ ¹Î°¨ÇÑ ´ÜÁ¡À» °¡Áö°í ÀÖ½À´Ï´Ù. ÀÌ·¯ÇÑ ´ÜÁ¡À» º¸¿ÏÇÏ´Â ÀÌ»óÄ¡¿¡ ¹Î°¨ÇÏÁö ¾ÊÀº ¼Õ½ÇÇÔ¼ö·Î´Â

Absolute loss ³ª Huber loss °¡ ÀÖ½À´Ï´Ù.

 

.

 

 





 

Reference

https://en.wikipedia.org/wiki/Gradient_descent

http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf

https://en.wikipedia.org/wiki/Gradient_boosting

 

 


 
 
 
ÈÞ´ëÆù ¹øÈ£
ÈÞ´ëÆù ÀÎÁõ¹øÈ£
 
 [EM] ±×·¡µð¾ðÆ® ºÎ½ºÆÃ(Gradient Boosting) ½Ç½À
 [EM] Neural Network (Activation Function)