Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Q Learning과 CNN을 이용한 Object Localization

8,940 views

Published on

Active Object Localization with Deep Reinforcement Learning을 설명하는 일본 관동 컴퓨터비젼 스터디모임의 다꾸야박사의 자료를 번역. 주로 Q Learning의 기본 개념과 실제 적용을 이해하기 쉽게 설명.

Published in: Technology
  • Yes you are right. There are many research paper writing services available now. But almost services are fake and illegal. Only a genuine service will treat their customer with quality research papers. ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • If u need a hand in making your writing assignments - visit ⇒ www.WritePaper.info ⇐ for more detailed information.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • There is a useful site for you that will help you to write a perfect and valuable essay and so on. Check out, please ⇒ www.HelpWriting.net ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Q Learning과 CNN을 이용한 Object Localization

  1. 1. Active Object Localization with Deep Reinforcement Learning 1 2016. 7. 김홍배, 한국항공우주연구원 第32回CV勉強会「ICCV2015読み会」, 皆川卓也
  2. 2. 소개 논문  Active Object Localization with Deep Reinforcement Learning • Juan C. Caicedo, and Svetlana Lazebnik • 물체검출 작업에 Deep Q-Network을 사용 2
  3. 3. DEEP Q-NETWORK (DQN)  Q Learning이라는 강화학습 알고리즘에 Convolutional Neural Network을 적용  아래 논문에서 기계에 컴퓨터게임을 하는 방법을 학습시켜 3/7로 인간을 이김 • Mnih, V., et al., “Playing Atari with Deep Reinforcement Learning”, NIPS Deep Learning Workshop, 2013 • Mnih, V., et al., “Human-level control through deep reinforcement learning”, Nature, 518 (7540), 529–533. 2015 3
  4. 4. 강화학습이란 ? 「어떤 환경에 있어서 Agent가 현재의 상태를 관 측하여 취하여야 하는 행동(Action)을 결정하는 문 제를 다루는 기계학습의 일종. Agent는 행동을 선 택함으로써 보수(Reward) 를 얻는다. 강화학습은 일련의 행동을 통하여 보수가 가장 많게 얻을 수 있 도록 정책(policy)을 학습한다.」(Wikipedia) 4
  5. 5. 강화학습이란? Agent 상태 : s 시간: t 행동: 𝑎 관측 정책: 𝜋(𝑠) 5
  6. 6. 상태 : s’ 강화학습이란? Agent 시간: t+1 행동: 𝑎 정책: 𝜋(𝑠) 6
  7. 7. 시간 : t+1 관측 보수 : 𝑟𝑡 학습으로 구함 강화학습이란? 정책: 𝜋(𝑠') 상태 : s’ Agent 7
  8. 8. 어떻게 정책을 학습할까 ? 아래와 같이 보수의 합의 기대치가 최대가 되도록 지금의 행동을 결정 𝑅𝑡 = 𝑟𝑡 + 𝛾𝑟𝑡 +1 + 𝛾2 𝑟𝑡 +2 + ⋯ + 𝛾 𝑇−𝑡 𝑟 𝑇 보수의 합 감쇄율 장래의 보수 𝜋∗(𝑠) = argmax 𝔼[𝑅𝑡 |𝑠𝑡 = 𝑠, 𝑎𝑡= 𝑎] 𝑎 상태 s일때、보수의 합의 기대치가 최대가 되도록 행동 a를 선택 강화학습이란? 8
  9. 9. Q LEARNING 𝑄∗(𝑠, a) 상태 s、행동 a의 조합의 “좋고” “나쁨”을 점수로 가르쳐주는 함수 상태 s일때、보수의 합의 기대치가 최대가 되도록 행동 a를 선택 𝜋∗(𝑠) = argmax 𝔼 𝑅𝑡 |𝑠𝑡 = 𝑠, 𝑎𝑡= 𝑎 𝑎 상태 s에서 행동 a를 취할 경우, 이후로 최적인 행동 을 계속 취할 경우에 얻게 되는 보수의 합의 기대치 𝜋를 대신해서 함수 Q를 학습 9
  10. 10. Q LEARNING 𝜋를 대신해서 함수 Q를 학습 𝜋∗(𝑠) = argmax 𝑄∗(𝑠, 𝑎) 𝑎 함수 𝑄의 변수 𝜃를 학습에 의해 구한다. 확률적 구배강하법(SGD, Stochastic Gradient Descent) 𝜃𝑖 +1 = 𝜃𝑖 − 𝛼𝛻𝜃 𝑖 𝐿(𝜃𝑖 ) Learning rate 손실함수(Loss function) Q(s, a; θ∗) 10
  11. 11. Q LEARNING 기준값(보수의 합의 최대 기대치) 존재하지 않음  손실함수의 정의 𝜃𝑖 +1 = 𝜃𝑖 − 𝛼𝛻𝜃 𝑖 𝐿(𝜃𝑖 ) L(ϴ𝑖)= 1 2 𝑄∗ 𝑠, 𝑎 − 𝑄 𝑠, 𝑎; ϴ𝑖 2 11
  12. 12. Q LEARNING 𝑄∗(𝑠, 𝑎) = 𝑟𝑡 + 𝛾 max 𝑄∗(𝑠′, 𝑎′) 𝑅𝑡 = 𝑟𝑡 + 𝛾𝑟𝑡+1 + 𝛾2 𝑟𝑡+2 + ⋯ + 𝛾 𝑇−𝑡 𝑟𝑇 보수의 합 𝑅𝑡 = 𝑟𝑡 + 𝛾𝑅𝑡+1 𝑄∗(𝑠, 𝑎) = 𝔼 [𝑅𝑡 |𝑠𝑡 = 𝑠, 𝑎𝑡= 𝑎] 𝑎에 의해 바뀐 상태𝑎′ 12
  13. 13. Q LEARNING  손실함수 L(ϴ𝑖)= 1 2 𝑄∗ 𝑠, 𝑎 − 𝑄 𝑠, 𝑎; ϴ𝑖 2 𝑄∗(𝑠, 𝑎) = 𝑟𝑡 + 𝛾 max 𝑄∗(s′, 𝑎′) 𝑎′ 13
  14. 14. Q LEARNING  손실함수 지금의 변수를 이용해서 근사화 L(ϴ𝑖) = 1 2 𝑄∗ 𝑠, 𝑎 − 𝑄 𝑠, 𝑎; ϴ𝑖 2 = 1 2 𝑟𝑡 + 𝛾 max 𝑄∗(s′, 𝑎′) − 𝑄 𝑠, 𝑎; ϴ𝑖 2 ≈ 1 2 𝑟𝑡 +𝛾 max 𝑄(𝑠′ , 𝑎′ ; 𝜃𝑖) − 𝑄 𝑠, 𝑎; ϴ𝑖 2 𝑎′ 𝑎′ 14
  15. 15. Q LEARNING  손실함수의 구배(Gradient) 𝛻𝜃 𝑖 𝐿 𝜃𝑖 = −( 𝑟𝑡 + 𝛾 max 𝑄(𝑠', 𝑎' ;𝜃𝑖) − 𝑄(𝑠,𝑎;𝜃𝑖 )) 𝛻𝜃 𝑖 𝑄(𝑠,𝑎;𝜃𝑖 ) L(ϴ𝑖) = 1 2 𝑟𝑡 +𝛾 max 𝑄(𝑠′ , 𝑎′ ; 𝜃𝑖 ) − 𝑄 𝑠, 𝑎; ϴ𝑖 2 𝑎′ 𝑎′ 15
  16. 16. Deep Q-Network의 정의 𝑄(𝑠, 𝑎2) Deep CNN 𝜃𝑖 ・・ 𝑄(𝑠, 𝑎 𝐿) s 𝑄(𝑠, 𝑎1)  함수 𝑄(𝑠, 𝑎; 𝜃 𝑖)를 CNN로 나타낸다.  입력이 상태 s, 출력이 각 행동에 따른 Q값 16
  17. 17. Deep Q-Network의 학습 𝑄(𝑠, 𝑎2) Deep CNN 𝜃𝑖 ・・ 𝑄(𝑠, 𝑎 𝐿) st 𝑄(𝑠, 𝑎1) 1. 상태 𝑆𝑡 를 입력시, 𝑄(𝑠𝑡 , 𝑎; 𝜃 𝑖)가 최대가 되는 행동 𝑎 𝑡를 선택 • 단 확률 𝜀으로 랜덤하게 선택 17
  18. 18. 2. 행동 𝑎𝑡 에 의 해 상 태 𝑆𝑡 +1과 보수 𝑟𝑡 를 얻는다. 𝑎𝑡 𝑠𝑡 → 𝑠𝑡+ 1 Agent 환경 𝑟𝑡 Deep Q-Network의 학습 18
  19. 19. 𝑄(𝑠𝑡 +1, 𝑎1) 𝑄(𝑠𝑡 +1, 𝑎2) Deep CNN 𝜃𝑖 ・・ ・・ ・ 𝑄(𝑠𝑡+1, 𝑎 𝐿) 𝑠 t+1 3. 상태 𝑆𝑡 +1를 입력하고 max 𝑎 𝑄(𝑠𝑡 +1, 𝑎; 𝜃𝑖 ) 를 구한다. Deep Q-Network의 학습 19
  20. 20. 𝑦𝑖Deep CNN 𝜃𝑖 ・ ・ 𝑄(𝑠𝑡 , 𝑎 𝐿) 𝑠𝑡 𝑄(𝑠𝑡 , 𝑎1) 4. 다음을 정답으로 해서 변수 𝜃 𝑖를 갱신한다. 𝑦𝑖 = 𝑟𝑡 + γ max 𝑎 𝑄(𝑠𝑡 +1, 𝑎; 𝜃𝑖 ) Deep Q-Network의 학습 L(ϴ𝑖) = 1 2 𝑦𝑖 − 𝑄 𝑠𝑡, 𝑎; ϴ𝑖 2 𝜃𝑖 +1= 𝜃𝑖 − 𝛼𝛻𝜃 𝑖 𝐿(𝜃𝑖 ) 20
  21. 21. 본론 21
  22. 22. DQN로 물체검출  Agent : Bounding Box  행동, a : Bounding Box의 이동/형상변경  상태, s : Box내의 이미지 특징 벡터 + 행동이력  보수, r : 정답(Ground Truth)과의 오버랩비율 22
  23. 23.  행동 (X9) • 상하좌우 이동, 확대/축소, 형상변화(가늘게/넙적하게), 종료 • 이동 및 크기변화의 크기는 Box 크기에 비례(α = 0.2) DQN로 물체검출 23
  24. 24.  상태 • Bounding Box내 이미지로부터 특징 벡터를 구함  CNN에서 구한 4,096x1 벡터 • 과거의 행동이력  직전 10회의 행동까지  각각의 행동은 9x1 Binary 벡터로 표시 (취한 행동에 “1”, 나머지는 “0”)  10 x 9 = 90x1 벡터 • 4096+90 = 4186x1 벡터를 DQN의 입력으로 한다. DQN로 물체검출 24
  25. 25.  보수 b g 𝐼𝑜𝑈 𝑏, 𝑔 = 𝑎𝑟𝑒𝑎(𝑏 ∩ 𝑔) 𝑎𝑟𝑒𝑎(𝑏 ∪ 𝑔) 𝑅 𝑎( 𝑠, 𝑠‘) = 𝑠𝑖𝑔𝑛 (𝐼𝑜𝑈(𝑏′, 𝑔) − 𝐼𝑜𝑈(𝑏, 𝑔)) Agent Ground Truth 정답과의 오버랩 비율 (Intersection over Union) Trigger(종료)이외의행동에대한보수 오버랩 비율이 커지면 1 , 작아지면 - 1 DQN로 물체검출 25
  26. 26. b g 𝐼𝑜𝑈 𝑏, 𝑔 = 𝑎𝑟𝑒𝑎(𝑏 ∩ 𝑔) 𝑎𝑟𝑒𝑎(𝑏 ∪ 𝑔) Agent Ground Trut h 오버랩 비율이 임계치, 𝜏 이상이면 +𝜂、이하면 −𝜂 +𝜂 𝑖𝑓 𝐼𝑜𝑈 𝑏, 𝑔 ≥ 𝜏 −𝜂 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒  보수 정답과의 오버랩 비율 Trigger(종료)에대한보수 𝑅w( 𝑠, 𝑠‘) = DQN로 물체검출 26
  27. 27. 네트웍 구성  이미지 크기는 224x224  특징추출은 Pre-trained CNN을 사용 90x1 224x224 pixels 27
  28. 28. Agent의 학습  Ε-GREEDY알고리즘  확률 ε로 랜덤하게 행동을 선택 • 그 이외에는 가장 Q값이 큰 행동으로 • 본 방법에서는 학습시 정답 데이터를 알고 있으므로 보수가 “+”인 행동 중에서 랜덤하게 선택  ε 은 학습이 진행됨에 따라 작아진다. 28
  29. 29.  Experience Replay  과거의 (𝑠 𝑡, 𝑎 𝑡, 𝑟𝑡, 𝑠𝑡 +1)를 replay-memory에 보존  DQN의 학습 시, 보존된 replay-memory로부터 랜덤하게 선택된 미니뱃치로 재학습 Agent의 학습 29
  30. 30. AGENT에 의한 물체검출 DQN 𝑠 Restart 25%smaller start box size Put it on an image corner Do Action Y Y total 200 steps over ? Y N 𝑎 Trigger ? N 40 steps over ? N 30
  31. 31. 실험 (PASCALVOC2007)  Pascal VOC 20 07 31
  32. 32. RECALL의 평가 32
  33. 33. 물체검출까지의 STEP수의 분포 33
  34. 34. 행동이력의 예 34
  35. 35. 복수 물체검출의 예 35
  36. 36. 검출실패의 예 36
  37. 37. SENSITIVITY ANALYSIS  물체의 외형(Occlusion, Truncation, Size, Aspect Ratio,Viewpoint of Objects, Visibility of parts)에 대한 민감도 해석 37
  38. 38. RUNTIME  K-40 GPU 1대  CNN에 의한 특징검출 (4.5ms)  Q-network (3.2 ms)  평균 1.54sec/image 38
  39. 39. 결론  강화학습을 사용해서 TOP-DOWN으로 물체 검출을 하는 방법을 제안 하나의 OBJECT에 대하여 11~25 정도의 영역밖에 보지 않았는데도 검출가능 39

×