This document discusses using OpenAI Gym to create reinforcement learning agents and summarizes various reinforcement learning algorithms. It introduces OpenAI Gym, an environment for developing and comparing reinforcement learning algorithms. It then provides overviews of model-free reinforcement learning algorithms like Q-learning, SARSA, policy iteration, and value iteration. Deep Q-networks which use neural networks to approximate the Q-function are also introduced. Examples of code implementations are provided.

1. OpenAI Gym을 이용한
강화학습 에이전트 만들기
파이콘 코리아 2017 튜토리얼
이의령, 이영무, 이웅원, 양혁렬, 김건우
RLCode

6. https://github.com/zzing0907/pycon-tutorial
•
•
https://github.com/rlcode/reinforcement-learning-
kr/tree/master/wiki

8. http://web.stanford.edu/class/cs234/slides/lecture1_introduction.pdf

9. à à
http://www.popsci.com/googles-alphago-ai-defeats-lee-se-dol-at-game-go

10. à à

12. •
à

13. •
à
https://www.youtube.com/watch?v=0CqoMwcqIbQ

14. •

16. à

17. à
•
•
•

19. à

20. à

21. 𝑠", 𝑎", 𝑟&, 𝑠&, 𝑎&, 𝑟', ⋯ , 𝑠)

23. à
•
•
•
•

25. •
•
•
• 𝑃,,-
.
𝑠/
• 𝛾
• 𝜋 𝑎 𝑠)

27. à

28. •

29. •
•
= 𝑅56& + 𝑅56' + ⋯ + 𝑅)

30. • à
0.1 + 0.1 + ⋯ = ∞
1 + 1 + ⋯ = ∞
= 𝑅56& + 𝛾𝑅56' + ⋯ + 𝛾)<5<&
𝑅)
0 ≤ 𝛾 ≤ 1

31. •
𝑣(𝑠) = 𝑬 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠

32. •
𝑞(𝑠, 𝑎) = 𝑬 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠, 𝐴5 = 𝑎

33. • à
•
𝑞E(𝑠, 𝑎) = 𝑬 𝝅 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠, 𝐴5 = 𝑎
𝑣E(𝑠) = 𝑬 𝝅 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠
𝜋(𝑎|𝑠) = 𝑷 𝐴5 = 𝑎|𝑆5 = 𝑠

35. •
𝑣E(𝑠) = 𝑬 𝝅 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠
𝑣E(𝑠) = H 𝜋 𝑎 𝑠)
.∈K
(𝑅56& + 𝛾𝑣E(𝑆56&))
𝑣E(𝑠) = 𝑬 𝝅 𝑅56& + 𝛾𝑣E(𝑆56&)|𝑆5 = 𝑠

36. 𝑣E(𝑠) = H 𝜋 𝑎 𝑠)
.∈K
(𝑅56& + 𝛾𝑃,,/
.
𝑣E(𝑆56&))

37. •
•
𝛾
𝑣∗ 𝑠 = 𝑚𝑎𝑥. 𝐸 𝑅56& + 𝛾𝑣∗ 𝑆56& 𝑆5 = 𝑠, 𝐴5 = 𝑎]
𝑞∗ 𝑠, 𝑎 = 𝐸 𝑅56& + 𝛾𝑚𝑎𝑥. 𝑞∗ 𝑆56&, 𝑎′ 𝑆5 = 𝑠, 𝐴5 = 𝑎]

38. • à
•
𝑞E(𝑠, 𝑎) = 𝑬 𝝅 𝑅56& + 𝛾𝑅56' + ⋯ |𝑆5 = 𝑠, 𝐴5 = 𝑎
𝑞E(𝑠, 𝑎) = 𝑬 𝝅 𝑅56& + 𝛾(𝑅56' + ⋯ )|𝑆5 = 𝑠, 𝐴5 = 𝑎
𝑞E(𝑠, 𝑎) = 𝑬 𝝅 𝑅56& + 𝛾𝑞E(𝑆56&, 𝐴56&)|𝑆5 = 𝑠, 𝐴5 = 𝑎

40. •
•

41. •
𝑣R
2. 𝑉R(𝑠/
) 𝛾
𝑅,
.
𝑅,
.
𝛾 𝑉R(𝑠/
)
𝜋(𝑎|𝑠)(𝑅,
.
𝛾 𝑉R(𝑠/
))
∑ 𝜋(𝑎|𝑠)(𝑅,
.
𝛾 𝑉R(𝑠/
))
∈ 𝑆에

42. •
𝑣E(𝑠) = H 𝜋 𝑎 𝑠)
.∈K
(𝑅56& + 𝛾𝑣E(𝑆56&))

44. •
𝜋′(𝑠) = 𝑎𝑟𝑔𝑚𝑎𝑥. 𝑞E(𝑠, 𝑎)

45. •
𝜋′(𝑠) = 𝑎𝑟𝑔𝑚𝑎𝑥. 𝑞E(𝑠, 𝑎)

46. https://github.com/zzing0907/pycon-tutorial/tree/master/policy_iteration

48. •
•
•
•
𝑣R6&(𝑠) = 𝑚𝑎𝑥.(𝑅,
. + 𝛾𝑣R(𝑆56&)

50. •
𝑣R6&(𝑠) = 𝑚𝑎𝑥.(𝑅,
. + 𝛾𝑣R(𝑆56&)

51. https://github.com/zzing0907/pycon-tutorial/tree/master/value_iteration

52. •
•
•

53. •
•

56. • à
ß
𝑞E(𝑠, 𝑎) = 𝑬 𝝅 𝑅56& + 𝛾𝑞E(𝑆56&, 𝐴56&)|𝑆5 = 𝑠, 𝐴5 = 𝑎
𝑞 𝑠, 𝑎 ← 𝑟 + 𝛾𝑞E(𝑠′, 𝑎′)
𝑞 𝑠, 𝑎 = 𝑞 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾𝑞 𝑠′, 𝑎′ − 𝑞 𝑠, 𝑎 )

57. •
à
𝑞 𝑠, 𝑎 = 𝑞 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾𝑞 𝑠′, 𝑎′ − 𝑞 𝑠, 𝑎 )
𝜀 − 𝜋(𝑠) = [
𝑎∗
= 𝑎𝑟𝑔𝑚𝑎𝑥. 𝑞(𝑠, 𝑎), 1 − 𝜀
𝑎 ≠ 𝑎∗
, 𝜀

58. 𝜀 −
• à
𝜀 − 𝜋(𝑠) = [
𝑎∗ = 𝑎𝑟𝑔𝑚𝑎𝑥. 𝑞(𝑠, 𝑎), 1 − 𝜀
𝑎 ≠ 𝑎∗, 𝜀
𝜋′(𝑠) = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑎 𝑞 𝜋(𝑠, 𝑎)

59. https://github.com/zzing0907/pycon-tutorial/tree/master/SARSA

60. •
à
𝑞 𝑠, 𝑎 = 𝑞 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾 max
./
𝑞(𝑠/
, 𝑎′) − 𝑞 𝑠, 𝑎 )

61. 𝜖
𝜖
https://github.com/rlcode/reinforcement-learning-kr/tree/master/1-grid-world/5-q-learning

62. (𝑆/) 𝑎/
(𝑆/)

63. https://github.com/zzing0907/pycon-tutorial/tree/master/Q-Learning

65. •
•
•
•

66. •

67. •

68. •
ℎ𝑖𝑠𝑡𝑜𝑟𝑦 → 𝑞 − 𝑛𝑒𝑡𝑤𝑜𝑟𝑘 → 𝑞 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑒𝑎𝑐ℎ 𝑎𝑐𝑡𝑖𝑜𝑛

69. •

70. •
𝑞 𝑠, 𝑎 = 𝑞 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾 max
./
𝑞(𝑠/
, 𝑎′) − 𝑞 𝑠, 𝑎 )
인공신경망 업데
이트

71. •
à

72. •
• 𝜃
•
𝑞 𝑠, 𝑎 = 𝑞 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾 max
./
𝑞(𝑠/, 𝑎′) − 𝑞 𝑠, 𝑎 )
𝑞o 𝑠, 𝑎 = 𝑞o 𝑠, 𝑎 + 𝛼(𝑟 + 𝛾 max
.-
𝑞o 𝑠/, 𝑎/ − 𝑞o 𝑠, 𝑎 )
𝑀𝑆𝐸 = 𝑟 + 𝛾 max
.-
𝑞o 𝑠/, 𝑎/ − 𝑞o 𝑠, 𝑎
'

73. •
https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

74. •
어떤 필터인지에 따라 이미지를 다양하게 변형 가능
https://en.wikipedia.org/wiki/Kernel_(image_processing)

75. •
DQN의 CNN 학습된 필터의 예시

76. • à

77. •
𝜃<
•
𝑀𝑆𝐸 = 𝑟 + 𝛾 max
.-
𝑞oq 𝑠/
, 𝑎/
− 𝑞o 𝑠, 𝑎
'

80. •
•
•
•
•

82. •
•
•
𝑥
𝑥̇
𝜃
𝜃̇
•
•
https://github.com/zzing0907/pycon-tutorial/tree/master/cartpole_dqn

83. •
•
•
from gym import wrappers
env = gym.make(‘CartPole-v1')
env = wrappers.Monitor(env, ‘/tmp/cartpole_upload’, force=True)

84. •
env.close()
gym.scoreboard.api_key = ‘your api key’
gym.upload(‘/tmp/cartpole_upload’)

85. http://wikibook.co.kr/reinforcement-learning/

86. •
•
•
•
•