You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

849 lines
50 KiB

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Collaboration and Competition\n",
"\n",
"---\n",
"\n",
"You are welcome to use this coding environment to train your agent for the project. Follow the instructions below to get started!\n",
"\n",
"### 1. Start the Environment\n",
"\n",
"Run the next code cell to install a few packages. This line will take a few minutes to run!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[31mtensorflow 1.7.1 has requirement numpy>=1.13.3, but you'll have numpy 1.12.1 which is incompatible.\u001b[0m\r\n",
"\u001b[31mipython 6.5.0 has requirement prompt-toolkit<2.0.0,>=1.0.15, but you'll have prompt-toolkit 3.0.18 which is incompatible.\u001b[0m\r\n"
]
}
],
"source": [
"!pip -q install ./python"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The environment is already saved in the Workspace and can be accessed at the file path provided below. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:unityagents:\n",
"'Academy' started successfully!\n",
"Unity Academy name: Academy\n",
" Number of Brains: 1\n",
" Number of External Brains : 1\n",
" Lesson number : 0\n",
" Reset Parameters :\n",
"\t\t\n",
"Unity brain name: TennisBrain\n",
" Number of Visual Observations (per agent): 0\n",
" Vector Observation space type: continuous\n",
" Vector Observation space size (per agent): 8\n",
" Number of stacked Vector Observation: 3\n",
" Vector Action space type: continuous\n",
" Vector Action space size (per agent): 2\n",
" Vector Action descriptions: , \n"
]
}
],
"source": [
"from unityagents import UnityEnvironment\n",
"import numpy as np\n",
"\n",
"env = UnityEnvironment(file_name=\"/data/Tennis_Linux_NoVis/Tennis\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# get the default brain\n",
"brain_name = env.brain_names[0]\n",
"brain = env.brains[brain_name]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Examine the State and Action Spaces\n",
"\n",
"Run the code cell below to print some information about the environment."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of agents: 2\n",
"Size of each action: 2\n",
"There are 2 agents. Each observes a state with length: 24\n",
"The state for the first agent looks like: [ 0. 0. 0. 0. 0. 0. 0.\n",
" 0. 0. 0. 0. 0. 0. 0.\n",
" 0. 0. -6.65278625 -1.5 -0. 0.\n",
" 6.83172083 6. -0. 0. ]\n",
"The state for the second agent looks like: [ 0. 0. 0. 0. 0. 0. 0.\n",
" 0. 0. 0. 0. 0. 0. 0.\n",
" 0. 0. -6.4669857 -1.5 0. 0.\n",
" -6.83172083 6. 0. 0. ]\n"
]
}
],
"source": [
"# reset the environment\n",
"env_info = env.reset(train_mode=True)[brain_name]\n",
"\n",
"# number of agents \n",
"NUM_AGENTS = len(env_info.agents)\n",
"print('Number of agents:', NUM_AGENTS)\n",
"\n",
"# size of each action\n",
"action_size = brain.vector_action_space_size\n",
"print('Size of each action:', action_size)\n",
"\n",
"# examine the state space \n",
"states = env_info.vector_observations\n",
"state_size = states.shape[1]\n",
"print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
"print('The state for the first agent looks like:', states[0])\n",
"print('The state for the second agent looks like:', states[1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. Take Random Actions in the Environment\n",
"\n",
"In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
"\n",
"Note that **in this coding environment, you will not be able to watch the agents while they are training**, and you should set `train_mode=True` to restart the environment."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total score (averaged over agents) this episode: -0.004999999888241291\n",
"Total score (averaged over agents) this episode: -0.004999999888241291\n",
"Total score (averaged over agents) this episode: -0.004999999888241291\n",
"Total score (averaged over agents) this episode: -0.004999999888241291\n",
"Total score (averaged over agents) this episode: -0.004999999888241291\n"
]
}
],
"source": [
"for i in range(5): # play game for 5 episodes\n",
" env_info = env.reset(train_mode=False)[brain_name] # reset the environment \n",
" states = env_info.vector_observations # get the current state (for each agent)\n",
" scores = np.zeros(NUM_AGENTS) # initialize the score (for each agent)\n",
" while True:\n",
" actions = np.random.randn(NUM_AGENTS, action_size) # select an action (for each agent)\n",
" actions = np.clip(actions, -1, 1) # all actions between -1 and 1\n",
" env_info = env.step(actions)[brain_name] # send all actions to tne environment\n",
" next_states = env_info.vector_observations # get next state (for each agent)\n",
" rewards = env_info.rewards # get reward (for each agent)\n",
" dones = env_info.local_done # see if episode finished\n",
" scores += env_info.rewards # update the score (for each agent)\n",
" states = next_states # roll over states to next time step\n",
" if np.any(dones): # exit loop if episode finished\n",
" break\n",
" print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When finished, you can close the environment."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"#env.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. It's Your Turn!\n",
"\n",
"Now it's your turn to train your own agent to solve the environment! A few **important notes**:\n",
"- When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
"```python\n",
"env_info = env.reset(train_mode=True)[brain_name]\n",
"```\n",
"- To structure your work, you're welcome to work directly in this Jupyter notebook, or you might like to start over with a new file! You can see the list of files in the workspace by clicking on **_Jupyter_** in the top left corner of the notebook.\n",
"- In this coding environment, you will not be able to watch the agents while they are training. However, **_after training the agents_**, you can download the saved model weights to watch the agents on your own machine! "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# Import necessary packages\n",
"\n",
"import gym\n",
"import random\n",
"import torch\n",
"import numpy as np\n",
"from collections import deque\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"import workspace_utils"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Actor and Critic Model"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# Actor and Critic Model\n",
"\n",
"import numpy as np\n",
"\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"\n",
"def hidden_init(layer):\n",
" fan_in = layer.weight.data.size()[0]\n",
" lim = 1. / np.sqrt(fan_in)\n",
" return (-lim, lim)\n",
"\n",
"class Actor(nn.Module):\n",
" \"\"\"Actor (Policy) Model.\"\"\"\n",
"\n",
" def __init__(self, state_size, action_size, seed, fc1_units=128, fc2_units=128):\n",
" \"\"\"Initialize parameters and build model.\n",
" Params\n",
" ======\n",
" state_size (int): Dimension of each state\n",
" action_size (int): Dimension of each action\n",
" seed (int): Random seed\n",
" fc1_units (int): Number of nodes in first hidden layer\n",
" fc2_units (int): Number of nodes in second hidden layer\n",
" \"\"\"\n",
" super(Actor, self).__init__()\n",
" self.seed = torch.manual_seed(seed)\n",
" self.bn1 = nn.BatchNorm1d(state_size)\n",
" self.fc1 = nn.Linear(state_size, fc1_units) # we have two agents in this environment\n",
" self.fc2 = nn.Linear(fc1_units, fc2_units)\n",
" self.fc3 = nn.Linear(fc2_units, action_size)\n",
" self.bn2 = nn.BatchNorm1d(fc2_units)\n",
" self.reset_parameters()\n",
"\n",
" def reset_parameters(self):\n",
" self.fc1.weight.data.uniform_(*hidden_init(self.fc1))\n",
" self.fc2.weight.data.uniform_(*hidden_init(self.fc2))\n",
" self.fc3.weight.data.uniform_(-3e-3, 3e-3)\n",
"\n",
" def forward(self, state):\n",
" \"\"\"Build an actor (policy) network that maps states -> actions.\"\"\" \n",
" x = self.bn1(state)\n",
" x = F.leaky_relu(self.fc1(x)) # F.relu\n",
" x = F.leaky_relu(self.fc2(x)) # F.relu\n",
" return F.tanh(self.fc3(x))\n",
"\n",
"\n",
"class Critic(nn.Module):\n",
" \"\"\"Critic (Value) Model.\"\"\"\n",
"\n",
" def __init__(self, state_size, action_size, seed, fcs1_units=128, fc2_units=128):\n",
" \"\"\"Initialize parameters and build model.\n",
" Params\n",
" ======\n",
" state_size (int): Dimension of each state\n",
" action_size (int): Dimension of each action\n",
" seed (int): Random seed\n",
" fcs1_units (int): Number of nodes in the first hidden layer\n",
" fc2_units (int): Number of nodes in the second hidden layer\n",
" \"\"\"\n",
" super(Critic, self).__init__()\n",
" self.seed = torch.manual_seed(seed)\n",
" \n",
" self.bn1 = nn.BatchNorm1d(state_size) \n",
" self.fcs1 = nn.Linear(state_size, fcs1_units) # take in the state size\n",
" self.fc2 = nn.Linear((fcs1_units+NUM_AGENTS*action_size), fc2_units) # add actions\n",
" self.fc3 = nn.Linear(fc2_units, 1) # 1 = a single value\n",
" self.bn2 = nn.BatchNorm1d(fc2_units)\n",
" self.reset_parameters()\n",
"\n",
" def reset_parameters(self):\n",
" self.fcs1.weight.data.uniform_(*hidden_init(self.fcs1))\n",
" self.fc2.weight.data.uniform_(*hidden_init(self.fc2))\n",
" self.fc3.weight.data.uniform_(-3e-3, 3e-3)\n",
"\n",
" def forward(self, state, action_agent_0, action_agent_1):\n",
" \"\"\"Build a critic (value) network that maps (state, action) pairs -> Q-values.\"\"\"\n",
" x = self.bn1(state)\n",
" x = F.leaky_relu(self.fcs1(x)) # F.relu\n",
" x = torch.cat((x, action_agent_0, action_agent_1), dim=1)\n",
" x = F.leaky_relu(self.fc2(x)) # F.relu\n",
" return self.fc3(x)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Multi Agent Deep Determinitic Policy Gradient Agent with Replay Buffer and Noise"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# Multi Agent Deep Determinitic Policy Gradient Agent with Replay Buffer and Noise\n",
"\n",
"import numpy as np\n",
"import random\n",
"import copy\n",
"from collections import namedtuple, deque\n",
"\n",
"import torch\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"\n",
"BUFFER_SIZE = int(1e5) # replay buffer size\n",
"BATCH_SIZE = 128 # minibatch size\n",
"GAMMA = 0.99 # discount factor\n",
"TAU = 1e-3 # for soft update of target parameters\n",
"LR_ACTOR = 1e-4 # learning rate of the actor\n",
"LR_CRITIC = 1e-3 # learning rate of the critic\n",
"NUM_AGENTS = 2 # Multi agent approach = 2; single agent approach = 1\n",
"LEARNING_LOOPS = 2 # Perform a second learning step in the step function\n",
"\n",
"device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
"\n",
"\n",
"class Agent(object):\n",
" \"\"\"Interacts with and learns from the environment.\"\"\"\n",
"\n",
" def __init__(self, state_size, action_size, random_seed):\n",
" \"\"\"Initialize an Agent object.\n",
" Params\n",
" ======\n",
" state_size (int): dimension of each state\n",
" action_size (int): dimension of each action\n",
" random_seed (int): random seed\n",
" \"\"\"\n",
" self.state_size = state_size\n",
" self.action_size = action_size\n",
" self.seed = random.seed(random_seed)\n",
"\n",
" # Actor Network (w/ Target Network)\n",
" self.actor_local = Actor(state_size, action_size, random_seed).to(device)\n",
" self.actor_target = Actor(state_size, action_size, random_seed).to(device)\n",
" self.actor_optimizer = optim.Adam(self.actor_local.parameters(), lr=LR_ACTOR)\n",
" print(\"Actor net - local: \", self.actor_local)\n",
" print(\"Actor net - target: \", self.actor_target)\n",
"\n",
" # Critic Network (w/ Target Network)\n",
" self.critic_local = Critic(state_size, action_size, random_seed).to(device)\n",
" self.critic_target = Critic(state_size, action_size, random_seed).to(device)\n",
" self.critic_optimizer = optim.Adam(self.critic_local.parameters(), lr=LR_CRITIC)\n",
" print(\"Critic net - local: \", self.critic_local)\n",
" print(\"Critic net - target: \", self.critic_target)\n",
" \n",
" # Init the weights of the target network with the weights of the local network\n",
" self.init_soft_update(self.actor_local, self.actor_target)\n",
" self.init_soft_update(self.critic_local, self.critic_target)\n",
"\n",
" # Noise process\n",
" self.noise = OUNoise(NUM_AGENTS * action_size, random_seed)\n",
"\n",
" # Replay memory\n",
" self.memory = ReplayBuffer(BUFFER_SIZE, BATCH_SIZE, random_seed)\n",
"\n",
" def step(self, states, actions_agent_0, actions_agent_1, rewards, next_state_agent_0, next_state_agent_1, dones):\n",
" \"\"\"Save experience in replay memory, and use random sample from buffer to learn.\"\"\"\n",
" # Save experiences / rewards\n",
" for state, action, action_1, reward, next_state, next_state_1, done \\\n",
" in zip(states, actions_agent_0, actions_agent_1, rewards, next_state_agent_0, next_state_agent_0, dones):\n",
" self.memory.add(state, action, action_1, reward, next_state, next_state_1, done)\n",
"\n",
" # Learn, if enough samples are available in memory\n",
" # Learn x times as specified in LEARNING_LOOPS\n",
" if len(self.memory) > BATCH_SIZE:\n",
" for _ in range(LEARNING_LOOPS):\n",
" experiences = self.memory.sample()\n",
" self.learn(experiences, GAMMA)\n",
"\n",
" def act(self, state, add_noise=True, noise_factor=1.0):\n",
" \"\"\"Returns actions for given state as per current policy.\"\"\"\n",
" state = torch.from_numpy(state).float().to(device)\n",
" self.actor_local.eval()\n",
" with torch.no_grad():\n",
" action = self.actor_local(state).cpu().data.numpy()\n",
" self.actor_local.train()\n",
" if add_noise:\n",
" action += noise_factor * self.noise.sample().reshape((-1, 2))\n",
" return np.clip(action, -1, 1)\n",
"\n",
" def reset(self):\n",
" self.noise.reset()\n",
"\n",
" def learn(self, experiences, gamma):\n",
" \"\"\"Update policy and value parameters using given batch of experience tuples.\n",
" Q_targets = r + γ * critic_target(next_state, actor_target(next_state), actor_target(next_state_other_player))\n",
" where:\n",
" actor_target(state) -> action\n",
" critic_target(state, action, action) -> Q-value\n",
" Params\n",
" ======\n",
" experiences (Tuple[torch.Tensor]): tuple of (s, a, a_2, r, s', s'_2, done) tuples\n",
" gamma (float): discount factor\n",
" \"\"\"\n",
" ##states, actions, actions_other_player, rewards, next_states, next_states_other_player, dones = experiences\n",
" states, actions_agent_0, actions_agent_1, rewards, next_state_agent_0, next_state_agent_1, dones = experiences\n",
"\n",
" # ---------------------------- update critic ---------------------------- #\n",
" # Get predicted next-state actions and Q values from target models\n",
" actions_next_agent_0 = self.actor_target(next_state_agent_0)\n",
" actions_next_agent_1 = self.actor_target(next_state_agent_1)\n",
" Q_targets_next = self.critic_target(next_state_agent_0, actions_next_agent_0, actions_next_agent_1)\n",
" # Compute Q targets for current states (y_i)\n",
" Q_targets = rewards + (gamma * Q_targets_next * (1 - dones))\n",
" # Current expected Q-values\n",
" Q_expected = self.critic_local(states, actions_agent_0, actions_agent_1)\n",
" # Compute critic loss\n",
" critic_loss = F.mse_loss(Q_expected, Q_targets)\n",
" # Minimize the loss\n",
" self.critic_optimizer.zero_grad()\n",
" critic_loss.backward()\n",
" # Gradient clipping\n",
" torch.nn.utils.clip_grad_norm_(self.critic_local.parameters(), 1)\n",
" self.critic_optimizer.step()\n",
"\n",
" # ---------------------------- update actor ---------------------------- #\n",
" # Compute actor loss\n",
" actions_pred = self.actor_local(states)\n",
" actor_loss = -self.critic_local(states, actions_pred, actions_agent_1).mean()\n",
" # Minimize the loss\n",
" self.actor_optimizer.zero_grad()\n",
" actor_loss.backward()\n",
" self.actor_optimizer.step()\n",
"\n",
" # ----------------------- update target networks ----------------------- #\n",
" self.soft_update(self.critic_local, self.critic_target, TAU)\n",
" self.soft_update(self.actor_local, self.actor_target, TAU)\n",
"\n",
" def soft_update(self, local_model, target_model, tau):\n",
" \"\"\"Soft update model parameters.\n",
" θ_target = τ*θ_local + (1 - τ)*θ_target\n",
" Params\n",
" ======\n",
" local_model: PyTorch model (weights will be copied from)\n",
" target_model: PyTorch model (weights will be copied to)\n",
" tau (float): interpolation parameter\n",
" \"\"\"\n",
" for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):\n",
" target_param.data.copy_(tau * local_param.data + (1.0 - tau) * target_param.data)\n",
"\n",
" def init_soft_update(self, local_net, target_net):\n",
" \"\"\" Init model parameters \n",
" Params\n",
" ======\n",
" local_model: PyTorch model (weights will be copied from)\n",
" target_model: PyTorch model (weights will be copied to)\n",
" \"\"\"\n",
" for target_param, local_param in zip(target_net.parameters(), local_net.parameters()):\n",
" target_param.data.copy_(local_param.data) \n",
" \n",
"\n",
"class OUNoise:\n",
" \"\"\"Ornstein-Uhlenbeck process.\"\"\"\n",
"\n",
" def __init__(self, size, seed, mu=0., theta=0.15, sigma=0.2):\n",
" \"\"\"Initialize parameters and noise process.\"\"\"\n",
" self.mu = mu * np.ones(size)\n",
" self.theta = theta\n",
" self.sigma = sigma\n",
" self.seed = random.seed(seed)\n",
" self.state = None\n",
" self.reset()\n",
"\n",
" def reset(self):\n",
" \"\"\"Reset the internal state (= noise) to mean (mu).\"\"\"\n",
" self.state = copy.copy(self.mu)\n",
"\n",
" def sample(self):\n",
" \"\"\"Update internal state and return it as a noise sample.\"\"\"\n",
" x = self.state\n",
" dx = self.theta * (self.mu - x) + self.sigma * np.array([random.random() for i in range(len(x))])\n",
" self.state = x + dx\n",
" return self.state\n",
"\n",
"\n",
"class ReplayBuffer:\n",
" \"\"\"Fixed-size buffer to store experience tuples.\"\"\"\n",
" def __init__(self, buffer_size, batch_size, seed):\n",
" \"\"\"Initialize a ReplayBuffer object.\n",
" Params\n",
" ======\n",
" buffer_size (int): maximum size of buffer\n",
" batch_size (int): size of each training batch\n",
" \"\"\"\n",
" self.memory = deque(maxlen=buffer_size) # internal memory (deque)\n",
" self.batch_size = batch_size\n",
" self.experience = namedtuple(\"Experience\", field_names=[\"state\", \"action\", \"action_1\", \"reward\", \"next_state\", \"next_state_1\", \"done\"])\n",
" self.seed = random.seed(seed)\n",
"\n",
" def add(self, state, action, action_1, reward, next_state, next_state_1, done):\n",
" \"\"\"Add a new experience to memory.\"\"\"\n",
" e = self.experience(state, action, action_1, reward, next_state, next_state_1, done)\n",
" self.memory.append(e)\n",
"\n",
" def sample(self):\n",
" \"\"\"Randomly sample a batch of experiences from memory.\"\"\"\n",
" experiences = random.sample(self.memory, k=self.batch_size)\n",
"\n",
" states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)\n",
" actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).float().to(device)\n",
" actions_1 = torch.from_numpy(np.vstack([e.action_1 for e in experiences if e is not None])).float().to(device)\n",
" rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)\n",
" next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(\n",
" device)\n",
" next_states_1 = torch.from_numpy(np.vstack([e.next_state_1 for e in experiences if e is not None])).float().to(\n",
" device)\n",
" dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(\n",
" device)\n",
"\n",
" return states, actions, actions_1, rewards, next_states, next_states_1, dones\n",
"\n",
" def __len__(self):\n",
" \"\"\"Return the current size of internal memory.\"\"\"\n",
" return len(self.memory)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Training of the agent (DDPG)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"num_agents: 2\n",
"state_size: 24\n",
"action_size: 2\n",
"Actor net - local: Actor(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fc1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=128, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=2, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Actor net - target: Actor(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fc1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=128, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=2, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Critic net - local: Critic(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fcs1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=132, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=1, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Critic net - target: Critic(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fcs1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=132, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=1, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Episode 100\t Average Score: 0.02rage maximum score over the last 10 episodes: 0.02\n",
"Episode 200\t Average Score: 0.00rage maximum score over the last 10 episodes: 0.00\n",
"Episode 300\t Average Score: 0.00rage maximum score over the last 10 episodes: 0.02\n",
"Episode 400\t Average Score: 0.05rage maximum score over the last 10 episodes: 0.10\n",
"Episode 500\t Average Score: 0.09rage maximum score over the last 10 episodes: 0.10\n",
"Episode 600\t Average Score: 0.10rage maximum score over the last 10 episodes: 0.10\n",
"Episode 669\tmax score: 2.60\t average maximum score over the last 10 episodes: 1.36\n",
"Environment solved in 569 episodes!\t Average Score: 0.52\n"
]
}
],
"source": [
"# Training of the agent (DDPG)\n",
"\n",
"def maddpg(n_episodes=1000, max_t=10000, print_every=100):\n",
" \"\"\"Deep Deterministic Policy Gradients DDPG\n",
" Params\n",
" ======\n",
" n_episodes (int) = maximum number of episodes\n",
" max_t (int) = max number of timesteps per episode\n",
" print_every (int) = orint results every n episodes\n",
" \"\"\"\n",
" scores_window = deque(maxlen=print_every)\n",
" scores = []\n",
" scores_mean = []\n",
" noise_decay = 1.0 # noise multiplication factor\n",
" \n",
" for i_episode in range(1, n_episodes+1):\n",
" env_info = env.reset(train_mode=True)[brain_name] # reset env\n",
" states = env_info.vector_observations # get current states of agent\n",
" score = np.zeros(NUM_AGENTS) # set score/ reward to zero if using multiple agents\n",
" agent.reset()\n",
" \n",
" for t in range(max_t):\n",
" if i_episode < 100:\n",
" actions_agent_0 = np.random.randn(2, 2) # use random actions for the first 100 episodes\n",
" else:\n",
" actions_agent_0 = agent.act(states, noise_factor=noise_decay) # let the agent select actions\n",
" \n",
" actions_agent_1 = np.flip(actions_agent_0, 0) # action of second agent/player\n",
" env_info = env.step(actions_agent_0)[brain_name] # send actions of both agents to env\n",
" rewards = env_info.rewards # get the rewards from env\n",
" next_state_agent_0 = env_info.vector_observations # get next states from env\n",
" next_state_agent_1 = np.flip(next_state_agent_0, 0) # get the resulting states for the second agent/player\n",
" dones = env_info.local_done # check if episode is done\n",
" \n",
" agent.step(states, actions_agent_0, actions_agent_1, rewards, next_state_agent_0, next_state_agent_1, dones) # perform step of agent\n",
" \n",
" # update statistical variables\n",
" states = next_state_agent_0\n",
" score += rewards\n",
" if np.any(dones):\n",
" break\n",
" \n",
" score_max = np.max(score) # the max score of the agents\n",
" scores_window.append(score_max)\n",
" scores.append(score_max)\n",
" scores_mean.append(np.mean(scores_window)) # mean score of the agent\n",
" \n",
" noise_decay = max(0.999 * noise_decay, 0.01) # reduce noise during training\n",
" \n",
" print('\\rEpisode {:d}\\tmax score: {:.2f}\\t average maximum score over the last 10 episodes: {:.2f}'.format(i_episode, scores_window[-1], np.mean(list(scores_window)[-10:])), end=\"\")\n",
" \n",
" if i_episode > 100 and np.mean(scores_window) > 0.5:\n",
" torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
" torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
" print('\\nEnvironment solved in {:d} episodes!\\t Average Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
" break\n",
" if i_episode % print_every == 0:\n",
" print('\\rEpisode {}\\t Average Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
" \n",
" return scores, scores_mean\n",
"\n",
"print(\"num_agents: \", NUM_AGENTS)\n",
"print(\"state_size: \", state_size)\n",
"print(\"action_size: \", action_size)\n",
"with workspace_utils.active_session():\n",
" agent = Agent(state_size=24, action_size=2, random_seed=0)\n",
" scores, scores_mean = maddpg()\n"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmYXHWd7/H3t9beu5N0E0J2dtFBwIggXodn3AC9OipewBkFxpErF7d7R+dB7x1R78y4zTjL44K443jRURwnKgoMMqCyJhDCEoEASWgCSSed3ru6uqq+949zulPdXV3pVLq27s/reeqpU2erb1WfPt863/M7v2PujoiIyGwi1Q5ARERqmxKFiIgUpUQhIiJFKVGIiEhRShQiIlKUEoWIiBSlRCEiIkUpUYiISFFKFCIiUlSs2gEcrs7OTl+3bl21wxARqSubN2/e5+5dpSxbd4li3bp1bNq0qdphiIjUFTPbWeqyKj2JiEhRShQiIlKUEoWIiBSlRCEiIkUpUYiISFFKFCIiUpQShYiIFKVEISJSZT97aDf9I+OTr3f3jXLbtj1VjGgqJQoRkSp6qmeID9zwIB/58UOT4972lbt4z3c34e5VjOwgJQoRkSraNzgGQN9IenLcCwMpAIbT2arENJ0ShYhIFQ2mMgC0NsRnTBsYHZ8xrhqUKEREqmhwLEgGrQ0zu94bSClRiIgsehNHFC3JAoliNFPpcApSohARqaJ6KD3VXTfjIiILwcTJ66f2DgGQc8fd6c9LDj1DY4yms+Tc2b53iKPbG1je1lDxWK1Wml/N1YYNG1z3oxCRenbbtj2857sz92Onrmpna3f/rMu97w+P4+rzTy7pPc1ss7tvKGVZlZ5ERCps084DBccXSxLVpEQhIlJh8YiVtJyVttgRU6IQEamwaKS0XW+V8kT5EoWZrTaz281sm5k9amYfKjDPuWbWb2ZbwscnyhWPiEitiEWrtcsvTTlbPWWAv3D3B8ysFdhsZre6+2PT5vuNu7+pjHGIiNSUeImJYsGVntz9eXd/IBweBLYBK8v1fiIi9aLU0lO1VCRaM1sHnA7cW2Dy2Wb2kJn90sxeXIl4RESqqeQjiiqdpSj7BXdm1gLcCHzY3QemTX4AWOvuQ2Z2AfBT4IQC67gCuAJgzZo1ZY5YRKS8YqWezF5opScAM4sTJInvu/tPpk939wF3HwqHbwLiZtZZYL7r3H2Du2/o6uoqZ8giImUXK7F5bLWUs9WTAd8Etrn7F2eZ5+hwPszszDCe/eWKSUSkFpTa6qla6aWcpadzgHcBD5vZlnDcx4E1AO5+LXAhcKWZZYBR4GKvtz5FREQOUyxa4m/0KtWeypYo3P23HCIBuvuXgC+VKwYRkVqk0pOIiBQVLbULj3mOY66UKEREKqy+jieUKEREKq7UE7ELsnmsiIjMn2pdcKdEISJSYfXWtlOJQkSk4krLFCo9iYhIUWr1JCKySKj0JCIiRanVk4iIlIVVKVMoUYiIVJhKTyIiUpSXXHyqDiUKEREpSolCRKTCSi096WS2iMgiUXKrJ3XhISIitUiJQkSkwkq9kadKTyIiUpS68BARkZqkRCEiUmFq9SQiIkWVesGdWj2JiEhNUqIQEakwlZ5ERKQodQooIiILihKFiEiFlX7jIp3MFhFZFEq9MrtalChEROrEgrsy28xWm9ntZrbNzB41sw8VmMfM7J/NbLuZbTWzM8oVj4hIrai3e2bHyrjuDPAX7v6AmbUCm83sVnd/LG+e84ETwscrgK+GzyIiC1d9VZ7Kd0Th7s+7+wPh8CCwDVg5bba3ANd74B6gw8xWlCsmEZF6tuBKT/nMbB1wOnDvtEkrgWfzXnczM5lgZleY2SYz29TT01OuMEVEKqLkLjwWaqsnM2sBbgQ+7O4D0ycXWGTGN+ju17n7Bnff0NXVVY4wRUQqps4aPZU3UZhZnCBJfN/df1Jglm5gdd7rVcDucsYkIlKvFlwXHhYcI30T2ObuX5xlto3Au8PWT2cB/e7+fLliEhGpBXV2QFHWVk/nAO8CHjazLeG4jwNrANz9WuAm4AJgOzACXF7GeEREakLJnQLObxhzVrZE4e6/5RCfy4PLE68qVwwiIgvKQj2ZLSIiU5Xa6qlalChERCqs3kpPShQiInViwbV6EhGRwuqr8KREISJSeSXWnqxKxSclChERKUqJQkSkwuqtm3ElChGRClOrJxERWVCUKEREKqzUe2ar9CQiskiUfI5CrZ5ERKQWKVGIiFRYyTcuUulJRGRxqLM8oUQhIiLFKVGIiFRY6a2edDJbRERqkBKFiEid0DkKEZFFouQuPNTqSURkcdCtUEVEpCx0RCEiskiU3nusWj2JiCwK9VV4UqIQEakbKj2JiCwSJff1VCVKFCIiFaZWTyIiUhYLrgsPM/uWme01s0dmmX6umfWb2Zbw8YlyxSIiUkvqrfQUK+O6vwN8Cbi+yDy/cfc3lTEGEZEFY8F14eHudwK95Vq/iMhiU/OtnszsVWZ2eTjcZWbr5+H9zzazh8zsl2b24nlYn4hIzSu1m/FqmVPpycyuATYAJwHfBuLAvwDnHMF7PwCsdfchM7sA+ClwwizvfwVwBcCaNWuO4C1FRKpvoV6Z/VbgzcAwgLvvBlqP5I3dfcDdh8Lhm4C4mXXOMu917r7B3Td0dXUdyduKiNStWi89pT04VnIAM2s+0jc2s6MtbOtlZmeGsew/0vWKiNS6+io8zb3V07+a2deADjN7L/BnwNeLLWBmNwDnAp1m1g1cQ1Cywt2vBS4ErjSzDDAKXOz1VrgTESlB6aWn6phTonD3vzOz1wEDBOcpPuHutx5imUsOMf1LBM1nRURkDqpVejpkojCzKHCzu78WKJocRETk0BZcFx7ungVGzKy9AvGIiCx4pRfZq3NIMddzFCngYTO7lbDlE4C7f7AsUYmISM2Ya6L4RfgQEZEjVOoBRc2eowBw9++aWQI4MRz1uLuPly8sEZEFrMTaU023ejKzc4HvAjsIYl1tZpeG/TmJiMgCNtfS098Dr3f3xwHM7ETgBuBl5QpMRGShKr30VNtdeMQnkgSAuz9BePGciIgcngV5wR2wycy+CXwvfP0nwObyhCQiIrVkroniSuAq4IMESe1O4CvlCkpEZCEr9YK7mm71FM73T+7+RZi8WjtZtqhERBaweuvVbq7nKG4DGvNeNwL/Mf/hiIjIbGq9m/GGiXtHAITDTeUJSURkYSu51VON37ho2MzOmHhhZhsIugYXEZHDVG+lp7meo/gw8CMz202QDI8BLipbVCIiMlMtlp7M7OVmdrS73w+cDPwQyAC/Ap6pQHwiIgtOya2e5jmOuTpU6elrQDocPhv4OPBl4ABwXRnjEhFZuBZY6Snq7r3h8EXAde5+I3CjmW0pb2giIpKvVrvwiJrZRDJ5DfDrvGlzPb8hIiJ5Sm/1VB2H2tnfANxhZvsIWjn9BsDMjgf6yxybiMiC5HXW7KloonD3vzGz24AVwC1+8NNFgA+UOzgRETmoZrvwcPd7Cox7ojzhiIgsfHV2QDHnC+5ERGSeLNQrs0VEpMpqva8nERGZJyo9iYhIUYWuzF7WnDjkcrV6ZbaIiFRALDqHNKDSk4jI4lCo9FStE9VzUbZEYWbfMrO9ZvbILNPNzP7ZzLab2db8bsxFRGSmhdjq6TvAeUWmnw+cED6uAL5axlhERGraXFo0LbhWT+5+J9BbZJa3ANd74B6gw8xWlCseEZFaUagLj9otPFX3HMVK4Nm8193hOBGRBa3eOgWsZqIo9JkLfn9mdoWZbTKzTT09PWUOS0Sk8qrVhfhcVDNRdAOr816vAnYXmtHdr3P3De6+oaurqyLBiYiUS6kX3NXq/SjKaSPw7rD101lAv7s/X8V4REQqouRbodZq77GlMrMbgHOBTjPrBq4B4gDufi1wE3ABsB0YAS4vVywiIrWuhitP5UsU7n7JIaY7cFW53l9EpFaVXHqa3zDmTFdmi4hUWKE8sSivoxARkblblF14iIhIYaV3M774Wj2JiCxSBa7MVulJRESKqd3CkxKFiEjFqdWTiIgUVfB+FDV8IYUShYhIDZhLmliMXXiIiCxKJXfhMc9xzJUShYhIhRU8R1G7lSclChGReqHmsSIii0SpBxQL8Z7ZIiJSgFo9iYhIWaj0JCKySBRq9VS7xxNKFCIilVew9FT5MOZKiUJEpE6o9CQiskgUbvVUu4cUShQiIhXmBZo9zambcTWPFRGRYlR6EhFZJEq+wV2VKFGIiFRYyfej0BGFiMjiNZcrs3WOQkRkkaizzmOVKEREKq1Qq6e5UOlJRGQR05XZIiIyqWDpaU7XUVSHEoWISKWp1dNBZnaemT1uZtvN7OoC0y8zsx4z2xI+/ryc8YiI1KpDtWhaY3sgl61QNFOVLVGYWRT4MnA+cApwiZmdUmDWH7r7aeHjG+WKR0SkVhTqZryYKFl+kfg4nb/7VJkiKq6cRxRnAtvd/Wl3TwM/AN5SxvcTEakLhe9wN/v8b4jcT6uNMrZiQ/mCKqKciWIl8Gze6+5w3HRvN7OtZvZjM1tdaEVmdoWZbTKzTT09PeWIVUSkqmbPE85n4kGxZfSYsysVzhTlTBSFPvf0PPozYJ27nwr8B/DdQity9+vcfYO7b+jq6prnMEVEKutwLqM4K7KNdhvhq5n/Srb5qPIFVUQ5E0U3kH+EsArYnT+Du+9397Hw5deBl5UxHhGRmlDwHEWB2pOR4/OxrzHmca7PvL4CkRVWzkRxP3CCma03swRwMbAxfwYzW5H38s3AtjLGIyJSswqVYP4u/jXWRHq4JnMpz7Os4jFNiJVrxe6eMbP3AzcDUeBb7v6omX0a2OTuG4EPmtmbgQzQC1xWrnhERGrF3EpPzrmRLTyUO5YfZs8tc0TFlS1RALj7TcBN08Z9Im/4Y8DHyhmDiEitOdSV2UaOv49fyzIb5DOZd+JVvjZaV2aLiJRZNufkckF6yOUN58svPZ1iO3lb9Lc8nFvHxuwrKxTl7JQoRETK7LiP38Qff+V3ALzzG/dw2+/3zpjngV19k8NnRYLTtX+e/ghp4pUJsgglChGRCtja3Q/APU/3HnLeV0S2sSO3nD0sLXdYc6JEISJSQ4wcL488zr25F1U7lElKFCIiNeRUe5olNsR9uZOrHcokJQoRkRpxku3i35NBw9B7dEQhIiLT/XE0OOH9vvSHeY7a6a6orNdRiIgsdpls7pDzNJHiH+Nf5vXRzWzOncCvcmdWILK5U6IQETlCtz++l3ue3s+qJU0AvOustZPTvnPXjsnhO54o3Pv1f4k8zOujm7krewp/k/nTssZaCiUKEZEjdPm375/yOj9R/PUvDnZhd+m37iu4/LmRLYx6gkvHr2a8BnfLOkchIlJFy+jnHdE7+Lfsq2oySYAShYhIVb0iso2Y5fjXKnf8V4wShYhIFZ0Z+T0jnuQRX1ftUGalRCEiMs9S49kpz7NpZ4jLYrewOXcCmRotO4EShYjIvBtIjQMwmMoUne89seAuDLflzih7TEdCiaIET/cMsXnngWqHISJV4O784L5d3P3UfgD2DKRmzPP5Xz3Oxod2MxgmjNmcHXmMbbnVfCd7XllinS9KFCX4o7+/g7d/9a5qhyEiVbCrd4Srf/Iwl3z9HgBuuG/XjHl+vLmbD97wIHsGxmZdT1tsnJfaU9yRO61ssc4XJQoRkcNwYGTqUUL/6OxHDc/1jQLwyuOC+12ffHQrAGeuW8pDr7qPhGVn9OkUmXbz7B2ffSPHdjYfadhHRIlCROQwDExLDAOjGY5pbyAW7uHbGw/eaOi5A0GiSMaCXW1XaxKAiGexB7+HN3XOSBSxSO3tlmsvIhGRGpZ/gjo1nmUgNU5bY5y2MEG0NR5svbQ7PKKIRYNd7UQSWZd+HMYG4IIvkCI5Zf3R6YcUNUCJQkTkMAzknaAeTGUYGA0SRWtDkCDaGvKOKMJEkQgTRUsyhpHjZWP3BjOsf/WM9ceitZcoarfhbh3I5rwms7+IlE9+6WkgNc5AKsPKjkZG01kSschkmQmCI4olkRFePnArJ8ce44+6B7kqsZXVIz2w4jSsuXPG+uPR2vv9vugTRfeBEZa3Ncz44+RyTiqTpSkRfEVDYxlaklO/rnQmR2Miesj3cHeG09nJ5fPXNbFxRSPG3oEUsWiEnDudLckZ68nmPNjwmhMzYpEjM/HdrlrSiNnB5D88lqEpEWUskyNiRiLcCaQzObI5n/z7Z7I5MjnHPfj7jmWyHNPeSCRiDI1liEWMaMSIRyMMpMZJxiIYRixipDJZ3IMdRN9omuZwm9szkKIxEaW9Mc541okYNCVi7B1MsaK9kd19oxzd1jC5fHMyxvBYhnQmRyIWoTkZIzWeZTAVfAYIauWZnJPNOTl3cjkYTmc4pqOR5/tHaUrE6BtJ09mSpDkZYzSdZSSdIRaJEIlAa0Oc4bEMZtAYj3JgZJze4TSrlzayfyhNW2McAxKxYDueGNeciDKSPnjxWe9wmvam+OSv79F0lmQswuh4ltHx4H9ib9hiKB411i5rZu9AioFUhuVtSbI5ZzCVYWlzYvJzN4f/E3sHUwyMZuhqTeLuHBgZZ1lLgmQsQjIWnfyORsezHNPRyHN9ozTFo/SNjpPNOQBrlzWxd3CM5kSUSMQYGcuScyc1nmVX7wgAEXI8set5GNrLCUujvMSf5ILEU7y4fzfLEjtYZT00DKZJJjKwB7JRY3D0GO731WxsfgdXXXxVwW0xVoM/Phf13uau7ft45zfu5b9tWMXnL3wpANv3DmJmfO/unXznrh1867INDI1l+eAND7Lx/edw6qqOyeXHMlkaE1F6BsfY1TtC30iaPzyxa7IeOeFHm7r5yxu38oULT2V9ZzMXXns3n33bH7C8rYHLv3M/bz9jFee95Gjee/2myWUe/uTrac07hAX47C+38fXfPAPAN969gdeesrxcX82i8w+3PsGXbt/O205fyecvPJVYNELvcJoz/u+tfPQNJ/GFmx/n2M5mPnfhqRzX1cLVN27llsf28Phfn8fwWJazP3MbY5kcrz6xizvDrqQ/+oaTeMX6pVx47d0AnL6mg/NfcjR/e9PvAWhORFnR0cj2vUOHHe8fntg1o8vqy165jlsefYHd/UG7/gtftoqHu/t5fM9gSet75yvWcNeTPfT29tBl/axqynLecUkeeHQbR9PLKR3j5Ab3Yp7liQLrjJElToY4GZY0xRgIWwuZBTvjPRFY2dFEBOf5/hTJmDE4lsEIpk/sLg0n3RAPr0lwBvPG7wdaG2IMpzIsbw8SQ89ACgd25a1nGEjGbDIRTqw7kwwSx8C093w6fD1cIJbLgfcnUyznAJGfOecDPHXwc/f50Tzoy9mUO5FhGmlobiN+/Llcs7mB6y46k/dev4l3Hr8G2ldNLvPydUu4f0dwbdaZ65fy863PF/+DVdiiThTdYYuErd39k+Ne+8U7gYNZ/c++c3Dn/VB3/7REEdyQ5K1f+d3kuv7na0/kQ689Ycr7/PKR4I/+0R9vnRx39U8enhy+8YFu+kbSU5bZN5SekSiezNuh/ObJHiWKefTM/mEAfvLgc6zvbOYDrzmBvYPBDvenDz4HwNP7hnnHtXdz8tGt/P6FYHe1eecBPv2zxya3had7Dv6Nug+MErGDF2Y+uKuPB3f1Tb4eTmeLJonLXrmO7XuH+O32fTOmFbqvQf59DyBoy2+z/DhtIsUq62GF9bKUATqf6ufUaJqORI5TWkaw/p2s2tLDNfSRbAhP3maBJ+CScLMcGGqkP9pBc1MT+4bGcJ/6HjmMNHHGiTE0mp3S7LOjKclgapwd+4P/GwdGMo6HuyQzY82yJpKx4Kjl+YExWhuSdDQleLZ3BMdY2pxg//A4faPB8vv7wDFaks10tjawY1+wm1/e1sALAyl83GAcwGiIR2hKxukeGmNZSwNDYxk6W5MsbUoEf5eeYXJ5n8eBzpYGmpMxdvenOG39CnY1Hs2wNZGNNnDc6pWkO45jsGU9K7qW0f5cP1/45n288rhlfPQNJ7Gus5lTzxrgpas7+PkHXsUJy1sm133zh1/NyiWNDI9lGM/m6GxJcuW5x7GivXGy6e27zl7Lp3722GSrqUpb1Ili4BBXTR7K2Hiwc5hIEgA7wh3O4eqdliimN8EDZvwjyvzJvwvZRNKY+PtON5EkAPpGxqe8HhrLcNGG1dy3o7ek7evGK8/mtNVLgKD1Sy7nvPUrv+Oh7n4+8voTufyc9TQlgp3nkqY4I+ks0bCsNZjK0BiPEs2NQd9OxvY+RWJkD/GRF/DeHVjfThjag40NYKO9Bd/fiWAsx9euJd36EiJtR5NtXU6uqQtPtpOKNtO4bDXZ5uU4cZbHo0FdPizRTexcJ/o4mhg3lMrQlAzKX0bQCig1np0sRzXGo4znciRjkcnmoRPn/7qAo0bStCRjxKIR1oRvErGgZDiQCkq5Q2NBQmtrCOZbG84XjRgvygVlo3QmR1MyimHEozb5PeZ8amujFdM+jwFmQQJ70SznJpuBJeHwGWuWsOUTr5tSXXjp6uBH5ktWtk9Z7qTw2or8cvKLjwnmWdqcAODyc9Zz+TnrC/7NKmGRJ4pgwyq0A57tl1i+sUzxDr8OruvQK+ufdhFPoT5iRtLF+42R0hX6vid29MXyc9+0v1vfyDgdzUFTycFUZk7bUb4lTYkpO6FIxCbPg3Q0BfV4clmWjr8AzzxD84Fn4MAOGNzD0r6dwfDAbsBJTK7FgjJHx1pY/XJIth583b4KmjqhpQvizVg0NrHElEabE2fiJtYZBxrypk+cH5hosNOct9OLGrQ3TT06BmiIR2mIHzzH18js5/s6mg5+mvzvJxa1yZ3p0lhiyjL580UjRnMyRvO0H+QTy05vaDT988y23mKml6DrWVkThZmdB/wTwXb2DXf/7LTpSeB64GXAfuAid99RzpjyTfxqn/gl4of4yT42np0yz0S5YT70Tb+Ip8Cv0YFRJYpyKfZ9j4zN/N4jBjmHvtH0jGltDXHaGmIM5J0cnavJHWJmDPp2Qe/TnDf0a94Qe4bXPTgK9z8PfTshm/e+kTg0d0HHmqC55ZL1sGQdLF0PbccE02LVKVnIwlC2RGFmUeDLwOuAbuB+M9vo7o/lzfYe4IC7H29mFwOfAy4qV0zTTewcpicMgPHsxD+4A0YLIwwNjzKWydHAGKuth1zfs7DMiJCjhREaSdM23gCpAYg3QXS2r9dJEJy0i5GliTFio85xNhLWjAdpf/o5iK0IdgKRGERjnDyymVMiPSy1AU7uWQfbngGLQiQaPkcgmoSGdog1BDuHhvZgOgTTZo1pcSuUhCe3jwJHGxP7/+lHghCUPtoa4zzXN5qXgJxmUrQwSqf102FDNJJmqQ2wjEEaLUUn/XT86NrgqKD/WSaOZS4DhqIN+Ng6WHEynHxBkAyWHhsmg5UH/8YiZVDOvcaZwHZ3fxrAzH4AvAXITxRvAT4ZDv8Y+JKZmR/qp/08SQ0PcX38MxzPc/i3X0Sk8Ri+HN9Jl/Uz5nHW2F66rB8Hmm0M7gF/oJmHkmMkLQM/CtbzdP4x+HZg4rgpmoB4E18Yj0FyjCTjRHASjBOzQxyNPBg+8vwTHDz27wZ+eJgfONYITUsPJpdINEhCydYwIUXDQmwULBK+jkCiOUh8sWSQgJJt4XAyWLbtmGBn1XJU3nqiB5c/3PpLFUzs0JOkYbQfRnpJ9++hiwNEx3IssQxtjJAgQ4wsSRtnGf2c/OwmLo/2kGScdhumhVFeu9V5zYGdJFL7aNqU4wPJNEnGidvspcqcGwdoITJ+Eqw5C5ZeMpkM/vsv9nPzzhzff9NZnHP8zHb3IuVWzkSxEng273U38IrZ5nH3jJn1A8uAmc08jtDW/7yRtjuvmTLur7IjLI/0clPuFazt7uWo3OO8MpLiKT+GTuvnCV/FXbkXM0qSfm8mHo+y3FOks8Ns9WNZ0hhnSXSU1HA/g95IigSxaJSjG3M0kKLBx2jIpsiOD5PKRRklQY4IaWKMeBIwMkQYoYEsEYa9gRd8KftpY0ljjM5GI0qWqGeJkmXXgRS9tLHfW1mZGGVtqxMhFzw8eE4wTosPEWecpKdp9mEMx3CWeS9NoyNEyBEN54+RoclHiZAiSg7Dp6wvQo4mRkn6GHHGSXiaRmbvEbOQLBH6rJ0hC1p6GPlHa1ObHk6Z7oWnz2nZWcYfTFk+ZZ5bc05DMk2rjcIzwOfhUuDS/B8BhTwPbw3L72mPMkAzDcNH8ULDcu4cXk3Okwxng795vzczRBO5pi6eGkowSoID3so+2hkjDhg73vvGGW+Rbbof2KuLO6VqypkoCm3V048U5jIPZnYFcAXAmjVrSgom0dxOb9PMVgOPrnkDt2TPIZMLfuGPpLNEzGiMR9k3NIZZcMKrdzjN0uYEjwG5XFCmGmyMsQswjH1DY0TMWNIcZ2+B9+8dTpNz6GpJsn94bHKdE+MSsQhXvvpYfrrlORpiUZ7cOzj5ZWTCxzGrI7xxdQe/ebJnyoV+ufABkAYOv1X+4Yl4hpiPE/NxGnNDLMn0sGS8h9bsASJkJ5OMhc9Rz9CS7aMpN4Tn7dqDzzfxzNTxNn2+qdOnLjt1M5q57CzvkbecmbGkrZVtg0HJLmdRckQYTDuNyQTDGSMTb2UgHaG5qRGLJVh61Eq27nOykSQDaRj1BMvbG/mrN51CqneE397xFDl3eofTRMxoa4hz/FEtnLG2g+/dvZM2M5YDqUywzf3xaSsLft9/+9Y/4MTlO3j5uqWz/k1EysnKVeUxs7OBT7r7G8LXHwNw98/kzXNzOM/dZhYDXgC6ipWeNmzY4Js2bZptsoiIFGBmm919QynLlrP91v3ACWa23swSwMXAxmnzbCQ4wge4EPh1pc5PiIjI3JSt9BSec3g/cDNB89hvufujZvZpYJO7bwS+CXzPzLYDvQTJREREakhZ20q6+03ATdPGfSJvOAW8o5wxiIjIkVk4lw6KiEhZKFGIiEhRShQiIlKUEoWIiBSlRCEiIkWV7YK7cjGzHmBniYt3UobuQSpAcVdOPcYM9Rl3PcYM9Rl3J9Ds7l2lLFx3ieJImNmmUq9MrCY2wHe9AAAGeklEQVTFXTn1GDPUZ9z1GDPUZ9xHGrNKTyIiUpQShYiIFLXYEsV11Q6gRIq7cuoxZqjPuOsxZqjPuI8o5kV1jkJERA7fYjuiEBGRw7RoEoWZnWdmj5vZdjO7utrx5DOzb5nZXjN7JG/cUjO71cyeDJ+XhOPNzP45/BxbzeyMKsW82sxuN7NtZvaomX2o1uM2swYzu8/MHgpj/lQ4fr2Z3RvG/MOwW3zMLBm+3h5OX1fpmKfFHzWzB83s5/USt5ntMLOHzWyLmW0Kx9XsNhLG0WFmPzaz34fb99l1EPNJ4Xc88Rgwsw/PW9zuvuAfBN2cPwUcS3DX6YeAU6odV158rwbOAB7JG/d54Opw+Grgc+HwBcAvCW7TdhZwb5ViXgGcEQ63Ak8Ap9Ry3OF7t4TDceDeMJZ/BS4Ox18LXBkO/w/g2nD4YuCHVd5O/hfw/4Cfh69rPm5gB9A5bVzNbiNhHN8F/jwcTgAdtR7ztPijBDeBWztfcVf1A1XwizsbuDnv9ceAj1U7rmkxrpuWKB4HVoTDK4DHw+GvAZcUmq/K8f878Lp6iRtoAh4guI/7PiA2fVshuJfK2eFwLJzPqhTvKuA24I+An4f/4PUQd6FEUbPbCNBGcNd0mza+ZmMu8BleD/xuPuNeLKWnlcCzea+7w3G1bLm7Pw8QPh8Vjq+5zxKWNk4n+IVe03GH5ZstwF7gVoIjzT53zxSIazLmcHo/sKyyEU/6R+AvOXh79GXUR9wO3GJmm83sinBcLW8jxwI9wLfDMt83zKyZ2o55uouBG8LheYl7sSQKKzCuXpt71dRnMbMW4Ebgw+4+UGzWAuMqHre7Z939NIJf6GcCLyo0W/hcEzGb2ZuAve6+OX90gVlrKu7QOe5+BnA+cJWZvbrIvLUQd4ygDPxVdz8dGCYo2cymFmKeFJ6nejPwo0PNWmDcrHEvlkTRDazOe70K2F2lWOZqj5mtAAif94bja+azmFmcIEl8391/Eo6u+bgB3L0P+E+C+myHmU3c7TE/rsmYw+ntBLfsrbRzgDeb2Q7gBwTlp3+k9uPG3XeHz3uBfyNIzrW8jXQD3e5+b/j6xwSJo5Zjznc+8IC77wlfz0vciyVR3A+cELYSSRAcmm2sckyHshG4NBy+lOAcwMT4d4etFs4C+icOLSvJzIzgnufb3P2LeZNqNm4z6zKzjnC4EXgtsA24HbhwlpgnPsuFwK89LOhWkrt/zN1Xufs6gm331+7+J9R43GbWbGatE8MEtfNHqOFtxN1fAJ41s5PCUa8BHqvlmKe5hINlJ5ivuKt50qXCJ3guIGiZ8xTwv6sdz7TYbgCeB8YJMv17CGrKtwFPhs9Lw3kN+HL4OR4GNlQp5lcRHKpuBbaEjwtqOW7gVODBMOZHgE+E448F7gO2ExyyJ8PxDeHr7eH0Y2tgWzmXg62eajruML6HwsejE/93tbyNhHGcBmwKt5OfAktqPeYwliZgP9CeN25e4taV2SIiUtRiKT2JiEiJlChERKQoJQoRESlKiUJERIpSohARkaKUKGTRM7PstJ43i/YubGbvM7N3z8P77jCzziNdj0i5qXmsLHpmNuTuLVV43x0E7df3Vfq9RQ6HjihEZhH+4v+cBfewuM/Mjg/Hf9LMPhIOf9DMHgv79P9BOG6pmf00HHePmZ0ajl9mZreEnc19jbz+dszsT8P32GJmXzOzaBU+skhBShQi0Dit9HRR3rQBdz8T+BJB/0rTXQ2c7u6nAu8Lx30KeDAc93Hg+nD8NcBvPehsbiOwBsDMXgRcRNCB3mlAFviT+f2IIqWLHXoWkQVvNNxBF3JD3vM/FJi+Ffi+mf2UoLsHCLo3eTuAu/86PJJoJ7hB1dvC8b8wswPh/K8BXgbcH3ShRSMHO28TqTolCpHifJbhCW8kSABvBv7KzF5M8S6cC63DgO+6+8eOJFCRclHpSaS4i/Ke786fYGYRYLW7305wU6EOoAW4k7B0ZGbnAvs8uFdH/vjzCTqbg6CztgvN7Khw2lIzW1vGzyRyWHREIRKeo8h7/St3n2gimzSzewl+VF0ybbko8C9hWcmAf3D3PjP7JMEd0rYCIxzs5vlTwA1m9gBwB7ALwN0fM7P/Q3AnuAhBL8JXATvn+4OKlELNY0VmoearIgGVnkREpCgdUYiISFE6ohARkaKUKEREpCglChERKUqJQkREilKiEBGRopQoRESkqP8Pd2bcfPFlT7kAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4484d544a8>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# plot the scores\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"plt.plot(np.arange(len(scores)), scores, label='Score')\n",
"plt.plot(np.arange(len(scores_mean)), scores_mean, label='Mean')\n",
"plt.ylabel('Score')\n",
"plt.xlabel('Episode')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Watch a Smart Agent"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Actor net - local: Actor(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fc1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=128, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=2, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Actor net - target: Actor(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fc1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=128, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=2, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Critic net - local: Critic(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fcs1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=132, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=1, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Critic net - target: Critic(\n",
" (bn1): BatchNorm1d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
" (fcs1): Linear(in_features=24, out_features=128, bias=True)\n",
" (fc2): Linear(in_features=132, out_features=128, bias=True)\n",
" (fc3): Linear(in_features=128, out_features=1, bias=True)\n",
" (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
")\n",
"Episode 1000\tAverage Score: 2.19"
]
}
],
"source": [
"import torch.optim as optim\n",
"\n",
"agent = Agent(state_size=24, action_size=2, random_seed=0)\n",
"\n",
"agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
"agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
"\n",
"n_episodes=1000\n",
"NUM_AGENTS=2\n",
"score = np.zeros(NUM_AGENTS)\n",
"\n",
"env_info = env.reset(train_mode=False)[brain_name] # reset env\n",
"states = env_info.vector_observations # get current states\n",
"\n",
"for i_eps in range(1,n_episodes+1):\n",
" \n",
" actions_agent_0 = agent.act(states, noise_factor=0) # let the agent select actions\n",
" actions_agent_1 = np.flip(actions_agent_0, 0) # action of second agent/player\n",
" env_info = env.step(actions_agent_0)[brain_name] # send actions of both agents to env\n",
" rewards = env_info.rewards # get the rewards from env\n",
" next_state_agent_0 = env_info.vector_observations # get next states from env\n",
" next_state_agent_1 = np.flip(next_state_agent_0, 0) # get the resulting states for the second agent/player\n",
" dones = env_info.local_done # check if episode is done\n",
"\n",
" agent.step(states, actions_agent_0, actions_agent_1, rewards, next_state_agent_0, next_state_agent_1, dones) # perform step of agent\n",
"\n",
" # update statistical variables\n",
" states = next_state_agent_0\n",
" score += rewards\n",
" #if np.any(dones):\n",
" # break \n",
"\n",
"print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_eps, np.mean(score)), end=\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}