manual.tex 30.2 KB
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649
\documentclass[12pt]{article}

%% Allows itemranges in enumerations
\def\itemrange#1{%
\addtocounter{enumi}{1}%
\edef\labelenumi{\theenumi--\noexpand\theenumi}%
\addtocounter{enumi}{-1}%
\addtocounter{enumi}{#1}%
\item
\def\labelenumi{\theenumi}}
\renewcommand*{\labelenumi}{\theenumi}

\usepackage{hyperref,graphicx}
\usepackage[cm]{fullpage}
\usepackage{enumitem}
\usepackage{subcaption}

\title{RoboCup 2D Half Field Offense \\ Technical Manual}
\author{Matthew Hausknecht}

\begin{document}

\maketitle
\tableofcontents

\section{Overview}

This document describes the installation, usage, state, and action spaces of the HFO domain.

\section{Installation}

Installation with CMake:

\begin{verbatim}
  > mkdir build && cd build
  > cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo ..
  > make -j4 # Replace 4 with the number of cores on your machine
  > make install # This just copies binaries to the HFO directory; no sudo required
\end{verbatim}

HFO installation has been tested on Ubuntu Linux and OSX. Successful
installation depends on
\verb+CMake, Boost-system, Boost-filesystem,+ and \verb+flex+.

These depedencies can be installed on Ubuntu using the following
command:\\
\verb+sudo apt-get install cmake libboost-filesystem libboost-system flex+

By default, the soccerwindow2 visualizer is also built and requires
\verb+Qt4+. Experimentally speaking, HFO is fully-functional without
the visualizer. To disable this component, use the following cmake
command:\\

\noindent \verb+  > cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo -DBUILD_SOCCERWINDOW=False ..+

\subsection{Python Interface}

The Python interface is required for interfacing Python agents to the
HFO domain. To install this interface, from the main HFO directory:

\noindent \verb+  > pip install .+

or

\noindent \verb+  > pip install --user .+

if you have limited permissions on the machine.

Successful installation depends on
\verb+Python 2.7, 3.2, or above+ (tested with 2.7 and 3.5) and \verb+numpy+.

\section{Uninstall}

The install is completely contained in the HFO directory. Simply
delete this directory to uninstall. If you have installed the python
interface, uninstall it as follows: \verb+pip uninstall hfo+.

\begin{figure}[htp]
  \centering
  \includegraphics[width=\textwidth]{figures/HFODiagram}
  \caption{HFO is comprised of several components which communicate
    over the network. Network connections are depicted with orange
    ovals. Calling the HFO executable starts the trainer, visualizer,
    and all the offensive and defensive npcs (Agent2d) as well as the
    offensive and defensive agent servers. Your code then uses the HFO
    interface to connect your agent to the server. Once all agents are
    connected, the game begins. The trainer oversees the game.}
  \label{fig:hfo}
\end{figure}

\section{Basic Usage}

RoboCup 2D soccer is designed to be played between two teams of
autonomous agents who communicate with a game server. Shown in Figure
\ref{fig:hfo}, the HFO domain reflects these design choices and allows
arbitrary teams to be created consisting of some mix of
non-player-controlled agents (agent2d npcs) and player-controlled
agents. These options are specified through the following flags:\\

\noindent
\verb+ > ./bin/HFO --offense-agents=1 --defense-agents=1 --offense-npcs=2 --defense-npcs=2+\\

This would create a 3v3 game with one player-controlled agent on each
team. In order for the game to start, you must connect your
player-controlled agents to the server. This is done through the
call:\\

\noindent \verb+  > hfo.connectToServer(FEATURE_SET, port, etc);+\\

The arguments to this function are provided by the HFO executable for
each player upon starting the game. For example
\verb|./bin/HFO --offense-agents=1| prints all the information needed
to connect the offensive player:\\

\noindent \verb+Waiting for player-controlled agent base_left-0: +\\
\noindent \verb+  config_dir=HFO/bin/teams/base/config/formations-dt,+\\
\noindent \verb+  server_port=6000, server_addr=localhost,+\\
\noindent \verb+  team_name=base_left, play_goalie=False+\\

By default, the server starts on port 6000, but may be changed as follows:\\

\noindent \verb+  > ./bin/HFO --port 12345+

\section{Visualizer}

The SoccerWindow2 Visualizer allows a live game to be viewed as it
progresses. By default, the visualizer is enabled. However, the game
will likely proceed at a pace too fast for meaningful watching. To
enforce a standard pace, disable sync-mode:\\

\noindent \verb+  > ./bin/HFO --no-sync+\\

To disable visualization altogether, run in headless mode:\\

\noindent \verb+  > ./bin/HFO --headless+\\

The visualizer may also be used after the end of a game by replaying
logs, as discussed in the next section.

\section{Logging}

By default, the soccer server generates game logs and stores them in
the \verb+log+ directory. The main game logs are rcg files:
\verb+log/*.rcg+. These log may be replayed using the soccerwindow2
visualizer. \\

\noindent To replay a log: \\
\verb+  > ./bin/soccerwindow2 -l log/incomplete.rcg+

\noindent To disable logging:\\
\verb+  > ./bin/HFO --no-logging +

\noindent To change the logging directory:\\
\verb+  > ./bin/HFO --log-dir /path/to/new/dir +

\section{Making Videos}
It is possible to make videos from logs by saving frames from
SoccerWindow2. It helps to full-screen SoccerWindow2 before making a
video as it will save higher quality images. There are also several
display options under View $\rightarrow$ View Preference $\rightarrow$
Show that toggle what will be displayed. Saving frames can be done by
File $\rightarrow$ Save Image. To convert the saved pngs into a
movie:\\

\noindent \verb+ avconv -r 10 -start_number 0 -i 3v3/image-%05d.png -f mp4 -c:v libx264+\\
\noindent \verb+ -s 1024x768 -vf "crop=iw/2.5:8.38*ih/10:iw/2:ih/10,transpose=1" +\\
\noindent \verb+ -pix_fmt yuv420p test.mp4 +\\

This command autocrops offensive half of the playfield and rotates it
90 degrees. Avconv can be replaced by ffmpeg. Start number specifies
the number of the first frame. yuv420p pix format for OSX
compatibility.

\section{Recording}

It is possible to record the state perceptions (low- or high-level depending on
the player), low-level actions, and game status of all players:\\

\noindent \verb+  > ./bin/HFO --record + \\

This will produce logs for all the offensive players
(\verb+log/left-[1-11].log+) and defensive players
(\verb+log/right-[1-11].log+). The first offensive player is left-11,
so in the case of single-agent offense, left-11.log will contain the
active player's record. Note that for player controlled agents, it is
necessary to specify a \verb+record_dir+ in the \verb+connectToServer+
function:\\
\\
\noindent \verb+std::string record_dir = "log/";+\\
\noindent \verb+hfo.connectToServer(features, config_dir, port, server_addr,+\\
\noindent \verb+                    team_name, goalie, record_dir);+\\

\section{Randomness}
A seed may be specified as follows:\\

\noindent \verb+  > ./bin/HFO --seed 123+\\

This seed will determine the placement of the players and the ball at
the beginning of each episode. Due to non-determinism in the player
policies, it is not sufficient to precisely replicate full games. It
\textit{only} replicates the starting conditions for each episode. The
player's behavior, observations, and physics all proceed
stochastically.

\section{Player On Ball}
By default, episodes begin with the ball being randomly positioned in
the offensive half of the playfield. Typically the first task for the
offense is to send a player to collect the ball. It is possible to
instead request that a certain offensive player is given the ball at
the start of each episode. This is accomplished as follows:\\

\noindent \verb+  > ./bin/HFO --offense-on-ball 1+\\

The above command will always give the ball to the first offensive
player (e.g. uniform number 11). If an offense-on-ball number is
specified that is larger than the number of offensive players, the
ball will be given to a random offensive player at the start of each
episode.

\section{Teams}
By default, offensive and defensive NPCs use the base Agent2D
policy. It is possible to use policies from different teams as
follows:\\

\noindent \verb+  > ./bin/HFO --offense-team helios+\\
\noindent \verb+  > ./bin/HFO --defense-team base+\\

This would take offense NPCs from Helios' 2013 Eindhoven release and
defensive NPCs from the default Agent2D-base. Currently the only
supported teams are Helios and Base.

\section{Communication}
HFO allows agents to receive and broadcast messages. This is
accomplished by the \verb hear \ and \verb say \ functions. The
maximum allowed message size is controlled by HFO's
\verb|--message-size| flag. See
\verb|examples/communication_agent.cpp| and
\verb|examples/communication_agent.py| for examples.

\section{Fullstate}
By default, perceptions and actions in HFO are noisy. The
\verb+ --fullstate+ flag in HFO removes noise from the agent's
perception of the world. Many tasks become significantly easier as a
result. Noise in actions remains. This flag is disabled by default.

\section{Controlling Trials}
HFO trials typically end with a goal, the defense capturing the ball,
the ball going out of bounds, or running out of time. The trials flag
specifies a maximum number of trials
\verb+ > ./bin/HFO --trials 500+. Instead, a maximum number of frames
may be specified: \verb+ > ./bin/HFO --frames 1000+ will stop the
server after 10,000 steps have passed. Each trial is run for a maximum
of \verb --frames-per-trial \ steps, but may stop early if no agent
approaches the ball within \verb --untouched-time \ steps.

\section{State Spaces}
The HFO domains provides a choice between a low and a high-level
feature set. Selecting between the different feature sets is
accomplished when connecting the agent to the server:

\begin{verbatim}
  > hfo.connectToServer(LOW_LEVEL_FEATURE_SET, ...);
  > hfo.connectToServer(HIGH_LEVEL_FEATURE_SET, ...);
\end{verbatim}

See \verb|examples/hfo_example_agent.cpp| and
\verb|examples/hfo_example_agent.py| for examples. As the choice of
feature set influences the challenge of learning, it is the
responsibility of the user to faithfully report which state space was
used. The following sections explain the feature sets.

\subsection{High Level Feature Set}
A set of high-level features is provided following the example given
by Barrett et al. pp. 159-160 \cite{THESIS14-Barrett}. Barrett writes
``There are many ways to represent the state of a game of half field
offense.  Ideally, we want a compact representation that allows the
agent to learn quickly by generalizing its knowledge about a state to
similar states without over-constraining the policy.'' All features
are encoded a floating point values and normalized to the range of
[-1,1]. Invalid features are given a value of -2. The features are as
follows:

\subsubsection{High Level State Feature List}
Let $T$ denote the number of teammates in the HFO game and $O$ the
number of opponents. There are a total of $10 + 6T + 3O$ high-level
features.

\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
\item{\textbf{X position} - The agent’s x-position on the field. See
  Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Y position} - The agent’s y-position on the field. See
  Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Orientation} - The global direction that the agent is facing.}
\item{\textbf{Ball X} - The ball's x-position on the field.}
\item{\textbf{Ball Y} - The ball's y-position on the field.}
\item{\textbf{Able to Kick} - Boolean indicating if the agent can kick the ball.}
\item{\textbf{Goal Center Proximity} - Agent's proximity to the center of the goal.}
\item{\textbf{Goal Center Angle} - Angle from the agent to the center of the goal.}
\item{\textbf{Goal Opening Angle} - The size of the largest open angle
  of the agent to the goal, shown as $\theta_g$ in Figure
  \ref{fig:openAngle}. Invalid if agent is not playing offense.}
\item{\textbf{Proximity to Opponent} - If an opponent is present,
  proximity to the closest opponent. Invalid if there are no
  opponents.}
\item [$T$] {\textbf{Teammate's Goal Opening Angle} - For each
  teammate $i$: $i$’s goal opening angle. Invalid if agent is not
  playing offense.}
\item [$T$] {\textbf{Proximity from Teammate i to Opponent} - For each
  teammate i: the proximity from the teammate to the closest
  opponent. This feature is invalid if there are no opponents or if
  teammates are present but not detected.}
\item [$T$] {\textbf{Pass Opening Angle} - For each teammate i: the open
  angle available to pass to teammate i. Shown as $\theta_p$ in Figure
  \ref{fig:openAngle}. If teammates are present but not detected, this
  feature is considered invalid and given the value of -2.}
\item [$3T$] {\textbf{X, Y, and Uniform Number of
    Teammates} - For each teammate: the x-position, y-position and
  uniform number of that teammate.}
\item [$3O$] {\textbf{X, Y, and Uniform Number of
    Opponents} - For each opponent: the x-position, y-position and
  uniform number of that opponent.}
\end{enumerate}

\begin{figure}[htp]
  \centering
  \includegraphics[width=.7\textwidth]{figures/playfieldCoords}
  \caption{\textbf{Normalized Coordinates in the HFO play field}:
    These coordinates are used for reporting the agent's position in
    the high-level feature set as well specifying targets for the
    mid-level actions (Section \ref{sec:mid_level_actions}). The
    red-rectangle shows the boundaries of the reported positions,
    which exceed the play field boundaries by 10\% in each
    direction. Positions exceeding this rectangle are bounded (via
    min/max) to the edges of the rectangle. All distance features are
    normalized against the max HFO distance shown in orange.}
  \label{fig:playfieldCoords}
\end{figure}

\begin{figure}[htp]
  \centering
  \includegraphics[width=.5\textwidth]{figures/openAngle}
  \caption{Open angle from ball to the goal $\theta_g$ avoiding the
    blue goalie and the open angle from the ball to the yellow
    teammate $\theta_p$. Figure reproduced with permission from Samuel
    Barrett.}
  \label{fig:openAngle}
\end{figure}

\subsection {Low Level Feature Set}
The state features used by HFO are designed with the mindset of
providing an overcomplete, basic, egocentric viewpoint. The features
are basic in the sense that they provide distances and angles to
relevant points of interest, but do not include higher level
perceptions such as the largest angle between a goal post and
goalkeeper.

All features are encoded as floating point values normalized to the
range of [-1,1]. Several different types of features exist:

\subsubsection{Boolean Features}
Boolean features assume either the minimum feature value of -1 or the
maximum feature value of 1.

\subsubsection{Valid Features}
Since feature information is attained from the Agent's world-model, it
is possible that, the world model's information may be stale or
incorrect. \textit{Valid features} are boolean features indicating
consistency of world model predictions. For example, if the world
model's estimate of the agent's position is known to be flawed, the
\textit{valid feature} for self position would assume the minimum
value of -1. Otherwise it will assume the maximum value of 1.

The features associated with a valid feature are given the value of
zero if an inconsistency is detected. For example, if the world model
detects that the agent's velocity is invalid, the feature that encodes
the magnitude of self velocity will be set to zero.

\subsubsection{Angular Features}
\textit{Angular features} (e.g. the angle to the ball), are encoded as
two floating point numbers -- the $sin(\theta)$ and $cos(\theta)$
where $\theta$ is the original angle in radians. Figure
\ref{fig:ang_example} provides examples of the angular encoding.

This encoding allows the angle to vary smoothly for all possible
angular values. Other encodings such as radians or degrees have a
discontinuity that when normalized, could cause the feature value to
flip between the maximum and minimum value in response to small
changes in $\theta$.

Given an angular feature $\langle \alpha_1, \alpha_2 \rangle$ we can
recover the original angle $\theta$ (in radians) by taking the
$cos^{-1}(\alpha_2)$ and multiplying by the sign of $\alpha_1$.
Another method uses the common 'atan2' function as
$atan2(\alpha_1, \alpha_2)$.

\begin{figure*}[htp]
  \centering
  \subcaptionbox{Angular Encoding}{
    \includegraphics[width=.4\textwidth]{figures/AngExample}
  }
  \hspace{3em}
  \subcaptionbox{Additional Examples}{
    \includegraphics[width=.3\textwidth]{figures/AngFeatExample}
  }
  \caption{\textbf{Angular Encoding:} Objects on the agents left/right
    side result in a negative/positive $sin(\theta)$. $cos(\theta)$ is
    positive in front of the player and negative behind. For example,
    an object directly in front of the player would have angular
    features of $sin(\theta)=0, cos(\theta)=1$. Additional examples:
    \textbf{Angle to ball} $\theta=60^\circ$ or $1.0472$ radians. This
    results in angular features $\langle sin(\theta)=.86,
    cos(\theta)=.49 \rangle$. \textbf{Angle to teammate}:
    $\theta=135^\circ, 2.35$ radians. $\langle sin(\theta)=.71,
    cos(\theta)=-.71 \rangle$. \textbf{Angle to Opponent}:
    $\theta=-90^\circ$ or $-1.57$ radians. $\langle sin(\theta)=-1,
    cos(\theta)=0 \rangle$.}
  \label{fig:ang_example}
\end{figure*}

\subsubsection{Proximity Features}
\textit{Proximity features} encode the proximity of the agent to an
object of interest. Unless otherwise indicated, they are normalized
against the maximum possible distance in the HFO playfield (defined as
$\sqrt{l^2 + w^2}$ where $l,w$ are the length and width of the HFO
playfield). A maximum proximity of 1 indicates the agent is co-located
with the object of interest, while a minimum proximity of -1 indicates
that the agent is across the field from the object of interest.

\subsubsection{Landmark Features}
Landmark features encode the relative angle and proximity of the agent
to a landmark of interest. Each landmark feature consists of three
floating point values, two to encode the agent's relative angle to the
landmark and one to encode the landmark's proximity. Note that if the
agent's self position is invalid, then the landmark feature values are
zeroed.

\subsubsection{Player Features}
Player features are used to encode the relationship of the agent to
another player or opponent. Each player feature is encoded as 1) a
landmark feature of that player's location 2) the global angle of that
player's body 3) the magnitude of the player's velocity and 4) the
global angle of the player's velocity. Eight floating point numbers
are used to encode each player feature.

\subsubsection{Uniform Number Features}
In the low-level feature space, unknown uniform numbers, or \textit{unums},
are encoded as -1, while known ones are encoded as $\frac{unum}{100}$, thus
remaining well within the $[-1, 1]$ range. (Note that roundoff error may need
to be allowed for when converting these back to integers, such as for use in
high-level actions; e.g., 0.0799 will need to be converted back to 8.)
Uniform number features, a later addition to the low-level feature space,
are positioned after all other features to hopefully ensure compatibility
with older programs.

\subsubsection{Other Features}
Some features, such as the agent's stamina, do not fall into any of
the above categories. These features are referred to as \textit{other
  features} and are normalized in the range $[-1, 1]$.

\subsubsection{Low Level State Feature List}
Let $T$ denote the number of teammates and $O$ denote the number of
opponents in the HFO game. Then there are a total of $58 + 9T + 9O$
low-level features:

\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
  \item{\textbf{Self\_Pos\_Valid} [Valid] Indicates if self position is valid.}
  \item{\textbf{Self\_Vel\_Valid} [Valid] Indicates if the agent's velocity is valid.}
  \itemrange{1}{\textbf{Self\_Vel\_Ang} [Angle] Angle of agent's velocity.}
  \item{\textbf{Self\_Vel\_Mag} [Other] Magnitude of the agent's velocity.}
  \itemrange{1}{\textbf{Self\_Ang} [Angle] Agent's Global Body Angle.}
  \item{\textbf{Stamina} [Other] Agent's Stamina: Low stamina slows movement.}
  \item{\textbf{Frozen} [Boolean] Indicates if the agent is Frozen. Frozen status can
    happen when tackling or being tackled by another player.}
  \item{\textbf{Colliding\_with\_ball} [Boolean] Indicates the agent
    is colliding with the ball.}
  \item{\textbf{Colliding\_with\_player} [Boolean] Indicates the agent
    is colliding with another player.}
  \item{\textbf{Colliding\_with\_post} [Boolean] Indicates the agent
    is colliding with a goal post.}
  \item{\textbf{Kickable} [Boolean] Indicates the agent is able to
    kick the ball.}
  \itemrange{2}{\textbf{Goal Center} [Landmark] Center point between the goal posts.}
  \itemrange{2}{\textbf{Goal Post Top} [Landmark] Top goal post.}
  \itemrange{2}{\textbf{Goal Post Bot} [Landmark] Bottom goal post.}
  \itemrange{2}{\textbf{Penalty Box Center} [Landmark] Center of the penalty box line.}
  \itemrange{2}{\textbf{Penalty Box Top} [Landmark] Top corner of the penalty box.}
  \itemrange{2}{\textbf{Penalty Box Bot} [Landmark] Bottom corner of the penalty box.}
  \itemrange{2}{\textbf{Center Field} [Landmark] The left middle point of the
    HFO play area.}
  \itemrange{2}{\textbf{Corner Top Left} [Landmark] Top left corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Top Right} [Landmark] Top right corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Bot Right} [Landmark] Bot right corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Bot Left} [Landmark] Bot left corner HFO Playfield.}
  \item{\textbf{OOB Left Dist} [Proximity] Proximity to the nearest
    point of the left side of the HFO playable area. E.g. distance
    remaining before the agent goes out of bounds in left field.}
  \item{\textbf{OOB Right Dist} [Proximity] Proximity to the right
    field line.}
  \item{\textbf{OOB Top Dist} [Proximity] Proximity to the top field line.}
  \item{\textbf{OOB Bot Dist} [Proximity] Proximity to the bottom field line.}
  \item{\textbf{Ball Pos Valid} [Valid] Indicates the ball position estimate is valid.}
  \itemrange{1}{\textbf{Ball Angle} [Angle] Agent's angle to the ball.}
  \item{\textbf{Ball Dist} [Proximity] Proximity to the ball.}
  \item{\textbf{Ball Vel Valid} [Valid] Indicates the ball velocity estimate is valid.}
  \item{\textbf{Ball Vel Mag} [Other] Magnitude of the ball's velocity.}
  \itemrange{1}{\textbf{Ball Vel Ang} [Angle] Global angle of ball velocity.}
  \item [$8T$] {\textbf{Teammate Features} [Player] One teammate feature set (8 features) for each teammate active in HFO, sorted by proximity to the agent.}
  \item [$8O$] {\textbf{Opponent Features} [Player] One opponent feature set (8 features) for each opponent present, sorted by proximity to the player.}
  \item [$T$]  {\textbf{Teammate Uniform Nums} [Unum] One uniform number for each teammate active in HFO, sorted by proximity to the agent.}
  \item [$O$]  {\textbf{Opponent Uniform Nums} [Unum] One uniform number for each opponent present, sorted by proximity to the player.}
\end{enumerate}

\section{Action Space}
The HFO domain provides support for both low-level primitive actions,
mid-level, and high-level strategic actions. Low-level, parameterized
actions are provided for locomotion and kicking. Mid-level actions are
still parameterized by capture high level activities such as
dribbling. Finally, high-level discrete, strategic actions are
available for moving, shooting, passing and dribbling. Control of the
agent's head and gaze is not provided and follows Agent2D's default
strategy. Low, medium, and high level actions are available through
the same interface. As the choice of action spaces greatly influences
the challenge of learning, it is the responsibility of the user to
faithfully report which action spaces were used.

\subsection{Low Level Actions}
\label{sec:low_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Dash}(power, degrees): Moves the agent with power [-100,
    100] where negative values move backwards. The relative direction
  of movement is given in degrees and varies between [-180,180] with 0
  degrees being a forward dash and 90 degrees dashing to the agent's
  right side. Note, dashing does not turn the agent.}
\item{\textbf{Turn}(degrees): Turns the agent in the
  specified direction. Valid values range between [-180, 180] degrees
  where 90 degrees turns the agent to directly to its right side.}
\item{\textbf{Tackle}(degrees): Contest the ball. Direction
  varies between [-180, 180].}
\item{\textbf{Kick}(power, degrees): Kick the ball with power [0, 100]
  in relative direction [-180, 180]. Has no effect if the agent does
  not possess the ball.}
\end{itemize}

\subsection{Mid Level Actions}
\label{sec:mid_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Kick$\_$To}(target$_x$, target$_y$, speed): Kicks the
  ball to the specified target point with the desired speed. Valid
  values for target$_{x,y} \in [-1,1]$ and speed $\in [0,3]$.}
\item{\textbf{Move$\_$To}(target$_x$, target$_y$): Moves to the
  specified target point using the max dash speed. Valid values for
  target$_{x,y} \in [-1,1]$.}
\item{\textbf{Dribble$\_$To}(target$_x$, target$_y$): Dribbles the
  ball to the specified target point. Attempts to fetch the ball if
  the agent doesn't already possess it. Performs some checks to avoid
  opponents and keeps good control of the ball. Valid values for
  target$_{x,y} \in [-1,1]$.}
\item{\textbf{Intercept}(): Moves to intercept the ball, taking into
  account the ball velocity. More efficient than chasing the ball.}
\end{itemize}

\subsection{High Level Actions}
\label{sec:high_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Move}(): Re-positions the agent according to the
  strategy given by Agent2D. The \textit{move} command works only when
  agent does not have the ball. If the agent has the ball, another
  command such as \textit{dribble}, \textit{shoot}, or \textit{pass}
  should be used.}
\item{\textbf{Shoot}(): Executes the best available shot. This command
  only works when the agent has the ball.}
\item{\textbf{Pass}(teammate\_uniform\_number): Passes to the teammate
  with the provided uniform number. Does nothing if the player does
  not have control of the ball or the requested teammate is not
  detected.}
\item{\textbf{Dribble}(): Advances the ball towards the goal using a
  combination of short kicks and moves.}
\item{\textbf{Catch}(): This goalie-specific action may be used to
  catch the ball.}
\item {\textbf{Reduce\_Angle\_To\_Goal} (): Moves the agent to a point on the field, such that the kicker has the least open angle to the goal. }


\item {\textbf{Defend\_Goal} () : Moves the agent to a point on a fixed line on the field, such that the kicker has the least open angle to the goal.}

\item {\textbf{Go\_To\_Ball} (): Makes the agent go towards the ball.}

\item {\textbf{Mark\_Player} (uniform\_number): Moves the agent so as to mark the player with the specified uniform number.}

\end{itemize}

\subsection{Special Actions}
\begin{itemize}[noitemsep]
\item{\textbf{NO-OP}: Indicates that the agent should take no action.}
\item{\textbf{Quit}: Indicates to the agent server that you wish to
  terminate the HFO environment.}
\end{itemize}

\subsection{Available Actions}
The Special Actions are always available. The below table indicates whether
other actions are available (only if there are no ``N''s indicated); check
below the table for the action abbreviations.

\begin{center}
{\footnotesize
\begin{tabular}{r         c     c      c    c     c     c      c    c    c    c    c     c     c     c     c     c     c}
Action                  & Da & Tu  &  Ta & K   & KT &  MT &  DT &  I  & M  & S  & P  &  Dr  & C  &  RG &  DG  & G  &  MP \\
\hline
Self position invalid   & Y  &  Y  &  Y  &  Y  & N  &  N  &  N? &  N? & N? & N? & ?  &  Y   & ?  &  N? &  N?  & ?  &  ? \\
Self velocity invalid   & N  &  Y? &  Y? &  Y? & ?  &  N  &  ?  &  N? & N? & N? & ?  &  Y   & ?  &  ?  &  ?   & ?  &  ? \\
Ball position invalid   & Y  &  Y  &  Y? &  N  & N  &  Y  &  N? &  N  & ?  & N  & N  &  Y?  & N  &  ?  &  ?   & N  &  ? \\
Ball velocity invalid   & Y  &  Y  &  Y  &  ?  & ?  &  Y  &  ?  &  N? & ?  & N  & N? &  Y?  & N? &  Y? &  Y?  & Y? &  Y? \\
Teammate loc invalid    & Y  &  Y  &  Y  &  Y  & Y  &  Y  &  Y  &  Y  & ?  & Y  & N  &  Y?  & Y  &  Y  &  Y   & Y  &  Y \\
Team. unum invalid      & Y  &  Y  &  Y  &  Y  & Y  &  Y  &  Y  &  Y  & Y  & Y  & N  &  Y   & Y  &  Y  &  Y   & Y  &  Y \\
Opponent loc invalid    & Y  &  Y  &  Y  &  Y  & Y  &  Y  &  Y? &  Y  & ?  & Y? & Y  &  Y?  & Y  &  N  &  N   & Y  &  N \\
Opp. unum invalid       & Y  &  Y  &  Y  &  Y  & Y  &  Y  &  Y  &  Y  & Y  & Y  & Y  &  Y   & Y  &  Y  &  Y   & Y  &  N \\
Ball kickable           & Y  &  Y  &  Y  &  Y  & Y  &  Y  &  Y  &  N? & N  & Y  & Y  &  Y   & Y  &  N? &  N?  & N? &  Y? \\
Ball not kickable       & Y  &  Y  &  Y  &  N  & N  &  Y  &  Y  &  Y  & Y  & N  & N  &  N   & Y  &  Y  &  Y   & Y  &  Y \\
Frozen                  & N  &  N  &  N  &  N  & N  &  N  &  N? &  N  & N? & N  & N  &  Y   & N? &  N  &  N   & N  &  N \\
Colliding w/ball        & ?  &  ?  &  ?  &  ?  & ?  &  ?  &  ?  &  N? & ?  & ?  & ?  &  ?   & ?  &  ?  &  ?   & ?  &  ? \\
Colliding w/player      & ?  &  ?  &  ?  &  N? & N? &  ?  &  ?  &  N? & ?  & ?  & ?  &  Y?  & ?  &  ?  &  ?   & ?  &  ? \\
Colliding w/post        & ?  &  ?  &  N? &  N? & N? &  ?  &  ?  &  N? & ?  & ?  & ?  &  ?   & ?  &  ?  &  ?   & ?  &  ? \\
Offense                 & Y  &  Y  &  N  &  Y  & Y  &  Y  &  Y  &  Y  & Y  & Y  & Y  &  Y   & N  &  N  &  N   & Y  &  N \\
Defense, not goalie     & Y  &  Y  &  Y  &  N? & N? &  Y  &  N  &  Y  & Y  & N  & N  &  N   & N  &  Y  &  ?   & Y  &  Y \\
Goalie (defense)        & Y  &  Y  &  Y? &  N? & N? &  Y  &  N  &  Y  & ?  & N  & N  &  N   & Y  &  ?  &  ?   & ?  &  ? \\
\end{tabular}
}
\end{center}
Da: Dash; Tu: Turn; Ta: Tackle; K: Kick \\
KT: Kick\_To; MT: Move\_To; DT: Dribble\_To; I: Intercept \\
M: Move; S: Shoot; P: Pass; Dr: Dribble; C: Catch; RG: Reduce\_Angle\_To\_Goal; DG: Defend\_Goal; G: Go\_To\_Ball; MP: Mark\_Player

\section{Developing a New Agent}

New agents may be developed in C++ or Python. In Python, as long as
the hfo interface has been installed, the agent needs only to
either \verb+from hfo import *+ or \verb+import hfo+. In C++ it is necessary to
\verb+#include <HFO.hpp>+ and also to link against the shared object
library \verb+lib/libhfo.so+ when compiling:

\begin{verbatim}
  > g++ example/your_new_agent.cpp -I src -L lib -Wl,-rpath=lib -lhfo
\end{verbatim}

\bibliographystyle{abbrv}
\bibliography{manual}

\end{document}