manual.tex

\documentclass[12pt]{article}

%% Allows itemranges in enumerations
\def\itemrange#1{%
\addtocounter{enumi}{1}%
\edef\labelenumi{\theenumi--\noexpand\theenumi}%
\addtocounter{enumi}{-1}%
\addtocounter{enumi}{#1}%
\item
\def\labelenumi{\theenumi}}
\renewcommand*{\labelenumi}{\theenumi}

\usepackage{hyperref,graphicx}
\usepackage[cm]{fullpage}
\usepackage{enumitem}
\usepackage{subcaption}

\title{RoboCup 2D Half Field Offense \\ Technical Manual}
\author{Matthew Hausknecht}

\begin{document}

\maketitle
\tableofcontents

\section{Overview}

This document describes the installation, usage, state, and action spaces of the HFO domain.

\section{Installation}

Installation with CMake:

\begin{verbatim}
  > mkdir build && cd build
  > cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo ..
  > make -j4 # Replace 4 with the number of cores on your machine
  > make install # This just copies binaries to the HFO directory; no sudo required
\end{verbatim}

HFO installation has been tested on Ubuntu Linux and OSX. Successful
installation depends on
\verb+CMake, Boost-system, Boost-filesystem,+ and \verb+flex+.

These depedencies can be installed on Ubuntu using the following
command:\\
\verb+sudo apt-get install cmake libboost-filesystem libboost-system flex+

By default, the soccerwindow2 visualizer is also built and requires
\verb+Qt4+. Experimentally speaking, HFO is fully-functional without
the visualizer. To disable this component, use the following cmake
command:\\

\noindent \verb+  > cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo -DBUILD_SOCCERWINDOW=False ..+

\subsection{Python Interface}

The Python interface is required for interfacing Python agents to the
HFO domain. To install this interface, from the main HFO directory:

\noindent \verb+  > pip install .+

or

\noindent \verb+  > pip install --user .+

if you have limited permissions on the machine.

Successful installation depends on
\verb+Python 2.7, 3.2, or above+ (tested with 2.7 and 3.5) and \verb+numpy+.

\section{Uninstall}

The install is completely contained in the HFO directory. Simply
delete this directory to uninstall. If you have installed the python
interface, uninstall it as follows: \verb+pip uninstall hfo+.

\begin{figure}[htp]
  \centering
  \includegraphics[width=\textwidth]{figures/HFODiagram}
  \caption{HFO is comprised of several components which communicate
    over the network. Network connections are depicted with orange
    ovals. Calling the HFO executable starts the trainer, visualizer,
    and all the offensive and defensive npcs (Agent2d) as well as the
    offensive and defensive agent servers. Your code then uses the HFO
    interface to connect your agent to the server. Once all agents are
    connected, the game begins. The trainer oversees the game.}
  \label{fig:hfo}
\end{figure}

\section{Basic Usage}

RoboCup 2D soccer is designed to be played between two teams of
autonomous agents who communicate with a game server. Shown in Figure
\ref{fig:hfo}, the HFO domain reflects these design choices and allows
arbitrary teams to be created consisting of some mix of
non-player-controlled agents (agent2d npcs) and player-controlled
agents. These options are specified through the following flags:\\

\noindent
\verb+ > ./bin/HFO --offense-agents=1 --defense-agents=1 --offense-npcs=2 --defense-npcs=2+\\

This would create a 3v3 game with one player-controlled agent on each
team. In order for the game to start, you must connect your
player-controlled agents to the server. This is done through the
call:\\

\noindent \verb+  > hfo.connectToServer(FEATURE_SET, port, etc);+\\

The arguments to this function are provided by the HFO executable for
each player upon starting the game. For example
\verb|./bin/HFO --offense-agents=1| prints all the information needed
to connect the offensive player:\\

\noindent \verb+Waiting for player-controlled agent base_left-0: +\\
\noindent \verb+  config_dir=HFO/bin/teams/base/config/formations-dt,+\\
\noindent \verb+  server_port=6000, server_addr=localhost,+\\
\noindent \verb+  team_name=base_left, play_goalie=False+\\

By default, the server starts on port 6000, but may be changed as follows:\\

\noindent \verb+  > ./bin/HFO --port 12345+

\section{Visualizer}

The SoccerWindow2 Visualizer allows a live game to be viewed as it
progresses. By default, the visualizer is enabled. However, the game
will likely proceed at a pace too fast for meaningful watching. To
enforce a standard pace, disable sync-mode:\\

\noindent \verb+  > ./bin/HFO --no-sync+\\

To disable visualization altogether, run in headless mode:\\

\noindent \verb+  > ./bin/HFO --headless+\\

The visualizer may also be used after the end of a game by replaying
logs, as discussed in the next section.

\section{Logging}

By default, the soccer server generates game logs and stores them in
the \verb+log+ directory. The main game logs are rcg files:
\verb+log/*.rcg+. These log may be replayed using the soccerwindow2
visualizer. \\

\noindent To replay a log: \\
\verb+  > ./bin/soccerwindow2 -l log/incomplete.rcg+

\noindent To disable logging:\\
\verb+  > ./bin/HFO --no-logging +

\noindent To change the logging directory:\\
\verb+  > ./bin/HFO --log-dir /path/to/new/dir +

Note that the logs can become quite large for a prolonged series of
episodes.

\section{Making Videos}
It is possible to make videos from logs by saving frames from
SoccerWindow2. It helps to full-screen SoccerWindow2 before making a
video as it will save higher quality images. There are also several
display options under View $\rightarrow$ View Preference $\rightarrow$
Show that toggle what will be displayed. Saving frames can be done by
File $\rightarrow$ Save Image. To convert the saved pngs into a
movie:\\

\noindent \verb+ avconv -r 10 -start_number 0 -i 3v3/image-%05d.png -f mp4 -c:v libx264+\\
\noindent \verb+ -s 1024x768 -vf "crop=iw/2.5:8.38*ih/10:iw/2:ih/10,transpose=1" +\\
\noindent \verb+ -pix_fmt yuv420p test.mp4 +\\

This command autocrops offensive half of the playfield and rotates it
90 degrees. Avconv can be replaced by ffmpeg. Start number specifies
the number of the first frame. yuv420p pix format for OSX
compatibility.

\section{Recording}

It is possible to record the state perceptions (low- or high-level depending on
the player), low-level actions, and game status of all players:\\

\noindent \verb+  > ./bin/HFO --record + \\

This will produce logs for all the offensive players
(\verb+log/left-[1-11].log+) and defensive players
(\verb+log/right-[1-11].log+). The first offensive player is left-11,
so in the case of single-agent offense, left-11.log will contain the
active player's record. Note that for player controlled agents, it is
necessary to specify a \verb+record_dir+ in the \verb+connectToServer+
function:\\
\\
\noindent \verb+std::string record_dir = "log/";+\\
\noindent \verb+hfo.connectToServer(features, config_dir, port, server_addr,+\\
\noindent \verb+                    team_name, goalie, record_dir);+\\

\section{Randomness}
A seed may be specified as follows:\\

\noindent \verb+  > ./bin/HFO --seed 123+\\

This seed will determine the placement of the players and the ball at
the beginning of each episode. Due to non-determinism in the player
policies, it is \textbf{not} sufficient to precisely replicate full games. It
\textit{only} replicates the starting conditions for each episode. The
players' behavior, observations, and physics all proceed
stochastically.

\section{Player On Ball}
By default, episodes begin with the ball being randomly positioned in
the offensive half of the playfield. Typically the first task for the
offense is to send a player to collect the ball. It is possible to
instead request that a certain offensive player is given the ball at
the start of each episode. This is accomplished as follows:\\

\noindent \verb+  > ./bin/HFO --offense-on-ball 1+\\

The above command will always give the ball to the first offensive
player (e.g. uniform number 11). If an offense-on-ball number is
specified that is larger than the number of offensive players, the
ball will be given to a random offensive player at the start of each
episode.

\section{Teams}
By default, offensive and defensive NPCs use the base Agent2D
policy. It is possible to use policies from different teams as
follows:\\

\noindent \verb+  > ./bin/HFO --offense-team helios+\\
\noindent \verb+  > ./bin/HFO --defense-team base+\\

This would take offense NPCs from Helios' 2013 Eindhoven release and
defensive NPCs from the default Agent2D-base. Currently the only
supported teams are Helios (on Linux only) and Base.

\section{Communication}
HFO allows agents to receive and broadcast messages. This is
accomplished by the \verb hear \ and \verb say \ functions. The
maximum allowed message size is controlled by HFO's
\verb|--message-size| flag. See
\verb|examples/communication_agent.cpp| and
\verb|examples/communication_agent.py| for examples.

\section{Fullstate}
By default, perceptions and actions in HFO are noisy. The
\verb|--fullstate| flag in HFO removes noise from the agent's
perception of the world. Many tasks become significantly easier as a
result. Noise in actions remains. This flag is disabled by default.

\section{Controlling Trials}
HFO trials typically end with a goal, the defense capturing
the ball, the ball going out of bounds, or running out of time.
The trials flag specifies a maximum number of
trials: \verb|> ./bin/HFO --trials 500|.
Instead, a maximum number of frames may be specified; for
instance, \verb|--frames 1000| will stop the
server after 1,000 steps have passed. Each trial is run for a
maximum of \verb|--frames-per-trial| steps, but may stop early
if no agent approaches the ball within \verb|--untouched-time| steps.

\section{State Spaces}
The HFO domains provides a choice between a low and a high-level
feature set. Selecting between the different feature sets is
accomplished when connecting the agent to the server:

\begin{verbatim}
  > hfo.connectToServer(LOW_LEVEL_FEATURE_SET, ...);
  > hfo.connectToServer(HIGH_LEVEL_FEATURE_SET, ...);
\end{verbatim}

See \verb|examples/hfo_example_agent.cpp| and
\verb|examples/hfo_example_agent.py| for examples. As the choice of
feature set influences the challenge of learning, it is the
responsibility of the user to faithfully report which state space was
used. The following sections explain the feature sets.

\subsection{High Level Feature Set}
A set of high-level features is provided following the example given
by Barrett pp. 159-160 \cite{THESIS14-Barrett}. Barrett writes:
``There are many ways to represent the state of a game of half field
offense.  Ideally, we want a compact representation that allows the
agent to learn quickly by generalizing its knowledge about a state to
similar states without over-constraining the policy.'' All features
are encoded a floating point values and normalized to the range of
[-1,1]. Invalid features are given a value of -2. The features are as
follows:

\subsubsection{High Level State Feature List}
Let $T$ denote the number of teammates in the HFO game and $O$ the
number of opponents. There are a total of $10 + 6T + 3O$ high-level
features.

\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
\item{\textbf{X position} - The agent’s x-position on the field. See
  Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Y position} - The agent’s y-position on the field. See
  Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Orientation} - The global direction that the agent is facing.}
\item{\textbf{Ball X} - The ball's x-position on the field.}
\item{\textbf{Ball Y} - The ball's y-position on the field.}
\item{\textbf{Able to Kick} - Boolean indicating if the agent can kick the ball.}
\item{\textbf{Goal Center Proximity} - Agent's proximity to the center of the goal.}
\item{\textbf{Goal Center Angle} - Angle from the agent to the center of the goal.}
\item{\textbf{Goal Opening Angle} - The size of the largest open angle
  of the agent to the goal, shown as $\theta_g$ in Figure
  \ref{fig:openAngle}. Invalid if agent is not playing offense.}
\item{\textbf{Proximity to Opponent} - If an opponent is present,
  proximity to the closest opponent. Invalid if there are no
  opponents.}
\item [$T$] {\textbf{Teammate's Goal Opening Angle} - For each
  teammate $i$: $i$’s goal opening angle. Invalid if agent is not
  playing offense.}
\item [$T$] {\textbf{Proximity from Teammate i to Opponent} - For each
  teammate i: the proximity from the teammate to the closest
  opponent. This feature is invalid if there are no opponents or if
  teammates are present but not detected.}
\item [$T$] {\textbf{Pass Opening Angle} - For each teammate i: the open
  angle available to pass to teammate i. Shown as $\theta_p$ in Figure
  \ref{fig:openAngle}. If teammates are present but not detected, this
  feature is considered invalid and given the value of -2.}
\item [$3T$] {\textbf{X, Y, and Uniform Number of
    Teammates} - For each teammate: the x-position, y-position and
  uniform number of that teammate.}
\item [$3O$] {\textbf{X, Y, and Uniform Number of
    Opponents} - For each opponent: the x-position, y-position and
  uniform number of that opponent.}
\end{enumerate}

\begin{figure}[htp]
  \centering
  \includegraphics[width=.7\textwidth]{figures/playfieldCoords}
  \caption{\textbf{Normalized Coordinates in the HFO play field}:
    These coordinates are used for reporting the agent's position in
    the high-level feature set as well specifying targets for the
    mid-level actions (Section \ref{sec:mid_level_actions}). The
    red-rectangle shows the boundaries of the reported positions,
    which exceed the play field boundaries by 10\% in each
    direction. Positions exceeding this rectangle are bounded (via
    min/max) to the edges of the rectangle. All distance features are
    normalized against the max HFO distance shown in orange.}
  \label{fig:playfieldCoords}
\end{figure}

\begin{figure}[htp]
  \centering
  \includegraphics[width=.5\textwidth]{figures/openAngle}
  \caption{Open angle from ball to the goal $\theta_g$ avoiding the
    blue goalie and the open angle from the ball to the yellow
    teammate $\theta_p$. Figure reproduced with permission from Samuel
    Barrett.}
  \label{fig:openAngle}
\end{figure}

\subsection {Low Level Feature Set}
The state features used by HFO are designed with the mindset of
providing an overcomplete, basic, egocentric viewpoint. The features
are basic in the sense that they provide distances and angles to
relevant points of interest, but do not include higher level
perceptions such as the largest angle between a goal post and
goalkeeper.

All features are encoded as floating point values normalized to the
range of [-1,1]. Several different types of features exist:

\subsubsection{Boolean Features}
Boolean features assume either the minimum feature value of -1 or the
maximum feature value of 1.

\subsubsection{Valid Features}
Since feature information is attained from the Agent's world-model, it
is possible that, the world model's information may be stale or
incorrect. \textit{Valid features} are boolean features indicating
consistency of world model predictions. For example, if the world
model's estimate of the agent's position is known to be flawed, the
\textit{valid feature} for self position would assume the minimum
value of -1. Otherwise it will assume the maximum value of 1.

The features associated with a valid feature are given the value of
zero if an inconsistency is detected. For example, if the world model
detects that the agent's velocity perception is invalid, the feature
that encodes the magnitude of self velocity will be set to zero.

\subsubsection{Angular Features}
\textit{Angular features} (e.g. the angle to the ball), are encoded as
two floating point numbers -- the $sin(\theta)$ and $cos(\theta)$
where $\theta$ is the original angle in radians. Figure
\ref{fig:ang_example} provides examples of the angular encoding.

This encoding allows the angle to vary smoothly for all possible
angular values. Other encodings such as radians or degrees have a
discontinuity that when normalized, could cause the feature value to
flip between the maximum and minimum value in response to small
changes in $\theta$.

Given an angular feature $\langle \alpha_1, \alpha_2 \rangle$ we can
recover the original angle $\theta$ (in radians) by taking the
$cos^{-1}(\alpha_2)$ and multiplying by the sign of $\alpha_1$.
Another method, possibly preferable in the presence of noise, uses
the common 'atan2' function as $atan2(\alpha_1, \alpha_2)$.

\begin{figure*}[htp]
  \centering
  \subcaptionbox{Angular Encoding}{
    \includegraphics[width=.4\textwidth]{figures/AngExample}
  }
  \hspace{3em}
  \subcaptionbox{Additional Examples}{
    \includegraphics[width=.3\textwidth]{figures/AngFeatExample}
  }
  \caption{\textbf{Angular Encoding:} Objects on the agents left/right
    side result in a negative/positive $sin(\theta)$. $cos(\theta)$ is
    positive in front of the player and negative behind. For example,
    an object directly in front of the player would have angular
    features of $sin(\theta)=0, cos(\theta)=1$. Additional examples:
    \textbf{Angle to ball} $\theta=60^\circ$ or $1.0472$ radians. This
    results in angular features $\langle sin(\theta)=.86,
    cos(\theta)=.49 \rangle$. \textbf{Angle to teammate}:
    $\theta=135^\circ, 2.35$ radians. $\langle sin(\theta)=.71,
    cos(\theta)=-.71 \rangle$. \textbf{Angle to Opponent}:
    $\theta=-90^\circ$ or $-1.57$ radians. $\langle sin(\theta)=-1,
    cos(\theta)=0 \rangle$.}
  \label{fig:ang_example}
\end{figure*}

\subsubsection{Proximity Features}
\textit{Proximity features} encode the proximity of the agent to an
object of interest. Unless otherwise indicated, they are normalized
against the maximum possible distance in the HFO playfield (defined as
$\sqrt{l^2 + w^2}$ where $l,w$ are the length and width of the HFO
playfield). A maximum proximity of 1 indicates the agent is co-located
with the object of interest, while a minimum proximity of -1 indicates
that the agent is across the field from the object of interest.

\subsubsection{Landmark Features}
Landmark features encode the relative angle and proximity of the agent
to a landmark of interest. Each landmark feature consists of three
floating point values, two to encode the agent's relative angle to the
landmark and one to encode the landmark's proximity. Note that if the
agent's self position is invalid, the landmark feature values are
zeroed.

\subsubsection{Player Features}
Player features are used to encode the relationship of the agent to
another player or opponent. Each player feature is encoded as 1) a
landmark feature of that player's location 2) the global angle of that
player's body 3) the magnitude of the player's velocity and 4) the
global angle of the player's velocity. Eight floating point numbers
are used to encode each player feature.

\subsubsection{Uniform Number Features}
In the low-level feature space, unknown uniform numbers, or \textit{unums},
are encoded as -1, while known ones are encoded as $\frac{unum}{100}$, thus
remaining well within the $[-1, 1]$ range. (Note that roundoff error may need
to be allowed for when converting these back to integers, such as for use in
high-level actions; \textit{e.g.}, 0.0799 will need to be converted back to 8.)
Uniform number features, a later addition to the low-level feature space,
are positioned after all other features to hopefully ensure compatibility
with older programs.

\subsubsection{Other Features}
Some features, such as the agent's stamina, do not fall into any of
the above categories. These features are referred to as \textit{other
  features} and are normalized in the range $[-1, 1]$.

\subsubsection{Low Level State Feature List}
Let $T$ denote the number of teammates and $O$ denote the number of
opponents in the HFO game. Then there are a total of $58 + 9T + 9O$
low-level features:

\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
  \item{\textbf{Self\_Pos\_Valid} [Valid] Indicates if self position is valid.}
  \item{\textbf{Self\_Vel\_Valid} [Valid] Indicates if the agent's velocity is valid.}
  \itemrange{1}{\textbf{Self\_Vel\_Ang} [Angle] Angle of agent's velocity.}
  \item{\textbf{Self\_Vel\_Mag} [Other] Magnitude of the agent's velocity.}
  \itemrange{1}{\textbf{Self\_Ang} [Angle] Agent's Global Body Angle.}
  \item{\textbf{Stamina} [Other] Agent's Stamina: Low stamina slows movement.}
  \item{\textbf{Frozen} [Boolean] Indicates if the agent is Frozen. Frozen status can
    happen when tackling or being tackled by another player.}
  \item{\textbf{Colliding\_with\_ball} [Boolean] Indicates the agent
    is colliding with the ball.}
  \item{\textbf{Colliding\_with\_player} [Boolean] Indicates the agent
    is colliding with another player.}
  \item{\textbf{Colliding\_with\_post} [Boolean] Indicates the agent
    is colliding with a goal post.}
  \item{\textbf{Kickable} [Boolean] Indicates the agent is able to
    kick the ball.}
  \itemrange{2}{\textbf{Goal Center} [Landmark] Center point between the goal posts.}
  \itemrange{2}{\textbf{Goal Post Top} [Landmark] Top goal post.}
  \itemrange{2}{\textbf{Goal Post Bot} [Landmark] Bottom goal post.}
  \itemrange{2}{\textbf{Penalty Box Center} [Landmark] Center of the penalty box line.}
  \itemrange{2}{\textbf{Penalty Box Top} [Landmark] Top corner of the penalty box.}
  \itemrange{2}{\textbf{Penalty Box Bot} [Landmark] Bottom corner of the penalty box.}
  \itemrange{2}{\textbf{Center Field} [Landmark] The left middle point of the RoboCup field
    (note that this is \textbf{not} the center of the HFO play area).}
  \itemrange{2}{\textbf{Corner Top Left} [Landmark] Top left corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Top Right} [Landmark] Top right corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Bot Right} [Landmark] Bottom right corner HFO Playfield.}
  \itemrange{2}{\textbf{Corner Bot Left} [Landmark] Bottom left corner HFO Playfield.}
  \item{\textbf{OOB Left Dist} [Proximity] Proximity to the nearest
    point of the left side of the HFO playable area. E.g. distance
    remaining before the agent goes out of bounds in left field.}
  \item{\textbf{OOB Right Dist} [Proximity] Proximity to the right
    field line.}
  \item{\textbf{OOB Top Dist} [Proximity] Proximity to the top field line.}
  \item{\textbf{OOB Bot Dist} [Proximity] Proximity to the bottom field line.}
  \item{\textbf{Ball Pos Valid} [Valid] Indicates the ball position estimate is valid.}
  \itemrange{1}{\textbf{Ball Angle} [Angle] Agent's angle to the ball.}
  \item{\textbf{Ball Dist} [Proximity] Proximity to the ball.}
  \item{\textbf{Ball Vel Valid} [Valid] Indicates the ball velocity estimate is valid.}
  \item{\textbf{Ball Vel Mag} [Other] Magnitude of the ball's velocity.}
  \itemrange{1}{\textbf{Ball Vel Ang} [Angle] Global angle of ball velocity.}
  \item [$8T$] {\textbf{Teammate Features} [Player] One teammate feature set (8 features)
	for each teammate active in HFO, sorted by proximity to the agent.}
  \item [$8O$] {\textbf{Opponent Features} [Player] One opponent feature set (8 features)
	for each opponent active in HFO, sorted by proximity to the player.}
  \item [$T$]  {\textbf{Teammate Uniform Nums} [Unum] One uniform number for each teammate active in HFO,
	sorted by proximity to the agent.}
  \item [$O$]  {\textbf{Opponent Uniform Nums} [Unum] One uniform number for each opponent active in HFO,
	sorted by proximity to the player.}
\end{enumerate}

\section{Action Space}
The HFO domain provides support for both low-level primitive actions,
mid-level, and high-level strategic actions. Low-level, parameterized
actions are provided for locomotion and kicking. Mid-level actions are
still mostly parameterized but capture high-level activities such as
dribbling. Finally, high-level discrete, strategic actions are
available for moving, shooting, passing and dribbling. Control of the
agent's head and gaze is not provided and follows Agent2D's default
strategy. Low, medium, and high level actions are available through
the same interface. As the choice of action spaces greatly influences
the challenge of learning, it is the responsibility of the user to
faithfully report which action spaces were used.

\subsection{Low Level Actions}
\label{sec:low_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Dash}(power, degrees): Moves the agent with power [-100,
    100] where negative values move backwards. The relative direction
  of movement is given in degrees and varies between [-180,180] with 0
  degrees being a forward dash and 90 degrees dashing to the agent's
  right side. Note, dashing does not turn the agent.}
\item{\textbf{Turn}(degrees): Turns the agent in the
  specified direction. Valid values range between [-180, 180] degrees
  where 90 degrees turns the agent to directly to its right side.}
\item{\textbf{Tackle}(degrees): Contest the ball. Direction
  varies between [-180, 180].}
\item{\textbf{Kick}(power, degrees): Kick the ball with power [0, 100]
  in relative direction [-180, 180]. Has no effect if the agent does
  not possess the ball.}
\end{itemize}

\subsection{Mid Level Actions}
\label{sec:mid_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Kick$\_$To}(target$_x$, target$_y$, speed): Kicks the
  ball to the specified target point with the desired speed. Valid
  values for target$_{x,y} \in [-1,1]$ and speed $\in [0,3]$.}
\item{\textbf{Move$\_$To}(target$_x$, target$_y$): Moves to the
  specified target point using the max dash speed. Valid values for
  target$_{x,y} \in [-1,1]$.}
\item{\textbf{Dribble$\_$To}(target$_x$, target$_y$): Dribbles the
  ball to the specified target point. Attempts to fetch the ball if
  the agent doesn't already possess it. Performs some checks to avoid
  opponents and keeps good control of the ball. Valid values for
  target$_{x,y} \in [-1,1]$.}
\item{\textbf{Intercept}(): Moves to intercept the ball, taking into
  account the ball velocity. More efficient than chasing the ball.}
\end{itemize}

\subsection{High Level Actions}
\label{sec:high_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Move}(): Re-positions the agent according to the
  strategy given by Agent2D. The \textit{move} command works only when
  agent does not have the ball. If the agent has the ball, another
  command such as \textit{dribble}, \textit{shoot}, or \textit{pass}
  should be used.}
\item{\textbf{Shoot}(): Executes the best available shot. This command
  only works when the agent has the ball.}
\item{\textbf{Pass}(teammate\_uniform\_number): Passes to the teammate
  with the provided uniform number. Does nothing if the player does
  not have control of the ball or the requested teammate is not
  detected.}
\item{\textbf{Dribble}(): Advances the ball towards the goal using a
  combination of short kicks and moves.}
\item{\textbf{Catch}(): This goalie-specific action may be used to
  catch the ball.}
\item{\textbf{Reduce\_Angle\_To\_Goal}(): Moves the agent to a point on the field,
	such that the kicker has the least open angle to the goal. }
\item{\textbf{Defend\_Goal}(): Moves the agent to a point on a fixed line on the field,
	such that the kicker has the least open angle to the goal.}
\item{\textbf{Go\_To\_Ball}(): Makes the agent go towards the ball.}
\item{\textbf{Mark\_Player}(uniform\_number): Moves the agent so as to mark the player
	with the specified uniform number.}

\end{itemize}

\subsection{Special Actions}
\begin{itemize}[noitemsep]
\item{\textbf{NO-OP}: Indicates that the agent should take no action.}
\item{\textbf{Quit}: Indicates to the agent server that you wish to
  terminate the HFO environment.}
\end{itemize}

\subsection{Applicable Actions}
The Special Actions are always applicable (potentially appropriate). The below table indicates whether
other actions are applicable (only if there are no ``N''s indicated); check
below the table for the action abbreviations and notes.

\begin{center}
{\footnotesize
\begin{tabular}{r       | c    c    c    c | c    c    c    c | c   c   c   c   c   c    c    c    c}
Action                  & Da & Tu & Ta & K & KT & MT & DT & I & M & S & P & D & C & RG & DG & G & MP \\
\hline \hline
Self position invalid   & Y  & Y  & Y  & Y & N  & N  & N  & N & N & N & N & Y & Y & N  & N  & N & N \\
Self velocity invalid   & N  & Y  & Y  & Y & N  & N  & N  & N & N & N & Y & Y & Y & N  & N  & N & N \\
Ball position invalid   & Y  & Y  & Y  & N & N  & Y  & Y  & N & N & N & N & Y & N & N  & N  & N & N \\
Ball velocity invalid   & Y  & Y  & Y  & Y & N  & Y  & N  & Y & Y & N & N & Y & Y & Y  & Y  & Y & Y \\
Teammate loc invalid    & Y  & Y  & Y  & Y & Y  & Y  & Y  & Y & Y & Y & N & Y & Y & Y  & Y  & Y & Y \\
Team. unum invalid      & Y  & Y  & Y  & Y & Y  & Y  & Y  & Y & Y & Y & N & Y & Y & Y  & Y  & Y & Y \\
Opponent loc invalid    & Y  & Y  & Y  & Y & Y  & Y  & N  & Y & N & Y & Y & N & Y & Y  & Y  & Y & N \\
Opp. unum invalid       & Y  & Y  & Y  & Y & Y  & Y  & Y  & Y & Y & Y & Y & Y & Y & Y  & Y  & Y & N \\
\hline
Ball kickable           & Y  & Y  & Y  & Y & Y  & N  & Y  & N & * & Y & Y & Y & Y & N  & N  & N & Y \\
Ball not kickable       & Y  & Y  & Y  & N & N  & Y  & Y  & Y & Y & N & N & N & Y & Y  & Y  & Y & Y \\
\hline
Frozen                  & N  & N  & N  & N & N  & N  & N  & N & N & N & N & N & N & N  & N  & N & N \\
Colliding w/ball        & Y  & Y  & N  & N & Y  & N  & Y  & Y & Y & Y & Y & Y & Y & N  & N  & N & N \\
Colliding w/player      & Y  & Y  & Y  & Y & Y  & Y  & Y  & Y & Y & Y & N & Y & Y & Y  & Y  & Y & Y \\
Colliding w/post        & Y  & Y  & Y  & Y & Y  & Y  & N  & Y & Y & Y & N & Y & Y & Y  & Y  & Y & Y \\
\hline
Offense                 & Y  & Y  & N  & Y & Y  & Y  & Y  & Y & Y & Y & Y & Y & N & N  & N  & Y & N \\
Defense, not goalie     & Y  & Y  & Y  & N & N  & Y  & N  & Y & Y & N & N & N & N & Y  & Y  & Y & Y \\
Goalie (defense)        & Y  & Y  & Y  & N & N  & Y  & N  & Y & Y & N & N & N & Y & N  & Y  & N & N \\
\end{tabular}
}
\end{center}
*: The Move command is not recommended for offensive players able to kick the ball, but can work for defensive players (usually by triggering a tackle).\\
\begin{itemize}[noitemsep]
\item{Da:\,Dash; Tu:\,Turn; Ta:\,Tackle; K:\,Kick}
\item{KT:\,Kick\_To; MT:\,Move\_To; DT:\,Dribble\_To; I:\,Intercept}
\item{M:\,Move; S:\,Shoot; P:\,Pass; D:\,Dribble; C:\,Catch; RG:\,Reduce\_Angle\_To\_Goal; DG:\,Defend\_Goal; G:\,Go\_To\_Ball; MP:\,Mark\_Player}
\end{itemize}

\section{Developing a New Agent}

New agents may be developed in C++ or Python. In Python, as long as
the hfo interface has been installed, the agent only needs to
\verb+import hfo+ (or \verb+from hfo import *+). In C++, it is necessary to
\verb+#include <HFO.hpp>+ and also link against the shared object
library \verb+lib/libhfo.so+ when compiling:

\begin{verbatim}
  > g++ example/your_new_agent.cpp -I src -L lib -Wl,-rpath=lib -lhfo
\end{verbatim}

\bibliographystyle{abbrv}
\bibliography{manual}

\end{document}