1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
\documentclass[12pt]{article}
%% Allows itemranges in enumerations
\def\itemrange#1{%
\addtocounter{enumi}{1}%
\edef\labelenumi{\theenumi--\noexpand\theenumi}%
\addtocounter{enumi}{-1}%
\addtocounter{enumi}{#1}%
\item
\def\labelenumi{\theenumi}}
\renewcommand*{\labelenumi}{\theenumi}
\usepackage{hyperref,graphicx}
\usepackage[cm]{fullpage}
\usepackage{enumitem}
\usepackage{subcaption}
\title{RoboCup 2D Half Field Offense \\ Technical Manual}
\author{Matthew Hausknecht}
\begin{document}
\maketitle
\tableofcontents
\section{Overview}
This document describes the installation, usage, state, and action spaces of the HFO domain.
\section{Installation}
Installation with CMake:
\begin{verbatim}
> mkdir build && cd build
> cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo ..
> make -j4 # Replace 4 with the number of cores on your machine
> make install # This just copies binaries to the HFO directory; no sudo required
\end{verbatim}
HFO installation has been tested on Ubuntu Linux and OSX. Successful
installation depends on
\verb+CMake, Boost-system, Boost-filesystem,+ and \verb+flex+.
These depedencies can be installed on Ubuntu using the following
command:\\
\verb+sudo apt-get install cmake libboost-filesystem libboost-system flex+
By default, the soccerwindow2 visualizer is also built and requires
\verb+Qt4+. Experimentally speaking, HFO is fully-functional without
the visualizer. To disable this component, use the following cmake
command:\\
\noindent \verb+ > cmake -DCMAKE_BUILD_TYPE=RelwithDebInfo -DBUILD_SOCCERWINDOW=False ..+
\subsection{Python Interface}
The Python interface is required for interfacing Python agents to the
HFO domain. To install this interface, from the main HFO directory:
\noindent \verb+ > pip install .+
or
\noindent \verb+ > pip install --user .+
if you have limited permissions on the machine.
Successful installation depends on
\verb+Python 2.7, 3.2, or above+ (tested with 2.7 and 3.5) and \verb+numpy+.
\section{Uninstall}
The install is completely contained in the HFO directory. Simply
delete this directory to uninstall. If you have installed the python
interface, uninstall it as follows: \verb+pip uninstall hfo+.
\begin{figure}[htp]
\centering
\includegraphics[width=\textwidth]{figures/HFODiagram}
\caption{HFO is comprised of several components which communicate
over the network. Network connections are depicted with orange
ovals. Calling the HFO executable starts the trainer, visualizer,
and all the offensive and defensive npcs (Agent2d) as well as the
offensive and defensive agent servers. Your code then uses the HFO
interface to connect your agent to the server. Once all agents are
connected, the game begins. The trainer oversees the game.}
\label{fig:hfo}
\end{figure}
\section{Basic Usage}
RoboCup 2D soccer is designed to be played between two teams of
autonomous agents who communicate with a game server. Shown in Figure
\ref{fig:hfo}, the HFO domain reflects these design choices and allows
arbitrary teams to be created consisting of some mix of
non-player-controlled agents (agent2d npcs) and player-controlled
agents. These options are specified through the following flags:\\
\noindent
\verb+ > ./bin/HFO --offense-agents=1 --defense-agents=1 --offense-npcs=2 --defense-npcs=2+\\
This would create a 3v3 game with one player-controlled agent on each
team. In order for the game to start, you must connect your
player-controlled agents to the server. This is done through the
call:\\
\noindent \verb+ > hfo.connectToServer(FEATURE_SET, port, etc);+\\
The arguments to this function are provided by the HFO executable for
each player upon starting the game. For example
\verb|./bin/HFO --offense-agents=1| prints all the information needed
to connect the offensive player:\\
\noindent \verb+Waiting for player-controlled agent base_left-0: +\\
\noindent \verb+ config_dir=HFO/bin/teams/base/config/formations-dt,+\\
\noindent \verb+ server_port=6000, server_addr=localhost,+\\
\noindent \verb+ team_name=base_left, play_goalie=False+\\
By default, the server starts on port 6000, but may be changed as follows:\\
\noindent \verb+ > ./bin/HFO --port 12345+
\section{Visualizer}
The SoccerWindow2 Visualizer allows a live game to be viewed as it
progresses. By default, the visualizer is enabled. However, the game
will likely proceed at a pace too fast for meaningful watching. To
enforce a standard pace, disable sync-mode:\\
\noindent \verb+ > ./bin/HFO --no-sync+\\
To disable visualization altogether, run in headless mode:\\
\noindent \verb+ > ./bin/HFO --headless+\\
The visualizer may also be used after the end of a game by replaying
logs, as discussed in the next section.
\section{Logging}
By default, the soccer server generates game logs and stores them in
the \verb+log+ directory. The main game logs are rcg files:
\verb+log/*.rcg+. These log may be replayed using the soccerwindow2
visualizer. \\
\noindent To replay a log: \\
\verb+ > ./bin/soccerwindow2 -l log/incomplete.rcg+
\noindent To disable logging:\\
\verb+ > ./bin/HFO --no-logging +
\noindent To change the logging directory:\\
\verb+ > ./bin/HFO --log-dir /path/to/new/dir +
\section{Making Videos}
It is possible to make videos from logs by saving frames from
SoccerWindow2. It helps to full-screen SoccerWindow2 before making a
video as it will save higher quality images. There are also several
display options under View $\rightarrow$ View Preference $\rightarrow$
Show that toggle what will be displayed. Saving frames can be done by
File $\rightarrow$ Save Image. To convert the saved pngs into a
movie:\\
\noindent \verb+ avconv -r 10 -start_number 0 -i 3v3/image-%05d.png -f mp4 -c:v libx264+\\
\noindent \verb+ -s 1024x768 -vf "crop=iw/2.5:8.38*ih/10:iw/2:ih/10,transpose=1" +\\
\noindent \verb+ -pix_fmt yuv420p test.mp4 +\\
This command autocrops offensive half of the playfield and rotates it
90 degrees. Avconv can be replaced by ffmpeg. Start number specifies
the number of the first frame. yuv420p pix format for OSX
compatibility.
\section{Recording}
It is possible to record the state perceptions (low- or high-level depending on
the player), low-level actions, and game status of all players:\\
\noindent \verb+ > ./bin/HFO --record + \\
This will produce logs for all the offensive players
(\verb+log/left-[1-11].log+) and defensive players
(\verb+log/right-[1-11].log+). The first offensive player is left-11,
so in the case of single-agent offense, left-11.log will contain the
active player's record. Note that for player controlled agents, it is
necessary to specify a \verb+record_dir+ in the \verb+connectToServer+
function:\\
\\
\noindent \verb+std::string record_dir = "log/";+\\
\noindent \verb+hfo.connectToServer(features, config_dir, port, server_addr,+\\
\noindent \verb+ team_name, goalie, record_dir);+\\
\section{Randomness}
A seed may be specified as follows:\\
\noindent \verb+ > ./bin/HFO --seed 123+\\
This seed will determine the placement of the players and the ball at
the beginning of each episode. Due to non-determinism in the player
policies, it is not sufficient to precisely replicate full games. It
\textit{only} replicates the starting conditions for each episode. The
player's behavior, observations, and physics all proceed
stochastically.
\section{Player On Ball}
By default, episodes begin with the ball being randomly positioned in
the offensive half of the playfield. Typically the first task for the
offense is to send a player to collect the ball. It is possible to
instead request that a certain offensive player is given the ball at
the start of each episode. This is accomplished as follows:\\
\noindent \verb+ > ./bin/HFO --offense-on-ball 1+\\
The above command will always give the ball to the first offensive
player (e.g. uniform number 11). If an offense-on-ball number is
specified that is larger than the number of offensive players, the
ball will be given to a random offensive player at the start of each
episode.
\section{Teams}
By default, offensive and defensive NPCs use the base Agent2D
policy. It is possible to use policies from different teams as
follows:\\
\noindent \verb+ > ./bin/HFO --offense-team helios+\\
\noindent \verb+ > ./bin/HFO --defense-team base+\\
This would take offense NPCs from Helios' 2013 Eindhoven release and
defensive NPCs from the default Agent2D-base. Currently the only
supported teams are Helios and Base.
\section{Communication}
HFO allows agents to receive and broadcast messages. This is
accomplished by the \verb hear \ and \verb say \ functions. The
maximum allowed message size is controlled by HFO's
\verb|--message-size| flag. See
\verb|examples/communication_agent.cpp| and
\verb|examples/communication_agent.py| for examples.
\section{Fullstate}
By default, perceptions and actions in HFO are noisy. The
\verb+ --fullstate+ flag in HFO removes noise from the agent's
perception of the world. Many tasks become significantly easier as a
result. Noise in actions remains. This flag is disabled by default.
\section{Controlling Trials}
HFO trials typically end with a goal, the defense capturing the ball,
the ball going out of bounds, or running out of time. The trials flag
specifies a maximum number of trials
\verb+ > ./bin/HFO --trials 500+. Instead, a maximum number of frames
may be specified: \verb+ > ./bin/HFO --frames 1000+ will stop the
server after 10,000 steps have passed. Each trial is run for a maximum
of \verb --frames-per-trial \ steps, but may stop early if no agent
approaches the ball within \verb --untouched-time \ steps.
\section{State Spaces}
The HFO domains provides a choice between a low and a high-level
feature set. Selecting between the different feature sets is
accomplished when connecting the agent to the server:
\begin{verbatim}
> hfo.connectToServer(LOW_LEVEL_FEATURE_SET, ...);
> hfo.connectToServer(HIGH_LEVEL_FEATURE_SET, ...);
\end{verbatim}
See \verb|examples/hfo_example_agent.cpp| and
\verb|examples/hfo_example_agent.py| for examples. As the choice of
feature set influences the challenge of learning, it is the
responsibility of the user to faithfully report which state space was
used. The following sections explain the feature sets.
\subsection{High Level Feature Set}
A set of high-level features is provided following the example given
by Barrett et al. pp. 159-160 \cite{THESIS14-Barrett}. Barrett writes
``There are many ways to represent the state of a game of half field
offense. Ideally, we want a compact representation that allows the
agent to learn quickly by generalizing its knowledge about a state to
similar states without over-constraining the policy.'' All features
are encoded a floating point values and normalized to the range of
[-1,1]. Invalid features are given a value of -2. The features are as
follows:
\subsubsection{High Level State Feature List}
Let $T$ denote the number of teammates in the HFO game and $O$ the
number of opponents. There are a total of $10 + 6T + 3O$ high-level
features.
\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
\item{\textbf{X position} - The agent’s x-position on the field. See
Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Y position} - The agent’s y-position on the field. See
Figure \ref{fig:playfieldCoords}.}
\item{\textbf{Orientation} - The global direction that the agent is facing.}
\item{\textbf{Ball X} - The ball's x-position on the field.}
\item{\textbf{Ball Y} - The ball's y-position on the field.}
\item{\textbf{Able to Kick} - Boolean indicating if the agent can kick the ball.}
\item{\textbf{Goal Center Proximity} - Agent's proximity to the center of the goal.}
\item{\textbf{Goal Center Angle} - Angle from the agent to the center of the goal.}
\item{\textbf{Goal Opening Angle} - The size of the largest open angle
of the agent to the goal, shown as $\theta_g$ in Figure
\ref{fig:openAngle}. Invalid if agent is not playing offense.}
\item{\textbf{Proximity to Opponent} - If an opponent is present,
proximity to the closest opponent. Invalid if there are no
opponents.}
\item [$T$] {\textbf{Teammate's Goal Opening Angle} - For each
teammate $i$: $i$’s goal opening angle. Invalid if agent is not
playing offense.}
\item [$T$] {\textbf{Proximity from Teammate i to Opponent} - For each
teammate i: the proximity from the teammate to the closest
opponent. This feature is invalid if there are no opponents or if
teammates are present but not detected.}
\item [$T$] {\textbf{Pass Opening Angle} - For each teammate i: the open
angle available to pass to teammate i. Shown as $\theta_p$ in Figure
\ref{fig:openAngle}. If teammates are present but not detected, this
feature is considered invalid and given the value of -2.}
\item [$3T$] {\textbf{X, Y, and Uniform Number of
Teammates} - For each teammate: the x-position, y-position and
uniform number of that teammate.}
\item [$3O$] {\textbf{X, Y, and Uniform Number of
Opponents} - For each opponent: the x-position, y-position and
uniform number of that opponent.}
\end{enumerate}
\begin{figure}[htp]
\centering
\includegraphics[width=.7\textwidth]{figures/playfieldCoords}
\caption{\textbf{Normalized Coordinates in the HFO play field}:
These coordinates are used for reporting the agent's position in
the high-level feature set as well specifying targets for the
mid-level actions (Section \ref{sec:mid_level_actions}). The
red-rectangle shows the boundaries of the reported positions,
which exceed the play field boundaries by 10\% in each
direction. Positions exceeding this rectangle are bounded (via
min/max) to the edges of the rectangle. All distance features are
normalized against the max HFO distance shown in orange.}
\label{fig:playfieldCoords}
\end{figure}
\begin{figure}[htp]
\centering
\includegraphics[width=.5\textwidth]{figures/openAngle}
\caption{Open angle from ball to the goal $\theta_g$ avoiding the
blue goalie and the open angle from the ball to the yellow
teammate $\theta_p$. Figure reproduced with permission from Samuel
Barrett.}
\label{fig:openAngle}
\end{figure}
\subsection {Low Level Feature Set}
The state features used by HFO are designed with the mindset of
providing an overcomplete, basic, egocentric viewpoint. The features
are basic in the sense that they provide distances and angles to
relevant points of interest, but do not include higher level
perceptions such as the largest angle between a goal post and
goalkeeper.
All features are encoded as floating point values normalized to the
range of [-1,1]. Several different types of features exist:
\subsubsection{Boolean Features}
Boolean features assume either the minimum feature value of -1 or the
maximum feature value of 1.
\subsubsection{Valid Features}
Since feature information is attained from the Agent's world-model, it
is possible that, the world model's information may be stale or
incorrect. \textit{Valid features} are boolean features indicating
consistency of world model predictions. For example, if the world
model's estimate of the agent's position is known to be flawed, the
\textit{valid feature} for self position would assume the minimum
value of -1. Otherwise it will assume the maximum value of 1.
The features associated with a valid feature are given the value of
zero if an inconsistency is detected. For example, if the world model
detects that the agent's velocity is invalid, the feature that encodes
the magnitude of self velocity will be set to zero.
\subsubsection{Angular Features}
\textit{Angular features} (e.g. the angle to the ball), are encoded as
two floating point numbers -- the $sin(\theta)$ and $cos(\theta)$
where $\theta$ is the original angle in radians. Figure
\ref{fig:ang_example} provides examples of the angular encoding.
This encoding allows the angle to vary smoothly for all possible
angular values. Other encodings such as radians or degrees have a
discontinuity that when normalized, could cause the feature value to
flip between the maximum and minimum value in response to small
changes in $\theta$.
Given an angular feature $\langle \alpha_1, \alpha_2 \rangle$ we can
recover the original angle $\theta$ (in radians) by taking the
$cos^{-1}(\alpha_2)$ and multiplying by the sign of $\alpha_1$.
Another method uses the common 'atan2' function as
$atan2(\alpha_1, \alpha_2)$.
\begin{figure*}[htp]
\centering
\subcaptionbox{Angular Encoding}{
\includegraphics[width=.4\textwidth]{figures/AngExample}
}
\hspace{3em}
\subcaptionbox{Additional Examples}{
\includegraphics[width=.3\textwidth]{figures/AngFeatExample}
}
\caption{\textbf{Angular Encoding:} Objects on the agents left/right
side result in a negative/positive $sin(\theta)$. $cos(\theta)$ is
positive in front of the player and negative behind. For example,
an object directly in front of the player would have angular
features of $sin(\theta)=0, cos(\theta)=1$. Additional examples:
\textbf{Angle to ball} $\theta=60^\circ$ or $1.0472$ radians. This
results in angular features $\langle sin(\theta)=.86,
cos(\theta)=.49 \rangle$. \textbf{Angle to teammate}:
$\theta=135^\circ, 2.35$ radians. $\langle sin(\theta)=.71,
cos(\theta)=-.71 \rangle$. \textbf{Angle to Opponent}:
$\theta=-90^\circ$ or $-1.57$ radians. $\langle sin(\theta)=-1,
cos(\theta)=0 \rangle$.}
\label{fig:ang_example}
\end{figure*}
\subsubsection{Proximity Features}
\textit{Proximity features} encode the proximity of the agent to an
object of interest. Unless otherwise indicated, they are normalized
against the maximum possible distance in the HFO playfield (defined as
$\sqrt{l^2 + w^2}$ where $l,w$ are the length and width of the HFO
playfield). A maximum proximity of 1 indicates the agent is co-located
with the object of interest, while a minimum proximity of -1 indicates
that the agent is across the field from the object of interest.
\subsubsection{Landmark Features}
Landmark features encode the relative angle and proximity of the agent
to a landmark of interest. Each landmark feature consists of three
floating point values, two to encode the agent's relative angle to the
landmark and one to encode the landmark's proximity. Note that if the
agent's self position is invalid, then the landmark feature values are
zeroed.
\subsubsection{Player Features}
Player features are used to encode the relationship of the agent to
another player or opponent. Each player feature is encoded as 1) a
landmark feature of that player's location 2) the global angle of that
player's body 3) the magnitude of the player's velocity and 4) the
global angle of the player's velocity. Eight floating point numbers
are used to encode each player feature.
\subsubsection{Uniform Number Features}
In the low-level feature space, unknown uniform numbers, or \textit{unums},
are encoded as -1, while known ones are encoded as $\frac{unum}{100}$, thus
remaining well within the $[-1, 1]$ range. (Note that roundoff error may need
to be allowed for when converting these back to integers, such as for use in
high-level actions; e.g., 0.0799 will need to be converted back to 8.)
Uniform number features, a later addition to the low-level feature space,
are positioned after all other features to hopefully ensure compatibility
with older programs.
\subsubsection{Other Features}
Some features, such as the agent's stamina, do not fall into any of
the above categories. These features are referred to as \textit{other
features} and are normalized in the range $[-1, 1]$.
\subsubsection{Low Level State Feature List}
Let $T$ denote the number of teammates and $O$ denote the number of
opponents in the HFO game. Then there are a total of $58 + 9T + 9O$
low-level features:
\begin{enumerate}[noitemsep]
\setcounter{enumi}{-1}
\item{\textbf{Self\_Pos\_Valid} [Valid] Indicates if self position is valid.}
\item{\textbf{Self\_Vel\_Valid} [Valid] Indicates if the agent's velocity is valid.}
\itemrange{1}{\textbf{Self\_Vel\_Ang} [Angle] Angle of agent's velocity.}
\item{\textbf{Self\_Vel\_Mag} [Other] Magnitude of the agent's velocity.}
\itemrange{1}{\textbf{Self\_Ang} [Angle] Agent's Global Body Angle.}
\item{\textbf{Stamina} [Other] Agent's Stamina: Low stamina slows movement.}
\item{\textbf{Frozen} [Boolean] Indicates if the agent is Frozen. Frozen status can
happen when tackling or being tackled by another player.}
\item{\textbf{Colliding\_with\_ball} [Boolean] Indicates the agent
is colliding with the ball.}
\item{\textbf{Colliding\_with\_player} [Boolean] Indicates the agent
is colliding with another player.}
\item{\textbf{Colliding\_with\_post} [Boolean] Indicates the agent
is colliding with a goal post.}
\item{\textbf{Kickable} [Boolean] Indicates the agent is able to
kick the ball.}
\itemrange{2}{\textbf{Goal Center} [Landmark] Center point between the goal posts.}
\itemrange{2}{\textbf{Goal Post Top} [Landmark] Top goal post.}
\itemrange{2}{\textbf{Goal Post Bot} [Landmark] Bottom goal post.}
\itemrange{2}{\textbf{Penalty Box Center} [Landmark] Center of the penalty box line.}
\itemrange{2}{\textbf{Penalty Box Top} [Landmark] Top corner of the penalty box.}
\itemrange{2}{\textbf{Penalty Box Bot} [Landmark] Bottom corner of the penalty box.}
\itemrange{2}{\textbf{Center Field} [Landmark] The left middle point of the
HFO play area.}
\itemrange{2}{\textbf{Corner Top Left} [Landmark] Top left corner HFO Playfield.}
\itemrange{2}{\textbf{Corner Top Right} [Landmark] Top right corner HFO Playfield.}
\itemrange{2}{\textbf{Corner Bot Right} [Landmark] Bot right corner HFO Playfield.}
\itemrange{2}{\textbf{Corner Bot Left} [Landmark] Bot left corner HFO Playfield.}
\item{\textbf{OOB Left Dist} [Proximity] Proximity to the nearest
point of the left side of the HFO playable area. E.g. distance
remaining before the agent goes out of bounds in left field.}
\item{\textbf{OOB Right Dist} [Proximity] Proximity to the right
field line.}
\item{\textbf{OOB Top Dist} [Proximity] Proximity to the top field line.}
\item{\textbf{OOB Bot Dist} [Proximity] Proximity to the bottom field line.}
\item{\textbf{Ball Pos Valid} [Valid] Indicates the ball position estimate is valid.}
\itemrange{1}{\textbf{Ball Angle} [Angle] Agent's angle to the ball.}
\item{\textbf{Ball Dist} [Proximity] Proximity to the ball.}
\item{\textbf{Ball Vel Valid} [Valid] Indicates the ball velocity estimate is valid.}
\item{\textbf{Ball Vel Mag} [Other] Magnitude of the ball's velocity.}
\itemrange{1}{\textbf{Ball Vel Ang} [Angle] Global angle of ball velocity.}
\item [$8T$] {\textbf{Teammate Features} [Player] One teammate feature set (8 features) for each teammate active in HFO, sorted by proximity to the agent.}
\item [$8O$] {\textbf{Opponent Features} [Player] One opponent feature set (8 features) for each opponent present, sorted by proximity to the player.}
\item [$T$] {\textbf{Teammate Uniform Nums} [Unum] One uniform number for each teammate active in HFO, sorted by proximity to the agent.}
\item [$O$] {\textbf{Opponent Uniform Nums} [Unum] One uniform number for each opponent present, sorted by proximity to the player.}
\end{enumerate}
\section{Action Space}
The HFO domain provides support for both low-level primitive actions,
mid-level, and high-level strategic actions. Low-level, parameterized
actions are provided for locomotion and kicking. Mid-level actions are
still parameterized by capture high level activities such as
dribbling. Finally, high-level discrete, strategic actions are
available for moving, shooting, passing and dribbling. Control of the
agent's head and gaze is not provided and follows Agent2D's default
strategy. Low, medium, and high level actions are available through
the same interface. As the choice of action spaces greatly influences
the challenge of learning, it is the responsibility of the user to
faithfully report which action spaces were used.
\subsection{Low Level Actions}
\label{sec:low_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Dash}(power, degrees): Moves the agent with power [-100,
100] where negative values move backwards. The relative direction
of movement is given in degrees and varies between [-180,180] with 0
degrees being a forward dash and 90 degrees dashing to the agent's
right side. Note, dashing does not turn the agent.}
\item{\textbf{Turn}(degrees): Turns the agent in the
specified direction. Valid values range between [-180, 180] degrees
where 90 degrees turns the agent to directly to its right side.}
\item{\textbf{Tackle}(degrees): Contest the ball. Direction
varies between [-180, 180].}
\item{\textbf{Kick}(power, degrees): Kick the ball with power [0, 100]
in relative direction [-180, 180]. Has no effect if the agent does
not possess the ball.}
\end{itemize}
\subsection{Mid Level Actions}
\label{sec:mid_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Kick$\_$To}(target$_x$, target$_y$, speed): Kicks the
ball to the specified target point with the desired speed. Valid
values for target$_{x,y} \in [-1,1]$ and speed $\in [0,3]$.}
\item{\textbf{Move$\_$To}(target$_x$, target$_y$): Moves to the
specified target point using the max dash speed. Valid values for
target$_{x,y} \in [-1,1]$.}
\item{\textbf{Dribble$\_$To}(target$_x$, target$_y$): Dribbles the
ball to the specified target point. Attempts to fetch the ball if
the agent doesn't already possess it. Performs some checks to avoid
opponents and keeps good control of the ball. Valid values for
target$_{x,y} \in [-1,1]$.}
\item{\textbf{Intercept}(): Moves to intercept the ball, taking into
account the ball velocity. More efficient than chasing the ball.}
\end{itemize}
\subsection{High Level Actions}
\label{sec:high_level_actions}
\begin{itemize}[noitemsep]
\item{\textbf{Move}(): Re-positions the agent according to the
strategy given by Agent2D. The \textit{move} command works only when
agent does not have the ball. If the agent has the ball, another
command such as \textit{dribble}, \textit{shoot}, or \textit{pass}
should be used.}
\item{\textbf{Shoot}(): Executes the best available shot. This command
only works when the agent has the ball.}
\item{\textbf{Pass}(teammate\_uniform\_number): Passes to the teammate
with the provided uniform number. Does nothing if the player does
not have control of the ball or the requested teammate is not
detected.}
\item{\textbf{Dribble}(): Advances the ball towards the goal using a
combination of short kicks and moves.}
\item{\textbf{Catch}(): This goalie-specific action may be used to
catch the ball.}
\item {\textbf{Reduce\_Angle\_To\_Goal} (): Moves the agent to a point on the field, such that the kicker has the least open angle to the goal. }
\item {\textbf{Defend\_Goal} () : Moves the agent to a point on a fixed line on the field, such that the kicker has the least open angle to the goal.}
\item {\textbf{Go\_To\_Ball} (): Makes the agent go towards the ball.}
\item {\textbf{Mark\_Player} (uniform\_number): Moves the agent so as to mark the player with the specified uniform number.}
\end{itemize}
\subsection{Special Actions}
\begin{itemize}[noitemsep]
\item{\textbf{NO-OP}: Indicates that the agent should take no action.}
\item{\textbf{Quit}: Indicates to the agent server that you wish to
terminate the HFO environment.}
\end{itemize}
\subsection{Available Actions}
The Special Actions are always available. The below table indicates whether
other actions are available (only if there are no ``N''s indicated); check
below the table for the action abbreviations.
\begin{center}
{\footnotesize
\begin{tabular}{r c c c c c c c c c c c c c c c c c}
Action & Da & Tu & Ta & K & KT & MT & DT & I & M & S & P & Dr & C & RG & DG & G & MP \\
\hline
Self position invalid & Y & Y & Y & Y & N & N & N? & N? & N? & N? & ? & Y & ? & N? & N? & ? & ? \\
Self velocity invalid & N & Y? & Y? & Y? & ? & N & ? & N? & N? & N? & ? & Y & ? & ? & ? & ? & ? \\
Ball position invalid & Y & Y & Y? & N & N & Y & N? & N & ? & N & N & Y? & N & ? & ? & N & ? \\
Ball velocity invalid & Y & Y & Y & ? & ? & Y & ? & N? & ? & N & N? & Y? & N? & Y? & Y? & Y? & Y? \\
Teammate loc invalid & Y & Y & Y & Y & Y & Y & Y & Y & ? & Y & N & Y? & Y & Y & Y & Y & Y \\
Team. unum invalid & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & N & Y & Y & Y & Y & Y & Y \\
Opponent loc invalid & Y & Y & Y & Y & Y & Y & Y? & Y & ? & Y? & Y & Y? & Y & N & N & Y & N \\
Opp. unum invalid & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & Y & N \\
Ball kickable & Y & Y & Y & Y & Y & Y & Y & N? & N & Y & Y & Y & Y & N? & N? & N? & Y? \\
Ball not kickable & Y & Y & Y & N & N & Y & Y & Y & Y & N & N & N & Y & Y & Y & Y & Y \\
Frozen & N & N & N & N & N & N & N? & N & N? & N & N & Y & N? & N & N & N & N \\
Colliding w/ball & ? & ? & ? & ? & ? & ? & ? & N? & ? & ? & ? & ? & ? & ? & ? & ? & ? \\
Colliding w/player & ? & ? & ? & N? & N? & ? & ? & N? & ? & ? & ? & Y? & ? & ? & ? & ? & ? \\
Colliding w/post & ? & ? & N? & N? & N? & ? & ? & N? & ? & ? & ? & ? & ? & ? & ? & ? & ? \\
Offense & Y & Y & N & Y & Y & Y & Y & Y & Y & Y & Y & Y & N & N & N & Y & N \\
Defense, not goalie & Y & Y & Y & N? & N? & Y & N & Y & Y & N & N & N & N & Y & ? & Y & Y \\
Goalie (defense) & Y & Y & Y? & N? & N? & Y & N & Y & ? & N & N & N & Y & ? & ? & ? & ? \\
\end{tabular}
}
\end{center}
Da: Dash; Tu: Turn; Ta: Tackle; K: Kick \\
KT: Kick\_To; MT: Move\_To; DT: Dribble\_To; I: Intercept \\
M: Move; S: Shoot; P: Pass; Dr: Dribble; C: Catch; RG: Reduce\_Angle\_To\_Goal; DG: Defend\_Goal; G: Go\_To\_Ball; MP: Mark\_Player
\section{Developing a New Agent}
New agents may be developed in C++ or Python. In Python, as long as
the hfo interface has been installed, the agent needs only to
either \verb+from hfo import *+ or \verb+import hfo+. In C++ it is necessary to
\verb+#include <HFO.hpp>+ and also to link against the shared object
library \verb+lib/libhfo.so+ when compiling:
\begin{verbatim}
> g++ example/your_new_agent.cpp -I src -L lib -Wl,-rpath=lib -lhfo
\end{verbatim}
\bibliographystyle{abbrv}
\bibliography{manual}
\end{document}