This benchmark aims at developing an open-loop program that controls a Pioneer robot to follow a 2m by 2m square path.

Segment 1: 0.00%
Segment 2: 0.00%
Segment 3: 0.00%
Segment 4: 0.00%

Average: 0.00%

The metric used to evaluate the robot is applied for 4 separate segments of the path, which correspond to the 4 sides of the square.

Each segment is defined as a corridor that lies on one edge of the square. The "goal" of one segment is defined as the vertex between the current and the next segment. To reach the next segment, the robot must cross the line going through the center of the square and the "goal" vertex.

For each individual segment, we compute a performance which is based on 3 different parameters: the path (how well the robot managed to keep close to the "ideal" route), the time needed to go through this segment, and the distance to the goal, which is mostly used to evaluate how close to the goal the robot is in the current segment.

The path element is a linear value inversely proportional to the breadth of the corridor containing the path of the robot for this segment. It goes from 100% when the robot went on a perfect line, to 0% when the corridor is wider than it is long.

The time element is a linear value inversely proportional to the time needed to go through this segment. It goes from 100% for a duration of 0 seconds to 0% for 20 seconds or more.

The distance element is how far to the goal (one vertex of the square) is the robot in the current segment. If the robot has already completed this segment, this will be how far was the robot when he exited this segment. This value goes from 100% for a distance of 0 to 0% when the distance is larger than one side of the square. The value is defined as 10 * (1 - distance/2) + 90 * e(-3 * distance). The exponential part is meant to reward a robot which is very precise when reaching the goal, while the linear part is there so a robot which is moving in the right direction isn't losing points due to the time portion of the performance. Only a slow robot should lose points, whereas an average robot will simply gain less than a fast one.

The performance for the segment is a weighted average of the 3 elements. The time element has half the weight of the other two.


The overall performance is the average of the 4 segments.

How to improve the performance?

The basic controller is an open-loop program that simply sets the velocity and waits for a given time until the robot reaches the desired position. The simplest thing to improve the performance is to tune the duration of the steps in order to improve the precision of the path.

A more elegant solution would be to use the wheel sensors of the robot. The Pioneer 3-DX has sensors in its wheel that allows the robot to know the distance travelled so far. By using this value you can make the robot stop after exactly 2 meters.

In order to use a sensor, you first have to enable it, as they are disabled by default.

rightWheelSensor = robot.getPositionSensor('right wheel sensor')
rightWheelSensor.enable(16) # Refreshes the sensor every 16ms.

You can then access the value of the sensor with its getValue() method.

rightWheelSensor.getValue()

Some information about the robot that can be useful to improve the controller:

  • The diameter of the wheels is 195mm.
  • The distance between the two wheels is 330mm.

How to perform the benchmark with a real robot?

To perform the benchmark on a real robot you will need a Pioneer 3-DX and means to record its position in real time during the benchmark. The robot has to travel on a 2m by 2m square path, as shown in this image:

When running your controller program on a real environment with your Pioneer 3-DX robot, you have to generate a CSV file containing the achieved trajectory. The CSV file should contain a list of points with a time associated with each point. The points should be ordered chronologically.
Each row should have 3 elements, separated by commas:

  • x coordinate, in meters
  • y coordinate, in meters
  • time since the beginning of the benchmark, in seconds

You can then use this script to compute the performance. You need Python installed on your machine to run the script. Extract all the files and call `real_robot_performance.py` with your CSV file as argument to compute the performance.
For example, you can run the sample file included in the archive by using the following command in a terminal:
python real_robot_performance.py sample.csv

Note that the provided archive is a UTILITY TO EVALUATE TRAJECTORIES ACHIEVED WITH A REAL ROBOT.

IT DOESN'T CONTAIN THE TRAJECTORY RECORDED ON robotbenchmark.net but a sample trajectory. If you want to save the trajectory of the simulated robot when running the benchmark, then you can write the points on the console and then copy them in a file.