Downloads

Evaluation tool for CARMEN-logfiles and relation files using our relation based metric:
Download the Metric Evaluator (.tar.gz, ANSI C++ source files).

Parameters:

Usage: ./metricEvaluator [options]

Options:
-------------------------------------------
-s <name>         slam log file containing ROBOTLASER1 data
-r <relations>    the relations file
-o <odometry>     raw odometry log file
-w "{wx, wy, wz, wphi, wtheta, wpsi}"    the error weight vector (units meter/rad)
-q                use squared error to compute averages (instead of sqrt of squared error)
-e <errorfile>    write avg, std deviation, number of measurements
-eu <errorfile>   write list of errors (unsorted)
-em <errorfile>   write list of errors (sorted by magnitude)
-e1 <errorfile>   write list of errors (sorted by first relation timestamp)
-e2 <errorfile>   write list of errors (sorted by second relation timestamp)
The two basic inputs are the SLAM logfile, that should contain ROBOTLASER1 or FLASER messages in carmen style and the relations file in the format described below. Additionally the program wants an error weight vector to compute one number. Reasonable values are 1.0, 1.0, 1.0, 0.0, 0.0, 0.0 to get translational and 0.0, 0.0, 0.0, 1.0, 1.0, 1.0 to get the rotational error. Optionally you can pass a logfile containing odometry data (as CARMEN ODOM), that will be used to correct SLAM positions, when no corresponding timestamp is found within 0.1s of a relation. By activating the -q option the squared error will be computed. Without -q you would get normal euclidian distance errors (for translations).

Outputs
There are two different outputs:
The averaged error, and additional parameters can be written to a file as documentation (-e).
Additionally the -eX options enabled to write the errors introduced by individual relations to a file using different sorting options. Unsorted will keep the order as in the relations file for a later matching.
Gnuplot can be used to easily display this data: 

gnuplot> plot "filename" with points

Documentation

Currently there are two file formats:
  • Our own .relations files
  • CARMEN log files

Relations files
Relations are given in a simple text based format as follows: 

# timestamp1 timestamp2 x y z roll pitch yaw
1232658478.161285 1232658644.149293 1.031273 -0.003531 0.000000 0.000000 -0.000000 -0.009755
1232658478.161285 1232658647.058497 2.039538 -0.014451 0.000000 0.000000 -0.000000 -0.016492
Each line describes a relation given by x, y, z, roll, pitch and yaw from timestamp1 to timestamp2. The timestamps have to match those in the logfile that is to be evaluated.

CARMEN logfiles
CARMEN style logfiles are text based files, that contain one data element per line. To parse odometry information, we support the standard ODOM format. To parse SLAM outputs we parse the position of either ROBOTLASER1 or the older FLASER messages.
Here is a quick overview of the specific formats: 

ODOM x y theta tv rv accel timestamp hostname logger_timestamp
in one line: 
ROBOTLASER1 laser_type start_angle field_of_view angular_resolution maximum_range accuracy
   remission_mode num_readings [range_readings] num_remissions [remission values] 
   laser_pose_x laser_pose_y laser_pose_theta robot_pose_x robot_pose_y robot_pose_theta laser_tv laser_rv 
   forward_safety_dist side_safty_dist turn_axis timestamp hostname logger_timestamp
FLASER num_readings [range_readings] x y theta odom_x odom_y odom_theta timestamp hostname logger_timestamp
As we are only concerned about the robot position, the other data, especially laser measurements can be incomplete, i.e, provide 0 range readings if the robot has no range finder.

Simple Example

The workflow when evaluating logs using relations is as follows:
  • Download a logfile (e.g. example.log) from the website and process a ROBOTLASER1 logfile with your algorithm. Name the result e.g. slam.log
  • Download the reference relations, e.g. example.relations.
  • Run metricEvaluator: 
    metricEvaluator -s slam.log -r example.relations -o example.log -w "{1.0,1.0,1.0,0.0,0.0,0.0}" -eu unsorted.errors
    This command will write translational errors to unsorted.errors and print out the mean error and standard deviation for slam.log. Passing the example.log as odometry works in most cases as the raw log should contain odometry.

License

MetricEvaluator is licensed under the Common Creative License, Attribution-NonCommercial-ShareAlike 3.0

You are free:
  • to Share - to copy, distribute and transmit the work
  • to Remix - to adapt the work
Under the following conditions:
  • Attribution. You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).
  • Noncommercial. You may not use this work for commercial purposes.
  • Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.
Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.

MetricEvaluator is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.