ParticleDetection.reconstruct_3D.matchND

Functions to reconstruct 3D rod endpoints from images of a stereo camera system by solving an n-partite matching problem between the rods within one frame and the respective previous frame .

Authors: Adrian Niemann (adrian.niemann@ovgu.de)

Date: 31.10.2022

assign(input_folder: str, output_folder: str, colors: Iterable[str], cam1_name: str = 'gp1', cam2_name: str = 'gp2', frame_numbers: Optional[Iterable[int]] = None, calibration_file: Optional[str] = None, transformation_file: Optional[str] = None, solver: LpSolver = pulp.PULP_CBC_CMD(msg=False)) → Tuple[ndarray]

Matches, triangulates and tracks rods over frames from *.csv data files.

The function matches rods over multiple frames using npartite matching and a combination of 3D displacement and 2D reprojection error as the weights. This results in 3D reconstructed rods, that are tracked over the course of the given frames. It takes *.csv files with the columns from DEFAULT_COLUMNS as input and also outputs the results in this format. The resulting dataset is saved in the given output folder.

Parameters

• input_folder (str) – Folder containing the *.csv files for all colors given in colors.

• output_folder (str) – Folder to write the output to. The parent folder of this must exist already.

• colors (Iterable[str]) – Names of the colors present in the dataset. See DEFAULT_COLUMNS.

• cam1_name (str, optional) –

  First camera’s identifier in the given dataset.

  By default "gp1".

• cam2_name (str, optional) –

  Second camera’s identifier in the given dataset.

  By default "gp2".

• frame_numbers (Iterable[int], optional) –

  An iterable of frame numbers present in the data.

  By default None.

• calibration_file (str, optional) –

  Path to a *.json file with stereocalibration data for the cameras which produced the images for the rod position data.

  By default the calibration constructed with Matlab for GP1 and GP2 is used.

• transformation_file (str, optional) –

  Path to a *.json file with transformation matrices expressing the transformation from the first camera’s coordinate system to the world/box coordinate system.

  By default the transformation constructed with Matlab is used.

• solver (LpSolver, optional) –

  Solver that is used for the n-partite matching problem.

  Default is PULP_CBC_CMD(msg=False).

Returns

Tuple[ndarray, ndarray] – [0]: reprojection errors

[1]: rod lengths

create_weights(p_3D: ndarray, p_3D_prev: ndarray, repr_errs: ndarray) → ndarray

Generate weights with 3D-displacement*reprojection-error.

Creates a weight matrix for matching rods between frames, using the rod endpoint displacement between frames times the reprojection-error of the rod endpoints as the cost of an assignment.

Parameters

• p_3D (np.ndarray) – Shape must be in (rod_ids(cam1), rod_ids(cam2), 4, 3) Dimension explanations: (rod_id(cam1), rod_id(cam2), endpoint-combination, 3D-coordinates)

• p_3D_prev (np.ndarray) – Shape must be in (rod_ids, 2, 3). Dimension explanations: (rod_id, end-point, 3D-coordinates)

• repr_errs (np.ndarray) – Shape must be in (rod_ids(cam1), rod_ids(cam2), 4, 2) Dimension explanations: (rod_id(cam1), rod_id(cam2), end-combo, err{cam1, cam2})

Returns

• weights (np.ndarray) – Weights in the shape of (rod_id, rod_ids(cam1), rod_ids(cam2))

• point_choices (np.ndarray) – Choices of minimizing endpoint combination (rod_id, rod_ids(cam1), rod_ids(cam2))

match_frame(data: DataFrame, cam1_name: str, cam2_name: str, frame: int, color: str, calibration: dict, P1: ndarray, P2: ndarray, rot: Rotation, trans: ndarray, r1: ndarray, r2: ndarray, t1: ndarray, t2: ndarray)

Matches, triangulates and tracks rods for one frame from a DataFrame.

Parameters

• data (DataFrame) – Dataset of rod positions.

• cam1_name (str) – First camera’s identifier in the given dataset, e.g. "gp1".

• cam2_name (str) – Second camera’s identifier in the given dataset, e.g. "gp2".

• frame (int) – Frame in data who’s endpoints shall be (re-)evaluated.

• color (str) – Color of the rods in data to match.

• calibration (dict) –

  Stereocamera calibration parameters with the required fields:

  "CM1": camera matrix of cam1

  "R": rotation matrix between cam1 & cam2

  "T": translation vector between cam1 & cam2

  "CM2": camera matrix of cam2

• P1 (ndarray) – Projection matrix for camera 1.

• P2 (ndarray) – Projection matrix for camera 2.

• rot (Rotation) – Rotation from camera 1 coordinates to world/experiment coordinates.

• trans (ndarray) – Translation vector as part of the transformation to world/experiment coordinates.

• r1 (ndarray) – Rotation matrix of camera 1.

• r2 (ndarray) – Rotation matrix of camera 2.

• t1 (ndarray) – Translation vector of camera 1.

• t2 (ndarray) – Translation vector of camera 2.

Returns

Tuple[DataFrame, ndarray, ndarray] – Returns the DataFrame with tracked rods for frame of color. Additionally, returns the assignment costs, i.e. the sum of end point reprojection errors per rod. Lastly, returns the lengths of the reconstructed rods.

Raises

ValueError – Is raised when an unbalanced dataset is encountered, i.e. on consecutive frames or camera angles an unequal number of rods is present.

npartite_matching(weights: ndarray, maximize: bool = True, solver: LpSolver = pulp.PULP_CBC_CMD(msg=False)) → Tuple[ndarray]

Solve an n-partite matching problem.

The weights given represent the cost of assigning the entities associated with the respective indices together.

Parameters

• weights (ndarray) – A n-dimensional matrix where each entry is the cost associated with choosing the combination of indices. The number of dimensions represent the number of groups and the size of each dimension the number of members in this group.

• maximize (bool, optional) – By default True.

• solver (LpSolver, optional) –

  A solver for the matching problem.

  By default PULP_CBC_CMD(msg=False).

Returns

Tuple[ndarray] – The length of the tuple is equal to the number of dimensions of weights. Each element of the tuple has a maximum size of the smallest dimension of weights. The elements of the arrays represent the index of the groups paired member.

Examples

The following weights are defined: weights.shape -> [12, 12, 4] This means three groups must be associated to each other, with four paths, because the smallest dimension has size 4. weights[0, 5, 2] then represents the cost of pairing element 0 of group 0 with element 5 of group 1 and element 2 of group 2.

>>> npartite_matching(np.random((12, 12, 4)))
(array([2, 9, 4, 11]), array([6, 9, 1, 0]), array([2, 1, 0, 3]))

Note

Adapted from: https://stackoverflow.com/questions/60940781/solving-the-assignment-problem-for-3-groups-instead-of-2