Training Workflow

The training script executes the control environment together with the reinforcement learning algorithm to generate a control policy. The trained model is then loaded by the inference script to control the payload, and the results are visualized and evaluated.

Physics Simulation

The physics engine simulates the motion of the payload, suspension ropes, control unit, and crane. To reproduce realistic suspended-load behavior, the simulation considers not only the payload geometry and mass, but also rope constraints, wind disturbances, and thruster forces.

Reinforcement Learning

A reinforcement learning model was used to generate continuous control outputs. The model receives the state of the payload and control unit as input and computes the thrust commands for the four thrusters.

The model is trained to rotate the payload to the target yaw angle and maintain its orientation after reaching the target. Under wind disturbances, it also learns how to generate thruster commands that compensate for external forces.

Training Targets

The HSteel, Plate, and Truss payloads differ in geometry, mass, rope attachment points, and rotational characteristics. Therefore, separate control models were trained for each payload under different disturbance conditions.

The same reinforcement learning method was applied to all payloads to evaluate its adaptability under different payload configurations and disturbance conditions.

Using the Trained Model

After training, only the trained model is extracted and loaded by the inference script. During control, the current state is provided to the model at fixed intervals, and the computed thruster commands are held until the next control cycle.

The inference results are saved as CSV files and visualized as evaluation charts showing the payload yaw angle and the thrust history of each thruster.

How to Read the Evaluation Charts

Control performance is evaluated by comparing the payload yaw angle without thruster control (Baseline) and with reinforcement learning control (Control), together with the thrust history of each thruster.

Payload / Disturbance

The payload type and disturbance condition are shown at the top of each chart.

Baseline

The payload yaw angle without thruster control. The red dashed line indicates the target yaw angle, and the blue line shows the payload yaw angle.

Control

The control result generated by the reinforcement learning model. The chart shows how quickly the payload reaches the target yaw angle and how well it maintains the desired orientation afterward.

Thruster Force Heatmap

The thrust history of the four thrusters. Yellow indicates higher thrust, while purple indicates lower thrust.

HSteel: 2-Point Suspension

HSteel is suspended by two ropes. The control performance was evaluated under three disturbance conditions: no wind, one-directional wind, and Two-Directional Wind.

X0Y0 (No Wind)

The payload reaches the target orientation in approximately 10 seconds and then maintains a yaw angle close to 90°.

After reaching the target yaw angle, the controller maintains the orientation with intermittent thrust, primarily using Thruster T3 near its maximum output.

X1Y0 (One-Directional Wind)

The payload reaches the target orientation in approximately 10 seconds and remains close to 90° thereafter.

After reaching the target yaw angle, the controller primarily uses Thrusters T1 and T2 near their maximum output.

X1Y1 (Two-Directional Wind)

The payload reaches the target orientation in approximately 10 seconds and remains close to 90° thereafter.

After reaching the target yaw angle, the controller uses Thrusters T1, T2, and T4, with T2 operating near its maximum output most of the time.

Plate: 4-Point Suspension

The Plate payload is suspended by four ropes. The control performance was evaluated under three disturbance conditions: no wind, one-directional wind, and two-directional wind.

X0Y0 (No Wind)

The payload reaches the target yaw angle in approximately 13 seconds and maintains it thereafter.

After reaching the target yaw angle, the controller primarily uses Thrusters T2 and T3 near their maximum output.

X1Y0 (One-Directional Wind)

The payload reaches the target yaw angle in approximately 13 seconds and maintains it thereafter.

After reaching the target yaw angle, the controller primarily uses Thrusters T1, T3, and T4, with T3 operating near its maximum output.

Similar thruster control behavior is observed under both X1Y0 and X1Y1 disturbance conditions.

X1Y1 (Two-Directional Wind)

The payload reaches the target yaw angle in approximately 13 seconds and maintains it thereafter.

After reaching the target yaw angle, the controller primarily uses Thrusters T1, T3, and T4, with T3 operating near its maximum output.

Similar thruster control behavior is observed under both X1Y0 and X1Y1 disturbance conditions.

Truss: 4-Point Suspension

The Truss payload is suspended by four ropes. The control performance was evaluated under three disturbance conditions: no wind, one-directional wind, and two-directional wind.

X0Y0 (No Wind)

The payload reaches the target yaw angle in approximately 20 seconds and maintains it thereafter.

After reaching the target yaw angle, all four thrusters are used, with Thruster T3 operating near its maximum output.

X1Y0 (One-Directional Wind)

The payload reaches the target yaw angle in approximately 20 seconds and maintains it thereafter.

After reaching the target yaw angle, the controller primarily uses Thruster T4, while the overall thruster output remains relatively low.

X1Y1 (Two-Directional Wind)

The payload reaches the target yaw angle in approximately 40 seconds and maintains it thereafter.This condition required the longest settling time among all evaluation cases.

After reaching the target yaw angle, all four thrusters operate near their maximum output.