Enhancing courier scheduling in crowdsourced last-mile delivery through dynamic shift extensions: a deep reinforcement learning approach

Crowdsourced delivery platforms face complex scheduling challenges to match couriers and customer orders. We consider two types of crowdsourced couriers, committed and occasional couriers, each with different compensation schemes. Crowdsourced delivery platforms usually schedule committed courier shifts based on predicted demand. Therefore, platforms may devise an “offline” schedule for committed couriers before the planning period. However, due to the unpredictability of demand, there are instances where it becomes necessary to make online adjustments to the offline schedule. In this study, we focus on the problem of dynamically adjusting the offline schedule through shift extensions for committed couriers. We develop a sequential decision process model to address this problem to maximize platform profit by determining courier shift extensions and the assignments of requests to couriers. To solve our sequential model, we employ deep Q-learning (DQN), a value-based reinforcement learning method, as a heuristic. Comparing DQN with a baseline policy, where no extensions are allowed, the courier assignment and shift extension (CASE) heuristic policy, and a myopic heuristic demonstrates the superior benefits platforms can achieve. Allowing shift extensions leads to increased rewards, reduced costs from lost orders, and fewer lost requests. Additionally, sensitivity analysis shows that the total extension compensation increases nonlinearly with the arrival rate of requests, and in a linear manner with the arrival rate of occasional couriers. For compensation sensitivity, the results showed that the average number of shift extensions has a maximum corresponding to the lowest average number of lost requests.