Double handoff for small single-packet RPCs

[breakertt]

Dear John,

I noticed that for small single-packet messages, when unloaded, homa_copy_to_user gets called roughly 1.5x per RPC.

After a rough check, I realized that the handoff in homa_rpc_alloc_server triggers the syscall thread and sets the RPC_PKTS_READY bit, but does not add the skb into the msgin.packets, which caused homa_copy_to_user exit and clear the RPC_PKTS_READY bit before the skb is added. This caused softirq call handoff again. This happens roughly 50% of the time for small (<MTU) unloaded RPCs.

I tried to elimitate thiswith two ways below

1. Add packet to msgin.packets in alloc_server
2. Check skb_queue_len > 0 before taking an RPC off ready_rpcs in homa_wait_shared, then look a while

Both solutions improved unloaded latency for 0.3 us but led to 10-30% loaded throughput decrease, which is not a good trade-off imo. Therefore, I wonder if you are aware of this problem already as intended design, or if you have a better fix that can come up with? Thanks.

Cheers,
TIanyi

[johnousterhout]

Interesting. I was not aware of the problem, and I'm surprised that the race could occur, because both homa_rpc_alloc_server and homa_wait_shared acquire the RPC lock. My expectation was that if a receiving thread wakes up before homa_data_pkt has finished adding the packet into the RPC, it would block on the RPC lock. By the time homa_copy_to_user is invoked, the packet should be queued on the RPC.

In looking at the code, I see one scenario where the RPC lock could get released before the packet is added to the RPC: this is if the incoming packet also contains an ack. Can you tell if that is what is happening in your test cases? If so, I think it might be possible to refactor homa_data_pkt to process the data packet before releasing the lock to handle the ack.

Adding the packet to msgin.packets in alloc_server sounds a bit complicated, but I'm surprised that it impacted throughput.

-John-

[breakertt]

Sorry for the late reply, yes, you were right, it's the piggy-backed ACK path where the lock gets released that opens the window. I'll follow up with a PR shortly.

To reproduce, I put a small probe on a branch in my fork: fix-handoff-twice-reproduce. It adds a single handoff_count counter to /proc/net/homa_metrics (incremented at homa_rpc_handoff entry) plus a util/handoff_ratio.sh that prints handoffs / requests_received.

Two machines. Server side:

~/HomaModule/util/cp_node server --protocol homa

Client side:

~/HomaModule/util/cp_node client \
    --protocol homa --workload 64 \
    --first-server 0 --server-nodes 1 --id 1 \
    --gbps 0 --client-max 1 --ports 1 --port-receivers 0

Let the client run for ~10 s and stop it, then read the metrics on the server:

~/HomaModule/util/handoff_ratio.sh

For single-packet messages each server RPC should trigger exactly one homa_rpc_handoff, so the no-race ratio is 1.000. On baseline I see:

handoffs=2892358 rpcs=2241715 ratio=1.290

i.e. ~29 % of server RPCs trigger a second redundant handoff after the receiver hits the empty queue and clears RPC_PKTS_READY.