Experimental Physics and
Industrial Control System

"Henrique F. Simoes via Tech-talk" <tech-talk at aps.anl.gov> · Wed, 10 Jan 2024 12:53:52 +0000

Hi tech-talkers,

I've been setting up a GigE camera from Allied Vision (Alvium G1-500c) with
ADAravis (areaDetector R3-12-1) in a GNU/Linux machine recently and had
struggled to have the machine and camera setup for its maximum throughput (2952
x 1944 x 3 @ 8fps). A typical problematic acquisition would be full of failed
frames due to timeouts.

For a local test setup, I've been able to manage achieve a high throughput by:

- Setting the GevSCPSPacketSize feature to 1500B (defaults to 576B), matching
  MTU [1]. This was not updated by Aravis auto packet size procedure (I haven't
  investigated why yet).
- Using Packet Socket (by enabling NET_RAW Linux capability for the process)
  [3, 4]
- Increasing receive buffer size (through sysctl net.core.rmem_max) [1]
- Increasing the number of RX ring buffers through ethtool --set-ring

For instance, this made me achieve 7.4Hz lossless traffic at full resolution
(~380k images acquired). I haven't yet checked how each of these influence
individually, though. Some of them might not be required for achieving this
result.

At even higher rates, specification limit (8.12Hz, in this case), I start to
see some timeouts until all frames become corrupted with something like this:

2024/01/09 20:22:00.427 ADAravis:lookupColorMode: Could not find a match for pixel format: 0
2024/01/09 20:22:00.427 ADAravis:processBuffer: unknown pixel format 0
2024/01/09 20:22:00.683 ADAravis::newBufferCallback bad frame status: Timeout

** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_pixel_format: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_width: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_height: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_x: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed
** (process:2): CRITICAL **: 20:22:00.683: arv_buffer_get_image_y: assertion 'arv_buffer_payload_type_has_aoi (buffer->priv->payload_type)' failed

I've also seen recommendations for setting the maximum possible MTU (jumbo
frame) [1, 3], but my NIC (a Broadcom NetXtreme BCM5764M) does not support more
than 1500B. Another recommendation is to enable link-layer flow control at the
NIC [1] (and gateway switch), but I couldn't capture any pause frame packet
sent in my setting (a misconfiguration, perhaps?).

Decreasing DeviceLinkThroughputLimit (and turning DeviceLinkThroughputLimitMode
on) showed to be helpful in some scenarios [2], but it actually restricts
transfer rate, making the maximum possible frame rate below the camera's
specification. Hence, its traffic becomes less prone to failures.

After replicating this configuration (as close as I could) in our deployment
machine, the failure rate raised to about 2.8% (of 940k continuously acquired
images) with a lower rate of 6.4Hz (with throughput limit set up to its
minimum). The machine has another NIC (an Intel 82583V), thus another driver
and configuration options. We also use another switch model (possibly with
other settings).

Is there anything else you do to tune GNU/Linux machines for supporting these
GigE cameras?

My intention here is twofold: increase our knowledge of configuration options
and document this for future reference.

Best regards,
Henrique F. Simões

[1]: Tips and tricks to connect 1000BASE-T in Alvium G1 User Guide. Available
at
<https://cdn.alliedvision.com/fileadmin/content/documents/products/cameras/Alvium_G1/techman/Alvium-G1_User-Guide.pdf>
[2]: https://github.com/AravisProject/aravis/issues/314#issuecomment-559459813
[3]: https://github.com/AravisProject/aravis/issues/434#issuecomment-718418648
[4]: https://www.kernel.org/doc/html/latest/networking/packet_mmap.html

Aviso Legal: Esta mensagem e seus anexos podem conter informações confidenciais e/ou de uso restrito. Observe atentamente seu conteúdo e considere eventual consulta ao remetente antes de copiá-la, divulgá-la ou distribuí-la. Se você recebeu esta mensagem por engano, por favor avise o remetente e apague-a imediatamente.

Disclaimer: This email and its attachments may contain confidential and/or privileged information. Observe its content carefully and consider possible querying to the sender before copying, disclosing or distributing it. If you have received this email by mistake, please notify the sender and delete it immediately.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System