ModemManager does not notice received SMS

Hello

I am using ModemManager 1.14.2 on a ARM64 board with 5 Quectel E21 modems. We are sending multiple SMS to all modems and a script listening to dbus messages from MM handles them. It generally works, but I found an issue: every now and then one of modems just stops receiving messages. Further investigation showed that it happens, because modem's internal memory is full. This happens because MM somehow misses some of incoming SMS.

For example checking list of messages via mmcli:

mmcli -m 3 --messaging-list-sms
No sms messages were found

The same modem directly via AT commands:

socat - /dev/ttyUSB10,crnl
AT+CMGL

+CMGL: 0,0,,67
0791448720003023200BD0E1FAD95DA603000002112231116100363AE1FB45D7E7F5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 1,0,,67
0791448720003023200BD0E1FAD95DA603000002112231215100363AE1FB45AFA7F5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 2,0,,67
0791448720003023200BD0E1FAD95DA60300000211223151720036BAFA3579764FF5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 3,0,,67
0791448720003023200BD0E1FAD95DA60300000211223151050036BAFA3579860FF5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 4,0,,67
0791448720003023240BD0E1FAD95DA60300000211223161610036BAFA35798EE7F5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 5,0,,67
0791448720003023200BD0E1FAD95DA60300000211223171510036BAFA357966A7F5C562B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 6,0,,67
0791448720003023200BD0E1FAD95DA603000002112231025200363A3191281D4EF5C662B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 7,0,,67
0791448720003023200BD0E1FAD95DA603000002112231020500363A3191282D0EF5C662B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 8,0,,67
0791448720003023200BD0E1FAD95DA603000002112231126100363A31912835E6F5C662B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 9,0,,67
0791448720003023200BD0E1FAD95DA603000002112231223100363A3191280DA6F5C662B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 10,0,,67
0791448720003023200BD0E1FAD95DA603000002112231525200363AE4DD98934EF5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 11,0,,67
0791448720003023200BD0E1FAD95DA603000002112231520500363AE4DD98A30EF5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 12,0,,67
0791448720003023200BD0E1FAD95DA603000002112231626100363AE4DD98ABE6F5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 13,0,,67
0791448720003023200BD0E1FAD95DA603000002112231725100363AE4DD9883A6F5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 14,0,,67
0791448720003023240BD0E1FAD95DA603000002112231034300363A18B9F84A4EF5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 15,0,,67
0791448720003023200BD0E1FAD95DA603000002112231030500363A18B9F85A0EF5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 16,0,,67
0791448720003023200BD0E1FAD95DA603000002112231136100363A18D9D886E6F5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 17,0,,67
0791448720003023200BD0E1FAD95DA603000002112231234100363A18B9F83AA6F5C762B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 18,0,,67
0791448720003023200BD0E1FAD95DA603000002112231538200363AF43259334FF5C862B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 19,0,,67
0791448720003023200BD0E1FAD95DA603000002112231530500363AF43259430FF5C862B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 20,0,,67
0791448720003023200BD0E1FAD95DA603000002112231636100363AF432594BE7F5C862B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 21,0,,67
0791448720003023200BD0E1FAD95DA603000002112231734100363AF4325923A7F5C862B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03
+CMGL: 22,0,,67
0791448720003023200BD0E1FAD95DA603000002112231045200363AE76CB98A4FF5C962B97E8CC7B032595BEB3E4775E6775D0E0A9FCFF2F2394C7FCBE7A0B7BB0CA2A7DB65D0F84D2E03

OK

So modem has a lot of received messages, but ModemManager does not know about them. It happens no matter if MM is talking to modem using cdc-wdm interface or AT commands (I disabled cdc-wdm using dbus rules). This problem does not happen immediately, it usually take about a day for first modem to fail, then several hours later others start to follow one by one. Restarting ModemManager fixes the problem (for some time) - it notices all messages stored in modem and reports them via dbus.