iova faults on IB1 cmdstream

I'm seeing some logged gpu devcore dumps that are faulting on the IB1 cmdstream address (see attached), which should be impossible unless there is a problem with the pgtables.

fault-info:
  - ttbr0=0000000002853000
  - iova=0000000100015000
  - dir=READ
  - type=TRANSLATION
  - source=CP

Since the previous resume, the sequence of cmds is:

t7              opcode: CP_ME_INIT (48) (9 dwords)
0001000000001000:               0000: 70c80008 0000002f 00000003 20000000 00000000 00000000 00000000 00000000
*
t7              opcode: CP_WHERE_AM_I (62) (3 dwords)
0001000000001024:               0000: 70620002 00011000 00010000
t7              opcode: CP_SMMU_TABLE_UPDATE (53) (5 dwords)
                        { TTBR0_LO = 0x2853000 }
                        { TTBR0_HI = 0x1 | ASID = 0 }
                        { CONTEXTIDR = 0 }
                        { CONTEXTBANK = 0 }
0001000000001030:               0000: 70d30004 02853000 00000001 00000000 00000000
t7              opcode: CP_MEM_WRITE (3d) (5 dwords)
                        { ADDR_LO = 0x808 }
                        { ADDR_HI = 0x10000 }
                gpuaddr:0001000000000808
0001000000001050:                       0000: 02853000 00000001
0001000000001044:               0000: 703d0004 00000808 00010000 02853000 00000001
t7              opcode: CP_EVENT_WRITE (46) (2 dwords)
                        { EVENT = CACHE_INVALIDATE }
                event CACHE_INVALIDATE
0001000000001058:               0000: 70460001 00000031
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x400 | CNT = 2 | 64B }
                        { DEST = 0x1a8 }
                        { DEST_HI = 0x10000 }
                base register: RBBM_PERFCTR_CP[0]+0
                gpuaddr:00010000000001a8
0001000000001060:               0000: 703e8003 40080400 000001a8 00010000
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x980 | CNT = 2 | 64B }
                        { DEST = 0x1b8 }
                        { DEST_HI = 0x10000 }
                base register: CP_ALWAYS_ON_COUNTER_LO
                gpuaddr:00010000000001b8
0001000000001070:               0000: 703e8003 40080980 000001b8 00010000
t7              opcode: CP_EVENT_WRITE (46) (2 dwords)
                        { EVENT = PC_CCU_INVALIDATE_DEPTH }
                event PC_CCU_INVALIDATE_DEPTH
0001000000001080:               0000: 70460001 00000018
t7              opcode: CP_EVENT_WRITE (46) (2 dwords)
                        { EVENT = PC_CCU_INVALIDATE_COLOR }
                event PC_CCU_INVALIDATE_COLOR
0001000000001088:               0000: 70460001 00000019
t7              opcode: CP_INDIRECT_BUFFER (3f) (4 dwords)
                ibaddr:0000000100017000
                ibsize:000000fc
0001000000001090:               0000: 70bf8003 00017000 00000001 000000fc
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x400 | CNT = 2 | 64B }
                        { DEST = 0x1b0 }
                        { DEST_HI = 0x10000 }
                base register: RBBM_PERFCTR_CP[0]+0
                gpuaddr:00010000000001b0
00010000000010a0:               0000: 703e8003 40080400 000001b0 00010000
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x980 | CNT = 2 | 64B }
                        { DEST = 0x1c0 }
                        { DEST_HI = 0x10000 }
                base register: CP_ALWAYS_ON_COUNTER_LO
                gpuaddr:00010000000001c0
00010000000010b0:               0000: 703e8003 40080980 000001c0 00010000
t4              write CP_SCRATCH[0x2].REG (0885)
                        CP_SCRATCH[0x2].REG: 32397
00010000000010c0:               0000: 48088501 00007e8d
t7              opcode: CP_EVENT_WRITE (46) (5 dwords)
                        { EVENT = CACHE_FLUSH_TS | IRQ }
                        { ADDR_0_LO = 0x4 }
                        { ADDR_0_HI = 0x10000 }
                        { 3 = 0x7e8d }
                event (null)
00010000000010c8:               0000: 70460004 80000004 00000004 00010000 00007e8d
t7              opcode: CP_WHERE_AM_I (62) (3 dwords)
00010000000010dc:               0000: 70620002 00011000 00010000
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x400 | CNT = 2 | 64B }
                        { DEST = 0x1c8 }
                        { DEST_HI = 0x10000 }
                base register: RBBM_PERFCTR_CP[0]+0
                gpuaddr:00010000000001c8
00010000000010e8:               0000: 703e8003 40080400 000001c8 00010000
t7              opcode: CP_REG_TO_MEM (3e) (4 dwords)
                        { REG = 0x980 | CNT = 2 | 64B }
                        { DEST = 0x1d8 }
                        { DEST_HI = 0x10000 }
                base register: CP_ALWAYS_ON_COUNTER_LO
                gpuaddr:00010000000001d8
00010000000010f8:               0000: 703e8003 40080980 000001d8 00010000
t7              opcode: CP_EVENT_WRITE (46) (2 dwords)
                        { EVENT = PC_CCU_INVALIDATE_DEPTH }
                event PC_CCU_INVALIDATE_DEPTH
0001000000001108:               0000: 70460001 00000018
t7              opcode: CP_EVENT_WRITE (46) (2 dwords)
                        { EVENT = PC_CCU_INVALIDATE_COLOR }
                event PC_CCU_INVALIDATE_COLOR
0001000000001110:               0000: 70460001 00000019
t7              opcode: CP_INDIRECT_BUFFER (3f) (4 dwords)
                ibaddr:0000000100015000
                ibsize:000000fc

Note that the first CP_INDIRECT_BUFFER (coming from a previous submit ioctl) seems ok. There isn't a second CP_SMMU_TABLE_UPDATE because it is the same context. The TTBR0 value (which arm-smmu-qcom.c reads back from smmu registers) matches what we expect it to be.

I'm also seeing a number of crashes from com.google.android.gms.unstable which look similar (ie. iova fault on IB1 address followed by GPU hang with opcode error (opcode=0x00000000). But these predate the roll-out of e25e92e0 so I don't have devcore's to go along with them.

instagram.devcore.gz com.google.android.gms.unstable.devcore.gz