REST API Importing results endpoint takes too long
I've been testing using importing results via REST API in production, this led to discovering some issue that couldn't be easily seen in small testing environment.
Importing test results takes in some cases more than 5min which is ok, importing speed is not my concern. Current uWSGI config is defaulted to terminate connection after 3min which is also ok. But I think we can improve how importing is handled. After some investigation I can say that this part code consumes the most time of importing results. For example in one case in takes 262s and rest of the code take less than 20s. And the whole function is wrapped with @transaction.atomic
hopefully Django/Postgres are smart and do not lock whole database for that much time. Anyway, what I think would be good solution is to return HTTP response as soon as we validate data and make sure it's correct. Then we create some queue system and we put execution of the rest of the code in queue. We could only put this last expensive block of code to queue. What do you think about that?
The easiest solution: Bump/parametrize uWSGI timeouts so connection and with that workers are not terminated. Then importing process can finish (as the whole function has to be executed without errors/termination because it's treated as transaction)