Workflow of bibo queries

[lingu]

The workflow of bibo queries use SageRun in Sage. SageRun drives an ATPA to perform a task for a QID.

The call path starts in scali invoking ProcRequestDefault. We optimize the timing of the steps and should make it complete in no more than 1s.

r1: scali
-> r3: getRequest
-> r5: ProcRequestDefalt
-> r7: queryMatch
-> r9: SageRun instances [1..llTaskCnt]
-> r11: atpa by user sage (atpa may log some info in atpa.log-$dd where dd is the datum number. Note that some log may be somewhere else.)
-> r13: atpa.sh by user sage (log in atpash.log-$dd, also check output files in the query's dir in fastdata))
-> r15: procone.py (log is perhaps in atpa.sh's log or procone2-$tycano.log)

Most of the log file locations are /thinker/local/today/users/$USER

We optimized r1 to r7 from 20+s to 3s by reducing memory footprint used by SageRun, skipping SageRunPrep and increasing atp count.

r9 and r11 still takes 13s. If we reduce the task count to 90, the time is reduced to 8s. With atpa.sh, it takes 10s.

---
20210813/lingu: log info
20210811/zhihao: link procone2.py
20201108/lingu: rename to be workflow.