Service management of Sage - D

[lingu]

(07-12-2016 05:54 PM)YU_Xinjie Wrote:  goal

1. the script can be used in both sage_portal & glad_portal to start/stop/status sage.
2. user can start sage before glad, and stop sage after glad.

Sage needs to be started automatically by glad for a batch of samples (dataflows).

It is okay to have a user manually start it at present. But this is not the way we want for the future. It adds a human operation. An extra human operation makes the software a bit less easy to use, and a whole lot easier to make a mistake.

So, when designing, you can make compromises now, but keep in mind what we want in the long run.

Quote:The script would replace the current start.sh/stop.sh/status.sh in the future.
The script may inspire us about how to set up regression tests for sage.

This is good thinking -- using one way, not two ways, to do one task.

Quote:[quote]
For shbio's project, I have already written a script /thinker/dstore/run/sage to support a part of the goal. I would try to generalize it.

config design

The brief idea is to use auntie to send a file to glad_portal. Then check the file content.

This is too heavyweight -- it may even disrupt Sage if it is busy.

Sending a file takes 2-3s. This is considered long/heavyweight.

OK to keep the current design if it has worked. But please be aware of latency in a pathetic way -- otherwise our software would be as slow as Hadoop in 2 years.

Quote:1.
require sage_user can password-less login sage_user@sage_portal

This is fine if all are sage_user.

Quote:copy private key of sage_user@sage_portal into $gb/conf/sage_key

$gb is owned by glad_user and all operative users can access it. So this is a security vulnerability. OK now. But make a TODO to fix it later.

Quote:Then glad operative user & sage_user can use sage_key to execute remote command to sage_user@sage_portal.

2. The glad_portal IP can be get from $gb/conf/config.sh

How (a convar like glad_portal?) Have we made sure such a varaible must exist in $gb/conf/config.sh. If not, please add a note there "need implement" so that we know to create a TODO item and implement this later after this design is endorsed.

Quote:3.
create a config file $gb/conf/sage_portal_ip to record the IP of sage_portal. Or we can also write it into $gb/conf/config.sh.

I prefer to create a convar like glad_sage_user and store it in $gb/conf/config.sh

Quote:4. let sage_user@sage_portal can password-less login glad_user@glad_portal.

That's fine -- we assume sage_user is no a human operator and can be a powerful guy.

Quote:start sage

Code:

remote_cmd () {
ssh -q -o StrictHostKeyChecking=no -i $gb/conf/sage_key $sage_user@$sage_portal "$1"
}
[/quote]

Don't have to specify this -- everybody can find a way to run a remote program without being tremendously wrong. So it's not a key design detail.

We require engineers use psudocode to write down the design if it involves some logic. But code is not appropriate psudocode.

[quote]
wait_sage_start () {
content="This is test content."
echo $content > $curdir/${USER}_sage_testfile
test_file=/thinker/bin/ephemeral/667.sagetest

# two minutes timeout
try_cnt=40
while [[ ("$try_cnt" -gt "0") && ((! -f "$test_file") || ("$(cat $test_file)" != "$content")) ]]; do
  timeout -s SIGINT 1 bash -c "sage_user=$sage_user $curdir/auntie ^$curdir/${USER}_sage_testfile $glad_portal 667 sagetest" || echo -n ''
  echo "Trying to start Sage..."
  sleep 2
  try_cnt=$((try_cnt-1))
done
rm -f $test_file
}
[/quote]

Too long wait. If it does not work in 30 sec, report an error and stop.

[quote]
start_sage () {
# check whether sage is really stopped.
local tmp="$(status_sage)"
if [[ "$tmp" != "Sage is stopped." ]]; then
  echo "Sage is already started."
  echo ""
  echo "Status: "
  echo "$tmp"
  return 0
fi

remote_cmd "screen -dmS sage_service bash -c 'cd ~/sage; ./start.sh'"
wait_sage_start
echo "Sage is started."
}

What if the wait_sage_start fails?

We need a way to check Sage has started. I thought this is what I thought you were doing. But it looks like here if auntie continues to fail the wait still completes without an error halt.

Quote:stop sage

Code:

stop_sage () {
# stop sage
remote_cmd "cd ~/sage; ./stop.sh" > /dev/null

# kill all sage_service screens
remote_cmd "
for session in \$(screen -ls | grep 'sage_service' | grep -o '[0-9]\+\.' | grep -o '[0-9]\+')
do
    screen -S \${session} -p 0 -X quit
done
"
[/quote]

Looks good.

[quote]
# check whether sage is really stopped.
local tmp="$(status_sage)"
if [[ "$tmp" == "Sage is stopped." ]]; then
  echo "Sage is stopped."
else
  echo "Sage failed to stop."
  echo ""
  echo "Status: "
  echo "$tmp"
fi
}

OK.

Quote:status sage

Code:

status_sage () {
local output="$(remote_cmd "cd ~/sage; ./status.sh")"
if [[ "$output" == "Sage is stopped." ]]; then
  echo "Sage is stopped."
else
  echo "$output"
fi
}

OK. In the future, we should consider using autnie to directly query Sage to get the status.