Experimental Physics and
Industrial Control System

Wesley Moore <[email protected]> · Mon, 24 Sep 2012 09:40:09 -0400 (EDT)

> Let me re-state the issue, so we are clear what's happening:

RTEMS is the client IOC running the sequencers and is rebooted for testing this issue.  VxWorks is the server AND is not rebooted.

> Scenario 1: you are using either static assign, or assign at runtime,
> but in
> the latter case for each channel you assign at most once. This works
> as
> expected.

True.

> Scenario 2: you re-assign at runtime. This fails (sometimes?), when
> some
> participant is rebootet. I guess it is not the IOC hosting the client
> (i.e.
> the SNL program) that gets re-bootet, but the server (where the PV
> resides),
> because you are talking about failing re-connects. It is not
> completely clear
> to me which side is VxWorks, which is RTEMS, and which causes
> problems when
> rebootet, but my guess is that: the SNL program runs on a VxWorks
> IOC; the
> servers are VxWorks and RTEMS; re-connect fails only if the server OS
> is RTEMS
> and if the channel gets re-assigned at runtime.

This is where we get out of sync...  The VxWorks server is not rebooted.  When RTEMS is using dynamic re-assign the trouble starts.  An attempt to illustrate what I mean:

string outputStatus;  assign outputStatus;        /* also tried the older assign to "" */
string pvName;
int    outputSelect;  assign outputSelect to "myOutputSelect";
...
pvGet(outputSelect);
sprintf(pvName, "myOutputStatus%d", outputSelect);  /* ex. pvName = "myOutputStatus0" */
pvAssign(outputStat, pvName);
/* set new value and pvPut */
/* repeat... (set pvName again, pvAssign, etc) */

I only have issues when re-using an assigned variable (i.e. outputStatus).  When using this re-assign scheme, sometimes the RTEMS client never connects to the VxWorks server.  Sometimes it does and then soon after dumps Virtual Disconnect errors.  De-assigning outputStatus in between pvAssigns produced the same results.  pvAssign always returns pvStatOK.  pvPut/pvGet returns pvStatDISCONN.

Using Scenario one, it never disconnects and all channels work every time...so far.  

> 
> Jeff:
> 
> Dynamic re-assign means in CA terms that the sequencer will issue a
> ca_clear_channel (for some connected channel), immediately followed
> by a
> ca_create_channel (to some other PV). Hmmm, what happens if there is
> a race
> between the ca_clear_channel call and the disconnect event caused by
> the IOC
> reboot? (Just think loud here.)
> 
> It should be possible to devise a simple test client that issues a
> long
> sequence of such create/clear calls to check whether this is a CA
> problem.
> Unfortunately my automatic test setup does not yet support RTEMS, so
> this will
> take some time for me to do.

I should be able to run this test and maybe help with your setup.  If you have something specific you want to try, get with me offline.  Provided it is a race condition, could we test with something simple like: de-assign, sleep, re-assign?

Wesley

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System