Experimental Physics and
Industrial Control System

Benjamin Franksen <[email protected]> · Mon, 24 Sep 2012 17:03:47 +0200

On Monday, September 24, 2012 09:40:09 you wrote:
> > Let me re-state the issue, so we are clear what's happening:
> RTEMS is the client IOC running the sequencers and is rebooted for testing
> this issue.  VxWorks is the server AND is not rebooted.

Ah... ok.

> > Scenario 2: you re-assign at runtime. This fails (sometimes?), when
> > some
> > participant is rebootet. I guess it is not the IOC hosting the client
> > (i.e.
> > the SNL program) that gets re-bootet, but the server (where the PV
> > resides),
> > because you are talking about failing re-connects. It is not
> > completely clear
> > to me which side is VxWorks, which is RTEMS, and which causes
> > problems when
> > rebootet, but my guess is that: the SNL program runs on a VxWorks
> > IOC; the
> > servers are VxWorks and RTEMS; re-connect fails only if the server OS
> > is RTEMS
> > and if the channel gets re-assigned at runtime.
> 
> This is where we get out of sync...  The VxWorks server is not rebooted.

Ok, this is a completely different scenario then.

> When RTEMS is using dynamic re-assign the trouble starts.  An attempt to
> illustrate what I mean:
>
> string outputStatus;  assign outputStatus; string pvName;
> int    outputSelect;  assign outputSelect to "myOutputSelect";
> ...
> pvGet(outputSelect);
> sprintf(pvName, "myOutputStatus%d", outputSelect); pvAssign(outputStat, 
pvName);
> /* set new value and pvPut */
> /* repeat... (set pvName again, pvAssign, etc) */

Do I read this correct: After the program starts, it enters a loop in which it 
reassigns one of the variables to another PV, then pvPuts, etc and does this 
every time the outputSelect PV changes. This works fine, all is well. Then you 
reboot the IOC. The IOC come up again, the rogram is started, initial assigns 
do connect, then some time later (maybe only a short time) one of the re-
assigns fails, and the client CA library says it's got a virtual circuit 
disconnect? This sounds like a CA problem to me, but I could be wrong of 
course. I wonder: is this maybe the first time the program accesses a PV on 
this specific IOC? In this case this could be the first time the server learns 
that the old client connection is dead.

> I only have issues when re-using an assigned variable (i.e. outputStatus). 
> When using this re-assign scheme, sometimes the RTEMS client never connects
> to the VxWorks server.

Does this mean: to whatever PV you want to switch, it fails to connect if the 
PV resides on the same (remote) VxWorks server? Does it work if you switch to 
a PV on another server?

> Sometimes it does and then soon after 

Soon after what? After it connects?

> dumps Virtual
> Disconnect errors.  De-assigning outputStatus in between pvAssigns produced
> the same results.  

(To be expected, as this is how re-assign is implemented.)

> pvAssign always returns pvStatOK.  

(Since pvAssign does not wait for the connect callback this is to be 
expected.)

> pvPut/pvGet returns pvStatDISCONN.

Hm, yes, it would.

> > It should be possible to devise a simple test client that issues a
> > long
> > sequence of such create/clear calls to check whether this is a CA
> > problem.
> > Unfortunately my automatic test setup does not yet support RTEMS, so
> > this will
> > take some time for me to do.
> 
> I should be able to run this test and maybe help with your setup.  If you
> have something specific you want to try, get with me offline.  Provided it
> is a race condition, could we test with something simple like: de-assign,
> sleep, re-assign?

Let us talk about tests as soon as I have understood what exactly goes wrong.

Could you cook your program down to small example that still exhibits the 
failure? Maybe switch between just two different PVs? It would also be 
interesting to know if the re-assign happens soon after the program (and the 
IOC) get started, and whether the amount of time between re-start and re-
assign matters at all (for whether the error happens or not).

Cheers
-- 
Ben Franksen
()  ascii ribbon campaign - against html e-mail 
/\  www.asciiribbon.org   - against proprietary attachments

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System