surprised by command ordering hanging a sequence program

[ted_barnes]

A question about the ordering of commands in a sequence. In this code snippet the ordering of 2 lines appears surprisingly crucial. This is the final part of a sequence program that earlier opened a camera, loaded a profile, and did many star hops and captured several images:

[many lines above]
WAIT UNTIL LOCALTIME "1:18 AM"
LIVE MODE
SET EXPOSURE TO 0.10
DELAY 10
CAPTURE 600 LIVE FRAMES
CAMERA CLOSE
COUNTDOWN FOR 10 SHOWING "Test Sequence testseq_011224_fin2 Completed in:"
PLAY SOUND Alert
END SEQUENCE

The above code hangs just after DELAY 10 and takes no live frames. However when I switch the order of just 2 lines, to

[unchanged]
WAIT UNTIL LOCALTIME "1:18 AM"
SET EXPOSURE TO 0.10
LIVE MODE
DELAY 10
CAPTURE 600 LIVE FRAMES
CAMERA CLOSE
COUNTDOWN FOR 10 SHOWING "Test Sequence testseq_011224_fin2 Completed in:"
PLAY SOUND Alert
END SEQUENCE

The program now runs through completion, and captures the 600 frames.

Why is the order of the EXPOSURE and LIVE MODE commands important? What is the general rule for when and where you can change the exposure time, for example when STILL MODE is involved as well?

Thanks!
Ted Barnes

[admin]

Hi Ted,

I suspect it may be an issue specific to your make (and perhaps model) of camera - I certainly wouldn't expect the ordering to cause an issue at that point, but maybe the change in exposure ends up running at a point where the camera is still getting started in live mode and that causes the problem...

Can you let me know what make/model of camera you are using so that I can test it out. It would also be interesting to know if

a) it happens in a simplified version that does the minimum needed to trigger the problem - probably something like set still mode, set long exposure, capture one frame, set live mode, set short exposure, etc

b) if adding a delay between live mode and set exposure helps.

cheers,

Robin

[ted_barnes]

Hello Robin,

I am using a ZWO ASI432MM, good for small images sizes and rapid transfers. Set to max gain and full field, timestamps on.

I did wonder if the finite time required for each step might sometimes result in the camera not scheduling its actions properly, and trying to do a new action before the other is finished, thus somehow causing itself to hang.

Here are 2 versions of a 20 line program snippet.

This one runs

SEQUENCE
COUNTDOWN FOR 10 SHOWING "snippet_012524_a Start in:"
TARGETNAME snippet_a
CAMERA OPEN "ZWO ASI432MM"
LOAD PROFILE "OC_ASI432MM (ZWO ASI432MM)"
LIVE MODE
DELAY 10
SET OUTPUT FORMAT TO "JPEG files (*.jpg)"
SET EXPOSURE TO 1
SET RESOLUTION TO 1608x1104
SET COLOUR SPACE TO MONO8
SET GAIN TO 350
STILL MODE
SET EXPOSURE TO 0.1
LIVE MODE
DELAY 10
CAPTURE 10 LIVE FRAMES
CAMERA CLOSE
COUNTDOWN FOR 10 SHOWING "snippet_012524_a Completed in:"
END SEQUENCE

but another (name changed to snippet_g), with lines 14 and 15 switched to opposite order,

LIVE MODE
SET EXPOSURE TO 0.1

reliably hangs.

I have been restarting SC before each test to avoid possible initial condition dependence.

###

I too wondered about inserting a delay in the hanging program after the LIVE MODE statement.

This variant of program g (hanger)

LIVE MODE
DELAY 20
SET EXPOSURE TO 0.1

gives a program that wonder of wonders now runs. But a reduced DELAY hangs it again. DELAY 0.01 hangs, DELAY 0.05 runs, and DELAY 0.02 looks like the outcome may depend on which way the wind was blowing. I hope not, I would like to think that this programming is deterministic.

I can see the timestamp stop on the ones that hang and do no captures. It's like they are stuck in one frame.

I look forward to hearing what you conclude about this.

Ted

[admin]

Hi Ted,

thanks for sharing - I have run through the version that freezes for you with each of the ZWO camera that I have here and for me, everything is fine.

Thinking about what could make my results different from yours...

* I don't have the 432MM camera, so maybe it is something specific to a particular model (I tried 174MC, 071MC, 1600MM)
* I don't have the capture profile settings that your sequence loads, so maybe something in those settings has an effect (I think unlikely - you set the exposure again after loading the profile and that would be the most likely candidate).
* Different computer - this could very well be something related to the speed that things happen

I think the last is most likely based on your experience with adding delays - SharpCap sends commands to the camera, which then gets on and tries to do what it is asked. SharpCap then runs more of the sequence and sends more commands, and if the new commands happen to arrive at a bad moment in relation to the ones already in progress, something hangs up. It could be in the camera itself or in the ZWO code that handles it, or even in the low level camera code in SharpCap.

Since I can't get it to happen here, the only realistic way to work out what is going on is to enable additional logging of camera communications and then run your sequence that causes the hang - this should then record in the log all the commands SharpCap sends to the camera, and hopefully I can spot why/how it gets stuck. It does make for a large log file though! See https://docs.sharpcap.co.uk/4.1/#Logging%20Settings for details on how to turn on additional logging.

cheers,

Robin

[ted_barnes]

Hello Robin,

Here is the capture profile that the snippet programs both load, specifically OC_ASI432MM. It's my generic for that camera.

(OC stands for Occultation Capture, which is what I have been doing. Nothing really funny except the max gain and short exposure time, one needs to capture rapid short frames of faintish stars to confirm and time an occultation. I wrote a procedure to do those automatically, which worked right up until the end.) Alas I missed an occultation by a Jupiter Trojan.

I will look into generating a fuller log file and get back to you.

My current suspicion about lessons learned is that I should switch to STILL MODE before doing a SET EXPOSURE.

I once wrote a nested loop sequence program that looped over camera gain AND exposure, to generate a pallet of frames to choose from. I wonder if more caution is advisable than I used at the time.

Ted

########################################
# OC_ASI432MM:

Pan=0
Tilt=0
Output Format=JPEG files (*.jpg)
Binning=1
Capture Area=1608x1104
Colour Space=MONO8
Temperature=27.2
Turbo USB=80(Auto)
Flip=None
Frame Rate Limit=120 fps
Gain=350
Exposure=100.0000ms
Timestamp Frames=On
Brightness=1
Auto Exp Max Gain=175
Auto Exp Max Exp M S=30000
Auto Exp Target Brightness=100
Trail Width=3
Minimum Trail Length=100
Trail Detection Sensitivity=9
Remove Satellite Trails=Off
Background Subtraction=Off
Planet/Disk Stabilization=Off
Banding Threshold=10
Banding Suppression=0
Apply Flat=None
Hot Pixel Sensitivity=5
Subtract Dark=None
NegativeDisplay=0
Display Black Point=0
Display MidTone Point=0.5
Display White Point=1
Notes=
ZWO Focuser (1)=9460
Focuser Temperature=22.9400005340576

########################################

[ted_barnes]

Robin,

I ran the version of the program than hangs (LIVE MODE followed immediately by SET EXPOSURE) with additional logging enabled, as you suggested. The log file is about 110K lines, 17MB. I chopped out the first ca. 1200 and last ca. 500 lines fyi and have attached those files. Are they helpful in understanding what the camera has done or failed to do?

Tnx
Ted

[admin]

Hi Ted,

in the 'last lines' part we can see the aftermath of the problem - SharpCap is repeatedly asking the camera for video data and always getting the response that there is none available. Whatever triggered that to happen is probably buried somewhere in the middle of the main part of the log, and is unlikely to be obvious to spot either I can also see from the end part of the log that the shutdown of the camera isn't going as it should - the last line but 4 suggests that the attempt to shut down the camera has actually frozen and is stopping SharpCap from responding.

All this ties in with a theory that the problem has somehow happened in the ZWO software to the control the camera, which has stopped responding as it should as a result of the earlier steps. Unfortunately, taking the diagnosis any further is going to be very hard, particularly since it is not affecting all models of camera - probably the best bet here is to just pick the order that works and stick with it

cheers,

Robin

[ted_barnes]

Hello Robin,

The "safe order" was 'SET EXPOSURE texp' followed by 'LIVE MODE'.

I suppose one can use the opposite "risky order" order as well, as long as a 'DELAY tdel' buffer is inserted to give the camera time to execute each command, e.g.

LIVE MODE
DELAY 10
SET EXPOSURE 0.1

I actually found that a 1 sec delay more than sufficed to stop my sequence program from hanging.

But when recording occultations one wants to avoid surprises in the last few seconds, so I will also do several test runs of the same program beforehand. Always a good idea ...

Ted

[ted_barnes]

Robin,

Sorry, I'm sure you have better things to do than staring at this log file. But can you have a quick look at this please?

I noticed a two funny lines in the first part of the log. They can be followed amid all the clutter ( I called ... ) by their ordered time stamps. One switches the file type to avi (a General preference I gave SC outside the sequence, but here i really wanted jpg frames), and the other announces an exposure time of 0 ms.

###############

in void SharpCap.Base.BaseImageProvider.Start()
...
Info 13:54:45.099835 #1 Changing output format to AVI files (*.avi) for short exposure based on user preference of avi

in void SharpCap.Base.CameraProxyBase.GrabThreadProc()
...
Info 13:54:45.146794 #22 Switching ZWO camera into video mode, exposure is now 0ms

###############

Might this be where a spurious exposure time of 0 is set?

Ted

[admin]

Hi Ted,

the first one is a response to the capture profile being loaded - the exposure being set by the profile is 100ms, and since the output format is (at least initially) in 'auto' mode, the short exposure triggers the code to pick a video file format (avi) rather than a still file format.

The second line showing a zero exposure is just an artifact of the ordering of some things in the SharpCap code - the value used for that logging is not yet initialized during the startup of the camera, so a default value of zero gets used during startup rather than the true exposure. The camera exposure is not really set to zero at that point, but it is a small bug since it means the camera gets set up in video mode briefly before switching to still mode if you start with a long exposure. I will look into fixing that.

cheers,

Robin