Question about Timestamp handling

[mcamilleri]

Hi Robin,

Thanks for implementing ADV format for non-GPS cameras. I have been using it for many months now for occultation observations and it has worked very well, with much smaller file sizes and less processing load and fewer dropped frames.

My question is about how timestamps are handled.
1. What rounding method is used to form the ms (3 d.p.) figure? Down, nearest or up or something else?
2. I think the timestamp is the time that SharpCap picks up the frame from the USB. Is that correct?
3. Is there any way with USB 3 to instead use the logged time that the frame hit the USB interface (assuming that it is logged)
4. Video vs Still mode. Video mode I think has the camera send a steady stream of images governed by the camera, so has a stable frame rate and exposure and minimal dead-time. What happens in Still mode? How is the next frame triggered? For example, is it before write to disk, or after write to disk is completed. I am trying to get an idea of how large and consistent a gap there may be in the frames.

Thanks
Michael

[admin]

Hi Michael,

Let's see on the ADV questions...

1. SharpCap doesn't round the timestamps itself, but it looks like the ADV library does when creating the milliseconds since epoch value - it just does integer division, so will always round down. By the look of things it may be storing a 'nanoseconds since epoch' value too - not sure if that actually gets to the file or not though.

2. & 3. The timestamp is set when SharpCap is delivered the frame from the camera SDK software. In general, timings before that point are unknown - nothing records how long the frame sat on the camera before beginning USB transmission, or how long the transmission took, or how long it was processed in the camera SDK before being handed to SharpCap.

4. You are correct, in video mode, the camera will start capturing a new frame by itself either immediately or soon after the previous frame finished. This will typically be close to immediate for longer exposures (where the exposure is longer than the USB download time for the image), but will be delayed for shorter exposures where the USB download time is limiting the frame rate to avoid having a second frame ready before the first has finished transferring to the PC.

In still mode, SharpCap requests each frame individually - this happens pretty quickly after the previous frame is handed to SharpCap (before any image processing, saving, etc), but there will still be some delay both in the SharpCap code and in the time for the 'new frame please' message to be passed to the camera and acted on.

From experiments done with GPS cameras, it seems that the frame rate and lag are pretty stable as long as the camera settings are kept the same, but when you start changing settings (particularly exposure, image/ROI size and 'turbo USB' settings), you can't count on the values that you measured/estimated at other settings.

Hope this helps,

Robin

[mcamilleri]

Thanks. That is very helpful.

So late or dropped frames are inevitable at fast rates using Video mode. In Still mode it seems like it will request the next frame as soon as the current frame is received then that will wait in the buffer until the SDK and Sharpcap can pick it up. Maybe that will give more consistent behaviour when the frame rate is too high.

I have done extensive testing using GPS flash timing and the delays are usually consistent to within 1-2 ms for the same camera settings. It varies depending on ROI, tilt, pan, binning, colour space and bit depth. It doesn't vary with gain. It does not vary with exposure as long as the exposure time is long enough to avoid dropped frames. I doesn't seem to vary with USB speed/mode unless the setting is very low, but I have not tested that much - I just run at full throttle or a high setting.

I have not done any testing of Still mode. I usually run in Video mode to ADV and I don't know if I can save to ADV using Still mode. Video mode seems to give a more consistent frame rate than Still mode, which is why I use Video. I'll do some testing to try to understand the behaviour.

Thanks again for your help and great customer support.

Michael

[admin]

Hi Michael,

you are correct that you cannot save to video file formats (ADV, SER, AVI) with the camera in still mode, so that rules out deliberately using the camera in still mode I think. On the other had, some cameras switch to still mode automatically above a threshold exposure length (about 2s) as it turns out to be easier to handle cancelling a frame in progress that way. Additionally, QHY cameras have an option to 'Force still mode' which makes them run internally in still mode when SharpCap is in video mode.

cheers,

Robin

[mcamilleri]

Hi Robin,

I am wanting to confirm how the timestamps and dropped frames behave when recording to a video format or FITS files at short exposure, frames rates ~3-25 fps.

This is what I have found from testing and checking the time between frame timestamps and against GPS flash timing. Would you please tell me if this is the expected behaviour.

As part of my Occultation Manager Tool I have timestamp checking and a PC Performance Testing process which shows how the timestamps are affected by PC processing load or USB traffic in real time. The purpose of these tools is to help occultation observers find out what frame rates and PC operating conditions are safe to use without running into problems with inaccurate timestamps or dropped frames.

Recording to ADV, SER or AVI

When recording to ADV, SER or AVI the camera sends a steady stream of frames. If the PC can keep up the time between timestamps is steady at the nominal exposure to 1-2 ms (rounding to 1 ms and some jitter).

Occasionally there may be a late timestamp where a single frame timestamp is delayed by a few ms. The next frame is then captured on time. So this gives an interframe sequence like 40, 40, 48, 32, 40, 40... for nominal 40 ms exposure. This is perhaps due to windows processes competing with SharpCap or USB traffic.

If the exposure time is too short then these late timestamps become much more frequent, and if pushed too far there start to be dropped frames. So the interframe sequence might be something like 40, 40, 63, 17, 40, 40. It seems that the camera keeps sending a steady stream, but windows drops the frame then just picks up the next available frame when it has recovered. So the timestamps are at or close to the actual time, but there will be gaps in the sequence due to dropped frame.

Recording to FITS

When recording at video rates to FITS format the behaviour is different and SharpCap will use the internal camera buffer if it can't keep up. Frames will not be dropped unless the buffer fills up, but the buffer will be read after the recording finishes. The interframe timestamp interval may be some ms longer than the nominal exposure and the frame timestamps could be many seconds later than the actual timestamp of the camera exposure. Timestamp jitter is higher than in video mode as the buffer is read out one frame at a time when SharpCap requests the frame

Behaviour under normal load

When the frame rate is not too high (<1/3 of max frame rate) and the PC and USB load is not high.

Video format: No dropped frames, jitter of 1-2 ms, occasional late frame that catches up.
FITS format: No dropped frames. Jitter of a few ms. occasional late frame that catches up.


Behaviour under high load
When the frame rate is higher than ~1/3 of max frame rate or the PC has some competing loads.

Video format: Same as before but occasional dropped frame. Timestamps are still accurate or only slightly delayed for some frames
FITS format: No dropped frames as they will be read from the camera buffer. Timestamps may be late by more than than one exposure.

Behaviour at excessive frame rate or excessive load

When the frame rate appoaches or exceeds the max frame rate, or very heavy PC loads.

Video format: Lots of late and dropped frames. Timestamps are still accurate or delayed by less than one exposure time.
FITS format: No dropped frames as they will be read from the camera buffer unless if fills. Timestamps will be late by more than one expoure and may be seconds or even minute late

[admin]

Hi,

that's not quite what I would expect given how the internals of SharpCap work... The handling of saving to file is completely separated from the mechanics of actually getting frames from the camera, so choosing to save as FITS will not (can not) directly affect how the onboard memory of the camera is used or not.

At a high level, things work like this

* SharpCap fetches frames from the camera as soon as they are available. In video mode the camera starts frames on its own and sends them to the PC and may buffer frames. In still mode SharpCap requests each frame individually and generally there is no buffering on the camera. Frames may be dropped on the camera or in transit via USB to the computer, or on very slow hardware if SharpCap cannot keep up with fetching the images. Timestamps are added at this point, immediately after the image data is delivered to SharpCap

* Each frame fetched from the camera passes into a processing pipeline where image processing (if any, such as dark/flat/etc) is applied. If the processing takes significant time then there can be dropped frames at this point if the next image arrives before the previous one has finished processing. If you have nothing turned on in the 'pre-processing' section of the controls then this pipeline is effectively doing nothing significant.

* If saving, frames are added to a queue of frames to be written out to file. If there is not enough memory available then frames can be dropped at this point - this indicates that the memory cache has filled because saving to file cannot keep up with the arrival of frames from the camera

* Separate code fetches the frames to be saved out of the queue and writes them to the file format you have chosen (FITS, SER, etc).

* (some) frames are sent to the display - if the frame rate is low then all are displayed, but as it rises, SharpCap will reduce the fraction sent to the display. SharpCap also reduces the display update if the PC load seems high, prioritizing doing stuff like saving or live stacking over updating the display 20 times per second.

The behavioural difference between FITS and other formats should only be noticeable if the memory cache fills because FITS is slower to write than SER. You can spot that happening by keeping an eye on the status bar at the bottom left, which will show the number of frames of memory cache in use. If this stays low during capture then the writing to disk is keeping up with the capture and you will not get dropped frames due to the speed of writing. If the frames in use climbs during capture, then the writing to disk is struggling to keep up and you will start to lose frames once the memory is full.

cheers,

Robin

[mcamilleri]

Thanks. I'll do some more tests to try to understand the behaviour better.

Michael