Possible timestamp error in SER files?

[mcamilleri]

My apologies for the long post.

I use Sharpcap to record SER videos for occultations with GPS flash timing. I use TANGRA to produce a light curve and analyse the light flashes from the video to correct the PC time offset to UTC. Examining the video and TANGRA the frame time reference time stamps do not match the on-screen Sharpcap time stamp (first screen shot). For this example the viewed frame is the first frame where the GPS flash pulse occurs so it overlaps the start of the UTC second. This is why the video screen is white as the 100ms long GPS flash is on. The timestamp from Sharpcap is 28:067 and is end of frame. The TANGRA time stamp does not match and shows 27.677 and the displayed calculated frame mid time 27.482. The exposure time was 390 ms. The calculated mid time from TANGRA is impossible as the frame must overlap the UTC second. So somewhere in the processing TANGRA is picking up an incorrect time stamp.

I upgraded Sharpcap sometime in Nov/Dec. I have recordings from Nov 2023 and before where the timestamps match so I think the problem is associated with the update. TANGRA has not been updated in this period.

This issue is urgent as if there is a problem it will affect many observers worldwide and reports may have to be redone.

Kind Regards
Michael

image.png (272.55 KiB) Viewed 544 times

[admin]

Hi Michael,

there have been a couple of changes in the handling of the timestamps over the last six months to try to bring more consistency to things.

1) August 2023 with the release of SharpCap 4.1 - change to SER file timestamps so that they are

a) GPS based start timestamp if available for GPS cameras or
b) Estimated start timestamp based on exposure and frame received time if exposure known or
c) Estimated end of frame timestamp based on frame received time if exposure unknown

Prior to this change, category b) did not exist, so you either got frame start time for GPS cameras or estimated end time for all other cameras

2) January 2024 - the same change made to the timestamp added to the image itself, since that had been missed back in August.

If you have captured images with a version of SharpCap 4.1 before 4.1.11757 then you have probably got files where fix 1) has been applied but fix 2) hasn't, so you will have the end of frame timestamps in the image but the start of frame timestamps in the metadata.

Hope that explains what is going on - please let me know if you discover things are not working to the above explanation.

cheers,

Robin

[mcamilleri]

Thanks for your quick and very informative reply. You have great user support!

That tallies with what I saw. After processing through TANGRA with END TIME timestamps set (incorrectly as they are in fact START timestamps) the times were incorrect by the exposure time. So the fix for TANGRA should be to select START TIME timestamps when doing the analysis.

From your description from Aug to Nov/Dec 2023 the SER file time stamp would be a START TIME as when recording using a CMOS astronomy camera the exposure time would be known by Sharpcap - is this correct? And the 'bug' from Aug to Nov/Dec was that the on screen time stamp was still an END TIME timestamp. Is that correct?

I will need to review some videos taken using previous versions of Sharpcap to fully understand the situation - the checks I have done so far had the SER file time stamp match the on screen time stamp using END TIME in TANGRA. I am unsure which version of Sharpcap I was using at the time so it might not have been 4.1.

My preference would be for SHARPCAP to keep END TIME timestamps in the meta data as that is what has actually been measured. Start time timestamps are an inference so are a secondary measurement so will be less accurate and subject to error.

I will inform the IOTA admins about this situation as it likely has caught out some observers and will have caused an error in their reporting timings. Milliseconds count in occultation observations so being out by 1 frame exposure time will likely require some reports to be redone. My initial assessment is that my time calculations will not (usually) be affected as it corrects relative to the incorrectly read timestamp so it ends up correct. Some other time calculation methods might be in error.

I have had a look at the release notes and found no mention of this change. The 4.1 documentation does not seem to describe this fix - it still describes the onscreen timestamp as being at or near the end of the exposure and no mention of the start/end time reference for SER timestamps that I could find.

Michael

[mcamilleri]

Have there been any changes to timestamps for other file formats, both video and still such as FITS?

[admin]

Hi,

unfortunately the update of the documentation is a periodic undertaking, and I suspect that I updated before changing the in-image timestamp. I will try to pick that one up the next time I do an update.

For other file formats :

FITS - basically the same as SER - DATE-OBS is the inferred start time and has been for a long time. DATE-END is an additional header which is the end time (time received by SharpCap). In any case, the comments in the fits headers describe each timestamp header.

TIFF (with ASTROFITS headers) - identical to FITS

ADV - oddly only records timestamps if they come from GPS. Not sure if this is correct or even if ADV gets much use any more

Other file formats - PNG,JPG,AVI,WMV, etc - do not store exact frame timing information.

My goal here is consistency across the formats, hence I have been making changes as I find issues where other formats are inconsistent with FITS.

cheers,

Robin

[mcamilleri]

Thanks for confirming the other formats.

Would you please enable ADV to be used for other camera types. It would be a better option for occultations as the time stamps are handled automatically by the analysis software and it has richer metadata. ADV is still used for occultations with the QHY174GPS, and I think ADV is increasing in use as there is a new GPS camera Astrid that creates ADV files directly from the camera module. If you do this for ADV then I am pretty sure it would become the recommended or required format for occultation observations.

There are likely some other observation groups relying on Sharpcap for accurate timing. Meteor observations and observations of artificial satellites spring to mind as being likely to use SER format rather than FITS.

Accurate timing is the single most important thing in occultations. We would greatly appreciate if you could keep us informed about any changes affecting timing - including that in the release notes would be sufficient, and an email to info@occultations.org would be helpful.

[mcamilleri]

Just to clarify, the onscreen timestamp bug with it showing the end time rather than the start time, does that affect other formats such as AVI? I truly hope that no-one is using those timestamps for manual time input but some observers might be.

Michael

[mcamilleri]

After some more digging I did find a mention in the release notes on Aug 2023. The wording could perhaps be more explicit. I didn't spot it at the time.

"Fix SER files using timestamp of end of frame not start"

[mcamilleri]

The change in the on screen time stamps for START TIME. Is this done for all formats and both video and still? Has this been changed for FITS?

[admin]

Hi,

the in-image timestamp affects all saved formats. Unfortunately I didn't mention that one in the release notes. The first version with that change was 4.1.11757 on Jan 29th.

I have just re-checked the ADV writing code, and currently the 'SystemTime' tag for each frame written to ADV is set to the estimated end time of the frame from the system clock. The 'SystemTime' tag is commented in the ADV writing library as

Code: Select all

// Lower accuracy system timestamp for the frame. Could be used as a backup time reference in case of a problem with the main timing hardware.

Unfortunately it doesn't say definitively if it should be the start or end timestamp, which is annoying. I did look for any further online documentation, but the AstroDigitalVideo documentation web site seems to have gone - both links relevant to ADV from this page are gone : http://www.hristopavlov.net/Tangra3/

I could also add both the start/end timestamps to the ADV in the data slots used for GPS timestamps, but that 'feels' wrong - what documentation I have hints very strongly that those timestamps are meant to be the high accuracy ones from a GPS source or similar.

Since the ADV format is only really used by the occultation community, I'm happy to accept feedback on the preferred options here.

cheers,

Robin