Another Offset Question

Discussion of using SharpCap for Deep Sky Imaging
SteveJP
Posts: 38
Joined: Sat Apr 28, 2018 8:35 am
Location: Melbourne, Australia

Re: Another Offset Question

#51

Post by SteveJP »

Hi turfpit
Thanks.
Your very comprehensive reply deserves a similar reply from me, but it might take me a day or two.
Just quickly, I'm not so much as concerned with how to deduce the correct Offset (aka Brightness), as much as trying to figure out why it should be increased with increasing gain. Ie what is the actual mechanism in the camera.
BTW Whilst I've been using FITS Liberator as suggested, I find that if I graph the SharpCap Histogram file, it's a lot more informative. I'll post an example soon.
Rgds
Steve
User avatar
turfpit
Posts: 1779
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: Another Offset Question

#52

Post by turfpit »

Steve

No problem, you probably have other priorities at this time of year.

The why I cannot explain but Brian can. He is following an AAVSO course and recently attended the CCD1 module which covers this aspect in great detail. When he resurfaces, expect a detailed explanation of what the mechanism is - expect some fried synapses after that.

I included the CCD and DSLR because these technologies demonstrate that the offset value is built into the camera electronics.
I graph the SharpCap Histogram file, it's a lot more informative
Yes I can see that. I use different capture packages on different operating systems, so FITS Liberator on Windows (or under WINE on Linux) gives me a generic way to deal with this aspect.

This topic is approaching ~2500 views which demonstrates how many people are interested/grappling with this 'what is the correct offset to use?' question.

Wishing you all the best for Christmas and the New Year.

Dave
SteveJP
Posts: 38
Joined: Sat Apr 28, 2018 8:35 am
Location: Melbourne, Australia

Re: Another Offset Question

#53

Post by SteveJP »

Hi Dave,

This is an example of the SharpCap Histogram plotted vs the FITS Liberator - ASI071MC Pro - 1ms exposure, Gain=90, Off=50 (Dark frame)

You can see from the mean/median, they represent the same data, but although the FITS Liberator has a lot of extra data, but the min and max histogram data only represents a tiny fraction of pixels and isn't really relevant. You can see from the Excel plot where the energy is. The mean corresponds pretty well to the offset as might be expected. 90 is about unity gain for this camera.

Cheers & Thanks
Steve
FITS-Liberator.PNG
FITS-Liberator.PNG (186.93 KiB) Viewed 3970 times
The attachment FITS-Liberator.PNG is no longer available
SteveJP
Posts: 38
Joined: Sat Apr 28, 2018 8:35 am
Location: Melbourne, Australia

Re: Another Offset Question

#54

Post by SteveJP »

... adding two pictures in-line didn't seem to work - here is the other one
SC-Hist-Plot.PNG
SC-Hist-Plot.PNG (24.37 KiB) Viewed 3969 times
User avatar
turfpit
Posts: 1779
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: Another Offset Question

#55

Post by turfpit »

Steve

There is a bug in the forum software which seems related to Previewing a post with more than one inline image. The workaround is to put the images inline but don't preview. Once the post is Submitted, subsequent edits do not mangle the inline images.

I understand the maths you have presented. For your gain setting of 90, I would investigate other offset values. How about changing the offset until the histogram is centralised and posting back 2 images equivalent to the ones you have just posted? How much horizontal scale is missing on the Excel?

The other interesting exercise would be to capture and process an image with both sets of settings and see what transpires.

My feeling is that if the DSLR and CCD designers use electronics to produce bias histograms that are balanced and centralised this doesn't happen by chance.

Dave
User avatar
turfpit
Posts: 1779
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: Another Offset Question

#56

Post by turfpit »

Steve

Can you get your FITS file and the SharpCap histogram CSV file to Robin please - I think this needs some investigation.

Dave
SteveJP
Posts: 38
Joined: Sat Apr 28, 2018 8:35 am
Location: Melbourne, Australia

Re: Another Offset Question

#57

Post by SteveJP »

Hi Dave,

I'm not sure what you mean by centralising the Histogram. The horizontal scale goes to 2^14-1 (16383), this being a 14 bit camera, I just ommited that picture there being not much to see. (Being just a photo of the inside of the lense cap!) If you integrate under the curve, you'll get 16M, this being a 16MegPixel camera.
The attachment SC-Hist-Plot-ALL.PNG is no longer available
The article you referenced http://www.stark-labs.com/help/blog/fil ... Offset.php was very interesting, particularly the discussion about gain. My full well (46,000) is much bigger than my bit depth (16383), so a set and forget gain is not on. I didn't think the discussion on offset was nearly as convincing as the discussion on gain, or maybe it just wasn't explained properly, or maybe I just don't get it. I'll be interested to hear Brian's explanation.

Cheers
Steve
PS Thanks for the tip about images - makes sense to me now.
CSV file is attached. Unfortunately, I'd deleted the FITS since I did dozens and it was filling my disk. I can do it again when I understand what you mean by "centralised"
Attachments
SC-Hist-Plot-ALL.PNG
SC-Hist-Plot-ALL.PNG (17.56 KiB) Viewed 3953 times
User avatar
turfpit
Posts: 1779
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: Another Offset Question

#58

Post by turfpit »

Steve

Centralised histograms are in the images in post Dec 22nd 11:20pm. I obtained centralised histograms on the 183 cameras by adjusting offset. At least I have a systematic method now to arrive at a sensible offset setting - better than the all too common 'play around with the sliders'. The 183s are 12-bit cameras. The DSLR is 14-bit and does it right for no effort, the CCD is 16-bit and does it right for no effort. At a guess, I think your FITS Liberator/excel anomaly will have something to do with the camera being 14-bit. We will get some more informed input on this when Brian is back in circulation.

Dave
SteveJP
Posts: 38
Joined: Sat Apr 28, 2018 8:35 am
Location: Melbourne, Australia

Re: Another Offset Question

#59

Post by SteveJP »

turfpit wrote: Mon Dec 24, 2018 12:56 am Steve

Centralised histograms are in the images in post Dec 22nd 11:20pm. I obtained centralised histograms on the 183 cameras by adjusting offset. At least I have a systematic method now to arrive at a sensible offset setting - better than the all too common 'play around with the sliders'. The 183s are 12-bit cameras. The DSLR is 14-bit and does it right for no effort, the CCD is 16-bit and does it right for no effort. At a guess, I think your FITS Liberator/excel anomaly will have something to do with the camera being 14-bit. We will get some more informed input on this when Brian is back in circulation.

Dave
Dave,
Dec 23 10:20 am for me, being 11 hours ahead of you.
I'm still not sure I understand your concept of balance. I'll get back to this in a few days. Christmas eve here now and I have to stop "doing" astronomy!
Seasons greeting to you.
Steve
User avatar
admin
Site Admin
Posts: 13173
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Another Offset Question

#60

Post by admin »

Hi folks,

so I guess I'm arriving a little late at the party here, but I'm going to do my best to provide hopefully definitive answers on how the offset control works and how and why it should be set.

I guess the first question to answer is why do we have an offset control? Essentially the purpose of the offset controlled is to avoid having any pixel readout a value of zero in any image. Why do we want to avoid a value of zero? Because if we get a value of zero we can never be sure that it was truly meant to be a value of zero i.e. that it shouldn't have been a negative pixel value that we didn't get because zero is the smallest value that pixel can readout. This is particularly important when you stack frames after capture.

Consider a situation where a dark area in an image has an average ADU 500 with a variation of between 200 and 800 due to various noise factors (see this thread – viewtopic.php?t=456 – for lots of detail on different types of noise that affect images). That setup would be fine as we would never see pixels with zero value in that region of the image and if we stacked many such images they would stack correctly giving an average value of about 500 in the stack. Now imagine that we turn the offset down so that the average value in that region of the image was no longer 500 but was 100, the random range of pixel values in that area would now really need to be between -200 and 400. However the pixel values that are read out from the camera would be between zero and 400 because any negative value would be reported as zero. When we stack these frames we won't get a correct average value in the stack because all the values that should have been negative being boosted up to 0. This can be a particular problem for master dark frames – if the histogram of the individual dark frames is clipped against the left-hand side then the average value of the master dark will be too high due to the effect described above.

Okay, so we now see why we want an offset control to allow us to ensure that we not getting zero pixel values read out from the sensor but we can still use as much of the sensors ADU ranges possible (a fixed, large offset would also remove zero values from the sensor, but would limit the amount of the ADU range we could use).

How do we set an offset control? The simplest guideline is to take dark frame with all the settings that you intend to use (i.e. correct exposure, temperature, gain et cetera) and adjust the offset to just separate the histogram of that dark frame from the left-hand side. If you don't intend to use dark frames then you could perform the same trick with a light frame (this will likely give you a smaller offset value as the sky background brightness will also tend to shift the histogram peak to the right and separated from the left-hand side of the graph).

The question at the start of this thread was whether the correct offset changes with the gain of the camera and why that should happen. The answer to this is – I think – it depends on the model and make of camera. As we know the width of the histogram peak increases as the gain is increased because gain is effectively just a multiplication factor that is boosting a weak signal into a stronger signal and at the same time changing a narrow histogram peak into a wider one. Inside the actual camera itself both the gain and the offset have to be applied before the analogue to digital conversion stage – the gain is basically a voltage multiplier that is applied to the signal coming from each pixel to make a stronger voltage signal to be fed into the ADC, while the offset is essentially a small extra voltage being added to every pixel to increase the pixel value.

Some camera brands/models seem to auto scale the offset value when the gain is adjusted without requiring adjustments of the actual offset value i.e. if you turn up the gain you will see the histogram peak not only widen but move to the right automatically in order to keep it separated from the left-hand side of the histogram graph. On other types of cameras when you turn up the gain the histogram peak stays in the same place and just gets wider and with such cameras there is a definite requirement to increase the offset value as the gain is increased to ensure separation.

Hope that all makes sense – if there are still unanswered questions lurking in the middle of this thread then please let me know and I'll do my best to answer them if I know the answer.

Cheers, Robin
Post Reply