Another Offset Question

[admin]

Hi folks,

so I guess I'm arriving a little late at the party here, but I'm going to do my best to provide hopefully definitive answers on how the offset control works and how and why it should be set.

I guess the first question to answer is why do we have an offset control? Essentially the purpose of the offset controlled is to avoid having any pixel readout a value of zero in any image. Why do we want to avoid a value of zero? Because if we get a value of zero we can never be sure that it was truly meant to be a value of zero i.e. that it shouldn't have been a negative pixel value that we didn't get because zero is the smallest value that pixel can readout. This is particularly important when you stack frames after capture.

Consider a situation where a dark area in an image has an average ADU 500 with a variation of between 200 and 800 due to various noise factors (see this thread – viewtopic.php?t=456 – for lots of detail on different types of noise that affect images). That setup would be fine as we would never see pixels with zero value in that region of the image and if we stacked many such images they would stack correctly giving an average value of about 500 in the stack. Now imagine that we turn the offset down so that the average value in that region of the image was no longer 500 but was 100, the random range of pixel values in that area would now really need to be between -200 and 400. However the pixel values that are read out from the camera would be between zero and 400 because any negative value would be reported as zero. When we stack these frames we won't get a correct average value in the stack because all the values that should have been negative being boosted up to 0. This can be a particular problem for master dark frames – if the histogram of the individual dark frames is clipped against the left-hand side then the average value of the master dark will be too high due to the effect described above.

Okay, so we now see why we want an offset control to allow us to ensure that we not getting zero pixel values read out from the sensor but we can still use as much of the sensors ADU ranges possible (a fixed, large offset would also remove zero values from the sensor, but would limit the amount of the ADU range we could use).

How do we set an offset control? The simplest guideline is to take dark frame with all the settings that you intend to use (i.e. correct exposure, temperature, gain et cetera) and adjust the offset to just separate the histogram of that dark frame from the left-hand side. If you don't intend to use dark frames then you could perform the same trick with a light frame (this will likely give you a smaller offset value as the sky background brightness will also tend to shift the histogram peak to the right and separated from the left-hand side of the graph).

The question at the start of this thread was whether the correct offset changes with the gain of the camera and why that should happen. The answer to this is – I think – it depends on the model and make of camera. As we know the width of the histogram peak increases as the gain is increased because gain is effectively just a multiplication factor that is boosting a weak signal into a stronger signal and at the same time changing a narrow histogram peak into a wider one. Inside the actual camera itself both the gain and the offset have to be applied before the analogue to digital conversion stage – the gain is basically a voltage multiplier that is applied to the signal coming from each pixel to make a stronger voltage signal to be fed into the ADC, while the offset is essentially a small extra voltage being added to every pixel to increase the pixel value.

Some camera brands/models seem to auto scale the offset value when the gain is adjusted without requiring adjustments of the actual offset value i.e. if you turn up the gain you will see the histogram peak not only widen but move to the right automatically in order to keep it separated from the left-hand side of the histogram graph. On other types of cameras when you turn up the gain the histogram peak stays in the same place and just gets wider and with such cameras there is a definite requirement to increase the offset value as the gain is increased to ensure separation.

Hope that all makes sense – if there are still unanswered questions lurking in the middle of this thread then please let me know and I'll do my best to answer them if I know the answer.

Cheers, Robin

[turfpit]

Thanks for that Robin - you did far more than you should have given your current medical condition.

Steve - in File > SharpCap Settings > General Tab, what is the state of Save 10/12/14 bit images in FITS files without stretching to 16-bit ?

If this is checked, then it needs unchecking for normal imaging use.

This will keep until you emerge from the fog of turkey and mince pies

Dave

[SteveJP]

Hi Robin,
I'm still working though your very comprehensive answer. I'll reply properly soon.
Steve'

[SteveJP]

Thanks for that Robin - you did far more than you should have given your current medical condition.

Steve - in File > SharpCap Settings > General Tab, what is the state of Save 10/12/14 bit images in FITS files without stretching to 16-bit ?

If this is checked, then it needs unchecking for normal imaging use.

This will keep until you emerge from the fog of turkey and mince pies

Dave

Hi Dave,

Well, as it happens, I do have that option checked since I thought it would be best to get a faithful representation of what the camera sees into my post processing. The effect is surely just a multiplication by 4? The non-stretched image is surely darker, but I think this just means the stretch needs to be harder which is probably just the same as the multiplication anyway. Working in PixInsight as I do, even in the linear processing stage, a default stretch is applied. If I import the 14 bit and 16 bit versions of the same thing into Pixinsight, the 14 bit image is dimmer of course, but as soon as the default stretch is applied, I can't tell them apart. Just wondering why you think it's important?

I note too, the histogram produced by SharpCap (which my Excel graphs come from) is independent of this setting. Ie for my 14 bit camera, the lines in the histogram go from 0 to 2^14 -1 regardless. So when I plot the histogram when SharpCap is set to not stretch to 16 bits, the graph I get from the SharpCap histogram agrees with FITS Liberator, but when I uncheck the option, FITS Liberator shows a multiplication by 4 whereas the SharpCap histogram does not.

I suspect this could be regarded as a bug, since the when the option is unchecked, the SharpCap histogram doesn't match the actual output file.

So much to consider !

Rgds
Steve

[SteveJP]

Thanks Robin,

That's a great explanation - I think I really get it now. Being a bit of a geek, I really feel I need to understand the underlying mechanism.
To that end, I've been running a few experiments here.

These are all graphs made from the histogram SharpCap outputs. All are 1ms darks.

The first image shows the effect of just changing the offset (aka brightness). The Gaussian distributions are clearly centred approximately around the offset setting.

The second image shows the effect of gain when the offset is constant. As you explained, the higher gain causes the distribution to spread out. In each case, the area under the curves will be the same, corresponding to my 16,000 pixels.

The third image shows the effect of even greater gain with constant offset. In this case the offset is now clearly insufficient and demonstrates why the offset should be increased with gain.

I suppose you could just set an offset big enough for all circumstances? I'll have to explore the downside of that.

Many thanks
Steve

1ms_G90_Off40+80.PNG (42.87 KiB) Viewed 6423 times

1ms_G90+150_Off40.PNG (39.59 KiB) Viewed 6423 times

1ms_G90+250_Off40.PNG (39.32 KiB) Viewed 6423 times

[SteveJP]

This is the correct third image

1ms_G90+250_Off40X.PNG (39.16 KiB) Viewed 6421 times

[turfpit]

Steve

Well, as it happens, I do have that option checked since

It needs to be unchecked unless you are doing astrometry. Have a look at this thread
viewtopic.php?t=1108 where I describe the use of the batch processing facility in ImageJ to resolve (thanks to @oopfan for that one - cheers Brian).


Robin - this setting again! A suggestion/request - a lot of time spent problem resolving this one on the forum. Would it be possible to have a means to dump all the Sharpcap Settings into a text file? This can be submitted with along with the forum request and we don't have to waste time working blind trying to solve issues.

Dave

[admin]

Hi folks,

the downside to selecting a constant high offset regardless of your gain value is that you are then limiting yourself to a smaller dynamic range in each image you take. For instance if you have a 12 bit ADC then the maximum ADU value is 4095. If you set an offset that puts the black level at a true ADU of perhaps 500 then you only have 3500 or so ADU available between the black point and saturation. If you can get away with setting a lower offset so that the black point is an ADU of maybe 200 then you have gained that little bit of extra dynamic range in each frame that you take.

Dave, all the SharpCap settings are in the registry and can probably be dumped out with the appropriate command prompt Reg.exe call.

Cheers, Robin

[turfpit]

Folks

the downside to selecting a constant high offset regardless of your gain value is that you are then limiting yourself to a smaller dynamic range in each image you take

For clarification, I haven't advocated a constant high offset just a systematic way to find the correct value to give a balanced histogram on a bias frame. See my post above showing CMOS, DSLR and CCD - two of them do it for free, one type has to have user intervention to achieve the same result.

Robin

all the SharpCap settings are in the registry and can probably be dumped out with the appropriate command prompt Reg.exe call

"probably" is not a solution. Dumping out the registry with reg.exe is ok for people like me and you, for the majority of people on the forum this is not a practical thing. Maybe program in an option in SharpCap to dump out the reg settings to a file which can then be posted?

What I am conscious of the is the large amount of time being put in to debugging issues, when, if the information was easily to hand the resolution would be faster and less time consuming. To keep recycling the same stuff again and again is not time productive for anyone.

Dave

[SteveJP]

Steve

Well, as it happens, I do have that option checked since

It needs to be unchecked unless you are doing astrometry. Have a look at this thread
viewtopic.php?t=1108 where I describe the use of the batch processing facility in ImageJ to resolve (thanks to @oopfan for that one - cheers Brian).


Robin - this setting again! A suggestion/request - a lot of time spent problem resolving this one on the forum. Would it be possible to have a means to dump all the Sharpcap Settings into a text file? This can be submitted with along with the forum request and we don't have to waste time working blind trying to solve issues.

Dave

Hi Dave/Robin,
I did read through that thread, or at least most of it. It seems folks using DSS may have trouble stacking since DSS expects to find stars with higher ADU's. I've had that option ticked for ages and stacking with PixInsight that doesn't seem worried about the values being a 1/4 of what they might otherwise be?
In regards to setting a constant offset, I realise the effect on dynamic range, but if I have a range of 16383 (2^14-1), does it really matter if I need to set an offset of say 50 to cover all circumstances? All depends on what offset you would need and the camera dynamic range.
Steve