Flats nightmare

Somewhere to ask questions about the best way to use SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Flats nightmare

#1

Post by clouzot »

Hi Robin, all,

I've been struggling for months with this issue, so after thorough testing I guess I'm ready to ask.

I'm currently using an SCT (C9) with a focal reducer and a ZWO ASI294MM Pro (all that follows also happens with my Altair 294c). An SCT with a focal reducer really has vignetting, so it may be uncovering an issue that also happens with optics that have a better field illumination.

I'm usually taking flats with Sharpcap, using a dimmable Lacerta 12V flatbox, and as the livestack goes I'm often seeing either undercorrection (residual vignetting) or, worse, overcorrection (inverse vignetting).

Here are some tests I ran lately:
- flatbox at maximum brightness
- 294MM set to unity gain (HCG = 120)
- set the exposure to get a decent histogram with no saturated pixels (either black or white) and ended up with 6ms or so
- captured various flats by either (a) leaving the "bias" unchecked, (b) checking the "bias" box, (c) separately capturing bias at the lowest possible exposure (around 40us), stacking them and using the obtained "master bias" as darks during the flat capture (effectively subtracting this master bias from each unitary flats).

I then applied those three different flats in order to see what would happen
Flat captured without bias
flatsansbias_short.jpg
flatsansbias_short.jpg (653.23 KiB) Viewed 3051 times
Flat captured with Sharpcap's "capture bias" feature
flatSCbias_short.jpg
flatSCbias_short.jpg (533.63 KiB) Viewed 3051 times
Flat captured by subtracting a "true" master bias
flattruebias_short.jpg
flattruebias_short.jpg (663.94 KiB) Viewed 3051 times
The bias-subtracted flat is overcorrecting, the two other seem OK at this point.

I then lowered the flatbox brightness in order to get a non saturated histogram for 4-second exposures, still at unity gain, more in line with I do during EAA sessions, and tested the above flats

Flat captured without bias
flatsansbias_longexpo.jpg
flatsansbias_longexpo.jpg (514.76 KiB) Viewed 3051 times
Flat captured with Sharpcap's "capture bias" feature
flatsSCbias_longexpo.jpg
flatsSCbias_longexpo.jpg (682.71 KiB) Viewed 3051 times
Flat captured by subtracting a "true" master bias
flatstruebias_longexpo.jpg
flatstruebias_longexpo.jpg (717.43 KiB) Viewed 3051 times
Now we have another issue!
-> The no-bias flat still undercompensates for vignetting.
-> The SC-captured "flat bias" undercalibrates slightly
-> The "master bias" flat seems to be compensating for vignetting, but both bias-corrected flats have this odd circular pattern.
-> The flat captured with Sharpcap's "capture bias" also has a rectangular zone at the top of the frame that is darker than the rest.

While I was there, I also tested "flat darks" by matching 4s darks with 4s flats. Same issue with those strange patterns.

Do you guys see where the issue could come from?
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Flats nightmare

#2

Post by admin »

Hi,

going right back to the beginning, the 294 sensor can have a bit of an issue when its place near the bottom of the HCG gain range. Basically, in certain settings configurations it can fail to fully saturate even when vastly overexposed. This is tripped up a lot of people capturing flat frames because it looks like they have good exposure with no pixels that are saturated, but in fact the camera is misleading them.

The way to check for this is to go back to your typical gain settings for capturing flats and gradually turn up the exposure to over saturate the frame. If you can get to a point where basically every pixel is at hundred percent brightness then you are okay. If find that increasing the exposure no longer affects the histogram and there are still pixels that are not saturated then you have the problem.

Here's an example histogram from an affected camera (with the gain set just above the LCG/HCG switch over point). This camera was taking 20s exposures, which should be fully saturated, but aren't. Note that I've seen a lot worse examples than this one with the peaks at about 70-80% and looking like wonderfully exposed flat frames when in fact overexposed badly.
Capture.JPG
Capture.JPG (55.99 KiB) Viewed 3039 times
If this problem is affecting you then the solution is to use a different gain value for your flat frame capture – either go down in gain to below the LCG/HCG switchover point (119) or go up in gain to about 180, by which point the problem should have disappeared.

cheers,

Robin
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#3

Post by clouzot »

Thanks Robin, I wasn't aware of that particular issue and I can confirm this is indeed the case with both my Altair 294c and my 294MM Pro (IMX492, but that's essentially the same sensor as far as I understand)
Gain 110
294MM_gain110_4s.jpg
294MM_gain110_4s.jpg (264.26 KiB) Viewed 3029 times
Gain 120 (HGC switch)
294MM_gain120_4s.jpg
294MM_gain120_4s.jpg (273.1 KiB) Viewed 3029 times
This bad behavior indeed disappears below gain 120 and above gain 135 with my 294MM. I'll try to capture flats (along with bias-calibrated flats and dark-calibrated flats) and will report here. Thanks again for your support!
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#4

Post by clouzot »

@Robin: while you're here, could you tell how does this odd HCG-unity-gain behavior would affect the capture of flats?
Does that mean that at (and just above) this gain, all reported pixel values shouldn't be trusted because of non-linearities, or does it only apply to a range of pixel values close to saturation?

I'm asking because if this gain yields inconsistent-non-linear pixel outputs across a wide range of brightness, then it should be avoided, be it for flats or lights frames.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Flats nightmare

#5

Post by admin »

Hi,

my opinion with these cameras is that HCG is dubious below 9.0x gain - that's 190 gain in ZWO units or 900 for Altair cameras. I've seen the data sheet for the 294 and it recommends not using HCG below that point (which comes from 4.5x gain from switching to HCG and 2.0x from normal gain). ZWO seem to have minimized the issue somewhat on the colour camera, but not so well on the mono.

If you image in this region then any values near the top of the brightness range will be non-linear - basically massively overexpose and then see where the histogram ends up - if you are seeing histogram at less than 100% when overexposed then that region is non-linear.

I just tweaked SharpCap to put the Altair 115M (294 mono) camera into HCG at gain 450 and then turned up the exposure. The histogram looks great for capturing a flat, but everything is actually washed out by the overexposure. Time to undo that change now - I don't want it getting into a release!

Robin
Capture.JPG
Capture.JPG (146.04 KiB) Viewed 3000 times
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#6

Post by clouzot »

It indeeds looks like a perfectly kosher histogram for taking flats...until you see the actual captured image.

I think I now understand why Altair set the unity gain at 900 for their 294c models...well above the LCG/HGC switch point (450). ZWO didn't do this.

So that essentially means that for all IMX492/294-based models, one shouldn't use anything lower than these values (190 for ZWOs, 900 for Altairs). Interesting! Time to redo all my dark library methinks. I'll test the (dark) flats captured around this gain as well.

Thanks again for your support Robin, much appreciated.
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#7

Post by clouzot »

Robin, last question if you don't mind: as you know those IMX294/492-based cameras pretty well, what kind of settings should one use for taking good flats?

1) I understand that the 120..190 gain range should be avoided at all costs so as to avoid HCG non-linearities.
2) There were also various technical threads advising not to use short exposure under 1-2 seconds (due to other oddities in the sensor behavior, perhaps unjustified)
3) while LGC gains are quite noisy due to substantial read noise levels below the LGC/HGC switch point

That essentially leaves us with gain>190 along with 2s+ exposures. Which, in turn, means
- flat darks (not bias-corrected flats, due to the already long exposure)
- lots of t-shirt layers in front of an already dimmed flatfield box in order to ensure a good histogram distribution.

Does the above look right to you?
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#8

Post by clouzot »

I tested those settings (dark-calibrated flats, 2s exposure, gain 190, same offset as I usually set for my lights).

I can report the so obtained flats calibrate out vignetting almost perfectly when I dim my flatbox to the typical brightness of my city skies and stack longer subs at unity gain.

So far, so good.
flats_again.png
flats_again.png (398.21 KiB) Viewed 2993 times
Now, I'd love to see if capturing short flats would be possible, because the very low brightness is quite impractical (quite a number of t-shirt layers were needed to achieve the proper brightness).
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Flats nightmare

#9

Post by admin »

Hi,

I'm unaware of any problems below 2s exposure (other than some cameras not doing exact exposures in the 1s to 2s range - ie you ask for 1.27s and get 1.4s). That's not really important for flats though. Also using LCG doesn't matter - for flats you are exposing the whole image at ~60-70% brightness, so you get 40,000 to 50,000 e per pixel. The stochastic (shot) noise on that is 200e, so the read noise of 7e really is tiny in comparison - not worth worrying about.

I would take flats at minimum gain, <1s exposure and see what happens.

cheers,

Robin
clouzot
Posts: 46
Joined: Thu Aug 08, 2019 11:13 am

Re: Flats nightmare

#10

Post by clouzot »

Thanks Robin,

I tried exactly what you advised (gain 0, 500ms flats, bias and non-bias, histogram averaging around 65% of the dynamic range, and of course no saturating pixels).

Applied to 15s exposures, the bias-corrected flats were overcorrecting for some reason, but I was glad to see the raw flats calibrate my lights beautifully.

So again I'd like to publicly thank you, because my original issue seems to be solved. The non-linearity of both my sensors (IMX294 and 492) above the HCG was likely the culprit.
Post Reply