Additive dark/flat application [SOLVED]

A place to report problems and bugs in SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs

Please also read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
gaar
Posts: 12
Joined: Sat Jul 04, 2020 8:42 pm

Additive dark/flat application [SOLVED]

Post by gaar »

After I use the Smart Histogram, dark and flat frames seem to apply additively; that is, selecting a new dark or flat acts as if the old AND the new frame are applied. I'm not 100% the Smart Histogram step is required, but that's my normal workflow.

SharpCap 3.2.6882 (and several previous versions) 32 bit
ZWO ASI294MC Pro

To reproduce:
* Capture a flat with the following settings:
- 4 sec exposure
- 121 gain
- appropriate dark selected
- confirm histogram OK
- bias grayed out as expected (because dark is selected; I understand bias doesn't work properly with the ASI294MC)
- monochrome
- average 20 frames
- apply flat when generated
* After this, dark and flat are selected
* Use the Smart Histogram
- target 5% read noise, max dynamic range
- measure
- after measuring, either click Apply or don't, it doesn't seem to matter
* After this, dark and flat selections are cleared
- (I can understand this for the dark, since the measurement change the exposure settings, but why is the flat cleared?)
* Set recommended exposure and gain
- (black level doesn't need changing)
* Set appropriate dark, and the flat generated earlier
* Live stack
- image shows faint vignetting (bright center), as if I hadn't applied a flat
* Reselect the same flat
- stacked image shows reverse vignetting, with bright edges
* Reselect same (or any) dark
- image shows hot pixels and intense color noise

To recover, I have to exit and restart SharpCap. Then I can select the same dark and flat, and they are applied perfectly, with no hot pixels or vignetting.

The attached doesn't precisely show the above sequence because I had generated the flat in a previous session. But it does capture the smart histogram and affected live stacking attempts. I'll try to get a cleaner log tonight.
Attachments
Log_2020-10-02T21_13_07-7872.log
(46.07 KiB) Downloaded 37 times
Last edited by gaar on Sat May 15, 2021 4:23 pm, edited 1 time in total.
User avatar
Menno555
Posts: 420
Joined: Mon Apr 20, 2020 2:19 pm
Location: The Netherlands
Contact:

Re: Additive dark/flat application

Post by Menno555 »

You first use the Histogram Brain. Then you use those settings and only then you create your flats and darks and apply them :)

Menno
gaar
Posts: 12
Joined: Sat Jul 04, 2020 8:42 pm

Re: Additive dark/flat application

Post by gaar »

I don't understand why or how the Smart Histogram should come first. In most situations (and definitely in mine), flat frame capture isn't possible with the recommended settings. The docs back this up: "There is no need to use the same exposure or gain for flat frames (in fact this would not normally work)."

Even if "Smart Histogram, then dark & flat" is a best practice, the behavior I see is still be a bug. Choosing a given dark or flat should always result in the same preprocessing regardless of what was selected previously.

Still, I'm game. I'll try that order tonight and see if anything changes.
User avatar
Menno555
Posts: 420
Joined: Mon Apr 20, 2020 2:19 pm
Location: The Netherlands
Contact:

Re: Additive dark/flat application

Post by Menno555 »

You have a point with the flats. Those are indeed independent on the exposure settings. But a setting like offset/brightness certainly is not. If the Brain suggests an Offset/Brightness setting of 4, your flats should have the same value.
But your darks are fully dependent on your initial settings. So, you use the Brain and that's giving an advice. You use those settings as initial and based upon those settings, you shoot your darks and those are then correctly used.
User avatar
admin
Site Admin
Posts: 6646
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Additive dark/flat application

Post by admin »

Hi,

that's certainly unexpected – the code for applying the dark and flat correction *should* have no memory of the selections that you've made previously. Live stack of course does tend to remember what's been going on previously, so I can imagine things like that appearing if you've previously had live stack active and haven't cleared. Is it possible to see the same effects while taking the live stack step out of the flow? (Perhaps replacing it with using the mini histogram stretch to increase the contrast enough to see any vignetting/reverse vignetting effects). If we can either prove that it requires live stack or prove that it happens without live stack it would greatly reduce the possible number of places that we would need to look to find the cause of the issue.

With respect to the measurement stage – both dark and flat correction are turned off while taking the measurement because the goal is to measure the true e/pixel/s value for the darkest part of the image being captured – having either of those corrections active would compromise that measurement. Having dark correction active would remove the contribution of thermal noise and also introduce confusion about the offset effect. Having flat correction active would tend to enhance the apparent rates in areas of the frame where the flat correction is boosting the brightness.

Cheers, Robin
gaar
Posts: 12
Joined: Sat Jul 04, 2020 8:42 pm

Re: Additive dark/flat application

Post by gaar »

I set the hobby aside for the winter, but I got back into it over the past week. I'm still seeing additive dark/flat application, now with SharpCap 3.2.6482.0. I didn't think to test without live stacking, but I did see a new behavior that I hadn't observed last fall. With live stacking underway, the histogram was accumulating overlapping and very different curves. With each new frame, a new RGB curve was rendered on top of the old curve, but with the peaks much lower than the previous frame. I regret that I didn't get a screenshot, but I will next time.

I tried resetting both dark and flat frames to None, and also reconnecting the camera, but the bizarre behavior continued. I had to restart SharpCap to stop the weird dark/flat application.

Next time I'm out, I plan to try a few things:

* See what happens without live stacking after using the Brain.
* Set dark and flat to None before using the Brain.
* Screencap the overlapping histogram curves, if I see them again.
User avatar
admin
Site Admin
Posts: 6646
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Additive dark/flat application

Post by admin »

Hi,

thanks for continuing to look into this - a screenshot (or even a screen capture video) showing the problem would certainly help .

cheers,

Robin
gaar
Posts: 12
Joined: Sat Jul 04, 2020 8:42 pm

Re: Additive dark/flat application

Post by gaar »

I have attached screenshots taken with SharpCap 3.2.6482. The screenshots show two examples of the problem. Each example was taken from an autostretched Live Stack view.

A1-reverse-vingette.png: Prior to starting this live stack, I used the Smart Histogram's Apply button and manually tweaked settings, including re-selecting Dark and Flat frames.
A2-no-reverse-vingette-same-curve.png: Same view and settings, but after restarting SharpCap. The Live Stack's histogram is still set from the previous session. Notice that the peak point in the histogram is much lower than the A1 shot.
A3-no-reverse-vingette-custom-curve.png: Same view and settings, but re-ran the histogram autostretch to confirm lack of reverse vingetting.

B1-reverse-vingette-2.png: Another example of the problem manifesting. This is from the same SharpCap session as shots A2 and A3, but after running the Smart Histogram. This time I did not click the Apply button, but manually applied settings myself.
B2-no-reverse-vingette-custom-curve-2.png: Same view and settings as B1 shot, but after restarting SharpCap and refreshing the histogram autostretch. Notice that the histogram is again much lower than in the B1 shot.

As always, I'm happy to collect any additional information, although it may take me awhile.
Attachments
B2-no-reverse-vingette-custom-curve-2.png
B2-no-reverse-vingette-custom-curve-2.png (548.57 KiB) Viewed 186 times
B1-reverse-vingette-2.png
B1-reverse-vingette-2.png (489.28 KiB) Viewed 186 times
A3-no-reverse-vingette-custom-curve.png
A3-no-reverse-vingette-custom-curve.png (476.81 KiB) Viewed 186 times
A2-no-reverse-vingette-same-curve.png
A2-no-reverse-vingette-same-curve.png (174.39 KiB) Viewed 186 times
A1-reverse-vingette.png
A1-reverse-vingette.png (507.25 KiB) Viewed 186 times
gaar
Posts: 12
Joined: Sat Jul 04, 2020 8:42 pm

Re: Additive dark/flat application

Post by gaar »

Let's just pretend that I spelled "vignette" correctly throughout my last post, shall we?
User avatar
admin
Site Admin
Posts: 6646
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Additive dark/flat application

Post by admin »

Hi,

thanks for the screenshots!

I can see that for each of the screenshots that have a strong stretch in the vicinity of the histogram peaks (B1 and A1) there is reverse vignetting visible. For the other screenshots the stretch is either weaker (B2, A3) or not positioned over the histogram peaks (A2). I can also see that you are always imaging at 4s exposure, but I cannot see whether the gain in use is always the same between the different scenarios.

My suspicion is that the potential for the reverse vignette is always there, it is just only made visible when the stretch is strong enough and in the right place relative to the histogram peaks. For instance, A2 looks dark because the black/mid levels are well to the right of the histogram peaks - if you moved the black/mid levels left then maybe you would have seen the reverse vignetting.

Ok, so now onto why reverse vignetting? I will first assume that it will go away if you remove the flat frame - that seems to be pretty much universal. Reverse vignetting is basically flat frame over-correction.

What causes flat frame over-correction?

1) Capturing flat frames with a dark that is unsuitable (ie longer exposure/higher gain than the flat frame or a higher offset/brightness)

or

2) Capturing light frames without a dark or with a dark that is unsuitable (shorter exposure/lower gain than the light frame or a lower offset/brightness).

Note that however perfect you try to make your flat correction, if you stretch far enough you will still see either some vignetting or some reverse vignetting.

Hope this makes sense,

Robin
Post Reply