Bug with dark frame + black level adjustment

A place to report problems and bugs in SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs

Please also read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Bug with dark frame + black level adjustment

Post by jrschmidt2 »

I believe I have found an apparent bug with the use of subtract dark frames in conjunction with the Live Stacking black level adjustment. In particular, this appears to happen when the dark frame is at the same offset/brightness as the light (as typically done), at least with dim light frames (e.g. low gain, short exposures).

Specifically, when apply the dark during live stack, MOVING the dark level off the default (zero position) yields very strange results, effectively clipping off most of the image. Interestingly, moving it BACK as far left as can be accomplished with the mouse (i.e. presumably that same zero) does NOT revert to the initial behavior -- the image remains strongly clipped. The "initial" image (before attempting to move the dark level) appears to yield a result consistent with what I get from other stacking programs (e.g. ASTAP) with the same dark substracted, in contrast to what I get after adjusting. Note that I am testing this with the virtual Folder Monitor camera for simplicity, after discovering the issue last night.

Note also that if I increase the offset of the lights a big (e.g. from 4->6) but keep the offset of the dark frame at 4, the issues does NOT appear (i.e. it is not clipped upon subtraction).
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Re: Bug with dark frame + black level adjustment

Post by jrschmidt2 »

An addendum to my prior post:

1) What ACTUALLY appears to be happening is that in the circumstances described below, SharpCap is making the initial "black" level to be NEGATIVE (after subtraction of darks). This can be seen by the shape of the "stretch" curve, which does not actually go to zero intensity at zero on the x-axis. You can get back to this initial behavior via "auto stretch" as well, although you cannot manually move the black level "negative".
2) Although the auto-stretch works OK in this case, you cannot adjust the black level to be e.g. a bit LESS negative, because as soon as you attempt to move it with the mouse it goes to zero (rather than a slightly smaller negative value).
3) This appears both with SharpCap 3.2 and the newest beta 4.
User avatar
admin
Site Admin
Posts: 6954
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Bug with dark frame + black level adjustment

Post by admin »

Hi,

can you share a dark frame and a light frame so that I can dig into this a bit further. Ideally if you have the capture settings files for both so I can see what settings were used.

SharpCap does track negative values in the stack, but normally this is not really relevant as even a small amount of background light pollution will comfortably lift the level of the light frames so that practically no negative values are created after dark subtraction. Do you have very dark skies (or are you using a narrowband filter?) those could lead to minimal background light pollution. Alternatively it's possible that some subtle camera setting difference between darks and lights could have elevated the brightness level of the dark frames. If it's neither of those then I may need to think about better handling of -ve values in the stretch adjustment.

cheers,

Robin
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Re: Bug with dark frame + black level adjustment

Post by jrschmidt2 »

Sure I'll post them both tonight. They should match since I saved the settings and then loaded/used those same settings in dark generation. I also check the FITS files myself and they appear to match (e.g. offset is OK). I think, as you say, the problem is that with low gain (121, ZWO 294mc) and short exposure (10s) + Optlong L-Pro the background is fairly dark. That said, these are the settings recommended by the "Brain" for max dynamic range and they do produce nice images (when stacked outside Sharpcap!).
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Re: Bug with dark frame + black level adjustment

Post by jrschmidt2 »

Here is a link to the light and dark frame FITS:

https://www.dropbox.com/sh/o2nbrdie0mdd ... 7aT1a?dl=0

The capture info for the flat (saved by SharpCap) is included as well. I couldn't find a corresponding information file for the lights, but both were generated from the 'Astrophotography' profile I made and saved (gain 121, 10s exposure, offset of 4, 50/50 white balance).
User avatar
admin
Site Admin
Posts: 6954
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Bug with dark frame + black level adjustment

Post by admin »

Hi,

thanks for sharing those frames - I have had a look at them and the dark is definitely a lower average brightness than the light frame, however both images have a certain amount of horizontal banding which is visible when you stretch them strongly. Where the bands interact in the wrong way (bright part in the dark frame, darker part in the light), there are some pixels where the dark frame is brighter than the light frame.

I think the best approach here is to try to reduce the banding in the dark frame (the banding in the light frame will vary from frame to frame, so as more frames get stacked it will tend to have a smaller and smaller effect as it averages out).

Reducing the banding in the dark can be achieved by averaging more dark frames to make the master dark or by using the banding suppresion function of SharpCap (https://docs.sharpcap.co.uk/4.0/#!2!Pre-processing).

I think that this approach should be sufficient to eliminate the problem with negative values in the stack (or at least reduce it to a level where it is hardly noticeable).

thanks,

Robin
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Re: Bug with dark frame + black level adjustment

Post by jrschmidt2 »

I don't think this is the root cause (although the banding is real -- a known issue with 294mc!). For example, I made a new dark with 200 frames. This is sufficient to essentially totally eliminate the banding in the dark. However, applying this new dark in the SharpCap live stack exhibits the same problem. (Again, it works fine in other stacking programs.)

The dark-subtracted image is FAR too clipped. For example, in this case (M51), after subtraction everything but the core of the galaxy is totally eliminated. I did some empirical testing and found that the "black" level (that is, the new zero of the histogram) after dark subtracting in SharpCap is equivalent to 300 adu in the original image. In comparison, the "offset" level in the dark (or equivalently, the peak in the dark histogram) is about 240 adu. However, the tail of the dark histogram goes to about 300. It is like SharpCap is subtracting off (roughly) the LARGEST value of the dark rather than the smallest.

I'm sure none of this would matter if the exposures where longer or the gain higher. In this case all the histograms would be far broader and the clipping negligible. Still, it would be nice to figure out what is going on here.
User avatar
admin
Site Admin
Posts: 6954
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Bug with dark frame + black level adjustment

Post by admin »

Hi,

I honestly don't see the clipping you describe when I test with the frames that you shared.

Here's what I see using the light frame and dark frame in the test camera (set to mono16).
Capture.JPG
Capture.JPG (133.03 KiB) Viewed 213 times
While there are a few pixels clipped to zero value, they are very much in the minority. The same happens if I run a live stack using those two frames as light + dark.

There is something more to this I expect that is causing the issue for you that we haven't worked out yet - right now though I am struggling to think of what it might be :(

cheers,

Robin
jrschmidt2
Posts: 10
Joined: Wed Oct 07, 2020 8:54 pm

Re: Bug with dark frame + black level adjustment

Post by jrschmidt2 »

A screen shot is worth a 1000 words. Here is what I get if I stack (using the dark) WITHOUT adusting the black level at all, or alternatively autostretching and tweak the midlevel (but leave the black):
Before adjusting black level
Before adjusting black level
autostretch_small.png (218.98 KiB) Viewed 169 times
As you can see, it is basically identical to what you see. So far so good!

But if I adjust the black level (in the live stack histogram) at ALL -- even if I then move it back all the way to the left -- I end up with this:
After histogram adjustment
After histogram adjustment
adjusted_small.png (80.78 KiB) Viewed 169 times
This makes it impossible to make any small adjustments to the black level. Either you take the autostretch result, or a massively clipped result, with no possible adjustment that can be made in between.

If you are unable to reproduce this then I guess I am stumped and confused as well! :D
User avatar
admin
Site Admin
Posts: 6954
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Bug with dark frame + black level adjustment

Post by admin »

Hi,

you were right, a picture does mean more than 1000 words...

It is something to do with using the 'Sigma Clipped' stacking option - when I turn that on I see the same as you, when I set the stacking to 'Default', everything is fine.

I will try to work out what is going on with the sigma clipped option and fix it if possible - unless you really need to deal with satellite trails, I'd switch to 'Default' stacking for now.

cheers,

Robin
Post Reply