Live Stack with Flat Frame Bug?

A place to report problems and bugs in SharpCap
Forum rules


If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs

Please also read about Troubleshooting USB Issues before posting.

*** Please do not post license keys - please report any problems with licensing to 'admin' by private message ***

Please include the following details in any bug report:

* Version of SharpCap
* Camera and other hardware being user
* Operating system version
* Contents of the SharpCap log after the problem has occurred.
[If SharpCap crashes, please send the bug report when prompted instead of including the log]
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#21

Post by admin »

Thanks for the further update and your experimental results. I can't think of much else to suggest at the moment but I am going to have a test myself when I get some more time to see if I can reproduce the same results in a proper Astro imaging workflow.

Cheers, Robin
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#22

Post by admin »

Hi,

been thinking through this again and trying to work out where the most likely places for any sort of problem would be. Unfortunately still cloudy here so no chance to test anything, but this would be my workflow.

1. Capturing flats. When capturing your flat frames you should either have no dark frame subtracted and use the subtract bias checkbox in the capture flat window, or you should be subtracting a dark frame that has been taken at the same exposure, gain and offset settings as will be used to capture the flats. Either of these approaches should generate flat frames that are properly scaled with no zero point offset. You should avoid capturing flat frames with a dark frame subtracted that was captured at different gain exposure or offset settings. In particular a dark frame that has been prepared for your actual imaging exposure set up would be completely inappropriate.

2. Capturing light frames. You should use dark subtraction in SharpCap while running live stacking with flats to ensure best results. Using dark subtraction will ensure that your images have no black offset, meaning that the flat correction will operate accurately. If you do not use dark subtraction then all the image pixel values in the frame offset upwards by whatever the offset level of the camera is and the flat correction multiplier is applied not only to the image brightness but also to this offset level, leading to somewhat incorrect flat correction.

You can see that the main point of both of these suggestions is to make sure that you are taking the appropriate steps to deal with the offset level of the camera both when taking the flat frames themselves and when using them. Needless to say it's important not to change the offset/brightness/black level setting during your capture workflow, as changing this setting between for instance capturing dark and capturing lights will reintroduce and offset.

I expect that you knew all of this anyway, but there is a chance that maybe something it will be helpful to you.

Cheers, Robin
BlackWikkett
Posts: 387
Joined: Sat Jun 23, 2018 8:48 pm
Contact:

Re: Live Stack with Flat Frame Bug?

#23

Post by BlackWikkett »

Hi Robin,

I think I follow what you're saying and I believe this is in deed been the case for the testing I've been doing. Here's been the methodology as of the latest testing that has produce the best results.

- Lights, Darks exposure, gain and offset the same
- flats gain the same, offset at zero, and exposure set to get max histogram value without saturation. No bias and color not mono. If MONO is selected we get back to the bad vignetting.

Using the ASI294 I and Don have found lower offset to give better results. Don has been using 8 offset but he's using different filter combo. I'm using the IDAS D1 and found 10 offset works well. If you have a look at my latest gallery submission viewtopic.php?f=16&t=1324 you'll see there's no vignetting evident in the images. These images are all Live Stack only using SharpCap histogram stretching with some images using SC noise reduction and SC sharpening.

The master flat created with the SC tool used 209ms exposure, 0 offset, 300 gain. (LED light panel and T-shirt) This was used for the successful tests and the images in the gallery thread linked above.

It looks like we'll have clear skies this Saturday 12-22-18. I'll be broadcasting on NSN https://www.nightskiesnetwork.com/ starting about 7:00 PM Eastern Time (weather permitting.) That's +5 GMT think midnight your time. If you're able to join us I can complete any testing you'd like. If this is possible you'll want to sign up for free account so you'll be able to communicate. Guest can tune into broadcast but don't have access voice or text.

Thanks for keeping this one on the radar and all you do! You have a lot of SC fans on NSN
-Wikkett
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#24

Post by admin »

Hi,

what histogram levels you get the peak at if you take the shortest possible dark frames when using your gain and offset values that you use for the flat capture? If that's above zero then the fact that you are not using bias frames would tend to indicate a problem with your flat frame correction due to the offset not being corrected in the flat frames.

Let me give you an example, if the vignetting level to the corner is 50% then you might expect an ADU of 50,000 in the centre of the frame and 25,000 in the corner which would mean that the flat frame correction would double the brightness of the corners to correct. However if there is a lack level ADU of say 500 then the 2 readings would be 50,000 and 25,500, which would give a correction factor of about 1.98 instead of a correction factor of 2. I.e. failing to subtract bias frame from the flats when capturing those would tend to lease the flats that under correct any problems.

The opposite tends to happen if you don't subtract dark frame from your lights – if you don't do that then applying a flat will tend over correct since it will also be applied to the bias level as well as the actual ADU value this is attributable to the image.

Although the difference between the correction ratios of 2.00 and 1.98 doesn't sound like much, it can be significant because we tend to stretch Astro images very significantly near the black point to bring out faint nebulous it in other features.

Cheers, Robin
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#25

Post by admin »

Hi,

further update… Although the weather hasn't incorporated to allow proper testing on Astro targets, I have been able to do some extensive testing using an electroluminescent panel and a 294C camera with a 50 mm lens that the vignettes quite significantly. My testing methodology was to capture a flat frame and then to apply that flat frame to exactly the same image with the aim of getting a perfectly even illumination showing in SharpCap once the flat frame was applied. To test this I was using the auto histogram stretch functionality to enhance any unevenness in the corrected image.

I found I got good results as long as I applied a dark frame (or use the bias option) when capturing the flat and similarly used a dark frame when capturing the final image along with the flat frame. Not using the dark/bias frame in either of those 2 steps does lead to significant problems with the correction once you stretch the resulting image.

I also discovered something of a gremlin in the way that SharpCap currently handles flat frame correction, particularly in the presence of saturated or nearly saturated pixels. Currently SharpCap uses the average pixel value of the flat frame as a reference level to aim for when applying the flat frame correction. That means that if the average value of the flat frame is 50,000 ADU, then SharpCap will brighten areas where the flat frame values less than 50,000 and reduce the brightness of areas where the flat frame values above 50,000. This approach seem to make sense to me at the time that I implemented flat frame correction, but in testing I found that the fact that it can reduce the brightness in some regions causes problems as it will drop the brightness of saturated pixels in areas where the flat frame is brightest. This can lead to issues like dimmer stars or colour casts in the centre of the frame.

I believe the correct approach here is to use the maximum value of the flat frame as the reference value instead of the mean value of the flat frame, so that the effect of the flat frame correction will always be to either brighten or leave unchanged image pixel values – it will never darken areas. I have made this change and uploaded a new version of SharpCap with it you can download from the link below. I'm fairly convinced that this should be an improvement but would welcome any feedback before making this new version public.

https://d.sharpcap.co.uk/download.html? ... 3.2.5900.0

Cheers, Robin
BlackWikkett
Posts: 387
Joined: Sat Jun 23, 2018 8:48 pm
Contact:

Re: Live Stack with Flat Frame Bug?

#26

Post by BlackWikkett »

admin wrote: Fri Dec 21, 2018 8:31 pm Hi,

what histogram levels you get the peak at if you take the shortest possible dark frames when using your gain and offset values that you use for the flat capture? If that's above zero then the fact that you are not using bias frames would tend to indicate a problem with your flat frame correction due to the offset not being corrected in the flat frames.

Sounds like your describing a "Dark-Flat" here Which I don't use. Although I have carefully watched the histogram during SC flat frame tool capture when "BIAS" is checked. This is the stage that SC turns the exposure to lowest possible exposure during the flat frame capture. I can't recall the numbers but did notice a spike in the histogram at 0% but don't recall the ADU value. I can repeat this test as needed since this can be done on the bench.

At one point in this journey we did try using a dark and a dark flat by selecting each in turn during a SC Flat tool creation run. Both options didn't return usable results.

I'm still on track to get some scope time tomorrow night as long as there is no significant changes to the weather forecast. I'll definitely try the new build and update the thread with results.

Thanks again Robin!
-Wikkett
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#27

Post by admin »

Hi Wikket,

Yes, the use of either a dark flat or (pretty much equivalently) the flat bias option is what I was getting at. Failure to do one of these is definitely going to cause issues with flat correction if your black level ADU is at all significant. This is because the calculated ratios between the dimmer and brighter areas the flat will be incorrect if the black level is not correctly accounted for.

For those who are using flats for dust doughnut removal on planetary, lunar and solar imaging this isn't really a big issue because the error introduced by ignoring the black level value is moderately small. However when using flats for deep sky imaging the small error introduced may just happen to end up in the very steep as part of the stretch function being used to enhance the faint stuff leading to visible artefacts of the type you have been seeing.

See the images below, which I took while testing using the electroluminescent panel.

In this case the bias option is being used for flat frame creation and dark frame is being used alongside the flat frame. This is the correct way of doing things and you can see that the resulting image is very close to being completely flat even with a stretch strong enough to show horizontal banding noise generated by the sensor.
bias used for flat, dark used for light.JPG
bias used for flat, dark used for light.JPG (348.57 KiB) Viewed 3709 times
in this case the flat is created without using an bias or dark frame, but I was using a dark frame when capturing. The flat tends to under correct corners of the image significantly (note that this image was taken with a significantly less strong stretch than the previous one)
dark used for light, no dark or bias for flat.JPG
dark used for light, no dark or bias for flat.JPG (276.04 KiB) Viewed 3709 times
Cheers, Robin
Attachments
bias used for flat, no dark for light.JPG
bias used for flat, no dark for light.JPG (281 KiB) Viewed 3709 times
BlackWikkett
Posts: 387
Joined: Sat Jun 23, 2018 8:48 pm
Contact:

Re: Live Stack with Flat Frame Bug?

#28

Post by BlackWikkett »

Just to make sure I'm understanding this correctly. Use a dark-flat while creating the flat or select bias in the flats tool. Also were you using mono or leaving this unchecked? I've found checking mono or using bias is what's been creating a lot of the problems but I'll give it try.

The first image, is this applying the dark and the flat created and just looking at the electroluminescent panel to show that image is relatively flat? In my previous testing I was able to have good results pointing back at my LED panel with the flat and or dark applied. This removed the dust and vignetting but once applying this to DSO in a Live Stacking run the results were terrible.

It looks like you were also using AWB to align all the color channels. Since I'm using an IDAS D1 LPS filter the color white balance will not be possible.

The example images you provided are all using build 3.2.5892. Would you like to test this in that build or the beta .5900?

-Wikkett
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Live Stack with Flat Frame Bug?

#29

Post by admin »

Hi,

firstly, please test with build 5900. That has the fixes to the code that applies the flat frame correction within SharpCap to avoid the problem that you can just see on the first of my images where there is a bit of circular feature in the centre of the image. The first image was also taken without the 'monochrome flat 'option ticked, which means that the flat frame created will automatically colour balance the output image to have aligned peaks in the histogram.

So in summary, yes use a dark flat always or alternatively use the bias option in the flats tool. Neither option is any better than the other but the bias option is considerably more convenient. Both will ensure that whatever offset level you have for black image will not be included in the flat frame calculations.

The result you report where the flat seemed to work well against the LED panel but gave poor results when imaging is typical of a case where the black level has not been removed accurately in either the flat frame or the light frame. For instance if I take a flat frame correctly and then apply it to my EL panel it looks pretty good regardless of whether I used dark subtraction if I use the same exposure that I used to capture the flats. However if I crank down the exposure to reduce the illumination level I get dreadful results for the flat correction if I don't use a dark frame. See the images below which both use the same flat frame and the same exposure settings.

Below image used a dark frame and gives proper flat correction even when the exposure is reduced
withdark.JPG
withdark.JPG (190.46 KiB) Viewed 3706 times
This image was using no dark frame and doesn't correct properly
no dark.JPG
no dark.JPG (196.58 KiB) Viewed 3706 times
Cheers, Robin
BlackWikkett
Posts: 387
Joined: Sat Jun 23, 2018 8:48 pm
Contact:

Re: Live Stack with Flat Frame Bug?

#30

Post by BlackWikkett »

Hi Robin,

Here are the results of testing flat frame correction in Live Stack in build 3.2.5900. All images used same master dark of 25 frames created just prior to testing Live stacking. For live stack used 30 sec, 300 gain, 10 offset, cooler at -15c. Each test image is 20 frames live stacked applying the master dark and one of 8 version of the flat frame. The flat frame used is in the title of each image. Each flat frame was 209 ms 300 gain 0 offset then made different flats enabling bias, mono and changing WB to create 8 different master flats.

My take away from this round of testing is that the heavy over correction causing extreme vignetting is no longer evident. I'm still seeing some dust not removed with the flats, in the images below the dust is at the 10 o'clock position in the image. I've noticed since build 3.2.5892 that setting offset to 0 when creating flats has given the best results. Don't think this has much to do with the build and more with the 0 offset but I've had the best result starting with the .5892 build.

I tried to keep the color balance, stretch as even as possible for these images but the full moon made this very difficult.

-Wikkett

ImageM45 FLAT-209ms-300g-0offs-5050WB-COL Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-5050WB-COL-BIAS Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-5050WB-MONO Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-5050WB-MONO-BIAS Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-AWB-COL Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-AWB-COL-BIAS Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-AWB-MONO Stack_20frames_600s_WithDisplayStretch by Black Wikkett, on Flickr

ImageM45 FLAT-209ms-300g-0offs-AWB-MONO-BIAS Stack_21frames_630s_WithDisplayStretch by Black Wikkett, on Flickr
Post Reply