Massive overcorrected master flat

[admin]

Hi,

ah, OK, now we are getting somewhere...

The magic percentage in the filename!

It's essential to correct for the 'pedestal level' in the image when performing flat correction (that is, the constant pixel value that would be in an image with exactly zero electrons collected across each pixel in the frame - this is usually controlled by the offset/black level/brightness control of the camera).

Basically, you take the image you want to correct, subtract the pedestal level value from each pixel value, leaving only the part of the image pixel values actually coming from photons hitting the sensor. You then apply the flat correction and optionally re-add the pedestal value. If you don't do this pedestal subtraction then the flat correction factor is also applied to the pedestal value, which leads to over correction. Alongside this, the flat correction has to be calculated to take account of the pedestal level in the flat frames that have been captured (failing to do that leads to under correction)

Best results when using flats (particularly for deep sky) are achieved by using dark frames - the dark subtraction removes the pedestal level, then the flat correction is applied to just the right value. However, that's not always possible or desireable, so SharpCap has a backup approach...

When capturing a flat, SharpCap stores the brightness of the pedestal level in the filename as a percentage (it can't go in something like FITS headers since it's possible to configure things so the flats are saved into PNG or TIFF). If the flat is being used without dark frames then that percentage is read from the file name and used to calculate an appropriate pedestal level to adjust out. If the flat is used with dark correction then the percentage is ignored.

This approach works as long as the offset/brightness/black level setting remains the same between flat capture and light frame capture. It's particularly handy for using flats to correct for dust donuts when doing high speed solar/lunar/planetary imaging (darks don't form part of the workflow for these sorts of imaging). In fact it was added to support that use case probably before SharpCap was widely used for deep sky or EAA imaging.

I think we still need to work out how it is that your workflow manages to activate this feature in a way that causes it to give the wrong results - normally that would only happen with a change in offset setting between flats and lights.

cheers,

Robin

[StarFlea]

Hmm,

The black level is set to 3 for both the lights and the flats. Everything else is identical, as you can see from the settings.

When I create the master flat, I usually use the following settings:

• Averaging over 25 frames

• monochrome

• subtracting bias frames (because it's quicker) to avoid overcorrection

It would be a mystery to me that the values ​​would change during this relatively short workflow, since the incorrectly corrected master flat is applied to the live image immediately after the process.

The following entry can be found in the log file:

Info 13:16:15.751250 #37 Updating dark frame offset: flatmean=0.465, darkmean=0.024, flatexposure=37, darkexposure=1, boost=1.0277777777777777, result=0.011364696451191314 in void SharpCap.UI.CreateFlatViewModel.ProcessFlatFrames(int frameCount, string deviceName, bool convertToMono, ColourSpaceId colourSpaceId, IFileNameProvider fnp, CancellationToken cancellationToken)

Info 13:16:15.813659 #1 Loading Flat frame from C:\CCD\Capture\Capture\2024-08-31\flats\MasterFlat_13_15_22_offset=1.136%.fits in bool SharpCap.Base.PropertyControls.FileSelectionControlBase.LoadFrame(string selectedFile)


So this offset in the file name is SharpCap-specific, right? Can you calculate the value 1.136 from the log entries above?

Best Regards!
StarFlea

[admin]

Hi,

the calculation works like this...

boost = flat exposure / (flat exposure - bias exposure)
pedestal level = flatmean - (flatmean - darkmean) * boost

Essentially it's treating it as two points on the graph at

x = bias exposure, y = darkmean and
x = flt exposure, y = flatmean

then extrapolating the straight line between them back to x=0 (zero exposure).

That then should calculate that the pedestal level is 1.136% of max ADU or ~744 (16 bit) / 3 (8 bit).

cheers,

Robin

[StarFlea]

So, there is news...

After many, many attempts with my camera (Omegon veTEC 571c), in which I made flats with a wide variety of settings, I was only able to create correct flats with a single setting, namely by setting the black level to 0.

This applies both when using the native camera driver (high speed) and for ASCOM.

For high-speed recordings, the black level is set to 3 (out of a maximum of 31) by default, and when using ASCOM it is set to 250 (out of a maximum of 7936).

The only other option was to completely switch off the black level offset correction when creating flats.

To rule out that this is a camera problem, I made the counter test with my planetary camera (Omegon veLOX 178c): Here too, there is an incorrect correction at black level 3, which disappears at a value of 0.

StarFlea

[admin]

Hi,

OK, good info - I have a ToupTek equivalent of that camera, so will be able to test myself when I get home (I am away at the moment). Are you using the ASCOM driver for the camera or the direct support (I think it should have direct support via the ToupTek camera support).

thanks,

Robin

[StarFlea]

The problem occurs both with the ASCOM driver and with direct access.
As a test, I have already integrated the latest Touptek.dll (https://www.touptek-astro.com/downloads ... av=box_win) in SharpCap, that did not solve the problem either...

[admin]

Hi,

I have just been testing today with my ToupTek 571M. I used the following steps.

* Camera with 25mm C mount lens - this vignettes badly, giving the flat something to do as you move away from the center (although the corners are basically fully vignetted - the flat cannot help there). This is just my setup - anything to generate some image and allow flat illumination is fine.

* Select native driver, MONO16
* capture flat with a piece of white paper over the lens, bias subtraction option selected, ignore the fact that the corners of the image are too dark (just make sure nothing over exposed)
* Once flat captured, check that the illuminated area of the sensor is even - adjust exposure and check remains even.
* Remove flat paper, check that the illuminated area looks good
* Close SharpCap, re-open, re-use flat saved in earlier steps and re-check
* Repeat for different black level settings.

All seems OK in this particular test scenario - certainly down to the accuracy of my 'flat panel'. As an example below

Without flat

noflat.jpg

With flat

withflat.jpg

You can see the flat evens out the brightness within the circle that is illuminated (although near the edges, the flat is boosting a lot, so the result is noisy).

I think the key now is to work out how your scenario (which is giving problems) differs from the steps above (which work for me - maybe you could confirm that they work for you too). By finding that key difference we will start to understand where the problem is coming from.

cheers,

Robin

[StarFlea]

Hmm, I see, normally you can't do much wrong with this process (or maybe I can)?

The only significant difference to my camera so far was that you used the MONO counterpart to my color camera.
To rule out that this is the problem, I also used a MONO version of a planetary camera (G3M178M). However, the overcorrection was also more than obvious here.

But I noticed one difference between your workflow and mine (please correct me if this is not the case): I set the black level of 3 when creating the master flat. From what I've read, you only set the black level afterwards when testing the lights.

Could this be the problem?

Regards!

StarFlea

[admin]

Hi,

yes, the change in black level is unfortunately precisely the wrong thing to do - you need to capture the flats and the lights at exactly the same black level for this to all work properly without needing separate dark frames. More details in the 'Getting Good Flat Frame Correction' section of the user manual - https://docs.sharpcap.co.uk/4.1/#Gettin ... Correction

If you change the black level between the flat capture and the light capture then SharpCap doesn't properly know the pedestal level for the light frames (no longer given by the magic % value in the flat frame file name), so either too much or too little pedestal is removed, leading to over or under correction.

Hope this is the solution,

Robin

[StarFlea]

I probably expressed myself a little confusingly: I set the black level for the flats to 3 (MONO8) - for example - and leave it at that value. I don't change anything there, including the lights.

I am well aware that the black level has to be the same for all the pictures taken.
I am currently testing the new version, but I'm not really making any progress!

The steps that need to be taken here are so elementary that I can't find the cause of the behavior.

StarFlea