Massive overcorrected master flat

[kaymann]

I probably expressed myself a little confusingly: I set the black level for the flats to 3 (MONO8) - for example - and leave it at that value. I don't change anything there, including the lights...
StarFlea

Are using flats with the camera set mono8 and the lights are being set in the camera to RAW16?

When you take flats and check take monochrome Flat Frame that does not mean you change your camera to Mono8. If I misunderstood your post I am sorry.

[StarFlea]

No, no, everything is fine!
I've tried the process with pretty much all color spaces. Between the flats and the lights, I leave all settings unchanged, both the black level and the color space. And in the current test runs, I'm not even changing the gain and exposure!

It's a mystery to me! What I also noticed is that the lower the black level is set from the start, the less the overcorrection is. At slightly higher values, the correction of the lights becomes so strong that the entire image turns black.

I don't think it's the cameras' fault, I've tried three different ones. I've now limited the creation of the flats to the essential points, I don't even load camera profiles anymore, I just make the most basic settings by hand. At the moment, I'm running out of ideas...
Are there any secret basic settings in SharpCap that could have been changed by something?

[StarFlea]

I'll now give you a detailed description of how I try to troubleshoot with my camera (here an Omegon veLOX 178M = ToupTek G3M178M), in the hope that someone might see what I'm doing wrong:

0. Setup: C8 with FocalReducer f/6.3 and an electronic flatfield mask (Aurora 220mm):

20240911_191951.jpg (88.17 KiB) Viewed 736 times

1. Start SharpCap and open the camera with the native driver (G3M178M):

1_Open.jpg (23.02 KiB) Viewed 736 times

2. Change the default value of the black level manually from 1 to 3 (with the default color space MONO8):

2_Blacklevel.jpg (80.91 KiB) Viewed 736 times

3. Settings for the master flat. I manually set the gain to 402 so that the histogram is in the right range:

3_FlatSettings.JPG (223 KiB) Viewed 736 times

4. I start the process... 25 flats and around 30 bias images are created. The resulting master flat is then automatically applied to the current live image (with the flat field mask) and the overcorrection becomes immediately apparent:

4_Result.JPG (286.49 KiB) Viewed 736 times

As I said: I don't know what could have gone wrong here!

[StarFlea]

I have now installed SharpCap version 4.1.11388 (first version with native ToupTek support) on another PC for testing purposes: Here too, I get the same overcorrection when creating the master flat.

I have also tried older SharpCap versions (4.0.8173, 4.0.9357, 4.0.9571) via ASCOM: Here too, the overcorrection is evident...

Greetings!

StarFlea

[StarFlea]

So, to rule out other potential sources of error, I tried a white, flat wall instead of my flat field mask to create the flats. Later on, I also replaced my C8 with a fisheye lens on the camera. But that doesn't change the fact that my master flats remain overcorrected! The lights I get are also very unclean as a result.

I naturally ask myself why I didn't notice this earlier. That's probably because I've only been creating flats for a relatively short time and the black level in my few previous attempts was always set to 0 or at least to a very low value, the effect of which I had hardly noticed visually at first. It was only during my last attempts to take solar photos and calibrate everything that it became really noticeable...

The number of possible causes is now pretty limited:

1. I'm caught in a serious mistake in my thinking or operation! In this case, please tell me what it is...
2. All three of my astrocams are defective...
3. The ToupTek drivers used (native and ASCOM) are unreliable. Here is a request to all readers of this thread: Please, maybe someone with a camera from another manufacturer and otherwise similar equipment can provide evidence to the contrary by reproducing the above process with a non-ToupTek (or its re-brand) camera...
4. The problem lies in SharpCap's internal processing. In this case, it must have something to do with the black level in combination with the bias frames / flat darks and the offset-value in the FITS-Filename.


Note: The cause of the overcorrection was not (as initially suspected) in the FITS header ("ROWORDER"). The fact that the overcorrection disappeared during my experiments with ROWORDER was simply because I renamed the file names and the offset entry was thus unconsciously overridden.

Regards
StarFlea

[admin]

Hi,

thanks for continuing to post your experiments.

Please can you test this...

Make sure the windows decimal separator is set to '.' not ',' and retest from start to finish.

I wonder if the output file is being named with a '.' as the decimal separator, but being re-read and interpreted as if a ',' was the separator or something.

On the other hand, if that *is* the case then I am surprised that it would only be noticed now - the code has been like that for about 5 years...

cheers,

Robin

[StarFlea]

BRILLIANT!


That seems to be the solution! In Germany, the symbols for decimal separators and digit grouping are swapped compared to the UK standard:

Format.JPG (79.38 KiB) Viewed 702 times

I've now done a test and a counter test, and it works!

Thank you! I was beginning to doubt myself!
StarFlea

[admin]

Hi,

thanks for confirming - I will fix for the next update, but I still don't understand quite how it could have been like that for 5+ years and no-one noticed...

cheers,

Robin

[StarFlea]

Well, I know from experience that many users here in Germany also run their PCs with UK or US versions, especially for "scientific applications". And with a low black level and the dark sky background of DS images, it might not be immediately noticeable...

In any case, I'm glad that this can now be fixed!

But I still have two questions that came up during the experiments:

1. The bias frames are apparently made with an exposure time of 1 ms by default, even if the camera supports lower exposure times. If I now create flats with an exposure < 1 ms, the bias frames will be brighter than the actual flats. As far as I know, bias frames should always be created with the smallest possible exposure. Would it make sense to read the camera values ​​in advance to generate the bias frames?

2. Can I use the calibrated master flat created by SharpCap without any problems when stacking with other software, e.g. with ASTAP, or does it have to be recalibrated because of the offset value in the file name?

Greetings!
StarFlea

[turfpit]

If I now create flats with an exposure < 1 ms

Flats with a very short exposure are not a good idea. I had a problem where there was horizontal banding in the flats. It was caused by the exposure and the frequency of the light panel interacting. Better to reduce the intensity of the light source or apply a layer of cloth or paper to force a longer exposure for the flat.

As far as I know, bias frames should always be created with the smallest possible exposure

I have seen references on Cloudy Nights to cameras with a Panasonic MN 34230 sensor (e.g. ZWO ASI1600, QHY163 or
Atik Horizon) where the recommendation is not to use <0.3s for bias frames as there can be a varying gradient across the frame with very short exposures.

Glad you got the the bottom of your problem.

Dave