Massive overcorrected master flat

[StarFlea]

Hi Robin!

Today I started a few attempts to optimize solar shots. To do this, I use a flat mask to generate a master flat (25 frames with subtracted bias frames) with the same gain (and temerature) as the light frames.
At first everything was great and I got a master flat, which I applied to the raw images from the flat mask: these were completely flattened as desired.

Through my own fault, however, I messed up the shot and then, to make matters worse, deleted my master flat irretrievably.
So: regenerate the master flat again! But no matter what I tried, from that point on, in many attempts I only got over-corrected master flats. I also restarted SharpCap several times, with no change!

For illustration purposes, I have uploaded the example images (reduced in resolution)...


Here is the raw image with the flat frame mask:

Raw-Frame.jpg (394.75 KiB) Viewed 2641 times

The master flat:

MasterFlat_13_15_22_offset=1.136%.jpg (254.38 KiB) Viewed 2641 times

The raw image over-corrected by the master flat:

Overcorrection.jpg (104.6 KiB) Viewed 2641 times

I don't know if I'm doing something wrong! Actually, I do everything as usual, but unfortunately I can no longer create a usable master flat!
I have also attached the camera profile and the log file used...

Greetigs!
StarFlea

[StarFlea]

When I posted here, I noticed that the master flat appears to be mirrored on the horizontal axis.

Could that have something to do with it?

[admin]

Hi,

the mirroring of the image could be related to the save formats (if they are different) and how the image has been processed, or it could be if you have used any 'flip' options for the camera - those settings must be the same when taking the flat as when using it, otherwise it will try to correct the wrong parts of the image.

Some things to check to get a good flat include

* Option for bias or dark flat chosen
* Use dark frames too for long exposure imaging with flats
* when capturing a master flat in SharpCap, leave the flat illumination in place when done and the flat is activated - the image should look very even in brightness, and should remain even (at different brightness levels) if you adjust the exposure down by a factor of 2 or so. If this is OK then your flat is working pretty well
* Applying a strong enough stretch to the final image will reveal that pretty much any flat is not 100% perfect - with a strong stretch you will start to see issues re-appearing that the flat has reduced, but not completely eliminated.

cheers,

Robin

[StarFlea]

Hmm,

So I have now mirrored the original master flat on the horizontal axis using an external program (AstraImage 6), saved it and then applied this in SharpCap to a raw frame (via the Folder Monitor Camera) evenly lit with a flat mask.

Result: Vignetting and dust donuts are now cleanly balanced:

Correct.jpg (428.34 KiB) Viewed 2609 times

It seems that the master frame was actually flipped around the horizontal axis, but I am not aware of having changed any settings here. I have attached the settings of the master flat, I cannot find anything here. What causes this flip?

The settings for the master flat generation explicitly state: "Flip=None" and "Flip (after dark/flat)=None"

I cannot understand what is going wrong here. If you look at the original master flat, you can see that it was already created mirrored. Is there an option that could cause this? Is it possible that this could be due to the file format of the master flat (FITS)?

Regards!
StarFlea

[StarFlea]

Another idea:
Is the cause perhaps the light frames? Perhaps they are mirrored for some reason.

I have also attached the settings of the light frame above for comparison...

[StarFlea]

I've tried many more times, but the image remains incorrectly corrected.
When I open the MasterFlat with ASTAP, you can see from the dust donuts that it is mirrored compared to the lights.

When I think back over the past few weeks, it's clear to me that I've probably had the problem with the Masterflat for a while, but I wasn't really aware of it.

I've now run additional logging for the current attempts, maybe this will provide new insights...

[StarFlea]

So, one more addendum: The "correct correction" only works if I mirror the original master flat vertically with an external program, but only if I then save the image as a TIF.
In the FITS format there are always some side effects or overcorrections in the application, whether I save the FITS with ASTAP or with GIMP...

Another possibility: I open the original master flat with the folder monitor camera in SharpCap (here it is already displayed in the correct orientation) and save it again explicitly as FITS using a snapshot and apply it.

[StarFlea]

I found something interesting:

https://pixinsight.com/forum/index.php? ... ile.20446/

Is it possible that there is an inconsistency with the keyword "ROWORDER" in the FITS header? That could explain the problem. In the header of my saved master flats, the keyword is set to "TOP-DOWN".
I don't know enough about the FITS format, but it might be worth considering...

For the sake of completeness, here is the complete FITS header (read with ASTAP):

Code: Select all

SIMPLE  =                    T / C# FITS: 09/01/2024 18:26:12                   
BITPIX  =                   16                                                  
NAXIS   =                    2 / Dimensionality                                 
NAXIS1  =                 6224                                                  
NAXIS2  =                 4168                                                  
SET-TEMP=                    0 /                                                
FLATDARK= 'Bias'               /                                                
BLKLEVEL=                    3 /                                                
OFFSET  =                    3 /                                                
GAIN    =                 6000 /                                                
JD_UTC  =   2460555.1848529247 / Julian Date at mid exposure                    
DATE-AVG= '2024-09-01T16:26:11.2926859' / System Clock:Est. Frame Mid Point     
DATE-END= '2024-09-01T16:26:11.3111859' / System Clock:Est. Frame End           
DATE-OBS= '2024-09-01T16:26:11.2741859' / System Clock:Est. Frame Start         
SWCREATE= 'SharpCap v4.1.12529.0, 64 bit' /                                     
IMAGETYP= 'Flat'               /                                                
CCD-TEMP=    6.300000000000001 / C                                              
EGAIN   =                3.232 / Electrons per ADU at true bit depth            
YBINNING=                    1 /                                                
XBINNING=                    1 /                                                
YPIXSZ  =                 3.76 / microns, includes binning if any               
XPIXSZ  =                 3.76 / microns, includes binning if any               
EXPTIME =                0.037 / seconds                                        
ROWORDER= 'TOP-DOWN'           /                                                
BSCALE  =                    1 /                                                
BZERO   =                32768 /                                                
EXTEND  =                    T / Extensions are permitted                       
FRAMETYP= 'Flat'               /                                                
INSTRUME= 'ATR2600C'           /                                                
END                                                                             

Addition: In a copy of the master flat, I used a hex editor to change the value of ROWORDER from "TOP-DOWN" to "BOTTOM-UP" and applied it to my light in SharpCap and then: Now the correction is perfect!

[admin]

Hi,

OK, I think we are getting close then... In theory, FITS files are all supposed to be in order from the bottom left corner upwards, but lots of software (SharpCap included) fails to spot this part of the FITS spec and writes files in the top-down rather than bottom-up format.

Unfortunately when I found out that SharpCap was writing FITS incorrectly, it was really too late to change - changing would have broken existing workflows for so many people that it wasn't an option. Instead SharpCap writes the ROWORDER header to indicate the non-compliance with the normal spec, which some software will read/understand.

SharpCap always writes ROWORDER, and reads ROWORDER in the sense that it affects the bayer pattern of a raw image, but does when reading a FITS image, it assumes that the image is written top-down (as SharpCap itself would write the image). That gives a perfect 'round trip' of save then reload for images in SharpCap, but will lead to an error if trying to load a bottom-up image into SharpCap as either an image or a flat frame.

I think the explanation of the problem lies somewhere in here - the fact that SharpCap is not dealing with bottom-up images correctly when reading, but I can't quite see how yet. I can fix that particular problem for the next update, but not 100% sure it will resolve your issue.

cheers,

Robin

[StarFlea]

The question is: has the problem always been like this?
If I find time tomorrow, I'll test one of the old versions of SharpCap.

Secondly, you have to make sure that the darks (if saved als FITS) are also affected!?

And thirdly, I loaded the original master flat into the Folder Monitor Camera and saved it again as FITS using a snapshot: then the orientation was correct. So obviously normal saving works. In any case, the header always said "TOP-DOWN". My manual change to "BOTTOM-UP" with the hex editor was just an attempt on my part to see what would happen.

When creating the master flat, in this case you should leave the flat mask on the telescope and apply the freshly created master flat to it. If a completely evenly lit image is created, you can be sure that the orientation is correct. That would be a good way to check.

Regards!
StarFlea