New Feature : Dark Frame Scaling

All the latest news about new features and improvements to SharpCap
Post Reply
User avatar
admin
Site Admin
Posts: 14657
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

New Feature : Dark Frame Scaling

#1

Post by admin »

Hi folks,

once again, a collaboration with @borodog (Mike) has helped forge a new SharpCap feature :)

Dark frame scaling is designed to help in situations where the dark frames in use don't exactly match the conditions for the light frames. The most obvious reason this might happen is if you are using an uncooled camera, so you have no real control over the sensor temperature. The same code can also _try_ to correct for changes in exposure and/or gain, but I would advise capturing matching darks if at all possible in those situations.

Anyway, in more detail...

What does Dark Scaing do?

When dark scaling is active, SharpCap will try to calculate a correlation between key pixels in the dark frame and the light frame to work out a scaling factor that needs to be applied to the dark frame to give best correction when the dark frame is applied. For instance, if the light frames are being taken at a colder temperature than the dark frame, it's likely that the scaling factor will be less than one, meaning that the dark frame will be scaled down before subtraction to get best results.

This calculation is done on a frame-by-frame basis, so if the camera temperature changes during a session, SharpCap's dark frame scaling will be constantly adjusting to the changes

How do I activate it?

If all the right conditions are in place to use Dark Frame Scaling (see below) then you will see a control appear two below the dark frame selection control, which you can use to turn dark frame scaling on or off
Screenshot 2024-11-04 150422.jpg
When is Dark frame scaling available?

Dark frame scaling is available (and the on/off control will show to enable/disable it) if all of the following conditions are met
  • The camera is in a high bit depth (12/14/16 bit) mode
  • Camera sensor analysis data is available for that camera in the selected mode
  • A dark frame has been selected in the 'Subtract Dark' control
There also need to be some dark regions in the light frames so that SharpCap can use these areas to calculate the dark scaling factor required - if nebulosity or a galaxy fills the whole frame (or it's not an astrophoto at all), the scaling calculation will probably fail to arrive at the right result.

Dark frame scaling may also be available when loading previously saved FITS files via the Virtual Folder Monitor camera (providing the FITS files have been saved with a recent version of SharpCap 4.1, which adds FITS headers related to the e/ADU and bias levels of the camera when the images were captured).

Why don't I need a bias frame?

Dark scaling techniques traditionally use a combination of a dark frame and a bias frame to calculate the new, scaled, dark frame data. Essentially you subtract the bias away from the dark frame, scale the result of that calculation then add the bias back on.

Right now, SharpCap isn't using a bias frame, instead it's using the measured bias level from camera sensor analysis data - using a single constant value across the whole frame.

In order to get benefit from using a true bias frame, you need to capture a very large number of individual bias frames to make a master bias (probably several hundred - with lower numbers random noise dominates over the pattern noise you hope to find in the master bias). Even if you do that, the variation of a master bias is low with modern CMOS cameras (I measured a standard deviation of 0.2e), meaning the effects are small and are reduced further if the dark scaling factor is close to one (small corrections made by dark scaling)

How do I find out if it is doing anything?

You can look in the SharpCap log, where the scaling applied to each frame will be recorded along with a measure of how strong the correlation that SharpCap found in the data was ...

Code: Select all

Info    15:22:24.123311 #44 Dark frame scaling of 0.6287x applied (scaling explains 86.0% of warm pixel variability) 
You will also see log entries if dark scaling fails to find a valid scale factor.

Alternatively, just hover your mouse over the 'Dark Scaling (Experimental)' control to see similar information in a tooltip...

Note... having left the camera used for the test above running for several minutes while writing this article, the sensor has warmed up, and now the results being seen are more like this...

Code: Select all

Info    15:28:20.773099 #44 Dark frame scaling of 0.8841x applied (scaling explains 89.8% of warm pixel variability)                             
The scaling factor has risen from 0.6287 to 0.8841 due to the temperature of the sensor rising.
Screenshot 2024-11-04 152447.png
What are the limitations of dark scaling?

If a scaling factor outside the range of 0.1 to 10.0 is calculated, it will be ignored and no scaling will be applied. Additionally, if the correlation calculated by SharpCap explains less than 10% of the variability in pixel brightnesses in the light frame, no scaling will be applied.[*]

What improvements can I expect?

Subtle reductions in noise in the darker parts of the image where dark frame correction is more accurate, better handling of warm/cool pixels, reducing their impact on the final image.

Why is it experimental?

Basically because of limited testing so far - while it's working as tested so far, there may be situations where it should work and it doesn't, or where it calculates the wrong scale factor for some reason and makes a mess of things. Please use with care and keep an eye on progress/results as you are capturing to ensure that everything is working as expected. Please do report any bugs/problems/poor results (ideally with captured images, darks, etc so the problem can be reproduced in testing).

cheers,

Robin
You do not have the required permissions to view the files attached to this post.
User avatar
admin
Site Admin
Posts: 14657
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: New Feature : Dark Frame Scaling

#2

Post by admin »

Hi,

anyone trying this who finds it does not become available with your camera, please try this version:

https://d.sharpcap.co.uk/download.html? ... 0&arch=x64

The previously uploaded version has a bug in the handling of cases where the true bit depth of the camera is not 16 (ie 12/14 bit sensors). This prevents the dark scaling option from showing for those sensors. The updated version in the link above resolves that issue.

cheers,

Robin
Borodog
Posts: 401
Joined: Fri Jan 01, 2021 7:25 pm

Re: New Feature : Dark Frame Scaling

#3

Post by Borodog »

Robin,

Tremendous thanks for implementing this. When I first started doing EAA in late 2020 or early 2021, I was using an SV305. The darks beginning to cut holes in my live stacks as the temperature fell was one of the first problems I ran into and I have been after ways to fix it ever since, going to pretty extreme lengths like making refrigerator and freezer darks, interpolated darks, a synthetic dark library, etc. I firmly believe that there are many more people out there trying to do EAA with uncooled cameras, simply because of their price point, than anyone realizes. This will improve the results for a lot of people, including me. Even though I have a cooled camera now, I still often choose one of my uncooled cameras for various reasons.

Cheers,

Mike
cdanil
Posts: 10
Joined: Sun Dec 26, 2021 4:04 am

Re: New Feature : Dark Frame Scaling

#4

Post by cdanil »

Hi Robin,

I'm afraid the new feature might be making a mess with my live stacking with dark subtraction. I tried two different sets of raw frames and two different darks, and I'm getting the same weird results, as if it's turning the image negative (the nebula appears dark) and the colors all off. 99.9% sure dark frame scaling was turned off as I wasn't even aware of it (it wouldn't activate with a Folder Monitor Camera anyway, right?). If I stack without the dark, everything is OK. I then uninstalled v12764 and went back to v12693 and the problem went away (same data). Happy to reinstall the current version and send any screenshots, logs, etc.

The raw frames and the darks were captured with a ZWO 294MC Pro cooled to -10C, and live stacked using the Folder Monitor Camera. 15s exposure, 300 gain.
User avatar
admin
Site Admin
Posts: 14657
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: New Feature : Dark Frame Scaling

#5

Post by admin »

Hi,

dark frame scaling can (in theory) be activated on the folder monitor camera if you are using FITS files and the light frame fits files contain enough special FITS headers for SharpCap to work out the camera sensor information. However, it's definitly off by default, so that's unlikely to be the precise cause.

However, I did update dark frame the way that hot pixel removal is handled when a dark frame is in use as part of the changes to allow dark frame scaling (the hot/warm pixels are the key data that allows dark frame scaling to happen, so they need to be preserved until the scaling calculation is done - this means that rather than being wiped out early in the processing, they are catalogued into a list early in the processing and the list is wiped out later). Maybe this change has had unexpected consequences?

Can you send me enough raw frames / dark frame / flat frame to let me test this out and try to reproduce the issue - that's probably the fastest way to get to the bottom of what is going on as I can quickly switch between the different version of the code to see what happens.

cheers,

Robin
cdanil
Posts: 10
Joined: Sun Dec 26, 2021 4:04 am

Re: New Feature : Dark Frame Scaling

#6

Post by cdanil »

https://www.dropbox.com/t/u4QCDYIQISTaiY59

Hi there, this should have the dark, the flat, five raw frames, and the log, as well as the resulting stacked image. Thanks Robin.
cdanil
Posts: 10
Joined: Sun Dec 26, 2021 4:04 am

Re: New Feature : Dark Frame Scaling

#7

Post by cdanil »

https://www.dropbox.com/t/Ju9ZJCGQoMcvSDOZ

Following up on what I just sent, the stacking was done without applying a flat. This is the one I've been using. With both dark and flat applied, I get a similar result as with dark only (what I previously sent). With flat and no dark, the stacked image looks overcorrected. Without either flat or dark, I get a nice image!
User avatar
admin
Site Admin
Posts: 14657
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: New Feature : Dark Frame Scaling

#8

Post by admin »

Hi,

thanks for the quick response - I have found the issue (at one point in the processing a flag was being lost from the processing). Update coming soon.

Robin
User avatar
admin
Site Admin
Posts: 14657
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: New Feature : Dark Frame Scaling

#9

Post by admin »

Please try version 4.1.12766 from the normal downloads page (https://www.sharpcap.co.uk/sharpcap/downloads), which should fix the problem.

thanks for the help!

Robin
cdanil
Posts: 10
Joined: Sun Dec 26, 2021 4:04 am

Re: New Feature : Dark Frame Scaling

#10

Post by cdanil »

Works great now. Thanks!
Post Reply