Hello everyone,
I've been experiencing a problem since I moved to 4.0. My SSD hard drive doesn't have much space and it's rarely above 50 GB free. After I moved to work with bin 1x1 (previously it was bin 2x2, so amount of data wasn't a problem), I keep filling this space very quickly. But the problem is, the free space left reduces too quickly. Each FITS frame I save have 32 MB (ASI1600), but it allows to save about 800 frames. That is approximately 25 gigabytes. My C drive space drops from 50 GB to 0 GB. Where are those missing 25 gigabytes? They are restored after I restart my computer. But I don't wanna restart it to continue imaging. I can't find myself in Settings, where is something like "save backup frame somewhere, but it requires twice amount of disk space" that it clearly would be.
Can anyone help me to track down how do I stop generating so many temporary files, so I can fully use 50 gigabytes (1600 frames) instead of half of it?
Thanks!
Temporary files?
Forum rules
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
If you have a problem or question, please check the FAQ to see if it already has an answer : https://www.sharpcap.co.uk/sharpcap-faqs
-
admin
- Site Admin
- Posts: 13362
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Temporary files?
Hi,
an odd one, SharpCap doesn't save temporary files when creating FITS files, so this is unexpected.
I just did a test using an ASI1600MM at 16bit, so 32Mb FITS files. Capturing 1000 frames reduced available disk space by approx 32Gb as expected, so the issue does not happen to me (tested with SharpCap 4.0 and 4.1 64 bit).
I wonder if it is something specific to your system setup that is causing this effect (antivirus or other software running in the background?)
cheers,
Robin
an odd one, SharpCap doesn't save temporary files when creating FITS files, so this is unexpected.
I just did a test using an ASI1600MM at 16bit, so 32Mb FITS files. Capturing 1000 frames reduced available disk space by approx 32Gb as expected, so the issue does not happen to me (tested with SharpCap 4.0 and 4.1 64 bit).
I wonder if it is something specific to your system setup that is causing this effect (antivirus or other software running in the background?)
cheers,
Robin
Re: Temporary files?
Thanks for answer!
So I took the camera and did some tests in my room. I tested on USB 2.0 and 3.0. With SharpCap 3.1 and 4.0. The problem appears in all cases. I added an exception to Windows Defender (by adding process "SharpCap"), but nothing changed.
100 frames take about 3.05 GB. In all cases it took 4.8-5.0 GB of disk space, except with SharpCap 3.1 (which also took much longer to obtain 1.0s frames in USB 2.0 compared to SC 4.0!), where it was 6.4 GB; I haven't tested 3.1 with USB 3.0.
4.8-4.9 GB actually fits the size of sum of Size (3.05 GB) and Size On Disk (last one is 1.71 GB), under Properties, or maybe that's a concidence. The space actually gets back itself after some time, but I still haven't figured how to force it without restarting (waiting just helped). But I noticed it cannot go back during hours of imaging, because it never happened during observations (otherwise, I would get GBs restored before finishing capturing frames). I waited for 20 minutes after capturing, didn't happen, closed SharpCap and I got GBs back after ~10 minutes. Now, I closed instantly after that and after 30 minutes I still don't have my GBs back. After deleting those frames instantly (whole folder), I indeed got 4.8-4.9 GBs instantly returned, even if these were 100 x 32 MBs. Antivirus is reasonable. But maybe I'm doing that exception wrong. Thought it will ask for a specified process, but I just had to type something, so I wrote "SharpCap"... I guess it's not that. Does this problem resemble anything? It's clearly not SC issue there, as GBs are returned after sometime having the program closed.
So I took the camera and did some tests in my room. I tested on USB 2.0 and 3.0. With SharpCap 3.1 and 4.0. The problem appears in all cases. I added an exception to Windows Defender (by adding process "SharpCap"), but nothing changed.
100 frames take about 3.05 GB. In all cases it took 4.8-5.0 GB of disk space, except with SharpCap 3.1 (which also took much longer to obtain 1.0s frames in USB 2.0 compared to SC 4.0!), where it was 6.4 GB; I haven't tested 3.1 with USB 3.0.
4.8-4.9 GB actually fits the size of sum of Size (3.05 GB) and Size On Disk (last one is 1.71 GB), under Properties, or maybe that's a concidence. The space actually gets back itself after some time, but I still haven't figured how to force it without restarting (waiting just helped). But I noticed it cannot go back during hours of imaging, because it never happened during observations (otherwise, I would get GBs restored before finishing capturing frames). I waited for 20 minutes after capturing, didn't happen, closed SharpCap and I got GBs back after ~10 minutes. Now, I closed instantly after that and after 30 minutes I still don't have my GBs back. After deleting those frames instantly (whole folder), I indeed got 4.8-4.9 GBs instantly returned, even if these were 100 x 32 MBs. Antivirus is reasonable. But maybe I'm doing that exception wrong. Thought it will ask for a specified process, but I just had to type something, so I wrote "SharpCap"... I guess it's not that. Does this problem resemble anything? It's clearly not SC issue there, as GBs are returned after sometime having the program closed.
-
admin
- Site Admin
- Posts: 13362
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Temporary files?
Ok, a few more things you could try...
* Check your SharpCap settings - look in the memory tab and see what you have set up there. If SharpCap wants to use large amounts of memory then it will allocate that via the swap file in some circumstances, and in theory that could grow and use up more disk space. However, SharpCap shouldn't keep the memory for every file written, so that is a bit of a long shot. The amount it uses should be limited by your memory settings anyway.
* Run Windows Resource Monitor while capturing and see what files are having large amounts of data written to in the 'Disk' tab (you can sort the lower panel by 'Write (B/sec)'. That will include the FITS files, but if it shows something else then maybe that will point you in the right direction.
* Once you end up in the situation where there is phantom space being used, run a disk space analysis tool (WinDirStat is a good, free one) to see what folders are using the space - leave that open then when the space has returned run another copy and see what differences you see.
cheers,
Robin
* Check your SharpCap settings - look in the memory tab and see what you have set up there. If SharpCap wants to use large amounts of memory then it will allocate that via the swap file in some circumstances, and in theory that could grow and use up more disk space. However, SharpCap shouldn't keep the memory for every file written, so that is a bit of a long shot. The amount it uses should be limited by your memory settings anyway.
* Run Windows Resource Monitor while capturing and see what files are having large amounts of data written to in the 'Disk' tab (you can sort the lower panel by 'Write (B/sec)'. That will include the FITS files, but if it shows something else then maybe that will point you in the right direction.
* Once you end up in the situation where there is phantom space being used, run a disk space analysis tool (WinDirStat is a good, free one) to see what folders are using the space - leave that open then when the space has returned run another copy and see what differences you see.
cheers,
Robin
Re: Temporary files?
libmar96
This is a way to try to track down where the disk space is being used, which hopefully will lead to what is producing the files.
Download TreeSize Free https://www.jam-software.com/treesize_free?ca=1. This program (needs to be run as administrator) will analyse the drive and produce a report in descending folder size. The report can be printed.
Carry out the SharpCap activity which brings on the reduction in disk space. Re-run the report and try to identify which folder has increased. This can be done in the house, without scope and during the day. This might take a few attempts and expansion of folders to identify the files.
This is a way to try to track down where the disk space is being used, which hopefully will lead to what is producing the files.
Download TreeSize Free https://www.jam-software.com/treesize_free?ca=1. This program (needs to be run as administrator) will analyse the drive and produce a report in descending folder size. The report can be printed.
- TreeSize.JPG (132.19 KiB) Viewed 611 times
Carry out the SharpCap activity which brings on the reduction in disk space. Re-run the report and try to identify which folder has increased. This can be done in the house, without scope and during the day. This might take a few attempts and expansion of folders to identify the files.
- What version of Windows?
- What antivirus software?
- What size is the SSD drive?
Re: Temporary files?
Thanks for answers! So, here are also answers to new questions:
- that memory settings value is set to 1.0 GB in both cases
- In Windows Resource Monitor I only see a bunch of new files that are FITS related, nothing else exceptional
- WinDirStat and TreeSize both show the same result - increase of Users folder by 1.7 GBs, slightly above 100 folders/files (capture folders are created as well), total memory used also increased by 1.7 GBs, nothing else from folders in C:/ got changed in size or amount of files
- In both tests above I got a decrease of 4.8 GBs free in bar under "My computer" (43.3 -> 38.5 | 43.0 -> 38.2)
- I use Windows 11, Windows Defender, my SSD is HFS256G3BTND-N210A
- Attachments
-
- WinDirStarReport.png (19.56 KiB) Viewed 592 times
-
admin
- Site Admin
- Posts: 13362
- Joined: Sat Feb 11, 2017 3:52 pm
- Location: Vale of the White Horse, UK
- Contact:
Re: Temporary files?
Hi,
so the conclusion is that no extra files are being written, but that extra space is being 'wiped' out on the disk and then comes back later, yes?
I wonder if this is the SSD? Early SSD devices stored one bit per cell, but were expensive, so manufacturers started with devices that can store 2, 3 or even 4 bits per cell by storing different voltage levels. It looks like yours is MLC, which is usually two bits per cell.
The trouble is that *writing* to this new multiple bits per cell SSD design is much slower than a single bit per cell SSD, so the trick is that part of the SSD is set up in one bit per cell mode to allow for fast writing, then the data gets moved to slower storage in the background, clearing out the fast write area for re-use. Some drives have a pre-set area for this fast write cache, others allocate it as required by using normal storage areas in single bit mode.
I'm guessing that your drive is of the second sort - that it allocates the fast write area dynamically. When you write lots of new data it temporarily uses twice as much space as normal because it is in the single bit mode. Over time, the drive (possibly with the co-operation of Windows) pushes that data to two bits per cell storage and the space re-appears. I guess there must be some way that the drive and Windows communicate to make the space appear to drop - maybe a special driver?
This is really a bit of speculation, but I can't think of anything else at the moment that even starts to make sense of it.
You could test by plugging in an external hard drive over USB (the old, spinning sort) if you have one and saving to that. If the same issue fails to appear with the free space of the hard drive then it would certainly point at the SSD explanation.
cheers,
Robin
so the conclusion is that no extra files are being written, but that extra space is being 'wiped' out on the disk and then comes back later, yes?
I wonder if this is the SSD? Early SSD devices stored one bit per cell, but were expensive, so manufacturers started with devices that can store 2, 3 or even 4 bits per cell by storing different voltage levels. It looks like yours is MLC, which is usually two bits per cell.
The trouble is that *writing* to this new multiple bits per cell SSD design is much slower than a single bit per cell SSD, so the trick is that part of the SSD is set up in one bit per cell mode to allow for fast writing, then the data gets moved to slower storage in the background, clearing out the fast write area for re-use. Some drives have a pre-set area for this fast write cache, others allocate it as required by using normal storage areas in single bit mode.
I'm guessing that your drive is of the second sort - that it allocates the fast write area dynamically. When you write lots of new data it temporarily uses twice as much space as normal because it is in the single bit mode. Over time, the drive (possibly with the co-operation of Windows) pushes that data to two bits per cell storage and the space re-appears. I guess there must be some way that the drive and Windows communicate to make the space appear to drop - maybe a special driver?
This is really a bit of speculation, but I can't think of anything else at the moment that even starts to make sense of it.
You could test by plugging in an external hard drive over USB (the old, spinning sort) if you have one and saving to that. If the same issue fails to appear with the free space of the hard drive then it would certainly point at the SSD explanation.
cheers,
Robin
Re: Temporary files?
libmar96
I agree with Robin that your test shows no extra files are created. The detail provided by Robin does open up a can of worms.
Maybe redo the test with 25Gb of data as per your first post in the thread?
I use a 240 Gb SSD drive (Crucial CT240BX300) as a secondary drive for processing data with PixInsight. For data generated with an Altair 183C the 16-bit FITS files are similar in size to your ASI1600 files. With all the various PI processing steps, a data set of 100 captures generates a huge amount of data. I have run close to the disk capacity and never seen the problem you are encountering.
Running a modern Windows system with 50 Gb free does not leave much room for operating system overhead when collecting data. The ASI1600 is a ~$1,000 dollar camera. Consider upgrading your SSD - a 1Tb SSD only costs ~£50 plus the cost of disk cloning software.
https://www.amazon.co.uk/Crucial-MX500- ... 9294&psc=1
Dave
I agree with Robin that your test shows no extra files are created. The detail provided by Robin does open up a can of worms.
Maybe redo the test with 25Gb of data as per your first post in the thread?
I use a 240 Gb SSD drive (Crucial CT240BX300) as a secondary drive for processing data with PixInsight. For data generated with an Altair 183C the 16-bit FITS files are similar in size to your ASI1600 files. With all the various PI processing steps, a data set of 100 captures generates a huge amount of data. I have run close to the disk capacity and never seen the problem you are encountering.
Running a modern Windows system with 50 Gb free does not leave much room for operating system overhead when collecting data. The ASI1600 is a ~$1,000 dollar camera. Consider upgrading your SSD - a 1Tb SSD only costs ~£50 plus the cost of disk cloning software.
https://www.amazon.co.uk/Crucial-MX500- ... 9294&psc=1
Dave