How to defeat the bottleneck?

Discussions of using SharpCap for Planetary Imaging
Post Reply
gfstallin
Posts: 2
Joined: Sat Dec 21, 2019 11:19 am

How to defeat the bottleneck?

Post by gfstallin » Sat Dec 21, 2019 11:47 am

I am using the ZWO ASI183MM for lunar imaging. At 20.18 megapixels, I know I'm pulling a lot of data. Typically through the first 1,000 frames captured, I'm averaging around 17 fps and zero dropped frames (good speed for this camera). As I inch past 1,100 frames captured, I start dropping frames. By 1,300 frames captured, I'm down to 3.2 fps and dropping a lot of frames. I've increased buffer to 7,102 Mb. I'm using an SSD drive and I have 16 GB RAM on a Windows 10 laptop and USB3.0. My speed test (within SharpCap) reads 475.972 Mb/s, 4,882 frames (1280X960) written in 12.02 seconds.

When I reduce the ROI to something more typical of a planetary camera (say 1280X720), I don't drop frames at all, no matter how many I capture. I suspect there is a bottleneck somewhere. Does anyone have any idea where that might be? Perhaps it is a clue that it only begins dropping frames after a certain number of frames are captured. I should note that as the hard drive begins to fill up, it appears that I start dropping frames earlier in the sequence; however, I am very uncertain about this. I've only used the camera a few times and this could be perception based on anxiety ("Arghhh...I'm running out of space!") rather an observation based in reality.

Short of buying a new laptop, I can add more RAM or a larger SSD (just ordered one). Let me know any ideas you might have. This is a great camera - I'd like to unlock its full potential.

George

User avatar
turfpit
Posts: 924
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: How to defeat the bottleneck?

Post by turfpit » Sun Dec 22, 2019 11:58 am

George

For general USB troubleshooting tips, have a look at this thread
viewtopic.php?f=18&t=349&p=1556#p1556

The first entry gives some systematic checks to carry out. The second post (by me) mentions USBtreeView. Using USBtreeView will enable you to check if the camera is actually running at USB3 (SuperSpeed, shows as SS at the port connection) or USB2 (HighSpeed, shows as H at the port connection). Best to assume nothing and establish the facts - not all USB chipsets and architectures are equal.

Having experience of an Altair 183C and 183M, I would say full frame (5000x3000) giving you 17fps is about right. If you look through my earlier lunar images (scroll down to the bottom) at https://www.astrobin.com/users/turfpit/ you will find some captures with an Altair 183C and M. Note the fps are low and are only improve when the capture area is reduced.

Exposure is going to affect fps - a larger exposure is going to result in a lower fps. This implies using higher gain to enable lower exposure. I usually use a gain of 30 - 40% when lunar imaging.

8-bit v 16-bit will have a large impact on fps - the higher bit capture will be approx. half the rate of the 8-bit capture. I would generally use 8-bit unless the atmosphere was very steady (rare).

These are steps I would take to try to establish where the bottleneck is:
  • Carry out the tests using only the capture laptop, manufacturer supplied cable, camera and nothing else connected to the machine.
  • Set the exposure to say 10ms and keep that for the duration of any tests.
  • There are 2 types of tests to consider: 1) just observing the fps at the bottom of the SharpCap screen without doing a capture. 2) Writing out a fixed capture say 2000 frames in SER format.
  • Use USBtreeView to check what speed the camera is actually connected to the USB bus.
  • Use Task Manager > Performance tab to check for bottlenecks with CPU, Memory, Disk e.g. 100% utilisation of a resource such as disk.
  • Establish which controls in the Camera Control panel on the right hand side are related to USB speed. For example with an ASI120MC, I have Turbo USB which ranges from 40 to 100 and has an Auto setting of 80. An Auto setting would be a good place to start, getting this wrong can result in USB bottlenecks - seen as dropped frames.
  • For a given exposure/gain/bit-depth, start with the largest Capture Area and gradually reduce the area. Record the fps shown at the bottom left in Sharpcap. Record the values. Reconnecting the camera will reset the dropped frame count.
  • Repeat the previous step but instead capture a set number of frames to disk. Record the fps shown during capture. Keep an eye on the Performance of CPU, memory, disk.
  • Try adjusting the USB settings for a given exposure/gain/bit-depth/capture area and watch the dropped frame count. This should help ascertain optimum settings - Auto has worked well from with the ASI120.
Post back your findings, include screen shots for USBtreeView and the Camera USB controls.

Don't forget these tests can all be carried out indoors, during the day, on a bench - no need for telescopes and clear nights.

This Astrobin search https://www.astrobin.com/search/?q=asi183MM is useful. Have a look at the lunar images and resolution sizes. I will leave you to draw your own conclusions.

Dave

User avatar
admin
Site Admin
Posts: 3445
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: How to defeat the bottleneck?

Post by admin » Sun Dec 22, 2019 3:31 pm

Hi,

The fact that you are only dropping frames during capture pretty much nails the cause as being something to do with disk write performance – if you are having USB issues then be dropping frames or getting slow frame rates when capturing wasn't active.

One thing to check is that you have lots of free space on your SSD device and that you have trimmed the device recently to ensure that all free sectors have been wiped clean ready for writing. In Windows you can do this by going to the place where you would defragment a hard disk and choosing to optimise the SSD drive.

Cheers, Robin

gfstallin
Posts: 2
Joined: Sat Dec 21, 2019 11:19 am

Re: How to defeat the bottleneck?

Post by gfstallin » Mon Dec 23, 2019 7:07 am

Dave and Robin,

Thank you both for your input. My new hard drive came in before your replies. I had needed more space. My laptop can accommodate two hard drives. Both drives were 256 GB SSDs. I bumped up the second drive to a 1 TB SSD. I just installed it this morning, after installing USB Tree View in an attempt to get to the root of the problem.

Bottom Line Up Front: the problem appears to be SOLVED.

Before doing anything, I did a "bench" test to see what the new drive was capable of doing using SharpCap's test feature (brilliant addition). Using the test function within SharpCap and keeping all parameters the same (gain, exposure), I saw that the test 6 GB file was writing 13% faster on average (an average of 6 tests) than either my old second internal drive or the C drive original to my laptop.

Here is a screenshot of performance with the old second internal drive:
Frames Captured on internal drive.jpg
Frames Captured on internal drive.jpg (184.97 KiB) Viewed 528 times
Both the SSD drive original to the computer, and the second drive SSD I had installed about a year ago had mostly the same performance. By around 1000 frames, I had run out of buffers and I would start dropping frames and fps would decrease dramatically.

Here is the performance with the new 1 TB SSD that I installed yesterday:
Frames Captured on new drive.jpg
Frames Captured on new drive.jpg (215.73 KiB) Viewed 528 times
WHOA! Through 1400 frames I had captured all of them. The 3 frames in buffer actually represents an anomaly. This was the only capture of six with more than 1000 frames that I had any frames in buffer on that new 1 TB SSD. Robin's assessment appears to have been correct. The drive that came with this computer, and the original 256 GB secondary drive I had installed on my laptop, with their 13% slower write speed on average, made the difference between dropping frames and not dropping frames in my case. Problem SOLVED!

Still, this problem was resolved by luck. Following these results, I did some research to see where I went wrong in having chosen the Samsung 850 256 GB SSD drive I just replaced. I think I've seen enough to tell me I wasn't going to be able to choose the right drive for this application no matter how much research I had done. The Samsung EVO 850 - which was solely dedicated to imaging and typically only had about 15 GB on it at any time - was perfectly adequate for imaging with the ASI224MC. It generated decent reviews by most users. Looking at some benchmark websites, it appears one of its weaknesses is write speed, something I hadn't seen at the time I bought it. The problem is, the drive I replaced it with, a Crucial MX500 1 TB drive, also has write speed listed as one of its weaknesses. At least one website rates the Crucial significantly faster than the Samsung 850, but other websites offer conflicting information. To me, this suggests there is likely a lot of variability in the speed of individual drive units regardless of brand and model (not a surprise) and that these cameras are pushing the limits of hard drives in general. Even if my own bench results hold up after testing the new Crucial MX500 drive in real-world imaging conditions, my own outcome might not be typical. With the variability I saw in some online benchmark results for my old Samsung 850 SSD, some samples appear to perform just as well as or better than the Crucial MX500.

Robin, I also want to thank you for your work on SharpCap. I'm leaning on it a little more than I had planned when I first subscribed. While the planets are mostly away, I've been thinking about EAA, which is why I even bought the Pro license in the first place. It is a great program and a real service to imagers everywhere, and kudos for the low subscription costs. Given the nature of our intellectual endeavor (not that many people do this), I suspect you aren't flying around on private jets with all of that SharpCap revenue coming in, but maybe you should be. :lol:

George

User avatar
turfpit
Posts: 924
Joined: Mon Feb 13, 2017 8:13 pm
Location: UK
Contact:

Re: How to defeat the bottleneck?

Post by turfpit » Mon Dec 23, 2019 1:53 pm

George

Good to hear you got sorted. If I carry out an hour of lunar imaging with a 1280x960 camera I expect to have ~100Gb of data. Storage does become a challenge.

Dave

User avatar
admin
Site Admin
Posts: 3445
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: How to defeat the bottleneck?

Post by admin » Tue Dec 24, 2019 1:55 pm

Hi,

Glad to hear you got it sorted. No private jets yet sadly, but thanks for the support and hope you continue to enjoy using the software.


Cheers, Robin

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest