Index number bug in sequencing

Discussions, Bug Reports and Issues related to Beta versions of SharpCap
Post Reply
zerolatitude
Posts: 156
Joined: Mon Mar 01, 2021 5:24 am

Index number bug in sequencing

#1

Post by zerolatitude »

Hi,

Normally in a capture there is a new subfolder for each time the loop starts, named as the time of start.

Since I was doing multiple loops, I removed the time subfolder to make it easier to process. The intention was to have all the files in 1 folder.

The file naming example is F:\AA_Captures\M42\2021-04-08\M42_OIII_2.5s_100_25_00001.png

The sequence loop is capture 20 live frames, resolve, and repeat 6 times. I expected to get one folder with files indexed from 1 to 120.

For the first loop, the files got indexed 1 to 20.

However, when a new loop started, the index is not incremented but starts again from 1. So for 3 loops, the same file gets overwritten 3 times.

Can you correct this on priority please? if there is no subfolder for the new time of start, the index should be incremented from the last number in that folder, and not be reset.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Index number bug in sequencing

#2

Post by admin »

Hi,

I'm more inclined to make it illegal (or at least have a warning) if you set a filename template that doesn't include a {Time} or {FrameTime} or {DateTime} somewhere - it's likely to cause file overwrite issues somewhere along the way. Even if you start the index from the existing number of frames in the folder then the problem isn't solved for video captures (which don't have an index in the file name).

Could you put the {Time} in the filename rather than the directory name? That will keep all the files in one directory and avoid the overwrite possibility. Alternatively you could use {FrameTime} or the 'Arrange files and folders to keep sequence captures together' step at the start of your sequence - that sets things up so each capture has its own subfolder.

cheers,

Robin
zerolatitude
Posts: 156
Joined: Mon Mar 01, 2021 5:24 am

Re: Index number bug in sequencing

#3

Post by zerolatitude »

Okay will do that.

I would still prefer index as it is easier to keep track of the number of files, and also to remember where you were if you do only some of them and then come back.

But can live with your suggestion.
kencraw
Posts: 4
Joined: Wed Oct 26, 2022 9:16 pm

Re: Index number bug in sequencing

#4

Post by kencraw »

I just ran into this bug last night and I'm *VERY* frustrated about it. So frustrated I'm afraid that this comment isn't going to come off appropriately charitable enough for the authors of this software. So let me start with a general statement that I'm very thankful for SharpCap and find the pricing for SharpCap Pro very reasonable. I also get that this is probably not a full-time job sort of software due to the limited size of the astronomy hobby and thus we should have reasonable expectations. It's more charity towards us, the users, and a vocation of love for the hobby than we the users tend to know.

So please keep the above in mind (that my heart is generally thankful) when I say...

This is *COMPLETELY* unacceptable. Destroying data is something that should be avoided at all costs. There are a numerous solutions to the "uh-oh, the index I'm intending to use already exists" and over-writing the data is the *LEAST* acceptable answer. Scratch that. It's *NEVER* an acceptable answer.

Obviously the best solutions involve a logic around integrating the existing indexes into the chosen indexes to use (a common one is find the highest index in the directory and start one above that but skipping over also works). But there are lots of other options: Append a 2nd index. Heck, even just stopping the session and throwing up an error message would be better than *DESTROYING DATA*! (Frankly, you should do that no matter what, if a file already exists when you open a file to write, you should either pick some other name or stop the sequence and display an error.)

I'm sure you can feel the frustration in my text. I suspect you can well imaging how frustrating it is to take 8 series of astronomy images over the course of many hours in the cold at a remote site in the middle of nowhere and for only one of those sets to be on your flash drive when you get home. Even if that specific problem hasn't happened to you you probably get it because the nature of astronomy is that we make lots of mistakes where imaging doesn't come out as expected and we don't realize it until we get home. It's frustrating, right?

So please, please, *PLEASE* think of that sort of frustration before you so casually dismiss the nature of this problem as if it is no big deal. Seriously, I'm sure you have a bug list you work and this needs to be on the URGENT list, not on the "when I get around to it" list. I can see why it might seem like it's on the lower end of the priority list, I really do, it seems like an odd corner case. And frankly it is. But *DESTROYING DATA* needs to be taken very seriously even if it's a rarely exposed bug.

So, now that I've allowed myself the cathartic release of venting about the lost data, I thank you for your software and I humbly ask that you consider fixing this problem in the near future so as not to ruin even one more night of astrophotography.

(And yes, I've now applied the fix you've suggested - from now on the data/time will always be in my filename. But there's some other poor sap out there like me who doesn't know that yet. Be kind to him and save him from my fate by fixing this bug.)
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Index number bug in sequencing

#5

Post by admin »

Hi,

thanks for posting in such a restrained manner, and sorry to hear about the lost data. It's also annoying from my point of view because one of the reasons that I started writing SharpCap was because AmCap (which was pretty much the only choice for webcam imaging back in the day) was very prone to overwrite files by mistake and I did it several times myself.

I presume that you are using manual filename templates, since I don't think its possible for this to happen with the filenames generated using the 'simple filename options'. With manual templates it is supposed to warn you in red that the options you have chosen may give you this problem, but having just reviewed the code, I think the logic for that warning on the 'sequence of still' image template is insufficient - just having {Index} in the template is insufficient unless you have {Time} or {DateTime} elsewhere.

I will update the warning code to deal with that situation (and also check to see if any of the live stack templates are affected). If you could share the templates that you are using with me, it would let me check that I have correctly detected the situation that you found yourself in.

thanks,

Robin
kencraw
Posts: 4
Joined: Wed Oct 26, 2022 9:16 pm

Re: Index number bug in sequencing

#6

Post by kencraw »

Thanks Robin. For clarity, my version is 4.0.9258.0 - 8/29/22.

My "Sequence (Stills)" Manual template was: {TargetName}\{FrameType}-{Filter}-{Index}

Yes, I'm using the "Edit Filename Templates Manually" checkbox

I re-created those settings and here's no warning of that being a problem. However, I started fiddling around, and if I changed the "Single File (Video)" template to remove the date and time I got a warning "Video Template must contain {Time} or {DateTime}". But once I fix that one, the "Sequence (Stills)" box is still without any date and/or time in the template and there's no warning. So my experience aligns with your analysis of the relevant code.

So making sure there's a warning for all 5 templates would be a good first step.

But may I also suggest also doing one of the following:
1. Disable the "OK" and "Apply" buttons when those warnings are listed
2. Always stop the sequence and display an error before overwriting a file
3. Add a checkbox (unchecked by default) titled "Allow overwriting of files" and either do #1 (disable buttons) or #2 (stop sequence and display error) when it is unchecked

Thanks again for being willing to add a fix.
kencraw
Posts: 4
Joined: Wed Oct 26, 2022 9:16 pm

Re: Index number bug in sequencing

#7

Post by kencraw »

Sorry, I should have included this in the previous reply:

I really think you should consider an index logic that won't overwrite files. Probably the simplest that will always work is doing a "File Exists" check before opening a file to write it and if the index already exists increment the index and try again (and thus repeat the check and thus looping through many indexes when a long sequence of indexes already exists). And the great thing about that is that (particularly if you'll add an index even if one isn't in the template) is that just doing that alone will fix *ALL* of these sorts of overwriting data problems in one shot. It will overall simplify your code by putting the logic in one place (where you open a file for writing).

Because honestly, I don't *want* a timestamp in my filename. That data is stored already in the filesystem. An index should be enough to prevent a conflict. Tons of cameras (which don't have a lot of computing horsepower, so it can't be hard) have an index-conflict avoidance logic built in. I know it may be a bit more work, but I think it's worth the effort and will save you from a lot of the other work/ideas I provided.

Thanks again for taking the time to reply and for being willing to consider my suggestions.
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Index number bug in sequencing

#8

Post by admin »

Hi,

I'm afraid I'm not convinced by the idea of just incrementing an index. All sorts of questions are missing answers in that approach (when do you reset the index, what if you have two copies of SharpCap running, do you fill in blanks if number 343 has been deleted, etc).

I do now have a more robust approach to validating the filename templates that checks a range of possibilities to see if any bring up duplicated filenames (different times on the same day, same times on different days, etc). Also the settings will not let you save changes if the possibility of duplicates exist.

I have also added a check at the point of creating a new output file - if the file already exists then SharpCap will give an error and stop capturing rather than overwrite the file (exception - zero byte files are allowed to be overwritten). I may have to make this optional, as it usually turns out that what is a bug for one person is a feature (and part of their everyday workflow) for someone else...

cheers,

Robin
kencraw
Posts: 4
Joined: Wed Oct 26, 2022 9:16 pm

Re: Index number bug in sequencing

#9

Post by kencraw »

I'm glad/thankful you've put in the creating output file check so you don't overwrite files. That's probably the most important change. And yeah, making it optional (one would assume a checkbox in the settings) would give the most flexibility and give users who (for some odd reason) like the existing overwriting behavior a way to maintain it.

I'm also glad/thankful for the additional warnings/checks/blocks you've implemented in the file template settings.

Thank you! (For both)

Regarding the index, while I could provide answers to all those questions (and I might at the end of this post), I think you're overthinking it. The way I see it, these indexes are just ID's. They're an arbitrary reference to separate one item from another one. While "true" indexes are logically ordered, it's not a requirement for ID's. (By way of example: databases almost never have well ordered ID's, even though every entry uses it as it's primary reference.) While I'd concede/agree that in an ideal world the ID's/index for SharpCap are well ordered, it's OK to have a situation where a poorly setup filename template and/or poorly maintained directory structure result in ID's that are not well ordered. And since many of the situations you've listed are ones that bring into question the precision/competence of the user... (why would they have gaps in their indexes? Why would they have two instances of SharpCap sending the exact same content (at least as far as the templating is concerned) to the same location?)... I think it's OK for the result to be somewhat jumbled when they do something odd/foolish. At best, they're smart and they'll detangle their self-created mess and put processes in place to prevent it in the future. At worst, it'll remain a jumbled mess.

So don't over think it. Just make sure they don't collide. With that in mind here's how the logic I provided would work:

1. When do you reset the index? The same as you do now, with every new run. My logic would mean that when you do multiple runs it will pick up where it left off as it will attempt to start with 1, see it already exists and then increment through all the existing numbers until it finally reaches the end of the preceding run, and then the first actually written file will be the next higher number.

2. What if you have two copies of SharpCap running? Just make sure the opening of the file for writing is atomic with the check of whether it exists and the multiple instances of SharpCap will automatically do some sort of index interleaving as they see the numbers that they were intending to use are taken when they go to open for writing.

3. Do you fill in blanks if number 343 has been deleted? The logic I suggested would have it fill in the blanks if those blanks were there when the run was started. If someone deleted files mid-run, then no, it wouldn't go back and re-fill in the earlier ones. However, the subsequent run would then start with where ever that hole was created and backfill it. That would also mean that later runs could have indexes before previous ones. I concede that results in a jumbled mess, but as I said, the user who left a directory in that state (with holes in their list of indexed files) is begging for that sort of mess.

But for people who well order maintain their directories the above logic works out quite well, don't you think?


A final question for you: How would one know what version of SharpCap I would need to wait for/download for a particular fix to be in it?
User avatar
admin
Site Admin
Posts: 13177
Joined: Sat Feb 11, 2017 3:52 pm
Location: Vale of the White Horse, UK
Contact:

Re: Index number bug in sequencing

#10

Post by admin »

Hi,

thanks for the additional explaination on the indexing - I will think about it a bit more (and also see if the idea picks up any additional support from other users).

The updated version of SharpCap will be available next Monday.

cheers,

Robin
Post Reply