PC Development

Back to Topic OverviewPost New Reply

Manage mode / Batch Set - very slow

(16 posts)

  • John Radcliffe
    Member

    Why is Batch Set so slow when copying metadata back from the image files to the database?

    I hadn't realised that this vast speed decrease had occurred with v2.5 compared with v2, so I tested on a folder. In each case I give two timings. For A I cleared all the Captions and Notes fields and then ran a template in Batch Set to copy them back. For B I just ran the same template again, but this time all it should have to do is check that the contents are the same.

    Pro2 .........A= 2sec / B= under 1sec
    Pro2.5 ..... A= 18sec / B= 18sec
    Pro3 ........ A= 55sec / B= 20sec

    These are massive increases, so what on earth are v2.5 and v3Beta doing that takes all that time?

    Or put another way, what changed between v2 and v2.5? I can think of 3 changes but can't understand why any of them should have had an impact on this exercise:

    (1) The ability to embed ACDSee data in the image files. I'm not using this, and the image files appeared to be unchanged afterwards, so I don't see why this should be relevant.

    (2) The "Originals" feature, but that should have no relevance either.

    (3) Unicode support for data fields, but I fail to see why this should slow things down as much as this (if at all).

    Does anyone have a suggestion? Does someone from the Development team have the answer? Is there anything I can do to speed it up?

    Posted On June 22, 2009 - 03:12 PM (5 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    I hadn't noticed that, but you're right, and those are excellent guesses as to what might have triggered this, although I wouldn't think any of those features *should* have caused this.  FWIW, though, I *have* noticed a possibly related thing: batch set from db *to* a set of RAW files also got much slower with 2.5 for RAW files (and remains so for 3.0).  Odd, since all it is doing is writing a small text file (XMP).  Similar numbers to what you're seeing - 10-100 times slower.

    FWIW, the embed operation is as fast as batch set used to be, so I know it's *possible* to write those XMP files quickly.

    Posted On June 22, 2009 - 06:14 PM (5 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Thanks Marc,

    I thought later that I should have Optimized the database first, so I repeated the test for v3Beta after doing so. It considerably improved the time, down to 35sec from 55! However, Optimized or not, v2 just flies in comparison.

    I haven't compared write times yet (database to IPTC fields), but I guess I should!

    Posted On June 23, 2009 - 01:40 PM (5 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Here are some comparative write times (i.e. when copying data from the database into IPTC fields in the image files). Please note that these are for only 50 images, not the full 350 in the previous exercise, so it's not actually writing quicker than reading.

    Pro2 = 25sec / 0sec
    Pro2.5 = 25sec / 2sec
    Pro3 = 35sec / 3sec

    Here v2.5 is just as quick as v2, except where no updating is required. Beta v3 takes about 40% longer than v2.5 in each case.

    These figures aren't exactly impressive, so would anyone from the Development team care to comment? It may be that I'm missing some factor out of the equation, and that the later versions aren't just dragging their feet.

    Posted On June 24, 2009 - 01:11 PM (5 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    Are those for RAW?  I am seeing literally a 10-100X slowdown in IPTC write time from 2 to 2.5 (and 3.0 is as bad as 2.5), but only for RAW.  That seems to be something different than the "slight" slowdown you are seeing.

    Posted On July 1, 2009 - 05:41 PM (4 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Sorry, Marc, I should have made that clear. My files are all Jpgs (so the data is being written into the image file itself).

    I was hoping that we would have got some response from the Development Team, as this is not the sort of change I expect (or want!) to see between versions, unless there is a very good reason that we are overlooking.

    As this issue also affects v2.5, I have submitted a support ticket for v2.5.

    Posted On July 2, 2009 - 12:56 PM (4 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Well, it's now 9 days since I submitted a support ticket (for v2.5, but with a link to this thread). Since receiving the auto-response acknowledging receipt I've heard nothing. So I guess either the developers aren't worried by a 20 times slowdown, or they know what the problem is but don't want to admit it.

    Personally, I don't find either approach particularly helpful (or, indeed, courteous).

    Is the slowdown significant? To me it is. With v2 I could simply select all the images in a folder and run Batch Set to copy my data back from the image fields. If only a few needed updating then, even if the folder contained around 1000 image files, that process might only take a 2 or 3 seconds. However, with v2.5 or v3 Beta I'm now faced with perhaps a minute's wait (or I have to just select the specific images first, if I know which they are).

    Posted On July 13, 2009 - 01:06 PM (4 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Well, I've had a response from Support, although I have to say that I find it disappointing.

    While ACDSee Pro 2 was slightly quicker than Pro 2.5 and Pro 3, Pro 2 was not performing all the functions that later versions are. This more complete processing provides users with greater stability but does result in fractionally longer wait time.

    Now I've no idea what else what else is being done (and how this is of benefit), but I have to disagree with the idea that the results I note above involve a "fractionally longer wait time".

    So, if the developers consider such an increase to be reasonable, I guess I can't expect a return to v2 levels of performance any time soon -- unless I go back to using v2 for image management and viewing, and either forget v3 completely or just use it for editing.

    Posted On July 17, 2009 - 03:29 PM (4 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    That would be disappointing.  But I've got an alternate read of the situation, not based on any inside knowledge of this particular issue or ACD, but of the software world in general.  My guess is that the response you received was from someone in support who may or may not have had any contact whatsoever with a developer before responding, and even if it was a developer, it might not be one affiliated with this particular tool, or even the Pro 3 project.  I'd further guess that the people responsbile for this tool on pro 3 *will* see this thread and, independently of what someone in support said about 2.5, may well look into it.  I guess we'll see.  I too am concerned about this slowdown, which I agree is not "slight" at all, but a factor of 10 or more (that's what, *1000%*?) when it comes to RAW files.

    Posted On July 22, 2009 - 08:48 PM (4 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Well I sure hope you're right. In some cases I'm getting times of 2 or 3 seconds for v2 and over 3 minutes for v3 (with v2.5 near to the latter), which simply ridiculous.

    In retrospect I wondered why I hadn't noticed this slowdown before, as I've been using v2.5 for some time. Then it occurred that it's because 90% of the time I'm reading the metadata back (the case that for me shows the biggest slowdown) on the computer I'm using at the moment. Because that's running Win2k, it still has Pro v2 on it.

    Posted On July 23, 2009 - 02:05 PM (4 months ago) (Permalink to this post)

  • Melanie Wood
    Community Manager

    Hello,

    While I recognize the times you have provided are longer, the durations we have been able to reproduce are not as long. The statement, based on our testing, is true. ACDSee Pro 2 was slightly quicker than Pro 2.5 and Pro 3 but it was not performing all the same functions. I would recommend replying to the ticket, ensuring you provide CS with all pertinent information, not just a link to a forum thread.

    When responding to users questions, CS endeavors to give the highest quality of service. There is consultation prior to responding to new issues and revision of old responses on a regular basis.

    -Melanie

    Posted On July 23, 2009 - 11:12 PM (4 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    Just to be clear, it wasn't my intent to criticize support.  As some here may know, I used to be a software engineer, and in my world, we considered it a *good* thing when support was able to answer questions without consulting us.  And given the number of engineers who may have worked on the same section of code for different releases over the years, it was by no means a given that any one person would be able to provide the full history of any given feature from release to release.  So I didn't mean it as a bad thing when I suggested support may have answered without having the fully story direct from the source - in my experience, it's practically inevitable that this will happen at times.

    Anyhow, this answer *does* seem basically sound to me - it just seems like it might not tell the whole story.  I've never measured the difference between 2.0 and later versions for JPEG, so I can't comment on how "slight" it might be - whether John's measurements are abnormal or not.  But for RAW, I can absolutely verify that IPTC write times have slowed down *enormously* - the same kind of factor of ten differences John refers to.  If ACD needs help reproducing this, I'm happy to do what I can.

    As a point of reference, the embed database operation runs in a fraction of a second per image, and hundred of images can be processed in just seconds.  That's the kind of speed that *used* to happen for propreitary RAW files when writing to IPTC, but now these files are as slow as DNG  which is to say, around 10 times slower.  So I do hope someone looks into that asect of the issue rather than considering the matter closed.

    Posted On July 24, 2009 - 12:09 AM (4 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    Hi Melanie,

    The reason I provided a link to this thread was that I thought this was the best way of providing as much information AND background as possible. From Ben's response (which I did acknowledge) it would appear that the information in this thread had indeed been read and taken into account.

    What I may NOT have made completely clear is that my "worst case" scenario involves processing a lot of image files where NO data actually needs to be copied back from the files to the database. In this case v2 can complete 1000 files in under a second, whereas v3 can take 180 seconds. It is obviously doing much more than just checking that the data IS the same: I just wonder whether really it needs to do this for the images for which no change is necessary (or whether it could do it optionally).

    The fact that I frequently run Batch Set in this way on entire folders, just to ensure that the data for a handful of images is updated, can be put down to my laziness. It's just so much easier to do that than to have to sort by Modified date, select the most recent files, run Batch Set on just those files, and then re-sort by Filename.

    So I blame the impressive speed of v2 for encouraging me to be lazy! It's an ease-of-use feature that I'd hate to lose.

    Posted On July 24, 2009 - 01:33 PM (4 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    John Radcliffe said:

    What I may NOT have made completely clear is that my "worst case" scenario involves processing a lot of image files where NO data actually needs to be copied back from the files to the database. In this case v2 can complete 1000 files in under a second, whereas v3 can take 180 seconds.

    Ah, that sounds like good info, and might explain why I hadn't noticed a big change with respect to JPEG.  Another thing to consider that I remember from testing this on previous versions - if you write certain fields like Keywords, it seems Batch Set Information (BSI) will write all files regardless of whether anything has changed.  If the problem fields like Keywords are *not* part of your template, then BSI can be a lot smarter about only updating files as necessary.  So do be sure the templates you are comparing both include Keywords (or both don't include them) to be sure you are comparing fairly.  Either way, BSI should be telling you how many files were updated - is the difference in speed is because in one version nothing is being udpated and in the other everything is, or just that updating nothing takes longer in 2.5 and 3.0?

    In any case, once again, I want to emphasize that what I am seeing is, as far as I know, something rather different, since I am shooting RAW, and am definitely talking about cases where everything is updated.  With RAW (proprietary - PEF - as opposed to DNG), the info is written to a small sidecar text file and *should* be incredibly fast.  That is how I remember it being with 2.0, and the Embed Database Information (EDI) facility of 2.5 and 3.0 is still that fast.  As in, hundreds of files in a matter of seconds.  But BSI, which in theory is doing no more work than EDI, is as I keep saying at least 10, probably more like 100 times slower than EDI for these proprietary RAW files - several *seconds per file* as opposed to dozens of *files per second*.  I can see no reason for that.

    Posted On July 27, 2009 - 07:07 PM (3 months ago) (Permalink to this post)

  • John Radcliffe
    Member

    One of the reasons I don't use the IPTC Keywords and Categories fields is the discovery that ACDSee always writes to these, even if the data it is writing is identical to what is already there. I keep to fields that ACDSee only writes to if the data has changed, so that it only updates (and I only have to back up) image files that needed to be changed, not every one in the batch. (As I'm the only person who accesses this info I don't find my non-standard usage an issue.)

    Of course when writing to the image files it's easy to see which files (if any) have been updated by checking the modified dates, but it's not so easy to see what ACDSee is doing when reading back to the database. However, judging by the time taken and by how the progress bar slows down or speeds up, it would seem that ACDSee s doing more work when more data has changed. (Even though I find v2.5 and v3 take longer to write to the image files, I have checked, and they aren't in fact updating more files than v2 did.)

    Where are you seeing reported how many images have been written to? I can't recall ever having seen this.

    Posted On July 28, 2009 - 02:13 PM (3 months ago) (Permalink to this post)

  • Marc Sabatella
    Moderator

    John Radcliffe said:

    Where are you seeing reported how many images have been written to? I can't recall ever having seen this.

    Good question.  Don't know where I got the idea there was count present.  I think when I was feeling my way through all this original, I probably just eyeballed it based on how long it took - if the progressi indicator blows right by a file, I know it didn't write, but if it takes several seconds, I know it's doing something.  That and/or I've looked at the file-modified dates.  Anyhow, it was easy enough to tell that writing Keywords forced it to write everything, but without that, it could be very quick and skip the fiule that hadn't changed.  I suppose you could use this same idea to get a handle on what's going on in your case.

    Posted On July 30, 2009 - 03:53 AM (3 months ago) (Permalink to this post)

Subscribe to this topic via RSS

Reply

You must log in to post.