A Script to solve a Workflow Problem

I have a problem in my photo workflow for quite some time, and finally I created a solution in the form of a script I wrote.
Maybe you have a similar problem, and you also work with Linux, then I may have a possible solution below.

However, even if you are a Windows user, scripts are still useful, and who doesn’t know the old DOS box.
You need basic computer knowledge, but you don’t need to be a programmer to use scripts.
And whether for Linux or Windows, Google is your best friend !
Just type your OS system, type in what you want to do, command line or terminal/console – you’ll be surprised how much you can still find about the old Dos or Shell Scripts.

The problem is as follows : I always shoot jpg + raw.
When I go through my photos on my computer the first time, my photo viewer is slow because it also has to open the raws, so I put all raws immediately after download in a folder that I create named “RAW”.
Then I view and clean up the jpgs almost immediately after – so all less good and bad jpgs are removed.
However, I still have all raws in my “RAW” folder,
both good and bad.
In the main folder I also create another subfolder named “EDIT”,
in this I put an edited jpg copy of my very best photos that I want to publish at some point, this copy can be an edit of a sooc jpg or raw, but in most cases only resized for web – the file name remains unchanged – only extra edits of the same jpg get renamed.

Now, what I want is : I want to keep all good to best jpgs,
and only from the very best (jpg or raw) of which I have edited and placed a jpg in the “EDIT” folder,
of that I want to keep the raw file,
all other raws will be removed.

Situation, description and solution :

First step – I copy all my photos from SD card to my computer with Rapid Photo Downloader – RPD creates a folder with file name “year month date”, and copies all files (jpg +raw) into this folder.

Step two – I create a subfolder from the start in that new folder “y m d” called “RAW” in which I place all raws – now my viewer has no problems anymore !
While viewing I delete all bad jpgs from the main folder.
I also create a second folder named “EDIT”,
in this folder I create a jpg copy of my very best jpgs or raws, that I want to publish at some point.



Step three – After fully processing a photo folder I would like to delete all raws – except ! all raws of which have been in an edit, or of which a jpg is in the “EDIT” folder, I want to keep !

For this I start a script which I named “RawCopier”.
It will first look in the “Edit” folder, remember all the file names of the jpgs that can be found there.
Then it will search for the same filename in the “RAW” folder with the Olympus .orf extension.
Then the script copies these files to the “EDIT” folder for when I later want to create another version and need the raw from these photos.


Result : In the “EDIT” folder I now have a raw file for every jpg it contains. It is safe to delete the “RAW” folder now.

Below is the text that makes up the script, you can adapt it to your own situation, other file extensions and/or other folders.

for f in /home/marc/Afbeeldingen/2022/20221015/Edit/.jpg; do filename=$(echo $f | sed -e ‘s/.\///’ -e ‘s/..*//’); cp -r /home/marc/Afbeeldingen/2022/20221015/Raw/${filename}.orf /home/marc/Afbeeldingen/2022/20221015/Edit/; done

As a script editor I use “Kate”, it’s simple to use and is standard in Kubuntu 22.04 LTS

Give the script a name and save the text file with the .sh extension,
in this case “RawCopier.sh”, in the “root” of the folder where you want it to run.
To make the script executable, open a terminal such as “Konsole” in the folder where you have the script.
Type : chmod +x followed by the exact full name of your script with extension:
chmod +x RawCopier.sh
Enter
Your Script is then executable.

If you want to run this script in a different or new folder – put a copy of the script in the “root” of that folder, open the script in Kate, and change the path names in the script to the current situation.

You can also use another folder to copy the Raws to instead of the Edit folder,
in the script just change the “copy to pad” in the last bit of the script (Orf) – this way it’s easier to check if you indeed have all the raws you need in the “orf” folder, so you can delete the “raw” folder.
The names I used are just an example, you can rename them in the script to whatever you want – just remember the correct order : first path is “lookup” folder, second is “copy from” folder, third is “copy to” folder.

for f in /home/marc/Afbeeldingen/2022/20221015/Edit/.jpg; do filename=$(echo $f | sed -e ‘s/.\///’ -e ‘s/..*//’); cp -r /home/marc/Afbeeldingen/2022/20221015/Raw/${filename}.orf /home/marc/Afbeeldingen/2022/20221015/Orf/; done

I found this very similar script at “https://www.unix.com/shell-programming-and-scripting/260878-copy-files-different-extensions-same-name.html” but this script didn’t work !
I edited it, changed a few things, and made it usable – You can easily adapt the script to a different situation.

La Gallerie

The photo gallery of Open Source Photography, Olympus micro 4/3 system, Vintage Lens Photograpy, Film Simulation, PictureFX, HDR – Photographer : Marc R.


Discover more from Open Source Photography

Subscribe to get the latest posts sent to your email.

Type your email…

9 thoughts on “A Script to solve a Workflow Problem

Add yours

  1. Zeker een handig script hoor en leuk dat je het beschikbaar stelt. Zelf gebruik ik Photoshop, die opent meteen RAW. Daar overloop ik meteen de RAW files (in series van bv. 5 of 10 stuks). De goeie worden meteen bewerkt en worden bewaard in hoogste resolutie jpeg files. Nadien open ik alle overgehouden high res. jpeg files, in Photoshop, pas ik de resolutie en de afmetingen aan, voeg er een logo aan toe en safe ze onder “web resolutie” in een afzonderlijke folder (die ik dan gebruik als beelden voor mijn blog). RAW files hou ik niet bij omdat ze massa’s ruimte in beslag nemen. Toegegeven dit is een keuze die je wel of niet wil maken.

    Like

    1. Bedankt Rudi !
      Wel ook in mijn software (rawtherapee-darktable-digikam) kan ik Raw en Jpg direct samen openen en verwerken,
      het probleem is dat ik nog steeds op een portable van 12 jaar geleden werk, een oude Dell Studio 1749.
      En ook al gaat dat perfect, het gaat traag – te traag zelfs voor mijn bescheiden 61 jarige leeftijd.
      Maar, rekeninghoudend met het milieu, het budget, de vrouw en nog andere beperkende maatregelen 🙂
      wil ik er geen afscheid van nemen.
      Ik heb hem opgevoerd tot het uiterste, en hij zal – als murphy het toelaat – nog wat jaren moeten volhouden.
      Dat is ook de reden dat ik alleen jpg’s bekijk bij het sorteren, die gaan een pak sneller,
      en dus zoek ik naar manieren om met minder rekenkracht meer te kunnen doen.
      In dat plaatje past Linux ook perfect, want het is veel minder veeleisend, en daardoor blijft mijn portable in leven.

      Liked by 1 person

  2. I randomly stumbled on your website when querying fujifilm + darktable. Your content looks interesting so I will have to check more of it out. Also, I like your script, coincidentally I wrote a little script yesterday to accomplish a similar task. I put my raw files in a base directory, within that a folder ending in “sooc” for jpg’s. I use darktable to prune the raws – but manually doing the same for sooc jpgs seemed overkill and a job for a script. I’ll attach a gist in case anyone stumbling here may have a similar workflow. Although its a simple script, It has not been tested much, so I would backup prior to testing.


    #!/usr/bin/sh
    # small script to prune extreneous sooc jpg's – we want raw_dir == jpg_dir
    # takes one arg: full path to raw_dir
    # requires 'trash-cli' package: https://github.com/andreafrancia/trash-cli
    RAW_DIR="${1}" # user passes raw directory path
    JPG_DIR=$(find "$RAW_DIR" -mindepth 1 -maxdepth 1 -name "*sooc" -type d) # only look dirs ending in "sooc"
    # get all files matching .{ext}, return only the basename
    # modified from: https://stackoverflow.com/questions/26935515/in-linux-how-to-compare-two-directories-by-filename-only-and-get-list-of-result
    getfiles() {
    find "$1" -maxdepth 1 -type f -regextype posix-extended -regex ".*\.(RAF|JPG)" \
    -exec bash -c 'for f in "$@"; do basename "${f%.*}"; done' "" {} + | sort;
    }
    # diff raw_dir -> jpg_dir. Compare basename only (minus ext)
    # returns files that exist in jpg_dir and not raw_dir
    diff=( $(grep -vxFf <(getfiles "$RAW_DIR") <(getfiles "$JPG_DIR")) )
    # stdout results of diff. Add results to "dirty" array
    dirty=()
    for filename in ${diff[@]}; do
    filepath="${JPG_DIR}/${filename}.JPG"
    dirty+=( "${filepath}" )
    echo "$filepath"
    done
    # ask before trashing extreneous sooc jpg's
    printf "> \e[31mTrash\e[39m the above \033[1m%s\033[0m jpg's from \e[33m%s\e[39m (Y/n) " ${#diff[@]} "$(basename "${JPG_DIR}")"
    read answer
    if [ "$answer" != "${answer#[Yy]}" ] ;then
    for filename in ${diff[@]}; do
    printf '>> \e[31mTrashing\e[39m \e[33m%s\e[39m\n' "${filename}"
    done
    trash "${dirty[@]}" # for extra safety, use trash-cli rather than rm
    printf '>> \e[31mTrashed\e[39m \033[1m%s\033[0m files\n' ${#diff[@]}
    fi

    view raw

    raw_diff.sh

    hosted with ❤ by GitHub

    Liked by 1 person

    1. Hello Justin,
      In the meantime I made a very small change to my script so that the script asks which folder to handle – that way I could use one script in the root of my photo folders.
      Well indeed, the essence of my script is to make a task easier, and that is exactly what you do in your script.
      I’ve seen the script, and it looks good – and the steps are well described too.
      Thanks for responding and for sharing your script !
      If you find my site interesting, just follow it please, the more followers, the more motivation to keep going!
      Marc.

      Like

      1. Followed! I have been using linux for years but only recently switched to a open source photography workflow. Your blog looks like a valuable resource in that regard so I’ll gladly subscribe.

        Justin

        Liked by 1 person

Leave a reply to mijnnikonenik Cancel reply

Website Powered by WordPress.com.

Up ↑

Discover more from Open Source Photography

Subscribe now to keep reading and get access to the full archive.

Type your email…

Continue reading