One day bash crash course: Functions that return, using echos with newlines in those functions, and converting *nix paths to Windows

I’m in the middle of writing a long, multi-part tutorial on Django authentication. Each part is a completely separate installation of Django, with its own virtualenv and Git repository. The only exception is the singular database, which all parts use (only one is started at a time, and I’m the only user).

After finishing one part, everything must be duplicated into the folder of the next part. This includes the code, the virtualenv and all its installed tools (and config files, like for Gunicorn), as well as things only needed by me (such as the Sublime Text project file, Git directory, and some scripts).

This got me started writing bash scripts to do some of the grunt work. Today I needed to translate a Unix path:

/home/jeffy/django_files/djauth_lifecycle_tutorial/part_04/djauth_root/django_auth_lifecycle/settings.py

to its corresponding Windows path:

Q:\django_files\djauth_lifecycle_tutorial\part_04\djauth_root\django_auth_lifecycle\settings.py

(I edit everything on Windows, and am connected to this Ubuntu machine via SFTP. The goal is to echo a Windows command that I can copy and quickly execute on my local machine.)

The first step was to determine how to search and replace a string, and then set the result to a variable. After many failed attempts, it turns out there are three ways to do it (with thanks to Stack Overflow user Cyrus, and Ask Ubuntu users Seth, and terdon):

pathOnNix="/home/django_auth_lifecycle/urls.py"

#Parameter expansion (alternate separators not permitted)
#https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html#Shell-Parameter-Expansion
path=${pathOnNix//\//\\}
echo "PX: $path"

#Command Substitution, via pipe and sed. Can also be s:'\/':'\\
#https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
path="$(echo "${pathOnNix}" | sed -e 's/\//\\/g')"
echo "CS1: $path"

#Can also be s:'\/':'\\
path=$(echo "$pathOnNix" | sed s/'\/'/'\\'/g)
echo "CS2: $path"

#A "Here string": Can also be s:\/:\\:g
#http://en.wikipedia.org/wiki/Here_document#Here_strings
path="$(sed -e 's/\//\\/g' <<< "${pathOnNix}")"
echo "HS: $path"


Output:

PX: \home\django_auth_lifecycle\urls.py
CS1: \home\django_auth_lifecycle\urls.py
CS2: \home\django_auth_lifecycle\urls.py
HS: \home\django_auth_lifecycle\urls.py

This also covers the first step in translating Unix to Windows: Replacing '/' with '\'. In fact, it’s not far from a fully working *nix-to-windows translator:

#!/bin/bash
pathOnNix="/home/django_auth_lifecycle/urls.py"
path=${pathOnNix//\/home\//Q:\\}
path=${path//\//\\}
echo $path

Output: Q:\django_auth_lifecycle\urls.py

However, since I’m doing this multiple times, I decided to place it into a function. Specifically, a function that returns a value, which is very unlike the languages that I’m used to (Java and Python).

#!/bin/bash
nix_to_win() {
   #echo "a $1"
   pathOnC=${1/\/home\/jeffy\//Q:\\}
   #echo "b $pathOnC"
   pathOnC=${pathOnC//\//\\}
   echo "$pathOnC"
}

sourceNixFile="/home/jeffy/django_files/djauth_lifecycle_tutorial/part_03/djauth_root/django_auth_lifecycle/urls.py"
destNixDir="/home/jeffy/django_files/djauth_lifecycle_tutorial/part_04/"

winSrcFile=$(nix_to_win "$sourceNixFile")
winDestDir=$(nix_to_win "$destNixDir")

echo "File to copy: $winSrcFile"
echo "Destination: $winDestDir"

Output:

File to copy: Q:\django_files\djauth_lifecycle_tutorial\part_03\djauth_root\django_auth_lifecycle\urls.py
Destination: Q:\django_files\djauth_lifecycle_tutorial\part_04\

This was not too hard. What was really hard was when I uncommented out those two debugging statements in the function. As mentioned, “returning” a value from a bash function is very different than in Java or Python. Every echo-ed line is returned. So the new output is

File to copy: a /home/jeffy/django_files/djauth_lifecycle_tutorial/part_03/djauth_root/django_auth_lifecycle/urls.py
b Q:\django_files/djauth_lifecycle_tutorial/part_03/djauth_root/django_auth_lifecycle/urls.py
Q:\django_files\djauth_lifecycle_tutorial\part_03\djauth_root\django_auth_lifecycle\urls.py
Destination: a /home/jeffy/django_files/djauth_lifecycle_tutorial/part_04/
b Q:\django_files/djauth_lifecycle_tutorial/part_04/
Q:\django_files\djauth_lifecycle_tutorial\part_04\

Critically, if you change those last four lines to:

echo "File to copy:"
echo $winSrcFile
echo "Destination:"
echo $winDestDir

Then the output becomes:

File to copy:
a /home/jeffy/django_files/djauth_lifecycle_tutorial/part_03/djauth_root/django_auth_lifecycle/urls.py b Q:\django_files/djauth_lifecycle_tutorial/part_03/djauth_root/django_auth_lifecycle/urls.py Q:\django_files\djauth_lifecycle_tutorial\part_03\djauth_root\django_auth_lifecycle\urls.py
Destination:
a /home/jeffy/django_files/djauth_lifecycle_tutorial/part_04/ b Q:\django_files/djauth_lifecycle_tutorial/part_04/ Q:\django_files\djauth_lifecycle_tutorial\part_04\

Now there’s only four lines of output, with the “a” and “b” debugging lines on the same line as the returned value (scroll to the right). What confused me was, just the action of debugging the function caused all the lines to be jumbled together (in my actual situation, I didn’t have those “File to copy”/”Destination” headers which make things clearer…what’s that concept of how looking at a problem changes it?).

Adding newlines explicitely to the debugging statements:

nix_to_win() {
   echo "a $1\n"
   pathOnC=${1/\/home\/jeffy\//Q:\\}
   echo "b $pathOnC\n"
   pathOnC=${pathOnC//\//\\}
   echo "$pathOnC"
}

does not change the output. Trying every possible solution in this Stack Overflow question makes no difference–the lines are still jumbled together.

The problem is, of course, that the debugging and return value are sharing the same output “pipeline”: The return variable itself. Moreso, the nature of bash is, when what is being echoed is not in double quotes, that all newlines and tabs are each replaced with a single space. Unless you really *really* know what you are doing, variable references ($varname) should always be placed in double-quotes. In fact, not doing so results in the ShellCheck linter warning:

SC2086: Double quote to prevent globbing and word splitting.

This is why

echo $winSrcFile

is all on one line, and

echo "$winSrcFile"

is on multiple lines (meaning the newlines are recognized). However, there’s still a problem. While each echo is placed on its own line, explicit newline characters ('\n') are printed literally. To print actual newlines in their place, the "-e" option is required.

nix_to_win() {
   echo "a $1\n"                    #Prints "a [value]\n"
   pathOnC=${1/\/home\/jeffy\//Q:\\}
   echo -e "b $pathOnC\n"           #Prints "b [value][an actual new line]"
   pathOnC=${pathOnC//\//\\}
   echo "$pathOnC"
}

As far as how to actually debug a function without messing up the return value, you can instead output to stderr. All stdout printed in the function (such as via echo or printf), *IS* its return value. stderr still defaults to displaying on the console, but since it is a different “pipeline” than stdout, it does not interfere with the return value.

(If you want the nitty gritty details, check out this fantastic answer by user John1024 to my Stack Overflow question.)

The final version:

#!/bin/bash
pathOnNix="/home/django_auth_lifecycle/urls.py"

nix_to_win() {
   #Print debugging to stderr, to *not* concat onto return value
   #echo "a $1" >&2

   pathOnC=${1//\/home\//Q:\\}
   #echo "b $pathOnC\n" >&2    #debug to stderr

   pathOnC=${pathOnC//\//\\}
   echo "$pathOnC"             #Return value to stdout
}

pathOnNix="/home/django_auth_lifecycle/urls.py"
pathOnWin=$(nix_to_win $pathOnNix)
echo "$pathOnWin"

Output: Q:\django_auth_lifecycle\urls.py

Phew.

With thanks also to Mateo_, and Ademan on askubuntu.com.

(Here’s an XYplorer script that does just the opposite.)

[reddit]

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s