June 19

Preparing Your Server for Vacation – The Unattended Server Access Checklist

Unattended Server Checklist showing steps of 1) Check VPN Connection 2) Reverse SSH Tunnel Backup 3) VNC available on backup devices 4) Test automatic server shutdown after loss of utility power 5) Test Server WOL form one of backup devices.

I am about to embark on a much-needed vacation this Friday. I travel relatively frequently for work and something that has always been a stressor in the back of my mind was leaving my homelab unattended. With my luck, as soon as I lock my door and get in the car, the server will wait for that exact minute to go down. The way I have managed this problem is by creating a checklist to ensure that, no matter where I am, I can access my network. Stuff goes wrong, that’s the nature of any real work, but I can fix anything so long as I have access. Best of all, this checklist can save you from having to call your wife and tell her to push a button on your computer!

In this post, I give you the checklist I use for unattended server access. In future posts, I will create how-to’s for implementing these features.

1) Check VPN Connection:

VPNing into my DMZ is my primary method of access for resources on my network when I am not at home. The advantages of being able to connect to my DMZ via VPN is enormous since it allows me to act just like another client on the network.

2) Check Reverse SSH Tunnel Backup (x2):

Let’s say my VPN server has gone down (in either a controlled or uncontrolled fashion); in that situation, my primary means of unattended access has gone with it- I’d be locked out of my own network. The Army has a phrase for this: “One is none, two is one, three is better.” Since we have currently only discussed one way of accessing the network remotely, applying the military’s logic, I have no way into my network to fix it; therefore, I need a backup. This is where a reverse SSH tunnel comes to the rescue. You can either create one manually or use a 3rd party service.

I use Remote.it’s connectd. With it, I can SSH into another device on my network (keeping with the idea of redundancy for critical systems, this device should not be the same one running your VPN server) and do what I need to do from there. (See WOL below).

3) VNC Server Available on Backup Devices:

My router uses a GUI, so being able to spin up a VNC server on demand from the SSH connection above is necessary.

4) Test Automatic Server Shutdown When Running on Backup Power (UPS):

I have an APC UPS directly connected to my server over USB. In the event of a power failure, the UPS tells my server that we are no longer on utility power and allows it to gracefully shutdown. Testing is essential here since it not only checks to make sure that this feature is still functional, but it also serves as a check on the UPS’s battery life.

5) Test Server Wake-on-LAN (WOL) from Backup Device:

Unlike my other network devices (router, hardware firewall, the above backup devices, etc.), my server is set up to NOT automatically restart upon the restoration of utility power. Since power outages often can have brief periods of power restoration, I don’t want it to continuously startup to then only lose power again. Therefore, after a graceful shutdown, I want the server to stay down until I bring it back up. I accomplish this via a WOL message (“magic packet”) from one of the backup devices to the server.

Category: Servers | LEAVE A COMMENT
June 11

Django Error Silver Bullets

I’ve recently been teaching myself Django. I was working on creating a new web app to handle the backend for some IoT projects I’ve been working on and recently came across two errors. In case anyone else comes across them, I will post the silver bullets below to hopefully save you some time.

1. no such table: main.auth_user__old

This was an unhandled exception that occurred whenever I tried to POST data from the Django admin:

Request Method: POST
Request URL: http://127.0.0.1:8000/admin/weatherstation/weather_recording/add/
Django Version: 2.1.1
Exception Type: OperationalError
Exception Value: no such table: main.auth_user__old
Exception Location: /home/engineer/anaconda3/envs/djangoEnv/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py in execute, line 296
Python Executable: /home/engineer/anaconda3/envs/djangoEnv/bin/python
Python Version: 3.5.6
Python Path: [‘/home/engineer/Projects/WEATHERSTATION/weathersite’, ‘/home/engineer/anaconda3/envs/djangoEnv/lib/python35.zip’, ‘/home/engineer/anaconda3/envs/djangoEnv/lib/python3.5’, ‘/home/engineer/anaconda3/envs/djangoEnv/lib/python3.5/plat-linux’, ‘/home/engineer/anaconda3/envs/djangoEnv/lib/python3.5/lib-dynload’, ‘/home/engineer/anaconda3/envs/djangoEnv/lib/python3.5/site-packages’]

Solution: This problem is described in Issue #21982. It has since been fixed. Since I use Anaconda (“conda”) as my package manager, the simple fix for me was to just update my Django environment to the latest version.

2. ModuleNotFoundError: No module named ‘sqlparse’

I thought I was home free at this point now that I had updated my Django environment to the latest version in conda, but alas that was not the case. This time, I received the following error:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/engineer/anaconda3/envs/djangoEnvNew/lib/python3.7/site-packages/django/core/management/__init__.py", line 381

...

File "/home/engineer/anaconda3/envs/djangoEnvNew/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 28,
 in <module>
    from .introspection import DatabaseIntrospection            # isort:skip
  File "/home/engineer/anaconda3/envs/djangoEnvNew/lib/python3.7/site-packages/django/db/backends/sqlite3/introspection.py",
 line 4, in <module>
    import sqlparse
ModuleNotFoundError: No module named 'sqlparse'

Thankfully the error stack is pretty helpful here. The most recent call with the error “ModuleNotFoundError: No module named ‘sqlparse'” is pretty descriptive. To resolve this error, you simply need to install the module with the following command (while in your virtual environment):

conda install sqlparse

That’s it! Hopefully this helps someone else. As always, let me know if you have any questions!

June 3

Building Blocks of Programming Languages

  • Four basic building blocks of programming languages:
  1. Expressions
  2. Statements
  3. Statement Blocks
  4. Function Blocks

1. Expressions

  • Expressions in computer programming have the same definition as expressions in math: they are a combination of an operator and its operand(s). In keeping with the mathematical definition of an expression, they are well-defined, meaning that an expression must ultimately resolve to a value.
    • An operator tells the computer to perform some kind of mathematical or logical manipulation and is performed on one or more operands
      • Examples:
        • a + b
          • + is the operator; a and b are operands
        • x – 2
          • – is the operator; a and b are the operands
        • a < b
          • < is the operator; a and b are the operands
      • With the above examples, two operands are in play, which is why you’ll hear them referred to as binary expressions
        • As a result, binary expressions use binary operators; binary operators operate on two operands
      • Operator Classification:
        • Operators can be classified based on the number of operands they perform their operation on:
          • Unary Operators
            • Take one operand
            • Example: & (address-of operator); see Pointer tutorial
          • Binary Operators
            • Operate on two operands and are by far the most common
            • Examples: +, -, <, =, etc.
          • Ternary Operators
            • Operate on three operands
        • Operators can also be classified based on the kind of function they perform:
          • Arithmetic (math) Operators:
            • i.e. Operators that perform math
            • Examples: +, -, /
          • Relational Operators:
            • Compare the values of two operands
            • Examples: >, <, ==
            • Return/resolve to a boolean: true (1) or false(0)
          • Logical Operators:
            • Combine logical expressions
            • Examples: && (AND), || (OR)
            • Return/resolve to a boolean: true; false

2. Statements:

  • Statements are syntactically complete instructions
    • In C, syntax dictates that all statements end with a semicolon (this semicolon is known as a statement terminator)
  • Example:
    • Variable assignment:
      • a = 4;
        • This is a syntactically complete instruction; note how it is simply an expression (a = 4) consisting of the assignment operator (=) with the operands a and 4, and terminating with a semicolon as required by C. By syntactically correct, we simply mean that the instruction complies with the rules of the language (i.e. syntax).

3. Statement Blocks:

  • Statement blocks group statements together so they act like a single statement (i.e. the statements act together as a block)
  • In C, statement blocks start with { and end with }. In Python, statement blocks are controlled by indentation. This is why whitespace matters in Python but not in C.
    • The code inside the statement block is known as the statement block body
  • Example:
Example of statement block

4. Function Blocks:

  • Function blocks are blocks of code that accomplish a single task
  • Functions allow you to reuse code so that if you need to do the same thing multiple times, you simply call the function tag wherever you need it; you separately define the function (with its function block) elsewhere in your code
    • This makes it easier to maintain your code since, if you need to update your function, you only have to update the code once in the function itself and not multiple times wherever the function was called
  • Functions also help when working on big projects with multiple developers by acting as a black box
    • Good functions act like a black box in the sense that you don’t have to waste your time, brainpower, or memory knowing exactly how the code inside the function (it’s function block) works; you just have to know that for a given input, you get a given output
    • This is also the basis of the idea behind libraries- you don’t have to know exactly how something is done. You just call the function (written by someone else) from the library; this forms the very basis of abstraction which allows for collaboration
  • Functions also help for code readability; instead of mentally having to parse out multiple lines of code you can look at the function name (like verifyPhoneNumber() ) and know that it verifies the phone number.
    • Example:
      • int addNum(int a, int b){//Function block here }
May 4

How to Install Fritzing and Fix Missing Dependency Error Messages Using Symlinks:

In our previous notebook entry, we completed our exploration into the I2C protocol and implemented an external EEPROM for the Arduino. In that post, I have a wiring diagram that was created using an app called Fritzing. In this tutorial, I will explain how to install Fritzing on Ubuntu as well as how to resolve the following missing dependency errors that I was greeted with when I first installed it:

  • /usr/share/fritzing-0.9.3b.linux.AMD64/lib/Fritzing: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory
  • /usr/share/fritzing-0.9.3b.linux.AMD64/lib/Fritzing: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

1. Download Fritzing:

Begin by downloading Fritzing available here: http://fritzing.org/download/

2. Unpack the .tar to a convenient directory:

Follow the directions for the install on that same page. I extracted the .tar to my /usr/share/ directory. You may have to run as sudo to do this.

3. Navigate to the directory where you extracted your Fritzing tar and try to launch Fritzing:

Fritzing extracted to /usr/share/. Launching Fritzing using ./Fritzing
Fritzing extracted to /usr/share/. Launching Fritzing using ./Fritzing

If it launches great, but it probably won’t, and will fail with the following error:

/usr/share/fritzing-0.9.3b.linux.AMD64/lib/Fritzing: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

4. Fix the “error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory” error:

This message tells us that Fritzing is missing a dependency- specifically the libssl1.so.1.0.0 library. Now, this is a very common library so it’s highly probably you already have it. Let’s find it with the Linux locate command:

locate libssl.so.1.0.0

Running this command should give you a list of locations that have this library. As you can see, I have quite a few duplicates of it:

locate command with locations of libssl.so.1.0.0
locate command with locations of libssl.so.1.0.0

Now that I know that I have the libssl.so.1.0.0 library and I know where it’s located, we can use a powerful trick available to us because we’re on Linux- the symlink. We’ve discussed symlinks (or “symbolic links”) before when I discussed how to create a separate Plex library that allows you to selectively share content. In short, symlinks are Unix’s equivalent of a “shortcut”. You can create a symbolic link to another file (or directory) and Linux will treat that shortcut just like it’s really there.

That’s exactly what we’re going to do here. So let’s create a symbolic link by picking one of the paths from our locate command above (it doesn’t really matter which one).

First, make sure we’re in the lib directory of your Fritzing directory:

cd ./lib

How do I know this is where we want to be? Well, it was in the first part of the error message:

/usr/share/fritzing-0.9.3b.linux.AMD64/lib/Fritzing: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

Now, create your symbolic link:

ln -s [path to directory you found above with your locate command] ./

In my case, I used:

ln -s /snap/core18/941/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 ./
Creating symbolic link ("symlink") to fix dependency error message.
Creating symbolic link (“symlink”) to fix dependency error message.

Now, try to launch Fritzing again. In my case, I was greeted by a new error:

/usr/share/fritzing-0.9.3b.linux.AMD64/lib/Fritzing: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

I am always excited to see a new error. It means I actually fixed something and now I get to move on to something else that’s broken!

5. Fix the “error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory” error:

Again, we’re going to start by finding where the missing dependency exists:

locate libcrypto.so.1.0.0

Once we have a viable library location, we’re going to create that symlink to point to it:

ln -s /snap/core18/941/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 ./lib/

Note that I messed up originally when I did this in my screenshot. I ran the above command with the output path in the main Fritzing directory, therefore I should have used ./lib/ as I show above, but if you’re already in the ./lib/ directory, you can just run the output with ./ as we did in the first one.

locate command with locations of libcrypto.so.1.0.0; followed with fix by creation of symlink
locate command with locations of libcrypto.so.1.0.0; followed with fix by creation of symlink

5. Rinse and repeat.

In my case, running ./Fritzing finally launched Fritzing, but you may have other dependencies that need addressing. Now that you know how to fix these missing dependencies this shouldn’t be too much of a problem. Enjoy!

As always, feel free to ask me any questions about any problems you run into.

Category: Linux | LEAVE A COMMENT
April 28

Expand Your Arduino’s Storage with an External EEPROM Part II: Reading from the AT24C256 – A Tutorial in How to Use the I2C Protocol Continued

Wiring Diagram Showing Connections between AT24C256 and Arduino

We first began our journey into learning the I2C protocol three weeks ago. In that post, we learned to write to an external EEPROM over the I2C protocol using nothing more than a datasheet and the Arduino’s built-in Wire library. Before learning to read from that EEPROM, which we will do today, we needed to gain the prerequisite knowledge of how data is stored in memory and how pointers work. From there, we learned how the data stored in these variables is passed along through to functions and what an array really is.

It’s been a daunting few weeks, but we’re finally there. Let’s read the data we wrote to our EEPROM armed with nothing more than the datasheet and the I2C protocol.

1. Review the Datasheet

Using the same strategy as before, we look for the command we’re interested in on the datasheet. Since we last wrote to the EEPROM using a page write, it should be pretty easy to guess that to read off our same data, we probably want a page read (also known as a sequential read).

Going to the Sequential Read section of the datasheet, we’re given the following description:

SEQUENTIAL READ: Sequential reads are initiated by either a current address read or a random address read. After the microcontroller receives a data word, it responds with an acknowledge. As long as the EEPROM receives an acknowledge, it will continue to increment the data word address and serially clock out sequential data words. When the memory address limit is reached, the data word address will “roll over” and the sequential read will continue. The sequential read operation is terminated when the microcontroller does not respond with a zero but does generate a following stop condition (see Figure 12 on page 12).

Atmel: https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

From here, the code practically writes itself, we just need to follow the directions Atmel has given us.

2. Write the Preamble:

Per the datasheet: “Sequential reads are initiated by either a current address read or a random address read.” Well, we wrote our data to a specific address, so we want to read from that address, so we’ll initiate the sequential read by using the random read. Referring to the Random Read section of the datasheet:

RANDOM READ: A random read requires a “dummy” byte write sequence to load in the data word address. Once the device address word and data word address are clocked in and acknowledged by the EEPROM, the microcontroller must generate another start condition. The microcontroller now initiates a current address read by sending a device address with the read/write select bit high. The EEPROM acknowledges the device address and serially clocks out the data word. The microcontroller does not respond with a zero but does generate a following stop condition (see Figure 11 on page 12).

Atmel: https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf
Random Read Preamble: dummy byte write with “the device address word and data word address are clocked in and acknowledged by the EEPROM”

The above-boxed section represents the “the device address word and data word address are clocked in and acknowledged by the EEPROM” portion. We’ll start by coding this section. Thankfully, we’ve already done it in the Page Write post- it’s everything in section 2:

Wire.beginTransmission(0b1010000);
Wire.write(0b0000000); // 7 bits of 0s; this method takes a byte though so it will still transmit a byte's worth of 0s.
Wire.write(0b00000000);
Wire.endTransmission();

Boom. Header done.

3. Read the Data:

RANDOM READ: A random read requires a “dummy” byte write sequence to load in the data word address. Once the device address word and data word address are clocked in and acknowledged by the EEPROM, the microcontroller must generate another start condition. The microcontroller now initiates a current address read by sending a device address with the read/write select bit high. The EEPROM acknowledges the device address and serially clocks out the data word. The microcontroller does not respond with a zero but does generate a following stop condition (see Figure 11 on page 12).

Atmel: https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

The start condition (with sending a device address) is common to the I2C protocol and is thankfully handled by the Wire library with the simple Wire.requestFrom method, where the first argument is the device address and the second argument is the number of bytes to request from the EEPROM:

Wire.requestFrom(deviceaddress,2);

Since I only wrote two bytes “Hi” to the address, I’m only requesting two bytes back.

Now to actually read the data, we’ll use a simple while loop:

while(Wire.available()) {
    Serial.print((char)Wire.read());
}

Putting together the full code in its entirety from the past two tutorials:

C source code for Arduino implementing the I2C protocol with an EEPROM.
Source code for both writePage and readPage.

Here is what she looks like on the Serial Monitor:

Serial monitor showing EEPROM read and write.
Serial output showing EEPROM read and write.

In the future, we’ll eventually revise this code to make it more versatile by putting it in a library.

April 27

Pointers, Arrays, and Functions in Arduino C

Array Memory Diagram

Now that we’ve completed our introduction to pointers, I had really wanted to move on and wrap up our section on using an EEPROM with the I2C protocol today. However, I feel like I would be doing a disservice to you without elaborating further on why we would even want to use pointers in the first place.

Just to recap, let’s look at some simple code to demo the syntax of using a pointer:

int myVar = 10;
int *myPointer;
myPointer = &myVar;
*myPointer = 20;

If you were to compile this code and run it, you would see that at the end myVar’s value would now be 20 even though you’ll notice we never set myVar itself to 20. We accomplished this by referencing our pointer, myPointer, to the memory address of myVar using the reference operator (&). We then dereferenced our pointer by using the dereference operator (*) with our pointer and setting its value to 20.

Now, the obvious question you probably have is, “Why in the heck would I want to do that?”

The example above is more of a toy, obviously contrived, but there are very real reasons why you would want to do this, especially when you’re running a microcontroller like the Arduino and you have to handle a lot more low-level operations. To see the value in pointers, you’ll first need to know something about how functions work in C.

I want to keep this explanation of functions at a high-level to keep the concepts easy to understand. For now, just know there are two ways to call a function: by value and by reference.

Function Call By Value:

Pass by Value lvalue-rvalue diagram
Pass By Value- Note how the rvalues are copied

We’ll start with the easiest one- easiest because it’s the one you’re most familiar with; you’ll see that a function call by reference isn’t particular difficult either.

int sum(int x, int y) {
	int sumZ = x + y;
	return sumZ;
}

 void main {
	int a = 2;
	int b = 3;
	int sumAB = sum(a,b);
}

Start by reading this kind of code like a computer would: with the main function. We set a = 2 and b =3, we then go to get sumAB by calling the function sum(a,b). When we call that function, we replace a with the value of a (i.e. 2) and b with the value of b (i.e. 3), so we could just as easily say int sumAB = sum(2, 3);

In the sum function we created, we set x = 2 and y = 3 inside the function due to the above arguments that have been passed to it. This is called passing by value because we’ve merely passed the values of the variables. Inside the function those values passed to it (the values of a and b in this case) are copied to its variables (x and y in our example).

You’ll see this can actually be a problem if we wanted the function to actually do something with those original variables (like change them). The function that has had its arguments passed by value can’t actually change the values of the arguments themselves because all it has access to is the function’s own copy of those values (i.e. x and y have a different lvalue from a and b even though their rvalues are the same).

So yeah, it can change the values of x and y, but it doesn’t affect the values of a and b because they reside at a different location in memory. Additionally, the values of x and y cease to exist as soon as we exit the stack (i.e. right after we return sumZ).

Thankfully, there’s a way around this: enter call by reference.

Function Call by Reference:

Pass by reference lvalue-rvalue diagram
Pass by reference: Note how the lvalue (memory address) is copied

So what if we want to actually change the parameters that we are passing to a function? Or what if we simply want to return more than one value? How can we escape the box that is the function’s call stack? That’s where call by reference (pointers) comes to the rescue.

void addOne (int *numA) {
	*numA = *numA + 1;
}

void main() {
	int varA = 15;
	addOne(&varA);
	Serial.println(varA);
}

Let’s again think about what’s happening here. We are taking a variable, varA, and extracting its memory address (its lvalue) with the reference operator, &. We are then passing that memory address in for the parameter numA in the function addOne. You can think of this as being the equivalent of declaring and initializing the pointer: int *numA = &varA. Inside the function, we are given direct access to the value stored in varA by dereferencing our numA pointer. The result of this program is the console prints 16 now stored in varA.

In a much more general (and I dare say enlightened sense), another way you can think of this is that in a way, this really is the same as call by value where we simply pass the rvalue of our parameter off to the function’s internal variables. The key difference is that for a pointer, its rvalue is simply the memory address of what it points to, therefore a memory address is what gets copied to the function!

Advanced Topic: This is the perfect opportunity to introduce this. In programming, particularly the C family of languages, there are two distinct categories of variables: value type variables and reference type variables. Now that you’re an expert on function call by value, function call by reference, and pointers, you can appreciate where the terms come from. Value type variables are variables where the rvalue stores an actual value (like an int storing the value 10). Value type variables tend to be associated with the “primitives”- primitive variables like int, char, byte, etc. Reference type variables store a memory address in their rvalue.

Arrays:

You’re undoubtedly familiar with the usual way of looping over an array, where we simply increment the index:

Standard for loop over an array
Standard for loop over an array’s index

But what if I told you, there was another way? Take a look at the following code:

Looping over an array using pointers
Looping over an array using pointers

They have the exact same output!

Serial output showing memory address and value of an array.

But why?! How?!

Look closely at the line where we print our value: Serial.println(*(numArray + i)); That looks like a pointer doesn’t it? Well, that’s because it is. Let’s dissect this a little more and look inside the parentheses. We know I is an int so as this loop progresses we’re adding + 0, +1, +2, etc. What does that tell us about numArray then? Well, that means it has to be a number of some sort for the addition operation to make any sense. But what kind of number? Well, we know this is a pointer so it must be what? That’s right, numArray is an address!

Now that you understand how pointers work, you now understand the implications of what this means. It means that when you define an array, what you’re actually doing is defining a pointer. The name of an array itself, such as numArray, is actually a memory address! If you read that advanced topic blurb above, the implication is that arrays are actually reference type variables and are therefore inherently passed by reference when used in functions.

Before we close this page of the notebook, I want to highlight a “gotcha”. Let’s say numArray had a memory address of 2288 as it apparently does from my screenshot above. If I = 1, why is it on the second iteration of the loop the address is 2290 and not 2289? The reason is because of how the compiler handles pointers. You see, when you define the array initially, the compiler is smart enough to allocate the memory based on the size of the data type used. In our case, we used ints which, in Arduino C, are two bytes long. When you iterate a pointer, the compiler is smart enough to multiply the iteration by the size of the data type for the next memory address. Therefore we start at 2288 and the next memory address for our next item in the array is 2290, followed by 2292, 2294, and so on:

Array memory diagram showing memory addresses.
April 25

Variables, Pointers, and Indirection in Arduino C

Before we continue on with learning about the I2C protocol and our EEPROM project, we need to discuss variables: what they are and what goes on behind the scenes. Knowledge of how variables work and the use of pointers and indirection with arrays will serve us well when it comes time to read from our EEPROM. Let’s begin.

Anatomy of a Variable:

1. What is a variable?

Simply put, variables hold data. More specifically, a variable holds data of a specific data type. For example, an int holds an integer, a string contains a collection of chars, etc.

2. What goes on behind the scenes when a variable is defined and when it is assigned?

When you define a variable, the compiler goes and checks the symbol table (basically a list of variables that have previously been declared) to see if that variable already exists. If it doesn’t, the compiler goes ahead and adds the new variable to the list.

Say, for example, you add the following statement:

int myVar;

Since our variable has not already been declared (it doesn’t already exist in the table), the compiler updates the symbol table so it now looks like this:

Symbol table with myVar declared (but not yet defined) since it lacks a location in memory (lvalue).
Symbol table after myVar declared- note the lack of an lvalue. This is because myVar is not yet defined. rvalue is also unknown because we haven’t assigned a value to myVar yet.

Now, technically, the variable has only been declared at this point- it’s missing an actual location in memory. To get this location in memory, the compiler requests a place to put this variable from the system’s memory manager. The memory manager then responds with a memory address which the compiler then adds to the symbol table for that variable. This memory address is known as an lvalue (lvalue = location value) and it merely represents where the variable can be found in memory. With this addition of the lvalue to the symbol table, our variable is now defined:

myVar now defined in the symbol table (myVar now has an lvalue).
Symbol table with myVar defined- this means that the variable now has a location in memory (lvalue).

With our new variable defined, we can now move on to storing a value in it. Fortunately, assigning a value to a variable is rather straightforward. When we assign a value to a variable, we directly navigate to the variable’s location in memory (the lvalue) and update the memory at that address with the new value. The data that’s actually stored in memory is known as the rvalue (rvalue = register value).

Continuing our example with the following assignment statement:

myVar = 10;

With this assignment, our symbol table now looks like this:

myVar after rvalue assignment
Symbol table after assignment- note the updated rvalue which holds our data value.

Another way to visualize what we have just gone over is with an lvalue-rvalue diagram:

lvalue-rvalue diagram for a value type variable
lvalue-rvalue diagram

This diagram is why you will see some people refer to the memory address as the “left value” and the actual data value as the “right value”.

  • There’s also an important caveat here: in Arduino, and C in general, there is no duty to clear that rvalue at our variable’s lvalue when we define it. Therefore you should always assume that a variable’s value contains whatever garbage was originally in that memory location unless we’ve explicitly assigned a value to the variable. (i.e., Don’t assume it’s 0 or null). Therefore it’s probably best to go ahead and initialize your variable with a value when you define it.
    Let’s summarize: Whenever your program needs to use the value stored in a variable, it uses the variable’s lvalue to go to that memory address and retrieves the data (rvalue) from that memory location.

Pointers:

Now that we’ve covered what variables are and how they really work, we’re ready to understand pointers. Simply put, a pointer is nothing more than a variable that references the memory address of another variable. Using the terminology that we’ve just learned, a pointer is a variable whose rvalue is the lvalue of another variable.

To visualize this, let’s take a look at two lvalue-rvalue diagrams representing the value type variable myVar and the reference type variable myPointer:

myPointer referencing myVar - Notice how the rvalue of myPointer is the memory address of myVar.
myPointer referencing myVar – Notice how the rvalue of myPointer is the memory address of myVar.

Declaring a Pointer:

Declaring a pointer variable is rather straightforward:

int *myPointer;

The type specifier (int in this case) must match the data type of the variable the pointer is to be used with. The asterisk indicates to the compiler that myPointer is a pointer. Since whitespace doesn’t really matter in C, the asterisk can be placed anywhere between the type specifier and the pointer variable name so you will sometimes also see: int* myPointer, int * myPointer, etc.

The Address-Of Operator:

By itself, a pointer that is defined but does not actually point to anything is a pretty pointless pointer (ha!). To point it to the memory address of another variable we simply need to assign the pointer the memory address of that variable. But where do we get the memory address from? That is, where do we get the lvalue of myVar from? Enter the address-of operator (&).

The address-of operator is a unary operator that returns the lvalue of a variable.

Pointer Assignment:

To point our new pointer at the memory location of our value type variable, myVar, we simply call the following statement:

myPointer = &myVar;

This completes the link shown in the previous diagram and is known as referencing. It is for this same reason that the address-of operator (&) is also known as the “referencing operator“.

Whenever you are learning a new concept, it’s a good idea to try it out yourself to prove to yourself what you’ve read. Let’s mock up an example of what we’ve learned so far in the Arduino IDE:

void setup() {
  Serial.begin(9600);
  
  int myVar = 10;  // Initialize a variable.
  
  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);  // Grab myVar's lvalue
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();
  
  int *myPointer;   // Declare your pointer.
  myPointer = &myVar; //Assign myVar's memory address to pointer.
  
  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);  //myPointer's lvalue
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);  //myPointer's rvalue
}

void loop() {
}

Watching the serial monitor, what you should see is something like this:

Serial log showing that the rvalue of a pointer is the memory address of the value type variable it references.
Note that the rvalue of myPointer is the same as myVar’s lvalue.

Notice that myPointer’s rvalue is the memory address of myVar (i.e. myVar’s lvalue), just like it shows in the diagram.

Indirection (Dereferencing):

We just saw that a pointer can reference a location in memory by assigning that pointer a variable’s memory address using the reference operator (&). We can take this a step further and obtain the actual value stored at that memory address by dereferencing the pointer. This is also known as indirection and is accomplished via the indirection operator (*) with your pointer. Example:

*myPointer = 5; // Go to memory addressed stored in myPointer's rvalue (myVar's lvalue) and place the value 5 in that memory address.

Continuing off our previous Arduino code example:

void setup() {
  Serial.begin(9600);
  
  int myVar = 10;
  
  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();
  
  int *myPointer;
  myPointer = &myVar;
  
  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);
  Serial.println();

  *myPointer = 5;  //THIS IS OUR DEREFRENCING ADDITION.
  Serial.println("-----------------------");
  Serial.println("Updating *myPointer = 5");
  Serial.println();

  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);
  Serial.println();

  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();

}

void loop() {
}
dereferencing the pointer and assigning a value; we are able to manipulate the data stored in myVar
Notice that by dereferencing the pointer and assigning a value, we are able to manipulate the data stored in myVar.

Notice that nothing changed to myPointer at all (blue). Neither its lvalue nor its rvalue changed. Contrast that with myVar (red) which had it’s rvalue changed to 5 by the indirection operator we applied to our pointer.

That is the power of pointers and indirection. In my next journal entry, I will discuss pointers and arrays which will then allow us to finally move on to the last part of our EEPROM I2C project!

April 11

A Short Introduction to Troubleshooting Docker Networks

Docker Network Diagram- depicting the major interfaces involved.

I recently just built an unRAID rig which has now been deployed to my DMZ. It’s great, but something I have been struggling with is that periodically my FileZilla docker container will be unable to connect to my FTP server, erroring out with “ENETUNREACH – Network unreachable”. It seems to be exacerbated by large file moves (like when I’m moving whole directories).

ENETUNREACH - Network unreachable error in FileZilla
ENETUNREACH – Network unreachable error in FileZilla

I started by researching the error message but there isn’t a whole lot on the formal definition of “ENETUNREACH – Network unreachable”. Presumably because “Network unreachable” is self-explanatory. There are a lot of forum posts about FileZilla being blocked by an antivirus suite’s software firewall, however, so I feel comfortable with the error description given. There are no subtleties to the message “ENETUNREACH – Network unreachable”- it simply means that FileZilla can’t connect to the network (i.e. it can’t dial out).

Docker Networking Overview:

Let’s begin with a brief introduction to Docker networking. At a high level, Docker containers are very similar to virtual machines (VMs) but without the overhead of having to run a duplicate OS. The diagram below, shows our Docker containers which run inside of our Host:

Basic Docker network diagram showing three interfaces: the interface between the container and the host, interface between the host and router, and interface between the router and internet.
Basic Docker Network Diagram

The above diagram shows a basic schematic of a typical Docker network. For my containers, I have the network configuration set to bridge mode which means that, at least for networking purposes, the Host acts like a glorified router to the Container. Put another way, when I want to communicate with the container over the network, I just put in the IP address of my Host and a port specific to that Container. The Host then takes that traffic and forwards it to the internal IP address of the Container at whatever port it’s listening on. This is also true for outbound traffic. In essence, bridge mode on Docker is nothing more than network address translation (NAT). Something you’re undoubtedly familiar with if you have a homelab or are self-hosted like I am.

Three network interfaces in Docker networks

Back to our problem. We know FileZilla can’t dial out. Referring back to the diagram, we see that there are three checkpoints where traffic could be failing: (1) the interface between the Docker container and Host, (2) the interface between the Host and the router, and (3) the interface between the router and the WAN/internet.

Troubleshooting the Connection:

Since we know our problem is in the outbound direction, let’s focus in that direction. What’s the simplest way to check for a connection? Ping it!

Testing each network interface we identified above in turn:

1) Ping the Host from the Container:

We can accomplish this by executing a command in the Container. This can be accomplished in unRAID by clicking on the Container and selecting console. If you aren’t running unRAID, this is the same as running <docker exec -it [container name here| FileZilla for me] sh>. Now you can simply ping the Host using <ping [insert IP address of Host on LAN here]. Pinging the Host in my case greeted me with a response so we know that connection isn’t the culprit:

Pinging Docker Host from Container.
Pinging the host from the container. We have a response so no problem here!

Note: You can also ping an address on the internet. When I did that here, I did not receive a response, confirming that the outbound connection is indeed broken somewhere along the chain:

Pinging website from Docker container.
Pinging a website from the container. No response. So we know something is broken along the chain.

2) Ping the Router from the Host:

This is straightforward enough. In unRAID you just open your console/terminal on the Host (“Tower”) and ping the router’s IP address. I also received a response here so we know that we have a connection to router. We probably already knew this since we could connect to the Host over the network in the first place though.

Pinging an address on the internet also received no response meaning my server was not able to access the internet. This further confirmed that the chain is broken still further upstream.

3) Ping an address on the WAN/internet from the Router:

This can be a little bit trickier. Thankfully I run dd-wrt on my routers so I can easily initiate a terminal session on the router with <terminal [insert router local IP address here]>. Pinging an address on the internet resulted in no response here as well! Well, we know this guy is the last interface in the chain so we know the problem is with him.

[There’s a little bit more to know here which is more specific to my network’s “unique” architecture. My server resides in a physically separate part of my home- away from my core network and my home isn’t physically wired for ethernet, therefore to adapt and overcome, I have dd-wrt set up to create a client bridge with this router (the router mentioned above is actually the client router in the client bridge). For those of you who don’t know what this is, it means that all clients connected to my secondary client router behave like they’re physically connected to the primary router. At least that’s the way it’s supposed to work in theory and it most often does- unfortunately, the client bridge mode is notoriously unreliable as is the case here. Had this been the DMZ’s main router at fault, as would be the case on your home network, a big clue would’ve been that I couldn’t access the internet from my computer.]

Note: I also want to point out another potential cause here. If, at any point when pinging an external site on the internet, you found that you didn’t get a response, try pinging an IP address (as opposed to the website address) you know is good. If that works, that suggests a problem with your DNS lookup and you should begin your investigation there.

Rebooting the router fixed the issue and allowed FileZilla to proceed without error. Honestly, the root cause here is that I am using client bridge mode but getting rid of it really isn’t an option for me at this point. I could try upgrading the router to better hardware but I need to let my wallet recover from this server build first. 🙂

Anyway, I thought this made for an interesting case study and demonstrates how breaking a system down into its simple parts allows for effective troubleshooting. With a methodical process and understanding, every problem can be overcome.

Improvise. Adapt. Overcome.

April 10

How to Share Part of Your Plex Libraries without Giving Users Complete Access to Your Full Library: Symbolic Links with Plex and Docker Containers

By now you’ve discovered some of the many shortcomings of Plex. One of which is that you can’t share individual videos with your friends or family- you have to give them access to the entire library! It’s even worse if you have kids and all of your movie content is stored under a single “Movies” library: “Saving Private Ryan” is going to be right next to “Tangled”.

Well, thankfully, there’s a way around that. Enter “symbolic links”. I’ll do another tutorial on what a symbolic link actually is in the future, but for now, think of them as Unix’s version of a shortcut- it merely points to the filename of another file located somewhere else. This means that to share specific content with users, all we have to do is create a library folder specific to that user and place a bunch of symbolic links in it referencing the content in the libraries we have stored already.

So yes, we do still have to create a new library to share with users, but the good news is that we can do this all without having to make a duplicate of the file and taking up valuable storage space!

Creating a Symbolic Link:

If your Plex Media Server is a normal install (i.e. not running in a Docker container), creating a symbolic link is pretty straightforward. Just create your new library directory and navigate to it in a terminal. Then just execute the following command:

ln -s [insert file source location here] ./

The above path in the [insert file path to video here] can be either a directory or a video file itself. Just create a symbolic link for each individual directory or video you want to share. That’s it!

Symbolic Links with Docker Containers:

Normally symbolic links (“symlinks”) are quite transparent to applications in Linux (that’s the whole point), but in the case of running Plex on an unRAID server, we have an additional challenge: Plex is running in a Docker container. Plex Docker containers have a volume mapping configured to map the media path in the Plex container to the user share on the unRAID host. The chances are that the two file paths specified for the container and the host don’t have the exact same parent directory (if they did that would partially defeat the point of the abstraction that Docker containers give you).

Docker Container Volume Mapping
Docker Container Volume Mapping: Host Path ‘= Container Path

The above configuration shows what I mean a little better. As you can see, the Host path does not equal the Container path. The consequence of this is that the Docker Host and the Docker Container have two different systems of reference. This problem we run into when we run the command “ln -s” above is that the symbolic link created follows an absolute path so when the Plex Docker container starts to follow this path that’s specified based off the Host’s directory tree, it fails. I think an example will help illustrate:

Normal "absolute" symlink with absolute file path.
Normal symlink with absolute file path.

The Fix: Relative Symlinks

Thankfully this is easily fixed with the addition of the relative (-r) argument to the ln command which will instead give us a relative symbolic link:

ln -rs [insert file source location here] ./

The example below demonstrates the difference compared to a “regular” symlink:

Relative symbolic link giving us a relative file path that will work in a Docker container.
Relative symlink giving us a relative file path that will work in a Docker container.

This allows the Docker container to translate the file path specified in the symbolic link into its own internal file structure. That’s it!

In a future tutorial I will explain more of the technical details behind symlinks and you’ll see equivalent alternatives to the command above.

April 6

Expand Your Arduino’s Storage with an External EEPROM (AT24C256): A Tutorial in How to Use the I2C Protocol

In a previous post, we covered how to expand your number of analog inputs by using an external ADC over the SPI bus. Today I want to demonstrate how to use the I2C protocol while simultaneously teaching you how to read a datasheet. After all, there will come a day when you want to use a new device and a library won’t already exist for it.

First of all, let’s establish the Arduino library we will be implementing to communicate with the AT24C256 EEPROM. For this tutorial, we will be using the Wire library to implement the I2C protocol (Mental shortcut: I2C protocol = Wire library). Let’s begin!

1. Review the Datasheet

First, let’s survey the land and have a look at the AT24C256 datasheet. When I look at a datasheet, one of the first things I do is figure out what commands I want to use. In this case, I know I want to do a page write. A page write allows you to write multiple bytes of data all at once, as opposed to a byte write which is all sorts of tedious. I’m a visual learner so once I know what command I want to use, I jump to the figure that parses it out. The figure for Page Write is shown below:

Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Just looking at this figure I see that every I2C message starts with three words: a device address, a first word address, and a second word address. This preamble is then followed with the actual data you want to write.

2. Writing Out the Code for the Address (i.e. the preamble)

From that diagram alone the code practically writes itself. It’s just a matter of knowing which methods to call in the Wire library. Let’s implement the code one word at a time:

Device Address:
Device Address Message
Device Address Highlighted in Red; Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Focusing on this part of the diagram, we see that the device address begins with a 1 (the line is high), followed by a 0 (line is low), 1, and so on so that we ultimately end up with 0b1010000. The last two least significant bits (LSBs) are 0s but can actually be changed to allow for up to four of these EEPROMs to be on the I2C bus. This is also shown separately in the datasheet:

Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

“But wait!” you say, “That’s only 7 bits. A byte is eight bits. What gives?!”

Well, you are technically correct- to make the full byte, we should have an eighth bit that tells the device whether we want to read or write (the R/W in the LSB position shown above), but we are using the Arduino Wire library which expects a 7-bit address. If you are given an 8-bit address like in Figure 7 above, you need to bit shift one position to the right to get a 7-bit address. Notice that you don’t have to do this minor bit of mental gymnastics if you just look at the original diagram in Figure 9. The Wire library will handle appending the R/W bit based on what method you end up calling.

So to begin our I2C transmission, we simply call the following:

Wire.beginTransmission(0b1010000);
First Word Address:
First Word Address Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Let’s just assume we’re starting off with a blank slate. So we’re just going to start from an initial address of 0. Honestly, we might as well go ahead and address the second word address here while we’re at it. We can think of the first and second word address as one giant address: i.e. two bytes together (a.k.a. an int). Looking at where the MSB of this “giant” 16-bit would fit in the diagram above (where the * is), we see that this value is a “DON’T CARE BIT”. Since I’m using a 256K EEPROM, our MSB for our address begins where the cross is (so we actually have a 15-bit address). Therefore our 15-bit address comes out to: 0b000000000000000 (i.e. 15 0’s).

Getting back to the first word address, we can see that that the AT24C256 still expects a full 8-bit byte even if it doesn’t particularly care what that first bit actually is. We can accomplish this with the following code:

Wire.write(0b0000000); // 7 bits of 0s; this method takes a byte though so it will still transmit a byte's worth of 0s.
Second Word Address:
Second Word Address Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Now we just transmit the second word address (i.e. the remaining 8 bits of our 15 bits from above):

Wire.write(0b00000000);

Granted this was a pretty trivial address, but I think you get the idea.

3. Write the Data

Now, I assume you probably didn’t come here to just address the EEPROM. You want to actually write some data to it right? Well, now you can do that using the same Wire.write calls we’ve done before. We’re finally in this part of the Page Write diagram:

Data Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

This part is rather simple compared to what we’ve done so far. To write data we simply use Wire.write again:

Wire.write(15);

The beauty of using Page Write is that we aren’t limited to just writing one byte at a time (which is what happens when you use Byte Write). With Page Write, once you’ve queued up the preamble, you can then queue up an entire array all at once. For example:

Wire.write("ABC");

4. Putting It All Together:

Putting it all together, the hard-coded Page Write function we just wrote could look something like this:

void writePage(){
  Wire.beginTransmission(0b1010000);
  Wire.write(0b0000000);
  Wire.write(0b00000000);
  byte a = Wire.write("Hi");
  Wire.endTransmission();
}

Don’t forget the Wire.endTransmission(); at the end so that we actually send out our data over the I2C bus. Also don’t forget to designate your Arduino as the master device in your setup block with Wire.begin();.

This tutorial has become a bit longer than I originally intended. Let’s call it a night and we’ll pick back up in the next post with how to request (read) data back off the EEPROM!