April 25

Variables, Pointers, and Indirection in Arduino C

Before we continue on with learning about the I2C protocol and our EEPROM project, we need to discuss variables: what they are and what goes on behind the scenes. Knowledge of how variables work and the use of pointers and indirection with arrays will serve us well when it comes time to read from our EEPROM. Let’s begin.

Anatomy of a Variable:

1. What is a variable?

Simply put, variables hold data. More specifically, a variable holds data of a specific data type. For example, an int holds an integer, a string contains a collection of chars, etc.

2. What goes on behind the scenes when a variable is defined and when it is assigned?

When you define a variable, the compiler goes and checks the symbol table (basically a list of variables that have previously been declared) to see if that variable already exists. If it doesn’t, the compiler goes ahead and adds the new variable to the list.

Say, for example, you add the following statement:

int myVar;

Since our variable has not already been declared (it doesn’t already exist in the table), the compiler updates the symbol table so it now looks like this:

Symbol table with myVar declared (but not yet defined) since it lacks a location in memory (lvalue).
Symbol table after myVar declared- note the lack of an lvalue. This is because myVar is not yet defined. rvalue is also unknown because we haven’t assigned a value to myVar yet.

Now, technically, the variable has only been declared at this point- it’s missing an actual location in memory. To get this location in memory, the compiler requests a place to put this variable from the system’s memory manager. The memory manager then responds with a memory address which the compiler then adds to the symbol table for that variable. This memory address is known as an lvalue (lvalue = location value) and it merely represents where the variable can be found in memory. With this addition of the lvalue to the symbol table, our variable is now defined:

myVar now defined in the symbol table (myVar now has an lvalue).
Symbol table with myVar defined- this means that the variable now has a location in memory (lvalue).

With our new variable defined, we can now move on to storing a value in it. Fortunately, assigning a value to a variable is rather straightforward. When we assign a value to a variable, we directly navigate to the variable’s location in memory (the lvalue) and update the memory at that address with the new value. The data that’s actually stored in memory is known as the rvalue (rvalue = register value).

Continuing our example with the following assignment statement:

myVar = 10;

With this assignment, our symbol table now looks like this:

myVar after rvalue assignment
Symbol table after assignment- note the updated rvalue which holds our data value.

Another way to visualize what we have just gone over is with an lvalue-rvalue diagram:

lvalue-rvalue diagram for a value type variable
lvalue-rvalue diagram

This diagram is why you will see some people refer to the memory address as the “left value” and the actual data value as the “right value”.

  • There’s also an important caveat here: in Arduino, and C in general, there is no duty to clear that rvalue at our variable’s lvalue when we define it. Therefore you should always assume that a variable’s value contains whatever garbage was originally in that memory location unless we’ve explicitly assigned a value to the variable. (i.e., Don’t assume it’s 0 or null). Therefore it’s probably best to go ahead and initialize your variable with a value when you define it.
    Let’s summarize: Whenever your program needs to use the value stored in a variable, it uses the variable’s lvalue to go to that memory address and retrieves the data (rvalue) from that memory location.

Pointers:

Now that we’ve covered what variables are and how they really work, we’re ready to understand pointers. Simply put, a pointer is nothing more than a variable that references the memory address of another variable. Using the terminology that we’ve just learned, a pointer is a variable whose rvalue is the lvalue of another variable.

To visualize this, let’s take a look at two lvalue-rvalue diagrams representing the value type variable myVar and the reference type variable myPointer:

myPointer referencing myVar - Notice how the rvalue of myPointer is the memory address of myVar.
myPointer referencing myVar – Notice how the rvalue of myPointer is the memory address of myVar.

Declaring a Pointer:

Declaring a pointer variable is rather straightforward:

int *myPointer;

The type specifier (int in this case) must match the data type of the variable the pointer is to be used with. The asterisk indicates to the compiler that myPointer is a pointer. Since whitespace doesn’t really matter in C, the asterisk can be placed anywhere between the type specifier and the pointer variable name so you will sometimes also see: int* myPointer, int * myPointer, etc.

The Address-Of Operator:

By itself, a pointer that is defined but does not actually point to anything is a pretty pointless pointer (ha!). To point it to the memory address of another variable we simply need to assign the pointer the memory address of that variable. But where do we get the memory address from? That is, where do we get the lvalue of myVar from? Enter the address-of operator (&).

The address-of operator is a unary operator that returns the lvalue of a variable.

Pointer Assignment:

To point our new pointer at the memory location of our value type variable, myVar, we simply call the following statement:

myPointer = &myVar;

This completes the link shown in the previous diagram and is known as referencing. It is for this same reason that the address-of operator (&) is also known as the “referencing operator“.

Whenever you are learning a new concept, it’s a good idea to try it out yourself to prove to yourself what you’ve read. Let’s mock up an example of what we’ve learned so far in the Arduino IDE:

void setup() {
  Serial.begin(9600);
  
  int myVar = 10;  // Initialize a variable.
  
  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);  // Grab myVar's lvalue
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();
  
  int *myPointer;   // Declare your pointer.
  myPointer = &myVar; //Assign myVar's memory address to pointer.
  
  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);  //myPointer's lvalue
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);  //myPointer's rvalue
}

void loop() {
}

Watching the serial monitor, what you should see is something like this:

Serial log showing that the rvalue of a pointer is the memory address of the value type variable it references.
Note that the rvalue of myPointer is the same as myVar’s lvalue.

Notice that myPointer’s rvalue is the memory address of myVar (i.e. myVar’s lvalue), just like it shows in the diagram.

Indirection (Dereferencing):

We just saw that a pointer can reference a location in memory by assigning that pointer a variable’s memory address using the reference operator (&). We can take this a step further and obtain the actual value stored at that memory address by dereferencing the pointer. This is also known as indirection and is accomplished via the indirection operator (*) with your pointer. Example:

*myPointer = 5; // Go to memory addressed stored in myPointer's rvalue (myVar's lvalue) and place the value 5 in that memory address.

Continuing off our previous Arduino code example:

void setup() {
  Serial.begin(9600);
  
  int myVar = 10;
  
  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();
  
  int *myPointer;
  myPointer = &myVar;
  
  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);
  Serial.println();

  *myPointer = 5;  //THIS IS OUR DEREFRENCING ADDITION.
  Serial.println("-----------------------");
  Serial.println("Updating *myPointer = 5");
  Serial.println();

  Serial.print("myPointer's lvalue: ");
  Serial.println((long) &myPointer, DEC);
  Serial.print("myPointer's rvalue: ");
  Serial.println((long) myPointer, DEC);
  Serial.println();

  Serial.print("myVar's lvalue: ");
  Serial.println((long) &myVar, DEC);
  Serial.print("myVar's rvalue: ");
  Serial.println(myVar, DEC);
  Serial.println();

}

void loop() {
}
dereferencing the pointer and assigning a value; we are able to manipulate the data stored in myVar
Notice that by dereferencing the pointer and assigning a value, we are able to manipulate the data stored in myVar.

Notice that nothing changed to myPointer at all (blue). Neither its lvalue nor its rvalue changed. Contrast that with myVar (red) which had it’s rvalue changed to 5 by the indirection operator we applied to our pointer.

That is the power of pointers and indirection. In my next journal entry, I will discuss pointers and arrays which will then allow us to finally move on to the last part of our EEPROM I2C project!

April 11

A Short Introduction to Troubleshooting Docker Networks

Docker Network Diagram- depicting the major interfaces involved.

I recently just built an unRAID rig which has now been deployed to my DMZ. It’s great, but something I have been struggling with is that periodically my FileZilla docker container will be unable to connect to my FTP server, erroring out with “ENETUNREACH – Network unreachable”. It seems to be exacerbated by large file moves (like when I’m moving whole directories).

ENETUNREACH - Network unreachable error in FileZilla
ENETUNREACH – Network unreachable error in FileZilla

I started by researching the error message but there isn’t a whole lot on the formal definition of “ENETUNREACH – Network unreachable”. Presumably because “Network unreachable” is self-explanatory. There are a lot of forum posts about FileZilla being blocked by an antivirus suite’s software firewall, however, so I feel comfortable with the error description given. There are no subtleties to the message “ENETUNREACH – Network unreachable”- it simply means that FileZilla can’t connect to the network (i.e. it can’t dial out).

Docker Networking Overview:

Let’s begin with a brief introduction to Docker networking. At a high level, Docker containers are very similar to virtual machines (VMs) but without the overhead of having to run a duplicate OS. The diagram below, shows our Docker containers which run inside of our Host:

Basic Docker network diagram showing three interfaces: the interface between the container and the host, interface between the host and router, and interface between the router and internet.
Basic Docker Network Diagram

The above diagram shows a basic schematic of a typical Docker network. For my containers, I have the network configuration set to bridge mode which means that, at least for networking purposes, the Host acts like a glorified router to the Container. Put another way, when I want to communicate with the container over the network, I just put in the IP address of my Host and a port specific to that Container. The Host then takes that traffic and forwards it to the internal IP address of the Container at whatever port it’s listening on. This is also true for outbound traffic. In essence, bridge mode on Docker is nothing more than network address translation (NAT). Something you’re undoubtedly familiar with if you have a homelab or are self-hosted like I am.

Three network interfaces in Docker networks

Back to our problem. We know FileZilla can’t dial out. Referring back to the diagram, we see that there are three checkpoints where traffic could be failing: (1) the interface between the Docker container and Host, (2) the interface between the Host and the router, and (3) the interface between the router and the WAN/internet.

Troubleshooting the Connection:

Since we know our problem is in the outbound direction, let’s focus in that direction. What’s the simplest way to check for a connection? Ping it!

Testing each network interface we identified above in turn:

1) Ping the Host from the Container:

We can accomplish this by executing a command in the Container. This can be accomplished in unRAID by clicking on the Container and selecting console. If you aren’t running unRAID, this is the same as running <docker exec -it [container name here| FileZilla for me] sh>. Now you can simply ping the Host using <ping [insert IP address of Host on LAN here]. Pinging the Host in my case greeted me with a response so we know that connection isn’t the culprit:

Pinging Docker Host from Container.
Pinging the host from the container. We have a response so no problem here!

Note: You can also ping an address on the internet. When I did that here, I did not receive a response, confirming that the outbound connection is indeed broken somewhere along the chain:

Pinging website from Docker container.
Pinging a website from the container. No response. So we know something is broken along the chain.

2) Ping the Router from the Host:

This is straightforward enough. In unRAID you just open your console/terminal on the Host (“Tower”) and ping the router’s IP address. I also received a response here so we know that we have a connection to router. We probably already knew this since we could connect to the Host over the network in the first place though.

Pinging an address on the internet also received no response meaning my server was not able to access the internet. This further confirmed that the chain is broken still further upstream.

3) Ping an address on the WAN/internet from the Router:

This can be a little bit trickier. Thankfully I run dd-wrt on my routers so I can easily initiate a terminal session on the router with <terminal [insert router local IP address here]>. Pinging an address on the internet resulted in no response here as well! Well, we know this guy is the last interface in the chain so we know the problem is with him.

[There’s a little bit more to know here which is more specific to my network’s “unique” architecture. My server resides in a physically separate part of my home- away from my core network and my home isn’t physically wired for ethernet, therefore to adapt and overcome, I have dd-wrt set up to create a client bridge with this router (the router mentioned above is actually the client router in the client bridge). For those of you who don’t know what this is, it means that all clients connected to my secondary client router behave like they’re physically connected to the primary router. At least that’s the way it’s supposed to work in theory and it most often does- unfortunately, the client bridge mode is notoriously unreliable as is the case here. Had this been the DMZ’s main router at fault, as would be the case on your home network, a big clue would’ve been that I couldn’t access the internet from my computer.]

Note: I also want to point out another potential cause here. If, at any point when pinging an external site on the internet, you found that you didn’t get a response, try pinging an IP address (as opposed to the website address) you know is good. If that works, that suggests a problem with your DNS lookup and you should begin your investigation there.

Rebooting the router fixed the issue and allowed FileZilla to proceed without error. Honestly, the root cause here is that I am using client bridge mode but getting rid of it really isn’t an option for me at this point. I could try upgrading the router to better hardware but I need to let my wallet recover from this server build first. 🙂

Anyway, I thought this made for an interesting case study and demonstrates how breaking a system down into its simple parts allows for effective troubleshooting. With a methodical process and understanding, every problem can be overcome.

Improvise. Adapt. Overcome.

April 10

How to Share Part of Your Plex Libraries without Giving Users Complete Access to Your Full Library: Symbolic Links with Plex and Docker Containers

By now you’ve discovered some of the many shortcomings of Plex. One of which is that you can’t share individual videos with your friends or family- you have to give them access to the entire library! It’s even worse if you have kids and all of your movie content is stored under a single “Movies” library: “Saving Private Ryan” is going to be right next to “Tangled”.

Well, thankfully, there’s a way around that. Enter “symbolic links”. I’ll do another tutorial on what a symbolic link actually is in the future, but for now, think of them as Unix’s version of a shortcut- it merely points to the filename of another file located somewhere else. This means that to share specific content with users, all we have to do is create a library folder specific to that user and place a bunch of symbolic links in it referencing the content in the libraries we have stored already.

So yes, we do still have to create a new library to share with users, but the good news is that we can do this all without having to make a duplicate of the file and taking up valuable storage space!

Creating a Symbolic Link:

If your Plex Media Server is a normal install (i.e. not running in a Docker container), creating a symbolic link is pretty straightforward. Just create your new library directory and navigate to it in a terminal. Then just execute the following command:

ln -s [insert file source location here] ./

The above path in the [insert file path to video here] can be either a directory or a video file itself. Just create a symbolic link for each individual directory or video you want to share. That’s it!

Symbolic Links with Docker Containers:

Normally symbolic links (“symlinks”) are quite transparent to applications in Linux (that’s the whole point), but in the case of running Plex on an unRAID server, we have an additional challenge: Plex is running in a Docker container. Plex Docker containers have a volume mapping configured to map the media path in the Plex container to the user share on the unRAID host. The chances are that the two file paths specified for the container and the host don’t have the exact same parent directory (if they did that would partially defeat the point of the abstraction that Docker containers give you).

Docker Container Volume Mapping
Docker Container Volume Mapping: Host Path ‘= Container Path

The above configuration shows what I mean a little better. As you can see, the Host path does not equal the Container path. The consequence of this is that the Docker Host and the Docker Container have two different systems of reference. This problem we run into when we run the command “ln -s” above is that the symbolic link created follows an absolute path so when the Plex Docker container starts to follow this path that’s specified based off the Host’s directory tree, it fails. I think an example will help illustrate:

Normal "absolute" symlink with absolute file path.
Normal symlink with absolute file path.

The Fix: Relative Symlinks

Thankfully this is easily fixed with the addition of the relative (-r) argument to the ln command which will instead give us a relative symbolic link:

ln -rs [insert file source location here] ./

The example below demonstrates the difference compared to a “regular” symlink:

Relative symbolic link giving us a relative file path that will work in a Docker container.
Relative symlink giving us a relative file path that will work in a Docker container.

This allows the Docker container to translate the file path specified in the symbolic link into its own internal file structure. That’s it!

In a future tutorial I will explain more of the technical details behind symlinks and you’ll see equivalent alternatives to the command above.

April 6

Expand Your Arduino’s Storage with an External EEPROM (AT24C256): A Tutorial in How to Use the I2C Protocol

In a previous post, we covered how to expand your number of analog inputs by using an external ADC over the SPI bus. Today I want to demonstrate how to use the I2C protocol while simultaneously teaching you how to read a datasheet. After all, there will come a day when you want to use a new device and a library won’t already exist for it.

First of all, let’s establish the Arduino library we will be implementing to communicate with the AT24C256 EEPROM. For this tutorial, we will be using the Wire library to implement the I2C protocol (Mental shortcut: I2C protocol = Wire library). Let’s begin!

1. Review the Datasheet

First, let’s survey the land and have a look at the AT24C256 datasheet. When I look at a datasheet, one of the first things I do is figure out what commands I want to use. In this case, I know I want to do a page write. A page write allows you to write multiple bytes of data all at once, as opposed to a byte write which is all sorts of tedious. I’m a visual learner so once I know what command I want to use, I jump to the figure that parses it out. The figure for Page Write is shown below:

Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Just looking at this figure I see that every I2C message starts with three words: a device address, a first word address, and a second word address. This preamble is then followed with the actual data you want to write.

2. Writing Out the Code for the Address (i.e. the preamble)

From that diagram alone the code practically writes itself. It’s just a matter of knowing which methods to call in the Wire library. Let’s implement the code one word at a time:

Device Address:
Device Address Message
Device Address Highlighted in Red; Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Focusing on this part of the diagram, we see that the device address begins with a 1 (the line is high), followed by a 0 (line is low), 1, and so on so that we ultimately end up with 0b1010000. The last two least significant bits (LSBs) are 0s but can actually be changed to allow for up to four of these EEPROMs to be on the I2C bus. This is also shown separately in the datasheet:

Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

“But wait!” you say, “That’s only 7 bits. A byte is eight bits. What gives?!”

Well, you are technically correct- to make the full byte, we should have an eighth bit that tells the device whether we want to read or write (the R/W in the LSB position shown above), but we are using the Arduino Wire library which expects a 7-bit address. If you are given an 8-bit address like in Figure 7 above, you need to bit shift one position to the right to get a 7-bit address. Notice that you don’t have to do this minor bit of mental gymnastics if you just look at the original diagram in Figure 9. The Wire library will handle appending the R/W bit based on what method you end up calling.

So to begin our I2C transmission, we simply call the following:

Wire.beginTransmission(0b1010000);
First Word Address:
First Word Address Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Let’s just assume we’re starting off with a blank slate. So we’re just going to start from an initial address of 0. Honestly, we might as well go ahead and address the second word address here while we’re at it. We can think of the first and second word address as one giant address: i.e. two bytes together (a.k.a. an int). Looking at where the MSB of this “giant” 16-bit would fit in the diagram above (where the * is), we see that this value is a “DON’T CARE BIT”. Since I’m using a 256K EEPROM, our MSB for our address begins where the cross is (so we actually have a 15-bit address). Therefore our 15-bit address comes out to: 0b000000000000000 (i.e. 15 0’s).

Getting back to the first word address, we can see that that the AT24C256 still expects a full 8-bit byte even if it doesn’t particularly care what that first bit actually is. We can accomplish this with the following code:

Wire.write(0b0000000); // 7 bits of 0s; this method takes a byte though so it will still transmit a byte's worth of 0s.
Second Word Address:
Second Word Address Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

Now we just transmit the second word address (i.e. the remaining 8 bits of our 15 bits from above):

Wire.write(0b00000000);

Granted this was a pretty trivial address, but I think you get the idea.

3. Write the Data

Now, I assume you probably didn’t come here to just address the EEPROM. You want to actually write some data to it right? Well, now you can do that using the same Wire.write calls we’ve done before. We’re finally in this part of the Page Write diagram:

Data Message
Source: Atmel https://www.mouser.com/datasheet/2/268/doc0670-1180619.pdf

This part is rather simple compared to what we’ve done so far. To write data we simply use Wire.write again:

Wire.write(15);

The beauty of using Page Write is that we aren’t limited to just writing one byte at a time (which is what happens when you use Byte Write). With Page Write, once you’ve queued up the preamble, you can then queue up an entire array all at once. For example:

Wire.write("ABC");

4. Putting It All Together:

Putting it all together, the hard-coded Page Write function we just wrote could look something like this:

void writePage(){
  Wire.beginTransmission(0b1010000);
  Wire.write(0b0000000);
  Wire.write(0b00000000);
  byte a = Wire.write("Hi");
  Wire.endTransmission();
}

Don’t forget the Wire.endTransmission(); at the end so that we actually send out our data over the I2C bus. Also don’t forget to designate your Arduino as the master device in your setup block with Wire.begin();.

This tutorial has become a bit longer than I originally intended. Let’s call it a night and we’ll pick back up in the next post with how to request (read) data back off the EEPROM!

March 31

Interfacing an 8-bit Microcontroller with a 10-bit Device over the SPI Protocol

10-Bit ADC MCP3008 Interfacing with Arduino over SPI
10-Bit ADC MCP3008 Interfacing with Arduino over SPI

In Simon Monk’s Programming Arduino Next Steps: Going Further with Sketches, Simon introduces the SPI protocol by way of interfacing the Arduino with a 10-bit ADC (specifically the MCP3008). Unfortunately, by choosing a 10-bit ADC to introduce the SPI protocol, he ultimately fails the main objective of getting students started with the SPI protocol. Additionally, without much explanation or reference to the MCP3008’s datasheet, his implementation of the algorithm feels awkward and clumsy. Mostly because the code is just unintuitive.

In this tutorial, I offer an improved implementation of the SPI protocol (at least in terms of readability for the new learner) for interfacing a 10-bit ADC with an 8-bit microcontroller such as the Arduino. Additionally, I will attempt a more thorough explanation behind how the code works. This tutorial assumes an understanding of basic SPI protocol implementation (I’ll save that for a future tutorial).

Background Information:

First off, let’s tease out the big assumptions. What does a 10-bit ADC mean? Simply put, it means that the analog-to-digital converter takes an analog voltage and represents it as a 10-bit digital value. What exactly does that mean? It means that it represents the analog input as a value between 0 (at 0 V) and 1024 (at saturation- i.e. your max voltage). Where does the number 1023 come from? Simple- it’s the maximum value that can be represented in binary by 10 bits all set to 1: (2^9)+(2^8)+(2^7)+(2^6)+(2^5)+(2^4)+(2^3)+(2^2)+(2^1)+(2^0). Add them all up and you get 1023.

Now, why is that a problem with a microcontroller such as an Arduino? Arduino, like most microcontrollers, speak in 8-bit (i.e. byte)-sized words. If our master device (the Arduino) is speaking on over the SPI protocol in byte-sized words and the slave chip speaks back in 10-bit words, we have an offset problem. The Arduino is done talking after 8 bits but our 10-bit ADC still has two bits left to transmit.

Outlining a Rough Algorithm:

As always, the best way to learn is by example. Let’s start by taking a look at MCP3008’s datasheet. Specifically read Section 5.0- Serial Communication and 6.1- Using the MCP3004/3008 with Microcontroller (MCU) SPI Ports.

Yeah, yeah, I know you aren’t going to read them, but you really should, I promise you they’re worth it. Here are the highlights:

The first clock received with CS low and DIN high will
constitute a start bit. The SGL/DIFF bit follows the start
bit and will determine if the conversion will be done
using single-ended or differential input mode. The next
three bits (D0, D1 and D2) are used to select the input
channel configuration.

Microchip: https://cdn-shop.adafruit.com/datasheets/MCP3008.pdf

Terminology notes: CS = “Chip Select” (aka SS- “Slave Select” in SPI nomenclature); Din = MOSI (Master Output Slave Input) in SPI nomenclature.

From just this part of the datasheet, we can start thinking about how we want to implement our algorithm for getting data off the ADC over the SPI bus. We are told that the chip recognizes the start bit based off SS going low and the first high MOSI bit it receives. Therefore our code could look something like the following:

digitalWrite(chipSelectPin, LOW); // Select the ADC by setting SS low.
SPI.transfer(0b00000001); // Fire off the start bit.

Now that we’ve woken up the ADC, we need to tell it what we want. We know from our datasheet that the next bit the ADC expects to see is the mode selection bit- in our case, we want to pass 1 since we want single-mode. The remaining three bits the ADC expects to see are the analog input address on the ADC. In my example, I am using input 0 so the three-bit address is simply 000. Knowing this we can then simply add the following line to our code:

SPI.transfer(0b10000000);

But wait, it turns out that our ADC will start responding and giving us an output within this byte, so we actually need what the transfer method returns. Therefore we need something more like this:

byte readingH = SPI.transfer(0b10000000);

Now that we’ve told our ADC what we want, and we’ve even started to receive data from the ADC, we’ll need to continue letting the ADC continue to speak its 10-bit message. We can accomplish this by flushing the ADC with a byte of zeros:

byte readingL = SPI.transfer(0b00000000);

Now the question is, at what point in the transfer’s output does an individual bit start to represent something useful? (A professor might ask this same question as, “What bit in the byte output is our MSB (most significant bit)?” You can read the paragraphs in the datasheet but honestly it’s surefire path to driving yourself mad; speech is too imprecise. As usual, a picture is worth a thousand words:

SPI Communication with the MCP3008 using 8-bit Segments: https://cdn-shop.adafruit.com/datasheets/MCP3008.pdf

The part we’re really interested in are the rows labeled “MCU Transmitted Data” (highlighted in red) and “MCU Received Data” (highlighted in green). This datasheet demonstrates the cleanest way to communicate with the ADC over SPI by using three one-byte words.

As you can see, we’ve already covered sending the first 8-bit word when we transmitted our start word (SPI.transfer(0b10000000);) and we have now just transmitted our controlByte (byte readingH = SPI.transfer(0b10000000);).

If you look at the corresponding byte (the second one) in MCU Received Data row, you’ll see that the last two bits of that byte (readingH) are our MSBs: B9 and B8. We then flushed the ADC with our third byte with nothing but zeros to get the remaining 8 bits.

Now it’s time to reconstruct the output message from the ADC in our Arduino. This can be accomplished with a combination of masking and bit shifting (You see?! Bit masking and shifting aren’t just a useless academic exercise! They actually have a purpose!)

Starting with readingH we identified the MSB as being the last two bits. Therefore, we simply need to mask readingH as so with a bitwise & so we only get the last two bits:

readingH = (readingH & 0b00000011)

Since these two bits represent the two most significant bits of our 10-bit value, we need to shuffle them over to the appropriate bit position with << 8.

readingL is even easier since the entire byte is the LSB. Now all we need to do is add readingH and readingL together to get our output. Remember, the data type of our addition operation is an int (i.e. two bytes) because we now have a 10-bit output. The whole operation can be done in a single line:

int reading = ((readingH & 0b00000011) << 8) + (readingL);

Putting It All Together:

Putting the above together in a single program:

#include <SPI.h>
const int chipSelectPin = 10;

void setup() {
  Serial.begin(9600);
  SPI.begin();
  pinMode(chipSelectPin, OUTPUT);
  digitalWrite(chipSelectPin, HIGH);  //Immediately set CS (slave select) high so the ADC isn't selected on startup.
}

void loop() {
  int reading = readADC(0);
  Serial.println(reading);
  delay(1000);
}

int readADC(byte channel){
  byte startBit = 0b00000001;
  byte controlByte = 0b10000000 | (channel << 4); // First bit 1 gives us single-ended mode on ADC; the next three bits represent the ADC's analog input
  byte flushByte = 0b00000000; // Flush the ADC to get the remaining byte output.
  
  digitalWrite(chipSelectPin, LOW); //Initiate SPI protocol by dropping slave select low.
  SPI.transfer(startBit); // Fire off our start bit.
  byte readingH = SPI.transfer(controlByte);  // Push in our control byte which tells the ADC what mode to use and what channel we want.
  byte readingL = SPI.transfer(flushByte);  // Get the rest of our output from the ADC by flushing it with a byte of 0s.
  digitalWrite(chipSelectPin, HIGH);  // After flushing, immediately deselect the chip so that it doesn't continue.

  int reading = ((readingH & 0b00000011) << 8) + (readingL); // Per datasheet, we know that only the last two bits of our first transfer contain useful info. The second byte is all useful.

  return reading;
}

When you pull up the serial monitor and start moving the potentiometer around, you should see something like this:

Arduino serial monitor displaying ADC input in decimal format
Arduino serial monitor displaying ADC input in decimal format

I hope this helps clear up any confusion with using the SPI protocol and especially when using a 10-bit ADC with an 8-bit microcontroller.

March 28

How to Upgrade to Ubuntu 19.04 (Disco Dingo) Beta (or any other Ubuntu Beta Version)

Ubuntu 19.04 (Disco Dingo) Beta is available today! I do this so infrequently that I always have to look up how to do it, so for future reference I’m tossing my notes up here. Here is how to upgrade to an Ubuntu beta version.

1. Make sure your current version of Ubuntu is up to date.

This can be accomplished by opening a terminal (Ctrl + Alt + T) and running the following command:

sudo apt-get update
sudo apt-get upgrade

2. Launch update manager

Next, launch update manager with the code below in the command box (Alt + F2). The -d option tells it to look for upgrade distributions:

update-manager -d
Prompt to upgrade to a new version of Ubuntu

Just click the “Upgrade” button and you’re done!

FYI, this can also be accomplished in the terminal (as opposed to the command box) with the following:

sudo update-manager -d

That’s it. Don’t forget to re-enable third party sources after you’ve upgraded.

Category: Linux | LEAVE A COMMENT
March 2

How to Fix ESP32 Compiler Error in Arduino IDE: /heltec/esp32/tools/esptool/esptool.py No module named serial.tools.list_ports

So you’ve installed one of the various ESP32 board managers for the Arduino IDE. You go to compile your code and you get the following message:

Trackback arduino line 35, in <module> import serial.tools.list_ports as list_ports

/heltec/esp32/tools/esptool/esptool.py No module named serial.tools.list_ports

You’ve already installed pyserial so what the heck is going on? Well, fun fact, your Arduino ESP32 board manager uses Python 2.7 and you probably have Python 3.+ installed. The Arduino IDE is trying to use pyserial and it can’t find it because it’s using Python 2.7 and you installed pyserial for use with Python 3. Thankfully the fix is simple enough.

  1. First, we need to download pyserial. Go to https://pypi.org/project/pyserial/#files and download the latest .tar.gz file you see there.
  2. Unpack the .tar.gz file you just downloaded. This will give you your install folder.
  3. Open this folder in your terminal and run the following command so we install pyserial for Python 2.x:
sudo python2 setup.py install

4. That’s it! Close the Arduino IDE and reopen it and you should be good to go!

February 2

How To Update/Install FileZilla on Ubuntu

FileZilla is an incredibly useful FTP client for transferring files between your workstation and servers. In this tutorial, I will walk you through updating/installing FileZilla on Ubuntu without using the repository. In general, if you want the latest and greatest features, try to avoid repositories- the apps in repositories are often outdated. I also feel like a repository is a crutch in that it obfuscates how your software is actually installed on your Linux system.

I can already hear the outrage now; I’m not saying that repositories are worthless. They greatly reduce maintenance when it comes to keeping your system (relatively) up-to-date. Downloading, extracting, and compiling every application from source would be hugely impractical. I use repositories for things that either I don’t use very often or that I don’t care about having the latest version of. For apps that I use often, where I care about having the latest, I handle those manually. Now, on to the tutorial.

1. Obtain your update files.

Obtain your update files. If you already have FileZilla installed, FileZilla checks automatically for updates at launch and downloads them to your home downloads folder. If you don’t have FileZilla already, download it here.

2. Navigate to your downloads folder.

Navigate to your Downloads folder and find your FileZilla tar file. I’ll admit I use the Files GUI app that comes with Ubuntu most of the time. Right click and select “Open in Terminal” (or just open a terminal with Ctrl + Alt + T and just type cd ~/Downloads/).

3. Extract your tar file.

Extract your tar file using the following command:

tar -vxjf FileZilla_3.40.0_x86_64-linux-gnu.tar.bz2

This will extract the file to a directory in your Downloads directory called FileZilla3. You should now have the following:

Extracted FileZilla3 files in ~/Downloads/ directory.
Extracted FileZilla3 files in ~/Downloads/ directory.

Notice this extraction contains a bin directory, implying that it’s ready to run (no compilation necessary).

4. Move your extracted files to their final location.

Let’s move this folder to our /opt/ directory with:

sudo mv ./FileZilla3/ /opt/

But wait! If you’ve already, installed, you’ll get the following error:

mv: cannot move './FileZilla3/' to '/opt/FileZilla3': Directory not empty

Even with sudo, mv will refuse to merge a directory. It’s a nice guardrail. In our case though, we do want to merge. For that, we’ll use rsync:

sudo rsync -a ./FileZilla3/ /opt/FileZilla3/

Warning: DO NOT FORGET to add the /FileZilla3/ directory to /opt/ like it shows above. If you simply did /opt/ you’d wipe out your entire /opt/ folder and be left with only FileZilla3. 

And with that, we’re done!

Category: Linux | LEAVE A COMMENT
January 16

Setup an NGINX Reverse Proxy on a Raspberry Pi (or any other Debian OS)

If you’re running a web server out of your homelab (and you should), you really should consider running your servers behind an NGINX reverse proxy. Honestly, this should be the first thing you build in your homelab. It doesn’t take a lot to setup- NGINX is so efficient it can even be run on something as simple as a Raspberry Pi and it pays you back in dividends once you’ve got it up and running.

What Does a Reverse Proxy Do?

A reverse proxy serves as a sort of dispatcher by acting as a central contact point for clients. Based on the information requested by the client, it then routes the request to the appropriate backend server and makes sure the backend server’s response makes it back to the appropriate client.

Prototypical NGINX reverse proxy diagram

What are these dividends you speak of?

A reverse proxy can give you additional flexibility, security, and even a performance bump. It can also greatly simplify your deployment:

  1. Flexibility: An NGINX reverse proxy can allow you to host multiple sites/domains with only one IP address. It accomplishes this by listening on a port (usually port 80 for HTTP traffic) and parsing the http request header for the host. Based on the host specified in the header, NGINX can route a request to the proper backend server (in a reverse proxy, this is also known as an upstream server).
  2. Security: By standing between the client and the backend server, the reverse proxy provides a degree of separation.
  3. Improved performance: NGINX can be used to cache static content which means that not only is content returned faster to the client, but since it often means that the upstream server doesn’t even need to be contacted, it can take a lot of the load off your backend servers.
  4. Simplifies your deployment: If you’re hosting multiple sites, an NGINX reverse proxy can greatly simplify your implementation by giving you a single point to manage your traffic. This means you only have to set up port forwarding once and whenever you create a new site, all you have to do is add an additional configuration listing to NGINX. When you implement HTTPS (and you should), instead of having to implement it on every individual web server you have setup, you can handle it all on your NGINX reverse proxy.

Installing NGINX:

Now that I’ve hopefully convinced you to implement an NGINX reverse proxy, let’s get started. I recommend using a dedicated device for this (again, it need not be expensive, even a Raspberry Pi will do) but it helps keep everything clean and compartmentalized. With a clean install of Ubuntu (if using an x86-64/AMD64 device) or Raspbian (if on the RPi), do the following:

1. Update your package list and make sure your device is updated:

sudo apt-get update
sudo apt-get upgrade

2. Depending on what Linux distro you’ve picked, it might have Apache already installed. We don’t want that so uninstall it with:

sudo apt-get remove apache2

3. Install NGINX:

 sudo apt-get install nginx

4. NGINX should start automatically, but just in case, you can start it manually with:

sudo systemctl start nginx

5. Confirm that NGINX is up and running by opening your browser and visiting your IP address. You should be able to get the default page to display by visiting your loopback IP address (127.0.0.1) or the actual IP assigned to your device (available by running the command ‘hostname -I’ in the terminal):

6. Good! Now we have NGINX up and running! If you don’t have a backend web server running yet, then we’re done since you don’t have anywhere for us to send traffic. Come back to this point when you do. But if you do have a web server for us to proxy traffic to/from, continue on!

Configuring the Reverse Proxy:

So you’ve made it this far and you now have an NGINX server running. Let’s set up the reverse proxy part to make this an NGINX reverse proxy and not just a simple NGINX web server:

1. Go to our NGINX sites-available directory:

cd /etc/nginx/sites-available/

2. Create the configuration file. You’ll eventually accumulate a lot of these, so I recommend naming it based on the site that you’re reverse proxying so you can easily find it again:

sudo nano example.com.conf

3. In nano, add the following:

server {
	listen 80;
	server_name: example.com
	location / {
	proxy_pass http://192.x.x.2;
	}
}

server_name is going to contain the domain name of the website clients are going to be requesting. proxy_pass is going to be the local (internal) IP address of the web server that you’re forwarding traffic to. You can also specify a particular port if your web server is running on a non-standard port (example: proxy_pass http://192.x.x.2:82300).

4. For NGINX to actually serve your site with your new configuration, you need to link it to /sites-enabled/ with:

ln -s /etc/nginx/sites-available/example.com.conf /etc/nginx/sites-enabled/example.com.conf

5. Test your configuration to make sure you aren’t getting any errors:

sudo nginx -t

6. Reload NGINX to tell it that the configuration has been updated:

sudo systemctl reload nginx

That’s all there is to it! In future posts, I’ll cover how to setup that upstream web server, how to configure the DNS for your domain, and port forwarding so that you can access your sites on the internet.

As always, feel free to comment if you run into any problems or need help!

Category: Servers | LEAVE A COMMENT
January 15

SQL Lessons to Remember

I had to write a few SQL queries today so I thought I would toss up a few notes as a sort of SQL FAQ. I’ll try to keep these notes updated.

  • Inner joins fail on null matches
    • When you are joining tables, if you want to keep null values from your foreign key (i.e. the thing you’re trying to match your second table on), you need to use a left outer join.
  • Inner joins on one-to-many
    • When joining one-to-many, an inner join will give you a line for each match to the second table; e.g. if I have one row in table A and it matches on two rows in table B, you will get two rows- one row for each match
  • What table should I start with when writing a query?
    • When writing a query, the best place to start is to figure out what the output should look like. This is important because it helps you determine the granularity of your report. Once you’ve done this, now, when you’re faced with a myriad of tables, you start with the table that matches the granularity of the report you wish to write.

Order Matters

Queries are written in the following order:

  • SELECT
  • FROM
  • WHERE
  • GROUP BY
  • HAVING
  • ORDER BY

However, they are processed in this order:

  • FROM
  • WHERE
  • GROUP BY
  • HAVING
  • SELECT
  • ORDER BY

This isn’t just an academic distinction, it can be helpful in troubleshooting your reports when you get either an error or results you don’t expect.