Building MicroPython Binary | ESP8266

Using Arduino IDE for microprocessors' programming in embedded C is something we've all been accustomed to by now. But, using Python with ESP8266? Sounds exciting, yet? Well, it definitely got me too excited when I first came across. The thought of using the libraries and syntax I've gotten so comfortable with using for so many years, now also can be used to interact with some hardware, was just mind-blowing.

In this blog post, we'll be exploring 2 of my favorite stuff from two different worlds:

1. ESP8266 - Hardware

2. Python - Software

Now, if you're already wondering why I'd want to rebuild the binary when its already available here, your thoughts are in the right place. I haven't mentioned the reason, yet.

P.S.: In this post, I'm going to concentrate on building the binary and not showing you how to make this project.

Project

I am coding an implementation of ESPAlexa to use my ESP8266 as switchboard controller using relay module. Sounds simple, doesn't it? Let's get to it.

Problem

Since the microcontroller using where this library must be used, doesn't get to utilize pip package manager, we are required to manually create this file on the filesystem and then import it into our code. Again, sounds simple, doesn't it?

In the screenshot above, espalexa.py file is available in the same folder as the current file, testalexa.py. Ideally this import should have been as successful as line 1, i.e., import time. But, here is what happens when this is executed on ESP8266

MemoryError! Bang, that's something we very very rarely see on our Windows machine (being a good programmer ;)). On digging a lil bit deeper, echoing with my inner voice, the cause was pretty clearly mentioned here. In simple words, we cannot import such a big file at runtime. Instead such big imports must be precompiled (cross-compile) and loaded with firmware files. And, how do we modify the firmware file?

Mm, its been a while that I've gotten out of my comfort zone. Let's do it!

SDK Setup

Here are the sequence of tasks I performed. You can follow the project's GitHub page's building section but you will need some help overcoming the issues and that's what I'm here for.

1. Create a Ubuntu VM - my version 20.04.4. I gave it 32 GB of HDD and 4 GB RAM - gcc seems to require 4 GB, I read.

2. As per the official Git page, here's the next command (check Git since this might be outdated)

sudo apt-get install make unrar-free autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python-dev python python-serial sed git unzip bash help2man wget bzip2

This throws an error

E: Unable to locate package python-serial

I modified the command to install python3-serial instead of python-serial. Seemed to have done the job, at least for now. So, the command is

sudo apt-get install make unrar-free autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python-dev python python3-serial sed git unzip bash help2man wget bzip2

3. Since our version of Ubuntu is greater than 14.04, run command

sudo apt-get install libtool-bin

4. Next,

git clone --recursive https://github.com/pfalcon/esp-open-sdk.git

5. Then,

cd esp-open-sdk
make

This gave me the following error:

Clearly, something was wrong here. Running bash --version yielded the version as 5.0.17

The issue seemed to have originated inside crosstool-NG folder. Checking the configure.ac file directly led me to culprit at line 193

replace the highlighted section with ([0-9\.]+). So, the resulting line

                     |$EGREP '^GNU bash, version ([0-9\.]+)')

save the file and execute this again

make

Execution goes on for a while. But then, as they say, every good looking workaround requires another workaround. The process went on without a problem for about 2 mins before throwing the next error

Upon checking the build.log file under esp-open-sdk/crosstool-NG folder, it seemed like there was a connectivity issue to server where the file was expected to be available

I looked up the file isl-0.14 and manually downloaded it to the folder:

/esp-open-sdk/crosstool-NG/.build/tarballs/

using command

wget https://src.fedoraproject.org/repo/pkgs/isl/isl-0.14.tar.xz/3d6b6a1cddd165fae2af5487c5531b09/isl-0.14.tar.xz

Running the command once again after moving back to /esp-open-sdk

make

To make it short, there was another file which could not be downloaded and moving back to

/esp-open-sdk/crosstool-NG/.build/tarballs/

I ran

wget https://github.com/libexpat/libexpat/releases/download/R_2_1_0/expat-2.1.0.tar.gz

and then while in root of this project, i.e., /esp-open-sdk

make

Finally, we've set up our SDK.

As mentioned in the second line of the screenshot above, run the command starting with export. Since, its going to be different for different machine, please don't just copy-paste this line from below:

export PATH=/home/megams/esp-open-sdk/xtensa-lx106-elf/bin:$PATH

Building Firmware

1. While still working from inside the esp-open-sdk folder, run the following

git clone --recursive https://github.com/micropython/micropython.git

This will take a long time and at the end of it, you will now have MicroPython folder created inside /esp-open-sdk with all the required files.

2. Then, add external dependencies

cd micropython
git submodule update --init

3. Building the cross-compiler

make -C mpy-cross

4. This is the step I've been working for this whole time. I'll now need to copy the libraries that I need to be frozen, i.e., written to the firmware.

As you may see, I've copied espalexa.py file to

/esp-open-sdk/micropython/ports/esp8266/modules 

Then we change the directory to

/esp-open-sdk/micropython/ports/esp8266

5. Run

make

This time, we have a new issue. Remember the workaround we took in Step 2 of SDK Setup section? It has now come back to bite us.

So, the workaround we tool was to install serial module for python3 (python3-serial). Now, to force our complication process to use Python3 instead of default python2, we'll be modifying

/esp-open-sdk/micropython/ports/esp8266/Makefile

As you'd see in the image below, on line 217, we're trying to execute esptool.py which is going to be executed by default using the default version on python on Linux, which in this case, is Python 2.7.

Hence, I've made two changes. Firstly, I've added python3 instead of python. Then, I've modified the file path to reflect the absolute path of the file.

(not important) The reason for this being, we'll no longer be able to access the path variable, which contains the esptool.py script (remember the export command), since we're now using it as an argument, and not an actual command.

After these changes, now its time to replace the esptool.py script with its python3 substitute, which you will find here. If you need to take a look at the official page for esptool, here.

Well, its now time to run the same command again, i.e.,

make

By now, it seems like an actual magic word which we've got to practice a million times to get right.

Anyways, that did the job for me.

Annnnnddddd, we finally have the binary!!

Tried flashing it on ESP8266 and was successful, though I still do have more classes to add for my project to take off (or even espalexa script's imports to be valid).

I'm too tired to type anything else now. See ya later!