Lo-Fi Python

Memory Monitoring Python Libraries + Tools

If you write Python code, there's probably been a time or two when you saw the dreaded "MemoryError". This happens after one of your Python scripts stops because your computer has no spare RAM to execute it. I recently experienced this frustration whilst trying to write hundreds of thousands of csv files. However, this time I grasped for tools that support smarter memory management. Now, I can watch my computer's memory bounce around with the Windows Resource Monitor. Python has quite a few memory profiling libraries for monitoring memory too!

Python Libraries and Guides

Memory Management Overview, Python documentation

Memory Profiler: "monitor memory usage of Python code"

psutil: "Cross-platform lib for process and system monitoring in Python"

py-spy: "Sampling profiler for Python programs"

pyinstrument: "🚴 Call stack profiler for Python. Shows you why your code is slow!"

Scalene: "a high-performance, high-precision CPU, GPU, and memory profiler for Python"

Glances: "Glances an Eye on your system. A top/htop alternative for GNU/Linux, BSD, Mac OS and Windows operating systems."

Yappi: "Yet Another Python Profiler, but this time thread & coroutine & greenlet aware."

Fil: "A Python memory profiler for data processing and scientific computing applications" (Video)

line_profiler: "Line-by-line profiling for Python"

pprofile: "Line-granularity, thread-aware deterministic and statistic pure-python profiler"

Guppy 3: "Python programming environment and heap analysis toolset"

See also: The Python Profilers, Python documentation

CPython standard distribution comes with three deterministic profilers. cProfile, Profile and hotshot. cProfile is implemented as a C module based on lsprof, Profile is in pure Python and hotshot can be seen as a small subset of a cProfile.

Yappi Github, https://github.com/sumerc/yappi

Windows Tools

Task Manager: Windows process management tool with some memory analytics

Collect Data in Windows with Performance Monitor

Resource Monitor: Windows tool with Memory, CPU, Disk and Network monitoring tabs

Resource Monitor can stop processes from running and view in use, standby (Cached) and free memory. This shows 7 Python scripts running and 49% of total memory is being consumed. Looks like we are running steady and safely below "MemoryError" overflow. We might be able to add a few more scripts with 51% of RAM available!

Memory Tips and Guides

• Use only the data you need. Any data you read in and aren't using is held in memory. The usecols argument in pandas is a great way to read a csv and only use the columns you need.
• Reading data in chunks with the chunksize argument is another way to reduce memory usage for large datasets.
• Measuring the memory usage of a Pandas dataframe
• Some tools are line oriented, others are function oriented. If your code contains large functions, you might favor a line based profiling tool.
• Be aware of the overhead some memory tools may incur. memory_profile was clocked with a whopping 270x slowdown per the Scalene PyCon talk below. The talk shows an awesome comparison of these Python profiling libraries:

Recommended Reading

• Profiling and Analyzing Performance of Python Programs
• profile, cProfile, and pstats – Performance analysis of Python programs.
• Profiling and Analyzing Performance of Python Programs
• 20 Linux Memory Management Command Line Tools
• Random-access Memory (RAM)
• Cache Memory

Conclusion

When you'll see "MemoryError" depends on your computer's hardware, the size of your dataset and what operations you need to script out. Generally speaking, I/O or file reads and writes are more expensive operations.

The tools in this post will help you anticipate how much computing power you have available, monitor your memory consumption more closely and avoid pushing your computer past its limits. You can do things like reading data in chunks and only using the columns you need to reduce your memory consumption. Realizing these tools and strategies can make getting things done with Python a smoother ride.

Installing Debian 9 Stretch Linux OS on a Dell Inspiron Laptop and Configuring the Wifi Network

Yesterday, I converted an 11-year old Dell Inspiron E1505 from Windows XP to Debian 9 Stretch. I may have overwrote my Windows XP OS. I do not care if I lost it since it's a vulnerable and outdated OS, which is no longer supported by Microsoft. I encountered difficulty with getting the wifi to work on Debian, but was able to find a solution using Wicd. Here are the steps I followed to do it all.

Using The Debian Installer-Loader

1. First, back up your Windows computer files. Then download the Debian-Installer Loader Windows executable from the Debian wiki.
2. Click the downloaded executable and follow instructions. I followed the default settings all the way through.
3. You may need to choose your own partitioning settings to ensure Windows is preserved if desired.
4. During installation, choose your Linux collection. I chose Xfce because it seems to be highest ranked among Linux users and "not just helpful for older computers where few system resources are available, but also simply for those who want to get the most out of their systems." Gnome and KDE are other popular options.

After completing installation, restart your computer and select your new OS on boot-up. The following error codes displayed for me while starting up, signaling missing wifi firmware.

ERROR Failed to load firmware!
b43ssb0:0: firmware: failed to load b43/ucode5.fw (-2)
b43ssb0:0: firmware: failed to load b43-open/ucode5.fw (-2)
b43-phy0 ERROR: You must go to https://wireless.wiki.kernel.org/en/users/drivers/b43#devicefirmware and download the correct firmware for this driver version.

It took me a few boot-ups before I realized what this error message meant. In the rest of this post I am trying to figure out and fix what is wrong before I saw the error message. I enjoyed learning how to introspect Linux networks, but if I were trying to fix this problem again, I'd go to directly to *`this page <http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/>`__*, which is linked to from the link in the error message, and try the solution there first.

su -
apt-get install sudo -y usermod -aG sudo yourusername

uninstall nm
sudo apt-get install -d --reinstall network-manager network-manager-gnome
install wicd
sudo apt-get install wicd-gtk

sudo ifconfig eth0 up

Troubleshooting Linux Wifi & Inspecting Your System

Now, let's check for enabled network interfaces. "wlan0" is usually the name of the wireless interface. Does wlan0 show when you enter this command? If not, then you may need to update your wifi firmware. This was the case for me. Below is an output where wlan0 is correctly configured.

sudo ifconfig

eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 00:25:a5:cf:38:7d txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 4 bytes 240 (240.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4 bytes 240 (240.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 191.142.1.297 netmask 255.255.255.0 broadcast 182.138.5.255
inet6 2601:241:8c00:50ea:21a:92ff:fe0d:7531 prefixlen 64 scopeid 0x0
inet6 fe80::22a:42tf:fe0d:7531 prefixlen 64 scopeid 0x20 ether 00:2a:92:2d:45:51 txqueuelen 1000 (Ethernet)
RX packets 8509 bytes 4639778 (4.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6206 bytes 923792 (902.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Check wlan0 is not blocked on kill list.

sudo rfkill list all

Some computers have a "kill switch" for wifi. This command lists any blocked interfaces. If it is blocked, this thread might be useful. If nothing shows when you run this command, or if you see wlan0 is not blocked, carry on.

Check which wifi controller you have. This thread provides more info on what this means.

lspci -nn | grep -e 0200 -e 0280

03:00.0 Ethernet controller [0200]: Broadcom Limited BCM4401-B0 100Base-TX [14e4:170c] (rev 02)
0b:00.0 Network controller [0280]: Broadcom Limited BCM4311 802.11b/g WLAN [14e4:4311] (rev 01)

Find your system architecture. This determines which firmware you should download in the next step.

sudo dpkg --print-architecture

First, read here to determine the right packages for your Linux system. Then download the appropriate missing wifi firmware. For Debian, I downloaded the two packages below.

1. b43-fwcutter 2) b43-installer

"cd" into the directory with .deb files. Run the below commands to install the new firmware, then reboot your computer. The last two commands are adapted from this thread.

sudo dpkg -i firmware-b43-installer_019-3_all.deb
sudo dpkg -i firmware-b43-fwcutter_019-3_i386.deb
sudo modprobe -r b43
sudo modprobe b43

Edit Wicd preferences to set wlan0 as the wireless interface if needed.

Success! Wireless networks are now showing.

Wrapping Up

I'd like to thank the awesome people who contributed to the Debian Installer-loader and all the help in Linux forums that enabled me figure this out. I'm new to the world of Linux but already enjoying diving into this operating system. Its ability to do just about anything from the command line are a lot of fun. I am now running two Linux systems, one on my Dell and another running Ubuntu that I installed on a Chromebook with Crouton. Both have been relatively painless to set up. It this case, it turned a sluggish laptop into a very capable machine. They should call it Lit-nux :)

Full Disclosure

This worked on my computer for a few days before the keyboard stopped working correctly on my computer. Typing became impossible because the keys didn't work or entered the wrong letters when pressed. I'm not sure what the cause of it was, but consider that before attempting this on a machine. Be prepared to lose it. If you really need the machine to be functional, it may not be a great idea to try this. This was attempted on an old beat up computer. I would try this method of porting a Windows machine to Linux again as a salvage project or on a low-risk Windows machine if I had one lying around.

Packaging Python as a Windows App via Pyinstaller

My research found that for creating a single-file Windows executable from a .py file, the front-running Python library is Pyinstaller. It worked on my Windows 7 PC. My program used a Gooey GUI, but many of the popular Python GUI libraries work as well.

Installation:

python -m pip install pyinstaller

Using the build.spec file and starting Pyinstaller:

Most examples I found used a build.spec file. Think of this as the blueprint by which your app will be constructed. It provides the compiling settings, images and any other data necessary for assembling your app. The format for passing the .spec file to Pyinstaller in pseudo-code:

pyinstaller (run mode options) (.spec file)

pyinstaller build.spec

Establishing a debugging loop with Pyinstaller

Debug mode can be set from the command line:

pyinstaller -debug build.spec

OR by passing debug=True to the EXE() function of the build.spec. I used the second option. See my full build.spec file at the bottom of this post. Pyinstaller displayed a lot of error messages while compiling my app, but it still compiled into a working .exe.

To see your app's error message, run the resulting your_app.exe from the command line. You can find it in the 'dist' folder that pyinstaller creates when you pass it the build.spec file. Set the dist folder as your working directory, type your_app.exe and hit enter. Once you are reading and fixing error messages, you're well on the way to creating your own desktop app.

Flushing sys.stdout/Python printing for Windows:

Python's design requires some code to play nice with Windows when it prints a statement. Simply add this to your .py file. I used write mode 'w'. What worked for me: don't pass 0 to fdopen(). This was contrary to Gooey's instructions.

1
2
3
4

import os
import sys
nonbuffered_stdout = os.fdopen(sys.stdout.fileno(), 'w')
sys.stdout = nonbuffered_stdout

Fetching the local user's system information:

In order to run on any user's system, we need to grab their local file paths. I accomplished this by referencing the sys._MEIPASS via the below code I found from a Stack Overflow post.

1
2
3
4
5
6
7

def resource_path(relative_path):
    """ Get absolute path to resource, works for dev and for PyInstaller"""
    base_path = getattr(sys, '_MEIPASS', os.path.dirname(os.path.abspath(__file__)))
    return os.path.join(base_path, relative_path)

temp_folder_path = getattr(sys, '_MEIPASS', os.getcwd())
relative_path = resource_path(temp_folder_path)

Adding library-specific patches:

1. Below is the fix I found for the Pandas library, which I added to my build.spec:

1
2

hiddenimports=['pandas._libs.tslibs.timedeltas','pandas._libs.tslibs.nattype',
'pandas._libs.tslibs.np_datetime','pandas._libs.skiplist']

2. The Gooey library needs some special code added to the build.spec for its images and languages data. More details are found in this blog post, written by the author of the Gooey library.

Last, but not least: don't panic.

Compiling Python to Windows binary code sounds like a dauntingtask, but it wasn't nearly as complex as I feared. The folks behind Pyinstaller have done a great job of making it possible and, dare I say, simple. Stay calm, drink some coffee, dig in and welcome the challenge with a trial and error mentality. I was able to figure it out over the span of one Saturday. Good luck.

Useful Resources:

1. Pyinstaller Github - If Things Go Wrong
2. Pyinstaller Documentation:Using Pyinstaller Arguments
3. Gooey Pyinstaller Instructions
4. Pandas hiddenimports Discussion

Caveats:

1. You should compile the program on the operating system it will be run on. There are options for creating a multi-os compatible package, but I did not attempt them.
2. Windows 7 is proven to work with Pyinstaller, per the documentation. It's also what I am running on. Other Windows systems older than Windows 10 may be less reliable.
3. I experienced trouble when passing arguments from the command line to pyinstaller and have so far been unable to get the console window to hide. Neither the -w, --windowed, --noconsole arguments seemed to work. I will update if I am able to find a solution.
4. Now that I am testing my compiled app, I am seeing 10x performance slowdowns when running as the final .exe vs. the original .py file. But at least the program runs correctly and still does so relatively quickly.
5. I also received the error: "Fatal error: Pyinstaller does not include a pre-compiled bootloader for your platform." I fixed this by upgrading to the latest version of Pyinstaller:

pip3 install --upgrade pyinstaller

My full build.spec file, modified from here :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# -*- mode: python -*-
import gooey
gooey_root = os.path.dirname(gooey.__file__)
gooey_languages = Tree(os.path.join(gooey_root, 'languages'), prefix = 'gooey/languages')
gooey_images = Tree(os.path.join(gooey_root, 'images'), prefix = 'gooey/images')

a = Analysis(['your_program.py'],
             pathex=['C:\\Python36\\Lib\\site-packages\\your_appdir'],
             hiddenimports=['pandas._libs.tslibs.timedeltas', 'pandas._libs.tslibs.np_datetime', 'pandas._libs.tslibs.nattype', 'pandas._libs.skiplist'],
             hookspath=None,
             runtime_hooks=None,
             )
options = [('u', None, 'OPTION')]
a.datas += [('program_icon.ico', 'C:\\Python36\\Lib\\site-packages\\your_appdir\\program_icon.ico',  'DATA'),
            ('config_icon.png', 'C:\\Python36\\Lib\\site-packages\\your_appdir\\config_icon.png','DATA')]

pyz = PYZ(a.pure)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          options,
          gooey_languages,
          gooey_images,
          name='ADD YOUR APP NAME HERE',
          debug=True,
          strip=False,
          upx=True,
          console=True,
          icon=os.path.join('program_icon.ico'))