Lo-Fi Python

There's a Linter for That! Python Linter and Formatter Libraries You Gotta Know

Linters exist for almost every kind of structured document. Python code, SQL, reStructuredText and many data formats can be improved with a linting library. Python's ecosystem has no shortage of formatters and linters.

In this post, I'll highlight the typical players and some niche linters you might not have heard of! The best quality of linters is their ability to find potential problems in your code by enforcing domain-specific rules. In return, you receive a list of hints for fixing anti-patterns, inconsistencies or unused code to remove. Many linters also offer document reformatting capabilities like the ubiquitous black library for Python code.

Python Linters and Formatters You Gotta Know

• ruff: the lean and fast linter library that's gotten a lot of people's attention. For good reason, it's the easiest and fastest Python code linter, running rust under the hood. In my experience, using ruff has made me a more efficient Python programmer. The ruff CLI's --fix argument is a nice touch for automatically fixing up your code. In addition to linting, ruff also now includes Python's fastest built-in formatter! ruff format is a high performance drop-in replacement for black.

Run ruff on your Python script for quick and easy linting.

pip install ruff
ruff do_stuff.py
# Find Python code errors and fix them with ruff.
ruff do_stuff.py --fix

Run ruff format on your Python script for fast code formatting.

ruff format do_stuff.py

• black: a must-mention, this Python code formatter with some linting-like qualities if your code has syntax errors. Typically, I run it on every script I write, but might consider using ruff format instead!

Reformat your code with black.

pip install black
python -m black do_stuff.py

• sqlfluff: "The SQL linter for humans", sqlfluff is a linter and code reformatting tool to tidy up your database queries. sqlfluff does the little things like uppercase your keywords, add whitespace where appropriate, check syntax and in general clean up your scripts. The CLI is configurable for nearly all dialects from DuckDB, T-SQL, Redshift, MySQL, to SQLlite and more. Check the sqlfluff Github repo for all their supported SQL dialects or use the sqlfluff dialects command to list them in your shell. This is the library I want to tell every Python programmer about right now, along with ruff. SQL is everywhere, we all use it. SQL linters are not something most people even know exist. Next time you need to fix a broken SQL script or clean up some legacy code, enter this into your shell:

pip install sqlfluff
# Lint your SQL code.
sqlfluff lint stuff.sql

# Fix your SQL file in the PostgreSQL dialect.
sqlfluff fix stuff.sql --dialect postgres

# Check available SQL dialects.
sqlfluff dialects

• json.tool: a Python standard library CLI tool with JSON validation abilities. If you're working with json, remember this stock Python tool.

Validate JSON with a built-in python CLI tool.

echo '{1.2:3.4}' | python -m json.tool

• pylint: commonly used static code analyser that enforces PEP-8 and other rules in your code.

Use pylint to improve to improve your Python scripts.

pip install pylint
pylint do_stuff.py

• yamllint: YAML is unavoidable as a configration staple in tons of modern software. It gets heaps of praise for its readability. I appreciated yamllint's instrospective linter when trying to figure out why my YAML config was broken.

Lint your YAML files.

pip install --user yamllint
yamllint config.yaml

• rstfmt and restructuredtext-lint (read more): Both of these reStructuredText format linter libaries offer code reformatting and linting abilities for reStructuredText files (.rst). I favored restructuredtext-lint, due to its rst-lint CLI tool. I used it to fix and tested it on old posts from this blog. Beware that using a reformatter might surface buried errors found by linting the RST, which you'll need to resolve by reading somewhat cryptic RST error messages given by your linter like "hyperlink mismatch, 1 reference but 0 targets". At least the line number is provided so you have a relative scope of where the error is coming from. rstfmt is another Python CLI tool in this space. One note of using these tools is to watch out for unwanted changes. One effect I saw was having code blocks auto-converted from Python formatting to regular code formatting.

Lint and reformat your .rst documents.

pip install restructuredtext-lint
rst-lint post.rst

Pelican + .rst or .md

RST is one of two pelican-import command line tool conversion options by the Pelican static site generator library, along with Markdown (.md). RST is the format this blog is composed in. Pelican has linting-like functionality when you run its pelican content command to compile your static site.

Although I haven't used them personally, it's worth mentioning popular libraries like pyflakes and flake8 for linting Python code. ruff supports some of these libraries also. Check out pymarkdownlint for linting Markdown documents. While researching for this post, I even stumbled upon an HTML linting command line tool that exists. html-linter applies linting to your HTML code. Starting to think that behind every seasoned Python programmer is a thick stack of linters! When it comes to fixing and refactoring old documents and code, linters and auto-formatters go hand in hand as invaluable tools.

Lint your Markdown documents.

pip install pymarkdownlnt
pymarkdown scan example.md

Lint your HTML documents.

pip install html-linter
html_lint.py filename.html

Supplementary Reading + Documentation

7 Python libraries for more maintainable code

reStructuredText documentation

sqlfluff CLI documentation reference

15 Cloudflare Pages Free Plan Features

Recently, this blog switched to Cloudflare Pages after years of hosting on WordPress. I'm now using the free plan and enjoying the various settings that allow you control aspects of security at a level not found in WordPress.

For example, you can activate the "bot fight mode", "browser integrity check" and "user agent blocking" settings to help you fend off bad actors.

Enable bot fight mode.

Run a speed test on your blog.

Turn on search engine hints.

See which search engines are crawling your blog.

Here are my top Cloudflare Pages features:

1. Zone analytics: the primary analytics to view unique visitors count, requests, bandwidth and network error logging
2. crawler hints: turn on hints to search engine crawlers and avoid wasteful compute
3. bot fight mode: enabling this mode will stop malicious bots
4. security center scan: validates your DNS configuration and tells you if anything needs fixed
5. run a speed test: test your site's speed
6. enable IP geolocation: includes the country code for each blog visitor
7. GraphQL API: extensible analytics HTTP API
8. AMP real url: ask Google to show your site's actual url in AMP
9. notifications: set email alerts to monitor your website
10. security events log: monitor managed challenges and bots
11. browser integrity check: looks for common http headers abused by spammers
12. user agent blocking: blocks users with malicious user agent
13. 0-RTT Connection Resumption: enable "0-Round Trip Time", optimized DNS for your blog
14. DNS Analytics: see DNS traffic visualizations
15. Onion Routing: serve your website's content in a tor-friendly way

These are all included in the free plan. Some are enabled out of the box and others need to be toggled on to activate. I'm really digging my two Cloudflare Pages blogs. It's taken some time to get used to writing in reStructuredText format. Using pelican for static site generation is working well.

One quirk of Cloudflare is that only the past 30 days of data is stored. It's not as convenient as Wordpress, which stores the entire analytics history of a blog's traffic. However, Cloudflare's GraphQL API is also an option for your data querying needs. Regardless, I'm very impressed at the level of configuration Cloudflare exposes out of the box!

How to Fix Pip Install Site-Packages --user Error

ERROR: Cannot perform a '--user' install. User site-packages are not visible in this virtualenv.

I found a Github thread that fixed the problem! You need to update your pyvenv.cfg in order to resolve this conflict.

Here is how to fix your Python environment from Github user jawalahe:

1. Go to the pyvenv.cfg file in the virtual environment folder.
2. Set include-system-site-packages to true and save the change.

After completing this, my pip installs worked again and no longer returned the error.

Wordpress to Pelican Blog Migration Complete

This blog is formerly known as "Python Marketer" from 2016 to 2023. In May 2023, I've begun the migration from Wordpress's "Personal" plan to a free, Pythonic Pelican static site. 100 posts are now hosted here exclusively on lofipython.com. I will be posting my projects, Python explorations and technical notes here going forward. Thank you for reading and I hope you enjoy these Python musings. If you want to keep up with my writing, I recommend using an RSS reader to follow the RSS or Atom feeds.

I’ve decided to continue blogging under a new Python moniker which is more fitting of who I am as a Pythonista in 2023. WordPress gave me my start as a blogger before I had the capabilities to make my own. Now, I’ve switched from their “Personal” plan to a Pelican + Cloudlare Pages free plan blog stack. It’s going great so far and will save me $48 per year vs. WordPress. Not to mention, I have a hard backup of all my content and host it on Github! I’m no longer dependent on a paid blogging platform to serve my blog. Huge win all around. Here’s to whatever Python projects are next!

The Next Chapter: Lo-Fi Python

Lo-fi (also typeset as lofi or low-fi; short for low fidelity) is a music or production quality in which elements usually regarded as imperfections in the context of a recording or performance are present, sometimes as a deliberate choice. Wikipedia

The Spirit of Low Fidelity

Lo-Fi Python aims to find the "lo-fi" spirit of Python. Doing more with less. Favoring the standard library. Lowest possible time to MVP (minimum viable product). Learning new libraries. Exploring the ecosystem with playful curiosity. Embrace helping others by helping yourself. This is the way.

Using Bing + GPT-4 to Write a Python Script for Windows Computer Maintenance

Today, I was granted access to Bing's large language model, the next generation GPT-4. Bing's chat is the "more powerful" successor to OpenAI's groundbreaking chat service that's generating heaps of hype in addition to its AI text generation abilities. I gained the new chat functionality a few weeks after joining the Bing waitlist.

A guilty pleasure of mine is fixing up old Windows machines by installing updates and running command prompt tools like SFC and chkdsk. There's also GUI based tools like disk cleanup and the disk defragmenter.

My HP computer on Windows 10 is running slow and steady. It's an old model, so I want to improve performance wherever possible and hopefully speed it up. Enter Bing. It did what I wanted and more, but I needed to rephrase my first question to get better results.

First, I asked how to improve performance on a Windows computer. Bing offered generic windows maintenance tips, then followed up to ask my operating system version, which is Windows 10. When I informed Bing it offered more targeted advice after I rephrased my question to focus on command line commands.

I then asked a new question in a fresh chat session. This time I included my OS and asked more specifically for Windows command prompt commands for improving performance. Bing listed a few options and gave a little context of what they do. Then I was presented options to show how to use powercfg or msconfig.

How to use powercfg according to Bing:

Bing chat helped me find and remember the SFC /scannow command. It caches copies of files, fixes corrupted files and may speed up your machine.

You can even prompt Bing for detailed examples of how to use specific windows commands. To get this kind of result from a search engine is incredible. It's like Bing is your own personal tutor. As someone who is constantly googling how to do computer and programming related things, this feels impressive. This was my first time using this service and it has raised the bar for searching the web. I then prompted Bing for a Python script capable of running the SFC command and had my first "wow" moment.

Then it asks if I want the Python code explained. I'm so chuffed at this point. It tapped the subprocess module to write the script and capture its output. Simply stunning.

SFC requires an administrator command prompt. You can start an admin command prompt from the start menu. I ran "python run_sfc.py" containing the below script on version Python 3.11. The command finished running approximately 20 minutes later. Additionally, if you get the error 'utf-8' bytes can't be decoded, you'll need to pass an encoding argument to decode(). Ok, here's the Python script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Import the subprocess module
import subprocess

# Define the SFC command as a list of arguments
sfc_command = ["sfc", "/scannow"]

# Run the SFC command using subprocess.run and capture the output
sfc_output = subprocess.run(sfc_command, capture_output=True)

# Print the output of the SFC command
print(sfc_output.stdout.decode())

If your text is using a non-English character encoding, I found you may need to pass a different encoding besides the default utf-8. For example, I found this to work if your command prompt characters are in Spanish:

1
2

# en español
print(sfc_output.stdout.decode(encoding="latin-1"))

You could also use a Windows batch file of course:

40 Open Source Libraries and Tools for the Modern Developer

I usually find a lot of these tools from viewing GitHub's trending repositories. When I find a novel repo, I'll star it to remember for later. This is a list of free, open source software spanning security, AI, cloud, HTTP, JSON, monitoring, dev ops and more.

I enjoy finding and cataloging new packages as much as using them sometimes. Some of these I'll never use. By filing them away, I can recall in it the future if I am faced with a problem that they can solve. Enjoy!

1. NeMo: a LLM (large language model) toolkit for conversational AI from Nvidia
2. LMOps: general technology for enabling LLMs from Microsoft
3. Prometheus: open source monitoring tool, built on top of Go
4. aws-cli: universal command line AWS interface
5. httptoolkit: "HTTP Toolkit is a beautiful & open-source tool for debugging, testing and building with HTTP(S) on Windows, Linux & Mac"
6. curl: ubiquitous command line tool for transferring data with URL syntax
7. curl-impersonate: a special build of curl that can impersonate Chrome and Firefox browsers
8. jq: touted JSON processor command line tool
9. jc: CLI tool and Python library that converts out of popular command line tools to other formats
10. JSONView: a Chrome web extension that applies syntax highlighting and folding to JSON in your browser, written in Typescript
11. thug: a Python based "honey client", used to mimic user behavior in a web browser in order to detect and emulate malicious content
12. OpenTelemetry: open source tracing and monitoring app
13. nocodb: an open source airtable alternative
14. dnstake: a fast tool to check for DNS vulnerabilities
15. croc: easily and securely send things from one computer to another
16. buku: inter-browser bookmark transfer tool
17. backstage: an open platform for building developer portals, written in Typescript
18. sentry: a "developer first" error tracking and performance monitoring platform
19. sidekick: free and open source live action debugging platform, like chrome dev tools for your backend
20. Prowler: an open source security tool to perform cloud best practices
21. Earthly: create a CI/CD continuous developer system
22. metabase: the easy, open source way for everyone in your company to ask questions and learn from data
23. Zulip: open source team chat
24. Joplin: an open source note taking and to do list application
25. fastup: a tool for gaining insights from large image collections and detecting anomalies
26. Upptime: an open source uptime monitor and status page connected to Github
27. Rembg: tool to remove the background from images
28. dyna-cli: convert Python functions to Linux shell commands
29. Penpot: the open source design & prototyping platform
30. pgcat: postgresQL pooler with sharding, load balancing and fail over support
31. pedalboard: a Python library for working with audio from Spotify
32. OpenRPA: free open source enterprise grade robot process automation software. See also: more free RPA tools
33. hook-slinger: manage webhooks with Python
34. vidcutter: a modern video editing tool
35. XSSstrike: most advanced Cross Site Scripting detector
36. wifiPhisher: wifi security testing tool with Python extensions
37. Salt: an "any infrastructure" automation tool built on Python
38. Locust: a scalable load testing Python library
39. Fabric: simple, Pythonic remote execution and deployment via ssh
40. ShareX: free and open source program that lets you capture and record your screen, written in C#

"Shutdown" a Windows Computer by Double-clicking a Batch File

Here is a quick and easy way to automate turning off your computer. This saves me about 15 seconds to manually click the start menu and restart buttons. It worked for me on an old, laggy HP computer running the Windows 10 operating system. Now, I can double-click a batch file on my Desktop and walk away while it struggles.

Batch files are executable via:

• double-clicking them
• right-clicking and selecting "Run"
• entering the batch file name in command prompt, ex: "shut down CPU.bat" if the current working directory is in the same folder as the batch file

Open a blank Notepad document and save as shut down CPU.bat with this text:

cmd /c shutdown -s

Source: Microsoft shutdown documentation

When this batch file runs, it will trigger a pop-up window warning that your computer is about to shut down. For my slow, slogging computer it shut off about 20 seconds later. This may also trigger queued automatic updates to install, which happened when I used the above command.

Source: Stack Overflow

App Review: Why You Should Use Pocket to Save Your Reads

I'm really enjoying using Pocket as my primary reading app on my cell phone. My top need is to download articles I find on the internet for offline viewing. This app does that and more in a smooth interface. This review reflects the perspective of using the Pocket Android version 8.1.1.0, pictured here:

dark theme with saved articles

Key Features and Benefits

• download articles & blogs to read offline
• quickly share a webpage to the Pocket app in 2 taps from your browser
• dark or light themes available
• free app, downloadable on your Android or iOS device
• converts and stores articles as podcast style listenable audio
• ability to archive or delete old articles
• includes tags to categorize your reads
• filter by length of the read, "short reads" or "long reads" and read or unread
• the length of the article is displayed in minutes it takes to read it
• chooses the best format to present the website, as "article view" or "web view"
• auto-bookmarking, aka you can close out of an article and it will remember where you were the next time you open it
• discover new stuff to read within the Pocket app
• highlight key points in a text to remember and save for later
• For developers, there is a Pocket API. Apps and integrations are available for other products like Zapier, Evernote, Amazon Alexa and Slack.
• created by Mozilla, a privacy friendly company that also created the Firefox web browser

saving is quick and easy

filter saved articles by length

The Pocket app is an overall great product and I felt compelled to endorse it here for the satisfaction it has brought me a in a short amount of time. It has increased my reading throughput. It's perfect for when you find yourself without an internet connection, in sparsely populated areas, on an airplane or while traveling in foreign countries.

I recommend this app to anyone who compulsively reads blogs and articles. It will help you keep up with the neverending stream of text also known as the internet. Might I suggest saving a few posts from this blog also?

Getting People to Use Software is Hard

You're sure you've created a stellar solution to a problem at hand. You took the time to think an idea out and execute. You iterated tirelessly, making tweaks and creating a tool that will sell itself to potential human users. Everyone loves their own creation. Others will realize the tool's value too!

There's only one problem. People's default setting is to not want to use your software. Whether it's low code, no code, Excel, PHP or Python driven, people are resistant to software. They don't want to use it unless it's so obvious and easy that the value is immediately visible.

Image Source

Getting humans to put in the time and effort to learn how to use your tools is really tough. It has to be excellent. The bar is high. You have to understand your users' needs and perceptions intimately. And even then, you'll probably still fail. Regardless, we keep trying because we are obsessed with solving a problem or simply paid to keep trying to solve the problem.

Building a solution is easy. Automating a task to maintain application state is also sometimes. In my experience, making something that other humans will actually trust and use is not. Keep trying. /rant

6 of the Hottest Python Libraries in 2023

Here's a short list of Python packages making a splash this year:

• pyscript: execute Python code in your web browser
• ruff: code linting package built with Rust and Python
• FastAPI: a rapidly adopted web framework for building APIs
• polars: high performance pandas drop in replacement, also built with Rust and Python
• buzz: Python package that accesses the Whisper API's text transcription of audio
• tomllib: new in the Python 3.11 standard library. It's quietly picking up steam amongst Python developers. Tom's Obvious Minimal Language, TOML, "a config file format for humans"

Python + Rust = High Performance

A common theme I've noticed is the emergence of Rust as a performance complement to Python code. It seems we can expect modularity between the two languages to strengthen their collective abilities.

example polars dataframe code

Python is still growing and evolving.

It's great to see. Including the ability to parse TOML config files in the standard library is a vote of confidence by the Python community. I will now seek to use TOML in my own projects.

Web frameworks and web browsers are king.

FastAPI is the new option in a space dominated by Flask and Django. Web frameworks allow developers to quickly create websites.

Running Python in a browser has been a movement in recent years. First Pyodide, WASM euphoria, and now a Python library that takes Python in a browser to new levels. Excited to see where these projects go and what new buzzy libraries will emerge this year!