Lo-Fi Python

Jul 23, 2023

There's a Linter for That! Python Linter and Formatter Libraries You Gotta Know

Linters exist for almost every kind of structured document. Python code, SQL, reStructuredText and many data formats can be improved with a linting library. Python's ecosystem has no shortage of formatters and linters.

In this post, I'll highlight the typical players and some niche linters you might not have heard of! The best quality of linters is their ability to find potential problems in your code by enforcing domain-specific rules. In return, you receive a list of hints for fixing anti-patterns, inconsistencies or unused code to remove. Many linters also offer document reformatting capabilities like the ubiquitous black library for Python code.

Python Linters and Formatters You Gotta Know

  • ruff: the lean and fast linter library that's gotten a lot of people's attention. For good reason, it's the easiest and fastest Python code linter, running rust under the hood. In my experience, using ruff has made me a more efficient Python programmer. The ruff CLI's --fix argument is a nice touch for automatically fixing up your code. In addition to linting, ruff also now includes Python's fastest built-in formatter! ruff format is a high performance drop-in replacement for black.

Run ruff on your Python script for quick and easy linting.

pip install ruff
ruff do_stuff.py
# Find Python code errors and fix them with ruff.
ruff do_stuff.py --fix

Run ruff format on your Python script for fast code formatting.

ruff format do_stuff.py
  • black: a must-mention, this Python code formatter with some linting-like qualities if your code has syntax errors. Typically, I run it on every script I write, but might consider using ruff format instead!

Reformat your code with black.

pip install black
python -m black do_stuff.py
sqlfluff SQL linting CLI tool
  • sqlfluff: "The SQL linter for humans", sqlfluff is a linter and code reformatting tool to tidy up your database queries. sqlfluff does the little things like uppercase your keywords, add whitespace where appropriate, check syntax and in general clean up your scripts. The CLI is configurable for nearly all dialects from DuckDB, T-SQL, Redshift, MySQL, to SQLlite and more. Check the sqlfluff Github repo for all their supported SQL dialects or use the sqlfluff dialects command to list them in your shell. This is the library I want to tell every Python programmer about right now, along with ruff. SQL is everywhere, we all use it. SQL linters are not something most people even know exist. Next time you need to fix a broken SQL script or clean up some legacy code, enter this into your shell:
pip install sqlfluff
# Lint your SQL code.
sqlfluff lint stuff.sql
# Fix your SQL file in the PostgreSQL dialect.
sqlfluff fix stuff.sql --dialect postgres
# Check available SQL dialects.
sqlfluff dialects
  • json.tool: a Python standard library CLI tool with JSON validation abilities. If you're working with json, remember this stock Python tool.

Validate JSON with a built-in python CLI tool.

echo '{1.2:3.4}' | python -m json.tool
  • pylint: commonly used static code analyser that enforces PEP-8 and other rules in your code.

Use pylint to improve to improve your Python scripts.

pip install pylint
pylint do_stuff.py
  • yamllint: YAML is unavoidable as a configration staple in tons of modern software. It gets heaps of praise for its readability. I appreciated yamllint's instrospective linter when trying to figure out why my YAML config was broken.

Lint your YAML files.

pip install --user yamllint
yamllint config.yaml
yamllint YAML linting CLI tool
  • rstfmt and restructuredtext-lint (read more): Both of these reStructuredText format linter libaries offer code reformatting and linting abilities for reStructuredText files (.rst). I favored restructuredtext-lint, due to its rst-lint CLI tool. I used it to fix and tested it on old posts from this blog. Beware that using a reformatter might surface buried errors found by linting the RST, which you'll need to resolve by reading somewhat cryptic RST error messages given by your linter like "hyperlink mismatch, 1 reference but 0 targets". At least the line number is provided so you have a relative scope of where the error is coming from. rstfmt is another Python CLI tool in this space. One note of using these tools is to watch out for unwanted changes. One effect I saw was having code blocks auto-converted from Python formatting to regular code formatting.

Lint and reformat your .rst documents.

pip install restructuredtext-lint
rst-lint post.rst

Pelican + .rst or .md

RST is one of two pelican-import command line tool conversion options by the Pelican static site generator library, along with Markdown (.md). RST is the format this blog is composed in. Pelican has linting-like functionality when you run its pelican content command to compile your static site.

Although I haven't used them personally, it's worth mentioning popular libraries like pyflakes and flake8 for linting Python code. ruff supports some of these libraries also. Check out pymarkdownlint for linting Markdown documents. While researching for this post, I even stumbled upon an HTML linting command line tool that exists. html-linter applies linting to your HTML code. Starting to think that behind every seasoned Python programmer is a thick stack of linters! When it comes to fixing and refactoring old documents and code, linters and auto-formatters go hand in hand as invaluable tools.

Lint your Markdown documents.

pip install pymarkdownlnt
pymarkdown scan example.md

Lint your HTML documents.

pip install html-linter
html_lint.py filename.html

Supplementary Reading + Documentation

7 Python libraries for more maintainable code

reStructuredText documentation

sqlfluff CLI documentation reference

Jul 09, 2023

15 Cloudflare Pages Free Plan Features

Recently, this blog switched to Cloudflare Pages after years of hosting on WordPress. I'm now using the free plan and enjoying the various settings that allow you control aspects of security at a level not found in WordPress.

For example, you can activate the "bot fight mode", "browser integrity check" and "user agent blocking" settings to help you fend off bad actors.

enable bot fight mode

Enable bot fight mode.

run a speed test on your blog

Run a speed test on your blog.

enable crawler hints

Turn on search engine hints.

see crawler traffic

See which search engines are crawling your blog.

Here are my top Cloudflare Pages features:

  1. Zone analytics: the primary analytics to view unique visitors count, requests, bandwidth and network error logging
  2. crawler hints: turn on hints to search engine crawlers and avoid wasteful compute
  3. bot fight mode: enabling this mode will stop malicious bots
  4. security center scan: validates your DNS configuration and tells you if anything needs fixed
  5. run a speed test: test your site's speed
  6. enable IP geolocation: includes the country code for each blog visitor
  7. GraphQL API: extensible analytics HTTP API
  8. AMP real url: ask Google to show your site's actual url in AMP
  9. notifications: set email alerts to monitor your website
  10. security events log: monitor managed challenges and bots
  11. browser integrity check: looks for common http headers abused by spammers
  12. user agent blocking: blocks users with malicious user agent
  13. 0-RTT Connection Resumption: enable "0-Round Trip Time", optimized DNS for your blog
  14. DNS Analytics: see DNS traffic visualizations
  15. Onion Routing: serve your website's content in a tor-friendly way

These are all included in the free plan. Some are enabled out of the box and others need to be toggled on to activate. I'm really digging my two Cloudflare Pages blogs. It's taken some time to get used to writing in reStructuredText format. Using pelican for static site generation is working well.

One quirk of Cloudflare is that only the past 30 days of data is stored. It's not as convenient as Wordpress, which stores the entire analytics history of a blog's traffic. However, Cloudflare's GraphQL API is also an option for your data querying needs. Regardless, I'm very impressed at the level of configuration Cloudflare exposes out of the box!

May 17, 2023

How to Fix Pip Install Site-Packages --user Error

While running Python version 3.8.10 today, out of the blue I saw this error attempting to pip install a package in my virtual environment:

ERROR: Cannot perform a '--user' install. User site-packages are not visible in this virtualenv.

I found a Github thread that fixed the problem! You need to update your pyvenv.cfg in order to resolve this conflict.

Here is how to fix your Python environment from Github user jawalahe:

  1. Go to the pyvenv.cfg file in the virtual environment folder.
  2. Set include-system-site-packages to true and save the change.

After completing this, my pip installs worked again and no longer returned the error.

May 15, 2023

Wordpress to Pelican Blog Migration Complete

This blog is formerly known as "Python Marketer" from 2016 to 2023. In May 2023, I've begun the migration from Wordpress's "Personal" plan to a free, Pythonic Pelican static site. 100 posts are now hosted here exclusively on lofipython.com. I will be posting my projects, Python explorations and technical notes here going forward. Thank you for reading and I hope you enjoy these Python musings. If you want to keep up with my writing, I recommend using an RSS reader to follow the RSS or Atom feeds.

I’ve decided to continue blogging under a new Python moniker which is more fitting of who I am as a Pythonista in 2023. WordPress gave me my start as a blogger before I had the capabilities to make my own. Now, I’ve switched from their “Personal” plan to a Pelican + Cloudlare Pages free plan blog stack. It’s going great so far and will save me $48 per year vs. WordPress. Not to mention, I have a hard backup of all my content and host it on Github! I’m no longer dependent on a paid blogging platform to serve my blog. Huge win all around. Here’s to whatever Python projects are next!

The Next Chapter: Lo-Fi Python

Lo-fi (also typeset as lofi or low-fi; short for low fidelity) is a music or production quality in which elements usually regarded as imperfections in the context of a recording or performance are present, sometimes as a deliberate choice. Wikipedia

The Spirit of Low Fidelity

Lo-Fi Python aims to find the "lo-fi" spirit of Python. Doing more with less. Favoring the standard library. Lowest possible time to MVP (minimum viable product). Learning new libraries. Exploring the ecosystem with playful curiosity. Embrace helping others by helping yourself. This is the way.

Feb 26, 2023

Using Bing + GPT-4 to Write a Python Script for Windows Computer Maintenance

Today, I was granted access to Bing's large language model, the next generation GPT-4. Bing's chat is the "more powerful" successor to OpenAI's groundbreaking chat service that's generating heaps of hype in addition to its AI text generation abilities. I gained the new chat functionality a few weeks after joining the Bing waitlist.

chatting with Bing

A guilty pleasure of mine is fixing up old Windows machines by installing updates and running command prompt tools like SFC and chkdsk. There's also GUI based tools like disk cleanup and the disk defragmenter.

My HP computer on Windows 10 is running slow and steady. It's an old model, so I want to improve performance wherever possible and hopefully speed it up. Enter Bing. It did what I wanted and more, but I needed to rephrase my first question to get better results.

First, I asked how to improve performance on a Windows computer. Bing offered generic windows maintenance tips, then followed up to ask my operating system version, which is Windows 10. When I informed Bing it offered more targeted advice after I rephrased my question to focus on command line commands.

Bing chat response
Bing chat suggestions

I then asked a new question in a fresh chat session. This time I included my OS and asked more specifically for Windows command prompt commands for improving performance. Bing listed a few options and gave a little context of what they do. Then I was presented options to show how to use powercfg or msconfig.

msconfig examples

How to use powercfg according to Bing:

how to use powercfg

Bing chat helped me find and remember the SFC /scannow command. It caches copies of files, fixes corrupted files and may speed up your machine.

You can even prompt Bing for detailed examples of how to use specific windows commands. To get this kind of result from a search engine is incredible. It's like Bing is your own personal tutor. As someone who is constantly googling how to do computer and programming related things, this feels impressive. This was my first time using this service and it has raised the bar for searching the web. I then prompted Bing for a Python script capable of running the SFC command and had my first "wow" moment.

Bing writing a Python script
Bing writes a Python script

Then it asks if I want the Python code explained. I'm so chuffed at this point. It tapped the subprocess module to write the script and capture its output. Simply stunning.

SFC requires an administrator command prompt. You can start an admin command prompt from the start menu. I ran "python run_sfc.py" containing the below script on version Python 3.11. The command finished running approximately 20 minutes later. Additionally, if you get the error 'utf-8' bytes can't be decoded, you'll need to pass an encoding argument to decode(). Ok, here's the Python script:

# Import the subprocess module
import subprocess

# Define the SFC command as a list of arguments
sfc_command = ["sfc", "/scannow"]

# Run the SFC command using subprocess.run and capture the output
sfc_output = subprocess.run(sfc_command, capture_output=True)

# Print the output of the SFC command
Bing Chat coding
Bing chat explains code
Bing chat explains code

If your text is using a non-English character encoding, I found you may need to pass a different encoding besides the default utf-8. For example, I found this to work if your command prompt characters are in Spanish:

# en español
Python encoding traceback

You could also use a Windows batch file of course:

Windows batch file instructions from Bing

Feb 25, 2023

40 Open Source Libraries and Tools for the Modern Developer

I usually find a lot of these tools from viewing GitHub's trending repositories. When I find a novel repo, I'll star it to remember for later. This is a list of free, open source software spanning security, AI, cloud, HTTP, JSON, monitoring, dev ops and more.

I enjoy finding and cataloging new packages as much as using them sometimes. Some of these I'll never use. By filing them away, I can recall in it the future if I am faced with a problem that they can solve. Enjoy!

  1. NeMo: a LLM (large language model) toolkit for conversational AI from Nvidia
  2. LMOps: general technology for enabling LLMs from Microsoft
  3. Prometheus: open source monitoring tool, built on top of Go
  4. aws-cli: universal command line AWS interface
  5. httptoolkit: "HTTP Toolkit is a beautiful & open-source tool for debugging, testing and building with HTTP(S) on Windows, Linux & Mac"
  6. curl: ubiquitous command line tool for transferring data with URL syntax
  7. curl-impersonate: a special build of curl that can impersonate Chrome and Firefox browsers
  8. jq: touted JSON processor command line tool
  9. jc: CLI tool and Python library that converts out of popular command line tools to other formats
  10. JSONView: a Chrome web extension that applies syntax highlighting and folding to JSON in your browser, written in Typescript
  11. thug: a Python based "honey client", used to mimic user behavior in a web browser in order to detect and emulate malicious content
  12. OpenTelemetry: open source tracing and monitoring app
  13. nocodb: an open source airtable alternative
  14. dnstake: a fast tool to check for DNS vulnerabilities
  15. croc: easily and securely send things from one computer to another
  16. buku: inter-browser bookmark transfer tool
  17. backstage: an open platform for building developer portals, written in Typescript
  18. sentry: a "developer first" error tracking and performance monitoring platform
  19. sidekick: free and open source live action debugging platform, like chrome dev tools for your backend
  20. Prowler: an open source security tool to perform cloud best practices
  21. Earthly: create a CI/CD continuous developer system
  22. metabase: the easy, open source way for everyone in your company to ask questions and learn from data
  23. Zulip: open source team chat
  24. Joplin: an open source note taking and to do list application
  25. fastup: a tool for gaining insights from large image collections and detecting anomalies
  26. Upptime: an open source uptime monitor and status page connected to Github
  27. Rembg: tool to remove the background from images
  28. dyna-cli: convert Python functions to Linux shell commands
  29. Penpot: the open source design & prototyping platform
  30. pgcat: postgresQL pooler with sharding, load balancing and fail over support
  31. pedalboard: a Python library for working with audio from Spotify
  32. OpenRPA: free open source enterprise grade robot process automation software. See also: more free RPA tools
  33. hook-slinger: manage webhooks with Python
  34. vidcutter: a modern video editing tool
  35. XSSstrike: most advanced Cross Site Scripting detector
  36. wifiPhisher: wifi security testing tool with Python extensions
  37. Salt: an "any infrastructure" automation tool built on Python
  38. Locust: a scalable load testing Python library
  39. Fabric: simple, Pythonic remote execution and deployment via ssh
  40. ShareX: free and open source program that lets you capture and record your screen, written in C#

Feb 21, 2023

"Shutdown" a Windows Computer by Double-clicking a Batch File

Here is a quick and easy way to automate turning off your computer. This saves me about 15 seconds to manually click the start menu and restart buttons. It worked for me on an old, laggy HP computer running the Windows 10 operating system. Now, I can double-click a batch file on my Desktop and walk away while it struggles.

Batch files are executable via:

  • double-clicking them
  • right-clicking and selecting "Run"
  • entering the batch file name in command prompt, ex: "shut down CPU.bat" if the current working directory is in the same folder as the batch file

Open a blank Notepad document and save as shut down CPU.bat with this text:

cmd /c shutdown -s

When this batch file runs, it will trigger a pop-up window warning that your computer is about to shut down. For my slow, slogging computer it shut off about 20 seconds later. This may also trigger queued automatic updates to install, which happened when I used the above command.

Feb 20, 2023

App Review: Why You Should Use Pocket to Save Your Reads

I'm really enjoying using Pocket as my primary reading app on my cell phone. My top need is to download articles I find on the internet for offline viewing. This app does that and more in a smooth interface. This review reflects the perspective of using the Pocket Android version, pictured here:

dark theme with saved articles

dark theme with saved articles

Key Features and Benefits

  • download articles & blogs to read offline
  • quickly share a webpage to the Pocket app in 2 taps from your browser
  • dark or light themes available
  • free app, downloadable on your Android or iOS device
  • converts and stores articles as podcast style listenable audio
  • ability to archive or delete old articles
  • includes tags to categorize your reads
  • filter by length of the read, "short reads" or "long reads" and read or unread
  • the length of the article is displayed in minutes it takes to read it
  • chooses the best format to present the website, as "article view" or "web view"
  • auto-bookmarking, aka you can close out of an article and it will remember where you were the next time you open it
  • discover new stuff to read within the Pocket app
  • highlight key points in a text to remember and save for later
  • For developers, there is a Pocket API. Apps and integrations are available for other products like Zapier, Evernote, Amazon Alexa and Slack.
  • created by Mozilla, a privacy friendly company that also created the Firefox web browser
saving is quick and easy

saving is quick and easy

filter saved articles by length

filter saved articles by length

The Pocket app is an overall great product and I felt compelled to endorse it here for the satisfaction it has brought me a in a short amount of time. It has increased my reading throughput. It's perfect for when you find yourself without an internet connection, in sparsely populated areas, on an airplane or while traveling in foreign countries.

I recommend this app to anyone who compulsively reads blogs and articles. It will help you keep up with the neverending stream of text also known as the internet. Might I suggest saving a few posts from this blog also?

Feb 15, 2023

Getting People to Use Software is Hard

You're sure you've created a stellar solution to a problem at hand. You took the time to think an idea out and execute. You iterated tirelessly, making tweaks and creating a tool that will sell itself to potential human users. Everyone loves their own creation. Others will realize the tool's value too!

There's only one problem. People's default setting is to not want to use your software. Whether it's low code, no code, Excel, PHP or Python driven, people are resistant to software. They don't want to use it unless it's so obvious and easy that the value is immediately visible.

`Image Source <https://community.spiceworks.com/topic/2181519-what-s-the-deal-with-open-source-software>`__

Image Source

Getting humans to put in the time and effort to learn how to use your tools is really tough. It has to be excellent. The bar is high. You have to understand your users' needs and perceptions intimately. And even then, you'll probably still fail. Regardless, we keep trying because we are obsessed with solving a problem or simply paid to keep trying to solve the problem.

Building a solution is easy. Automating a task to maintain application state is also sometimes. In my experience, making something that other humans will actually trust and use is not. Keep trying. /rant

Jan 22, 2023

6 of the Hottest Python Libraries in 2023

Here's a short list of Python packages making a splash this year:

  • pyscript: execute Python code in your web browser
  • ruff: code linting package built with Rust and Python
  • FastAPI: a rapidly adopted web framework for building APIs
  • polars: high performance pandas drop in replacement, also built with Rust and Python
  • buzz: Python package that accesses the Whisper API's text transcription of audio
  • tomllib: new in the Python 3.11 standard library. It's quietly picking up steam amongst Python developers. Tom's Obvious Minimal Language, TOML, "a config file format for humans"
Python + Rust logos

Python + Rust = High Performance

A common theme I've noticed is the emergence of Rust as a performance complement to Python code. It seems we can expect modularity between the two languages to strengthen their collective abilities.

example polars dataframe code

example polars dataframe code

Python is still growing and evolving.

It's great to see. Including the ability to parse TOML config files in the standard library is a vote of confidence by the Python community. I will now seek to use TOML in my own projects.

Web frameworks and web browsers are king.

FastAPI is the new option in a space dominated by Flask and Django. Web frameworks allow developers to quickly create websites.

Running Python in a browser has been a movement in recent years. First Pyodide, WASM euphoria, and now a Python library that takes Python in a browser to new levels. Excited to see where these projects go and what new buzzy libraries will emerge this year!

pyscript landing page
← Previous Next → Page 2 of 12