Lo-Fi Python

pymarketer: an HTTP + Spreadsheet Wrangling Python package

Typically, this blog reviews the other Python libraries in its vast ecosystem. This time, it's my own package I made for fun, pymarketer. This was created in a single day and can be installed from the Github repo. Have a go at my most read post if you need help with pip.

Install with pip from the source Github repo:

python -m pip install git+https://github.com/erickbytes/pymarketer.git

The pymarketer package helps you do things like:

• merging all the tabs of an Excel file into one CSV
• generate HTTP code
• make a word cloud image
• splitting a CSV
• merging CSVs

Generating a Word Cloud with the pymarketer Package** via wordcloud

1
2
3
4
5
6
7
8

import pandas as pd
import pymarketer as pm

xl = "Chicago Breweries.xlsx"
df = pd.read_excel(xl)
# Make a wordcloud from a pandas dataframe.
wordcloud = pm.word_cloud(df)
wordcloud.to_file("Text Word Cloud Visualization.jpg")

This package relied on several Python libraries to complete:

• chardet
• pandas
• numpy
• wordcloud
• ftfy

I'll likely expand on this in the future. Anyone who wrangles data might be able to apply this package to good profit. At minimum, you might find it interesting to take a look at the project's __init__.py to see how some of the functions are implemented.

Additional Resources

• How to Make an Internal Python Package
• pymarketer Github Examples

The Things They Don't Tell You About Ampersands and XML

In an XML document, you need to escape any ampersands in your text as &

I began a new coding project. Sure, there's documentation for the API that solves my problem. I find out it uses XML. Extensible Markup Language, a classic API format. Cool. I craft a beautiful script that works at first. Or so it seems!

Later on, I realize it doesn't work as well as I believed. It turns out, if I want a server to accept my XML document, escaping certain characters might be required. The documentation didn't mention this. It was my first time using XML, how would I know?

I noticed a script only worked for a handful of requests. It failed for most, returning a 400 status code. Suspecting the issue was likely in my payload, I studied the data of the request bodies that failed compared to the others that succeeded. All of the payload bodies that failed contained text with an ampersand.

Suspecting it might be an XML + ampersand related issue, I googled this Stack Overflow post which explains the ampersand escaping situation. There are a handful of characters that must be escaped. Otherwise, you might not be able to connect to the server.

These are the things they often don't tell you. Those little details you must sometimes realize for yourself, unless someone bothers to mention it or write it down. Now you know something that cost me an hour or two of tinkering to realize!

Image Source

Want to read more on HTTP? Check out my guide on making HTTP requests with Python to read more about HTTP requests.

Hammock-Driven Development Notes

Occasionally you will find a video or talk that connects or resonates with you in a great way. Rich Hickey's "Hammock Driven Development", a self-described "rant" is packed with wisdom. I keep coming back to re-watch and today, I have written down some key points from this amazing rant!

Key Ideas

Take more time to think through your problem.

When was the last time you...

thought about something for a whole day?

thought about something for a whole month or year?

Hammock Driven Development, https://www.youtube.com/watch?v=f84n5oFoZBc

On Bugs

• Bugs are cheaper to fix in development.
• Least expensive to avoid in design
• Most expensive in to fix in production

Analysis & Design, Simplified

• Identify problem trying to solve.
• Assess whether it solves that problem.

On Problem Solving

Problem Solving (cont.)

• State the problem out loud.
• Understand the problem's facts, context and constraints.
• What don't you know?
• Find problems in your solution.
• Write it all down.

More Input, Better Output

• Read in and around your space.
• Look critically at other solutions.
• You can't connect things you don't know about.

On Focus

• On the hammock, no one knows if you're sleeping and they don't bother you because of this.
• Computers are distracting.
• Let loved ones know you are going to be "gone", focusing deeply for some time.

Waking Mind vs Background Mind

• The waking mind is good at critical thinking.
• Use waking time to assign tasks to background mind.
• The background mind is good at making connections and good at strategy.

Sleep According to Scientific American:

• The brain processes info learned while sleeping.
• Sleep makes memories stonger and weeds out irrelevant details.
• Our brain finds hidden relations among memories to solve waking problems.

Closing Ideas

Write the proposed solution down. Hammock time is important "mind's eye time". We switch from "input mode" to "recall mode" during hammock time. Wait overnight, or sometimes months, to think about your problem, sleep sober for best results! Eventually coding is required, and your feedback loop is important, but "don't lean on it too much". You will be wrong, facts and requirements will change. Mistakes happen. That's fine, do not be afraid of being wrong. /rant

The notes in this blog post are paraphrased from this rant.

Python Requests SSLCertVerificationError Solution for Windows

I initially installed my Python version from the Windows Store and it worked fine for almost a year on my Windows 10 computer. Then I started getting this error message when trying to use the requests library on all HTTP requests:

1

requests.exceptions.SSLError: HTTPSConnectionPool(host='api.example.com', port=443): Max retries exceeded with url: /oauth/token?grant_type=client_credentials (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

Apparently my Python certificates were not valid or up to date on my computer. These are ".PEM" or ".cert" files that certify your connection for the SSL protocol. I googled this error until I found the python-certifi-win32 library. I only needed to pip install this library and it fixed the problem:

pip install python-certifi-win32

Huge thank you to the maintainer of this package. It solved my issues and now I can make HTTP requests again!

Reflections on 5 Years of Solving Problems with Python

Prior to learning Python, I had no programming experience. I worked in marketing for a book publisher and did not perform well at my job. It was not a good fit. They eventually fired me. As my previous job unraveled, I discovered Python and the Coursera course, Programming for Everybody (Getting Started with Python). Fortunately, that course jump-started me onto a path of learning and reading each day. My aim was to make my own website, a goal that I accomplished. I needed to know how the sausage was made.

Looking back from 2020, I can safely say Python changed my life. Because of it, I now have a fulfilling marketingdata-oriented career. I'm also grateful for the financial stability that came with it. I love to learn about the language and continue to improve my abilities to solve problems with new tools, not only Python.

Below are pieces of wisdom picked up from my experiences. They are the result of many hours of study, reading, mistakes, luck, toil and eventual glory.

These are thought-provoking adages and guidelines, not absolute truths in all cases.

1. Developing a habit of learning pays off over time, no matter what the subject is. It is an investment in yourself that compounds.
2. Follow your own curiosity. It's less important to compare what you know to others. Compare what you know today to what you knew yesterday. Don't worry about how long it takes to learn.
3. Watch educational or technical conference talks on sites like YouTube or InfoQ. Rich Hickey, Brandon Rhodes and David Beazley are some of my favorite speakers. Watch talks from all languages, not just Python. Often the concepts apply to any programming language.
4. Use an RSS reader. Anytime you find a good blog, subscribe by RSS or email to get new posts. I use the Feeder Chrome extension\Android app.
5. The Zen of Python contains a lot of wisdom. I like the concept of Explicit is better than implicit. This implies declaring your actions in written or oral fashion, providing additional context. Consider favoring easier to read solutions over clever one-liners. For example:
   • List comprehensions are useful and "pythonic", use them! But sometimes it's easier to use a for loop to hash out an idea. (Contrarily, avoiding the Initialize Then Modify pattern benefits those comfortable with comprehensions.)
   • Explicitly using keyword arguments versus positional arguments is another way to make your code easier to understand.
6. Can you explain the solution simply? If not, try to clarify your understanding or maybe there's a simpler way. In Python, there are often several ways to accomplish the same goal. But keep in mind the Zen of Python: There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Look for the obvious way. An example of this is string formatting. I've heard f-strings are the hot new way to do this now.
7. Don't be afraid to change course if things don't feel right. Ask yourself while coding, "Does this feel efficient?" Recently I was trying to format a json string so I approached it like I had in the past, by exporting the request from Postman and formatting the json string with python's format() built-in. But this time, the curly braces were confusing me, I was struggling and it wasn't working. I googled and around and saw python's json module and df.to_json() in pandas. They were a much easier and better-looking solution. But it still wasn't working. Finally, i used the Postman approach and f-strings to format a successful payload. The third try worked! F-strings are super nice and clean.
8. If you're stuck, there's probably a free online course or blog post that explains whatever is confusing you. Use the Googles. When in doubt, Google the error message.
9. Begin your project by writing a list of requirements. This often leads to good questions and cases that may need to be addressed. The book Code Complete 2 covers establishing project requirements in great detail, along with the other stages in the life-cycle of a software project. I'm really enjoying this book and highly recommend it.
10. Names are really important. Take time to think about a good name for your variables and functions. Also, name your scripts well. I name my scripts using action verbs. For example, my script that organizes accumulated files on my desktop into folders is named clean_desktop_files.py. When I see this script months later, its name reminds me the action the script is performing. I believe it's better to err on the side of longer, more descriptive names for variables and functions. It makes code easier to understand. But there is a trade-off with length to keep in check.
11. Moving a block of code into a function can abstract away repetitive code and increase its readability.
12. Each function should do one thing only. Follow the single-responsibility principle.
13. Train yourself to think in data structure conversions. The Python dictionary is very useful and can be converted to and from lists, tuples, sets, etc. I often find it more efficient to convert to a different structure to efficiently organize it. Usually I am googling things like "convert class object to python dictionary" because dictionaries are easy to work with or convert to other structures. The vars() built-in is great for converting objects to a dictionary. For example, once you have a dictionary, you might be able to solve your problem by converting it to a dataframe.
14. Use only the data you need. Reading in just the essential data helps avoid memory issues and hanging programs. In pandas, the usecols argument in pd.read_csv() is great for this. This creates a dataframe with 2 columns:

1

df = pd.read_csv('emails.csv', usecols=['name','email'])

15. Assume that if something is broken, it's because of something you've done. Start from the assumption that your code contains the bug and work outward by eliminating possibilities. Avoid jumping to quick conclusions. Instead, carefully consider possible reasons for why something is happening. Many times, I find my 2nd or 3rd hypothesis is actually true.
16. There will be times when you'll look at someone else's choices and wonder why they did things a certain way. Consider the possibility that they know more than you in this domain.
17. Beware of sequencing errors. Are your tasks, scripts or functions executing in an efficient order to reach your end goal? Look to unblock bottlenecks and correct chronological mistakes in your processes.
18. Before you send that email asking for help, go back and take another look. There's also no shame in asking for help. Be sure you proofread your email before sending.
19. Status code 200 does not guarantee your API request was successful. You may want to write a test to confirm success that doesn't rely on response status codes.
20. Unfortunately, testing gets shunned sometimes. Make it a priority. I enjoy writing pytest tests more than most other code. Why? Because tests confirm my scripts are working to some degree, detect bugs and provide a refactoring safety net.
21. Refactoring your code is a crucial step in making it better. Coming back to my code after a few weeks, months or years brings clarity, experience and a new perspective. It feels good to improve the quality of my old work.
22. Consolidate your tasks. Bundling things can save you a bundle of time! Identify redundant patterns and remove if possible. Observe yourself while working. Any repetitive manual process can probably be automated away. Recently, I figured out how to use a Windows batch file to instantly activate my Python virtual environment. It took me a few years of tediously pasting the cd and activate commands into command prompt every day to realize. Now it's a snap.
23. Stack Overflow is a useful resource. But the top answers may be outdated. Check the other less popular answers sometimes. Or...
24. Read the documentation! An updated or more elegant solution might be there. I recently found os.makedirs(path, exist_ok=True) in the os docs. I didn't know about the exist_ok argument. I was creating folders with a more complicated alternative from Stack Overflow for years. I use this way all the time now. In the same vein, if you need the local system username, the Python docs state getpass.getuser() is preferred over os.getlogin().
25. Write documentation explaining how to use your projects. Even if you can only muster a quick README text file, that's better than nothing. Within your code, docstrings are a nice addition. I have yet to use Sphinx, but it is a popular choice for generating documentation.
26. Teaching others feels good and solidifies your knowledge. Writing and pair programming are great ways to improve your understanding and pass your skills along to other people. While we're on the subject of writing...
27. Write everything down! Your head is not good at storing information in memory. Computers are. This frees your mind to come up with new ideas rather than expending energy to remember what you've already done. It also helps you plan. I use a Notepad text file to keep a running to-do list. You could also use services like Trello or Microsoft Planner. While writing code, use comments and docstrings conservatively for quick notes, clarifications or reminders. The important thing is to write it down somewhere.
28. When editing your writing, continually ask yourself, "Do I need this word or phrase?" for every word you write.

"Brevity is the soul of wit." - William Shakespeare (Hamlet)

29. Draw inspiration from culture, nature and professional disciplines outside of your own. Insights can be mined from anything. Don't dismiss a situation as mundane without first scanning for knowledge nuggets and gems.
30. Better solutions often come to me after gaining time and experience with a problem. Building software is an iterative cycle of adjustment towards consistently fulfilling the needs of those it serves in 100% of cases. In a perfect world, you'd never have bugs. But edge cases tend to pop up in ways you didn't think of when you first wrote a solution. There will also be projects where requirements or business rules change. Consider that possibility when you are designing your solution.
31. It's possible to find a job that you're excited about and genuinely enjoy the work.
32. Respect your craft, whether it's coding or another profession. A skilled carpenter needs precision, practice and focus to make something beautiful. Approach your craft with the same mindset and pride in making your best art.
33. We all have holes in our knowledge. Be receptive to other ways of thinking. The best way to learn is from other humans. Everyone has different backgrounds and experiences. I have never used object oriented programming, classes or certain command line tools like ssh. I have a loose understanding of these things but have not yet applied them to my projects. Working with paths (os and pathlib) still gives me fits sometimes. These are knowledge gaps that I want to fill in. Additionally, we don't know what we don't know. Try to illuminate the fog of your unknown.
34. Choosing to dedicate to learning Python is among the best decisions I've made.
35. Attitude is more important than intelligence. Anyone can learn to program, play guitar or fly an airplane. You can become an adept problem solver. Acquire an attitude to support your determination and persistence.

[caption id="attachment_2981" align="alignnone" width="959"] Brandon Rhodes: Stopping to Sharpen Your Tools - PyWaw Summit 2015[/caption]

I'll leave you with the 4 P's and 4 C's from my Programming for Everybody Coursera course graduation ceremony. Cultivating these principles will guide you to growing your education and finding a positive course in life:

4 P's: Passion, Purpose, Persistence, Playfulness

4 C's: Choice, Commitment, Connection, Completion

Thank you for reading and I hope this post helps you on your own educational journey.

Lessons Learned from "Lost in Space" on Netflix

Being a developer has similarities to the life of a family surviving in space. Sure, the stakes are lower when writing code versus being chased by alien life forms, but the core idea is the same. One is faced with a problem and it must be solved, with varying degrees of consequences for failure.

I enjoyed watching the Robinsons solve problem after problem to stay alive in alien worlds and the vacuum of space. I believe you can learn from anything. So in that spirit, here are some lessons from the show for us problem solving humans:

1. Don't panic when you're in trouble. A cool head is vitally important to your survival chances when you're in a bind. Judy demonstrates this quality in evading the raptor-like creatures chasing her.
2. Problems solved often lead to more problems to solve. We would be wise to anticipate what might happen if we do succeed.
3. Be cautious when venturing into unfamiliar territories. You never know what could be there to trip you up. Which leads me to my next point...
4. Respect nature's ability to f*ck you up. The environment you're in, other organisms, and bacteria can wreak havoc. Despite our perceived self-importance, nature is our god. We are but specks in its creation. Cases in point: the metal-dissolving virus that destroys Judy's chariot and the poisonous algae Don accidentally discovers that immobilizes him.
5. Know when to go for broke, or when to re-group and wait to make your next move. The Robinsons spent 7 months stocking up at their new beach home. Until they suffered a setback...
6. Adapt your plan when new information becomes available or unexpected events happen. They will happen.
7. Think of unconventional ways to use the resources you do have to fix the problem at hand. Who would have thought a spaceship could be retrofitted into a boat?
8. Things often work 83% of the way you expect them to work. Sometimes that's enough to move on. Or maybe it buys you enough time to figure out your next move. The electrical field stalled the fleet of robots long enough for the Robinsons to realize the children should "Lord of the Flies" their way to Alpha Centauri.
9. Question your own motivations, and be open to changing course if you are wrong. Adler realized he should help Will save Scarecrow's life after first trying to stop him. Scarecrow then exhibits this same quality when he turns against his robot comrades to save the children.
10. Ethics and morals may conflict with each other. Was Maureen Robinson wrong for fraudulently saving her son Will? Ethically, yes. But morally, as a mother saving her son? No.
11. Make time for your family and friends. Maureen was so busy saving the family that she didn't read Penny's book. Stop and smell the roses every once in a while. Or read your daughter's damn book.
12. Humans are capable of both great compassion and horrible cruelty. We're running from alien robots, but sometimes we're the monsters? Cough cough Smith and Hastings... cough. Excuse me. (smh 🤦)
13. I'll spare you any parallels about modern artificial intelligence. The best, most grounded definition I've heard of modern AI is "automated cognition".
14. Self-sacrifice is an honorable death to save the lives of others, but can be frustrating to watch on TV. "Did you really have to kill yourself there?" - me, several times during the show 😆