Tag: python

Should you learn C/C++ in 2018 ?

Published on: 15.02.2018

Conclusion

I would not recommend learning C/C++ as your first programing language just for learning to programme, better go with Python.

Also, if you do not know why you want to learn C/C++, better do not spend time on it.

My C/C++ background

Most of my professional programming career, around 10 years I have spent writing C/C++ programs.

By professional programming career, I mean that other people have paid me to write code.

I have spent more time in C/C++, altho I would say that I know Python better than C/C++.

My knowledge of Python is better because Python has fewer features than C/C++ and you do not need to do memory management.

Why C/C++ is hard

IMHO C/C++ is hard due to many features and memory management.

By the time you understand memory management, you can learn 80% of Python.

Just look at this image of code that is concentrating 2 strings in C vs. Python.

Yes, there can be less C code (error checking, free, few const less) and the code would still work.

But then I would be showing bad C production code and not be demonstrating how real-world C code should be written.

It is not fun to debug segmentation fault and that is what you get if there is no proper error checking in C.

Python code could, and in real life should also have if __name__ == "__main__":, but I have removed it to because it is not necessary like in C.

Point is that higher level programing language (Python in this example) is doing a lot of low-level things automatically.

So, you need fewer lines of code, what means fewer possible bugs, resulting in increased developer productivity.

The tradeoff is more CPU and memory for faster time to market.

C and C++ are two different programming languages

Here I am using a language construct C/C++.

Because most of the code bases (at least that I have seen in a corporate environment) are some hybrid of C and C++.

My theory is that the older code was written in C and then later they added object-oriented programming (C++).

But C and C++ are not the same programming languages, mental programming models are quite different.

C code is about structure, functions, and pointers.

In C++ you have object-oriented programming and lot of other features.

In the real world, you need to know C and C++, so that is why I use C/C++.

When you should learn C/C++

The only positive thing that I can see that person will learn from learning C/C++ is manual memory management and pointers.

Basically better understanding how the computer (on software level) is working.

If your only reason is to have a better understanding how the computer (on software level) is working, better learn C programing language.

C has fewer features than C++.

It is easier to learn C++ if you know C because then you should know manual memory management and pointers.

But for anything else than few niches (system, embedded, banking, game engines, etc), I do not see a much practical use of C/C++ in the year 2018.

Probably you ain’t gonna use C/C++

I am not saying C/C++ is not in use anywhere in 2018.

I personally (almost every month) get some interview requests for C/C++ positions.

Usually, they are either for embedded software (C) or banking industries (C++).

I am just arguing that beginners (persons that do not know any programing language) should not start with C/C++.

The only exception is if you are planning to get a job (or start your business) in C/C++ environment.

If it is not for the job/business, I do not see why you should learn C/C++, except for hobby.

But even then, get a better hobby 🙂.

P.S. If you, dear reader, think that I am missing some point, please add it in comments.

Always start with simple solution (ConfigParser vs. JSON for Python configuration file)

Published on: 01.01.2018

Conclusion

Sometimes perfect is the enemy of the good.

Why use configuration file

I had Python program, that needed to access device via IP address and to do some diagnostic and commands on it.

At that time, I was accessing only one device (only one IP address).

But could see that in future (in few months to one year), I will need to do the same set of command on more devices.

One solution is to add IP address as a parameter to CLI program.

In my use case number of IP address that needs to be accessed will never be bigger than 34.

And writing 34 IP addresses as CLI parameter, that is around 373 letters, is not a nice solution.

When you need to read it, to see are all IP addressee included, it is not easy to read.

Python code as configuration was not possible

I was distributing my Python code as EXE, so use of Python code as configuration was not possible.

Altho, I think that Python code as the configuration is a good solution if you are executing source code, and only developer (not average user who does not know what Notepad is) will edit it.

Sometimes perfect is enemy of good

Sometimes I do have a problem that I tend to unnecessarily complicate things.

Because I think about all possible edge cases and all possible future uses.

And from these two, all possible future uses are the real problem.

Edge cases can happen, but is there a positive cost benefit to solve them and how?

This need to be determined on a case by case basis.

But “all possible future use” is trying to anticipate the future.

That is impossible.

From my experience, whenever I add code for future use cases, usually it was waste of time.

Even when I am the only user, so I can argue to my self that I know will need it.

Usually, I do not need it.

So, I have decided to eliminate waste when developing software, starting from this project.

For example, here I needed a configuration file that will have a list of IP addresses, which I will iterate in for loop.

This was the smallest requirement that I needed for my problem.

But immediately I was thinking that it would be nice to have:

  • checking if IP address is in the valid format
  • if I have 5 address that I need to write all 5, but I could add special syntax for that. Like 192.168.1.1 --- 192.168.1.5, and this can really get more complicated if you want to cover all edge cases.
  • I only need IPv4, but can I also ad IPv6
  • if there is some error in the configuration file, there can be a useful error message and suggested a solution
  • another configuration file can be added where I would define what are valid IP addresses, and then I can validate requested IP against valid IP

And the list can go on.

But I said to myself, NO.

You will just make code that can read a list of items as a list in Python from the configuration file.

Nothing more.

If when you have the real need, you will add additional features when needed.

When I think about this, maybe this is an example of premature design?

Premature design is deciding too early what a program should do.

Why JSON is better than Configparser for list datatype

After investigating possible solution, decision was to try ConfigParser.

ConfigParser configuration file was:

Code was:

I found few problems with ConfigParser:

  1. It did not have build in functionality to read data as Python List
    • so even for this simple example I had to write custom code (this is why I have get_as_list() function)
  2. There were no rules in format, eg. "192.168.1.36" and 192.168.1.36 was both valid.
    • this was not big problem, but I just did not like it

Let try JSON (as configuration files in Python)

JSON configuration file was:

Code was:

There is much less code, and line count is important.

One thing that I did not like about JSON is "ValueError: No JSON object could be decoded", this is error message that you get if you JSON is invalid.

I was hoping to get some more details, eg. what token in which line.

But you can not have it all, and ConfigParser was no bettter.

Decision was to use JSON as configuration file for Python, because code was less complicated (less lines of code).

I did not tried YAML.

Syntax, at least to me, is less readable than JSON.

Questions and remarks please leave in comments.

Automatic backup of git repositories to Dropbox with Python

Published on: 01.12.2017

Intro

I will show how to upload files to Dropbox from Python code.

Why do I need this?

Currently, I am only using WebFaction for all my web services and also as my private git server.

I wanted to make an automatic backup of my git repositories to Dropbox.

Dropbox App

I order to upload files to Dropbox you need to have an access token.

And for the access token, you need to register your app on DBX platform.

All of this must be done on Dropbox website.

The first step is to go to https://www.dropbox.com/developers/apps/ and press “Create App” button.

Step 1

Just click

Just click “Create app” button

Step 2

New app on DBX Platform

We will use Dropbox API.

We will choose “App folder” because we will just upload one backup to Dropbox, we do not need full access to all our files.

Name your app and click “Create App” button.

Step 3

Use defaults settings

We will use defaults settings, here we will get the access token so, click “Generate access token” button.

Step 4

Access token generated

Now you have your access token, you will need it in your code, so copy it.

Step 5

Dropbox app

Now we have our “my_git_backup” Dropbox app.

pip install

It is always recommended to use virtual environments inside python.

At least I use them always.

Code

I am using fabric to make my life(code) easier.

I use fabric every time when I am calling CLI command from Python.

I will explain NUMBER_OF_BACKUP_TO_KEEP later, I use it at the end of the program.

All code that follows is inside with lcd(remote_directory): Python context manager.

The context manager is used so all code that follows is executed inside remote_directory directory.

Name of backup file will be YYYYMMDD_HHMM_git_backup.zip where upper case letters are date and time when a program was executed.

Eg. 20171121_1856_git_backup.zip so that we know from when is this backup file.

For making an actual backup, zip CLI command is used, we are only doing the backup of files that end on *.git (in my case only git repositories).

I also have LAME_PASSWORD for basic protection.

This is why I used fabric, just by calling local() function you can execute CLI commands.

The first line is the opening connection to your Dropbox application, you need to add your own access token as an argument.

Next two lines are for upload, you are: opening file, reading it and uploading bytes to Dropbox.

In Dropbox documentation is mention that this is only working for files till 150MB in size.

With last line program is deleting the local backup.

First for loop is getting all files from your backup folder in a list.

Second for loop is deleting all files except, last few files.

How many files to keep (otherwise we need manually to delete old backup files) is define in NUMBER_OF_BACKUP_TO_KEEP from the beginning of the code.

I keep it at 10, more than that I do not need.

Because we have date and time in our filename we can use Python sort function to sort files by when the backup was done.

The program can be run with
fab -f fabfile git_backup_to_dropbox

First is fab because we used fabric, fabfile because fabfile.py is file of our source code and git_backup_to_dropboxis name of the function that we are executing from fabfile.py file.

How I run this automaticaly

I personally run this command from crontab once per day.
35 02 * * * /home/user_name/code/venv/bin/fab -f /home/user_name/code/fabfile git_backup_to_dropbox

Conclusion

This can be used for backup of any folder as zip file automaticaly to Dropbox.

For any questions, please write them in comments.

Controlling NEC display from Python with nec-pd-sdk

Update on: 07.03.2013
Updated version available on Twilio blog.

Published on: 01.11.2017

Conclusion

pip install nec-pd-sdk

My Story

I was responsible for maintenance of one spectacular 17 meters tall audio/video system on a cruise ship.

The system had 34 NEC X551UN screens among other components.

Waterfall from top

Behind each screen, there is a SDI-to-DVI converter.

If a picture on the screen was black, usually there was some problem with SDI-to-DVI converter, mostly power supply was broken.

Or NEC screen was broken, but I never had it in practice.

Special NEC screen

But, also there was one special screen, it was black from time to time.

After restart mostly fine, and SDI-to-DVI converter was fine.

After one month of troubleshooting, I have come to the conclusion that problem is with NEC screen.

It just got stuck every few days (sometimes every second day, some time was fine for a week), and simple restart (sometimes of 5 seconds and sometimes of 5 minutes) would solve issue till next time.

I also know that when there is a black picture on this screen, then screen diagnostic was “No signal.”

NEC screen no signal

I have come to the conclusion that the following code could solve the problem:

And then to run this code on a schedule, like every hour.

Existing NEC software

NEC have two software applications for managing their products.

First is PD Comms Tool, you can remotely get and set all values to a screen.

It also has a scripting language.

I have used it for setting scheduler for all 34 screens and change of time.

It is much faster than manually doing it for each screen.

The second one is NaViSet Administrator 2 it is much more powerful than “PD Comms Tool”.

It can be used for monitoring all your NEC screens and also some additional equipment (like projectors and Windows PC)

It also has a visual scripting language where you can set and get multiple parameters according to some condition.

And then you also can set specific scheduler for each script.

I could have used this tool for my problem, but there was just one problem, it did not have sleep/pause command.

Design

I know that existing NEC software is communicating with the screen via TCP/IP.

Full protocol documentation is at http://www.necdisplay.com/documents/UserManuals/External_Control_P.V.X-series.pdf, but I was not so eager to write custom TCP/IP packets.

I wanted something more readable and simple.

I googled “NEC python” and found about nec-pd-sdk, what is python SDK for NEC screens.

There is no textual documentation, but there are few examples.

Most useful for me was test_routines_example.py, it is showing how to get every parameter.

The command for turning screen ON and OFF was found in source code.

Code for turning NEC screen ON and OFF

Here is the code:

Last Words

When I was investigating NaViSet Administrator 2 for my use-case.

I contacted NEC support, to ask can them can I use it for this purpose.

They told me not and suggested to use TCP/IP External_Control.

So, even support from NEC does not know that they have NEC python SDK.

What is sad, considering that their NEC python SDK is useful software.

Questions and remarks, please leave in comments.

Full Stack Python Guide to Deployments, book review

full_stack_python_guide_to_deployments-Book_Cover

Published on: 01.10.2017

Number of pages: 262 with code example, 185 without
Written by: Matthew Makai
Publish by: Self-publish

Conclusion
Useful to read, if you want to learn/see how to deploy python web application.

Review
Topic of book is described in its subject, full stack python deployments.

In appendix C of book is full source code of Flask application that you are deploying.

Topic of book is not source code from Flask application, but just how to deploy that code.

Deployment is done on virtual server on Linode (Ubuntu).

Deployment process is automated with Fabric and Ansible.

I liked this automation because you can use it for your own project also.

Other technologies used are: Nginx, Git, PostgreSQL, Redis, Green Unicorn, Celery and Jenkins

Make standalone executable from Python code with PyInstaller

Published on: 01.09.2017

I wanted to create single file that person could run on Windows machine, from my Pythone code.

After some investigation I found PyInstaller and 1 hour later I had my EXE file from Python code.

Process for generating EXE files from Python code with PyInstaller was quite easy, at least from my experience.

I have used it on Windows 7 64-bit and had no problems.

My program was one file script with 300 line and dependencies to docopt and pyautogui.

Steps for generating exe file with PyInstaller

This will install PyInstaller
pip install pyinstaller

This will generate script.exe in dist directory
pyinstaller --onefile script.py

After this you have your EXE program.

How PyInstaller is working

Here I have used --onefile option for PyInstaller what will make one file EXE program.

If you just use pyinstaller script.py, with out --onefile option, than in dist folder you will get folder script with EXE file and all additional files for your EXE file to work.

If you use --onefile option, then your one file EXE program need every time to uncompresses all files every time when it starts.

Uncompression is described in details in official documentation, temporary directory for uncompression in Windows is %TEMP%.

Some other solutions:
http://nsis.sourceforge.net/Main_Page
http://nuitka.net/
https://pypi.python.org/pypi/pynsist
https://cx-freeze.readthedocs.io/en/latest/
http://www.py2exe.org/

Comparison of some others solutions.

Grokking Algorithms, book review

Grokking_Algorithms-Book_Cover

Published on: 01.07.2017

Number of pages: 256
Written by: Aditya Bhargava
Publish by: Manning

Conclusion
Interesting book, visualizations are very helpful.

Review
When I saw this book first time I was not interested.
Book cover was too childish for me.

But after listening podcast about it, I changed my mind.

Illustrations in book are used to explain concepts, and they are very useful.
All code is in python, that was bonus point for me.

Here is video example of illustration:

I do recommend this book for reading, you can understand Big O notation from it and algorithms.

Learning Selenium Testing Tools with Python, book review

Learning_Selenium_Testing_Tools_with_Python-Book_Cover

Published on: 01.03.2017

Number of pages: 216
Written by: Unmesh Gundecha
Publish by: Packt Publishing

Conclusion
Book about Selenium with Python examples.

Review
Only book about Selenium with Python examples that I have found so far.
I found much more books with Java examples, alto it is not difficult to translate syntax from Java to Python, because Selenium API is similar, I prefer to read code in Python.

Usually I read programing books for two reasons:
– when I do not know anything about topic, so that I get basic understanding
– when I have already used technology, even did some project, but want to see what I do not know about it

With Selenium is second reason.

Book is good, it have lot of code examples how to use Selenium API.
Good to have as reference.

Automate the Boring Stuff with Python, book review

automate-the-boring-stuff-with-python-Book_Cover

Published on: 01.02.2017

Number of pages: 504
Written by: Al Sweigart
Publish by: No Starch Press

Conclusion
Best book for learning programming.

Review
I am very impress with this book.
I am recommending it to anybody who want to learn programming.

Book is available for free online.
Book is also available as video tutorial and first 15 videos are on YouTube for free.

This book is going to teach you how to use programming knowledge to automate tasks on computer.
Some examples are: send emails, automate web page interaction, automate GUI interaction.
All other books on programing for beginners just teach you basic programming, but after that people have no idea how to use it in everyday life.
This book is different because it will show you real examples how to use it in everyday life.