Tag: python

Automatic backup of git repositories to Dropbox with Python

Published on: 01.12.2017

Intro

I will show how to upload files to Dropbox from Python code.

Why do I need this?

Currently, I am only using WebFaction for all my web services and also as my private git server.

I wanted to make an automatic backup of my git repositories to Dropbox.

Dropbox App

I order to upload files to Dropbox you need to have an access token.

And for the access token, you need to register your app on DBX platform.

All of this must be done on Dropbox website.

The first step is to go to https://www.dropbox.com/developers/apps/ and press “Create App” button.

Step 1

Just click

Just click “Create app” button

Step 2

New app on DBX Platform

We will use Dropbox API.

We will choose “App folder” because we will just upload one backup to Dropbox, we do not need full access to all our files.

Name your app and click “Create App” button.

Step 3

Use defaults settings

We will use defaults settings, here we will get the access token so, click “Generate access token” button.

Step 4

Access token generated

Now you have your access token, you will need it in your code, so copy it.

Step 5

Dropbox app

Now we have our “my_git_backup” Dropbox app.

pip install

It is always recommended to use virtual environments inside python.

At least I use them always.

Code

I am using fabric to make my life(code) easier.

I use fabric every time when I am calling CLI command from Python.

I will explain NUMBER_OF_BACKUP_TO_KEEP later, I use it at the end of the program.

All code that follows is inside with lcd(remote_directory): Python context manager.

The context manager is used so all code that follows is executed inside remote_directory directory.

Name of backup file will be YYYYMMDD_HHMM_git_backup.zip where upper case letters are date and time when a program was executed.

Eg. 20171121_1856_git_backup.zip so that we know from when is this backup file.

For making an actual backup, zip CLI command is used, we are only doing the backup of files that end on *.git (in my case only git repositories).

I also have LAME_PASSWORD for basic protection.

This is why I used fabric, just by calling local() function you can execute CLI commands.

The first line is the opening connection to your Dropbox application, you need to add your own access token as an argument.

Next two lines are for upload, you are: opening file, reading it and uploading bytes to Dropbox.

In Dropbox documentation is mention that this is only working for files till 150MB in size.

With last line program is deleting the local backup.

First for loop is getting all files from your backup folder in a list.

Second for loop is deleting all files except, last few files.

How many files to keep (otherwise we need manually to delete old backup files) is define in NUMBER_OF_BACKUP_TO_KEEP from the beginning of the code.

I keep it at 10, more than that I do not need.

Because we have date and time in our filename we can use Python sort function to sort files by when the backup was done.

The program can be run with
fab -f fabfile git_backup_to_dropbox

First is fab because we used fabric, fabfile because fabfile.py is file of our source code and git_backup_to_dropboxis name of the function that we are executing from fabfile.py file.

How I run this automaticaly

I personally run this command from crontab once per day.
35 02 * * * /home/user_name/code/venv/bin/fab -f /home/user_name/code/fabfile git_backup_to_dropbox

Conclusion

This can be used for backup of any folder as zip file automaticaly to Dropbox.

For any questions, please write them in comments.

Controlling NEC display from Python with nec-pd-sdk

Published on: 01.11.2017

Conclusion

pip install nec-pd-sdk

My Story

I was responsible for maintenance of one spectacular 17 meters tall audio/video system on a cruise ship.

The system had 34 NEC X551UN screens among other components.

Waterfall from top

Behind each screen, there is a SDI-to-DVI converter.

If a picture on the screen was black, usually there was some problem with SDI-to-DVI converter, mostly power supply was broken.

Or NEC screen was broken, but I never had it in practice.

Special NEC screen

But, also there was one special screen, it was black from time to time.

After restart mostly fine, and SDI-to-DVI converter was fine.

After one month of troubleshooting, I have come to the conclusion that problem is with NEC screen.

It just got stuck every few days (sometimes every second day, some time was fine for a week), and simple restart (sometimes of 5 seconds and sometimes of 5 minutes) would solve issue till next time.

I also know that when there is a black picture on this screen, then screen diagnostic was “No signal.”

NEC screen no signal

I have come to the conclusion that the following code could solve the problem:

And then to run this code on a schedule, like every hour.

Existing NEC software

NEC have two software applications for managing their products.

First is PD Comms Tool, you can remotely get and set all values to a screen.

It also has a scripting language.

I have used it for setting scheduler for all 34 screens and change of time.

It is much faster than manually doing it for each screen.

The second one is NaViSet Administrator 2 it is much more powerful than “PD Comms Tool”.

It can be used for monitoring all your NEC screens and also some additional equipment (like projectors and Windows PC)

It also has a visual scripting language where you can set and get multiple parameters according to some condition.

And then you also can set specific scheduler for each script.

I could have used this tool for my problem, but there was just one problem, it did not have sleep/pause command.

Design

I know that existing NEC software is communicating with the screen via TCP/IP.

Full protocol documentation is at http://www.necdisplay.com/documents/UserManuals/External_Control_P.V.X-series.pdf, but I was not so eager to write custom TCP/IP packets.

I wanted something more readable and simple.

I googled “NEC python” and found about nec-pd-sdk, what is python SDK for NEC screens.

There is no textual documentation, but there are few examples.

Most useful for me was test_routines_example.py, it is showing how to get every parameter.

The command for turning screen ON and OFF was found in source code.

Code for turning NEC screen ON and OFF

Here is the code:

Last Words

When I was investigating NaViSet Administrator 2 for my use-case.

I contacted NEC support, to ask can them can I use it for this purpose.

They told me not and suggested to use TCP/IP External_Control.

So, even support from NEC does not know that they have NEC python SDK.

What is sad, considering that their NEC python SDK is useful software.

Questions and remarks, please leave in comments.

Full Stack Python Guide to Deployments, book review

full_stack_python_guide_to_deployments-Book_Cover

Published on: 01.10.2017

Number of pages: 262 with code example, 185 without
Written by: Matthew Makai
Publish by: Self-publish

Conclusion
Useful to read, if you want to learn/see how to deploy python web application.

Review
Topic of book is described in its subject, full stack python deployments.

In appendix C of book is full source code of Flask application that you are deploying.

Topic of book is not source code from Flask application, but just how to deploy that code.

Deployment is done on virtual server on Linode (Ubuntu).

Deployment process is automated with Fabric and Ansible.

I liked this automation because you can use it for your own project also.

Other technologies used are: Nginx, Git, PostgreSQL, Redis, Green Unicorn, Celery and Jenkins

Make standalone executable from Python code with PyInstaller

Published on: 01.09.2017

I wanted to create single file that person could run on Windows machine, from my Pythone code.

After some investigation I found PyInstaller and 1 hour later I had my EXE file from Python code.

Process for generating EXE files from Python code with PyInstaller was quite easy, at least from my experience.

I have used it on Windows 7 64-bit and had no problems.

My program was one file script with 300 line and dependencies to docopt and pyautogui.

Steps for generating exe file with PyInstaller

This will install PyInstaller
pip install pyinstaller

This will generate script.exe in dist directory
pyinstaller --onefile script.py

After this you have your EXE program.

How PyInstaller is working

Here I have used --onefile option for PyInstaller what will make one file EXE program.

If you just use pyinstaller script.py, with out --onefile option, than in dist folder you will get folder script with EXE file and all additional files for your EXE file to work.

If you use --onefile option, then your one file EXE program need every time to uncompresses all files every time when it starts.

Uncompression is described in details in official documentation, temporary directory for uncompression in Windows is %TEMP%.

Some other solutions:
http://nsis.sourceforge.net/Main_Page
http://nuitka.net/
https://pypi.python.org/pypi/pynsist
https://cx-freeze.readthedocs.io/en/latest/
http://www.py2exe.org/

Comparison of some others solutions.

Grokking Algorithms, book review

Grokking_Algorithms-Book_Cover

Published on: 01.07.2017

Number of pages: 256
Written by: Aditya Bhargava
Publish by: Manning

Conclusion
Interesting book, visualizations are very helpful.

Review
When I saw this book first time I was not interested.
Book cover was too childish for me.

But after listening podcast about it, I changed my mind.

Illustrations in book are used to explain concepts, and they are very useful.
All code is in python, that was bonus point for me.

Here is video example of illustration:

I do recommend this book for reading, you can understand Big O notation from it and algorithms.

Learning Selenium Testing Tools with Python, book review

Learning_Selenium_Testing_Tools_with_Python-Book_Cover

Published on: 01.03.2017

Number of pages: 216
Written by: Unmesh Gundecha
Publish by: Packt Publishing

Conclusion
Book about Selenium with Python examples.

Review
Only book about Selenium with Python examples that I have found so far.
I found much more books with Java examples, alto it is not difficult to translate syntax from Java to Python, because Selenium API is similar, I prefer to read code in Python.

Usually I read programing books for two reasons:
– when I do not know anything about topic, so that I get basic understanding
– when I have already used technology, even did some project, but want to see what I do not know about it

With Selenium is second reason.

Book is good, it have lot of code examples how to use Selenium API.
Good to have as reference.

Automate the Boring Stuff with Python, book review

automate-the-boring-stuff-with-python-Book_Cover

Published on: 01.02.2017

Number of pages: 504
Written by: Al Sweigart
Publish by: No Starch Press

Conclusion
Best book for learning programming.

Review
I am very impress with this book.
I am recommending it to anybody who want to learn programming.

Book is available for free online.
Book is also available as video tutorial and first 15 videos are on YouTube for free.

This book is going to teach you how to use programming knowledge to automate tasks on computer.
Some examples are: send emails, automate web page interaction, automate GUI interaction.
All other books on programing for beginners just teach you basic programming, but after that people have no idea how to use it in everyday life.
This book is different because it will show you real examples how to use it in everyday life.

Effective Python, book review

Effective_Python-Book_Cover

Published on: 02.09.2016

Number of pages: 256
Written by: Brett Slatkin
Publish by: Addison-Wesley

Conclusion
Collection of 59 good ideas on how to write better Python code.

Review
Book contains 59 items on how to written better Python code.

Items are grouped in 8 chapters: pythonic thinking, functions, classes and inheritance, metaclasses and attributes, concurrency and parallelism, built-in modules, collaboration, production.

Useful to read.