Category: software

Automatic backup of git repositories to Dropbox with Python

Published on: 01.12.2017

Intro

I will show how to upload files to Dropbox from Python code.

Why do I need this?

Currently, I am only using WebFaction for all my web services and also as my private git server.

I wanted to make an automatic backup of my git repositories to Dropbox.

Dropbox App

I order to upload files to Dropbox you need to have an access token.

And for the access token, you need to register your app on DBX platform.

All of this must be done on Dropbox website.

The first step is to go to https://www.dropbox.com/developers/apps/ and press “Create App” button.

Step 1

Just click

Just click “Create app” button

Step 2

New app on DBX Platform

We will use Dropbox API.

We will choose “App folder” because we will just upload one backup to Dropbox, we do not need full access to all our files.

Name your app and click “Create App” button.

Step 3

Use defaults settings

We will use defaults settings, here we will get the access token so, click “Generate access token” button.

Step 4

Access token generated

Now you have your access token, you will need it in your code, so copy it.

Step 5

Dropbox app

Now we have our “my_git_backup” Dropbox app.

pip install

It is always recommended to use virtual environments inside python.

At least I use them always.

Code

I am using fabric to make my life(code) easier.

I use fabric every time when I am calling CLI command from Python.

I will explain NUMBER_OF_BACKUP_TO_KEEP later, I use it at the end of the program.

All code that follows is inside with lcd(remote_directory): Python context manager.

The context manager is used so all code that follows is executed inside remote_directory directory.

Name of backup file will be YYYYMMDD_HHMM_git_backup.zip where upper case letters are date and time when a program was executed.

Eg. 20171121_1856_git_backup.zip so that we know from when is this backup file.

For making an actual backup, zip CLI command is used, we are only doing the backup of files that end on *.git (in my case only git repositories).

I also have LAME_PASSWORD for basic protection.

This is why I used fabric, just by calling local() function you can execute CLI commands.

The first line is the opening connection to your Dropbox application, you need to add your own access token as an argument.

Next two lines are for upload, you are: opening file, reading it and uploading bytes to Dropbox.

In Dropbox documentation is mention that this is only working for files till 150MB in size.

With last line program is deleting the local backup.

First for loop is getting all files from your backup folder in a list.

Second for loop is deleting all files except, last few files.

How many files to keep (otherwise we need manually to delete old backup files) is define in NUMBER_OF_BACKUP_TO_KEEP from the beginning of the code.

I keep it at 10, more than that I do not need.

Because we have date and time in our filename we can use Python sort function to sort files by when the backup was done.

The program can be run with
fab -f fabfile git_backup_to_dropbox

First is fab because we used fabric, fabfile because fabfile.py is file of our source code and git_backup_to_dropboxis name of the function that we are executing from fabfile.py file.

How I run this automaticaly

I personally run this command from crontab once per day.
35 02 * * * /home/user_name/code/venv/bin/fab -f /home/user_name/code/fabfile git_backup_to_dropbox

Conclusion

This can be used for backup of any folder as zip file automaticaly to Dropbox.

For any questions, please write them in comments.

Controlling NEC display from Python with nec-pd-sdk

Published on: 01.11.2017

Conclusion

pip install nec-pd-sdk

My Story

I was responsible for maintenance of one spectacular 17 meters tall audio/video system on a cruise ship.

The system had 34 NEC X551UN screens among other components.

Waterfall from top

Behind each screen, there is a SDI-to-DVI converter.

If a picture on the screen was black, usually there was some problem with SDI-to-DVI converter, mostly power supply was broken.

Or NEC screen was broken, but I never had it in practice.

Special NEC screen

But, also there was one special screen, it was black from time to time.

After restart mostly fine, and SDI-to-DVI converter was fine.

After one month of troubleshooting, I have come to the conclusion that problem is with NEC screen.

It just got stuck every few days (sometimes every second day, some time was fine for a week), and simple restart (sometimes of 5 seconds and sometimes of 5 minutes) would solve issue till next time.

I also know that when there is a black picture on this screen, then screen diagnostic was “No signal.”

NEC screen no signal

I have come to the conclusion that the following code could solve the problem:

And then to run this code on a schedule, like every hour.

Existing NEC software

NEC have two software applications for managing their products.

First is PD Comms Tool, you can remotely get and set all values to a screen.

It also has a scripting language.

I have used it for setting scheduler for all 34 screens and change of time.

It is much faster than manually doing it for each screen.

The second one is NaViSet Administrator 2 it is much more powerful than “PD Comms Tool”.

It can be used for monitoring all your NEC screens and also some additional equipment (like projectors and Windows PC)

It also has a visual scripting language where you can set and get multiple parameters according to some condition.

And then you also can set specific scheduler for each script.

I could have used this tool for my problem, but there was just one problem, it did not have sleep/pause command.

Design

I know that existing NEC software is communicating with the screen via TCP/IP.

Full protocol documentation is at http://www.necdisplay.com/documents/UserManuals/External_Control_P.V.X-series.pdf, but I was not so eager to write custom TCP/IP packets.

I wanted something more readable and simple.

I googled “NEC python” and found about nec-pd-sdk, what is python SDK for NEC screens.

There is no textual documentation, but there are few examples.

Most useful for me was test_routines_example.py, it is showing how to get every parameter.

The command for turning screen ON and OFF was found in source code.

Code for turning NEC screen ON and OFF

Here is the code:

Last Words

When I was investigating NaViSet Administrator 2 for my use-case.

I contacted NEC support, to ask can them can I use it for this purpose.

They told me not and suggested to use TCP/IP External_Control.

So, even support from NEC does not know that they have NEC python SDK.

What is sad, considering that their NEC python SDK is useful software.

Questions and remarks, please leave in comments.

Make standalone executable from Python code with PyInstaller

Published on: 01.09.2017

I wanted to create single file that person could run on Windows machine, from my Pythone code.

After some investigation I found PyInstaller and 1 hour later I had my EXE file from Python code.

Process for generating EXE files from Python code with PyInstaller was quite easy, at least from my experience.

I have used it on Windows 7 64-bit and had no problems.

My program was one file script with 300 line and dependencies to docopt and pyautogui.

Steps for generating exe file with PyInstaller

This will install PyInstaller
pip install pyinstaller

This will generate script.exe in dist directory
pyinstaller --onefile script.py

After this you have your EXE program.

How PyInstaller is working

Here I have used --onefile option for PyInstaller what will make one file EXE program.

If you just use pyinstaller script.py, with out --onefile option, than in dist folder you will get folder script with EXE file and all additional files for your EXE file to work.

If you use --onefile option, then your one file EXE program need every time to uncompresses all files every time when it starts.

Uncompression is described in details in official documentation, temporary directory for uncompression in Windows is %TEMP%.

Some other solutions:
http://nsis.sourceforge.net/Main_Page
http://nuitka.net/
https://pypi.python.org/pypi/pynsist
https://cx-freeze.readthedocs.io/en/latest/
http://www.py2exe.org/

Comparison of some others solutions.