Archive for category Programming

Using AWS Lambda to copy RDS snapshots between regions

AWS Lambda

At work we needed to make MySQL database on RDS backups between regions without having a running instance in the destination region, I mean, no read replicas wanted. Someone suggested to use a cron to copy the backups between regions. I thought this had to been done so I decided to do a research and I ran into this excellent post that explains how to make the copy using Lambda functions with Python: Copying RDS snapshot to another region for cross-region recovery

This function get the last snapshots for all RDS databases in the source region and copies them to the destination region. Then it deletes old snapshots in the destination region to save space. The function can be triggered using CloudWatch or RDS events, for example when the database backup is finished.

Paulina Budzon, the post author, commented that the function can be improved so I made some changes:

  • Added database list to be backup-ed, instead of all databases in RDS
  • Changed variable naming to avoid reference to the destination region
  • Removed source region example reference in SourceDBSnapshotIdentifier string
  • Added variables for source and destination regions

I shared the code here but it can be got from my fork https://github.com/lgallard/aws-maintenance, or you can get it from Paulina’s https://github.com/pbudzon/aws-maintenance, because she merged my  pull request.

I hope it helps somebody else:

import boto3
 import operator

aws_account = 'XXXX'
 source = 'us-east-1'
 destination = 'sa-east-1'
 databases = ['mysqldb01', 'pgdb01']

def copy_latest_snapshot():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = client.describe_db_snapshots(
 SnapshotType='automated',
 IncludeShared=False,
 IncludePublic=False
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No automated snapshots found")

snapshots_per_project = {}

for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)

copy_name = project + "-" + sorted_list[0][1].strftime("%Y-%m-%d")

print("Checking if " + copy_name + " is copied")

try:
 foreign_client.describe_db_snapshots(
 DBSnapshotIdentifier=copy_name
 )
 except:
 response = foreign_client.copy_db_snapshot(
 SourceDBSnapshotIdentifier='arn:aws:rds:' + source + ':' + aws_account + ':snapshot:' + sorted_list[0][0],
 TargetDBSnapshotIdentifier=copy_name,
 CopyTags=True
 )

if response['DBSnapshot']['Status'] != "pending" and response['DBSnapshot']['Status'] != "available":
 raise Exception("Copy operation for " + copy_name + " failed!")
 print("Copied " + copy_name)

continue

print("Already copied")

def remove_old_snapshots():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = foreign_client.describe_db_snapshots(
 SnapshotType='manual'
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No manual snapshots in "+ destination + " found")

snapshots_per_project = {}
 for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 if len(snapshots_per_project[project]) > 1:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)
 to_remove = [i[0] for i in sorted_list[1:]]

for snapshot in to_remove:
 print("Removing " + snapshot)
 foreign_client.delete_db_snapshot(
 DBSnapshotIdentifier=snapshot
 )

def lambda_handler(event, context):
 copy_latest_snapshot()
 remove_old_snapshots()

if __name__ == '__main__':
 lambda_handler(None, None)

Reference: Copying RDS snapshot to another region for cross-region recovery


, , ,

No Comments

Including Apache HTTP Client on Android Studio

 

Android Studio - Apache HTTP Client

In order to use Apache HTTP Client (httpclient y httpmime) in Eclipse I used to download the Android port and then include the dependencies specifying the jar files: /home/lgallard//Android/libs/httpcore-4.3.2.jar /home/lgallard//Android/libs/httpmime-4.3.5.jar I kept this practice after migrating my projects to Android Studio, but I used Gradle to resolve the remaining libraries. Then to uniform everything, I did a research on how to include them. Just add the following lines to your app’s build.gradle file:


apply plugin: 'com.android.application'

android {

    dependencies {
        compile group: 'org.apache.httpcomponents' , name: 'httpclient-android' , version: '4.3.5.1'
        compile (group: 'org.apache.httpcomponents' , name: 'httpmime' , version: '4.3.5') {
        exclude module: 'org.apache.httpcomponents:httpclient'}
    }

    android {
        useLibrary 'org.apache.http.legacy'
    }

}

Then sync gradle and compile your project again.

Reference: Apache HttpClient Android (Gradle)

,

18 Comments

It’s back…

Luis’ Blog is back after some months in pause, and thanks to a friend of mine who is hosting the site. Several people asked me about the blog or information that only can find in my blog, therefore I decided to reactivate again. I hope it helps you, you learn something new or just be a way to change information. Enjoy it!

, , ,

2 Comments

How to enable the Web UI on qBittorrent

If you want to manage your qBittorrent server using a web interface, follow these steps:

  1. On the menu bar, go to Tools > Options qBittorrent WEB UI
  2. In the new window, choose Web UI option
  3. Check Enable the Web User Interface (Remote control) option
  4. Choose a port (default 8080)
  5. Set username and password (default username: admin / password: adminadmin) WEB UI
  6. Click on Ok to save settings.

Now in a browser you can check your qBittorrent server if you put your server’s IP address and port, example: 192.168.1.100:8080 as show below: qBittorrent Web UI You can also access it from your Android device by installing qBittorrent Client or qBittorrent Client Pro: qBitttoren Client Pro

, ,

No Comments

Enabling HTML5 for Blipblip.tv in WordPress plugin Video Sidebar Widgets

English Begin E

The problem

Visiting my blog from a tablet I noticed that theBlip.Tv videos I set in Video Sidebar Widgets were not shown. Doing a research I realized the plugin loaded the old flash-based player  instead of the HTML5 version, so videos coudn’t be played from mobile devices with not flash support, which are the majority of the handheld systems nowadays.

The solution

I just had to add some conditionals in files helper-functions.php and class-videosidebarwidget.php to consider the Blipblip.tv case and thus I added the HTM5 player. In particular, I added the following lines to helper-functions.php:

elseif($admin=="true"){
 if($source == "Blip"){
 echo "\n<iframe src=\"$value.html?p=1\" width=\"250\" height=\"250\" 
 frameborder=\"0\" allowfullscreen>
 </iframe> 
 
 <embed type=\"application/x-shockwave-flash\" src=\"http://a.blip.tv/api.swf#$v_id2\" 
 style=\"display:none\">
 </embed>\n\n"; 
 }else{ 
 // echo video in admin
 echo "\n<object width=\"212\" height=\"172\">\n";
 echo $flashvar;
 echo "<param name=\"allowfullscreen\" value=\"true\" />\n";
 echo "<param name=\"allowscriptaccess\" value=\"always\" />\n";
 echo "<param name=\"movie\" value=\"$value\" />\n";
 echo "<param name=\"wmode\" value=\"transparent\">\n";
 echo "<embed src=\"$value\" type=\"application/x-shockwave-flash\" wmode=\"transparent\" ";
 echo "allowfullscreen=\"true\" allowscriptaccess=\"always\" ";
 echo $flashvar2;
 echo "width=\"212\" height=\"172\">\n";
 echo "</embed>\n";
 echo "</object>\n\n";
 }
}else{

And in file class-videosidebarwidget.php I edited the below code:


case 'Blip': $rv_value = "http://blip.tv/play/$Embed_id"; $rv_flashvar = ""; $rv_flashvar2 = ""; $rv_cap = $Embed_cap;

And I added the following lines:

if($select_source == "Blip"){ 
 echo "\n<iframe align=\"left\" src=\"$rv_value.html?p=1\" width=\"$RV_width\" height=\"$RV_height\" 
 frameborder=\"0\" allowfullscreen>
 </iframe>
 
 <embed type=\"application/x-shockwave-flash\" src=\"http://a.blip.tv/api.swf#$Embed_id\" 
 style=\"display:none\">
 </embed>\n\n";
}else{
 echo "\n<object width=\"$RV_width\" height=\"$RV_height\">\n";
 echo $rv_flashvar;
 echo "<param name=\"allowfullscreen\" value=\"true\" />\n";
 echo "<param name=\"allowscriptaccess\" value=\"always\" />\n";
 echo "<param name=\"movie\" value=\"$rv_value\" />\n";
 echo "<param name=\"wmode\" value=\"transparent\">\n";
 echo "<embed src=\"$rv_value\" type=\"application/x-shockwave-flash\" wmode=\"transparent\" ";
 echo "allowfullscreen=\"true\" allowscriptaccess=\"always\" ";
 echo $rv_flashvar2;
 echo "width=\"$RV_width\" height=\"$RV_height\">\n";
 echo "</embed>\n";
 echo "</object>\n\n";
 }
 if(!empty($rv_cap)){echo "<p class=\"VideoCaption\">$rv_cap</p>\n\n";};
 
 
 echo $after_widget;
 }

After these changes random HTM5 videos are loaded without problems.

Plugin support

By the way, I tried to contact the plugin developer to include these changes and he said he had not time for doing it. It a shame it’s not open-sourced so somebody else can maintain the code.

,

No Comments

Programming Mobile Applications for Android Handheld Systems course

Programming Mobile Applications for Android Handheld Systems

Coursera just sent me the certificate for completing with distinction the course Programming Mobile Applications for Android Handheld Systems. This course is endorsed by the university of Maryland and lectured by Dr. Adan Porter.

Something worthy to mention about this Android course is that it’s really well structured, and the good quality of the videos and assignments. It makes me recall my time at college because the projects are very organized with clear objectives. In fact project’s evaluations are done automatically with JUnits. But the final project is evaluated by other peers (4 in total).

For this course I took the Signature Track, which is the paid option where the identity is checked by a keyboard pattern recognition software and verified by checking  the student’s photo (at the end of the course some assistants check those pics). With these validations they grant a certificate that can be verified using the following links:

If you are interested in this course, you can see the course’s content follow the above link.

,

6 Comments

Gitlab full Postgresql backup and restore

GitLab and Posgresql

GitLab can be used with Postgresql as its database engine  (beside MySQL),  and all come ready to make backups and restores of its databases and git repositories by using a ruby script (rake). This is the recommended way because it lets you recover everything wit not pending transactiond on database and Git sides.

The problem

The documentation explains all you have to perform a manual backup and restore, but the latter fails because it tries to insert existing registers that generate conflicts like this one:

ALTER SEQUENCE
psql:/home/git/gitlab/tmp/backups/db/database.sql:812: ERROR: relation "users" already exists

The Solution

The postgresql backup must erase all tables before trying to recreate them and insert the registers. That can be done by including the  –clean r -c  to the pg_dump command. This option must be edited in the ruby script that makes the backup, which is  /home/git/gitlab/lib/backup/database.rb by default. In this file you have to find this part of code and substitute the following bold line:

require 'yaml'
module Backup
 class Database
 attr_reader :config, :db_dir
def initialize
 @config = YAML.load_file(File.join(Rails.root,'config','database.yml'))[Rails.env]
 @db_dir = File.join(Gitlab.config.backup.path, 'db')
 FileUtils.mkdir_p(@db_dir) unless Dir.exists?(@db_dir)
 end
def dump
 success = case config["adapter"]
 when /^mysql/ then
 print "Dumping MySQL database #{config['database']} ... "
 system('mysqldump', *mysql_args, config['database'], out: db_file_name)
 when "postgresql" then
 print "Dumping PostgreSQL database #{config['database']} ... "
 pg_env
 system('pg_dump', config['database'], out: db_file_name)
 end
 report_success(success)
 end

This line must be replaced with this other line:

system('pg_dump', config['database'], '-c', out: db_file_name)

Here you can see that option -c is passed as argument to command pg_dump. This will force to include all DROPS needed in the generated .sql  for the GitLab backup.

,

No Comments

Course Creative, Serious and Playful Science of Android Apps

Coursera androidapps101 2014 Computer Science and Programming Badge

Recently I received the awards for the Android course Creative, Serious and Playful Science of Android Apps lectured by Lawrence Angrave from University of Illinois at Urbana-Champaign. The course was designed to learn Android programing from scratch, therefore if you are interesting in developing for this platform this is a good starting point. The video lectures were made in English, but there are subtitles for Spanish and even English, so language is not barrier.

The Awards

The awards I earned were these:

  • Statement of Accomplishment:  This is a sort of of certificate which you earn when accomplishing 70% or higher. I got 94.6%
  • Computer Science and Programming Badge: This is a kind of badge they gave you also when you accomplished 70% or higher.

This awards are not part of the Signature Track, which let you share and validate this grade,  but you have to pay $49 for this feature.

,

No Comments

qBittorrent Controller for Android

Download application

If you are interested in this application you can install it from Google Play or download qBittorrent Controller apk on your device. Also you can download the code from GitHub because it was released as LGPL.

Features

  • See two panel view (fragments), for torrent list and details in the same window
  • Drawer menu for switching between All, Downloading, Completed, Paused, Active and Inactive torrent list
  • Pause or resume all torrents from the selected list
  • Auto refresh torrent list after performing an action on a torrent (pause, resume, or delete)
  • Add URL directly or by clicking the torrent link on your device’s browser
  • Pause, resume, delete or delete individual torrents with its downloaded data
  • Set and save a connection account

Phone’s view



Tablet’s view


,

6 Comments

My first application on Android: BatteryReporter

batteryreporter

From a while I wanted to learn how to develop Android applications and when I saw that open a course at Coursera for developing Android appslications so I took the chance to learn. I must confess that it not  a piece of cake, I mean, it just not “copy and paste” code (although you can find example codes out there) and you must learn new concepts, because although you have programmed before (even Java) now you must learn how to do it the “Android’s way”.

After following the video lectures, read and read a lot, I build my first application, which in deed is part of the first assignment for the course. It’s a simple application which reads the device’s battery, and reports if it is charging or discharging, as well as the percentage of charge of the device.

For those interested in try my first application, or just take a look to the code for learninf, here I leave the apk file and the main code.

The code (MainActivity.java)

Here you can take a look to the code for the application:

package com.lgallardo.batteryreporter;
import android.os.BatteryManager;
import android.os.Bundle;
import android.app.Activity;
import android.content.Intent;
import android.content.IntentFilter;
import android.content.res.Resources;
import android.view.Menu;
import android.view.View;
import android.widget.ImageView;
import android.widget.TextView;
public class MainActivity extends Activity {
@Override
 protected void onCreate(Bundle savedInstanceState) {
 super.onCreate(savedInstanceState);
 setContentView(R.layout.activity_main);
 }
@Override
 public boolean onCreateOptionsMenu(Menu menu) {
 // Inflate the menu; this adds items to the action bar if it is present.
 getMenuInflater().inflate(R.menu.main, menu);
 return true;
 }
public void getStatus(View view) {
TextView statusValueTextView, chargingValueTextView, levelValueTextView;
 ImageView iconImageView;

 String charging = "";
int level, scale;
 float batteryPct;
// Get resources reference
 Resources res = getResources();
// Get values TextViews
 statusValueTextView = (TextView) findViewById(R.id.statusValue);
 chargingValueTextView = (TextView) findViewById(R.id.chargingValue);
 levelValueTextView = (TextView) findViewById(R.id.levelValue);

 // Get ImageView (icon)

 iconImageView = (ImageView) findViewById(R.id.imageView1);

 // Get battery's status
 IntentFilter ifilter = new IntentFilter(Intent.ACTION_BATTERY_CHANGED);
 Intent batteryStatus = registerReceiver(null, ifilter);
// Check if the battery is charging or is charged?
 int status = batteryStatus.getIntExtra(BatteryManager.EXTRA_STATUS, -1);
 boolean isCharging = status == BatteryManager.BATTERY_STATUS_CHARGING
 || status == BatteryManager.BATTERY_STATUS_FULL;
// Update UI status
 statusValueTextView.setText(Integer.toString(status));
// Update UI charging
 if (isCharging) {
 charging = res.getString(R.string.yes);
// Get charging method
 int chargePlug = batteryStatus.getIntExtra(
 BatteryManager.EXTRA_PLUGGED, -1);
 boolean usbCharge = chargePlug == BatteryManager.BATTERY_PLUGGED_USB;
 // boolean acCharge = chargePlug ==
 // BatteryManager.BATTERY_PLUGGED_AC;
if (usbCharge) {
 charging = charging + " " + res.getString(R.string.usb);
 } else {
 charging = charging + " " + res.getString(R.string.ac);
 }

 iconImageView.setImageResource(R.drawable.charging);
 } else {
 charging = res.getString(R.string.no);
 iconImageView.setImageResource(R.drawable.discharging);
 }
chargingValueTextView.setText(charging);
// Update UI level
 level = batteryStatus.getIntExtra(BatteryManager.EXTRA_LEVEL, -1);
 scale = batteryStatus.getIntExtra(BatteryManager.EXTRA_SCALE, -1);
 batteryPct = 100 * level / (float) scale;
levelValueTextView.setText(Float.toString(batteryPct)+"%");
}
}

,

2 Comments