Using AWS Lambda to copy RDS snapshots between regions

AWS Lambda

At work we needed to make MySQL database on RDS backups between regions without having a running instance in the destination region, I mean, no read replicas wanted. Someone suggested to use a cron to copy the backups between regions. I thought this had to been done so I decided to do a research and I ran into this excellent post that explains how to make the copy using Lambda functions with Python: Copying RDS snapshot to another region for cross-region recovery

This function get the last snapshots for all RDS databases in the source region and copies them to the destination region. Then it deletes old snapshots in the destination region to save space. The function can be triggered using CloudWatch or RDS events, for example when the database backup is finished.

Paulina Budzon, the post author, commented that the function can be improved so I made some changes:

  • Added database list to be backup-ed, instead of all databases in RDS
  • Changed variable naming to avoid reference to the destination region
  • Removed source region example reference in SourceDBSnapshotIdentifier string
  • Added variables for source and destination regions

I shared the code here but it can be got from my fork https://github.com/lgallard/aws-maintenance, or you can get it from Paulina’s https://github.com/pbudzon/aws-maintenance, because she merged my  pull request.

I hope it helps somebody else:

import boto3
 import operator

aws_account = 'XXXX'
 source = 'us-east-1'
 destination = 'sa-east-1'
 databases = ['mysqldb01', 'pgdb01']

def copy_latest_snapshot():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = client.describe_db_snapshots(
 SnapshotType='automated',
 IncludeShared=False,
 IncludePublic=False
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No automated snapshots found")

snapshots_per_project = {}

for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)

copy_name = project + "-" + sorted_list[0][1].strftime("%Y-%m-%d")

print("Checking if " + copy_name + " is copied")

try:
 foreign_client.describe_db_snapshots(
 DBSnapshotIdentifier=copy_name
 )
 except:
 response = foreign_client.copy_db_snapshot(
 SourceDBSnapshotIdentifier='arn:aws:rds:' + source + ':' + aws_account + ':snapshot:' + sorted_list[0][0],
 TargetDBSnapshotIdentifier=copy_name,
 CopyTags=True
 )

if response['DBSnapshot']['Status'] != "pending" and response['DBSnapshot']['Status'] != "available":
 raise Exception("Copy operation for " + copy_name + " failed!")
 print("Copied " + copy_name)

continue

print("Already copied")

def remove_old_snapshots():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = foreign_client.describe_db_snapshots(
 SnapshotType='manual'
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No manual snapshots in "+ destination + " found")

snapshots_per_project = {}
 for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 if len(snapshots_per_project[project]) > 1:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)
 to_remove = [i[0] for i in sorted_list[1:]]

for snapshot in to_remove:
 print("Removing " + snapshot)
 foreign_client.delete_db_snapshot(
 DBSnapshotIdentifier=snapshot
 )

def lambda_handler(event, context):
 copy_latest_snapshot()
 remove_old_snapshots()

if __name__ == '__main__':
 lambda_handler(None, None)

Reference: Copying RDS snapshot to another region for cross-region recovery

, , ,

No Comments

How to load VirtualBox driver

VirtualBox
If for any reason (for instance Genymotion complaining about not finding VirtualBox) and you need to reload the VirtualBox driver keep in mind that the script that do this has been moved in newest Debian/Ubuntu versions, so if you used to run this:

/etc/init.d/vboxdrv.sh setup

Now you must reload the module this way:

/usr/lib/virtualbox/vboxdrv.sh setup

,

No Comments

How to update Linux time zones

time zone map
On May 1st, Venezuela will change its time zone to UTC-4, therefore in Linux you must update your tzdata package to be ready to this change. To do so just just follow these steps:

Debian/Ubuntu

aptitude update
aptitude safe-upgrade tzdata

Red Hat

yum check-update
yum update tzdata

In order to check whether the update has the time zone changes you can run this command:

zdump -v /usr/share/zoneinfo/right/America/Caracas  | grep 2016

/usr/share/zoneinfo/right/America/Caracas  Sun May  1 06:59:59 2016 UT = Sun May  1 02:29:59 2016 VET isdst=0 gmtoff=-16200
/usr/share/zoneinfo/right/America/Caracas  Sun May  1 07:00:00 2016 UT = Sun May  1 03:00:00 2016 VET isdst=0 gmtoff=-14400

Reference: How to check if the Time Zone database

2 Comments

AWS Certified Solutions Architect

AWS CSA

At work we are migrating many services to Amazon’s cloud, therefore we need to formalize the AWS knowledge. In that sense we found an online course site called Linux Academy, which I consider helped us with the certification and to understand AWS in general, but it wasn’t the only resource we used, thus we complemented with quizzes, and the AWS documentation.

If you are interested in achieve this certification and the resources we used, here you are some links:

Online courses

 

Quizzes

 

AWS Documentation

4 Comments

Including Apache HTTP Client on Android Studio

 

Android Studio - Apache HTTP Client

In order to use Apache HTTP Client (httpclient y httpmime) in Eclipse I used to download the Android port and then include the dependencies specifying the jar files: /home/lgallard//Android/libs/httpcore-4.3.2.jar /home/lgallard//Android/libs/httpmime-4.3.5.jar I kept this practice after migrating my projects to Android Studio, but I used Gradle to resolve the remaining libraries. Then to uniform everything, I did a research on how to include them. Just add the following lines to your app’s build.gradle file:


apply plugin: 'com.android.application'

android {

    dependencies {
        compile group: 'org.apache.httpcomponents' , name: 'httpclient-android' , version: '4.3.5.1'
        compile (group: 'org.apache.httpcomponents' , name: 'httpmime' , version: '4.3.5') {
        exclude module: 'org.apache.httpcomponents:httpclient'}
    }

    android {
        useLibrary 'org.apache.http.legacy'
    }

}

Then sync gradle and compile your project again.

Reference: Apache HttpClient Android (Gradle)

,

18 Comments

Installing Java on Debian

centOS 6

If you need to install Oracle Java on Debian, you can follow the next steps once the binary file had been downloaded from Oracles’s download page:

Once it has been downloaded, proceed to extract it into the /opt directory:

# tar xvzf jdk-8u45-linux-x64.tar.gz

First let’s check the current java version:

# java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

Here you can see it points to OpenJDK’s 1.7.0_79 version. Now let’s add a path to the version to use:

# update-alternatives --install /usr/bin/java java /opt/jdk1.8.0_45/bin/java 1

Last line adds the /opt/jdk1.8.0_45/bin/java path to /usr/bin/java java  binary, in other words, it creates a symbolic link to use in case that options is chosen. Let’s set an alternative that points to the new path:

# update-alternatives --config java
  Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1071 auto mode
 1 /opt/jdk1.8.0_45/bin/java 1 manual mode
 2 /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java 1061 manual mode
 3 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1071 manual mode
 Enter to keep the current selection[+], or type selection number: |

After choosing the option 1, check the java version one more time:

# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

Now the binary points to the Oracle’s version. You can check it with the symbolic links:

# ls -l /usr/bin/java
lrwxrwxrwx 1 root root 22 jun  9  2014 /usr/bin/java -> /etc/alternatives/java

# ls -l /etc/alternatives/java
lrwxrwxrwx 1 root root 25 jun 15 09:21 /etc/alternatives/java -> /opt/jdk1.8.0_45/bin/java

 

 

,

2 Comments

It’s back…

Luis’ Blog is back after some months in pause, and thanks to a friend of mine who is hosting the site. Several people asked me about the blog or information that only can find in my blog, therefore I decided to reactivate again. I hope it helps you, you learn something new or just be a way to change information. Enjoy it!

, , ,

2 Comments

How to enable the Web UI on qBittorrent

If you want to manage your qBittorrent server using a web interface, follow these steps:

  1. On the menu bar, go to Tools > Options qBittorrent WEB UI
  2. In the new window, choose Web UI option
  3. Check Enable the Web User Interface (Remote control) option
  4. Choose a port (default 8080)
  5. Set username and password (default username: admin / password: adminadmin) WEB UI
  6. Click on Ok to save settings.

Now in a browser you can check your qBittorrent server if you put your server’s IP address and port, example: 192.168.1.100:8080 as show below: qBittorrent Web UI You can also access it from your Android device by installing qBittorrent Client or qBittorrent Client Pro: qBitttoren Client Pro

, ,

No Comments

How to extend a volume on LVM

Sometimes it is necessary to extend a logical volume, expanding the physical volume where it is hosted. If this is the case, you can follow this example extending a volume of 450 GB to 500 GB: First, Let’s check the disk geometry:

:~# fdisk -l /dev/sdb
Disk /dev/sdb: 483.2 GB, 483183820800 bytes
 255 heads, 63 sectors/track, 58743 cylinders, total 943718400 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x0001178e

After expanding the volume you must make the system to rescan the volume:

:~# echo 1 > /sys/block/sdb/device/rescan

Then check the disk geometry one more time:

:~# fdisk -l /dev/sdb
Disk /dev/sdb: 536.9 GB, 536870912000 bytes
 255 heads, 63 sectors/track, 65270 cylinders, total 1048576000 sectors
 Units = sectors of 1 * 512 = 512 bytes
 Sector size (logical/physical): 512 bytes / 512 bytes
 I/O size (minimum/optimal): 512 bytes / 512 bytes
 Disk identifier: 0x0001178e

As you can see, the disk size grew up but this is only known by the operating system, because the LVM still has the same information since the physical volume hasn’t been expanded yet:

:~# pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda5 debian lvm2 a-- 3.76g 0
 /dev/sda6 debian lvm2 a-- 13.00g 0
 /dev/sdb svn_vg lvm2 a-- 450.00g 0

If you extend the physical volume with the following command:

:~# pvresize /dev/sdb
 Physical volume "/dev/sdb" changed
 1 physical volume(s) resized / 0 physical volume(s) not resized

And check the physical volume sizes you will see the added space:

:~# pvs
 PV VG Fmt Attr PSize PFree
 /dev/sda5 debian lvm2 a-- 3.76g 0
 /dev/sda6 debian lvm2 a-- 13.00g 0
 /dev/sdb svn_vg lvm2 a-- 500.00g 50.00g

Now you have to extend the logical volume and the system file size on the fly. You can do both actions using just one command:

:~# lvresize -r -L +50G /dev/mapper/svn_vg-svn
 Extending logical volume svn to 500.00 GiB
 Logical volume svn successfully resized
 resize2fs 1.42.5 (29-Jul-2012)
 Filesystem at /dev/mapper/svn_vg-svn is mounted on /var/data/svn; on-line resizing required
 old_desc_blocks = 29, new_desc_blocks = 32
 Performing an on-line resize of /dev/mapper/svn_vg-svn to 131070976 (4k) blocks.
 The filesystem on /dev/mapper/svn_vg-svn is now 131070976 blocks long.

Finally, check the free space available to see the added 50 GB in the system file:

:~# df -h
 Filesystem Size Used Avail Use% Mounted on
 rootfs 993M 287M 655M 31% /
 udev 10M 0 10M 0% /dev
 tmpfs 397M 228K 397M 1% /run
 /dev/mapper/debian-root 993M 287M 655M 31% /
 tmpfs 5.0M 0 5.0M 0% /run/lock
 tmpfs 794M 0 794M 0% /run/shm
 /dev/sda1 228M 34M 183M 16% /boot
 /dev/mapper/debian-home 2.0G 39M 1.9G 3% /home
 /dev/mapper/debian-tmp 3.9G 7.1M 3.7G 1% /tmp
 /dev/mapper/debian-usr 2.0G 869M 1.1G 46% /usr
 /dev/mapper/debian-var 6.7G 2.1G 4.3G 33% /var
 /dev/mapper/svn_vg-svn 493G 432G 62G 88% /var/data/svn

,

2 Comments

Enabling HTML5 for Blipblip.tv in WordPress plugin Video Sidebar Widgets

English Begin E

The problem

Visiting my blog from a tablet I noticed that theBlip.Tv videos I set in Video Sidebar Widgets were not shown. Doing a research I realized the plugin loaded the old flash-based player  instead of the HTML5 version, so videos coudn’t be played from mobile devices with not flash support, which are the majority of the handheld systems nowadays.

The solution

I just had to add some conditionals in files helper-functions.php and class-videosidebarwidget.php to consider the Blipblip.tv case and thus I added the HTM5 player. In particular, I added the following lines to helper-functions.php:

elseif($admin=="true"){
 if($source == "Blip"){
 echo "\n<iframe src=\"$value.html?p=1\" width=\"250\" height=\"250\" 
 frameborder=\"0\" allowfullscreen>
 </iframe> 
 
 <embed type=\"application/x-shockwave-flash\" src=\"http://a.blip.tv/api.swf#$v_id2\" 
 style=\"display:none\">
 </embed>\n\n"; 
 }else{ 
 // echo video in admin
 echo "\n<object width=\"212\" height=\"172\">\n";
 echo $flashvar;
 echo "<param name=\"allowfullscreen\" value=\"true\" />\n";
 echo "<param name=\"allowscriptaccess\" value=\"always\" />\n";
 echo "<param name=\"movie\" value=\"$value\" />\n";
 echo "<param name=\"wmode\" value=\"transparent\">\n";
 echo "<embed src=\"$value\" type=\"application/x-shockwave-flash\" wmode=\"transparent\" ";
 echo "allowfullscreen=\"true\" allowscriptaccess=\"always\" ";
 echo $flashvar2;
 echo "width=\"212\" height=\"172\">\n";
 echo "</embed>\n";
 echo "</object>\n\n";
 }
}else{

And in file class-videosidebarwidget.php I edited the below code:


case 'Blip': $rv_value = "http://blip.tv/play/$Embed_id"; $rv_flashvar = ""; $rv_flashvar2 = ""; $rv_cap = $Embed_cap;

And I added the following lines:

if($select_source == "Blip"){ 
 echo "\n<iframe align=\"left\" src=\"$rv_value.html?p=1\" width=\"$RV_width\" height=\"$RV_height\" 
 frameborder=\"0\" allowfullscreen>
 </iframe>
 
 <embed type=\"application/x-shockwave-flash\" src=\"http://a.blip.tv/api.swf#$Embed_id\" 
 style=\"display:none\">
 </embed>\n\n";
}else{
 echo "\n<object width=\"$RV_width\" height=\"$RV_height\">\n";
 echo $rv_flashvar;
 echo "<param name=\"allowfullscreen\" value=\"true\" />\n";
 echo "<param name=\"allowscriptaccess\" value=\"always\" />\n";
 echo "<param name=\"movie\" value=\"$rv_value\" />\n";
 echo "<param name=\"wmode\" value=\"transparent\">\n";
 echo "<embed src=\"$rv_value\" type=\"application/x-shockwave-flash\" wmode=\"transparent\" ";
 echo "allowfullscreen=\"true\" allowscriptaccess=\"always\" ";
 echo $rv_flashvar2;
 echo "width=\"$RV_width\" height=\"$RV_height\">\n";
 echo "</embed>\n";
 echo "</object>\n\n";
 }
 if(!empty($rv_cap)){echo "<p class=\"VideoCaption\">$rv_cap</p>\n\n";};
 
 
 echo $after_widget;
 }

After these changes random HTM5 videos are loaded without problems.

Plugin support

By the way, I tried to contact the plugin developer to include these changes and he said he had not time for doing it. It a shame it’s not open-sourced so somebody else can maintain the code.

,

No Comments