Posts Tagged Programming

Using AWS Lambda to copy RDS snapshots between regions

AWS Lambda

At work we needed to make MySQL database on RDS backups between regions without having a running instance in the destination region, I mean, no read replicas wanted. Someone suggested to use a cron to copy the backups between regions. I thought this had to been done so I decided to do a research and I ran into this excellent post that explains how to make the copy using Lambda functions with Python: Copying RDS snapshot to another region for cross-region recovery

This function get the last snapshots for all RDS databases in the source region and copies them to the destination region. Then it deletes old snapshots in the destination region to save space. The function can be triggered using CloudWatch or RDS events, for example when the database backup is finished.

Paulina Budzon, the post author, commented that the function can be improved so I made some changes:

  • Added database list to be backup-ed, instead of all databases in RDS
  • Changed variable naming to avoid reference to the destination region
  • Removed source region example reference in SourceDBSnapshotIdentifier string
  • Added variables for source and destination regions

I shared the code here but it can be got from my fork https://github.com/lgallard/aws-maintenance, or you can get it from Paulina’s https://github.com/pbudzon/aws-maintenance, because she merged my  pull request.

I hope it helps somebody else:

import boto3
 import operator

aws_account = 'XXXX'
 source = 'us-east-1'
 destination = 'sa-east-1'
 databases = ['mysqldb01', 'pgdb01']

def copy_latest_snapshot():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = client.describe_db_snapshots(
 SnapshotType='automated',
 IncludeShared=False,
 IncludePublic=False
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No automated snapshots found")

snapshots_per_project = {}

for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)

copy_name = project + "-" + sorted_list[0][1].strftime("%Y-%m-%d")

print("Checking if " + copy_name + " is copied")

try:
 foreign_client.describe_db_snapshots(
 DBSnapshotIdentifier=copy_name
 )
 except:
 response = foreign_client.copy_db_snapshot(
 SourceDBSnapshotIdentifier='arn:aws:rds:' + source + ':' + aws_account + ':snapshot:' + sorted_list[0][0],
 TargetDBSnapshotIdentifier=copy_name,
 CopyTags=True
 )

if response['DBSnapshot']['Status'] != "pending" and response['DBSnapshot']['Status'] != "available":
 raise Exception("Copy operation for " + copy_name + " failed!")
 print("Copied " + copy_name)

continue

print("Already copied")

def remove_old_snapshots():
 client = boto3.client('rds', source)
 foreign_client = boto3.client('rds', destination)

response = foreign_client.describe_db_snapshots(
 SnapshotType='manual'
 )

if len(response['DBSnapshots']) == 0:
 raise Exception("No manual snapshots in "+ destination + " found")

snapshots_per_project = {}
 for snapshot in response['DBSnapshots']:
 if snapshot['DBInstanceIdentifier'] not in databases or snapshot['Status'] != 'available' :
 continue

if snapshot['DBInstanceIdentifier'] not in snapshots_per_project.keys():
 snapshots_per_project[snapshot['DBInstanceIdentifier']] = {}

snapshots_per_project[snapshot['DBInstanceIdentifier']][snapshot['DBSnapshotIdentifier']] = snapshot[
 'SnapshotCreateTime']

for project in snapshots_per_project:
 if len(snapshots_per_project[project]) > 1:
 sorted_list = sorted(snapshots_per_project[project].items(), key=operator.itemgetter(1), reverse=True)
 to_remove = [i[0] for i in sorted_list[1:]]

for snapshot in to_remove:
 print("Removing " + snapshot)
 foreign_client.delete_db_snapshot(
 DBSnapshotIdentifier=snapshot
 )

def lambda_handler(event, context):
 copy_latest_snapshot()
 remove_old_snapshots()

if __name__ == '__main__':
 lambda_handler(None, None)

Reference: Copying RDS snapshot to another region for cross-region recovery


, , ,

No Comments

Including Apache HTTP Client on Android Studio

 

Android Studio - Apache HTTP Client

In order to use Apache HTTP Client (httpclient y httpmime) in Eclipse I used to download the Android port and then include the dependencies specifying the jar files: /home/lgallard//Android/libs/httpcore-4.3.2.jar /home/lgallard//Android/libs/httpmime-4.3.5.jar I kept this practice after migrating my projects to Android Studio, but I used Gradle to resolve the remaining libraries. Then to uniform everything, I did a research on how to include them. Just add the following lines to your app’s build.gradle file:


apply plugin: 'com.android.application'

android {

    dependencies {
        compile group: 'org.apache.httpcomponents' , name: 'httpclient-android' , version: '4.3.5.1'
        compile (group: 'org.apache.httpcomponents' , name: 'httpmime' , version: '4.3.5') {
        exclude module: 'org.apache.httpcomponents:httpclient'}
    }

    android {
        useLibrary 'org.apache.http.legacy'
    }

}

Then sync gradle and compile your project again.

Reference: Apache HttpClient Android (Gradle)

,

18 Comments

It’s back…

Luis’ Blog is back after some months in pause, and thanks to a friend of mine who is hosting the site. Several people asked me about the blog or information that only can find in my blog, therefore I decided to reactivate again. I hope it helps you, you learn something new or just be a way to change information. Enjoy it!

, , ,

2 Comments

My first application on Android: BatteryReporter

batteryreporter

From a while I wanted to learn how to develop Android applications and when I saw that open a course at Coursera for developing Android appslications so I took the chance to learn. I must confess that it not  a piece of cake, I mean, it just not “copy and paste” code (although you can find example codes out there) and you must learn new concepts, because although you have programmed before (even Java) now you must learn how to do it the “Android’s way”.

After following the video lectures, read and read a lot, I build my first application, which in deed is part of the first assignment for the course. It’s a simple application which reads the device’s battery, and reports if it is charging or discharging, as well as the percentage of charge of the device.

For those interested in try my first application, or just take a look to the code for learninf, here I leave the apk file and the main code.

The code (MainActivity.java)

Here you can take a look to the code for the application:

package com.lgallardo.batteryreporter;
import android.os.BatteryManager;
import android.os.Bundle;
import android.app.Activity;
import android.content.Intent;
import android.content.IntentFilter;
import android.content.res.Resources;
import android.view.Menu;
import android.view.View;
import android.widget.ImageView;
import android.widget.TextView;
public class MainActivity extends Activity {
@Override
 protected void onCreate(Bundle savedInstanceState) {
 super.onCreate(savedInstanceState);
 setContentView(R.layout.activity_main);
 }
@Override
 public boolean onCreateOptionsMenu(Menu menu) {
 // Inflate the menu; this adds items to the action bar if it is present.
 getMenuInflater().inflate(R.menu.main, menu);
 return true;
 }
public void getStatus(View view) {
TextView statusValueTextView, chargingValueTextView, levelValueTextView;
 ImageView iconImageView;

 String charging = "";
int level, scale;
 float batteryPct;
// Get resources reference
 Resources res = getResources();
// Get values TextViews
 statusValueTextView = (TextView) findViewById(R.id.statusValue);
 chargingValueTextView = (TextView) findViewById(R.id.chargingValue);
 levelValueTextView = (TextView) findViewById(R.id.levelValue);

 // Get ImageView (icon)

 iconImageView = (ImageView) findViewById(R.id.imageView1);

 // Get battery's status
 IntentFilter ifilter = new IntentFilter(Intent.ACTION_BATTERY_CHANGED);
 Intent batteryStatus = registerReceiver(null, ifilter);
// Check if the battery is charging or is charged?
 int status = batteryStatus.getIntExtra(BatteryManager.EXTRA_STATUS, -1);
 boolean isCharging = status == BatteryManager.BATTERY_STATUS_CHARGING
 || status == BatteryManager.BATTERY_STATUS_FULL;
// Update UI status
 statusValueTextView.setText(Integer.toString(status));
// Update UI charging
 if (isCharging) {
 charging = res.getString(R.string.yes);
// Get charging method
 int chargePlug = batteryStatus.getIntExtra(
 BatteryManager.EXTRA_PLUGGED, -1);
 boolean usbCharge = chargePlug == BatteryManager.BATTERY_PLUGGED_USB;
 // boolean acCharge = chargePlug ==
 // BatteryManager.BATTERY_PLUGGED_AC;
if (usbCharge) {
 charging = charging + " " + res.getString(R.string.usb);
 } else {
 charging = charging + " " + res.getString(R.string.ac);
 }

 iconImageView.setImageResource(R.drawable.charging);
 } else {
 charging = res.getString(R.string.no);
 iconImageView.setImageResource(R.drawable.discharging);
 }
chargingValueTextView.setText(charging);
// Update UI level
 level = batteryStatus.getIntExtra(BatteryManager.EXTRA_LEVEL, -1);
 scale = batteryStatus.getIntExtra(BatteryManager.EXTRA_SCALE, -1);
 batteryPct = 100 * level / (float) scale;
levelValueTextView.setText(Float.toString(batteryPct)+"%");
}
}

,

2 Comments

Repairing tables on MySQL

 

MySQL fix

If for any reason you come across next message when querying a table:

mysql> select * from data_values_queued;
ERROR 1194 (HY000): Table 'data_values_queued' is marked as crashed and should be repaired

You can fix this problem as follow.

CHECK TABLE

First, proceed to check the table to determine the problem:

mysql> check table data_values_queued;
+----------------------------+-------+----------+----------------------------------------------------------+
| Table                      | Op    | Msg_type | Msg_text                                                 |
+----------------------------+-------+----------+----------------------------------------------------------+
| datbas1.data_values_queued | check | warning  | 4 clients are using or haven't closed the table properly |
| datbas1.data_values_queued | check | error    | Size of datafile is: 4200         Should be: 4220        |
| datbas1.data_values_queued | check | error    | Corrupt                                                  |
+----------------------------+-------+----------+----------------------------------------------------------+
3 rows in set (0.05 sec)

Here you can see the table wasn’t closed properly, for instance due to a abrupt shutdown. In my case a discard the first message because I’m using InnoDB as engine, which allows concurrency on tables.

REPAIR TABLE

Now for repairing the table you can use the following sentence:

mysql> repair table data_values_queued ;
+----------------------------+--------+----------+----------------------------------------+
| Table                      | Op     | Msg_type | Msg_text                               |
+----------------------------+--------+----------+----------------------------------------+
| datbas1.data_values_queued | repair | warning  | Number of rows changed from 144 to 143 |
| datbas1.data_values_queued | repair | status   | OK                                     |
+----------------------------+--------+----------+----------------------------------------+
2 rows in set (0.00 sec)

 

With this you will have repaired the table . Let’s check it:

mysql> select * from data_values_queued;
+---------+--------+----------+---------------------+
| id      | iodbId | value    | timestamp           |
+---------+--------+----------+---------------------+
| 1476194 |    170 | 297      | 2012-01-11 08:40:45 |
| 1476193 |    170 | 296      | 2012-01-11 08:40:17 |
| 1476176 |     71 | 11692    | 2012-01-11 05:06:46 |
| 1476177 |     72 | 12061    | 2012-01-11 05:06:46 |
| 1476178 |     73 | 11820    | 2012-01-11 05:06:46 |
| 1476179 |    107 | 11703    | 2012-01-11 05:06:46 |
...
| 1476258 |    170 | 282      | 2012-01-11 11:07:43 |
+---------+--------+----------+---------------------+
143 rows in set (0.00 sec)

References

,

4 Comments

MySQL Prepared statements for C (complete example)

MySql logo

At work I had to make a interface to MySQL from pure C, and searching on the net I came across the book MySQL Developer’s Library by Paul DuBois which has really good example in chapter 7 on how to do SQL prepared statements using the C connector for MySQL. Although the book explains how to execute a SQL sentence with prepared parameters (prepared statements) and how to get data from a query, at the end you can’t see the whole thing together so I decided to post a complete example:

void iodb_getNumAddresses(int id, int *numAddresses){

 MYSQL *conn;
 MYSQL_STMT *stmt;
 char *sql;

 // Bind variables
 MYSQL_BIND param[1], result[1];

 int myId, myNumAddresses;
 my_bool is_null[1];

 sql = "select count(*) from addresses where id = ?";

 // Open Database
 openDB(&conn);

 // Allocate statement handler
 stmt = mysql_stmt_init(conn);

 if (stmt == NULL) {
  print_error(conn, "Could not initialize statement handler");
  return;
 }

 // Prepare the statement
 if (mysql_stmt_prepare(stmt, sql, strlen(sql)) != 0) {
  print_stmt_error(stmt, "Could not prepare statement");
  return;
 }

 // Initialize the result column structures
 memset (param, 0, sizeof (param)); /* zero the structures */
 memset (result, 0, sizeof (result)); /* zero the structures */

 // Init param structure
 // Select
 param[0].buffer_type     = MYSQL_TYPE_LONG;
 param[0].buffer         = (void *) &myId;
 param[0].is_unsigned    = 0;
 param[0].is_null         = 0;
 param[0].length         = 0;

 // Result
 result[0].buffer_type     = MYSQL_TYPE_LONG;
 result[0].buffer         = (void *) &myNumAddresses;
 result[0].is_unsigned    = 0;
 result[0].is_null         = &is_null[0];
 result[0].length         = 0;

 // Bind param structure to statement
 if (mysql_stmt_bind_param(stmt, param) != 0) {
  print_stmt_error(stmt, "Could not bind parameters");
  return;
 }

 // Bind result
 if (mysql_stmt_bind_result(stmt, result) != 0) {
  print_stmt_error(stmt, "Could not bind results");
  return;
 }

 // Set bind parameters
 myId            = id;

 // Execute!!
 if (mysql_stmt_execute(stmt) != 0) {
  print_stmt_error(stmt, "Could not execute statement");
  return;
 }

 if (mysql_stmt_store_result(stmt) != 0) {
  print_stmt_error(stmt, "Could not buffer result set");
  return;
 }

 // Init data
 (*numAddresses) = 0;

 // Fetch
 if(mysql_stmt_fetch (stmt) == 0){
  (*numAddresses) = myNumAddresses;
 }

 // Deallocate result set
 mysql_stmt_free_result(stmt); /* deallocate result set */

 // Close the statement
 mysql_stmt_close(stmt);

 // Close Database
 closeDB(conn);

}

The important thing here is the fact that there are two special structures from the MySQL API: param which set the parameters fro the query,  and result which will be used to store the data retrieved from the database. In this example, after querying the database numAddresses will have the value queried.

References

4 Comments

Installing Subclipse on Debian (and spinoffs)


Link to the video

If you are interested in installing Subclipse as a SVN client for Eclipse, you can use the following steps, which are OS independent, so they are valid for Windows, Mac and Linux, except the procedure for installing and setting JavaHL library on Debian (and spin-off like Ubuntu).

Installing Subclipse:

In general terms, these are the steps for installing Subclipse in Eclipse:

  1. Open Eclipse, got to  Help > Install New Software.
  2. Add the Subclipse’s URL  to the specific Eclipse version you have. For instance, in Galileo you must add this one http://subclipse.tigris.org/update_1.6.x
  3. Choose all shown items: Core SVNkit Library, Optional JNA Library (recommended) and Subclipse.
  4. Accept the user agreement to star the installation process. At the end a message will pop up to warn you about the fact you are installing third-party software. Just say yes.
  5. When finished you will be asked to restart Eclipse.

JavaHL library on Debian

If you don’t want to use  SVNKit library but JavaHL, then you must install this package:

 aptitude install libsvn-java

You have two alternatives to notify Eclipse about the existence of that library:

  1. The right thing is to make Eclipse read the configuration file with the path to the JavaHL library. To be honest I couldn’t find the user’s eclipse.ini file, just the general /etc/eclipse.ini. There you can add the lines shown in bold:
  2. -startup
    plugins/org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar
    --launcher.library
    plugins/org.eclipse.equinox.launcher.gtk.linux.x86_1.0.200.v20090520
    -showsplash
    org.eclipse.platform
    --launcher.XXMaxPermSize
    256m
    -vmargs
    -Djava.library.path=/usr/lib/jni
    -Xms128m
    -Xmx512m
  3. The other solution is to make a symbolic link to the library, somethink like:
  4. ln -s /usr/lib/jni/libsvnjavahl-1.so /usr/lib/
  5. Reestar Eclipse and check changes by going to Preferences > Team > SVN, where you will be able to see the library version.

SVNKit library

With the installation process explained at the beginning you have downloaded SVNKit, which be used as a substitution of JavaHL, without needing to configure any file at all and with the advantage of be an multi-platform solutions, I mean, it will work on Windows and Mac as well. To do so just go to Window > Preferences > Team > SVN and choose  at the SVN section the SVNKit option:

SVNKit

With all these now you will be able to use this SVN client in Eclipse!!

, ,

6 Comments

Sociable plugin hack for bit.ly

Sociable WordPress Plugin

I was testing some plugins for allowing my visitor to share my posts in social networks like Twitter ot Facebook and the one I liked it the most was Sociable. ButI realized it used awe.sw for shorten URLs with Twitter and Identi.ca, but awe.sw service is colapsed and they’re not longer inviting new people. Thus, I started to research some ways to make it work with Bit.ly and I found this post that explains exactly what I wanted to, however they use the file_get_contents function, which throws this error in my server:

Warning: file_get_contents() [function.file-get-contents]: URL file-access is disabled in the server configuration in /home/g/gallardo/web/wp-content/plugins/sociable/sociable.php on line 813

This is because the allow_url_fopen variable is not enabled in PHP. After asking my hosting provider, they recommended me to use cURL instead of file_get_contents,so I had to make some modifications to the sociable.php file, and I had to add a new one called sociable-bitly.php where all URLs are shorten with cURL.

I put both files into the sociable.3.5.2-hacked.zip zip, just in case somebody needs it. But be aware, it’s not the whole plugin, I just put these file to be copied into the /sociable folder in the official plugin.  Another thing to keep in mind is that you have to edit the sociable-bitly.php with your Bit.ly login and API Key information in the following lines:

$bitlylogin = 'yourbitlylogin';
$bitlyapikey= 'yourbitlyapikey';

Other things can be done with this plugin, for instance all awe.se code can be removed, the Bit.ly input files can be added to the setting panel, or it can be integrated with tinyurl. I hope next official version comes with some of these features.

References:

,

2 Comments

Packing Python code for Symbian

PyS60

Let’s say you want to pack an application in Python for Symbian phones in order to deliver you software in an installation .sis file. How can you do it? Easy, with ensymble

Installing ensymble

In Debian you can install it form repositories by typing:

aptitude install  ensymble

Packing an application

If you have a file called application.py and want to make an installation file, just type:

ensymble py2sis application.py

Packing directories

Sometimes you need to pack a directory to deliver libraries. In this case you have to copy the directory structure you want to have on your phone and copy your files there. For instance, let’s say you want to make a .sis file for a Python library you created, called mylib where files settings.py and networing.py will be store:

mkdir MyLib/pyhon/lib/mylib
cp settings.py MyLib/pyhon/lib/mylib
cp networking.py MyLib/pyhon/lib/mylib

ensymble simplesis --caption="MyLib" --version=1.0.2 MyLib

With the last command you will end up with a file called MyLib_v1_0_2.sis. This file will install a directory called E:\\Python\Lib\mylib (or C:\\Python\Lib\mylib depending on where you install it) with your two Python files.

,

2 Comments

Octave: An alternative to MATLAB

sombrero

I’ve always wondered if there was any Software Libre alternative to MATLAB for doing some lab’s experiments  at collage, like those you might find on subjects such as signal processing or communication. Browsing on Internet I came across Octave, so I decided to test it and give it a try. Let’s see how to install it, how it looks like and my comments about it…

Installing Octave

On Debian just install the following package:

aptitude install qtoctave

It installs Octave’s command interpreter, some of its components and a graphic interface based on Qt to be able to use Octave as  you would use MATLAB. If other components are required, they can be installed later according to everybody’s needs. In my case I had to install the communication component:

aptitude install octave-communications

QtOctave interface

The graphic interface let you, among other things, work with Octave’s interpreter, see the variable list, edit a .m file, and see the command list as you would do in MATLAB.

QtOctave

Editing a .m file

In the above picture you can see a script for one of the experiments on the “Comunication II” subject (Universidad Simón Bolívar, Caracas – Venezuela), where students have to generate some line codes to see how they look like in time, their power spectral densities, how inmune they are to noise and their eye patterns; using functions provided by the teacher, which were made in MATLAB. Let’s see some picture I could generate:

Tiempo Frecuencia

Subplot Patron del Ojo

Compatibility with MATLAB

Although many of the MATLAB commands and functions are recognized by Octave, not everything is implemented the same way or in fact it might not be coded in Octave. For instance, I had to edit the scope.m file to comment the cla command (clear current axes) and some global variables didn’t work  in some of the Toolbox’s scripts provided for this experiment. If you have any problem with a MATLAB’s function or command you can check out the Octave <-> Matlab Compatibility Database.

Final comments

I think Octave is a real  alternative to MATLAB. People can say it’s not as complete as MATLAB is…and it’s true, but who use MATLAB at 100%? On the other hand, I think it would be an option to eradicate other operating systems and some bad habits like buying licenses, because students end it up buying a pirated license to finish their assignments at home, even though university’s labs have 100% legal licenses.

Can those experiments be adapated to Octave? I do think so. Furthermore, post-grade students could write code (like fft6) to contribute to the Octave project by growing its function library up…For now I’m going to share my findings with my teacher to see what she thinks!

,

6 Comments