[DPS Class] Contributions WEEK 15

This week I work with Rafael Lopez and we implements a monitoring system for the cluster Benchmarking, also, we reconfigure all the cluster from zero, and we simplify the configuration of the nodes. Also, we write some script for automate the nodes configuration.

Download the scripts: scripts_cluster.tar.gz

Nominations

• Rafael
• Emmanuel
• Gaby

[DPS Class] Benchmarks

A Benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it. The term 'benchmark' is also mostly utilized for the purposes of elaborately-designed benchmarking programs themselves.

Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems.

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.

Wikipedia: Benchmark

Supercomputing Benchmark

The number and type of benchmark suites used to study supercomputer performance has varied widely over the years. In early studies, an ad hoc collection of programs was typically used to measure the performance of a given system relative to a known performance benchmark. Eventually, this practice evolved into groups of programs explicitly designed as supercomputer benchmark suites.

The most widely used benchmarks for performance on supercomputing clusters are: the SPEChpc96 suite; the Livermore Loops; and for scientific machines, the Linpack Kernels.

Some general examples of individual computer benchmarks:

• Dhrystone - Integer benchmark for UNIX systems
• Whetstone - Floating point benchmark for minicomputers
• I/O benchmarks
• MIPS
• Synthetic benchmarks
• Kernel benchmarks
• SPECint / SPECfp
• Summarizing

Types of Benchmarks

1. Real program:
Word processing software tool software of CDA user's application software (i.e.: MIS)
2. Microbenchmark:
Designed to measure the performance of a very small and specific piece of code.
3. Kernel:
Contains key codes normally abstracted from actual program popular kernel: Livermore loop linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language) results are represented in MFLOPS
4. Component Benchmark/ micro-benchmark:
Programs designed to measure performance of a computer's basic components automatic detection of computer's hardware parameters like number of registers, cache size, memory latency
5. Synthetic Benchmark:
Procedure for programming synthetic benchmark: take statistics of all types of operations from many application programs get proportion of each operation write program based on the proportion above Types of Synthetic Benchmark Whetstone, Dhrystone. These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
6. I/O benchmarks
7. Database benchmarks: to measure the throughput and response times of database management systems (DBMS')
8. Parallel benchmarks: used on machines with multiple cores, processors or systems consisting of multiplemachines

Linpack

This is the benchmark used to test the members of Top500 list.

The LINPACK Benchmark was introduced by Jack Dongarra. A detailed description as well as a list of performance results on a wide variety of machines is available in postscript form from netlib. Here you can download the latest version of the LINPACK Report: performance.ps. A parallel implementation of the Linpack benchmark and instructions on how to run it can be found at http://www.netlib.org/benchmark/hpl/.

The benchmark used in the LINPACK Benchmark is to solve a dense system of linear equations. TOP500 uses a version of the benchmark that allows the user to scale the size of the problem and to optimize the software in order to achieve the best performance for a given machine. This performance does not reflect the overall performance of a given system, as no single number ever can. It does, however, reflect the performance of a dedicated system for solving a dense system of linear equations. Since the problem is very regular, the performance achieved is quite high, and the performance numbers give a good correction of peak performance.

By measuring the actual performance for different problem sizes n, a user can get not only the maximal achieved performance Rmax for the problem size Nmax but also the problem size N1/2 where half of the performance Rmax is achieved. These numbers together with the theoretical peak performance Rpeak are the numbers given in the TOP500. In an attempt to obtain uniformity across all computers in performance reporting, the algorithm used in solving the system of equations in the benchmark procedure must conform to LU factorization with partial pivoting. In particular, the operation count for the algorithm must be 2/3 n^3 + O(n^2) double precision floating point operations. This excludes the use of a fast matrix multiply algorithm like "Strassen's Method" or algorithms which compute a solution in a precision lower than full precision (64 bit floating point arithmetic) and refine the solution using an iterative approach.

Top500: LinPack Benchmark

If you want to use Linpack in your system, follow these steps

1. Download the LinPack Benchmark: LinPack 10.3.4
2. Extract the content of the packet.
3. Open the path: ~/linpack_10.3.4/benchmarks/linpack/
4. Run the executable xlinpack_xeon64 or xlinpack_xeon32 according your architecture
5. Set these inputs:

   Input data or print help ? Type [data]/help : data
   Number of equations to solve (problem size): 5000
   Leading dimension of array: 10000
   Number of trials to run: 10
   Data alignment value (in Kbytes): 1000000
   

6. Press enter and wait to the finish

This is my output

There are also a package to run Linpack in HP Clusters, but I will review that when the cluster are ready :)

References

• Wikipedia: Benchmark
• Top500 LinPack Benchmark
• Supercomputing Benchmarking

[DPS Class] Contributions WEEK 12

This week I did a research about High Availability Techniques in order for maintain the communications between the nodes of our cluster, in that research I include a very simple definitions, and some programs that we can use on our cluster.

Link: http://elisa.dyndns-web.com/progra/HASoftware

Nominations

• Emmanuel.
• Saul.
• Alex

The references are in the page.

[DPS Class] Contributions WEEK 11

This week I did a research about Fault Tolerance for the wiki, in that research I include some general information, concepts about the theme and a simple procedure that we can implement in our distributed system

Link: http://elisa.dyndns-web.com/progra/faultTolerance

For the laboratory I did a research about Petri Nets, here I include some general concepts [Spanish].

Link: http://jcecdps.blogspot.mx/2012/04/dps-lab-red-de-petri.html

Nominations

• Roberto Martinez, because his research about the Petri Net Simulator and my research about Petri Nets complements each other very well
• Cecilia Urbina, because she stills investigating about very good python tools to improve the cluster.

The references are in each page.

[DPS Class] Contributions WEEK 9

This week I did a research about RPyC or Remote Python Call, that is a library to perform Distributed Computing, Disttibuted Testing, Remote Administration and Tunneling or Chaining.

All the information about this tool is in the following link:

Link: http://elisa.dyndns-web.com/progra/rpyc

I think this is another usefull tool to improve our cluster because, as well we know, python is a powerfull but simple programming languaje, and the most of us know how to code in this languaje.

All the references are also in the Wiki Page.

[DPS Class] Contributions WEEK 8

Last week, the members of the cluster team did a work meeting. We talked about the importance of join our efforts to build only one cluster instead of work separately in small groups.

We decided the specifications of each computer that are in cluster:

• Operating System: Ubuntu 10.04 64bits
• User: cluster
• Password: (1)

Also, we did the schedule of the activities that we need to do to configure the cluster.

1. Install all the software and packages.
2. Configure VPN.
3. Configure a computing grid.

In a second stage, we are going to integrate a website, where will be possible to upload the tasks, which will be validated, executed and processed by the cluster, and at the final we can download the results, maybe in a compressed file.

This task will be done by Emmanuel, with some tools like:

1. Apache, PHP
2. Ruby on Rails
3. Configure a computing grid.

I think this was necessary. Now we are working together and I hope that the results are even more significant than before.

Nominations

• Cecy Urbina: Because she remains active in exam weeks
• Roberto Martinez: Because he remains active in exam weeks
• Rafael Lopez: Because he participate actively in the meeting, distributing task and clarifying doubts.

(1): Check the Google Document posted in our Facebook page to check this information. (Security :P)

[DPS Class] Contributions WEEK 6

This week, my partner Rafael Lopez and work on the interconnection of two VPN, one in each home, using two different methods:

• OpenVPN: I've updated this entry, check the new information at the bottom.
• HamuchiBy Rafael Lopez

this will serve to us in the future to connect 2 completely separate cluster and make a grid.

For the laboratory, I make the entry: Grid Computing & Cluster Computing

I was unable to login to the wiki because the server stills down, so I wrote the entries in my blog and I will transcribe them to the wiki later.

NOMINATIONS

• Rafael Lopez For the implementation of a Hamuchi Network, and he still helps me in all the work, and also because he called the group for the first meeting to check the progress of the proyect.
• Cecilia Urbina for the implementation of Parallel PI Calculation
• Roberto Martinez for the implemtation of PseudoRMI in Python using Pyro

[DPS Class] Contributions WEEK 5

This week, my partner Rafael Lopez and me were working on the installation and configuration of a VPN server, one in each home, this will serve to us in the future to connect 2 completely separate cluster. This is known as GRID COMPUTING

I was unable to login to the wiki, so I wrote the entries in my blog and when I fix the problem with my account I will transcribe them to the wiki.

LINKS:

• How to setting up and configure OpenVPN
• How to open ports on a router

NOMINATIONS

• Rafael Lopez: It can be very repetitive, but I think he's one of those who most actively participated iand contributed to the creation of the cluster, and which has given more results.

[DPS Class] OpenVPN

"A virtual private network (VPN) is a network that uses primarily public telecommunication infrastructure, such as the Internet, to provide remote offices or traveling users an access to a central organizational network.

VPNs typically require remote users of the network to be authenticated, and often secure data with encryption technologies to prevent disclosure of private information to unauthorized parties.

VPNs may serve any network functionality that is found on any network, such as sharing of data and access to network resources, printers, databases, websites, etc. A VPN user typically experiences the central network in a manner that is identical to being connected directly to the central network. VPN technology via the public Internet has replaced the need to requisition and maintain expensive dedicated leased-line telecommunication circuits once typical in wide-area network installations.

Virtual private network technology reduces costs because it does not need physical leased lines to connect remote users to an Intranet."

For more information about VPN, visit VPN (Wikipedia)

OpenVPN

"OpenVPN is a software based connectivity solution, uses SSL (Secure Sockets Layer) and Virtual Private Network VPN (virtual private network) technology.
OpenVPN offers point-to-point connectivity with hierarchical validation of remotely connected users and host, is a very good option for Wi-Fi environments (wireless networks IEE 802.11) and supports a wide settings, including load balancing and more. It is released under the GPL, free software.

No other solution offers such a mix of enterprise-level security, safety, ease of use and rich features.
OpenVPM simplified the configuration of VPN's reducing the difficult to configure other solutions such as IPsec and making it more accessible for people inexperienced in this type of technology."

For more information, visit OpenVPN Site

Installing and configuring OpenVPN on Ubuntu 10.04 LTS 32Bits

NOTE: I don't include screenshots of the installation because I had some problems during the process, however, I will put the correct commands that if you execute it correctly, you will have no problems. Also, all of the following instructions, unless otherwise indicated, must be executed in the VPN server.


First, the installation of the packages:

suda apt-get install openvpn openssl

Our working directory will be /etc/openvpn/

cd /etc/openvpn

Now that the openvpn package is installed, the certificates for the VPN server need to be created.
First, copy the easy-rsa directory to /etc/openvpn. This will ensure that any changes to the scripts will not be lost when the package is updated. You will also need to adjust permissions in the easy-rsa directory to allow the current user permission to create files.

sudo mkdir /etc/openvpn/easy-rsa/
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/
sudo chown -R $USER /etc/openvpn/easy-rsa/

Next, edit /etc/openvpn/easy-rsa/vars adjusting the values to your environment, this is my configuration:

export KEY_COUNTRY="MX"
export KEY_PROVINCE="NL"
export KEY_CITY="Monterrey"
export KEY_ORG="JuanCarlos"
export KEY_EMAIL="your_email@example.com"

Now, enter the following commands in order to create the server certificates:

cd /etc/openvpn/easy-rsa/
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server
cd keys
openvpn --genkey --secret ta.key
sudo cp server.crt server.key ca.crt dh1024.pem ta.key /etc/openvpn/

Now, enter the following commands in order to create the client certificates, replace CLIENTNAME with the hostname of the client, to know which it are, run hostname in a terminal of a client:

cd /etc/openvpn/easy-rsa/
source vars
./pkitool CLIENTNAME

Now configure the openvpn server by creating /etc/openvpn/server.conf from the example file:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
sudo gzip -d /etc/openvpn/server.conf.gz

Backup /etc/openvpn/server.conf:

sudo mv /etc/openvpn/server.conf /etc/openvpn/server.conf.bak

Create a new /etc/openvpn/server.conf with the following options to:

dev tun
proto tcp
port 1194
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
user nobody
group nogroup
server 10.8.0.0 255.255.255.0
persist-key
persist-tun
status openvpn-status.log
#verb 3
client-to-client
push "redirect-gateway def1"
#log-append /var/log/openvpn
#comp-lzo 

After configuring the server, restart openvpn by entering:

sudo /etc/init.d/openvpn restart

Configuring the Clients

First, the installation of the packages:

suda apt-get install openvpn openssl

Also, our working directory will be /etc/openvpn/

cd /etc/openvpn

Make some directories:

sudo mkdir /etc/openvpn/easy-rsa/
sudo mkdir /etc/openvpn/easy-rsa/keys

From the server, copy the following files to the client, and place them in the appropriate folder:

/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.key
/etc/openvpn/ta.key

Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before.
Then with the server configured and the client certificates copied to the /etc/openvpn/ directory, create a client configuration file by copying the example. In a terminal on the client machine enter:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /etc/openvpn

Backup /etc/openvpn/client.conf:

sudo mv /etc/openvpn/client.conf /etc/openvpn/client.conf.bak

Create a new /etc/openvpn/client.conf with the following options to, in the line remote 123.456.789.000 1194 , replace 123.456.789.000 with the public IP of your server or the hostname of your server:

dev tun
client
proto tcp
remote 123.456.789.000 1194
resolv-retry infinite
nobind
user nobody
group nogroup
# Try to preserve some state across restarts.
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
key /etc/openvpn/easy-rsa/keys/CLIENTNAME.key
comp-lzo
# Set log file verbosity.
verb 3 

Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before and which already must be copied to the client.

Finally, restart openvpn:

sudo /etc/init.d/openvpn restart

Now you should now be able to connect to the remote LAN through the VPN. If you run ifconfig command, you should see a new tunneled interface, something like this:

If you look, a new IP address is assigned to the client, that is a private IP address of our VPN.

Also, you can test the network with some pings :)

UPDATE (11/mar/2012) Configuring and Connecting 2 remote VPN

Firts, follow all the previous steps to build a fully functional local VPN.

We do our cluster, a client of another one. For this, in the remote VPN server, open a terminal and make the corresponding certificates for the client, replace CLIENTNAME with the hostname of the local VPN server, to know which it are, run hostname in a terminal of a the local VPN server:

cd /etc/openvpn/easy-rsa/
source vars
./pkitool CLIENTNAME

Back in our local VPN, copy the following files from the remote VPN server to the local VPN server, and place them in the appropriate folder:

/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.key
/etc/openvpn/ta.key

Remember CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before.
Then create a the client configuration file by copying the example. In a terminal of the local VPN server enter:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /etc/openvpn

Backup /etc/openvpn/client.conf:

sudo mv /etc/openvpn/client.conf /etc/openvpn/client.conf.bak

Create a new /etc/openvpn/client.conf with the following options to, in the line remote 123.456.789.000 1194 , replace 123.456.789.000 with the domain of the remote VPN server, is highly recommended to use a service like dyndns:

dev tun
client
proto tcp
remote remote.vpn.domain 1194
resolv-retry infinite
nobind
user nobody
group nogroup
# Try to preserve some state across restarts.
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
key /etc/openvpn/easy-rsa/keys/CLIENTNAME.key
comp-lzo
# Set log file verbosity.
verb 3 

Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before and which already must be copied to the client.

Finally, restart openvpn in the local VPN server:

sudo /etc/init.d/openvpn restart

Now you should be able to connect to the remote LAN through the VPN, also, you can see how start the client and server daemon in the local VPN server:

[IMAGE]

You must have the tunneled interface like before, and now, if you run the comand route, you should be able to see the IP range of your VPN and also the IP range of the remote VPN.

[IMAGE] https://help.ubuntu.com/10.04/serverguide/openvpn.html

[DPS Class] Contributions WEEK 4

We might made this some day...

For this week, I was researching with my partner Rafael Lopez about Beowulf cluster and MPI systems. So we made two blog entries where we share the gathered information.

• MPI: Theory and Installation
• MPI Samples
• High Performance Clusters Español  English

Also, we implement the John The Ripper application, you saw the live execution the past tuesday.

For the next week, we are beggining to research about Parallel CUDA, so Rafael Lopez and me we are working in the construction of a GPU cluster for the next week, some examples and maybe a live execution.

NOMINATIONS:

• Rafael Lopez, because he help me a lot to understand some cluster concepts and configurations.
• Emmanuel, because he improve the entry "MPI Samples" with an extra example.

That's all.

[DPS Class] Contributions WEEK 3

In the picture, CUDA summary of my computer :)

For this week, I extend my previous entry for the Wiki. I add a new method to install CUDA because all the other tutorials of my partners won't work for me, I think this will be helpful because the problems that I had with the installation are common in the internet portals.
LINK: GPU Computing with CUDA | Installation Section

Also, Rafael Lopez and me we are working in the construction of a cluster using John The Ripper Architecture. This are under developement

This week, I`m going to nominate Gaby Garcia because she made some compilations of examples in CUDA , the next step after my installation tutorial.


References

• Install CUDA on Ubuntu
• Problems with nVidia Optimus Graphics Cards
• Problems with GPL Libraries

[DPS Class] Contributions WEEK 2

As a contribution for this week, I wrote some information about Compute Unified Device Architecture (CUDA) in the Wiki, with some images and a short description in each one.

Link: http://elisa.dyndns-web.com/progra/CUDA


Also, I wrote a little entry about Supercomputing in Mexico.

Link:


Also, I do the translation of the content of the Cluster Team page to english. :D

That's all for this week. :)

[DPS Class] Monitoring the efficiency of a computing cluster

This week I did a research of various systems that can help us to monitor the efficiency of our cluster and the available resources.

Because these are monitoring systems, the contribution is in the software section.

The link to this contribution is Monitoring the efficiency of a computing cluster

Where you can find more details and links for various applications.

Greetings:).