jueves, 24 de mayo de 2012

[DPS Class] Contributions WEEK 16



This week I work with Rafael Lopez, we look for some testing codes for the cluster, also, we help our parterns to configure their computers. We add some new nodes to the cluster.

The examples are the shown the past thursday.


Nominations
  • Rafael
  • Emmanuel
  • Victor

jueves, 17 de mayo de 2012

[DPS Class] Contributions WEEK 15



This week I work with Rafael Lopez and we implements a monitoring system for the cluster Benchmarking, also, we reconfigure all the cluster from zero, and we simplify the configuration of the nodes. Also, we write some script for automate the nodes configuration.

Download the scripts: scripts_cluster.tar.gz


Nominations
  • Rafael
  • Emmanuel
  • Gaby

miércoles, 9 de mayo de 2012

[DPS Class] Contributions WEEK 14



This week I did a research about Benchmarking in that research I include a very simple definitions, and a practice that I did with Linpack test



Nominations
  • Rafael.
  • Cecy.
  • Alex
The references are in the page.

[DPS Class] Benchmarks

A Benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it. The term 'benchmark' is also mostly utilized for the purposes of elaborately-designed benchmarking programs themselves.

Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems.

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures.


Wikipedia: Benchmark



Supercomputing Benchmark


The number and type of benchmark suites used to study supercomputer performance has varied widely over the years. In early studies, an ad hoc collection of programs was typically used to measure the performance of a given system relative to a known performance benchmark. Eventually, this practice evolved into groups of programs explicitly designed as supercomputer benchmark suites.

The most widely used benchmarks for performance on supercomputing clusters are: the SPEChpc96 suite; the Livermore Loops; and for scientific machines, the Linpack Kernels.

Some general examples of individual computer benchmarks:
  • Dhrystone - Integer benchmark for UNIX systems
  • Whetstone - Floating point benchmark for minicomputers
  • I/O benchmarks
  • MIPS
  • Synthetic benchmarks
  • Kernel benchmarks
  • SPECint / SPECfp
  • Summarizing

Types of Benchmarks

  1. Real program:
  2. Word processing software tool software of CDA user's application software (i.e.: MIS)
  3. Microbenchmark:
  4. Designed to measure the performance of a very small and specific piece of code.
  5. Kernel:
  6. Contains key codes normally abstracted from actual program popular kernel: Livermore loop linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language) results are represented in MFLOPS
  7. Component Benchmark/ micro-benchmark:
  8. Programs designed to measure performance of a computer's basic components automatic detection of computer's hardware parameters like number of registers, cache size, memory latency
  9. Synthetic Benchmark:
  10. Procedure for programming synthetic benchmark: take statistics of all types of operations from many application programs get proportion of each operation write program based on the proportion above Types of Synthetic Benchmark Whetstone, Dhrystone. These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
  11. I/O benchmarks
  12. Database benchmarks: to measure the throughput and response times of database management systems (DBMS')
  13. Parallel benchmarks: used on machines with multiple cores, processors or systems consisting of multiplemachines

Linpack


This is the benchmark used to test the members of Top500 list.

The LINPACK Benchmark was introduced by Jack Dongarra. A detailed description as well as a list of performance results on a wide variety of machines is available in postscript form from netlib. Here you can download the latest version of the LINPACK Report: performance.ps. A parallel implementation of the Linpack benchmark and instructions on how to run it can be found at http://www.netlib.org/benchmark/hpl/.

The benchmark used in the LINPACK Benchmark is to solve a dense system of linear equations. TOP500 uses a version of the benchmark that allows the user to scale the size of the problem and to optimize the software in order to achieve the best performance for a given machine. This performance does not reflect the overall performance of a given system, as no single number ever can. It does, however, reflect the performance of a dedicated system for solving a dense system of linear equations. Since the problem is very regular, the performance achieved is quite high, and the performance numbers give a good correction of peak performance.

By measuring the actual performance for different problem sizes n, a user can get not only the maximal achieved performance Rmax for the problem size Nmax but also the problem size N1/2 where half of the performance Rmax is achieved. These numbers together with the theoretical peak performance Rpeak are the numbers given in the TOP500. In an attempt to obtain uniformity across all computers in performance reporting, the algorithm used in solving the system of equations in the benchmark procedure must conform to LU factorization with partial pivoting. In particular, the operation count for the algorithm must be 2/3 n^3 + O(n^2) double precision floating point operations. This excludes the use of a fast matrix multiply algorithm like "Strassen's Method" or algorithms which compute a solution in a precision lower than full precision (64 bit floating point arithmetic) and refine the solution using an iterative approach.

Top500: LinPack Benchmark



If you want to use Linpack in your system, follow these steps
  1. Download the LinPack Benchmark: LinPack 10.3.4
  2. Extract the content of the packet.
  3. Open the path: ~/linpack_10.3.4/benchmarks/linpack/
  4. Run the executable xlinpack_xeon64 or xlinpack_xeon32 according your architecture
  5. Set these inputs:
    Input data or print help ? Type [data]/help : data
    Number of equations to solve (problem size): 5000
    Leading dimension of array: 10000
    Number of trials to run: 10
    Data alignment value (in Kbytes): 1000000
    
  6. Press enter and wait to the finish
This is my output



There are also a package to run Linpack in HP Clusters, but I will review that when the cluster are ready :)

References

martes, 8 de mayo de 2012

[DPS Lab] Extra Points 3

  • Energy: is an indirectly observed quantity. It is often understood as the ability a physical system has to do work on other physical systems. Since work is defined as a force acting through a distance (a length of space), energy is always equivalent to the ability to exert pulls or pushes against the basic forces of nature, along a path of a certain length.
  • Storage: refers to computer components and recording media that retain digital data. Data storage is a core function and fundamental component of computers.
  • Attention: is the cognitive process of selectively concentrating on one aspect of the environment while ignoring other things. Attention has also been referred to as the allocation of processing resources.
  • Industry: is the production of an economic good or service within an economy.
  • Expert: is someone widely recognized as a reliable source of technique or skill whose faculty for judging or deciding rightly, justly, or wisely is accorded authority and status by their peers or the public in a specific well-distinguished domain.
  • Powerful: Having or capable of exerting power. Effective or potent.
  • Simulate: is the imitation of the operation of a real-world process or system over time.[1] The act of simulating something first requires that a model be developed; this model represents the key characteristics or behaviors of the selected physical or abstract system or process. The model represents the system itself, whereas the simulation represents the operation of the system over time.
  • Peak: The highest, or sometimes the highest and lowest points or value on some measurement, result, graph, etcetera.

domingo, 29 de abril de 2012

[DPS Class] Contributions WEEK 12



This week I did a research about High Availability Techniques in order for maintain the communications between the nodes of our cluster, in that research I include a very simple definitions, and some programs that we can use on our cluster.



Nominations
  • Emmanuel.
  • Saul.
  • Alex
The references are in the page.

cluster.sh

Para automatizar algunas tareas...

domingo, 22 de abril de 2012

[DPS Lab] Extra Points 2

  • Transition: The process of change from one form, state, style or place to another.
  • Event: Is an action that is usually initiated outside the scope of a program and that is handled by a piece of code inside the program. Typical sources of events include the user (who presses a key on the keyboard, in other words, through a keystroke).
  • Execution: Is the process by which a computer or a virtual machine carries out the instructions of a computer program. The instructions in the program trigger sequences of simple actions on the executing machine. Those actions produce effects according to the semantics of the instructions in the program.
  • Graph: Is an abstract representation of a set of objects where some pairs of the objects are connected by links. The interconnected objects are represented by mathematical abstractions called vertices, and the links that connect some pairs of vertices are called edges.
  • Interrumption: Is an asynchronous signal indicating the need for attention or a synchronous event in software indicating the need for a change in execution. A hardware interrupt causes the processor to save its state of execution and begin execution of an interrupt handler. Software interrupts are usually implemented as instructions in the instruction set, which cause a context switch to an interrupt handler similar to a hardware interrupt.
  • Node: A vertex (plural vertices) or node is the fundamental unit out of which graphs are formed: an undirected graph consists of a set of vertices and a set of edges (unordered pairs of vertices), while a directed graph consists of a set of vertices and a set of arcs (ordered pairs of vertices).
  • Disallow: Deny the access or use of something.
  • Condition: Is a feature of a programming language which perform different computations or actions depending on whether a programmer-specified boolean condition evaluates to true or false. Apart from the case of branch predication, this is always achieved by selectively altering the control flow based on some condition.
  • Input: Any data entered into a computer or data processing system,
  • Reachable: Be able to touch or take something with our arms
  • Precise: Is a measure of the detail in which the quantity is expressed. This is usually measured in bits, but sometimes in decimal digits. It is related to precision in mathematics, which describes the number of digits that are used to express a value.

miércoles, 18 de abril de 2012

[DPS Class] Contributions WEEK 11



This week I did a research about Fault Tolerance for the wiki, in that research I include some general information, concepts about the theme and a simple procedure that we can implement in our distributed system


For the laboratory I did a research about Petri Nets, here I include some general concepts [Spanish].


Nominations
  • Roberto Martinez, because his research about the Petri Net Simulator and my research about Petri Nets complements each other very well
  • Cecilia Urbina, because she stills investigating about very good python tools to improve the cluster.
The references are in each page.

[DPS Lab] Red de Petri


Es una representación matemática o gráfica de un sistema a eventos discretos en el cual se puede describir la topología de un sistema distribuido, paralelo o concurrente. La red de Petri esencial fue definida en la década de los años 1960 por Carl Adam Petri. Son una generalización de la teoría de autómatas que permite expresar un sistema a eventos concurrentes. 

Una red de Petri es un grafo orientado con dos tipos de nodos: lugares (representados mediante circunferencias) y transiciones (representadas por segmentos rectos verticales).


Los lugares y las transiciones se unen mediante arcos o flechas.

Fundamentos

  • Una Red de Petri es un modelo gráfico, formal y abstracto para describir y analizar el flujo de información.
  • El análisis de las Redes de Petri ayuda a mostrar información importante sobre la estructura y el comportamiento dinámico de los sistemas modelados.
  • La teoría de las Redes de Petri permite la representación matemática del sistema a ser modelado.
  • Las Redes de Petri son de utilidad en el diseño de sistemas de hardware y software, para especificación, simulación y diseño de diversos problemas de ingeniería.
  • Las Redes de Petri pueden considerarse como autómatas formales o como generadores de lenguajes formales y tienen asociación con la teoría de grafos.
  • Son excelentes para representar procesos concurrentes, así como, procesos donde pueden existir restricciones sobre la concurrencia, precedencia, o frecuencia de esas ocurrencias.

Ventajas
  • Tratamiento individual de procesos independientes.
  • Procesos paralelos o concurrentes.
  • Recursos compartidos.

Estructuras Básicas



Componentes y caracteristicas


Las Redes de Petri están compuestas de cuatro componentes básicos que forman su estructura:
  • Un conjunto de plazas P
  • Un conjunto de transiciones T
  • La función de entrada I
  • La función de salida O
Las funciones de entrada y salida relacionan las transiciones y las plazas.

La función de entrada I es un mapeo a partir del conjunto de plazas de entrada hacia la transición tj, la función se puede escribir como I(tj).

La función de salida O es un mapeo a partir de la transición tj hacia el conjunto de plazas de salida, la función de salida se puede escribir como O(tj).

Construcción


  1. Un arco une siempre lugares con transiciones y nunca dos lugares o dos transiciones.
  2. Una transición puede ser destino de varios lugares y un lugar puede ser el destino devarias transiciones.
  3. Una transición puede ser origen de varios lugares y un lugar puede ser origen de varias transiciones.
  4. Los lugares pueden presentar marcas (una marca se representa mediante un punto en el interior del círculo).


  5. Cada lugar tiene asociada una acción o salida. Los lugares que contiene marcas se consideran lugares activos. Cuando un lugar está activo sus salidas están a uno.
  6. A las transiciones se les asocia eventos (funciones lógicas de las variables de entrada). Una transición se dice que está sensibilizada cuando todos su lugares origen están marcados.
  7. Cuando ocurre un evento asociado a una transición (la función lógica se hace uno), se dice que la transición está validada.

Validación


Consiste en comprobar que se cumplen las propiedades de vivacidad, limitación y reversibilidad:
Hay que considerar :

M0: marcado inicial. De este se desprende el comportamiento del sistema.
[M0] : vector de marcados posibles a partir de un marcado inicial. (marcados alcanzables).
  • Vivacidad: Se trata de un concepto relacionado con la idea de “no bloqueo”. 
    Una transición se dice viva si para un marcado inicial existe una secuencia de franqueos para la cual se puede franquear esa transición. Si todas las transiciones de una red son vivas, la RdP se llama viva y así la red nunca se bloquea.
  • Limitación: Se dice que la red está k- limitada si para todo marcado alcanzable tenemos que ningún lugar tiene un número de marcas mayor que k. Las redes 1-limitadas pueden implementarse mediante biestables, estas redes son conocidas como binarias.
    Si la red diseñada generar más marcas que las que su limitación permite el modelado será erroneo
  • Reversibilidad: Una RdP es reversible si para cualquier marcado alcanzable es posible volver al marcado inicial.
Referencias

miércoles, 4 de abril de 2012

[DPS Class] Contributions WEEK 9



This week I did a research about RPyC or Remote Python Call, that is a library to perform Distributed Computing, Disttibuted Testing, Remote Administration and Tunneling or Chaining.

All the information about this tool is in the following link:

Link: http://elisa.dyndns-web.com/progra/rpyc

I think this is another usefull tool to improve our cluster because, as well we know, python is a powerfull but simple programming languaje, and the most of us know how to code in this languaje.

All the references are also in the Wiki Page.

lunes, 2 de abril de 2012

[DPS Lab] Extra Points

  1. Authentication: is the act of confirming the truth of an attribute of a datum or entity. This might involve confirming the identity of a person or software program, tracing the origins of an artifact, ensuring that a product is what its packaging and labeling claims to be.

  2. Identity: Is a set of data that uniquely describes a person or a thing (sometimes referred to as subject or entity) and contains information about the subject's relationships to other entities

  3. Consistent: When the data in a storage system remains unchanged over a period of time.

  4. DistributionThe act of divide the tasks in several computers.

  5. Scalable: When a system, network, or process, is capable to be enlarged to accommodate that growth in amount of work and performance.

  6. Allocalte: The action of set a portion of a system memory or system storage for specific purposes.

  7. Transaction: The act of exchange or transfer something from one place to another.

  8. Driver: Is a computer program allowing higher-level computer programs to interact with a hardware device through the system bus.

  9. Replica: Is a copy of a system data shared and stored in another computer system.

  10. Sorting: Is the act of arrange a set of data in a particular order.

  11. Security: Is the degree of protection against danger, damage, loss, and crime.

  12. Task: Is an execution path through address space. In other words, a set of program instructions that are loaded in memory. The address registers have been loaded with the initial address of the program. At the next clock cycle, the CPU will start execution, in accord with the program.

jueves, 29 de marzo de 2012

[DPS Lab] Consistency and Replication

Replication




"Is the process of sharing information so as to ensure consistency between redundant resources, such as processes, storage devices and systems, etcetera. 
The same data is shared and/or stored in several devices. 
The replication itself should be transparent to an external user. Also, in a failure scenario, a failover of replicas is hidden as much as possible. "


Types
  • Active: is performed by processing the same request at every replica.
  • Passive: each single request is processed on a single replica and then its state is transferred to the other replicas.

Why perform Replication?
  • Enhance reliability
    • If one replica is unavailable or crashes, use another.
    • Protect against corrupted data
  • Improve performance
    • Scale with size of the distributed system (replicated web servers).
    • Scale in geographically distributed systems
  • Key issue: need to maintain consistency of replicated data.
Keeping replicas up to date needs a lot of network use, so, we need a high speed broadband connection between nodes. The updates need to be atomic (transaction) The replicas need to be synchronized (time consuming)

Consistency


Is the process of keep the same information in all the systems all the time.

All the replicas must be consistent, if one copy is modified, others become inconsistent; so:
  • Modifications have to be carried out on all copies.
  • Problems with network performance.
  • It is needed to handling concurrency.
Consistency models are used in distributed systems like distributed shared memory systems or distributed data stores (such as a filesystems, databases, optimistic replication systems or Web caching). The system supports a given model, if operations on memory follow specific rules. The data consistency model specifies a contract between programmer and system, wherein the system guarantees that if the programmer follows the rules, memory will be consistent and the results of memory operations will be predictable. The most common models are:
  • Casual Consistency: Writes that are potentially casually related must be seen by all processes in the same order. Concurrent writes may be seen in a different order on different machines.
  • FIFO Consistency: Writes done by a single process are seen by all other processes in the order in which they were issued, but writes from different processes may be seen in a different order by different processes.
  • Strict Consistency: Any read always returns the result of the most recent write, implicitly assumes the presence of a global clock. A write is immediately visible to all processes.
  • Secuential Consistency: Weaker than strict consistency. Assumes all operations are executed in some sequential order and each process issues operations in program order. Any valid interleaving is allowed. All agree on the same interleaving. Each process preserves its program order. Nothing is said about “most recent write”
Consistency Model Comparison



References

miércoles, 28 de marzo de 2012

[DPS Class] Contributions WEEK 8



Last week, the members of the cluster team did a work meeting. We talked about the importance of join our efforts to build only one cluster instead of work separately in small groups.

We decided the specifications of each computer that are in cluster:
  • Operating System: Ubuntu 10.04 64bits
  • User: cluster
  • Password: (1)
Also, we did the schedule of the activities that we need to do to configure the cluster.
  1. Install all the software and packages.
  2. Configure VPN.
  3. Configure a computing grid.
In a second stage, we are going to integrate a website, where will be possible to upload the tasks, which will be validated, executed and processed by the cluster, and at the final we can download the results, maybe in a compressed file.
This task will be done by Emmanuel, with some tools like:
  1. Apache, PHP
  2. Ruby on Rails
  3. Configure a computing grid.

I think this was necessary. Now we are working together and I hope that the results are even more significant than before.

Nominations
  • Cecy Urbina: Because she remains active in exam weeks
  • Roberto Martinez: Because he remains active in exam weeks
  • Rafael Lopez: Because he participate actively in the meeting, distributing task and clarifying doubts.

(1): Check the Google Document posted in our Facebook page to check this information. (Security :P)

lunes, 12 de marzo de 2012

[DPS Class] Contributions WEEK 6



This week, my partner Rafael Lopez and work on the interconnection of two VPN, one in each home, using two different methods:
  • OpenVPN: I've updated this entry, check the new information at the bottom.
  • HamuchiBy Rafael Lopez
this will serve to us in the future to connect 2 completely separate cluster and make a grid.

For the laboratory, I make the entry: Grid Computing & Cluster Computing

I was unable to login to the wiki because the server stills down, so I wrote the entries in my blog and I will transcribe them to the wiki later.

NOMINATIONS
  • Rafael Lopez For the implementation of a Hamuchi Network, and he still helps me in all the work, and also because he called the group for the first meeting to check the progress of the proyect.
  • Cecilia Urbina for the implementation of Parallel PI Calculation
  • Roberto Martinez for the implemtation of PseudoRMI in Python using Pyro

sábado, 10 de marzo de 2012

[DPS Lab] Cluster Computing & Grid Computing

Grid computing is focused on the ability to support computation across administrative domains sets it apart from traditional computer clusters or traditional distributed computing. Grids offer a way of using the information technology resources optimally inside an organization. In short, it involves virtualizing computing resources. Grid computing is often confused with cluster computing. Functionally, one can classify grids into several types: Computational Grids (including CPU scavenging grids), which focuses primarily on computationally-intensive operations, and Data grids, or the controlled sharing and management of large amounts of distributed data.

Definition


There are many definitions of the term Grid computing:
  • A service for sharing computer power and data storage capacity over the Internet
  • An ambitious and exciting global effort to develop an environment in which individual users can access computers, databases and experimental facilities simply and transparently, without having to consider where those facilities are located.
  • Grid compting is a model for allowing companies to use a large number of computing resources on demand, no matter where they are located.

Cluster Computing VS. Grid Computing


When two or more computers are used together to solve a problem, it is called a computer cluster . Then there are several ways of implementing the cluster, Beowulf is maybe the most known way to do it, but basically it is just cooperation between computers in order to solve a task or a problem. Cluster Computing is then just the thing you do when you use a computer cluster.

Grid computing is something similar to cluster computing, it makes use of several computers connected is some way, to solve a large problem. There is often some confusion about the difference between grid vs. cluster computing. The big difference is that a cluster is homogenous while grids are heterogeneous. The computers that are part of a grid can run different operating systems and have different hardware whereas the cluster computers all have the same hardware and OS. A grid can make use of spare computing power on a desktop computer while the machines in a cluster are dedicated to work as a single unit and nothing else.

Grid are inherently distributed by its nature over a LAN, metropolitan or WAN. On the other hand, the computers in the cluster are normally contained in a single location or complex.

Another difference lies in the way resources are handled. In case of Cluster, the whole system (all nodes) behave like a single system view and resources are managed by centralized resource manager. In case of Grid, every node is autonomous i.e. it has its own resource manager and behaves like an independent entity.

Characteristics of Grid Computing

  • Loosely coupled (Decentralization)
  • Diversity and Dynamism
  • Distributed Job Management & scheduling

Characteristics of Cluster computing

  • Tightly coupled systems
  • Single system image
  • Centralized Job management & scheduling system

Areas of Grid Computing and it's applications for modeling and computing


  1. Predictive Modeling and Simulations
  2. Predictive Modeling is done through extensive computer simulation experiments, which often involve large-scale computations to achieve the desired accuracy and turnaround time. It can also be called "Modeling the Future". Such numerical modeling requires state-of-the-art computing at speeds approaching 1 GFLOPS and beyond. In case of Computational Biology , It is the modeling and simulation of self-organizing adaptive response of systems where spatial and proximal information is of paramount importance. Areas of Research include:
    • Numerical Weather Forecasting.
    • Flood Warning
    • Semiconductor Simulation
    • Oceanography
    • Astrophysics (Modeling of Black holes and Astronomical formations)
    • Sequencing of the human genome
    • Socio-economic and Government use
  3. Engineering Design and Automation
    • Finite-element analysis
    • Computational aerodynamics
    • Remote Sensing Applications
    • Artificial Intelligence and Automation
    This areas requires parallel processing for the following intelligence functions:
    • Image Processing
    • Pattern Recognition
    • Computer Vision
  4. Energy Resources Exploration
    • Seismic Exploration
    • Reservoir Modeling
    • Plasma Fusion Power
    • Nuclear Reactor Safety
  5. Medical, Military and Basic Research
    • Medical Imaging
    • Quantum Mechanics problems
    • Polymer Chemistry
    • Nuclear Weapon Design
  6. Visualization
    • Computer-generated graphics, films and animations
    • Data Visualization
    i

This is an excelent document that explain the diferences and characteristics of Cluster, Grid ans Cloud computing: Cluster, Grid and Cloud Computing

References

jueves, 1 de marzo de 2012

[DPS Class] Contributions WEEK 5



This week, my partner Rafael Lopez and me were working on the installation and configuration of a VPN server, one in each home, this will serve to us in the future to connect 2 completely separate cluster. This is known as GRID COMPUTING

I was unable to login to the wiki, so I wrote the entries in my blog and when I fix the problem with my account I will transcribe them to the wiki.

LINKS:
NOMINATIONS
  • Rafael Lopez: It can be very repetitive, but I think he's one of those who most actively participated iand contributed to the creation of the cluster, and which has given more results.

miércoles, 29 de febrero de 2012

[DPS Class] OpenVPN



"A virtual private network (VPN) is a network that uses primarily public telecommunication infrastructure, such as the Internet, to provide remote offices or traveling users an access to a central organizational network.

VPNs typically require remote users of the network to be authenticated, and often secure data with encryption technologies to prevent disclosure of private information to unauthorized parties.

VPNs may serve any network functionality that is found on any network, such as sharing of data and access to network resources, printers, databases, websites, etc. A VPN user typically experiences the central network in a manner that is identical to being connected directly to the central network. VPN technology via the public Internet has replaced the need to requisition and maintain expensive dedicated leased-line telecommunication circuits once typical in wide-area network installations.

Virtual private network technology reduces costs because it does not need physical leased lines to connect remote users to an Intranet."


For more information about VPN, visit VPN (Wikipedia)

OpenVPN



"OpenVPN is a software based connectivity solution, uses SSL (Secure Sockets Layer) and Virtual Private Network VPN (virtual private network) technology.
OpenVPN offers point-to-point connectivity with hierarchical validation of remotely connected users and host, is a very good option for Wi-Fi environments (wireless networks IEE 802.11) and supports a wide settings, including load balancing and more. It is released under the GPL, free software.

No other solution offers such a mix of enterprise-level security, safety, ease of use and rich features.
OpenVPM simplified the configuration of VPN's reducing the difficult to configure other solutions such as IPsec and making it more accessible for people inexperienced in this type of technology."


For more information, visit OpenVPN Site

Installing and configuring OpenVPN on Ubuntu 10.04 LTS 32Bits


NOTE: I don't include screenshots of the installation because I had some problems during the process, however, I will put the correct commands that if you execute it correctly, you will have no problems. Also, all of the following instructions, unless otherwise indicated, must be executed in the VPN server.


First, the installation of the packages:
suda apt-get install openvpn openssl
Our working directory will be /etc/openvpn/
cd /etc/openvpn

Now that the openvpn package is installed, the certificates for the VPN server need to be created.
First, copy the easy-rsa directory to /etc/openvpn. This will ensure that any changes to the scripts will not be lost when the package is updated. You will also need to adjust permissions in the easy-rsa directory to allow the current user permission to create files.
sudo mkdir /etc/openvpn/easy-rsa/
sudo cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* /etc/openvpn/easy-rsa/
sudo chown -R $USER /etc/openvpn/easy-rsa/

Next, edit /etc/openvpn/easy-rsa/vars adjusting the values to your environment, this is my configuration:
export KEY_COUNTRY="MX"
export KEY_PROVINCE="NL"
export KEY_CITY="Monterrey"
export KEY_ORG="JuanCarlos"
export KEY_EMAIL="your_email@example.com"

Now, enter the following commands in order to create the server certificates:
cd /etc/openvpn/easy-rsa/
source vars
./clean-all
./build-dh
./pkitool --initca
./pkitool --server server
cd keys
openvpn --genkey --secret ta.key
sudo cp server.crt server.key ca.crt dh1024.pem ta.key /etc/openvpn/

Now, enter the following commands in order to create the client certificates, replace CLIENTNAME with the hostname of the client, to know which it are, run hostname in a terminal of a client:
cd /etc/openvpn/easy-rsa/
source vars
./pkitool CLIENTNAME

Now configure the openvpn server by creating /etc/openvpn/server.conf from the example file:
sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
sudo gzip -d /etc/openvpn/server.conf.gz
Backup /etc/openvpn/server.conf:
sudo mv /etc/openvpn/server.conf /etc/openvpn/server.conf.bak
Create a new /etc/openvpn/server.conf with the following options to:
dev tun
proto tcp
port 1194
ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh1024.pem
user nobody
group nogroup
server 10.8.0.0 255.255.255.0
persist-key
persist-tun
status openvpn-status.log
#verb 3
client-to-client
push "redirect-gateway def1"
#log-append /var/log/openvpn
#comp-lzo 

After configuring the server, restart openvpn by entering:
sudo /etc/init.d/openvpn restart

Configuring the Clients


First, the installation of the packages:
suda apt-get install openvpn openssl
Also, our working directory will be /etc/openvpn/
cd /etc/openvpn
Make some directories:
sudo mkdir /etc/openvpn/easy-rsa/
sudo mkdir /etc/openvpn/easy-rsa/keys

From the server, copy the following files to the client, and place them in the appropriate folder:
/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.key
/etc/openvpn/ta.key
Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before.
Then with the server configured and the client certificates copied to the /etc/openvpn/ directory, create a client configuration file by copying the example. In a terminal on the client machine enter:
sudo cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /etc/openvpn
Backup /etc/openvpn/client.conf:
sudo mv /etc/openvpn/client.conf /etc/openvpn/client.conf.bak
Create a new /etc/openvpn/client.conf with the following options to, in the line remote 123.456.789.000 1194 , replace 123.456.789.000 with the public IP of your server or the hostname of your server:
dev tun
client
proto tcp
remote 123.456.789.000 1194
resolv-retry infinite
nobind
user nobody
group nogroup
# Try to preserve some state across restarts.
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
key /etc/openvpn/easy-rsa/keys/CLIENTNAME.key
comp-lzo
# Set log file verbosity.
verb 3 
Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before and which already must be copied to the client.

Finally, restart openvpn:
sudo /etc/init.d/openvpn restart
Now you should now be able to connect to the remote LAN through the VPN. If you run ifconfig command, you should see a new tunneled interface, something like this:



If you look, a new IP address is assigned to the client, that is a private IP address of our VPN.

Also, you can test the network with some pings :)




UPDATE (11/mar/2012) Configuring and Connecting 2 remote VPN


Firts, follow all the previous steps to build a fully functional local VPN.

We do our cluster, a client of another one. For this, in the remote VPN server, open a terminal and make the corresponding certificates for the client, replace CLIENTNAME with the hostname of the local VPN server, to know which it are, run hostname in a terminal of a the local VPN server:
cd /etc/openvpn/easy-rsa/
source vars
./pkitool CLIENTNAME

Back in our local VPN, copy the following files from the remote VPN server to the local VPN server, and place them in the appropriate folder:
/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
/etc/openvpn/easy-rsa/keys/CLIENTNAME.key
/etc/openvpn/ta.key
Remember CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before.
Then create a the client configuration file by copying the example. In a terminal of the local VPN server enter:
sudo cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /etc/openvpn
Backup /etc/openvpn/client.conf:
sudo mv /etc/openvpn/client.conf /etc/openvpn/client.conf.bak
Create a new /etc/openvpn/client.conf with the following options to, in the line remote 123.456.789.000 1194 , replace 123.456.789.000 with the domain of the remote VPN server, is highly recommended to use a service like dyndns:
dev tun
client
proto tcp
remote remote.vpn.domain 1194
resolv-retry infinite
nobind
user nobody
group nogroup
# Try to preserve some state across restarts.
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/easy-rsa/keys/CLIENTNAME.crt
key /etc/openvpn/easy-rsa/keys/CLIENTNAME.key
comp-lzo
# Set log file verbosity.
verb 3 
Where CLIENTNAME.crt and CLIENTNAME.key are the certificates created in the server before and which already must be copied to the client.

Finally, restart openvpn in the local VPN server:
sudo /etc/init.d/openvpn restart
Now you should be able to connect to the remote LAN through the VPN, also, you can see how start the client and server daemon in the local VPN server:

[IMAGE]

You must have the tunneled interface like before, and now, if you run the comand route, you should be able to see the IP range of your VPN and also the IP range of the remote VPN.

[IMAGE] https://help.ubuntu.com/10.04/serverguide/openvpn.html

[DPS Lab] How to open ports on router

This is a guide where I explain how to open ports on a router, in this case, a router Thompson TG585v7



First, we need to login to our modem. Open a web browser and type the following address:

192.168.1.254

We are going to see the following screen:



In the fields, type the following data:

User name: TELMEX
Password: Your WEP KEY. (check the labels of your router)

If the information is correct we will see the main configuration screen:



Then, in the sidebar we will select the option "Herramientas" ("tools"), we will see this screen:



In Herramientas (tools) screen, we are going to select the option "Comparticion de juegos y aplicaciones" (Games and Applications Sharing). We will enter to this section:



Here is where we have control over ports, we can open, close and assign to any device.
To do this, at the bottom of the screen we will select the option "Crear neuvo juego o aplicacion" ("Create new game or application").
We will see this screen:



Here we will enter a name for our new game or application. In the options below we will select the one that says "Entrada manual de mapas de puertos" ("Manual Entry of Port Maps") and we click Next.

On the next screen:



we are going to define the range of ports, which can be as wide as we need it. In this case I just need to open only one port the 1194, so my interval is from 1194 to 1194.
The other options are left with the default values ​​that have, we click on "Agregar" ("Add"), the screen will refresh and we will see a table that was not previously with the newly opened ports.



Next, we just need to assign that port to a device, basically, this is a type of redirection, because if a request comes to our router with a specific port, the request will be redirected to the device that is assigned that port.
For this we go to the options at the bottom of the modem setup screen and select the option "Asignar un juego o aplicacion a un dispositivo de red local" ("Assign a game or application to a local network device").
Then, in the next screen:



in the table, we will choose the game which we are going to assign, and the device which we are going to assign a game.
Once we have specified these data, we will click the button "Agregar" ("add").



And that is all, we have opened a port and also have set a condition to redirect traffic to the computer or device that will -serve the request.

miércoles, 22 de febrero de 2012

[DPS Class] Contributions WEEK 4

We might made this some day...

For this week, I was researching with my partner Rafael Lopez about Beowulf cluster and MPI systems. So we made two blog entries where we share the gathered information.




Also, we implement the John The Ripper application, you saw the live execution the past tuesday.

For the next week, we are beggining to research about Parallel CUDA, so Rafael Lopez and me we are working in the construction of a GPU cluster for the next week, some examples and maybe a live execution.

NOMINATIONS:

  • Rafael Lopez, because he help me a lot to understand some cluster concepts and configurations.
  • Emmanuel, because he improve the entry "MPI Samples" with an extra example.

That's all.

jueves, 16 de febrero de 2012

[DPS Class] Contributions WEEK 3

In the picture, CUDA summary of my computer :)


For this week, I extend my previous entry for the Wiki. I add a new method to install CUDA because all the other tutorials of my partners won't work for me, I think this will be helpful because the problems that I had with the installation are common in the internet portals.
LINK: GPU Computing with CUDA | Installation Section

Also, Rafael Lopez and me we are working in the construction of a cluster using John The Ripper Architecture. This are under developement

This week, I`m going to nominate Gaby Garcia because she made some compilations of examples in CUDA , the next step after my installation tutorial.


References

miércoles, 8 de febrero de 2012

[DPS Class] Contributions WEEK 2



As a contribution for this week, I wrote some information about Compute Unified Device Architecture (CUDA) in the Wiki, with some images and a short description in each one.

Link: http://elisa.dyndns-web.com/progra/CUDA


Also, I wrote a little entry about Supercomputing in Mexico.

Link:


Also, I do the translation of the content of the Cluster Team page to english. :D

That's all for this week. :)

[DPS Lab] Supercomputing in Mexico

Since 2003, Mexico has been in the TOP500, this classification list 2 times a year the ranking of the 500 most powerful computers in the world.

In first we have the supercomputer "K Computer", K comes from the japanese word "Kei" which means 10 quadrillion.

Characteristics
  • Location: RIKEN Advanced Institute for Computational Science (Kobe, Japan)
  • Manufacturer: Fujitsu
  • Date of Operation: June 2011 (Fully Operational November 2011)
  • Operating System: Linux
Technical Specifications
  • 88.128 2.0GHz 8-core SPARC64 processors VIIIfx packed in 864 cabinets (705.024 cores)
  • 1410048 GB of memory
  • 12659.89 kW of Power Consumption
  • Water Cooling System
Taked from http://i.top500.org/system/177232


In the center of the country was created the LANCAD project (Laboratorio Nacional de Computo de Alto Desempeño [National Laboratory of High Performance Computing]) with the contribution of the Universidad Autónoma Metropolitana, Instituto Politécnico Nacional (CINVESTAV), and Universidad Nacional Autónoma de México.

Each of these 3 institutions host a cluster of computation:

UNAM: Kan Balam

The UNAM host the supercomputer Kan Balam, it was developed by HP and has 1368 AMD 2.6 GHz Opteron Core processors, 3016 Gigabytes of RAM and a 160 terabytes storage, stored in 19 racks, which together, they use an area of 15 to 20 meters square.
KanBalam comunications operates through a high speed network with Infiniband technology at 10 gigabits per second.
Kan Balam has processing capacity to perform 7,113 trillion arithmetic operations per second and is capable of delivering this capability up to 350 users. The computer can increase their capacity in the future if necessary since this consists of servers, each with 4 processors.



UAM: Aitzaloa

The Aitzaloa cluster consists in 3 main parts.

High Performance Computing Node (HPC)
  • Number of Nodes: 270 (135 Supermicro's Twin) nodes
  • Processors: Quad-Core Intel Xeon 3.0 Ghz with 1600 MHz Front Side Bus
  • Number of Processors: Quad-Core 540 (2160 CORES)
  • Memory: 16GB RAM per node (4320 Total Distributed)
  • Computing Capacity: 18.4 TFlops.
  • Communications: Infiniband
  • Operating System: Linux Centos 5.2
Storage System
  • Comprising: 4 servers HP-Proliant DL380G5
  • Storage System: 100TB Luste
  • Hard Drives: 150 1TB drives in Raid 1 and Raid 6
  • Communications: Infiniband and Gigabit Ethernet
Master node
  • Processors: 2 Intel Xeon 2.8GHz Quad-Core
  • Front Side Bus: 1600 MHz
  • Memory: 32GB RAM
  • Communications: Infiniband and Gigabit Ethernet
  • Operating System: Linux
  • Distribution: Centos 5.2
  • Local Storage: 9TB

The companies that contributed to this project are HP and Sun Microsystems



IPN (CINVESTAV): Xiuhcoatl

In late January, CINVESTAV present Xiuhcoatl supercomputer, which in Nahuatl means "fire serpent"

It has 3.480 Intel and AMD processors with a capacity of 24.97 teraflops and 7,200 GB of RAM.
It has a storage capacity of 45.350 GB hard disk and reaching peak computing capacity would require between 70 and 80 kilowatt hours of power consumption.
This includes 170 servers supercomputer capable of performing 18 trillion mathematical calculations per second.
At U.S. cluster hybrid type, which integrates Intel and AMD processors, as well as graphics processing units contecnología GPGPU (General-Purpose Computation on Graphics Processing Units).




The "grid" has a capacity of about 50 Teraflops.

Xiuhcoatl, Kan Balam and Aitzaloa totaling 7 000 cores, its are interconnected by a fiber optic network from the Metro IPN to University, to UAM.

Some planned projects for this cluster range from modeling of proteins that cause Alzheimer's disease (interaction between the atoms), simulations of earth's climate, tsunamis and the formation of stars

The country is moving in the field of Supercomputers, however, the center are getting all the credit. Will the Northern students able to contribute to the supercomputing?

References

miércoles, 1 de febrero de 2012

[DPS Lab] Computing Cluster and Parallel Computing

Taked from http://www.phys.ntu.edu.tw



In simple words, a computing cluster is a group of computers that work together, distributing tasks and sharing hardware and software. By these methods the computing capabilities can grow significantly.
Each computer runs its own operating system, and all are interconnected via a local area network (LAN).





Types of Clusters:

Based on its characteristics, the clusters are classified into 3 types:

  • 1. High Availability Clusters
  HA Clusters are designed to ensure constant access to service applications. The clusters are designed to maintain redundant nodes that can act as backup systems in the event of failure. The minimum number of nodes in a HA cluster is two – one active and one redundant – though most HA clusters will use considerably more nodes. 
HA clusters aim to solve the problems that arise from mainframe failure in an enterprise. Rather than lose all access to IT systems, HA clusters ensure 24/7 access to computational power. This feature is especially important in business, where data processing is usually time-sensitive. 

  • 2. Load-balancing Clusters
  Load-balancing clusters operate by routing all work through one or more load-balancing front-end nodes, which then distribute the workload efficiently between the remaining active nodes. Load-balancing clusters are extremely useful for those working with limited IT budgets. Devoting a few nodes to managing the workflow of a cluster ensures that limited processing power can be optimized. 

  • 3. High-performance Clusters
  HPC clusters are designed to exploit the parallel processing power of multiple nodes. They are most commonly used to perform functions that require nodes to communicate as they perform their tasks – for instance, when calculation results from one node will affect future results from another. 


Cluster Benefits

Some of the benefits of cluster computing are:

Reduced Cost:  The price of off-the-shelf consumer desktops has plummeted in recent years, and this drop in price has corresponded with a vast increase in their processing power and performance. The average desktop PC today is many times more powerful than the first mainframe computers. 

Processing Power:  The parallel processing power of a high-performance cluster can, in many cases, prove more cost effective than a mainframe with similar power. This reduced price per unit of power enables enterprises to get a greater ROI from their IT budget. 

Improved Network Technology: Driving the development of computer clusters has been a vast improvement in the technology related to networking, along with a reduction in the price of such technology. 

Scalability: Perhaps the greatest advantage of computer clusters is the scalability they offer. While mainframe computers have a fixed processing capacity, computer clusters can be easily expanded as requirements change by adding additional nodes to the network. 

Availability: When a mainframe computer fails, the entire system fails. However, if a node in a computer cluster fails, its operations can be simply transferred to another node within the cluster, ensuring that there is no interruption in service. 



Parallel Computing

Parallel computing is based on the concept of "divide and conquer", divide a task into smaller parts and execute them simultaneously. When each of these small parts ends, generates a part of the final result, when all tasks complete the sub-results are merged to make the final result.

You need to have multi-core computers to carry out this type of calculation, because each core is responsible for carrying out one of the subtasks.
Our world is highly parallelized, take for example an anthill, to keep alive the colony is a common task, and each of the individuals in the colony do their part to complete.

A computer instructions commonly performed in series. When a algorithm is parallelizable it's usually divided into small works called threads, then we apply some synchronization tasks to execute the tasks at the time and order (eg, read or modify a variable). Some synchronization methods are:
  • Locks
  • Condition variables
  • Semaphores

These methods prevent the execution of a critical part of the code at the same time by two or more threads causing an error in the execution or errors in the final result.

Parallelism Levels

  • Bit: Increases the processor word size. Increasing the word size reduces the number of instructions the processor must execute in order to perform an operation on variables whose sizes are greater than the length of the word. (For example, consider a case where an 8-bit processor must add two 16-bit integers.
  • Instruction: Is a measure of how many of the operations in a computer program can be performed simultaneously.
  • Data: Focuses on distributing the data across different parallel computing nodes. Is achieved when each processor performs the same task on different pieces of distributed data
  • Task: Focuses on distributing execution processes (threads) across different parallel computing nodes.

Classes of parallel computers

Parallel computers can be roughly classified according to the level at which the hardware supports parallelism. This classification is broadly analogous to the distance between basic computing nodes. These are not mutually exclusive; for example, clusters of symmetric multiprocessors are relatively common.
  • Multicore computing
  • Symmetric multiprocessing
  • Distributed computing
    • Cluster computing: Interconnected computers.
    • Grid computing: Interconnected clusters.

Referencias:

[DPS Class] Monitoring the efficiency of a computing cluster



This week I did a research of various systems that can help us to monitor the efficiency of our cluster and the available resources.

Because these are monitoring systems, the contribution is in the software section.

The link to this contribution is Monitoring the efficiency of a computing cluster

Where you can find more details and links for various applications.

Greetings:).