Tag: linux

New Amazon Instance: High I/O for NoSQL

Latest from the Amazon Cloud-

hi1.4xlarge instances come with eight virtual cores that can deliver 35 EC2 Compute Units (ECUs) of CPU performance, 60.5 GiB of RAM, and 2 TiB of storage capacity across two SSD-based storage volumes. Customers using hi1.4xlarge instances for their applications can expect over 120,000 4 KB random write IOPS, and as many as 85,000 random write IOPS (depending on active LBA span). These instances are available on a 10 Gbps network, with the ability to launch instances into cluster placement groups for low-latency, full-bisection bandwidth networking.

High I/O instances are currently available in three Availability Zones in US East (N. Virginia) and two Availability Zones in EU West (Ireland) regions. Other regions will be supported in the coming months. You can launch hi1.4xlarge instances as On Demand instances starting at $3.10/hour, and purchase them as Reserved Instances

http://aws.amazon.com/ec2/instance-types/

High I/O Instances

Instances of this family provide very high instance storage I/O performance and are ideally suited for many high performance database workloads. Example applications include NoSQL databases like Cassandra and MongoDB. High I/O instances are backed by Solid State Drives (SSD), and also provide high levels of CPU, memory and network performance.

High I/O Quadruple Extra Large Instance

60.5 GB of memory
35 EC2 Compute Units (8 virtual cores with 4.4 EC2 Compute Units each)
2 SSD-based volumes each with 1024 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
Storage I/O Performance: Very High*
API name: hi1.4xlarge

*Using Linux paravirtual (PV) AMIs, High I/O Quadruple Extra Large instances can deliver more than 120,000 4 KB random read IOPS and between 10,000 and 85,000 4 KB random write IOPS (depending on active logical block addressing span) to applications. For hardware virtual machines (HVM) and Windows AMIs, performance is approximately 90,000 4 KB random read IOPS and between 9,000 and 75,000 4 KB random write IOPS. The maximum sequential throughput on all AMI types (Linux PV, Linux HVM, and Windows) per second is approximately 2 GB read and 1.1 GB write.

Revolution R Enterprise 6.0 launched!

Just got the email-more software is good news!

Revolution R Enterprise 6.0 for 32-bit and 64-bit Windows and 64-bit Red Hat Enterprise Linux (RHEL 5.x and RHEL 6.x) features an updated release of the RevoScaleR package that provides fast, scalable data management and data analysis: the same code scales from data frames to local, high-performance .xdf files to data distributed across a Windows HPC Server cluster or IBM Platform Computing LSF cluster.  RevoScaleR also allows distribution of the execution of essentially any R function across cores and nodes, delivering the results back to the user.

Detailed information on what’s new in 6.0 and known issues:
http://www.revolutionanalytics.com/doc/README_RevoEnt_Windows_6.0.0.pdf

and from the manual-lots of function goodies for Big Data

 

  • IBM Platform LSF Cluster support [Linux only]. The new RevoScaleR function, RxLsfCluster, allows you to create a distributed compute context for the Platform LSF workload manager.
  •  Azure Burst support added for Microsoft HPC Server [Windows only]. The new RevoScaleR function, RxAzureBurst, allows you to create a distributed compute context to have computations performed in the cloud using Azure Burst
  • The rxExec function allows distributed execution of essentially any R function across cores and nodes, delivering the results back to the user.
  • functions RxLocalParallel and RxLocalSeq allow you to create compute context objects for local parallel and local sequential computation, respectively.
  • RxForeachDoPar allows you to create a compute context using the currently registered foreach parallel backend (doParallel, doSNOW, doMC, etc.). To execute rxExec calls, simply register the parallel backend as usual, then set your compute context as follows: rxSetComputeContext(RxForeachDoPar())
  • rxSetComputeContext and rxGetComputeContext simplify management of compute contexts.
  • rxGlm, provides a fast, scalable, distributable implementation of generalized linear models. This expands the list of full-featured high performance analytics functions already available: summary statistics (rxSummary), cubes and cross tabs (rxCube,rxCrossTabs), linear models (rxLinMod), covariance and correlation matrices (rxCovCor),
    binomial logistic regression (rxLogit), and k-means clustering (rxKmeans)example: a Tweedie family with 1 million observations and 78 estimated coefficients (categorical data)
    took 17 seconds with rxGlm compared with 377 seconds for glm on a quadcore laptop

     

    and easier working with R’s big brother SAS language

     

    RevoScaleR high-performance analysis functions will now conveniently work directly with a variety of external data sources (delimited and fixed format text files, SAS files, SPSS files, and ODBC data connections). New functions are provided to create data source objects to represent these data sources (RxTextData, RxOdbcData, RxSasData, and RxSpssData), which in turn can be specified for the ‘data’ argument for these RevoScaleR analysis functions: rxHistogram, rxSummary, rxCube, rxCrossTabs, rxLinMod, rxCovCor, rxLogit, and rxGlm.


    example, 

    you can analyze a SAS file directly as follows:


    # Create a SAS data source with information about variables and # rows to read in each chunk
    sasDataFile <- file.path(rxGetOption(“sampleDataDir”),”claims.sas7bdat”)
    sasDS <- RxSasData(sasDataFile, stringsAsFactors = TRUE,colClasses = c(RowNum = “integer”),rowsPerRead = 50)

    # Compute and draw a histogram directly from the SAS file
    rxHistogram( ~cost|type, data = sasDS)
    # Compute summary statistics
    rxSummary(~., data = sasDS)
    # Estimate a linear model
    linModObj <- rxLinMod(cost~age + car_age + type, data = sasDS)
    summary(linModObj)
    # Import a subset into a data frame for further inspection
    subData <- rxImport(inData = sasDS, rowSelection = cost > 400,
    varsToKeep = c(“cost”, “age”, “type”))
    subData

 

The installation instructions and instructions for getting started with Revolution R Enterprise & RevoDeployR for Windows: http://www.revolutionanalytics.com/downloads/instructions/windows.php

Oracle R Updated!

Interesting message from https://blogs.oracle.com/R/ the latest R blog

 

_——–_

Oracle just released the latest update to Oracle R Enterprise, version 1.1. This release includes the Oracle R Distribution (based on open source R, version 2.13.2), an improved server installation, and much more.  The key new features include:

  • Extended Server Support: New support for Windows 32 and 64-bit server components, as well as continuing support for Linux 64-bit server components
  • Improved Installation: Linux 64-bit server installation now provides robust status updates and prerequisite checks
  • Performance Improvements: Improved performance for embedded R script execution calculations

In addition, the updated ROracle package, which is used with Oracle R Enterprise, now reads date data by conversion to character strings.

We encourage you download Oracle software for evaluation from the Oracle Technology Network. See these links for R-related software: Oracle R Distribution, Oracle R Enterprise, ROracle, Oracle R Connector for Hadoop.  As always, we welcome comments and questions on the Oracle R Forum.

 

 

Oracle R Distribution 2-13.2 Update Available

Oracle has released an update to the Oracle R Distribution, an Oracle-supported distribution of open source R. Oracle R Distribution 2-13.2 now contains the ability to dynamically link the following libraries on both Windows and Linux:

  • The Intel Math Kernel Library (MKL) on Intel chips
  • The AMD Core Math Library (ACML) on AMD chips

 

To take advantage of the performance enhancements provided by Intel MKL or AMD ACML in Oracle R Distribution, simply add the MKL or ACML shared library directory to the LD_LIBRARY_PATH system environment variable. This automatically enables MKL or ACML to make use of all available processors, vastly speeding up linear algebra computations and eliminating the need to recompile R.  Even on a single core, the optimized algorithms in the Intel MKL libraries are faster than using R’s standard BLAS library.

Open-source R is linked to NetLib’s BLAS libraries, but they are not multi-threaded and only use one core. While R’s internal BLAS are efficient for most computations, it’s possible to recompile R to link to a different, multi-threaded BLAS library to improve performance on eligible calculations. Compiling and linking to R yourself can be involved, but for many, the significantly improved calculation speed justifies the effort. Oracle R Distribution notably simplifies the process of using external math libraries by enabling R to auto-load MKL orACML. For R commands that don’t link to BLAS code, taking advantage of database parallelism usingembedded R execution in Oracle R Enterprise is the route to improved performance.

For more information about rebuilding R with different BLAS libraries, see the linear algebra section in the R Installation and Administration manual. As always, the Oracle R Distribution is available as a free download to anyone. Questions and comments are welcome on the Oracle R Forum.

Teradata updates Teradata-R

The Teradata add-on package for R

teradataR is a package or library that allows R users to easily connect to Teradata, establish data frames (R data formats) to Teradata and to call in-database analytic functions within Teradata.  This allows R users to work within their R console environment while leveraging the in-database functions developed with Teradata Warehouse Miner. This package provides 44 different analytical functions and an additional 20 data connection and R infrastructure functions.  In addition, we’ve added a function that will list the stored procedures within Teradata provide the capability to call functions from R.

  • 20 Functions to enable R infrastructure to operate with Teradata
  • tdConnect – Connect to Teradata via ODBC
  • Td.data.frame – Establish data frame connections to a Teradata table
  • 44 in-database analytical functions callable from R.  Sample of the functions include:
  • Descriptive statistics: Overlap, histogram, frequency, statistics, matrix functions, and values analysis
  • Reorganization functions: join, merge and samples
  • Transformations: bincode, recode, rescale, sigmoid, zscore and null replacement
  • K-Means clustering and Score K-Means
  • Statistical tests: ks, dagostino.pearson, shapiro.wilk, bionomial, and wilcoxon
  • R language features nrow, ncol, min, max, summary, as.dataframe, and dim
  • Tool and R functions that allow users to create their own custom analytic functions that’s callable by R.
  • Teradata Warehouse Miner can capture any analytic stream including UDFs and create a stored procedure
    • Analytic process to create new derived predictive variables can be captured as a stored procedure.
    • Entire process to create or update an analytical data set can be captured as a stored procedure.
    • R function can list all the stored procedures within Teradata.
    • R function can call a stored procedure that runs in-database

TeradataR allows R users to leverage all the benefits of in-database processing with Teradata:

  • Eliminate data movement from Teradata to the R framework for key data intensive tasks.
  • Leverage the speed of Teradata database’s parallel processing to run analytics against big data.
  • Ability to operate within the R console environment.
  • Embed your frequently performed tasks to run in-database.
  • R and TeradataR are free downloads.

Source- http://developer.teradata.com/applications/articles/in-database-analytics-with-teradata-r

This package allows users of R to interact with a Teradata database. R is an open source language for statistical computing and graphics. R provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering) and graphical techniques, and is highly extensible. Users can use many statistical functions directly against the Teradata system without having to extract the data into memory.

Enhancements included with this new 1.0.1 release include:

  • teradataR User Guide
  • addition of Mac OS X Package
  • addition of Red Hat Linux Package (added 2/23/12)
  • summary has been enhanced to run faster
  • JDBC support added to allow Windows or Mac users to run the package with JDBC
  • td.data.frame enhanced to allow support for manipulation to add columns and expressions
  • td.data.frame enhanced to use Teradata 14.0 Fastpath Transform Functions (see Appendix B)
  • td.tapply function added to apply a select group of functions to columns of an array

From-http://downloads.teradata.com/download/applications/teradata-r

and

A new R package for Red Hat Linux has been added to the teradataR 1.0.1 release. This new package provides the same functionality as in the previously released Windows and Mac OS X packages, but is built for Red Hat Linux. This version was built and tested on Red Hat Linux 6.2 32-bit. (The R version for Red Hat Linux is 2.14.1)

Installing this package is the same as any normal R package; just extract it into your R library area, or use the install.packagescommand with the file path.

from- http://developer.teradata.com/tag/r

and

With plenty of prolific and enthusiastic developers, the number of packages for R is expected to grow tremendously. Statisticians and analysts using these packages will find innovative ways to use data to answer their research and business questions. And as organizations become more willing to rely on open-source software for mission-critical tasks, R is poised to become an essential tool for analyzing our complex world.

Source-http://www.teradatamagazine.com/v09n03/Connections/R-you-ready/

From the user guide-

http://downloads.teradata.com/download/applications/teradata-r

teradataR allows R users to easily connect to Teradata, establish td data frames (virtual R data frames) to
Teradata and to call in-database analytic functions within Teradata. This allows R users to work within their R
console environment while leveraging the in-database functions

A Function List
teradataR-package  Allow access to Teradata via R
as.data.frame.td.data.frame Convert td data frame to a data frame
as.td.data.frame Coerce to a td data frame
dim.td.data.frame Dimensions of a td data frame
hist.td.data.frame Histograms
Is.td.data.frame Is an Object a Teradata Data Frame
Is.td.expression Is an Object a Teradata Expression
mean.td.data.frame Arithmetic Mean
median.td.data.frame Median Value
min.td.data.frame Minima
predict.kmeans Kmeans Model Prediction
print.td.data.frame Show contents of a td data frame
sum.td.data.frame Sum of column
summary.td.data.frame Summary of Teradata Data Frame
Td.bincode Create Table of Bincode Values
Td.binomial Binomial Test
Td.binomialsign Binomial Sign Test
Td.call.sp Locate and call stored procedure
Td.cor Correlation Matrix
Td.cov Covariance Matrix
Td.dagostino.pearson D’Agostino Pearson Test
Td.data.frame Teradata Data Frames
Td.f.oneway One way F Test
Td.factanal Factor Analysis
Td.freq Frequency Analysis
Td.hist Histograms
Td.join Join Tables in Teradata
Td.kmeans K-Means Clustering
Td.ks Kolmogorov Smirnov Test
Td.lilliefors Lilliefors Test
Td.merge Merge Rows of Teradata Tables
Td.mode Mode Value of Column
Td.mwnkw Mann-Whitney/Kruskal Wallis Test
Td.nullreplace Replace Null Values
Td.overlap Overlap
Td.quantiles Quantile Values
Td.rank Rank
Td.recode Recode
Td.rescale Rescale Values of Column
Td.sample Sample Rows
Td.shapiro.wilk Shapiro Wilk
Td.sigmoid Sigmoid Transformation
Td.smirnov Smirnov Test
Td.solve Solve a system of equations
Td.stats General Statistics
Td.t.paired T Test Paired
Td.t.unpaired T Test Unpaired
Td.t.unpairedi T Test – Unpaired Indicator
Td.values Values
Td.wilcoxon Wilcoxon Test
Td.zscore Zscore Transformation
tdClose Close connection
tdConnect Connect to Teradata database
tdMetadataDB Set metadata database
tdQuery Query Teradata Database
teradataR Allow access to Teradata via R
[.td.data.frame Extract Teradata Data Frame
[<-.td.data.frame Replace value of Teradata Data Frame

How to add or change the %PATH variable in Windows 7

See this for a simple 5 step way to add or change the %PATH variable in Windows 7 if you need to install an application that shows error while installing (because that APP was built for Linux based systems… )

How to learn to be a hacker easily

1) Are you sure. It is tough to be a hacker. And football players get all the attention.

2) Really? Read on

3) Read Hacker’s Code

http://muq.org/~cynbe/hackers-code.html

The Hacker’s Code

“A hacker of the Old Code.”

  • Hackers come and go, but a great hack is forever.
  • Public goods belong to the public.*
  • Software hoarding is evil.
    Software does the greatest good given to the greatest number.
  • Don’t be evil.
  • Sourceless software sucks.
  • People have rights.
    Organizations live on sufferance.
  • Governments are organizations.
  • If it is wrong when citizens do it,
    it is wrong when governments do it.
  • Information wants to be free.
    Information deserves to be free.
  • Being legal doesn’t make it right.
  • Being illegal doesn’t make it wrong.
  • Subverting tyranny is the highest duty.
  • Trust your technolust!

4) Read How to be a hacker by

Eric Steven Raymond

http://www.catb.org/~esr/faqs/hacker-howto.html

or just get the Hacker Attitude

The Hacker Attitude

1. The world is full of fascinating problems waiting to be solved.
2. No problem should ever have to be solved twice.
3. Boredom and drudgery are evil.
4. Freedom is good.
5. Attitude is no substitute for competence.
5) If you are tired of reading English, maybe I should move on to technical stuff
6) Create your hacking space, a virtual disk on your machine.
You will need to learn a bit of Linux. If you are a Windows user, I recommend creating a VMWare partition with Ubuntu
If you like Mac, I recommend the more aesthetic Linux Mint.
How to create your virtual disk-
read here-
Download VM Player here
http://www.vmware.com/support/product-support/player/
Down iso image of operating system here
http://ubuntu.com
Downloading is the longest thing in this exercise
Now just do what is written here
http://www.vmware.com/pdf/vmware_player40.pdf
or if you want to try and experiment with other ways to use Windows and Linux just read this
http://www.decisionstats.com/ways-to-use-both-windows-and-linux-together/
Moving data back and forth between your new virtual disk and your old real disk
http://www.decisionstats.com/moving-data-between-windows-and-ubuntu-vmware-partition/
7) Get Tor to hide your IP address when on internet
https://www.torproject.org/docs/tor-doc-windows.html.en
8a ) Block Ads using Ad-block plugin when surfing the internet (like 14.95 million other users)
https://addons.mozilla.org/en-US/firefox/addon/adblock-plus/
 8b) and use Mafiafire to get elusive websites
https://addons.mozilla.org/en-US/firefox/addon/mafiaafire-redirector/
9) Get a  Bit Torrent Client at http://www.utorrent.com/
This will help you download stuff
10) Hacker Culture Alert-
This instruction is purely for sharing the culture but not the techie work of being a hacker
The website Pirate bay acts like a search engine for Bit torrents 
http://thepiratebay.se/
Visiting it is considered bad since you can get lots of music, videos, movies etc for free, without paying copyright fees.
The website 4chan is considered a meeting place to meet other hackers. The site can be visually shocking
http://boards.4chan.org/b/
You need to do atleast set up these systems, read the websites and come back in N month time for second part in this series on how to learn to be a hacker. That will be the coding part.
END OF PART  1
Updated – sorry been a bit delayed on next part. Will post soon.

Amazon gives away 750 hours /month of Windows based computing

and an additional 750 hours /month of Linux based computing. The windows instance is really quite easy for users to start getting the hang of cloud computing. and it is quite useful for people to tinker around, given Google’s retail cloud offerings are taking so long to hit the market

But it is only for new users.

http://aws.typepad.com/aws/2012/01/aws-free-usage-tier-now-includes-microsoft-windows-on-ec2.html

WS Free Usage Tier now Includes Microsoft Windows on EC2

The AWS Free Usage Tier now allows you to run Microsoft Windows Server 2008 R2 on an EC2 t1.micro instance for up to 750 hours per month. This benefit is open to new AWS customers and to those who are already participating in the Free Usage Tier, and is available in all AWS Regions with the exception of GovCloud. This is an easy way for Windows users to start learning about and enjoying the benefits of cloud computing with AWS.

The micro instances provide a small amount of consistent processing power and the ability to burst to a higher level of usage from time to time. You can use this instance to learn about Amazon EC2, support a development and test environment, build an AWS application, or host a web site (or all of the above). We’ve fine-tuned the micro instances to make them even better at running Microsoft Windows Server.

You can launch your instance from the AWS Management Console:

We have lots of helpful resources to get you started:

Along with 750 instance hours of Windows Server 2008 R2 per month, the Free Usage Tier also provides another 750 instance hours to run Linux (also on a t1.micro), Elastic Load Balancer time and bandwidth, Elastic Block Storage, Amazon S3 Storage, and SimpleDB storage, a bunch of Simple Queue Service and Simple Notification Service requests, and some CloudWatch metrics and alarms (see the AWS Free Usage Tier page for details). We’ve also boosted the amount of EBS storage space offered in the Free Usage Tier to 30GB, and we’ve doubled the I/O requests in the Free Usage Tier, to 2 million.