A DataFrame is basically a table with rows and columns.
Columns are named, rows are numbered (but can be named) and can be easily selected and calculated upon. Internally, columns are stored as 1d numpy arrays. If you set row names, they’re converted into a dictionary for fast access. There is a rich subselection/slicing API, see help(DataFrame.get_item) (it also works for setting values). Please note that any slice get’s you another DataFrame, to access individual entries use get_row(), get_column(), get_value().
DataFrames also understand basic arithmetic and you can either add (multiply,…) a constant value, or another DataFrame of the same size / with the same column names, like this:
#multiply every value in ColumnA that is smaller than 5 by 6.
my_df[my_df[:,'ColumnA'] < 5, 'ColumnA'] *= 6
#you always need to specify both row and column selectors, use : to mean everything
my_df[:, 'ColumnB'] = my_df[:,'ColumnA'] + my_df[:, 'ColumnC']
#let's take every row that starts with Shu in ColumnA and replace it with a new list (comprehension)
select = my_df.where(lambda row: row['ColumnA'].startswith('Shu'))
my_df[select, 'ColumnA'] = [row['ColumnA'].replace('Shu', 'Sha') for row in my_df[select,:].iter_rows()]
Dataframes talk directly to R via rpy2 (rpy2 is not a prerequiste for the library!)
A fast and efficient DataFrame object for data manipulation with integrated indexing;
Tools for reading and writing data between in-memory data structures and different formats: CSV and text files, Microsoft Excel, SQL databases, and the fast HDF5 format;
Intelligent data alignment and integrated handling of missing data: gain automatic label-based alignment in computations and easily manipulate messy data into an orderly form;
Flexible reshaping and pivoting of data sets;
Intelligent label-based slicing, fancy indexing, and subsetting of large data sets;
Columns can be inserted and deleted from data structures for size mutability;
Aggregating or transforming data with a powerful group by engine allowing split-apply-combine operations on data sets;
High performance merging and joining of data sets;
Hierarchical axis indexing provides an intuitive way of working with high-dimensional data in a lower-dimensional data structure;
Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data;
The library has been ruthlessly optimized for performance, with critical code paths compiled to C;
Python with pandas is in use in a wide variety of academic and commercial domains, including Finance, Neuroscience, Economics, Statistics, Advertising, Web Analytics, and more.
Why not R?
First of all, we love open source R! It is the most widely-used open source environment for statistical modeling and graphics, and it provided some early inspiration for pandas features. R users will be pleased to find this library adopts some of the best concepts of R, like the foundational DataFrame (one user familiar with R has described pandas as “R data.frame on steroids”). But pandas also seeks to solve some frustrations common to R users:
R has barebones data alignment and indexing functionality, leaving much work to the user. pandas makes it easy and intuitive to work with messy, irregularly indexed data, like time series data. pandas also provides rich tools, like hierarchical indexing, not found in R;
R is not well-suited to general purpose programming and system development. pandas enables you to do large-scale data processing seamlessly when developing your production applications;
Hybrid systems connecting R to a low-productivity systems language like Java, C++, or C# suffer from significantly reduced agility and maintainability, and you’re still stuck developing the system components in a low-productivity language;
The “copyleft” GPL license of R can create concerns for commercial software vendors who want to distribute R with their software under another license. Python and pandas use more permissive licenses.
This module allows access to comma- or other delimiter separated files as if they were tables, using a dictionary-like syntax. DataMatrix objects can be manipulated, rows and columns added and removed, or even transposed
Oracle just released the latest update to Oracle R Enterprise, version 1.1. This release includes the Oracle R Distribution (based on open source R, version 2.13.2), an improved server installation, and much more. The key new features include:
Extended Server Support: New support for Windows 32 and 64-bit server components, as well as continuing support for Linux 64-bit server components
Improved Installation: Linux 64-bit server installation now provides robust status updates and prerequisite checks
Performance Improvements: Improved performance for embedded R script execution calculations
In addition, the updated ROracle package, which is used with Oracle R Enterprise, now reads date data by conversion to character strings.
Oracle has released an update to the Oracle R Distribution, an Oracle-supported distribution of open source R. Oracle R Distribution 2-13.2 now contains the ability to dynamically link the following libraries on both Windows and Linux:
The Intel Math Kernel Library (MKL) on Intel chips
The AMD Core Math Library (ACML) on AMD chips
To take advantage of the performance enhancements provided by Intel MKL or AMD ACML in Oracle R Distribution, simply add the MKL or ACML shared library directory to the LD_LIBRARY_PATH system environment variable. This automatically enables MKL or ACML to make use of all available processors, vastly speeding up linear algebra computations and eliminating the need to recompile R. Even on a single core, the optimized algorithms in the Intel MKL libraries are faster than using R’s standard BLAS library.
Open-source R is linked to NetLib’s BLAS libraries, but they are not multi-threaded and only use one core. While R’s internal BLAS are efficient for most computations, it’s possible to recompile R to link to a different, multi-threaded BLAS library to improve performance on eligible calculations. Compiling and linking to R yourself can be involved, but for many, the significantly improved calculation speed justifies the effort.Oracle R Distribution notably simplifies the process of using external math libraries by enabling R to auto-load MKL orACML. For R commands that don’t link to BLAS code, taking advantage of database parallelism usingembedded R execution in Oracle R Enterprise is the route to improved performance.
For more information about rebuilding R with different BLAS libraries, see the linear algebra section in the R Installation and Administration manual. As always, the Oracle R Distribution is available as a free download to anyone.Questions and comments are welcome on the Oracle R Forum.
Following the successful one-day master class on Rcpp preceding this year’s R/Finance conference, a full-day master class on Rcpp and related topics which will be held on Saturday, October 8, in San Francisco.
The full-day format allows combining an introductory morning session with a more advanced afternoon session while leaving room for sufficient breaks. We plan on having about six hours of instructions, a one-hour lunch break and two half-hour coffee breaks (and lunch and refreshments will be provided).
Morning session: “A Hands-on Introduction to R and C++”
The morning session will provide a practical introduction to the Rcpp package (and other related packages). The focus will be on simple and straightforward applications of Rcpp in order to extend R and/or to significantly accelerate the execution of simple functions.
The tutorial will cover the inline package which permits embedding of self-contained C, C++ or FORTRAN code in R scripts. We will also discuss RInside, to easily embed the R engine code in C++ applications, as well as standard Rcpp extension packages such as RcppArmadillo and RcppEigen for linear algebra (via highly expressive templated C++ libraries) and RcppGSL.
Afternoon session: “Advanced R and C++ Topics”
The afternoon tutorial will provide a hands-on introduction to more advanced Rcpp features. It will cover topics such as writing packages that use Rcpp, how Rcpp modules and the new R ReferenceClasses interact, and how Rcpp sugar lets us write C++ code that is often as expressive as R code. Another possible topic, time permitting, may be writing glue code to extend Rcpp to other C++ projects.
We also expect to leave some time to discuss problems brought by the class participants.
October 8, 2011 – San Franciso
AMA Executive Conference Center
@ the Marriott Hotel
55 4th Street, 2nd Level
San Francisco, CA 94103
Tel. 415-442-6770
Instructor Bio
Dirk E has been contributing packages to CRAN for nearly a decade. Among these are RQuantLib, digest, littler, random, RPostgreSQL, as well the Rcpp family of packages comprising Rcpp, RInside, RcppClassic, RcppExamples, RcppDE, RcppArmadillo and RcppEigen. He maintains the CRAN Task Views for Finance as well as High-Performance Computing, and is a founding co-organiser of the annual R / Finance conferences in Chicago. He has Ph.D. in Financial Econometrics from EHESS (Paris), and works in Chicago as a Quantitative Strategist.
and as per http://cran.r-project.org/src/base/NEWS
the answer is plenty is new in the newR.
While you and me, were busy writing and reading blogs, or generally writing code for earning more money, or our own research- Uncle Peter D and his band of merry men have been really busy in a much more upgraded R.
————————————–
CHANGES————————-
NEW FEATURES:
• Reading a packages's CITATION file now defaults to ASCII rather
than Latin-1: a package with a non-ASCII CITATION file should
declare an encoding in its DESCRIPTION file and use that encoding
for the CITATION file.
• difftime() now defaults to the "tzone" attribute of "POSIXlt"
objects rather than to the current timezone as set by the default
for the tz argument. (Wish of PR#14182.)
• pretty() is now generic, with new methods for "Date" and "POSIXt"
classes (based on code contributed by Felix Andrews).
• unique() and match() are now faster on character vectors where
all elements are in the global CHARSXP cache and have unmarked
encoding (ASCII). Thanks to Matthew Dowle for suggesting
improvements to the way the hash code is generated in unique.c.
• The enquote() utility, in use internally, is exported now.
• .C() and .Fortran() now map non-zero return values (other than
NA_LOGICAL) for logical vectors to TRUE: it has been an implicit
assumption that they are treated as true.
• The print() methods for "glm" and "lm" objects now insert
linebreaks in long calls in the same way that the print() methods
for "summary.[g]lm" objects have long done. This does change the
layout of the examples for a number of packages, e.g. MASS.
(PR#14250)
• constrOptim() can now be used with method "SANN". (PR#14245)
It gains an argument hessian to be passed to optim(), which
allows all the ... arguments to be intended for f() and grad().
(PR#14071)
• curve() now allows expr to be an object of mode "expression" as
well as "call" and "function".
• The "POSIX[cl]t" methods for Axis() have been replaced by a
single method for "POSIXt".
There are no longer separate plot() methods for "POSIX[cl]t" and
"Date": the default method has been able to handle those classes
for a long time. This _inter alia_ allows a single date-time
object to be supplied, the wish of PR#14016.
The methods had a different default ("") for xlab.
• Classes "POSIXct", "POSIXlt" and "difftime" have generators
.POSIXct(), .POSIXlt() and .difftime(). Package authors are
advised to make use of them (they are available from R 2.11.0) to
proof against planned future changes to the classes.
The ordering of the classes has been changed, so "POSIXt" is now
the second class. See the document ‘Updating packages for
changes in R 2.12.x’ on for
the consequences for a handful of CRAN packages.
• The "POSIXct" method of as.Date() allows a timezone to be
specified (but still defaults to UTC).
• New list2env() utility function as an inverse of
as.list() and for fast multi-assign() to existing
environment. as.environment() is now generic and uses list2env()
as list method.
• There are several small changes to output which ‘zap’ small
numbers, e.g. in printing quantiles of residuals in summaries
from "lm" and "glm" fits, and in test statisics in print.anova().
• Special names such as "dim", "names", etc, are now allowed as
slot names of S4 classes, with "class" the only remaining
exception.
• File .Renviron can have architecture-specific versions such as
.Renviron.i386 on systems with sub-architectures.
• installed.packages() has a new argument subarch to filter on
sub-architecture.
• The summary() method for packageStatus() now has a separate
print() method.
• The default summary() method returns an object inheriting from
class "summaryDefault" which has a separate print() method that
calls zapsmall() for numeric/complex values.
• The startup message now includes the platform and if used,
sub-architecture: this is useful where different
(sub-)architectures run on the same OS.
• The getGraphicsEvent() mechanism now allows multiple windows to
return graphics events, through the new functions
setGraphicsEventHandlers(), setGraphicsEventEnv(), and
getGraphicsEventEnv(). (Currently implemented in the windows()
and X11() devices.)
• tools::texi2dvi() gains an index argument, mainly for use by R
CMD Rd2pdf.
It avoids the use of texindy by texinfo's texi2dvi >= 1.157,
since that does not emulate 'makeindex' well enough to avoid
problems with special characters (such as (, {, !) in indices.
• The ability of readLines() and scan() to re-encode inputs to
marked UTF-8 strings on Windows since R 2.7.0 is extended to
non-UTF-8 locales on other OSes.
• scan() gains a fileEncoding argument to match read.table().
• points() and lines() gain "table" methods to match plot(). (Wish
of PR#10472.)
• Sys.chmod() allows argument mode to be a vector, recycled along
paths.
• There are |, & and xor() methods for classes "octmode" and
"hexmode", which work bitwise.
• Environment variables R_DVIPSCMD, R_LATEXCMD, R_MAKEINDEXCMD,
R_PDFLATEXCMD are no longer used nor set in an R session. (With
the move to tools::texi2dvi(), the conventional environment
variables LATEX, MAKEINDEX and PDFLATEX will be used.
options("dvipscmd") defaults to the value of DVIPS, then to
"dvips".)
• New function isatty() to see if terminal connections are
redirected.
• summaryRprof() returns the sampling interval in component
sample.interval and only returns in by.self data for functions
with non-zero self times.
• print(x) and str(x) now indicate if an empty list x is named.
• install.packages() and remove.packages() with lib unspecified and
multiple libraries in .libPaths() inform the user of the library
location used with a message rather than a warning.
• There is limited support for multiple compressed streams on a
file: all of [bgx]zfile() allow streams to be appended to an
existing file, but bzfile() reads only the first stream.
• Function person() in package utils now uses a given/family scheme
in preference to first/middle/last, is vectorized to handle an
arbitrary number of persons, and gains a role argument to specify
person roles using a controlled vocabulary (the MARC relator
terms).
• Package utils adds a new "bibentry" class for representing and
manipulating bibliographic information in enhanced BibTeX style,
unifying and enhancing the previously existing mechanisms.
• A bibstyle() function has been added to the tools package with
default JSS style for rendering "bibentry" objects, and a
mechanism for registering other rendering styles.
• Several aspects of the display of text help are now customizable
using the new Rd2txt_options() function.
options("help_text_width") is no longer used.
• Added \href tag to the Rd format, to allow hyperlinks to URLs
without displaying the full URL.
• Added \newcommand and \renewcommand tags to the Rd format, to
allow user-defined macros.
• New toRd() generic in the tools package to convert objects to
fragments of Rd code, and added "fragment" argument to Rd2txt(),
Rd2HTML(), and Rd2latex() to support it.
• Directory R_HOME/share/texmf now follows the TDS conventions, so
can be set as a texmf tree (‘root directory’ in MiKTeX parlance).
• S3 generic functions now use correct S4 inheritance when
dispatching on an S4 object. See ?Methods, section on “Methods
for S3 Generic Functions†for recommendations and details.
• format.pval() gains a ... argument to pass arguments such as
nsmall to format(). (Wish of PR#9574)
• legend() supports title.adj. (Wish of PR#13415)
• Added support for subsetting "raster" objects, plus assigning to
a subset, conversion to a matrix (of colour strings), and
comparisons (== and !=).
• Added a new parseLatex() function (and related functions
deparseLatex() and latexToUtf8()) to support conversion of
bibliographic entries for display in R.
• Text rendering of \itemize in help uses a Unicode bullet in UTF-8
and most single-byte Windows locales.
• Added support for polygons with holes to the graphics engine.
This is implemented for the pdf(), postscript(),
x11(type="cairo"), windows(), and quartz() devices (and
associated raster formats), but not for x11(type="Xlib") or
xfig() or pictex(). The user-level interface is the polypath()
function in graphics and grid.path() in grid.
• File NEWS is now generated at installation with a slightly
different format: it will be in UTF-8 on platforms using UTF-8,
and otherwise in ASCII. There is also a PDF version, NEWS.pdf,
installed at the top-level of the R distribution.
• kmeans(x, 1) now works. Further, kmeans now returns between and
total sum of squares.
• arrayInd() and which() gain an argument useNames. For arrayInd,
the default is now false, for speed reasons.
• As is done for closures, the default print method for the formula
class now displays the associated environment if it is not the
global environment.
• A new facility has been added for inserting code into a package
without re-installing it, to facilitate testing changes which can
be selectively added and backed out. See ?insertSource.
• New function readRenviron to (re-)read files in the format of
~/.Renviron and Renviron.site.
• require() will now return FALSE (and not fail) if loading the
package or one of its dependencies fails.
• aperm() now allows argument perm to be a character vector when
the array has named dimnames (as the results of table() calls
do). Similarly, array() allows MARGIN to be a character vector.
(Based on suggestions of Michael Lachmann.)
• Package utils now exports and documents functions
aspell_package_Rd_files() and aspell_package_vignettes() for
spell checking package Rd files and vignettes using Aspell,
Ispell or Hunspell.
• Package news can now be given in Rd format, and news() prefers
these inst/NEWS.Rd files to old-style plain text NEWS or
inst/NEWS files.
• New simple function packageVersion().
• The PCRE library has been updated to version 8.10.
• The standard Unix-alike terminal interface declares its name to
readline as 'R', so that can be used for conditional sections in
~/.inputrc files.
• ‘Writing R Extensions’ now stresses that the standard sections in
.Rd files (other than \alias, \keyword and \note) are intended to
be unique, and the conversion tools now drop duplicates with a
warning.
The .Rd conversion tools also warn about an unrecognized type in
a \docType section.
• ecdf() objects now have a quantile() method.
• format() methods for date-time objects now attempt to make use of
a "tzone" attribute with "%Z" and "%z" formats, but it is not
always possible. (Wish of PR#14358.)
• tools::texi2dvi(file, clean = TRUE) now works in more cases (e.g.
where emulation is used and when file is not in the current
directory).
• New function droplevels() to remove unused factor levels.
• system(command, intern = TRUE) now gives an error on a Unix-alike
(as well as on Windows) if command cannot be run. It reports a
non-success exit status from running command as a warning.
On a Unix-alike an attempt is made to return the actual exit
status of the command in system(intern = FALSE): previously this
had been system-dependent but on POSIX-compliant systems the
value return was 256 times the status.
• system() has a new argument ignore.stdout which can be used to
(portably) ignore standard output.
• system(intern = TRUE) and pipe() connections are guaranteed to be
avaliable on all builds of R.
• Sys.which() has been altered to return "" if the command is not
found (even on Solaris).
• A facility for defining reference-based S4 classes (in the OOP
style of Java, C++, etc.) has been added experimentally to
package methods; see ?ReferenceClasses.
• The predict method for "loess" fits gains an na.action argument
which defaults to na.pass rather than the previous default of
na.omit.
Predictions from "loess" fits are now named from the row names of
newdata.
• Parsing errors detected during Sweave() processing will now be
reported referencing their original location in the source file.
• New adjustcolor() utility, e.g., for simple translucent color
schemes.
• qr() now has a trivial lm method with a simple (fast) validity
check.
• An experimental new programming model has been added to package
methods for reference (OOP-style) classes and methods. See
?ReferenceClasses.
• bzip2 has been updated to version 1.0.6 (bug-fix release).
--with-system-bzlib now requires at least version 1.0.6.
• R now provides jss.cls and jss.bst (the class and bib style file
for the Journal of Statistical Software) as well as RJournal.bib
and Rnews.bib, and R CMD ensures that the .bst and .bib files are
found by BibTeX.
• Functions using the TAR environment variable no longer quote the
value when making system calls. This allows values such as tar
--force-local, but does require additional quotes in, e.g., TAR =
"'/path with spaces/mytar'".
DEPRECATED & DEFUNCT:
• Supplying the parser with a character string containing both
octal/hex and Unicode escapes is now an error.
• File extension .C for C++ code files in packages is now defunct.
• R CMD check no longer supports configuration files containing
Perl configuration variables: use the environment variables
documented in ‘R Internals’ instead.
• The save argument of require() now defaults to FALSE and save =
TRUE is now deprecated. (This facility is very rarely actually
used, and was superseded by the Depends field of the DESCRIPTION
file long ago.)
• R CMD check --no-latex is deprecated in favour of --no-manual.
• R CMD Sd2Rd is formally deprecated and will be removed in R
2.13.0.
PACKAGE INSTALLATION:
• install.packages() has a new argument libs_only to optionally
pass --libs-only to R CMD INSTALL and works analogously for
Windows binary installs (to add support for 64- or 32-bit
Windows).
• When sub-architectures are in use, the installed architectures
are recorded in the Archs field of the DESCRIPTION file. There
is a new default filter, "subarch", in available.packages() to
make use of this.
Code is compiled in a copy of the src directory when a package is
installed for more than one sub-architecture: this avoid problems
with cleaning the sources between building sub-architectures.
• R CMD INSTALL --libs-only no longer overrides the setting of
locking, so a previous version of the package will be restored
unless --no-lock is specified.
UTILITIES:
• R CMD Rprof|build|check are now based on R rather than Perl
scripts. The only remaining Perl scripts are the deprecated R
CMD Sd2Rd and install-info.pl (used only if install-info is not
found) as well as some maintainer-mode-only scripts.
*NB:* because these have been completely rewritten, users should
not expect undocumented details of previous implementations to
have been duplicated.
R CMD no longer manipulates the environment variables PERL5LIB
and PERLLIB.
• R CMD check has a new argument --extra-arch to confine tests to
those needed to check an additional sub-architecture.
Its check for “Subdirectory 'inst' contains no files†is more
thorough: it looks for files, and warns if there are only empty
directories.
Environment variables such as R_LIBS and those used for
customization can be set for the duration of checking _via_ a
file ~/.R/check.Renviron (in the format used by .Renviron, and
with sub-architecture specific versions such as
~/.R/check.Renviron.i386 taking precedence).
There are new options --multiarch to check the package under all
of the installed sub-architectures and --no-multiarch to confine
checking to the sub-architecture under which check is invoked.
If neither option is supplied, a test is done of installed
sub-architectures and all those which can be run on the current
OS are used.
Unless multiple sub-architectures are selected, the install done
by check for testing purposes is only of the current
sub-architecture (_via_ R CMD INSTALL --no-multiarch).
It will skip the check for non-ascii characters in code or data
if the environment variables _R_CHECK_ASCII_CODE_ or
_R_CHECK_ASCII_DATA_ are respectively set to FALSE. (Suggestion
of Vince Carey.)
• R CMD build no longer creates an INDEX file (R CMD INSTALL does
so), and --force removes (rather than overwrites) an existing
INDEX file.
It supports a file ~/.R/build.Renviron analogously to check.
It now runs build-time \Sexpr expressions in help files.
• R CMD Rd2dvi makes use of tools::texi2dvi() to process the
package manual. It is now implemented entirely in R (rather than
partially as a shell script).
• R CMD Rprof now uses utils::summaryRprof() rather than Perl. It
has new arguments to select one of the tables and to limit the
number of entries printed.
• R CMD Sweave now runs R with --vanilla so the environment setting
of R_LIBS will always be used.
C-LEVEL FACILITIES:
• lang5() and lang6() (in addition to pre-existing lang[1-4]())
convenience functions for easier construction of eval() calls.
If you have your own definition, do wrap it inside #ifndef lang5
.... #endif to keep it working with old and new R.
• Header R.h now includes only the C headers it itself needs, hence
no longer includes errno.h. (This helps avoid problems when it
is included from C++ source files.)
• Headers Rinternals.h and R_ext/Print.h include the C++ versions
of stdio.h and stdarg.h respectively if included from a C++
source file.
INSTALLATION:
• A C99 compiler is now required, and more C99 language features
will be used in the R sources.
• Tcl/Tk >= 8.4 is now required (increased from 8.3).
• System functions access, chdir and getcwd are now essential to
configure R. (In practice they have been required for some
time.)
• make check compares the output of the examples from several of
the base packages to reference output rather than the previous
output (if any). Expect some differences due to differences in
floating-point computations between platforms.
• File NEWS is no longer in the sources, but generated as part of
the installation. The primary source for changes is now
doc/NEWS.Rd.
• The popen system call is now required to build R. This ensures
the availability of system(intern = TRUE), pipe() connections and
printing from postscript().
• The pkg-config file libR.pc now also works when R is installed
using a sub-architecture.
• R has always required a BLAS that conforms to IE60559 arithmetic,
but after discovery of more real-world problems caused by a BLAS
that did not, this is tested more thoroughly in this version.
BUG FIXES:
• Calls to selectMethod() by default no longer cache inherited
methods. This could previously corrupt methods used by as().
• The densities of non-central chi-squared are now more accurate in
some cases in the extreme tails, e.g. dchisq(2000, 2, 1000), as a
series expansion was truncated too early. (PR#14105)
• pt() is more accurate in the left tail for ncp large, e.g.
pt(-1000, 3, 200). (PR#14069)
• The default C function (R_binary) for binary ops now sets the S4
bit in the result if either argument is an S4 object. (PR#13209)
• source(echo=TRUE) failed to echo comments that followed the last
statement in a file.
• S4 classes that contained one of "matrix", "array" or "ts" and
also another class now accept superclass objects in new(). Also
fixes failure to call validObject() for these classes.
• Conditional inheritance defined by argument test in
methods::setIs() will no longer be used in S4 method selection
(caching these methods could give incorrect results). See
?setIs.
• The signature of an implicit generic is now used by setGeneric()
when that does not use a definition nor explicitly set a
signature.
• A bug in callNextMethod() for some examples with "..." in the
arguments has been fixed. See file
src/library/methods/tests/nextWithDots.R in the sources.
• match(x, table) (and hence %in%) now treat "POSIXlt" consistently
with, e.g., "POSIXct".
• Built-in code dealing with environments (get(), assign(),
parent.env(), is.environment() and others) now behave
consistently to recognize S4 subclasses; is.name() also
recognizes subclasses.
• The abs.tol control parameter to nlminb() now defaults to 0.0 to
avoid false declarations of convergence in objective functions
that may go negative.
• The standard Unix-alike termination dialog to ask whether to save
the workspace takes a EOF response as n to avoid problems with a
damaged terminal connection. (PR#14332)
• Added warn.unused argument to hist.default() to allow suppression
of spurious warnings about graphical parameters used with
plot=FALSE. (PR#14341)
• predict.lm(), summary.lm(), and indeed lm() itself had issues
with residual DF in zero-weighted cases (the latter two only in
connection with empty models). (Thanks to Bill Dunlap for
spotting the predict() case.)
• aperm() treated resize = NA as resize = TRUE.
• constrOptim() now has an improved convergence criterion, notably
for cases where the minimum was (very close to) zero; further,
other tweaks inspired from code proposals by Ravi Varadhan.
• Rendering of S3 and S4 methods in man pages has been corrected
and made consistent across output formats.
• Simple markup is now allowed in \title sections in .Rd files.
• The behaviour of as.logical() on factors (to use the levels) was
lost in R 2.6.0 and has been restored.
• prompt() did not backquote some default arguments in the \usage
section. (Reported by Claudia Beleites.)
• writeBin() disallows attempts to write 2GB or more in a single
call. (PR#14362)
• new() and getClass() will now work if Class is a subclass of
"classRepresentation" and should also be faster in typical calls.
• The summary() method for data frames makes a better job of names
containing characters invalid in the current locale.
• [[ sub-assignment for factors could create an invalid factor
(reported by Bill Dunlap).
• Negate(f) would not evaluate argument f until first use of
returned function (reported by Olaf Mersmann).
• quietly=FALSE is now also an optional argument of library(), and
consequently, quietly is now propagated also for loading
dependent packages, e.g., in require(*, quietly=TRUE).
• If the loop variable in a for loop was deleted, it would be
recreated as a global variable. (Reported by Radford Neal; the
fix includes his optimizations as well.)
• Task callbacks could report the wrong expression when the task
involved parsing new code. (PR#14368)
• getNamespaceVersion() failed; this was an accidental change in
2.11.0. (PR#14374)
• identical() returned FALSE for external pointer objects even when
the pointer addresses were the same.
• L$a@x[] <- val did not duplicate in a case it should have.
• tempfile() now always gives a random file name (even if the
directory is specified) when called directly after startup and
before the R RNG had been used. (PR#14381)
• quantile(type=6) behaved inconsistently. (PR#14383)
• backSpline(.) behaved incorrectly when the knot sequence was
decreasing. (PR#14386)
• The reference BLAS included in R was assuming that 0*x and x*0
were always zero (whereas they could be NA or NaN in IEC 60559
arithmetic). This was seen in results from tcrossprod, and for
example that log(0) %*% 0 gave 0.
• The calculation of whether text was completely outside the device
region (in which case, you draw nothing) was wrong for screen
devices (which have [0, 0] at top-left). The symptom was (long)
text disappearing when resizing a screen window (to make it
smaller). (PR#14391)
• model.frame(drop.unused.levels = TRUE) did not take into account
NA values of factors when deciding to drop levels. (PR#14393)
• library.dynam.unload required an absolute path for libpath.
(PR#14385)
Both library() and loadNamespace() now record absolute paths for
use by searchpaths() and getNamespaceInfo(ns, "path").
• The self-starting model NLSstClosestX failed if some deviation
was exactly zero. (PR#14384)
• X11(type = "cairo") (and other devices such as png using
cairographics) and which use Pango font selection now work around
a bug in Pango when very small fonts (those with sizes between 0
and 1 in Pango's internal units) are requested. (PR#14369)
• Added workaround for the font problem with X11(type = "cairo")
and similar on Mac OS X whereby italic and bold styles were
interchanged. (PR#13463 amongst many other reports.)
• source(chdir = TRUE) failed to reset the working directory if it
could not be determined - that is now an error.
• Fix for crash of example(rasterImage) on x11(type="Xlib").
• Force Quartz to bring the on-screen display up-to-date
immediately before the snapshot is taken by grid.cap() in the
Cocoa implementation. (PR#14260)
• model.frame had an unstated 500 byte limit on variable names.
(Example reported by Terry Therneau.)
• The 256-byte limit on names is now documented. • Subassignment by [, [[ or $ on an expression object with value
NULL coerced the object to a list.
and as per http://cran.r-project.org/src/base/NEWS
the answer is plenty is new in the newR.
While you and me, were busy writing and reading blogs, or generally writing code for earning more money, or our own research- Uncle Peter D and his band of merry men have been really busy in a much more upgraded R.
————————————–
CHANGES————————-
NEW FEATURES:
• Reading a packages's CITATION file now defaults to ASCII rather
than Latin-1: a package with a non-ASCII CITATION file should
declare an encoding in its DESCRIPTION file and use that encoding
for the CITATION file.
• difftime() now defaults to the "tzone" attribute of "POSIXlt"
objects rather than to the current timezone as set by the default
for the tz argument. (Wish of PR#14182.)
• pretty() is now generic, with new methods for "Date" and "POSIXt"
classes (based on code contributed by Felix Andrews).
• unique() and match() are now faster on character vectors where
all elements are in the global CHARSXP cache and have unmarked
encoding (ASCII). Thanks to Matthew Dowle for suggesting
improvements to the way the hash code is generated in unique.c.
• The enquote() utility, in use internally, is exported now.
• .C() and .Fortran() now map non-zero return values (other than
NA_LOGICAL) for logical vectors to TRUE: it has been an implicit
assumption that they are treated as true.
• The print() methods for "glm" and "lm" objects now insert
linebreaks in long calls in the same way that the print() methods
for "summary.[g]lm" objects have long done. This does change the
layout of the examples for a number of packages, e.g. MASS.
(PR#14250)
• constrOptim() can now be used with method "SANN". (PR#14245)
It gains an argument hessian to be passed to optim(), which
allows all the ... arguments to be intended for f() and grad().
(PR#14071)
• curve() now allows expr to be an object of mode "expression" as
well as "call" and "function".
• The "POSIX[cl]t" methods for Axis() have been replaced by a
single method for "POSIXt".
There are no longer separate plot() methods for "POSIX[cl]t" and
"Date": the default method has been able to handle those classes
for a long time. This _inter alia_ allows a single date-time
object to be supplied, the wish of PR#14016.
The methods had a different default ("") for xlab.
• Classes "POSIXct", "POSIXlt" and "difftime" have generators
.POSIXct(), .POSIXlt() and .difftime(). Package authors are
advised to make use of them (they are available from R 2.11.0) to
proof against planned future changes to the classes.
The ordering of the classes has been changed, so "POSIXt" is now
the second class. See the document ‘Updating packages for
changes in R 2.12.x’ on for
the consequences for a handful of CRAN packages.
• The "POSIXct" method of as.Date() allows a timezone to be
specified (but still defaults to UTC).
• New list2env() utility function as an inverse of
as.list() and for fast multi-assign() to existing
environment. as.environment() is now generic and uses list2env()
as list method.
• There are several small changes to output which ‘zap’ small
numbers, e.g. in printing quantiles of residuals in summaries
from "lm" and "glm" fits, and in test statisics in print.anova().
• Special names such as "dim", "names", etc, are now allowed as
slot names of S4 classes, with "class" the only remaining
exception.
• File .Renviron can have architecture-specific versions such as
.Renviron.i386 on systems with sub-architectures.
• installed.packages() has a new argument subarch to filter on
sub-architecture.
• The summary() method for packageStatus() now has a separate
print() method.
• The default summary() method returns an object inheriting from
class "summaryDefault" which has a separate print() method that
calls zapsmall() for numeric/complex values.
• The startup message now includes the platform and if used,
sub-architecture: this is useful where different
(sub-)architectures run on the same OS.
• The getGraphicsEvent() mechanism now allows multiple windows to
return graphics events, through the new functions
setGraphicsEventHandlers(), setGraphicsEventEnv(), and
getGraphicsEventEnv(). (Currently implemented in the windows()
and X11() devices.)
• tools::texi2dvi() gains an index argument, mainly for use by R
CMD Rd2pdf.
It avoids the use of texindy by texinfo's texi2dvi >= 1.157,
since that does not emulate 'makeindex' well enough to avoid
problems with special characters (such as (, {, !) in indices.
• The ability of readLines() and scan() to re-encode inputs to
marked UTF-8 strings on Windows since R 2.7.0 is extended to
non-UTF-8 locales on other OSes.
• scan() gains a fileEncoding argument to match read.table().
• points() and lines() gain "table" methods to match plot(). (Wish
of PR#10472.)
• Sys.chmod() allows argument mode to be a vector, recycled along
paths.
• There are |, & and xor() methods for classes "octmode" and
"hexmode", which work bitwise.
• Environment variables R_DVIPSCMD, R_LATEXCMD, R_MAKEINDEXCMD,
R_PDFLATEXCMD are no longer used nor set in an R session. (With
the move to tools::texi2dvi(), the conventional environment
variables LATEX, MAKEINDEX and PDFLATEX will be used.
options("dvipscmd") defaults to the value of DVIPS, then to
"dvips".)
• New function isatty() to see if terminal connections are
redirected.
• summaryRprof() returns the sampling interval in component
sample.interval and only returns in by.self data for functions
with non-zero self times.
• print(x) and str(x) now indicate if an empty list x is named.
• install.packages() and remove.packages() with lib unspecified and
multiple libraries in .libPaths() inform the user of the library
location used with a message rather than a warning.
• There is limited support for multiple compressed streams on a
file: all of [bgx]zfile() allow streams to be appended to an
existing file, but bzfile() reads only the first stream.
• Function person() in package utils now uses a given/family scheme
in preference to first/middle/last, is vectorized to handle an
arbitrary number of persons, and gains a role argument to specify
person roles using a controlled vocabulary (the MARC relator
terms).
• Package utils adds a new "bibentry" class for representing and
manipulating bibliographic information in enhanced BibTeX style,
unifying and enhancing the previously existing mechanisms.
• A bibstyle() function has been added to the tools package with
default JSS style for rendering "bibentry" objects, and a
mechanism for registering other rendering styles.
• Several aspects of the display of text help are now customizable
using the new Rd2txt_options() function.
options("help_text_width") is no longer used.
• Added \href tag to the Rd format, to allow hyperlinks to URLs
without displaying the full URL.
• Added \newcommand and \renewcommand tags to the Rd format, to
allow user-defined macros.
• New toRd() generic in the tools package to convert objects to
fragments of Rd code, and added "fragment" argument to Rd2txt(),
Rd2HTML(), and Rd2latex() to support it.
• Directory R_HOME/share/texmf now follows the TDS conventions, so
can be set as a texmf tree (‘root directory’ in MiKTeX parlance).
• S3 generic functions now use correct S4 inheritance when
dispatching on an S4 object. See ?Methods, section on “Methods
for S3 Generic Functions†for recommendations and details.
• format.pval() gains a ... argument to pass arguments such as
nsmall to format(). (Wish of PR#9574)
• legend() supports title.adj. (Wish of PR#13415)
• Added support for subsetting "raster" objects, plus assigning to
a subset, conversion to a matrix (of colour strings), and
comparisons (== and !=).
• Added a new parseLatex() function (and related functions
deparseLatex() and latexToUtf8()) to support conversion of
bibliographic entries for display in R.
• Text rendering of \itemize in help uses a Unicode bullet in UTF-8
and most single-byte Windows locales.
• Added support for polygons with holes to the graphics engine.
This is implemented for the pdf(), postscript(),
x11(type="cairo"), windows(), and quartz() devices (and
associated raster formats), but not for x11(type="Xlib") or
xfig() or pictex(). The user-level interface is the polypath()
function in graphics and grid.path() in grid.
• File NEWS is now generated at installation with a slightly
different format: it will be in UTF-8 on platforms using UTF-8,
and otherwise in ASCII. There is also a PDF version, NEWS.pdf,
installed at the top-level of the R distribution.
• kmeans(x, 1) now works. Further, kmeans now returns between and
total sum of squares.
• arrayInd() and which() gain an argument useNames. For arrayInd,
the default is now false, for speed reasons.
• As is done for closures, the default print method for the formula
class now displays the associated environment if it is not the
global environment.
• A new facility has been added for inserting code into a package
without re-installing it, to facilitate testing changes which can
be selectively added and backed out. See ?insertSource.
• New function readRenviron to (re-)read files in the format of
~/.Renviron and Renviron.site.
• require() will now return FALSE (and not fail) if loading the
package or one of its dependencies fails.
• aperm() now allows argument perm to be a character vector when
the array has named dimnames (as the results of table() calls
do). Similarly, array() allows MARGIN to be a character vector.
(Based on suggestions of Michael Lachmann.)
• Package utils now exports and documents functions
aspell_package_Rd_files() and aspell_package_vignettes() for
spell checking package Rd files and vignettes using Aspell,
Ispell or Hunspell.
• Package news can now be given in Rd format, and news() prefers
these inst/NEWS.Rd files to old-style plain text NEWS or
inst/NEWS files.
• New simple function packageVersion().
• The PCRE library has been updated to version 8.10.
• The standard Unix-alike terminal interface declares its name to
readline as 'R', so that can be used for conditional sections in
~/.inputrc files.
• ‘Writing R Extensions’ now stresses that the standard sections in
.Rd files (other than \alias, \keyword and \note) are intended to
be unique, and the conversion tools now drop duplicates with a
warning.
The .Rd conversion tools also warn about an unrecognized type in
a \docType section.
• ecdf() objects now have a quantile() method.
• format() methods for date-time objects now attempt to make use of
a "tzone" attribute with "%Z" and "%z" formats, but it is not
always possible. (Wish of PR#14358.)
• tools::texi2dvi(file, clean = TRUE) now works in more cases (e.g.
where emulation is used and when file is not in the current
directory).
• New function droplevels() to remove unused factor levels.
• system(command, intern = TRUE) now gives an error on a Unix-alike
(as well as on Windows) if command cannot be run. It reports a
non-success exit status from running command as a warning.
On a Unix-alike an attempt is made to return the actual exit
status of the command in system(intern = FALSE): previously this
had been system-dependent but on POSIX-compliant systems the
value return was 256 times the status.
• system() has a new argument ignore.stdout which can be used to
(portably) ignore standard output.
• system(intern = TRUE) and pipe() connections are guaranteed to be
avaliable on all builds of R.
• Sys.which() has been altered to return "" if the command is not
found (even on Solaris).
• A facility for defining reference-based S4 classes (in the OOP
style of Java, C++, etc.) has been added experimentally to
package methods; see ?ReferenceClasses.
• The predict method for "loess" fits gains an na.action argument
which defaults to na.pass rather than the previous default of
na.omit.
Predictions from "loess" fits are now named from the row names of
newdata.
• Parsing errors detected during Sweave() processing will now be
reported referencing their original location in the source file.
• New adjustcolor() utility, e.g., for simple translucent color
schemes.
• qr() now has a trivial lm method with a simple (fast) validity
check.
• An experimental new programming model has been added to package
methods for reference (OOP-style) classes and methods. See
?ReferenceClasses.
• bzip2 has been updated to version 1.0.6 (bug-fix release).
--with-system-bzlib now requires at least version 1.0.6.
• R now provides jss.cls and jss.bst (the class and bib style file
for the Journal of Statistical Software) as well as RJournal.bib
and Rnews.bib, and R CMD ensures that the .bst and .bib files are
found by BibTeX.
• Functions using the TAR environment variable no longer quote the
value when making system calls. This allows values such as tar
--force-local, but does require additional quotes in, e.g., TAR =
"'/path with spaces/mytar'".
DEPRECATED & DEFUNCT:
• Supplying the parser with a character string containing both
octal/hex and Unicode escapes is now an error.
• File extension .C for C++ code files in packages is now defunct.
• R CMD check no longer supports configuration files containing
Perl configuration variables: use the environment variables
documented in ‘R Internals’ instead.
• The save argument of require() now defaults to FALSE and save =
TRUE is now deprecated. (This facility is very rarely actually
used, and was superseded by the Depends field of the DESCRIPTION
file long ago.)
• R CMD check --no-latex is deprecated in favour of --no-manual.
• R CMD Sd2Rd is formally deprecated and will be removed in R
2.13.0.
PACKAGE INSTALLATION:
• install.packages() has a new argument libs_only to optionally
pass --libs-only to R CMD INSTALL and works analogously for
Windows binary installs (to add support for 64- or 32-bit
Windows).
• When sub-architectures are in use, the installed architectures
are recorded in the Archs field of the DESCRIPTION file. There
is a new default filter, "subarch", in available.packages() to
make use of this.
Code is compiled in a copy of the src directory when a package is
installed for more than one sub-architecture: this avoid problems
with cleaning the sources between building sub-architectures.
• R CMD INSTALL --libs-only no longer overrides the setting of
locking, so a previous version of the package will be restored
unless --no-lock is specified.
UTILITIES:
• R CMD Rprof|build|check are now based on R rather than Perl
scripts. The only remaining Perl scripts are the deprecated R
CMD Sd2Rd and install-info.pl (used only if install-info is not
found) as well as some maintainer-mode-only scripts.
*NB:* because these have been completely rewritten, users should
not expect undocumented details of previous implementations to
have been duplicated.
R CMD no longer manipulates the environment variables PERL5LIB
and PERLLIB.
• R CMD check has a new argument --extra-arch to confine tests to
those needed to check an additional sub-architecture.
Its check for “Subdirectory 'inst' contains no files†is more
thorough: it looks for files, and warns if there are only empty
directories.
Environment variables such as R_LIBS and those used for
customization can be set for the duration of checking _via_ a
file ~/.R/check.Renviron (in the format used by .Renviron, and
with sub-architecture specific versions such as
~/.R/check.Renviron.i386 taking precedence).
There are new options --multiarch to check the package under all
of the installed sub-architectures and --no-multiarch to confine
checking to the sub-architecture under which check is invoked.
If neither option is supplied, a test is done of installed
sub-architectures and all those which can be run on the current
OS are used.
Unless multiple sub-architectures are selected, the install done
by check for testing purposes is only of the current
sub-architecture (_via_ R CMD INSTALL --no-multiarch).
It will skip the check for non-ascii characters in code or data
if the environment variables _R_CHECK_ASCII_CODE_ or
_R_CHECK_ASCII_DATA_ are respectively set to FALSE. (Suggestion
of Vince Carey.)
• R CMD build no longer creates an INDEX file (R CMD INSTALL does
so), and --force removes (rather than overwrites) an existing
INDEX file.
It supports a file ~/.R/build.Renviron analogously to check.
It now runs build-time \Sexpr expressions in help files.
• R CMD Rd2dvi makes use of tools::texi2dvi() to process the
package manual. It is now implemented entirely in R (rather than
partially as a shell script).
• R CMD Rprof now uses utils::summaryRprof() rather than Perl. It
has new arguments to select one of the tables and to limit the
number of entries printed.
• R CMD Sweave now runs R with --vanilla so the environment setting
of R_LIBS will always be used.
C-LEVEL FACILITIES:
• lang5() and lang6() (in addition to pre-existing lang[1-4]())
convenience functions for easier construction of eval() calls.
If you have your own definition, do wrap it inside #ifndef lang5
.... #endif to keep it working with old and new R.
• Header R.h now includes only the C headers it itself needs, hence
no longer includes errno.h. (This helps avoid problems when it
is included from C++ source files.)
• Headers Rinternals.h and R_ext/Print.h include the C++ versions
of stdio.h and stdarg.h respectively if included from a C++
source file.
INSTALLATION:
• A C99 compiler is now required, and more C99 language features
will be used in the R sources.
• Tcl/Tk >= 8.4 is now required (increased from 8.3).
• System functions access, chdir and getcwd are now essential to
configure R. (In practice they have been required for some
time.)
• make check compares the output of the examples from several of
the base packages to reference output rather than the previous
output (if any). Expect some differences due to differences in
floating-point computations between platforms.
• File NEWS is no longer in the sources, but generated as part of
the installation. The primary source for changes is now
doc/NEWS.Rd.
• The popen system call is now required to build R. This ensures
the availability of system(intern = TRUE), pipe() connections and
printing from postscript().
• The pkg-config file libR.pc now also works when R is installed
using a sub-architecture.
• R has always required a BLAS that conforms to IE60559 arithmetic,
but after discovery of more real-world problems caused by a BLAS
that did not, this is tested more thoroughly in this version.
BUG FIXES:
• Calls to selectMethod() by default no longer cache inherited
methods. This could previously corrupt methods used by as().
• The densities of non-central chi-squared are now more accurate in
some cases in the extreme tails, e.g. dchisq(2000, 2, 1000), as a
series expansion was truncated too early. (PR#14105)
• pt() is more accurate in the left tail for ncp large, e.g.
pt(-1000, 3, 200). (PR#14069)
• The default C function (R_binary) for binary ops now sets the S4
bit in the result if either argument is an S4 object. (PR#13209)
• source(echo=TRUE) failed to echo comments that followed the last
statement in a file.
• S4 classes that contained one of "matrix", "array" or "ts" and
also another class now accept superclass objects in new(). Also
fixes failure to call validObject() for these classes.
• Conditional inheritance defined by argument test in
methods::setIs() will no longer be used in S4 method selection
(caching these methods could give incorrect results). See
?setIs.
• The signature of an implicit generic is now used by setGeneric()
when that does not use a definition nor explicitly set a
signature.
• A bug in callNextMethod() for some examples with "..." in the
arguments has been fixed. See file
src/library/methods/tests/nextWithDots.R in the sources.
• match(x, table) (and hence %in%) now treat "POSIXlt" consistently
with, e.g., "POSIXct".
• Built-in code dealing with environments (get(), assign(),
parent.env(), is.environment() and others) now behave
consistently to recognize S4 subclasses; is.name() also
recognizes subclasses.
• The abs.tol control parameter to nlminb() now defaults to 0.0 to
avoid false declarations of convergence in objective functions
that may go negative.
• The standard Unix-alike termination dialog to ask whether to save
the workspace takes a EOF response as n to avoid problems with a
damaged terminal connection. (PR#14332)
• Added warn.unused argument to hist.default() to allow suppression
of spurious warnings about graphical parameters used with
plot=FALSE. (PR#14341)
• predict.lm(), summary.lm(), and indeed lm() itself had issues
with residual DF in zero-weighted cases (the latter two only in
connection with empty models). (Thanks to Bill Dunlap for
spotting the predict() case.)
• aperm() treated resize = NA as resize = TRUE.
• constrOptim() now has an improved convergence criterion, notably
for cases where the minimum was (very close to) zero; further,
other tweaks inspired from code proposals by Ravi Varadhan.
• Rendering of S3 and S4 methods in man pages has been corrected
and made consistent across output formats.
• Simple markup is now allowed in \title sections in .Rd files.
• The behaviour of as.logical() on factors (to use the levels) was
lost in R 2.6.0 and has been restored.
• prompt() did not backquote some default arguments in the \usage
section. (Reported by Claudia Beleites.)
• writeBin() disallows attempts to write 2GB or more in a single
call. (PR#14362)
• new() and getClass() will now work if Class is a subclass of
"classRepresentation" and should also be faster in typical calls.
• The summary() method for data frames makes a better job of names
containing characters invalid in the current locale.
• [[ sub-assignment for factors could create an invalid factor
(reported by Bill Dunlap).
• Negate(f) would not evaluate argument f until first use of
returned function (reported by Olaf Mersmann).
• quietly=FALSE is now also an optional argument of library(), and
consequently, quietly is now propagated also for loading
dependent packages, e.g., in require(*, quietly=TRUE).
• If the loop variable in a for loop was deleted, it would be
recreated as a global variable. (Reported by Radford Neal; the
fix includes his optimizations as well.)
• Task callbacks could report the wrong expression when the task
involved parsing new code. (PR#14368)
• getNamespaceVersion() failed; this was an accidental change in
2.11.0. (PR#14374)
• identical() returned FALSE for external pointer objects even when
the pointer addresses were the same.
• L$a@x[] <- val did not duplicate in a case it should have.
• tempfile() now always gives a random file name (even if the
directory is specified) when called directly after startup and
before the R RNG had been used. (PR#14381)
• quantile(type=6) behaved inconsistently. (PR#14383)
• backSpline(.) behaved incorrectly when the knot sequence was
decreasing. (PR#14386)
• The reference BLAS included in R was assuming that 0*x and x*0
were always zero (whereas they could be NA or NaN in IEC 60559
arithmetic). This was seen in results from tcrossprod, and for
example that log(0) %*% 0 gave 0.
• The calculation of whether text was completely outside the device
region (in which case, you draw nothing) was wrong for screen
devices (which have [0, 0] at top-left). The symptom was (long)
text disappearing when resizing a screen window (to make it
smaller). (PR#14391)
• model.frame(drop.unused.levels = TRUE) did not take into account
NA values of factors when deciding to drop levels. (PR#14393)
• library.dynam.unload required an absolute path for libpath.
(PR#14385)
Both library() and loadNamespace() now record absolute paths for
use by searchpaths() and getNamespaceInfo(ns, "path").
• The self-starting model NLSstClosestX failed if some deviation
was exactly zero. (PR#14384)
• X11(type = "cairo") (and other devices such as png using
cairographics) and which use Pango font selection now work around
a bug in Pango when very small fonts (those with sizes between 0
and 1 in Pango's internal units) are requested. (PR#14369)
• Added workaround for the font problem with X11(type = "cairo")
and similar on Mac OS X whereby italic and bold styles were
interchanged. (PR#13463 amongst many other reports.)
• source(chdir = TRUE) failed to reset the working directory if it
could not be determined - that is now an error.
• Fix for crash of example(rasterImage) on x11(type="Xlib").
• Force Quartz to bring the on-screen display up-to-date
immediately before the snapshot is taken by grid.cap() in the
Cocoa implementation. (PR#14260)
• model.frame had an unstated 500 byte limit on variable names.
(Example reported by Terry Therneau.)
• The 256-byte limit on names is now documented. • Subassignment by [, [[ or $ on an expression object with value
NULL coerced the object to a list.
Just checked out cool new series from NVidia servers.
Now though SAS Inc/ Jim Goodnight thinks HP Blade Servers are the cool thing- the GPU takes hardware high performance computing to another level. It would be interesting to see GPU based cloud computers as well – say for the on Demand SAS (free for academics and students) but which has had some complaints of being slow.
To give users hands-on experience, the program is underpinned by a virtual computing lab (VCL), a remote access service that allows users to reserve a computer configured with a desired set of applications and operating system and then access that computer over the Internet. The lab is powered by an IBM BladeCenter infrastructure, which includes more than 500 blade servers, distributed between two locations. The assignment of the blade servers can be changed to meet shifts in the balance of demand among the various groups of users. Laura Ladrie, MSA Classroom Coordinator and Technical Support Specialist, says, “The virtual computing lab chose IBM hardware because of its quality, reliability and performance. IBM hardware is also energy efficient and lends itself well to high performance/low overhead computing.
Thats interesting since IBM now competes (as owner of SPSS) and also cooperates with SAS Institute
You’re effectively turbo-charging through deployment of many processors within the blade servers?
Yes. We’ve got machines with 192 blades on them. One of them has 202 or 203 blades. We’re using Hewlett-Packard blades with 12 CP cores on each, so it’s a total 2300 CPU cores doing the computation.
Our idea was to give every one of those cores a little piece of work to do, and we came up with a solution. It involved a very small change to the algorithm we were using, and it’s just incredible how fast we can do things now.
I don’t think of it as a grid, I think of it as essentially one computer. Most people will take a blade and make a grid out of it, where everything’s a separate computer running separate jobs.
We just look at it as one big machine that has memory and processors all over the place, so it’s a totally different concept.
GPU servers can be faster than CPU servers, though , Professor G.
TESLA GPU COMPUTING SOLUTIONS FOR DATA CENTERS
Supercharge your cluster with the Tesla family of GPU computing solutions. Deploy 1U systems from NVIDIA or hybrid CPU-GPU servers from OEMs that integrate NVIDIA® Tesla™ GPU computing processors.
When compared to the latest quad-core CPU, Tesla 20-series GPU computing processors deliver equivalent performance at 1/20th the power consumption and 1/10th the cost. Each Tesla GPU features hundreds of parallel CUDA cores and is based on the revolutionary NVIDIA® CUDA™ parallel computing architecture with a rich set of developer tools (compilers, profilers, debuggers) for popular programming languages APIs like C, C++, Fortran, and driver APIs like OpenCL and DirectCompute.
NVIDIA’s partners provide turnkey easy-to-deploy Preconfigured Tesla GPU clusters that are customizable to your needs. For 3D cloud computing applications, our partners offer the Tesla RS clusters that are optimized for running RealityServer with iray.
Usage of accelerated BLAS libraries seems to shrouded in some mystery, judging from somewhat regularly recurring requests for help on lists such as r-sig-hpc(gmane version), the R list dedicated to High-Performance Computing. Yet it doesn’t have to be; installation can be really simple (on appropriate systems).
Another issue that I felt needed addressing was a comparison between the different alternatives available, quite possibly including GPU computing. So a few weeks ago I sat down and wrote a small package to run, collect, analyse and visualize some benchmarks. That package, called gcbd (more about the name below) is now onCRAN as of this morning. The package both facilitates the data collection for the paper it also contains (in the vignette form common among R packages) and provides code to analyse the data—which is also included as a SQLite database. All this is done in the Debian and Ubuntu context by transparently installing and removing suitable packages providing BLAS implementations: that we can fully automate data collection over several competing implementations via a single script (which is also included). Contributions of benchmark results is encouraged—that is the idea of the package.
And from his paper on the same-
Analysts are often eager to reap the maximum performance from their computing platforms.
A popular suggestion in recent years has been to consider optimised basic linear algebra subprograms (BLAS). Optimised BLAS libraries have been included with some (commercial) analysis platforms for a decade (Moler 2000), and have also been available for (at least some) Linux distributions for an equally long time (Maguire 1999). Setting BLAS up can be daunting: the R language and environment devotes a detailed discussion to the topic in its Installation and Administration manual (R Development Core Team 2010b, appendix A.3.1). Among the available BLAS implementations, several popular choices have emerged. Atlas (an acronym for Automatically Tuned Linear Algebra System) is popular as it has shown very good performance due to its automated and CPU-speci c tuning (Whaley and Dongarra 1999; Whaley and Petitet 2005). It is also licensed in such a way that it permits redistribution leading to fairly wide availability of Atlas.1 We deploy Atlas in both a single-threaded and a multi-threaded con guration. Another popular BLAS implementation is Goto BLAS which is named after its main developer, Kazushige Goto (Goto and Van De Geijn 2008). While `free to use’, its license does not permit redistribution putting the onus of con guration, compilation and installation on the end-user. Lastly, the Intel Math Kernel Library (MKL), a commercial product, also includes an optimised BLAS library. A recent addition to the tool chain of high-performance computing are graphical processing units (GPUs). Originally designed for optimised single-precision arithmetic to accelerate computing as performed by graphics cards, these devices are increasingly used in numerical analysis. Earlier criticism of insucient floating-point precision or severe performance penalties for double-precision calculation are being addressed by the newest models. Dependence on particular vendors remains a concern with NVidia’s CUDA toolkit (NVidia 2010) currently still the preferred development choice whereas the newer OpenCL standard (Khronos Group 2008) may become a more generic alternative that is independent of hardware vendors. Brodtkorb et al. (2010) provide an excellent recent survey. But what has been lacking is a comparison of the e ective performance of these alternatives. This paper works towards answering this question. By analysing performance across ve di erent BLAS implementations|as well as a GPU-based solution|we are able to provide a reasonably broad comparison.
Performance is measured as an end-user would experience it: we record computing times from launching commands in the interactive R environment (R Development Core Team 2010a) to their completion.
And
Basic Linear Algebra Subprograms (BLAS) provide an Application Programming Interface
(API) for linear algebra. For a given task such as, say, a multiplication of two conformant
matrices, an interface is described via a function declaration, in this case sgemm for single
precision and dgemm for double precision. The actual implementation becomes interchangeable
thanks to the API de nition and can be supplied by di erent approaches or algorithms. This
is one of the fundamental code design features we are using here to benchmark the di erence
in performance from di erent implementations.
A second key aspect is the di erence between static and shared linking. In static linking,
object code is taken from the underlying library and copied into the resulting executable.
This has several key implications. First, the executable becomes larger due to the copy of
the binary code. Second, it makes it marginally faster as the library code is present and
no additional look-up and subsequent redirection has to be performed. The actual amount
of this performance penalty is the subject of near-endless debate. We should also note that
this usually amounts to only a small load-time penalty combined with a function pointer
redirection|the actual computation e ort is unchanged as the actual object code is identi-
cal. Third, it makes the program more robust as fewer external dependencies are required.
However, this last point also has a downside: no changes in the underlying library will be
reected in the binary unless a new build is executed. Shared library builds, on the other
hand, result in smaller binaries that may run marginally slower|but which can make use of
di erent libraries without a rebuild.
Basic Linear Algebra Subprograms (BLAS) provide an Application Programming Interface(API) for linear algebra. For a given task such as, say, a multiplication of two conformantmatrices, an interface is described via a function declaration, in this case sgemm for singleprecision and dgemm for double precision. The actual implementation becomes interchangeablethanks to the API de nition and can be supplied by di erent approaches or algorithms. Thisis one of the fundamental code design features we are using here to benchmark the di erencein performance from di erent implementations.A second key aspect is the di erence between static and shared linking. In static linking,object code is taken from the underlying library and copied into the resulting executable.This has several key implications. First, the executable becomes larger due to the copy ofthe binary code. Second, it makes it marginally faster as the library code is present andno additional look-up and subsequent redirection has to be performed. The actual amountof this performance penalty is the subject of near-endless debate. We should also note thatthis usually amounts to only a small load-time penalty combined with a function pointerredirection|the actual computation e ort is unchanged as the actual object code is identi-cal. Third, it makes the program more robust as fewer external dependencies are required.However, this last point also has a downside: no changes in the underlying library will bereected in the binary unless a new build is executed. Shared library builds, on the otherhand, result in smaller binaries that may run marginally slower|but which can make use ofdi erent libraries without a rebuild.
And summing up,
reference BLAS to be dominated in all cases. Single-threaded Atlas BLAS improves on the reference BLAS but loses to multi-threaded BLAS. For multi-threaded BLAS we nd the Goto BLAS dominate the Intel MKL, with a single exception of the QR decomposition on the xeon-based system which may reveal an error. The development version of Atlas, when compiled in multi-threaded mode is competitive with both Goto BLAS and the MKL. GPU computing is found to be compelling only for very large matrix sizes. Our benchmarking framework in the gcbd package can be employed by others through the R packaging system which could lead to a wider set of benchmark results. These results could be helpful for next-generation systems which may need to make heuristic choices about when to compute on the CPU and when to compute on the GPU.
Hardware solutions or atleast need to be a part of Revolution Analytic’s thinking as well. SPSS does not have any choice anymore though 😉
It would be interesting to see how the new SAS Cloud Computing/ Server Farm/ Time Sharing facility is benchmarking CPU and GPU for SAS analytics performance – if being done already it would be nice to see a SUGI paper on the same at http://sascommunity.org.
Multi threading needs to be taken care automatically by statistical software to optimize current local computing (including for New R)
Acceptable benchmarks for testing hardware as well as software need to be reinforced and published across vendors, academics and companies.
IPSUR stands for Introduction to Probability and Statistics Using R, ISBN: 978-0-557-24979-4, which is a textbook written for an undergraduate course in probability and statistics. The approximate prerequisites are two or three semesters of calculus and some linear algebra in a few places. Attendees of the class include mathematics, engineering, and computer science majors.
IPSUR is FREE, in the GNU sense of the word. Hard copies are available for purchase here from Lulu and will be available (coming soon) from the other standard online retailers worldwide. The price of the book is exactly the manufacturing cost plus the retailers’ markup. You may be able to get it even cheaper by downloading an electronic copy and printing it yourself, but if you elect this route then be sure to get the publisher-quality PDF from theDownloads page. And double check the price. It was cheaper for my students to buy a perfect-bound paperback from Lulu and have it shipped to their door than it was to upload the PDF to Fed-Ex Kinkos and Xerox a coil-bound copy (and on top of that go pick it up at the store).
If you are going to buy from anywhere other than Lulu then be sure to check the time-stamp on the copyright page. There is a 6 to 8 week delay from Lulu to Amazon and you may not be getting the absolute latest version available.
Refer to the Installation page for instructions to install an electronic copy of IPSUR on your personal computer. See the Feedback page for guidance about questions or comments you may have about IPSUR.
This plugin for the R Commander accompanies the text Introduction to Probability and Statistics Using R by G. Jay Kerns. The plugin contributes functions unique to the book as well as specific configuration and functionality to R Commander, the pioneering work by John Fox of McMaster University.
RcmdrPlugin.IPSUR’s primary goal is to provide a user-friendly graphical user interface (GUI) to the open-source and freely available R statistical computing environment. RcmdrPlugin.IPSUR is equipped to handle many of the statistical analyses and graphical displays usually encountered by upper division undergraduate mathematics, statistics, and engineering majors. Available features are comparable to many expensive commercial packages such as Minitab, SPSS, and JMP-IN.
Since the audience of RcmdrPlugin.IPSUR is slightly different than Rcmdr’s, certain functionality has been added and selected error-checks have been disabled to permit the student to explore alternative regions of the statistical landscape. The resulting benefit of increased flexibility is balanced by somewhat increased vulnerability to syntax errors and misuse; the instructor should keep this and the academic audience in mind when usingRcmdrPlugin.IPSUR in the classroom
DECISION STATS 