Installing Ipython Notebook on Ubuntu 12

I ran into a series of errors and finally managed to make Ipython run on my Ubuntu 12. Notice I am adding some extra stuff in terms of mathjax and pandoc but that is just for a smoother install. Note the trouble point was the package pyzmq but it was troubleshooted by both the –upgrade option as well as the installing of python-dev

sudo apt-get install python-pip
sudo apt-get install python-dev

sudo pip install --upgrade ipython[all]
sudo pip install invoke
sudo pip install jinja2
sudo pip install --upgrade pyzmq

sudo python -m IPython.external.mathjax
sudo apt-get install pandoc
sudo pip install tornado jsonschema

ipython notebook

Screenshot from 2015-04-07 01:03:33

 

Sources-

  1. http://askubuntu.com/questions/390457/how-to-install-the-latest-ipython-notebook-in-ubuntu-12-04
  2. http://askubuntu.com/questions/100529/how-to-install-python-package-pyzmq-properly
  3. http://stackoverflow.com/questions/25318766/gcc-failed-when-pip-upgrading-pyzmq

Running R GUI on Google Compute

I wanted to run R GUIs ( rattle, Rcmdr, Deducer) on my Google Compute Instance, but didnt know how to figure out how to enable x11.

Initially I just tried to enable x11 forwarding in the local ssh (Ubuntu) and remote sshd( GCE), but it still needed some more.

Note I use gedit to edit files locally ( since it is easier) and vi to edit files remotely ( because I didnt have a graphical environment there yet) . I used vi help from the link here  (basically sudo vi filename opens the file in Linux, you scroll down and press Insert to write your changes, then hit escape, then write this to save and quit :qw ( or :q! to NOT save and quit), your mouse is quite useless and the arrow keys dont help much in vi- I assure you that)

[local]
/etc/ssh_config or ~/.ssh/config
ForwardX11 yes

restarted local ssh

[remote]
/etc/sshd_config
X11Forwarding yes

restarted remote sshd

Well this is how it is done- following is a copy and paste from actual discussion-

here are two steps you have to do in order to run X-windows applications on your instance.

1) You have to install some X-windows applications on your instance.  I used the command
sudo apt-get install xterm
which works on Ubuntu.  On Centos, you would use the command
yum install xterm
but I didn’t test that.
2) You have to create an X-windows tunnel through SSH.  You do that with the -X switch to the gcutil ssh command:
 gcutil ssh –ssh_arg -X INSTANCE
When you login to the instance, verify that the tunnel is in place.
$rman@test-pd:~$ echo $DISPLAY
localhost:10.0
rman@test-pd:~$
By way of contrast, this is what it looks like if the tunnel didn’t work:
rman@test-pd:~$ echo $DISPLAY
rman@test-pd:~$

Hat Tip- gce discussion group on google groups  https://groups.google.com/forum/#!forum/gce-discussion  and Jeff Silverman from the GCE team.

Modifying Sources File in Linux for Monet DB Installation

If you like me are a Linux newbie, the following helps in modifying the source file . i.e for adding in sources for Monet DB installation

sudo sh -c ‘echo “deb http://dev.monetdb.org/downloads/deb/ precise monetdb” > /etc/apt/sources.list.d/monetdb.list’
sudo sh -c ‘echo “deb-src http://dev.monetdb.org/downloads/deb/ precise monetdb” > /etc/apt/sources.list.d/monetdb.list’

 

ps- I was on Google compute/ Linux

 

Cyber Cold War

I try to write on cyber conflict without getting into the politics of why someone is hacking someone else. I always get beaten by someone in the comments thread when I write on politics.

But recent events have forced me to update my usual “how-to” cyber conflict to “why” cyber conflict. This is because of a terrorist attack in my hometown Delhi.

(updated-

http://www.nytimes.com/2012/02/14/world/middleeast/israeli-embassy-officials-attacked-in-india-and-georgia.html?_r=1&hp

Iran allegedly tried  (as per Israel) to assassinate the wife of Israeli Defence Attache in Delhi using a magnetic bomb, India as she went to school to pick up her kids, somebody else put a grenade in Israeli embassy car in Georgia which was found in time. 

Based on reports , initial work suggests the bomb was much more sophisticated than local terrorists, but the terrorists seemed to have some local recce work done.

India has 0 history of antisemitism but this is the second time Israelis have been targeted since 26/11 Mumbai attacks. India buys 12 % of oil annually from Iran (and refuses to join the oil embargo called by US and Europe)

Cyber Conflict is less painful than conflict, which is inevitable as long as mankind exists. Also the Western hemisphere needs a moon shot (cyber conflict could be the Sputnik like moment) and with declining and aging populations but better technology, Western Hemisphere govts need cyber conflict as they are running out of humans to fight their wars. Eastern govt. are even more obnoxious in using children for conflict propaganda, and corruption.

Last week CIA.gov website went down

This week Iranian govt is allegedly blocking https traffic on eve of Annual Revolution Day (what a coincidence!)

 

Some resources to help Internet users in Iran (or maybe this could be a dummy test for the big one – hacking the great firewall of China)

News from Hacker News-

http://news.ycombinator.com/item?id=3575029

 

I’m writing this to report the serious troubles we have regarding accessing Internet in Iran at the moment. Since Thursday Iranian government has shutted down the https protocol which has caused almost all google services (gmail, and google.com itself) to become inaccessible. Almost all websites that reply on Google APIs (like wolfram alpha) won’t work. Accessing to any website that replies on https (just imaging how many websites use this protocol, from Arch Wiki to bank websites). Also accessing many proxies is also impossible. There are almost no official reports on this and with many websites and my email accounts restricted I can just confirm this based on my own and friends experience. I have just found one report here:

Iran Shut Down Gmail , Google , Yahoo and sites using “Https” Protocol

The reason for this horrible shutdown is that the Iranian regime celebrates 1979 Islamic revolution tomorrow.

I just wanted to let you guys know about this. If you have any solution regarding bypassing this restriction please help!

 

The boys at Tor think they can help-

but its not so elegant, as I prefer creating a  batch file rather than explain coding to newbies. 

this is still getting to better and easier interfaces

https://www.torproject.org/projects/obfsproxy-instructions.html.en

Obfsproxy Instructions

client torrc

Step 1: Install dependencies, obfsproxy, and Tor

 

You will need a C compiler (gcc), the autoconf and autotools build system, the git revision control system, pkg-config andlibtoollibevent-2 and its headers, and the development headers of OpenSSL.

On Debian testing or Ubuntu oneiric, you could do:
# apt-get install autoconf autotools-dev gcc git pkg-config libtool libevent-2.0-5 libevent-dev libevent-openssl-2.0-5 libssl-dev

If you’re on a more stable Linux, you can either try our experimental backport libevent2 debs or build libevent2 from source.

Clone obfsproxy from its git repository:
$ git clone https://git.torproject.org/obfsproxy.git
The above command should create and populate a directory named ‘obfsproxy’ in your current directory.

Compile obfsproxy:
$ cd obfsproxy
$ ./autogen.sh && ./configure && make

Optionally, as root install obfsproxy in your system:
# make install

If you prefer not to install obfsproxy as root, you can instead just modify the Transport lines in your torrc file (explained below) to point to your obfsproxy binary.

You will need Tor 0.2.3.11-alpha or later.


Step 2a: If you’re the client…

 

First, you need to learn the address of a bridge that supports obfsproxy. If you don’t know any, try asking a friend to set one up for you. Then the appropriate lines to your tor configuration file:

UseBridges 1
Bridge obfs2 128.31.0.34:1051
ClientTransportPlugin obfs2 exec /usr/local/bin/obfsproxy --managed

Don’t forget to replace 128.31.0.34:1051 with the IP address and port that the bridge’s obfsproxy is listening on.
 Congratulations! Your traffic should now be obfuscated by obfsproxy. You are done! You can now start using Tor.

For old fashioned tunnel creation under Seas of English Channel-

http://dag.wieers.com/howto/ssh-http-tunneling/

Tunneling SSH over HTTP(S)
This document explains how to set up an Apache server and SSH client to allow tunneling SSH over HTTP(S). This can be useful on restricted networks that either firewall everything except HTTP traffic (tcp/80,tcp/443) or require users to use a local (HTTP) proxy.
A lot of people asked why doing it like this if you can just make sshd listen on port 443. Well, that might work if your environment is not hardened like I have seen at several companies, but this setup has a few advantages.

  • You can proxy to anywhere (see the Proxy directive in Apache) based on names
  • You can proxy to any port you like (see the AllowCONNECT directive in Apache)
  • It works even when there is a layer-7 protocol firewall
  • If you enable proxytunnel ssl support, it is indistinguishable from real SSL traffic
  • You can come up with nice hostnames like ‘downloads.yourdomain.com’ and ‘pictures.yourdomain.com’ and for normal users these will look like normal websites when visited.
  • There are many possibilities for doing authentication further along the path
  • You can do proxy-bouncing to the n-th degree to mask where you’re coming from or going to (however this requires more changes to proxytunnel, currently I only added support for one remote proxy)
  • You do not have to dedicate an IP-address for sshd, you can still run an HTTPS site

Related-

http://opensourceandhackystuff.blogspot.in/2012/02/captive-portal-security-part-1.html

and some crypto for young people

http://users.telenet.be/d.rijmenants/en/onetimepad.htm

 

Me- What am I doing about it? I am just writing poems on hacking at http://poemsforkush.com

Moving data between Windows and Ubuntu VMWare partition

I use Windows 7 on my laptop (it came pre-installed) and Ubuntu using the VMWare Player. What are the advantages of using VM Player instead of creating a dual-boot system? Well I can quickly shift from Ubuntu to Windows and bakc again without restarting my computer everytime. Using this approach allows me to utilize software that run only on Windows and run software like Rattle, the R data mining GUI, that are much easier installed on Linux.

However if your statistical software is on your Virtual Disk , and your data is on your Windows disk, you need a way to move data from Windows to Ubuntu.

The solution to this as per Ubuntu forums is –http://communities.vmware.com/thread/55242

Open My Computer, browse to the folder you want to share.  Right-click on the folder, select Properties.  Sharing tab.  Select the radio button to “Share this Folder”.  Change the default generated name if you wish; add a description if you wish.  Click the Permissions button to modify the security settings of what users can read/write to the share.

On the Linux side, it depends on the distro, the shell, and the window manager.

Well Ubuntu makes it really easy to configure the Linux steps to move data within Windows and Linux partitions.

 

NEW UPDATE-

VMmare makes it easy to share between your Windows (host) and Linux (guest) OS

 

Step 1

and step 2

Do this

 

and

Start the Wizard

when you finish the wizard and share a drive or folder- hey where do I see my shared ones-

 

see this folder in Linux- /mnt/hgfs (bingo!)

Hacker HW – Make this folder //mnt/hgfs a shortcut in Places your Ubuntu startup

Hacker Hw 2-

Upload using an anon email your VM dark data to Ubuntu one

Delete VM

Purge using software XX

Reinstall VM and bring back backup

 

Note time to do this

 

 

 

-General Sharing in Windows

 

 

Just open the Network tab in Ubuntu- see screenshots below-

Windows will now ask your Ubuntu user for login-

Once Logged in Windows from within Ubuntu Vmware, this is what happens

You see a tab called “users on “windows username”- pc appear on your Ubuntu Desktop  (see top right of the screenshot)

If you double click it- you see your windows path

You can now just click and drag data between your windows and linux partitions , just the way you do it in Windows .

So based on this- if you want to build  decision trees, artifical neural networks, regression models, and even time series models for zero capital expenditure- you can use both Ubuntu/R without compromising on your IT policy of Windows only in your organization (there is a shortage of Ubuntu trained IT administrators in the enterprise world)

Revised Installation Procedure for utilizing both Ubuntu /R/Rattle data mining on your Windows PC.

Using VMWare to build a free data mining system in R, as well as isolate your analytics system (thus using both Linux and Windows without overburdening your machine)

First Time

  1. http://downloads.vmware.com/d/info/desktop_end_user_computing/vmware_player/4_0Download and Install
  2. http://www.ubuntu.com/download/ubuntu/downloadDownload Only
  3. Create New Virtual Image in VM Ware Player
  4. Applications—–Terminal——sudo apt get-install R (to download and install)
  5.                                          sudo R (to open R)
  6. Once R is opened type this  —-install.packages(rattle)—– This will install rattle
  7. library(rattle) will load Rattle—–
  8. rattle() will open the GUI—-
Getting Data from Host to Guest VM
Next Time
  1. Go to VM Player
  2. Open the VM
  3. sudo R in terminal to bring up R
  4. library(rattle) within R
  5. rattle()
At this point even if you dont know any Linux and dont know any R, you can create data mining models using the Rattle GUI (and time series model using E pack in the R Commander GUI) – What can Rattle do in data mining? See this slideshow-http://www.decisionstats.com/data-mining-with-r-gui-rattle-rstats/
If Google Docs is banned as per your enterprise organizational IT policy of having Windows Explorer only- well you can see these screenshots http://rattle.togaware.com/rattle-screenshots.html

Using Two Operating Systems for RATTLE, #Rstats Data Mining GUI

Using a virtual partition is slightly better than using a dual boot system. That is because you can keep the specialized operating system (usually Linux) within the main operating system (usually Windows), browse and alternate between the two operating system just using a simple command, and can utilize the advantages of both operating system.

Also you can create project specific discs for enhanced security.

In my (limited ) Mac experience, the comparisons of each operating system are-

1) Mac-  Both robust and aesthetically designed OS, the higher price and hardware-lockin for Mac remains a disadvantage. Also many stats and analytical software just wont work on the Mac

2) Windows- It is cheaper than Mac and easier to use than Linux. Also has the most compatibility with applications (usually when not crashing)

3) Linux- The lightest and most customized software in the OS class, free to use, and has many lite versions for newbies. Not compatible with mainstream corporate IT infrastructure as of 2011.

I personally use VMWare Player for creating the virtual disk (as much more convenient than the wubi.exe method)  from http://www.vmware.com/support/product-support/player/  (and downloadable from http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0)

That enables me to use Ubuntu on the alternative OS- keeping my Windows 7 for some Windows specific applications . For software like Rattle, the R data mining GUI , it helps to use two operating systems, in view of difficulties in GTK+.

Installing Rattle on Windows 7 is a major pain thanks to backward compatibility issues and version issues of GTK, but it installs on Ubuntu like a breeze- and it is very very convenient to switch between the two operating systems

Download Rattle from http://rattle.togaware.com/ and test it on the dual OS arrangement to see yourself.

 

 

 

 

 

RStudio 3- Making R as simple as possible but no simpler

From the nice shiny blog at http://blog.rstudio.org/, a shiny new upgraded software (and I used the Cobalt theme)–this is nice!

awesome coding!!!

 

http://www.rstudio.org/download/

Download RStudio v0.94

Diagram desktop

If you run R on your desktop:

Download RStudio Desktop

OR

Diagram server

If you run R on a Linux server and want to enable users to remotely access RStudio using a web browser:

Download RStudio Server

 

RStudio v0.94 — Release Notes

June 15th, 2011

 

New Features and Enhancements

Source Editor and Console

  • Run code:
    • Run all lines in source file
    • Run to current line
    • Run from current line
    • Redefine current function
    • Re-run previous region
    • Code is now run line-by-line in the console
  • Brace, paren, and quote matching
  • Improved cursor placement after newlines
  • Support for regex find and replace
  • Optional syntax highlighting for console input
  • Press F1 for help on current selection
  • Function navigation / jump to function
  • Column and line number display
  • Manually set/switch document type
  • New themes: Solarized and Solarized Dark

Plots

  • Improved image export:
    • Formats: PNG, JPEG, TIFF, SVG, BMP, Metafile, and Postscript
    • Dynamic resize with preview
    • Option to maintain aspect ratio when resizing
    • Copy to clipboard as bitmap or metafile
  • Improved PDF export:
    • Specify custom sizes
    • Preview before exporting
  • Remove individual plots from history
  • Resizable plot zoom window

History

  • History tab synced to loaded .Rhistory file
  • New commands:
    • Load and save history
    • Remove individual items from history
    • Clear all history
  • New options:
    • Load history from working directory or global history file
    • Save history always or only when saving .RData
    • Remove duplicate entries in history
  • Shortcut keys for inserting into console or source

Packages

  • Check for package updates
  • Filter displayed packages
  • Install multiple packages
  • Remove packages
  • New options:
    • Install from repository or local archive file
    • Target library
    • Install dependencies

Miscellaneous

  • Find text within help topic
  • Sort file listing by name, type, size, or modified
  • Set working directory based on source file, files pane, or browsed for directory.
  • Console titlebar button to view current working directory in files pane
  • Source file menu command
  • Replace space and dash with dot (.) in import dataset generated variable names
  • Add decimal separator preference for import dataset
  • Added .tar.gz (Linux) and .zip (Windows) distributions for non-admin installs
  • Read /etc/paths.d on OS X to ensure RStudio has the same path as terminal sessions do
  • Added manifest to rsession.exe to prevent unwanted program files and registry virtualization

Server

  • Break PAM auth into its own binary for improved compatibility with 3rd party PAM authorization modules.
  • Ensure that AppArmor profile is enforced even after reboot
  • Ability to add custom LD library path for all sessions
  • Improved R discovery:
    • Use which R then fallback to scanning for R script
    • Run R discovery unconfined then switch into restricted profile
  • Default to uncompressed save.image output if the administrator or user hasn’t specified their own options (improved suspend/resume performance)
  • Ensure all running sessions are automatically updated during server version upgrade
  • Added verify-installation command to rstudio-server utility for easily capturing configuration and startup related errors

 

Bug Fixes

Source Editor

  • Undo to unedited state clears now dirty bit
  • Extract function now captures free variables used on lhs
  • Selected variable highlight now visible in all themes
  • Syncing to source file updates made outside of RStudio now happens immediately at startup and does not cause a scroll to the bottom of the document.
  • Fixed various issues related to copying and pasting into word processors
  • Fixed incorrect syntax highlighting issues in .Rd files
  • Make sure font size for printed source files matches current editor setting
  • Eliminate conflict with Ctrl+F shortcut key on OS X
  • Zoomed Google Chrome browser no longer causes cursor position to be off
  • Don’t prevent opening of unknown file types in the editor

Console

  • Fixed sporadic missing underscores (and other bottom clipping of text) in console
  • Make sure console history is never displayed offscreen
  • Page Up and Page Down now work properly in the console
  • Substantially improved console performance for both rapid output and large quantities of output

Miscellaneous

  • Install successfully on Windows with special characters in home directory name
  • make install more tolerant of configurations where it can’t write into /usr/share
  • Eliminate spurious stderr output in forked children of multicore package
  • Ensure that file modified times always update in the files pane after a save
  • Always default to installing packages into first writeable path of .libPaths()
  • Ensure that LaTeX log files are always preserved after compilePdf
  • Fix conflicts with zap function from epicalc package
  • Eliminate shortcut key conflicts with Ubuntu desktop workspace switching shortcuts
  • Always prompt when attempting to save files of the same name
  • Maximized main window now properly restored when reopening RStudio
  • PAM authorization works correctly even if account has password expiration warning
  • Correct display of manipulate panel when Plots pane is on the left

 

Previous Release Notes