Exciting Contest at CrowdANALYTIX

A new contest from a relatively new website. This one is fast and furious and has a decent chunk of money!

 

From

 

http://www.crowdanalytix.com/contests/airport-guest-sentiment-analysis-1544282253/view/

 

Submission Deadline:
Sun, 26 February 2012 05:00 AM UTC
Results Announced by:
Mon, 05 March 2012 05:00 AM UTC
Category: Text Analytics Function: Aerospace & Aviation

 

Title
Analysis of sentiment and its intensity – feedback from airport guests
Description
ABC (name intentionally obfuscated) is one of the best managed and highly profitable airports  in India. As with all well managed airports, ABC would like to understand what guests feel about their experience when traveling, using or transiting through their airport. ABC has a website in which guests can visit and leave behind a comment, agree or disagree with others’ comments, or respond to a comment confirming or negating the expressed opinion.

The goal of this contest is to create a summarization of the opinions, feelings and sentiments expressed in the comments left behind by guests on the website. This information is being provided as data for solvers. Some understanding of the intensity of the opinion, feeling or sentiment will also be useful. For example, if there is a consistent demand for more spas across guest conversations, it needs to be highlighted.  Consistent positive or negative sentiments and opinions need to be discovered and highlighted.
Data
Guest comments have been crawled and provided to you. The data consists approximately 1000 comments from guests including the timestamp of those comments.  Personal information (name, email etc) have been hidden. This data is publicly available
Solver Expectations:
Participants may submit entries before the deadline. If a participant submits multiple entries, the entry submitted last before the deadline will be considered as the participant’s submission.
The following deliverables are expected to be submitted:
  • a report expressing the results of the opinion and sentiment mining
  • documentation about how you approached the problem, what tools, technologies and languages you used, and what problems you encountered
Timeline and Prizes: 
This contest begin on 16 Feb 2012 and will last for a duration of 9 days.
Prizes:
  • One 1st Prize – $1000
  • One 2nd Prize – $500
  • Two 3rd Prizes – $250

Radoop 0.3 launched- Open Source Graphical Analytics meets Big Data

What is Radoop? Quite possibly an exciting mix of analytics and big data computing

 

http://blog.radoop.eu/?p=12

What is Radoop?

Hadoop is an excellent tool for analyzing large data sets, but it lacks an easy-to-use graphical interface. RapidMiner is an excellent tool for data analytics, but its data size is limited by the memory available, and a single machine is often not enough to run the analyses on time. In this project, we combine the strengths of both projects and provide a RapidMiner extension for editing and running ETL, data analytics and machine learning processes over Hadoop.

We have closely integrated the highly optimized data analytics capabilities of Hive and Mahout, and the user-friendly interface of RapidMiner to form a powerful and easy-to-use data analytics solution for Hadoop.

 

and what’s new

http://blog.radoop.eu/?p=198

Radoop 0.3 released – fully graphical big data analytics

Today, Radoop had a major step forward with its 0.3 release. The new version of the visual big data analytics package adds full support for all major Hadoop distributions used these days: Apache Hadoop 0.20.2, 0.20.203, 1.0 and Cloudera’s Distribution including Apache Hadoop 3 (CDH3). It also adds support for large clusters by allowing the namenode, the jobtracker and the Hive server to reside on different nodes.

As Radoop’s promise is to make big data analytics easier, the 0.3 release is also focused on improving the user interface. It has an enhanced breakpointing system which allows to investigate intermediate results, and it adds dozens of quick fixes, so common process design mistakes get much easier to solve.

There are many further improvements and fixes, so please consult the release notes for more details. Radoop is in private beta mode, but heading towards a public release in Q2 2012. If you would like to get early access, then please apply at the signup page or describe your use case in email (beta at radoop.eu).

Radoop 0.3 (15 February 2012)

  • Support for Apache Hadoop 0.20.2, 0.20.203, 1.0 and Cloudera’s Distribution Including Apache Hadoop 3 (CDH3) in a single release
  • Support for clusters with separate master nodes (namenode, jobtracker, Hive server)
  • Enhanced breakpointing to evaluate intermediate results
  • Dozens of quick fixes for the most common process design errors
  • Improved process design and error reporting
  • New welcome perspective to help in the first steps
  • Many bugfixes and performance improvements

Radoop 0.2.2 (6 December 2011)

  • More Aggregate functions and distinct option
  • Generate ID operator for convenience
  • Numerous bug fixes and improvements
  • Improved user interface

Radoop 0.2.1 (16 September 2011)

  • Set Role and Data Multiplier operators
  • Management panel for testing Hadoop connections
  • Stability improvements for Hive access
  • Further small bugfixes and improvements

Radoop 0.2 (26 July 2011)

  • Three new algoritms: Fuzzy K-Means, Canopy, and Dirichlet clustering
  • Three new data preprocessing operators: Normalize, Replace, and Replace Missing Values
  • Significant speed improvements in data transmission and interactive analytics
  • Increased stability and speedup for K-Means
  • More flexible settings for Join operations
  • More meaningful error messages
  • Other small bugfixes and improvements

Radoop 0.1 (14 June 2011)

Initial release with 26 operators for data transmission, data preprocessing, and one clustering algorithm.

Note that Rapid Miner also has a great R extension so you can use R, a graphical interface and big data analytics is now easier and more powerful than ever.


Cyber Cold War

I try to write on cyber conflict without getting into the politics of why someone is hacking someone else. I always get beaten by someone in the comments thread when I write on politics.

But recent events have forced me to update my usual “how-to” cyber conflict to “why” cyber conflict. This is because of a terrorist attack in my hometown Delhi.

(updated-

http://www.nytimes.com/2012/02/14/world/middleeast/israeli-embassy-officials-attacked-in-india-and-georgia.html?_r=1&hp

Iran allegedly tried  (as per Israel) to assassinate the wife of Israeli Defence Attache in Delhi using a magnetic bomb, India as she went to school to pick up her kids, somebody else put a grenade in Israeli embassy car in Georgia which was found in time. 

Based on reports , initial work suggests the bomb was much more sophisticated than local terrorists, but the terrorists seemed to have some local recce work done.

India has 0 history of antisemitism but this is the second time Israelis have been targeted since 26/11 Mumbai attacks. India buys 12 % of oil annually from Iran (and refuses to join the oil embargo called by US and Europe)

Cyber Conflict is less painful than conflict, which is inevitable as long as mankind exists. Also the Western hemisphere needs a moon shot (cyber conflict could be the Sputnik like moment) and with declining and aging populations but better technology, Western Hemisphere govts need cyber conflict as they are running out of humans to fight their wars. Eastern govt. are even more obnoxious in using children for conflict propaganda, and corruption.

Last week CIA.gov website went down

This week Iranian govt is allegedly blocking https traffic on eve of Annual Revolution Day (what a coincidence!)

 

Some resources to help Internet users in Iran (or maybe this could be a dummy test for the big one – hacking the great firewall of China)

News from Hacker News-

http://news.ycombinator.com/item?id=3575029

 

I’m writing this to report the serious troubles we have regarding accessing Internet in Iran at the moment. Since Thursday Iranian government has shutted down the https protocol which has caused almost all google services (gmail, and google.com itself) to become inaccessible. Almost all websites that reply on Google APIs (like wolfram alpha) won’t work. Accessing to any website that replies on https (just imaging how many websites use this protocol, from Arch Wiki to bank websites). Also accessing many proxies is also impossible. There are almost no official reports on this and with many websites and my email accounts restricted I can just confirm this based on my own and friends experience. I have just found one report here:

Iran Shut Down Gmail , Google , Yahoo and sites using “Https” Protocol

The reason for this horrible shutdown is that the Iranian regime celebrates 1979 Islamic revolution tomorrow.

I just wanted to let you guys know about this. If you have any solution regarding bypassing this restriction please help!

 

The boys at Tor think they can help-

but its not so elegant, as I prefer creating a  batch file rather than explain coding to newbies. 

this is still getting to better and easier interfaces

https://www.torproject.org/projects/obfsproxy-instructions.html.en

Obfsproxy Instructions

client torrc

Step 1: Install dependencies, obfsproxy, and Tor

 

You will need a C compiler (gcc), the autoconf and autotools build system, the git revision control system, pkg-config andlibtoollibevent-2 and its headers, and the development headers of OpenSSL.

On Debian testing or Ubuntu oneiric, you could do:
# apt-get install autoconf autotools-dev gcc git pkg-config libtool libevent-2.0-5 libevent-dev libevent-openssl-2.0-5 libssl-dev

If you’re on a more stable Linux, you can either try our experimental backport libevent2 debs or build libevent2 from source.

Clone obfsproxy from its git repository:
$ git clone https://git.torproject.org/obfsproxy.git
The above command should create and populate a directory named ‘obfsproxy’ in your current directory.

Compile obfsproxy:
$ cd obfsproxy
$ ./autogen.sh && ./configure && make

Optionally, as root install obfsproxy in your system:
# make install

If you prefer not to install obfsproxy as root, you can instead just modify the Transport lines in your torrc file (explained below) to point to your obfsproxy binary.

You will need Tor 0.2.3.11-alpha or later.


Step 2a: If you’re the client…

 

First, you need to learn the address of a bridge that supports obfsproxy. If you don’t know any, try asking a friend to set one up for you. Then the appropriate lines to your tor configuration file:

UseBridges 1
Bridge obfs2 128.31.0.34:1051
ClientTransportPlugin obfs2 exec /usr/local/bin/obfsproxy --managed

Don’t forget to replace 128.31.0.34:1051 with the IP address and port that the bridge’s obfsproxy is listening on.
 Congratulations! Your traffic should now be obfuscated by obfsproxy. You are done! You can now start using Tor.

For old fashioned tunnel creation under Seas of English Channel-

http://dag.wieers.com/howto/ssh-http-tunneling/

Tunneling SSH over HTTP(S)
This document explains how to set up an Apache server and SSH client to allow tunneling SSH over HTTP(S). This can be useful on restricted networks that either firewall everything except HTTP traffic (tcp/80,tcp/443) or require users to use a local (HTTP) proxy.
A lot of people asked why doing it like this if you can just make sshd listen on port 443. Well, that might work if your environment is not hardened like I have seen at several companies, but this setup has a few advantages.

  • You can proxy to anywhere (see the Proxy directive in Apache) based on names
  • You can proxy to any port you like (see the AllowCONNECT directive in Apache)
  • It works even when there is a layer-7 protocol firewall
  • If you enable proxytunnel ssl support, it is indistinguishable from real SSL traffic
  • You can come up with nice hostnames like ‘downloads.yourdomain.com’ and ‘pictures.yourdomain.com’ and for normal users these will look like normal websites when visited.
  • There are many possibilities for doing authentication further along the path
  • You can do proxy-bouncing to the n-th degree to mask where you’re coming from or going to (however this requires more changes to proxytunnel, currently I only added support for one remote proxy)
  • You do not have to dedicate an IP-address for sshd, you can still run an HTTPS site

Related-

http://opensourceandhackystuff.blogspot.in/2012/02/captive-portal-security-part-1.html

and some crypto for young people

http://users.telenet.be/d.rijmenants/en/onetimepad.htm

 

Me- What am I doing about it? I am just writing poems on hacking at http://poemsforkush.com

C4ISTAR for Hacking and Cyber Conflict

As per http://en.wikipedia.org/wiki/C4ISTAR

C2I stands for command, control, and intelligence.

C3I stands for command, control, communications, and intelligence.

C4I stands for command, control, communications, computers, and (military) intelligence.

C4ISTAR is the British acronym used to represent the group of the military functions designated by C4 (command, control, communications, computers), I (military intelligence), and STAR (surveillance, target acquisition, and reconnaissance) in order to enable the coordination of operations

I increasingly believe that cyber conflict will develop its own terminology and theory and paradigms in due time. In the meantime, it will adopt paradigms from existing military literature and adapt it to the unique sub culture of cyber conflict for both offensive, defensive as well as pre-emptive actions. Here I am theorizing for a case of targeted hacking attacks rather than massive attacks that bring down a website for a few hours and achieve nothing but a few press headlines . I would also theorize on countering such attacks.

So what would be the C4ISTAR for –

1) Media company supporting SOPA/PIPA/Take down Mega Upload-

Command and Control refers to the ability of commanders to direct forces-

This will be the senior executives including the members of board, legal officers, and public relationship/marketing people. Their name is available from corporate websites, and social media scraping can ensure both a list of contact addresses (online) as well as biases for phishing /malware attacks. This could also include phone (flooding or voicemail hacking ) attacks , and attacks against the email server of the company rather than the corporate website.

Communications– This will include all online and social media channels including websites of the media company , but also  those of the press relations firms handling communications , phones,websites- anything which the target is likely to communicate externally (and if possible internal communication)

Timing is everything- coordinating attacks immediately is juevenile, but it might be more mature to attack on vulnerable days like product launches or just before a board of directors meeting

Intelligence

Most corporates have an in-house research team, they can be easily targeted using social media channels, but also offline research and digging deep. Targeting intelligence corps of the target corporate is likely to produce a much better disruption. Eventually they can be persuaded to stop working for that corporate.

Computers– Anything that runs on electricity and can be disabled – should be disabled. This might require much more creativity than just flooding.

 surveillance-  This can be both online as well as offline, and would be of electronic assets, likely responses for the attack, and the key people who are to be disrupted.

target acquisition-  at least ten people within each corporate can and should be ideally disrupted, rather than just the website. this would call for social media scraping, and prior planning. even email in-boxes can be disrupted (if all else fails)

and reconnaissance-

study your target companies, target employees, and their strategies.

Then segment and prioritize in a list of  matrix of 10  to 10, who is more vulnerable and who is more valuable to attack.

the C4ISTAR for -a hacker activist organization is much more complicated but forensics reveal that most hackers tend to leave a signature style (in terms of computers,operating systems,machine ids,communication, tools, or even port numbers used)

the best defense for a media rich company to prevent hacking attacks is to first identify its own C4ISTAR structure for its digital content strategy and then fortify as well as scrub vulnerabilities (including from online information regarding its own employees)

(to be continued)

http://www.catb.org/~esr/faqs/hacker-howto.html

The Hacker Attitude

Product Review – Revolution R 5.0

So I got the email from Revolution R. Version 5.0 is ready for download, and unlike half hearted attempts by many software companies they make it easy for the academics and researchers to get their free copy. Free as in speech and free as in beer.

Some thoughts-

1) R ‘s memory problem is now an issue of marketing and branding. Revolution Analytics has definitely bridged this gap technically  beautifully and I quote from their documentation-

The primary advantage 64-bit architectures bring to R is an increase in the amount of memory available to a given R process.
The first benefit of that increase is an increase in the size of data objects you can create. For example, on most 32-bit versions of R, the largest data object you can create is roughly 3GB; attempts to create 4GB objects result in errors with the message “cannot allocate vector of length xxxx.”
On 64-bit versions of R, you can generally create larger data objects, up to R’s current hard limit of 231 􀀀 1 elements in a vector (about 2 billion elements). The functions memory.size and memory.limit help you manage the memory used byWindows versions of R.
In 64-bit Revolution R Enterprise, R sets the memory limit by default to the amount of physical RAM minus half a gigabyte, so that, for example, on a machine with 8GB of RAM, the default memory limit is 7.5GB:

2) The User Interface is best shown as below or at  https://docs.google.com/presentation/pub?id=1V_G7r0aBR3I5SktSOenhnhuqkHThne6fMxly_-4i8Ag&start=false&loop=false&delayms=3000

-(but I am still hoping for the GUI ,Revolution Analytics promised us for Christmas)

3) The partnership with Microsoft HPC is quite awesome given Microsoft’s track record in enterprise software penetration

but I am also interested in knowing more about the Oracle version of R and what it will do there.

Preview- Google Cloud SQL

From –http://code.google.com/apis/sql/

What is Google Cloud SQL?

Google Cloud SQL is web service that allows you to create, configure, and use relational databases with your App Engine applications. It is a fully-managed service that maintains, manages, and administers your databases, allowing you to focus on your applications and services.

By offering the capabilities of a MySQL database, the service enables you to easily move your data, applications, and services into and out of the cloud. This allows for high data portability and helps in faster time-to-market because you can quickly leverage your existing database (using JDBC and/or DB-API) in your App Engine application.

Here is where you can get an invite to the beta only Google Cloud SQL

Sign up for Limited Preview

Google Cloud SQL is available to a limited number of users. To sign up for the service:

  1. Visit the Google APIs Console. The console opens the All services pane.
  2. Find the SQL Service line in the Services table and click Request access…
  3. Fill out the enrollment form.
  4. Our team will review your enrollment information and respond by email to the address associated with your Google Account.
  5. Follow the link in the email to view the Terms of Service. Please read these carefully before accepting.
  6. Sign up for the google-cloud-sql-announce group to receive important announcements and product news. (NOTE- Members: 384)
and after all that violence and double talk, a walk in the clouds with SQL.
1. There are three kinds of instances in the beta view
2. Wait for the Instance to be created note- the Design of the Interface uptil now is much better than Amazon’s.  
Note you need to have an appspot application from Google Apps and can choose between the Python and Java versions. Quite clearly there is a play for other languages too. I think GO is also supported.
3. You can import your data from your Google Storage bucket
4. I am not that hot at coding or maybe the interface was too pretty. Anyways- the log tells me that import of the text file has failed from Google Storage to Google Cloud SQL 
5. Incidentally the Google Cloud Storage interface is also much better than the Amazon GUI for transferring data- Note I was using the classical statistical dataset Boston Housing Data as the test case. 
6. The SQL prompt is the weakest part of the design process of the Interphase. There is no Query builder and the SELECT FROM WHERE prompt is slightly amusing/ insulting . I mean guys either throw in a fully fledged GUI for query builder similar to the MYSQL Workbench , than create a pretty white command prompt.
7. You can also export your data back to your Google Storage bucket 
These are early days, and I am trying to see if there is a play for some cloud kind of ODBC action between R, Prediction API , and the cloud SQL… so try it out yourself at http://code.google.com/apis/sql/ and see if there is any juice you can build  here.

Google Doodle for Diwali Greetings

Hey,

If you like Diwali the festival and want Google to create a doodle for it, just send an email to proposals@google.com

 

http://en.wikipedia.org/wiki/Diwali

Diwali or Deepavali[note 1], popularly known as the “festival of lights,” is a festival celebrated between mid-October and mid-November for different reasons. For Hindus, Diwali is one of the most important festivals of the year and is celebrated in families by performing traditional activities together in their homes. For Jains, Diwali marks the attainment of moksha or nirvana by Mahavira in 527 BC.[1][2]

Diwali is an official holiday in India,[3] NepalSri LankaMyanmarMauritiusGuyanaTrinidad & TobagoSurinameMalaysiaSingapore,[4] andFiji.

The name “Diwali” is a contraction of “Dipawali” (Sanskrit: दीपावली Dīpāwalī), which translates into “row of lamps”.[5] Diwali involves the lighting of small clay lamps (diyas or dīpas) in Sanskrit: दीप) filled with oil to signify the triumph of good over evil. During Diwali, all the celebrants wear new clothes and share sweets and snacks with family members and friends.

The festival starts with Dhanteras on which most Indian business communities begin their financial year. The second day of the festival, Naraka Chaturdasi, marks the vanquishing of the demon Naraka by Lord Krishna and his wife SatyabhamaAmavasya, the third day of Diwali, marks the worship of Lakshmi, the goddess of wealth in her most benevolent mood, fulfilling the wishes of her devotees. Amavasya also tells the story of Lord Vishnu, who in his dwarf incarnation vanquished the Bali, and banished him to Patala. It is on the fourth day of Diwali, Kartika Shudda Padyami, that Bali went to patala and took the reins of his new kingdom in there. The fifth day is referred to as Yama Dvitiya (also called Bhai Dooj), and on this day sisters invite their brothers to their homes.

While the story behind Diwali and the manner of celebration varies from region to region (festive fireworks, worship, lights, sharing of sweets), the essence is the same – to rejoice in the Inner Light

from-

http://www.google.com/doodle4google/history.html

How can Google users/the public submit ideas for doodles?

The doodle team is open to user ideas; requests for doodles can be sent to proposals@google.com