Tag: US

Interview Rapid-I -Ingo Mierswa and Simon Fischer

Here is an interview with Dr Ingo Mierswa , CEO of Rapid -I and Dr Simon Fischer, Head R&D. Rapid-I makes the very popular software Rapid Miner – perhaps one of the earliest leading open source software in business analytics and business intelligence. It is quite easy to use, deploy and with it’s extensions and innovations (including compatibility with R )has continued to grow tremendously through the years.

In an extensive interview Ingo and Simon talk about algorithms marketplace, extensions , big data analytics, hadoop, mobile computing and use of the graphical user interface in analytics.

Special Thanks to Nadja from Rapid I communication team for helping coordinate this interview.( Statuary Blogging Disclosure- Rapid I is a marketing partner with Decisionstats as per the terms in https://decisionstats.com/privacy-3/)

Ajay- Describe your background in science. What are the key lessons that you have learnt while as scientific researcher and what advice would you give to new students today.

Ingo: My time as researcher really was a great experience which has influenced me a lot. I have worked at the AI lab of Prof. Dr. Katharina Morik, one of the persons who brought machine learning and data mining to Europe. Katharina always believed in what we are doing, encouraged us and gave us the space for trying out new things. Funnily enough, I never managed to use my own scientific results in any real-life project so far but I consider this as a quite common gap between science and the “real world”. At Rapid-I, however, we are still heavily connected to the scientific world and try to combine the best of both worlds: solving existing problems with leading-edge technologies.

Simon: In fact, during my academic career I have not worked in the field of data mining at all. I worked on a field some of my colleagues would probably even consider boring, and that is theoretical computer science. To be precise, my research was in the intersection of game theory and network theory. During that time, I have learnt a lot of exciting things, none of which had any business use. Still, I consider that a very valuable experience. When we at Rapid-I hire people coming to us right after graduating, I don’t care whether they know the latest technology with a fancy three-letter acronym – that will be forgotten more quickly than it came. What matters is the way you approach new problems and challenges. And that is also my recommendation to new students: work on whatever you like, as long as you are passionate about it and it brings you forward.

Ajay-  How is the Rapid Miner Extensions marketplace moving along. Do you think there is a scope for people to say create algorithms in a platform like R , and then offer that algorithm as an app for sale just like iTunes or Android apps.

 Simon: Well, of course it is not going to be exactly like iTunes or Android apps are, because of the more business-orientated character. But in fact there is a scope for that, yes. We have talked to several developers, e.g., at our user conference RCOMM, and several people would be interested in such an opportunity. Companies using data mining software need supported software packages, not just something they downloaded from some anonymous server, and that is only possible through a platform like the new Marketplace. Besides that, the marketplace will not only host commercial extensions. It is also meant to be a platform for all the developers that want to publish their extensions to a broader community and make them accessible in a comfortable way. Of course they could just place them on their personal Web pages, but who would find them there? From the Marketplace, they are installable with a single click.

Ingo: What I like most about the new Rapid-I Marketplace is the fact that people can now get something back for their efforts. Developing a new algorithm is a lot of work, in some cases even more that developing a nice app for your mobile phone. It is completely accepted that people buy apps from a store for a couple of Dollars and I foresee the same for sharing and selling algorithms instead of apps. Right now, people can already share algorithms and extensions for free, one of the next versions will also support selling of those contributions. Let’s see what’s happening next, maybe we will add the option to sell complete RapidMiner workflows or even some data pools…

Ajay- What are the recent features in Rapid Miner that support cloud computing, mobile computing and tablets. How do you think the landscape for Big Data (over 1 Tb ) is changing and how is Rapid Miner adapting to it.

Simon: These are areas we are very active in. For instance, we have an In-Database-Mining Extension that allows the user to run their modelling algorithms directly inside the database, without ever loading the data into memory. Using analytic databases like Vectorwise or Infobright, this technology can really boost performance. Our data mining server, RapidAnalytics, already offers functionality to send analysis processes into the cloud. In addition to that, we are currently preparing a research project dealing with data mining in the cloud. A second project is targeted towards the other aspect you mention: the use of mobile devices. This is certainly a growing market, of course not for designing and running analyses, but for inspecting reports and results. But even that is tricky: When you have a large screen you can display fancy and comprehensive interactive dashboards with drill downs and the like. On a mobile device, that does not work, so you must bring your reports and visualizations very much to the point. And this is precisely what data mining can do – and what is hard to do for classical BI.

Ingo: Then there is Radoop, which you may have heard of. It uses the Apache Hadoop framework for large-scale distributed computing to execute RapidMiner processes in the cloud. Radoop has been presented at this year’s RCOMM and people are really excited about the combination of RapidMiner with Hadoop and the scalability this brings.

 Ajay- Describe the Rapid Miner analytics certification program and what steps are you taking to partner with academic universities.

Ingo: The Rapid-I Certification Program was created to recognize professional users of RapidMiner or RapidAnalytics. The idea is that certified users have demonstrated a deep understanding of the data analysis software solutions provided by Rapid-I and how they are used in data analysis projects. Taking part in the Rapid-I Certification Program offers a lot of benefits for IT professionals as well as for employers: professionals can demonstrate their skills and employers can make sure that they hire qualified professionals. We started our certification program only about 6 months ago and until now about 100 professionals have been certified so far.

Simon: During our annual user conference, the RCOMM, we have plenty of opportunities to talk to people from academia. We’re also present at other conferences, e.g. at ECML/PKDD, and we are sponsoring data mining challenges and grants. We maintain strong ties with several universities all over Europe and the world, which is something that I would not want to miss. We are also cooperating with institutes like the ITB in Dublin during their training programmes, e.g. by giving lectures, etc. Also, we are leading or participating in several national or EU-funded research projects, so we are still close to academia. And we offer an academic discount on all our products 🙂

Ajay- Describe the global efforts in making Rapid Miner a truly international software including spread of developers, clients and employees.

Simon: Our clients already are very international. We have a partner network in America, Asia, and Australia, and, while I am responding to these questions, we have a training course in the US. Developers working on the core of RapidMiner and RapidAnalytics, however, are likely to stay in Germany for the foreseeable future. We need specialists for that, and it would be pointless to spread the development team over the globe. That is also owed to the agile philosophy that we are following.

Ingo: Simon is right, Rapid-I already is acting on an international level. Rapid-I now has more than 300 customers from 39 countries in the world which is a great result for a young company like ours. We are of course very strong in Germany and also the rest of Europe, but also concentrate on more countries by means of our very successful partner network. Rapid-I continues to build this partner network and to recruit dynamic and knowledgeable partners and in the future. However, extending and acting globally is definitely part of our strategic roadmap.

Biography–

Dr. Ingo Mierswa is working as Chief Executive Officer (CEO) of Rapid-I. He has several years of experience in project management, human resources management, consulting, and leadership including eight years of coordinating and leading the multi-national RapidMiner developer team with about 30 developers and contributors world-wide. He wrote his Phd titled “Non-Convex and Multi-Objective Optimization for Numerical Feature Engineering and Data Mining” at the University of Dortmund under the supervision of Prof. Morik.

Dr. Simon Fischer is heading the research & development at Rapid-I. His interests include game theory and networks, the theory of evolutionary algorithms (e.g. on the Ising model), and theoretical and practical aspects of data mining. He wrote his PhD in Aachen where he worked in the project “Design and Analysis of Self-Regulating Protocols for Spectrum Assignment” within the excellence cluster UMIC. Before, he was working on the vtraffic project within the DFG Programme 1126 “Algorithms for large and complex networks”.

http://rapid-i.com/content/view/181/190/ tells you more on the various types of Rapid Miner licensing for enterprise, individual and developer versions.

(Note from Ajay- to receive an early edition invite to Radoop, click here http://radoop.eu/z1sxe)

 

Credit Downgrade of USA and Triple A Whining

As a person trained , deployed and often asked to comment on macroeconomic shenanigans- I have the following observations to make on the downgrade of US Debt by S&P

1) Credit rating is both a mathematical exercise of debt versus net worth as well as intention to repay. Given the recent deadlock in United States legislature on debt ceiling, it is natural and correct to assume that holding US debt is slightly more risky in 2011 as compared to 2001. That means if the US debt was AAA in 2001 it sure is slightly more risky in 2011.

2) Politicians are criticized the world over in democracies including India, UK and US. This is natural , healthy and enforced by checks and balances by constitution of each country. At the time of writing this, there are protests in India on corruption, in UK on economic disparities, in US on debt vs tax vs spending, Israel on inflation. It is the maturity of the media as well as average educational level of citizenry that amplifies and inflames or dampens sentiment regarding policy and business.

3) Conspicuous consumption has failed both at an environmental and economic level. Cheap debt to buy things you do not need may have made good macro economic sense as long as the things were made by people locally but that is no longer the case. Outsourcing is not all evil, but it sure is not a perfect solution to economics and competitiveness. Outsourcing is good or outsourcing is bad- well it depends.

4) In 1944 , the US took debt to fight Nazism, build atomic power and generally wage a lot of war and lots of dual use inventions. In 2004-2010 the US took debt to fight wars in Iraq, Afghanistan and bail out banks and automobile companies. Some erosion in the values represented by a free democracy has taken place, much to the delight of authoritarian regimes (who have managed to survive Google and Facebook).

5) A Double A rating is still quite a good rating. Noone is moving out of the US Treasuries- I mean seriously what are your alternative financial resources to park your government or central bank assets, euro, gold, oil, rare earth futures, metals or yen??

6) Income disparity as a trigger for social unrest in UK, France and other parts is an ominous looming threat that may lead to more action than the poor maths of S &P. It has been some time since riots occured in the United States and I believe in time series and cycles especially given the rising Gini coefficients .

Gini indices for the United States at various times, according to the US Census Bureau:[8][9][10]

  • 1929: 45.0 (estimated)
  • 1947: 37.6 (estimated)
  • 1967: 39.7 (first year reported)
  • 1968: 38.6 (lowest index reported)
  • 1970: 39.4
  • 1980: 40.3
  • 1990: 42.8
    • (Recalculations made in 1992 added a significant upward shift for later values)
  • 2000: 46.2
  • 2005: 46.9
  • 2006: 47.0 (highest index reported)
  • 2007: 46.3
  • 2008: 46.69
  • 2009: 46.8

7) Again I am slightly suspicious of an American Corporation downgrading the American Governmental debt when it failed to reconcile numbers by 2 trillion and famously managed to avoid downgrading Lehman Brothers.  What are the political affiliations of the S &P board. What are their backgrounds. Check the facts, Watson.

The Chinese government should be concerned if it is holding >1000 tonnes of Gold and >1 trillion plus of US treasuries lest we have a third opium war (as either Gold or US Treasuries will burst)

. Opium in 1850 like the US Treasuries in 2010 have no inherent value except for those addicted to them.

8   ) Ron Paul and Paul Krugman are the two extremes of economic ideology in the US.

Reminds me of the old saying- Robbing Peter to pay Paul. Both the Pauls seem equally unhappy and biased.

I have to read both WSJ and NYT to make sense of what actually is happening in the US as opinionated journalism has managed to elbow out fact based journalism. Do we need analytics in journalism education/ reporting?

9) Panic buying and selling would lead to short term arbitrage positions. People like W Buffet made more money in the crash of 2008 than people did in the boom years of 2006-7

If stocks are cheap- buy. on the dips. Acquire companies before they go for IPOs. Go buy your own stock if you are sitting on  a pile of cash. Buy some technology patents in cloud , mobile, tablet and statistical computing if you have a lot of cash and need to buy some long term assets.

10) Follow all advice above at own risk and no liability to this author 😉

 

US-CERT Incident Reporting System

Here are some resources if your cyber resources have been breached. Note the form doesnot use CAPTCHA at all

US-CERT Incident Reporting System (their head Randy Vickers quit last week)

https://forms.us-cert.gov/report/

Using the US-CERT Incident Reporting SystemIn order for us to respond appropriately, please answer the questions as completely and accurately as possible. Questions that must be answered are labeled “Required”. As always, we will protect your sensitive information. This web site uses Secure Sockets Layer (SSL) to provide secure communications. Your browser must allow at least 40-bit encryption. This method of communication is much more secure than unencrypted email.  Continue reading “US-CERT Incident Reporting System”

Best of Google Plus-Week 2-Top 1/0

Stuff I like from week  2 of Google Plus meme- animated GIFS,jokes,nice photos  are just some of them-

Here is week 1 in case you missed it

https://decisionstats.com/best-of-google-plus-week-1-top10/

 

Continue reading “Best of Google Plus-Week 2-Top 1/0”

Cyber Attacks-Protecting your assets and people from cyber attacks

Cyber Attacks-Protecting your assets and people from cyber attacks

Everyday we hear of new cyber attacks on organizations and countries. The latest attacks were on IMF and 200,000 accounts of Citibank and now the website of the US Senate. If some of the most powerful and technologically advanced organizations could not survive targeted attacks, how effective is your organization in handling cyber security. Sony Playstation, Google Gmail, PBS website are other famous targets that have been victimized.

Before we play the blame game by pointing to China for sponsoring hacker attacks, or Russian spammers for creating Bot Nets or ex Silicon Valley /American technology experts rendered jobless by off-shoring, we need to both understand which companies are most vulnerable, which processes need to be fine tuned and what is the plan of action in case your cyber security is breached.

Which companies are most vulnerable?

If you have valuable data, confidential in nature , in electronic form AND connectivity to internet, you have an opening. Think of data as water, if you have a small leakage all the water can be leaked away. To add to complexity, the attackers are mostly unknown, and extremely difficult to catch, and can take a big chunk of your credibility and intellectual property in a very short time.

The best people in technology are not the ones attending meetings in nicely pressed suits– and your IT guy is rarely a match for the talent that is now available on freelance hire for cyber corporate espionage.

Any company or organization that has not undergone through one real time simulated cyber attack or IT audit that focuses on data security is very vulnerable.

Which organizational processes need to be fine tuned ?
Clearly employee access even at senior management needs to be ensured for both technological as well as social vulnerability. Does your reception take the name of senior management if cold called. Do your senior managers surf the internet and use a simple password on the same computer and laptop. Do you have disaster management and redundancy plans.
A wall is only as strong as its weakest brick and the same is true of organizational readiness for cyber attacks.

What is the plan of action in case your cyber security is breached?
Lean back, close your eyes and think your website has just been breached, someone has just stolen confidential emails from your corporate email server, and complete client as well as the most confidential data in your organization has been lost.

Do you have a plan for what to do next? Or are you waiting for an actual cyber event to occur to make that plan.

Top 25 Errors in Programming that lead to hacker attacks

I am elaborating an earlier article on https://decisionstats.com/top-25-most-dangerous-software-errors/ based on my continued research into cyber conflict and strategy. My inputs are in italics – the rest is a condensed article for further thought.

This is thus a very useful initiative for the world to follow and upgrade their cyber security.

It is in accordance with the US policy to secure its cyber infrastructure (http://www.whitehouse.gov/the-press-office/remarks-president-securing-our-nations-cyber-infrastructure)  and countries like India, and even Europe as well as other nations could do well to atleast benchmark their own security practices in software and digital infrastructure with it. There seems to much better technical coordination between rogue hackers than patriotic hackers imho 😉


The Department of Homeland Security of the United States of America has just launched a list of top 25 errors in programming or creating software that increase vulnerability to hacking attacks. The list which is available at http://cwe.mitre.org/top25/index.html lists down a methodology fo measuring vulnerability called Common Weakness Scoring System (CWSS) and uses that score to rank the various errors as well as suggestions to eliminate these weaknesses or errors.
Measuring Weaknesses

The importance of a weakness (that arises due to software bugs) may vary depending on business usage or project implementation, the technologies , operating systems and computing environments in use, and the risk or threat perception.The Common Weakness Scoring System (CWSS) provides a mechanism for scoring weaknesses. and provides a framework for prioritizing security errors (“weaknesses”) that are discovered in software applications.
Identifying Weaknesses
For example the number 1 weakness is shown with
1CWE-89: Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’).
The rest of the weaknesses are

RANK SCORE ID NAME
[1] 93.8 CWE-89 Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’)
[2] 83.3 CWE-78 Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’)
[3] 79.0 CWE-120 Buffer Copy without Checking Size of Input (‘Classic Buffer Overflow’)
[4] 77.7 CWE-79 Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’)
[5] 76.9 CWE-306 Missing Authentication for Critical Function
[6] 76.8 CWE-862 Missing Authorization
[7] 75.0 CWE-798 Use of Hard-coded Credentials
[8] 75.0 CWE-311 Missing Encryption of Sensitive Data
[9] 74.0 CWE-434 Unrestricted Upload of File with Dangerous Type
[10] 73.8 CWE-807 Reliance on Untrusted Inputs in a Security Decision
[11] 73.1 CWE-250 Execution with Unnecessary Privileges
[12] 70.1 CWE-352 Cross-Site Request Forgery (CSRF)
[13] 69.3 CWE-22 Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’)
[14] 68.5 CWE-494 Download of Code Without Integrity Check
[15] 67.8 CWE-863 Incorrect Authorization
[16] 66.0 CWE-829 Inclusion of Functionality from Untrusted Control Sphere
[17] 65.5 CWE-732 Incorrect Permission Assignment for Critical Resource
[18] 64.6 CWE-676 Use of Potentially Dangerous Function
[19] 64.1 CWE-327 Use of a Broken or Risky Cryptographic Algorithm
[20] 62.4 CWE-131 Incorrect Calculation of Buffer Size
[21] 61.5 CWE-307 Improper Restriction of Excessive Authentication Attempts
[22] 61.1 CWE-601 URL Redirection to Untrusted Site (‘Open Redirect’)
[23] 61.0 CWE-134 Uncontrolled Format String
[24] 60.3 CWE-190 Integer Overflow or Wraparound
[25] 59.9 CWE-759 Use of a One-Way Hash without a Salt
Details of each weakness is given by http://cwe.mitre.org/top25/index.html#Details
It includes Summary , Weakness Prevalence, Consequences, Remediation Cost, Ease of Detection ,Attacker Awareness and Attack Frequency .In addition the following sections describe each software vulnerability in detail- Technical Details ,Code Examples ,Detection Methods ,References,Prevention and Mitigation, Related CWEs and Related Attack Patterns.
Other important software weaknesses are –

[26] CWE-770: Allocation of Resources Without Limits or Throttling
[27] CWE-129: Improper Validation of Array Index
[28] CWE-754: Improper Check for Unusual or Exceptional Conditions
[29] CWE-805: Buffer Access with Incorrect Length Value
[30] CWE-838: Inappropriate Encoding for Output Context
[31] CWE-330: Use of Insufficiently Random Values
[32] CWE-822: Untrusted Pointer Dereference
[33] CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization (‘Race Condition’)
[34] CWE-212: Improper Cross-boundary Removal of Sensitive Data
[35] CWE-681: Incorrect Conversion between Numeric Types
[36] CWE-476: NULL Pointer Dereference
[37] CWE-841: Improper Enforcement of Behavioral Workflow
[38] CWE-772: Missing Release of Resource after Effective Lifetime
[39] CWE-209: Information Exposure Through an Error Message
[40] CWE-825: Expired Pointer Dereference
[41] CWE-456: Missing Initialization
Mitigating Weaknesses
Here is an example of the new matrix for migrations that also list the top 25 errors . This thus shows a way to fix the weaknesses and relative impact on each weakness by the following mitigations.
http://cwe.mitre.org/top25/mitigations.html#MitigationMatrix

Effectiveness ratings include:

  • High: The mitigation has well-known, well-understood strengths and limitations; there is good coverage with respect to variations of the weakness.
  • Moderate: The mitigation will prevent the weakness in multiple forms, but it does not have complete coverage of the weakness.
  • Limited: The mitigation may be useful in limited circumstances, only be applicable to a subset of this weakness type, require extensive training/customization, or give limited visibility.
  • Defense in Depth (DiD): The mitigation may not necessarily prevent the weakness, but it may help to minimize the potential impact when an attacker exploits the weakness.

Within the matrix, the following mitigations are identified:

 

  • M1: Establish and maintain control over all of your inputs.
  • M2: Establish and maintain control over all of your outputs.
  • M3: Lock down your environment.
  • M4: Assume that external components can be subverted, and your code can be read by anyone.
  • M5: Use industry-accepted security features instead of inventing your own.

The following general practices are omitted from the matrix:

  • GP1: Use libraries and frameworks that make it easier to avoid introducing weaknesses.
  • GP2: Integrate security into the entire software development lifecycle.
  • GP3: Use a broad mix of methods to comprehensively find and prevent weaknesses.
  • GP4: Allow locked-down clients to interact with your software.

 

M1 M2 M3 M4 M5 CWE
High DiD Mod CWE-22: Improper Limitation of a Pathname to a Restricted Directory (‘Path Traversal’)
Mod High DiD Ltd CWE-78: Improper Neutralization of Special Elements used in an OS Command (‘OS Command Injection’)
Mod High Ltd CWE-79: Improper Neutralization of Input During Web Page Generation (‘Cross-site Scripting’)
Mod High DiD Ltd CWE-89: Improper Neutralization of Special Elements used in an SQL Command (‘SQL Injection’)
Mod DiD Ltd CWE-120: Buffer Copy without Checking Size of Input (‘Classic Buffer Overflow’)
Mod DiD Ltd CWE-131: Incorrect Calculation of Buffer Size
High DiD Mod CWE-134: Uncontrolled Format String
Mod DiD Ltd CWE-190: Integer Overflow or Wraparound
High CWE-250: Execution with Unnecessary Privileges
Mod Mod CWE-306: Missing Authentication for Critical Function
Mod CWE-307: Improper Restriction of Excessive Authentication Attempts
DiD CWE-311: Missing Encryption of Sensitive Data
High CWE-327: Use of a Broken or Risky Cryptographic Algorithm
Ltd CWE-352: Cross-Site Request Forgery (CSRF)
Mod DiD Mod CWE-434: Unrestricted Upload of File with Dangerous Type
DiD CWE-494: Download of Code Without Integrity Check
Mod Mod Ltd CWE-601: URL Redirection to Untrusted Site (‘Open Redirect’)
Mod High DiD CWE-676: Use of Potentially Dangerous Function
Ltd DiD Mod CWE-732: Incorrect Permission Assignment for Critical Resource
High CWE-759: Use of a One-Way Hash without a Salt
DiD High Mod CWE-798: Use of Hard-coded Credentials
Mod DiD Mod Mod CWE-807: Reliance on Untrusted Inputs in a Security Decision
High High High CWE-829: Inclusion of Functionality from Untrusted Control Sphere
DiD Mod Mod CWE-862: Missing Authorization
DiD Mod CWE-863: Incorrect Authorization