How does cryptography work?

How does cryptography work?

by Jeroen Ooms

This page attempts to give a very basic conceptual introduction to cryptographic methods. Before we start the usual disclaimer:

I am not a cryptographer. This document is only for educational purposes. Crypto is hard, you should never trust your home-grown implementation. Unless you’re a cryptographer you will probably overlook some crucial details. Developers should only use the high-level functions that have been implemented by an actual cryptographer.

Now that we got this is out of the way, let’s start hacking :)

The XOR operator

The logical XOR operator outputs true only when both inputs differ (one is true, the other is false). It is sometimes called an invertor because the output of x gets inverted if and only if y is true:

# XOR two (8bit) bytes 'x' and 'y'
x <- as.raw(0x7a)
y <- as.raw(0xe4)
z <- base::xor(x, y)
# Show the bits in each byte
cbind(x = rawToBits(x), y = rawToBits(y), z = rawToBits(z))
      x  y  z
[1,] 00 00 00
[2,] 01 00 01
[3,] 00 01 01
[4,] 01 00 01
[5,] 01 00 01
[6,] 01 01 00
[7,] 01 01 00
[8,] 00 01 01

In cryptography we xor a message x with secret random data y. Because each bit in y is randomly true with probability 0.5, the xor output is completely random and uncorrelated to x. This is called perfect secrecy. Only if we know y we can decipher the message x.

# Encrypt message using random one-time-pad
msg <- charToRaw("TTIP is evil")
one_time_pad <- random(length(msg))
ciphertext <- base::xor(msg, one_time_pad)

# It's really encrypted
[1] "(8\xd7ȉ%\u035f\x81\xbb\023\xa2"
# Decrypt with same pad
rawToChar(base::xor(ciphertext, one_time_pad))
[1] "TTIP is evil"

This method is perfectly secure and forms the basis for most cryptograhpic methods. However the challenge is generating and communicating unique pseudo-random y data every time we want to encrypt something. One-time-pads as in the example are not very practical for large messages. Also we should never re-use a one-time-pad y for encrypting multiple messages, as this compromises the secrecy.

Stream ciphers

A stream cipher generates a unique stream of pseudo-random data based on a secret key and a unique nonce. For a given set of parameters the stream cipher always generates the same stream of data. Sodium implements a few popular stream ciphers:

password <- "My secret passphrase"
key <- hash(charToRaw(password))
nonce <- random(8)
chacha20(size = 20, key, nonce)
 [1] 51 c6 c9 45 c6 13 6b 3d 6f 5c e3 ab 9f 16 f2 46 ce cb 19 f3

Each stream requires a key and a nonce. The key forms the shared secret and should only be known to trusted parties. The nonce is not secret and is stored or sent along with the ciphertext. The purpose of the nonce is to make a random stream unique to protect gainst re-use attacks. This way you can re-use a your key to encrypt multiple messages, as long as you never re-use the same nonce.

salsa20(size = 20, key, nonce)
 [1] df 7d 13 ca ea 7c ff 93 e5 b6 fe b6 6b e2 91 14 ed ae 17 eb

Over the years cryptographers have come up with many more variants. Many stream ciphers are based on a block cipher such as AES: a keyed permutation of fixed length amount of data. The block ciphers get chained in a particular mode of operation which repeatedly applies the cipher’s single-block operation to securely transform amounts of data larger than a block.

We are not going to discuss implementation details, but you could probably come up with something yourself. For example you could use a hash function such sha256 as the block cipher and append counter which is incremented for each block (this is called CTR mode).

# Illustrative example.
sha256_ctr <- function(size, key, nonce){
  n <- ceiling(size/32)
  output <- raw()
  for(i in 1:n){
    counter <- packBits(intToBits(i))
    block <- sha256(c(key, nonce, counter))
    output <- c(output, block)

This allows us to generate an arbitrary length stream from a single secret key:

password <- "My secret passphrase"
key <- hash(charToRaw(password))
nonce <- random(8)
sha256_ctr(50, key, nonce)
 [1] 07 01 96 02 7e c7 37 b4 8c b1 6a ec 4e 2d 56 34 7d 39 13 bc 72 e0 19
[24] ad b3 44 0e 9f 88 bb 3d 26 94 aa 66 01 2e bd 46 55 2c 04 99 1e af a9
[47] 91 cd 53 b4

In practice, you should never write your own ciphers. A lot of research goes into studying the properties of block ciphers under various modes of operation. In the remainder we just use the standard Sodium ciphers: chacha20, salsa20, xsalsa20 or aes128. See sodium documentation for details.

Symmetric encryption

Symmetric encryption means that the same secret key is used for both encryption and decryption. All that is needed to implement symmetric encryption is xor and a stream cipher. For example to encrypt an arbitrary length message using password:

# Encrypt 'message' using 'password'
myfile <- file.path(R.home(), "COPYING")
message <- readBin(myfile, raw(),$size)
passwd <- charToRaw("My secret passphrase")

A hash function converts the password to a key of suitable size for the stream cipher, which we use to generate a psuedo random stream of equal length to the message:

# Basic secret key encryption
key <- hash(passwd)
nonce8 <- random(8)
stream <- chacha20(length(message), key, nonce8)
ciphertext <- base::xor(stream, message)

Now the ciphertext is an encrypted version of the message. Only those that know the key and the nonce can re-generate the same keystream in order to xor the ciphertext back into the original message.

# Decrypt with the same key
key <- hash(charToRaw("My secret passphrase"))
stream <- chacha20(length(ciphertext), key, nonce8)
out <- base::xor(ciphertext, stream)

# Print part of the message
cat(substring(rawToChar(out), 1, 120))
               Version 2, June 1991

 Copyright (C) 1989, 1991 Free Software Foundation, Inc.

The Sodium functions data_encrypt and data_decrypt provide a more elaborate implementation of the above. This is what you should use in practice for secret key encryption.

Symmetric encryption can be used for e.g. encrypting local data. However because the same secret is used for both encryption and decryption, it is impractical for communication with other parties. For exchanging secure messages we need public key encryption.

Public-key encryption and Diffie-Hellman

Rather than using a single secret-key, assymetric (public key) encryption requires a keypair, consisting of a public key for encryption and a private-key for decryption. Data that is encrypted using a given public key can only be decrypted using the corresponding private key.

The public key is not confidential and can be shared on e.g. a website or keyserver. This allows anyone to send somebody a secure message by encrypting it with the receivers public key. The encrypted message will only be readable by the owner of the corresponding private key.

# Create keypair
key <- keygen()
pub <- pubkey(key)

# Encrypt message for receiver using his/her public key
msg <- serialize(iris, NULL)
ciphertext <- simple_encrypt(msg, pub)

# Receiver decrypts with his/her private key
out <- simple_decrypt(ciphertext, key)
identical(msg, out)
[1] TRUE

How does this work? Public key encryption makes use of Diffie-Hellman (D-H): a method which allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure channel. In the most simple case, both parties generate a temporary keypair and exchange their public key over the insecure channel. Then both parties use the D-H function to calculcate the (same) shared secret key by combining their own private key with the other person’s public key:

# Bob generates keypair
bob_key <- keygen()
bob_pubkey <- pubkey(bob_key)

# Alice generates keypair
alice_key <- keygen()
alice_pubkey <- pubkey(alice_key)

# After Bob and Alice exchange pubkey they can both derive the secret
alice_secret <- diffie_hellman(alice_key, bob_pubkey)
bob_secret <- diffie_hellman(bob_key, alice_pubkey)
identical(alice_secret, bob_secret)
[1] TRUE

Once the shared secret has been established, both parties can discard their temporary public/private key and use the shared secret to start encrypting communications with symmetric encryption as discussed earlier. Because the shared secret cannot be calculated using only the public keys, the process is safe from eavesdroppers.

The classical Diffie-Hellman method is based on the discrete logarithm problem with large prime numbers. Sodium uses curve25519, a state-of-the-art D-H function by Daniel Bernsteinan designed for use with the elliptic curve Diffie–Hellman (ECDH) key agreement scheme.



(Ajay- I really liked this very nice tutorial on cryptography and hope it helps bring more people in the debate. This is just to share this very excellent vignette based on the Sodium package in R)


I read a chapter from How to Win Friends and Influence People as part of my Holiday reading. It is a remarkably well written book and I am trying to summarize a few key early learnings.

  1. Use lucid examples that people can relate to while writing a book.
  2. Base a book based on what works or does not work in real life.
  3. Do not criticize people (Chapter 1)

Since I criticize a lot, that is my new year resolution. To stop changing other people by criticism.

I also started re-reading from one of my favorite authors. Hemigway lived, died and wrote by a code of his own. Some learnings from him

  1. Keep words simple and sentences short
  2. Write a lot
  3. Be passionate
  4. Be honorable

Honor and self respect seems to be the underlying code for Hemingway.

To cap off , I watched this documentary Code I was really horrified how we hackers have been so busy trying to change the world we forgot to address some issues in the hacker culture

  1. We need more ethnic diversity
  2. We need more gender diversity
  3. Diversity brings better creative mix and stable teams

In addition I learnt that balancing funding with creative creation is essential to survival. Well funded creative projects will be better produced than less funded. What is shown more, sells more. (Jo Dikhta hain woh bikta hain)

Well thats all the code. But yes the movie convinced me to try and lift a  finger to help bring more women and African-Hispanic coders in my small way. I hope you try something like that too.

How geeks can help defeat terrorists part 1

There is a lot of money in defeating terrorism by using analytics, much more than in internet ads alone.

I personally don’t think you can make the world safer by collecting data from my Facebook Twitter WordPress or Gmail account- indeed precious resources are diverted into signal intelligence (sigint) when they could have been used in human intelligence ( humint). But signal intelligence interests the lobbyists of the military-industrial complex more than humint does and post Manning and Snowden it would be questionable to increase the number of analysts without a thorough screening. There are no easy solutions to this unfortunately as the  attacks in Paris and California show the limits of signal intelligence.

If you collect data from Internet in bulk- the terrorists will adapt. Now that everyone terrorist and civilian knows data is being collected. Surprise as a key element ahs been lost due to Snowden.

Perhaps of greater utility is for linking databases of law enforcement across the world, and better  interfaces for querying and analyzing huge data  and automated alerts for reminding global law enforcement when they fail to follow up. Better analytics is needed, not more data. That’s just old news and lazy data analytics.

Interface design is key to solving Big Data. I have a huge pile of intelligence reports to read as a decision maker. What kind of data visualisation extracts signal from noise and gives it in a timely automated manner.  They are all just documents and documents.

Thats a Big problem to solve in Big Data Analytics.

Note from Ajay – These are the author’s personal views.



India used to be a Superpower but we declined. China was a superpower then it declined. So did Britain. So did Soviet Russia. The United States remains the aging Rocky Balboa of the superpowers, but you can see some decline in influence compared to when Clinton was President.

What do superpowers do?

  • They invest a lot of money in arms and defence
  • They earn a lot of money from trade so they can invest it in arms
  • They put their own interest ahead the interest of their neighbours and competitors
  • They pretend to go to war if you hurt a single citizen, but they themselves do not do much when thousands of their citizens are mal-treated by pollution, by exploitative working conditions, by small arms and guns, by crime, by inequality

Ultimately I think Switzerland is the only superpower. Their superpower lies in not pretending to be super at all.

During trade and now climate negotiations, the past and the present and the future superpowers collide. The needs of the many are more important than the egos of a few politicians , the brilliance of their advisers and the theatrics of a few.

Does the planet need a CEO? Probably yes, and the United Nations has failed to be a superpower or any power at all. It is just a conference holding organization.

The greatest generation that won Word War 2 in the West and defeated Colonialism in the East was succeeded by the Baby Boomer generation that just boomed and consumed. The next generation will pay the price of the past few generations. The country that has the best care of the next generation for a healthy productive workforce for both economic and defence deployment will win the race to be the Superbpower. Thats not a typo. Stop being a superpower and start being a superb power.

In the meantime, I would rather see Matt Damon colonize Mars and Rocky Balbao teach boxing to the nest generation.

The planet is fine. The polar bears are fucked.

What has George Carlin got to do with Climate Change.  Well apparently plenty of vision came from Carlin. Those days we used to call Climate Change as Global Warming. The Ozone layer was a separate problem.

To hack climate change you may want to hack the way people think. The way people think is the way the climate ended up changing itself.

See this at NYTIMES Greeland is melting away. “Every scientist, Dr. Smith said, is keenly aware that the research costs “a tremendous amount of taxpayer money.”


Rather the spend a lot of money to exotic places in Greenland to collect data, maybe we can use the Internet of Things to just keep some chips to collect data and transmit it via the Internet.


Newer forms of climate change data might need newer forms of mathematics rather than brute forcing older mathematical techniques. It might need newer languages in terms of computer software. There is a lot of carbon locked up in ice and it will increase exponentially not linearly. We dont even have a proper way to forecast yet.

Rising temperatures would also mean newer species maybe at a smaller microbial or bacterial level. There will be change. Summer is coming. or maybe a newer Ice Age is coming. Who knows. We were too busy spending money on Christmas trees. We forgot to do the math.

The planet will be fine. The polar bears are fucked. The people … the people… Oh the people. (or read it here)


You got people like this around you? Country’s full of ’em now. People walkin’ around all day long every minute of the day, worried about everything. Worried about the air, worried about the water, worried about the soil. Worried about insecticides, pesticides, food additives, carcinogens, worried about radon gas, worried about asbestos, worried about saving endangered species.

Lemme tell ya bout endangered species, awright? Saving endangered species is just one more arrogant attempt by humans to control Nature. It’s arrogant meddling. It’s what got us in trouble in the first place. Doesn’t anybody understand that? Interfering with Nature. Over 90 percent, over, way over 90 percent, of the species that have ever lived on this planet, ever lived, are gone. Wooosh! They’re extinct. We didn’t kill them all. They just disappeared. That’s what nature does. They disappear these days at the rate of 25 a day—and I mean regardless of our behavior. Irrespective of how we act on this planet, 25 species that were here today will be gone tomorrow. Let them go gracefully. Leave Nature alone. Haven’t we done enough? We’re so self-important, so self-important. Everybody’s gonna save something now. Save the trees, save the bees, save the whales, save those snails. And the greatest arrogance of all, save the planet. What? Are these fucking people kidding me? Save the planet? We don’t even know how to take care of ourselves yet. We haven’t learned to care for one another—we’re gonna save the fuckin’ planet? I’m gettin’ tired of that shit. Tired of that shit. Tired.

I’m tired of fuckin’ Earth Day, I’m tired of these self-righteous environmentalists, these white bourgeoise liberals who think the only thing wrong with this country is there aren’t enough bicycle paths. People trying to make the world safe for their Volvos. Besides, environmentalist don’t give a shit about the planet, they don’t care about the planet, not in the abstract they don’t, not in the abstract they don’t. You know what they’re interested in? A clean place to live. Their own habitat. They’re worried that someday in the future they might be personally inconvenienced. Narrow, unenlightened self-interest doesn’t impress me. Besides, there is nothing wrong with the planet, nothing wrong with the planet. The planet is fine. The people are fucked. Difference. Difference. The planet is fine. Compared to the people, the planet is doin’ great! It’s been here four and a half billion years. Did you ever think about the arithmetic? The planet has been here four and a half billion years. We’ve been here, what? A hundred thousand? Maybe two hundred thousand and we’ve only been engaged in heavy industry for a little over two hundred years. Two hundred years versus four and a half billion. And we have the conceit to think that somehow we’re a threat? That somehow we’re gonna put in jeopardy this beautiful little blue-green ball that’s just a floatin’ around the sun? The planet has been through a lot worse than us. Been through all kinds of things worse than us. Been through earthquakes, volcanoes, plate tectonics, continental drift, solar flares, sunspots, magnetic storms, the magnetic reversal of the poles, hundreds of thousands of years of bombardment by comets and asteroids, and meteors, world-wide floods, tidal waves, world-wide fires, erosion, cosmic rays, recurring ice ages, and we think some plastic bags and some aluminum cans are going to make a difference?

The planet isn’t going anywhere. We are! We’re goin’ away. Pack your shit, Folks, we’re goin’ away. We won’t leave much of a trace either, thank god for that. Maybe a little styrofoam, maybe, little styrofoam. Planet’ll be here and we’ll be long gone. Just another failed mutation. Just another closed-end biological mistake, an evolutionary cul de sac. The planet will shake us off like a bad case of fleas, a surface nuisance. You wanna know how the planet’s doin’? Ask those people at Pompeii, who were frozen into position from volcanic ash. How the planet’s doin’. Wanna know if the planet’s alright, ask those people in Mexico City or Armenia, or a hundred other places buried under thousands of tons of earthquake rubble if they feel like a threat to the planet this week. How about those people in Kilauea, Hawaii who built their homes right next to an active volcano and then wonder why they have lava in the living room. The planet will be here for a long, long, long time after we’re gone and it will heal itself, it will cleanse itself ’cuz that’s what it does. It’s a self-correcting system. The air and the water will recover, the earth will be renewed, and if it’s true that plastic is not degradable well, the planet will simply incorporate plastic into a new paradigm: the earth plus plastic. The earth doesn’t share our prejudice towards plastic. Plastic came out of the earth. The earth probably sees plastic as just another one of its children. Could be the only reason the earth allows us to be spawned from it in the first place: it wanted plastic for itself. Didn’t know how to make it, needed us. Could be the answer to our age-old philosophical question, “Why are we here?” “Plastic, assholes.”

So, so, the plastic is here, our job is done, we can be phased out now. And I think that’s really started already, don’t you? I mean, to be fair, the planet probably sees us as a mild threat, something to be dealt with, but I’m sure the planet will defend itself in the manner of a large organism like a bee hive or an ant colony can muster a defense. I’m sure the planet will think of something. What would you do, if you were the planet trying to defend against this pesky, troublesome species? Let’s see, what might, viruses, viruses might be good, they seem vulnerable to viruses. And, viruses are tricky, always mutating and forming new strains whenever a vaccine is developed. Perhaps this first virus could be one that compromises the immune system in these creatures. Perhaps a human immuno deficiency virus making them vulnerable to all sorts of other diseases and infections that might come along, and maybe it could be spread sexually, making them a little reluctant to engage in the act of reproduction.

Well, that’s a poetic note. And it’s a start. But I can dream, can’t I? I don’t worry about the little things, bees, trees, whales, snails. I think we’re part of a greater wisdom than we’ll ever understand, a higher order, call it what you want. You know what I call it? The Big Electron. The Big Electron. Woooohhhh, woooohhhh, woooohhhh. It doesn’t punish, it doesn’t reward, it doesn’t judge at all. It just is, and so are we, for a little while. Thanks for being here with me for a little while tonight.

Change the Mirror

Things I can change
Rate of Change
Things I can change slowly Things I can change quicker
Executing Change Training Build Buy
Need for Change Ambition Money Passion
Things I cant change
Things I need not change Things that are too expensive for me to change
Executing Non- Change Ignore the thing to be changed Outsource the task to be changed

Heaven is a place on Mars

A few years ago, on a flight, in a land far far away, I was asked by a another person, What do you think we should do. I said, lets go to Mars.Why is that? the man asked. Because, I paused and said, this planet is going to run out of resources.

Perhaps that is what we should tell the Russians and Chinese and the Anglo-Saxons to do. Kill ISIS together, stop fighting over the Arctic, and reignite the battle to make empires but on Mars.

One more thing, climate change is the 100 billion dollar opportunity. Any startups willing to hack climate change?

Think of the market opportunities, yall.