Some Thoughts

Graduate student life

2012-02-09T01:52:00.000-08:00

I get asked many times "What was it like doing a PhD?". Initially I didn't have a good answer, but after coming out of academia for a few years I have a better idea. There are many thoughts around how adults lose the childlike wonder about learning new things. That adults just go from one task that they have to do to the next task that they have to do. Never stopping to learn and wonder about the world they live in. Working in the industry is sort of like that. There are business goals and after we solve the number one problem, the number two problem becomes the number one problem and we repeat the process. This can get quickly boring. While I was doing my PhD, I had the freedom to learn anything that interested me. I think this was one of the most important things that distinguishes what I do now from what I did then. There were no barriers to learning. Sure, I had a goal in mind and I had to produce papers and results but even then, the requirements were to produce something new and not necessarily practical. Learning for the fun of it is really enjoyable. Whether the learning comes from reading books or taking classes or watching videos or visiting people and places.It feels good to build something and apply all the learnings, but is always feels good to keep learning.

Personalized Education

2011-04-11T02:29:00.000-07:00

Everyone always complains that not much teaching happens in classrooms. And that the only thing that increases learning is individual attention. Standing on the stage and parroting text from notes is not going to help. This is only one of the reasons why teaching at home is better. Kids get individual attention. The other important part is context and material tailored to each student. When a mom teachers her kid, she knows everything the kid knows and the examples she gives are from areas the kid understands. In a classroom, that is not the case. Many of the kids might not be familiar with the exmaples at all. E.g., in school, there was an english lesson (short story) where people are sitting at a table having lunch and one of the characters says "pass the mustard". Each and every word in that sentence is understandable but who on earth eats mustard by itself? In the context of Indian cooking that does't make any sense. Only if you have been to the western world where mustard is a yellow paste that comes in bottles, the setence makes ense. Another e.g., while teaching the shapes of electron's orbitals, the chemistry text said that the d-oribtals look like a dumbell surrounded by a doughnut. Who knows what a doughnut is? The teacher said a doughnut looks like a torus. That made much more sense! Without proper context, half the text is incomprehensible. Also, different kids learn at different paces. Sites such as Khan Academy [1] have made individual learning much easier by allowing the student to set his own pace. But we need more. There needs to be more personalization in education. If a kid has never heard of doughnuts but knows about vada, then the content needs to be modified to use examples from the vocabulary the kid already knows. Replacing words with its synonyms is a good start but translating idioms and other common phrases is much more tricky, even if it is between one style of English to another.

[1] http://www.khanacademy.org/

One year later a.k.a. To the moon and back

2011-04-07T04:49:00.000-07:00

Houston, we have touchdown. It has been over an year since I came back to India and not only did I survive reentry without burning up, I seem to have assimilated back successfully.

I spent six years in the US give or take a few months spent vacationing. When I first got home, the population density took a bit of getting used to. Having stayed alone or with one roommate, I had to get used staying together again. Today, even if one of the family members is out for more than a day, the house feels empty.

I never had major culture shock when I went to US. Any home sickness I had was in reverse. The first few years were fun and as the years progressed, the question of returning to India became a question of when rather than if. Overall, the transition to and from US has been very gradual and smooth.

I am starting to feel that I have never left India. The memories of Purdue and Seattle are still fresher than the memories of say IITM, but when I drive on the road, I am no longer reminded of the smooth roads of I-90 that I used to drive on or the 200kmph driving in Germany.

Another surprising thing is that Bengaluru is becoming more and more like the US itself. I went to see the movie Robot in a multiplex which looked no different than any I visited in US. The brands, the restaurants, the people everything is a smaller scale copy. In some malls, the price is in USD too! Stay in one of the posh apartment complexes and go to the dollar malls and you never have to feel that you are NOT in US.

Neo Indian [1] posted an entry called "20 signs that you’ve successfully made the transition to living in India" [2]. I am already at 16 :) 90% of the transition is complete. Like in software and construction, the other 10% is going to take another 90% of the time :)

[1] http://neoindian.org/ -- highly recommended. Has ton of funny posts and useful information too.
[2] http://neoindian.org/2010/01/07/20-signs-that-youve-successfully-made-the-transition-to-living-in-india/

One year in the US (old post)

2011-04-04T08:21:00.000-07:00

This is an old post from my blog. The old blog doesn't exist anymore and I will be posting a few of the old articles here, along with the comments.

It has been a year since I landed in US on Aug 11, 2003 (11PM). Had attended the
convocation (commencement in US terms) just a couple of days before boarding the
flight. Feel like going home now, but I have got some research to do. Life has
changed. I specifically didn't want to be a software engineer and now all I am
doing is coding. This is much better coding, of course, when compared to the
code ~~monkeys~~ warriors. But its similar though. Every now and
then I get stuck with some obscure bug (which is demoralizing) and after a few
days (which feels like eternity), it disappears as mysteriously as it appeared.
Then a sigh of relief not long lived as another obscure bug surfaces the next
day.

When I was preparing to come to the US, so many people were giving "free"
advice, without themselves having been here and experienced anything. Most of
the advices were just threatenings directed towards me, such as "You (being a
vegetarian) will not be able to eat anything in the US and hence will starve to
death", "You (being under weight) will not be able to carry luggage or travel to
the US" (somehow they were thinking that I will fall off the aircraft on to the
ocean below or get crushed between the wheels of the trolley in the airport).
Many more. Some were directed towards my parents in an attempt to prevent them
from "losing" me : "your son will tell you now that he will come back. But once
he goes there, he will (they were so certain) marry a white girl and never
return home. He will forget you as soon as he goes there" etc. etc. After
telling all of these people to STFU, they were all praising me for going to the
US alone (somehow they thought it was an adventure).

One good thing that has happened after experiencing a different culture is that
it has put a lot (I mean a LOT) of things in perspective. I am able to evaluate
things better than I did before. The way different cultures handle disasters,
public affairs, education and so on are so much more different.

I am getting so many thoughts about how things (especially social science which
includes history, geography, economics and civics) should have been taught in
schools. Prof. M S Ananth (current director of IITM and Professor of Chemical
Engg.) once told in class (while teaching Thermodynamics) that we will go
through the exact same phase. He said that during our stay at IITM, most of us
think that we did not learn anything. But when we enter grad school or a job, we
do feel that we had learnt something and some respect arises for the Professors
who taught the courses. Then we consult the Professors to get our problems
solved and finally, we we are successful, we come back and tell them how to run
the institute. I feel that I should have started writing my journal from the
beginning of my 10th standard. Thats when a lot of "dramatic" changes have been
happening. I keep dreaming of the day when I will revisit my past and record all
the important incidents and PJs, but I don't know when will that happen.

Now that I have spent an year in the US, when I go back to India (hopefully in
December), it will be a very interesting thing to listen to the people who gave
all those free "advices".

Comments:
deepak

machi, nice digging at all ur 'wellwishers' ;-) What these ppl don't understand is how difficult it is to get laid by a white girl, leave alone marry one ;-)

About ur point on gaining perspective - u r bang on target man. I feel exactly the same way.

And do not despair abt nothing interesting happening in ur life since u left IIT. I bet 10 years hence, u will be thinking, 'I should've written more in my journal when I went to the US' :) You just don't notice the good things now. While if you look back, although it's a 3dimensional view, everything in past time gets projected on a single 2d plane - which means all the cool experiences over time appear as if they happened at the same instance .

Book reviews

2011-01-21T21:32:00.000-08:00

I have long since been a fan of science fiction books and some time back I realized that there is a specific term for the type of sci-fi I like. It is called hard science fiction. Thanks to my kindle, I have been able to finish reading four such books. The general thing that fascinates about these books is that all of the things mentioned in the books are actually possible and there are very few assumptions and violations of the laws of physics.

Dragon's Egg by Robert L. Forward

In the near future, humans discover a neutron star and plan a manned expedition to it. Once at the neutron star, the humans realize that there is life on the star and that it is intelligent. We get a fast forwarded look at how a civilization can evolve from cave man era to a sophisticated one in a matter of days and even surpassing human intelligence and capabilities a short while later. All throughout the book, the narration is nice and the descriptive enough to imagine what is going on. The appendix has many helpful pictures which help with the imagination greatly.

Tau Zero by Poul Anderson

This book is about the effects of long term isolation on people, how they react to it, what they do to stay sane etc. The journey of a few dozen people in a bussard jet powered star craft take them very close the speed of light and the time dilation they experience makes them isolated from everyone else.

Mission of Gravity by Hal Clement

Humans send a probe to a planet with very hundreds of g gravity and it gets lost. They enlist the help of the indigenous centipede like life, which is intelligent, to get parts of it back. The book is very scientifically accurate and doesn't require any magic to explain the different phenomena. Regular, everyday physics of gravity becomes an engaging read from low gravity of 3g to a high of 700g at the end.

Between the Strokes of Night by Charles Sheffield

This is a story about people who start out with sleep research and end up examining the universe till its very end. The way time is extended for humans is by going to a state of reduced consciousness similar to hibernation. While the rest of the world goes on at its own speed, people in this new state experience it a thousand times slower, thus extending their life span in to the millenia.

Three out of the four books are about what happens when humans experience time differently from what we are used to. It is interesting that we devote so much time into understanding various possibilities and effects of varied time perception. Well, time passed very quickly when I reading them :)

Intelligent Monkeys of IITM

2010-04-17T21:54:00.000-07:00

Within two days of my stay at Ganga Hostel, IIT Madras, I became aware of the monkey problem. I came back to my room to find it completely ransacked and lacking in all food items painstakingly prepared by mom. I learned that it was the monkeys that did it. Thankfully a small part of the sweets I brought were untouched, so I locked the kit bag and went to class. When I came back, even the food in the locked bag was missing. The monkeys were very clever. They were able to slide the zip and make a little opening between the two zips that were tied together and reach for the food. How did the monkeys get so smart? Well, they had stayed at IIT Madras for many years whereas we had been there only for a few days.

The eventual solution came from my dad who wired the bars in the windows so that the monkeys wouldn't get in.

Publishing shenanigans

2009-09-30T14:48:00.000-07:00

Saw this in the beginning of the online edition of Successful Lisp

NOTICE: As you know, Successful Lisp has been available as a printed edition since 2004. I regret to inform my readers that my publisher has not paid royalties on 2007 and 2008 sales. As the publisher has not responded affirmatively to any of my inquiries regarding royalty payments, I can only assume that he has no intentions of honoring his contractual agreement.

The income from the book is not sufficient to support a response in the courts, considering that such a response would have to be conducted in Germany. (I am in the USA.) As a result, I ask you to consider not buying the printed edition of Successful Lisp. I seem to have have no reasonable recourse to compel the publisher to release past-due royalty payments. However, with your understanding and cooperation I hope to prevent the inequity of this unfortunate situation from becoming worse.

At a time when print is being taken over by digital world, why would publishers do stupid things like this? Perhaps print is going to the cleaners *because* of such stupidity. Time for David Lamkins to try out print on demand.

What if everybody was a vegetarian?

2009-09-21T11:41:00.000-07:00

When I was in school, there was a mandatory discussion on the topic of feeding the increasing population of the world. The teacher made a statement that if everybody was vegetarian, there wouldn't be enough food to feed all the people, so it is necessary for some people to be non vegetarians. I took as the truth. Years later, I learned about food webs and then I realized how wrong that answer was. As the section on flow of food chains explains, only 10% of the energy at any given level is passed on to the next level. Which means that if plants capture 100 joules (or calories if you are american) of energy from the Sun and I eat the plant, I get 10 joules of energy. If on the other hand, if someone (not me, I'm vegetarian) lets an animal eat the plants and eats the animal, they get only 1 joule of energy! Clearly, by everyone becoming vegetarians, there will be more food available to feed the world's population :) I am not saying this will be enough to fill everyone, just that it will be more food than the case where people eat meat. By logical extension, we could process solar energy to generate all our dietary requirements, then there would be even more food available for everyone.

Before people start fighting about the nutritional value (proteins are the primary concern), there is enough of everything in different plants to provide for all dietary requirements. You just have to know what to eat. Read Vegetarianism for more details.

Caveat #1: The above theory assumes that the land that is used to grow food for animals can be used to grow food for humans. If this is not true, then clearly, that land can only be used to feed animals and though limited, provide meat for humans. This is immediately obvious for fisheries. Until people can grow plants in the sea and harvest them, all sea food is essentially coming from land that can't be used to grow plants for human consumption.

Caveat #2: I am not interested in the philosophical question of whether or not it is OK to eat meat grown in a vat. Growing meat in vats still takes the same or more amount of energy (proof needed) and hence the same as meat from killing an animal for the purposes of this discussion.

Three ways to screw up life

2009-09-09T22:51:00.000-07:00

A wise woman once said about three ways that people end up screwing up their lives:

1. Cannot forget about the past.
2. Can't live the present.
3. And can't stop being worried about the future.

Future of humans

2009-08-25T20:00:00.000-07:00

The other day one of my colleagues started talking about something he heard on NPR about how humans might be evolving or de-evolving and asked for my take on it. Poor guy had to listen to ten minutes of me explaining to me this idea I have had for many months now. So instead of inflicting my held up ideas on some unsuspecting people, I am going to inflict it upon the readers of this blog.

Humans have constantly evolved to have their knowledge for survival learned during the course of their childhood and later life rather than something pre programmed. This is both good and bad. This is good because this allows us to adapt faster to a changing environment. We are no longer bound by evolution's slow rate to adapt to a new environment. Instead, we program our brain to do things we see fit, such as eating donuts and are thus able to learn what is immediately needed, such as attacking each other, rather than having to wait for some random mutation to do it for us. The ability to program our brains is bad because while we can do it faster, we don't always do it better than nature. Nature is slow but everything we represent is a result of survival over all the previous generations and we are very well adapted to surviving in harsh conditions. When we started programming our brains, we took that responsibility and the end result is that most humans today can live only in the cities with all the technology helping them. If a solar wind takes out a few satellites the entire world communication and the air industry will come to halt. Just imagine twitter being down. Wait, that happens already. Just imagine google being down. The ability to hack our brains allows us to reprogram our brains quickly and we can adapt to start talking again instead of texting each other. But it is all short term. Very, very few people have tried to program their brains to aid the long term survival. Mostly, we end up learning things that aid the immediate survival and not long term survival.

If our lack of ability to see the longer term health of earth gets us in trouble, then I suspect humans will be gone and it will be many thousands or millions of years before evolution produces something resembling humans again. Life is very resilient, species are not.

How to use technology to reduce arrests

2009-08-10T10:04:00.000-07:00

I heard on NPR today that the number of arrests made by the Seattle police has decreased by about 35%. One of the reasons mentioned was that they raised the bar for what is arrestable. Anyone possessing less than 3 grams of drugs will be warned and let go instead of arrested and put in jail. This made sense. The other reason, the funny one, stated that the Seattle police thought it took much time to fill out the report in the new software they have installed. Although the chief of police stated that he doesn't know anyone who complains about the software, I have to wonder how much truth is in that report. I knew technology would solve crime but I never expected it to do it this way!

Watching videos on the bus

2009-06-27T14:53:00.000-07:00

I started out with the simple goal of playing TED videos on the bus, so that I could use the time a little better than staring out of the window. So I saw a (relatively) cheap used sony walkman for $50 and bought it. It was really nice but it had a severe limitation of being capable of playing videos of 320x240 resolution only. So I decided to convert the videos using Amazon's EC2. The effort is detailed in my previous post. The conversion program was cut short as it was not cost effective.

So, as I mentioned in the previous post, I decided to do the easy thing and order stuff from the internet. I ordered a Zune. Which was a bigger disaster than I predicted. Not only did the device fail miserably (as documented in great detail on the web), the software also managed to delete my collection of TED videos I had downloaded. Luckily I had bought it from Amazon who promptly refunded my money when I returned it to them.

With Zune failing, the hunt for a new mp3 video player was on. Around that time, Brian Carper, a lisp blogger, blogged about his Cowon D2 being a much better player than iPod. Which prompted me to look at what Cowon has to offer. There are a number of cowon offerings that are quite attractive, but I found only the Cowon A3 meet the requirements of being able to play high resolution video (at least mp4 format) AND FM radio.

Cowon A3 has all the features that I could imagine and more. I plays a dozen audio and video formats, is able to receive FM radio, has speakers, has a microphone and can record audio, allows one to view pictures and read documents. There is also an optioned called Mobile TV but I haven't figured out what to do with it yet or how to get it to work. The setting menu for mobile TV only has options in Japanese (I think). There is a joy stick instead of individual buttons that make it easier to move around. Creating playlists is easy. Randomized playing is also available on a per directory basis. Copying files is as simple as hooking it up to the computer with the USB drive and dropping files in a folder (MOVIES for movies and MUSIC for music). There are a few cons as well. It is bulky and somewhat expensive. The joy stick takes some getting used to. And when the player shuts down and restarts it doesn't remember what was playing. The hold button is not easily accessible after the protective leather cover (sold separately) is put on and there is no way to easily pause/mute the thing without opening the cover, which means that I can't manipulate it is in my pocket. I have enjoyed it very much this past month and easily been overloaded with the wonders and problems of the world as told by the wonderful TED speakers. Did I mention I am like TED talks?

What I find absolutely amazing is that the small sony walkman mp3 video player has a feature called AVLS -- Automatic Volume Limiting System which, coincidentally, automatically limits the volume coming out, so that I don't go deaf. I have longed for this type of feature (limit volume to 70dB) in every single device that spits out audio that I have used. But none of them have it. Just imagine the TV not shouting when the ads come. The Cowon A3 doesn't have it either. I wonder how much longer I have to wait to get such helpful features.

Update 1 [2009-07-10]: As my sister rightly pointed out, my mother wanted the AVLS feature a decade and a half ago. Still no TV with AVLS :(

Update 2 [2009-07-10]: The Cowon A3 does remember what was being played before it was switched off. The feature is not on by default.

Stack Overflow reputation: Reputation, Interest - which is it?

2009-06-03T13:34:00.000-07:00

The Stack Overflow faq and the blog post by Jeff Atwood on voting analysis on SO clearly state that the reputation is not a related to the skill of a given person but indicates how much invested they are in the site. In other words, I can get a lot of reputation if I answer a lot of very easy and common questions that don't require much skill. It is also possible to have high reputation by answering tough questions although by looking at the reputation alone, there is no way to know.

Recently, SO started a job board and Jeff's partner in this endeavor Joel claims that people putting SO reputation on their resumes is informative. This implicitly implies that reputation is tied to skill, which is not necessarily true. The only way to know the applicant's ability is to read the questions and answers he posted on SO. Given that most recruiters only scan the resume, I wonder who is going to take the time to actually read through the resume.

The idea of putting the reputation on the resume is painful to me because it creates a random metric that people want to because it is a convenient single number but doesn't necessarily reflect skill.

Software Subscriptions

2009-04-24T17:02:00.000-07:00

For years, Microsoft and other companies have been trying to get users to switch to a subscription based software distribution and usage model. The reasoning is simple. If users buy a particular software version, they can continue running it with the parent company having little control over it. Forcing an upgrade generates a lot of pain and anger. A large part of the bad name that companies like Microsoft and many others get is due to such non-user friendly acts. If the software were to be subscription based, then the companies have a much better defense for turning off the software. Which is why I think the latest IBM and Amazon EC2 combination is a prelude to software subscriptions. The idea is simple. Simply start an instance which has a predefined set of IBM applications and pay hourly rates for the usage. Of course the usage rates are higher than the baseline $.10/hour. On the one hand, more people are going to use IBM's products as they are not forced to buy the whole thing. One the other hand, IBM loses revenue that it used to get by forcing people to do things. In the long run, I think the small revenue will be bigger than the big revenue. Coming back to software subscriptions, the only thing stopping OS providers like Microsoft or interactive application providers like Adobe is the latency. Even today, the latency is simply too much to run an interactive application. If latency issues are solved, then we will be back to the days of thin clients except that this time around, each person will have their own server controlled by someone else.

Is free energy really good?

2009-04-03T09:36:00.000-07:00

This is another post in my continuing series of arm chair theories. I didn't label my previous theories as such, but given that I don't have degrees on most topics I talk about here, very few are NOT arm chair theories.

Today's topic is energy. Specifically cheap and clean energy. The entire world today is focused on energy. Both its source and its lack of abundance are a great source of grief for a lot of people. The source is a problem because it is predominantly fossil fuels and using fossil fuels has led us to a lot of problems. The lack of abundance is a problem because, well, most people don't have as much as energy as they want or in many countries, even need. The many solutions being touted as the panacea to all of our energy problems are (1) fusion, (2) fission and (3) renewable resources (solar, wind, oceanic etc.). Note that hydrogen (via fuel cells) is not an energy source by itself as we need energy to get that hydrogen in the first place. It is only a means of storing and distributing the energy that comes from some other source such as options 1, 2 and 3.

As mentioned earlier there are two aspects to the energy problem. We want cheap sources of energy and we want clean sources of energy. The necessity of cheap sources of energy is obvious. Cheaper sources mean more can be done easily. The cleaner part of the energy problem is more dirty. Most people couldn't care less about whether or not their energy source is clean. The pollution problem all over the world and the climate changes affecting us are evidence of this fact. The only way humans are going to use clean energy is if it is the cheapest energy as well. Unfortunately the priorities of most people are reversed, as always. What we need is a "reverse the polarity" button a la Star Trek. The reason for this is that humans strive towards efficiency only when a particular resource is scarce. Which means that if we get a cheaper source, we are going to be profoundly stupid about using it. Even in Star Trek, they only work on increasing efficiency because their power is limited. All those miners getting stuck in dilithium mines and in need rescue are not just plot devices! To be sure, the scarcity of a energy will severely limit growth, so the solution is not to have cheap sources at all, but to be careful about using it. Our body is excellent in making use of the limited resources it has got, just not us.

Downloading youtube videos

2009-04-03T09:02:00.000-07:00

I have been using keepvid.com to save youtube videos and wanted to reduce the process from two clicks (one click to click on the bookmarklet and one click to download) to one. Hence the greasemonkey script. All it does is put a "Save Video" link just below the video itself. Click on the source file to install it. Give it a try and if it doesn't work, let me know. The script allows one to download the video as an mp4 file.

Diveintogreasemonkey.org is still quite helpful, even though it was last updated many years back. The only key to the script is using unsafeWindow to get the swfArgs variable that youtube defines.

Double Entry Book Keeping

2009-03-21T10:36:00.000-07:00

In an earlier post I complained that double entry book keeping amounted to duplication of data and is therefore bad. Since then, I have come to the realization that the system is a good thing. The system is just like a double stranded DNA with complementary entries except that different people are writing to different strands. Eventually the two strands should match up and when they don't the company is in for a rough ride.

Civilization

2009-02-20T14:38:00.000-08:00

The book Guns, Germs and Steel takes about 500 pages, among other things, to explain how the European civilizations were able to do so much more than the other civilizations. I watched the documentary before I read the book and after watching the documentary, I didn't feel like reading it. The documentary gets many things right but I wasn't totally convinced about a few explanations. For one thing there is a conflict between the stories related to hygiene. The Europeans, having domesticated a few animals, were living in filthy conditions and when they came to south, they survived at the expense of the indigenous population by spreading the disease. Sometime later, the documentary goes on to say that when the Europeans came near the equator, they were unable to deal with the diseases spread by the mosquitoes. So, the lack of hygiene didn't help. Ultimately I wasn't convinced by Jared Diamond's arguments that guns, germs and steel were the causes for the rise of the European colonization.

As you may have guessed, I have a theory of my own. My theory is called the theory of laziness. Places like South Asia or most of Africa are pretty warm. For survival, one has to find food. And that's it. The elements of nature don't bother the population too much. Whereas Europe gets cold. Which means that to survive, simple procurement of food is not enough. People need clothes, heat, a roof etc. Which means that to survive, a minimum amount of resourcefulness and effort. And succeeding among the filthy, resourceful people took even more intelligence. Which is what led them to do crazy things like go on voyages. Whereas people in other continents were just happy the way things were. Food => happiness and there was really no need to do crazy things. Of course, for even colder places or places the Sahara desert, survival takes way too much energy so even though people might not be happy, they don't have any energy left to do crazy things.

There is a possibility that having found food easily, people could have used the extra time to do conquering. Why that didn't happen, I don't know.

All that said, Jared Diamond's next book about the fall of societies called Collapse is quite good. Again I have only seen the documentary and not read the book but the arguments for collapse are much more well founded than the arguments in the first one.

Unicode

2009-02-16T20:40:00.000-08:00

I recently attended a talk on Unicode and the presenter said this as a take home message (even if we forgot everything): "Everything has an encoding". That sentence alone goes a long way in understanding what Unicode, UTF-8, ASCII etc. etc. are. Let me explain.

Introduction

When people first stared working with computers all known text was in English and a mere 128 bits were sufficient to represent the alphabet including punctuation, upper and lower case letters and other print characters. But the need to represent more characters arose. Up until now, the character A was represented by the value 65 and it fit in one byte. Note that the value was the character is the same as the representation i.e., the value 65 represents A and on disk, when stored in a file, it is stored as the value 65. This is called the ASCII encoding. ASCII is very simple minded. The character value and the encoding are the same.

Unicode

For representing say Indian languages, more than 256 bits (one byte) is needed. So, Unicode essentially gives a number to all characters possible. I mean that all characters from all languages in the world. It is a giant lookup table that says number 5000 looks like this. Now we come to storing those character on disk. We can no longer treat one byte as one character because clearly the number 5000 needs more than one byte to represent it. What I have been calling character up until now is called as a code point in Unicode. Which means that Unicode has a big table of corresponding code points and pictures.

Encoding Unicode

So what does it say about storing the text on disk? Clearly ASCII is not sufficient. The simple and easy solution is to see the largest code point in use, calculate how many bytes are needed to represent it and use that many bytes to represent one "character". The character is in quotes because I wanted it to mean a character like A. That is what UTF-32 is. It uses four bytes to represent a code point and it can store all of the 1.1 million Unicode code points. That's a lot of code points. In ASCII one character was one byte. But in UTF-32, one character is four bytes. Which means that UTF-32 is a multi-byte encoding. But it is highly wasteful. If the text is largely English, then more than three quarters of the space is taken up by zeros. Which is why the intelligent people at Unicode variable length encoding schemes such as UTF-8 and UTF-16. In UTF-8, code points are encoded in one, two, three or four bytes and in UTF-16 each code point is represented in a two or four bytes. Let us examine how code points map to bytes in UTF-8. UTF-16 is similar. Since UTF-8 is a variable length encoding, when we store code points, we have to indicate whether or not the byte we are looking at is a single byte or a part of a bigger byte sequence. This is accomplished by leading bytes. Which means that when we have to store a code point and it requires three bytes, we don't simply convert its value into three bytes and store it. Instead we mark the first few bits in the first byte that indicates is the beginning of a three byte code point, we mark the first few bits in the second byte that indicates it is the second byte in a three byte code point etc. Wikipedia has a nice example to explain this:

The $ sign has a code point of U+0024. This only needs one byte and it is represented as 00100100. The first zero indicates that is a one byte code point. The subsequent seven bits, indicated in bold, actually store the code point. This is the key difference between ASCII and UTF. In ASCII, the code point and the byte representations are the same. But in UTF they are not.

The Unicode code point U+00A2 is represented using two bytes: 11000010, 10100010.

The code point U+20AC is represented using three bytes: 11100010, 10000010, 10101100.

Finally the code point U+10ABCD is represented using four bytes: 11110100, 10001010, 10101111, 10001101.

The nice thing about UTF-8 is that ASCII characters require one byte only and their encoding matches the code point.

More Encodings

This is what the presenter meant by "Everything is encoded". There is no such thing as plain text. All there is, is text encoded in some format. ASCII and UTF-* are well known encoding standards. There is nothing preventing me from creating my own encoding and start using it. Which is what a lot of companies did when they had to support more characters than ASCII. Hence the existence of ad-hoc encodings such as WinLatin1 or MacRoman. The wicked part is that there is easy way to know what the encoding is just by looking at a string of ones and zeros.

XML solves this problem by having the encoding attribute. Which makes the XML readers read up until they find the encoding line and go back reparse the file using whatever encoding is specified.

There is more.

The next piece of information involves combination characters. For example, say we want to represent a with a dot on top. This can be a code point in itself or it can be a combination of 'a' and the dot on top. For frequently used combination characters, Unicode stores the combination as a single code point in addition to storing the individual parts as different code points. When a piece of software encounters the text, it is up to the software to decide how to display it. So if you don't have the respective fonts installed you will get question marks or boxes.

There is even more.

What I have said above is only beginning to scratch the surface. For example, if the twitter box says 100 characters remaining, do they mean 100 bytes or in fact 100 code points? If I suddenly type something in Kannada, then store 100 characters, the software will definitely need more than 100 bytes. The fun begins when you have bidirectional text. Some arabic languages start on the right side of the page and go left. If you two characters from that language and two from a left-to-right language, then the display and highlighting becomes tricky.

So, the take home message is that everything has an encoding. When storing text, make sure you know what encoding you are storing it in.

References

Unicode FAQ

UTF FAQ

Comparison of Unicode encodings

Summary of "A programmer's view of the Universe, part 2: Mario Kart"

2009-01-01T07:45:00.000-08:00

After spending a reasonable amount of time reading it, I finally finished the article A programmer's view of the Universe, part 2: Mario Kart. Here is a summary:

A system is a physical and/or virtual space defined by its boundaries. Physical systems include a fish tank, a phone, house, etc. A virtual system is a system that exists in a simulation such as a video game.
Every system is embedded in its host system. A person lives in a room, a video game exists in a game console or computer.
For someone inside the system, the question of what is beyond the system is meaningless. For example, in our universe, most of the space is composed of nothing, but beyond the universe, there is not even nothing. Steve calls it undefined.
There may be tunnels in and out of the system. For a virtual system, this includes mouse/keyboard input.
The host system typically imposes a strict set of rules on the embedded system.
Sometimes it is not possible to know the the existence of the host system and/or the existence of the tunnels in and out of the system.

I believe this is it. Please feel free to use the comments to correct me. For those of you who end up discovering this article before Steve's, this will help you know where the thread is going.

Wikipedia featured article feed

2008-12-25T20:19:00.000-08:00

For some time now, I have been searching for a feed for wikipedia's featured article and featured picture. There wasn't any that I could find easily, so I created one. Basic idea is to fetch the main page, mess around with the html, extract the relevant parts and push it as a feed. I decided to use google's app engine to do this. About 200 lines of python, 50 lines of kid template and six hours later, the app was ready. Most of the time was spent in looking at the html structure of the Main_Page and retrieving the relevant bits. The feed generation needs to be only once a day as the featured article/picture is changed once per day. So as an optimiztion, I stored the feeds in the datastore, and serve it from there. If the time stamp on the feed in the store shows it to be more than 24 hours old, then a fresh round of screen scraping is done. The feeds are probably not pretty, but they seem to work. Here is the url: http://wp-tfap.appspot.com/?feed=tfa&type=rss2 . The feed parameter can take two values: tfa for today's featured article and tfp for today's featured picture. The type parameter can two values as well: rss2 for rss 2.0 feed and atom for atom feed. Any comments and criticisms are welcome.

Update: Corrected typo in the url.

Accounting and Me

2008-12-13T18:18:00.003-08:00

Having obtained three degrees in chemical engineering, I am now writing software that accountants want to use. This means that I am learning about accounting and a bunch of new programming techniques as well. My understanding of accounting principles is very limited but currently I am viewing it through the lens of a programmer as well as a chemical engineer. There are a couple of things that I would not have done.

1. Double entry bookkeeping: As a programmer, this is not something I would have done. The concept of double entry bookkeeping is that all transactions are written in two books - one book that keeps track of positives (assets) and a second book that keeps track of negatives (liabilities). This means that if I spend a dollar, it is entered as a negative amount in the assets book and a positive amount in the liabilities book. Data duplication is something all programmers try to avoid at all costs. And the data duplication via the double entry is at the heart of accounting. Companies spend a lot of time closing the books and making sure the two versions agree but I have a feeling that if there was one copy of the data then there would be no need to close books.

2. Accrual system: This one strikes me as odd from a chemical engineering (and in general a physical) perspective. There are basically two forms of accounting - one is cash flow and the other is accrual system. In the cash flow system, you define a boundary for the company/household and write down all transactions for the system when they happen. That's it. If I spend a dollar, I write it down as one dollar spent. If I make a dollar, I write it down immediately as one dollar made. This is very obvious to me as a chemical engineer. I draw a boundary around a system. And then I use the simple formula of "in - out + generation = accumulation" to account for material movement in and out of the system. If I make two dollars, spend one dollar and get a dime as interest for the two dollars I made, then in = 2, out = 1 and generation = 0.1, which leaves accumulation = 1.10, which is what I am left with. Not so in the accrual system of accounting. In the accrual system, the process of making/spending money and accounting for it is decoupled. This means that even though I might spend a dollar for the whole year for getting news papers, I write down only one twelfth of a dollar spent every month. Keep in mind that I have already spent the dollar in the first month. After the first month, I pretend to have eleven twelfths of a dollar for the rest of year. And so on. By decoupling accounting and the actual cash flow, I am now free to account for the cash flow as I wish. If chemical engineers were to do this, they would recognize a toxic spill over a century there by making each recognition of the spill non toxic and non lethal. I believe the decoupling of actual cash flow and the recognition of the cash flow makes it very easy to forge books and claim that a company is making profit when in reality it is not. I am sure there is a sound need for coming up with such a system and hopefully I will discover it as I learn more.

Photography lessons

2008-12-05T20:09:00.008-08:00

Having been prodded by my sister (who blogs at http://lakshmibk.blogspot.com/), I decided to write whatever I have learned so far about photography. Having started photography only two years back, I don't have a lot of experience, so I will say what I have understood of the sayings of other famous photographers.

The big secret

The really big secret to taking great photographs is to be able to recognize and tell great photographs from crappy ones. Anyone can say "that looks great", but until you can say precisely why a photograph looks great, you won't be able to make one yourself. That is the real secret that I have learned in the past two years of taking photographs and in my previous life (pre high school days) of making art. Everything else is only a means to achieve what you can picture in your mind.

Summary

The number one lesson is to be at the right place at the right time. I can't stress this enough. There really is no substitute for it. The space-time of taking a photography is half the picture. The other half is deciding what to take. With all the powerful image editors available, post processing makes up another half of the picture. I try to minimize any editing afterwards but, I shoot for a ratio more like space-time:shooting:editing::60:30:10. There are already a lot of good sources about how the shutter and aperture lets us take different types of pictures, so I am not going to talk about them. Let me explain what I have done so far with a couple of examples.

Space-time

As Ken Rockwell says, the best times to shoot are dawn and dusk. I completely agree with him. When shooting cityscapes, twilight is the best time. The blue of the sky is very rich and the lit buildings give a very good light. The orange light from the buildings contrast nicely with the blue of the sky resulting in a very pleasing image. At twilight, the light changes very quickly. It is often necessary to show up at the place before dusk, setup the camera and wait until the light gets interesting. A little bit before, the sky is a very pale blue and the buildings are not lit. A little bit later and the sky is totally dark with the light from the buildings completely blown. Here is a short time lapse of the same building to show how quickly the light changes:

Shooting

Next comes actually taking the picture. Even the low end consumer point and shoot cameras have a lot of configurable options these days and can be exploited to get good shots. Most of the time, the automatic setting works just fine. Here are a few places where the defaults might have to be changed:

When shooting a twilight city scene or a night scene, it might be necessary to overexpose a bit. The image looks a bit brighter and not so dull.
During long exposure night shots, the sky shows up as a dark brown/red color. To counter this, set the white balance to tungsten.
Sunset scenes get an enhanced look of orange if the white balance is set to cloudy or shade. The scene is already red/orange/yellow, so it may not be necessary to tweak the white balance.

On a photography trip, I spend most of the time just looking around until I find something interesting. I then spend some time looking through the viewfinder and compose the image. It is quite hard to decide which part to fit in the image and which part to leave out. In spite of that, I can only decide whether or not my shot was good by looking at the results. I usually end up recomposing and shooting again.

Editing

With the powerful image editors available today, no picture is complete, much like the Star Wars movies. People can't resist the temptation to change some settings here, tweak some colors there until they are satisfied. Most of the time, the satisfaction is hard to come by because the primary way to be able to change the photograph is to be at the spot and re shoot. This is possible if you can reproduce the conditions but most of the time, it is not possible. I am not against editing. There are a couple of photos that I have spent more than an hour editing. And I came away happy with the results. There are also times when no matter what I did, the photograph did not improve and worse, the image quality deteriorated because of all the fiddling. Just goes to show that we have to be careful while editing.

Editing an image is a way to get an existing image to what you have pictured in your mind. Setting the various control in a camera is one form of editing. The other form is with the image editor. I will mostly talk about editing with an image editor. With the digital camera, you can take a picture, look at the preview, change setting and re shoot until you are satisfied. Some of the times when I edit and why I do so are listed below:

Sometimes a shot gets too busy and lost in the details. At such times, converting the image to black and white reduces the clutter. The ground level of Seattle's space needle looks somewhat like this in color:
.
I didn't like it very much that way, so I edited it, primarily converting it to B&W. This is the result:
Portraits also look better in B&W as the subtle imperfections in the skin are hidden when converted to B&W.
Cropping a picture is also editing although most people don't consider it so. There was once a time when I didn't do cropping as it would result in a smaller image with lesser mega pixels. And I ended up with badly composed shots. These days I chop off all unwanted parts until I get something I like. With no pressure of printing the image in large size, I don't have to worry about mega pixels.
Conversion to B&W is not the only edit I end up doing. Sometimes, something as simple as having a high contrast image could work as well. This some random motor bike that was collecting dust in front of a house.

This obviously looks very dull. I wanted to highlight the old dusty nature of the bike and I increased the contrast until the transfer function in the curves dialog looked like an S with square corners. This is the result:

Another big secret

OK. I forgot about this one. When I read about this secret, it made perfect sense and yet I had to hear it from Ken Rockwell and not think of it myself. And the big secret this time is: post only your best pictures and not everything you click. In hindsight this is obvious. Not even the National Geographic photographers produce publication quality material every time they aim their camera and click. In fact, I saw a documentary where they said that a NG photographer shoots about one hundred rolls of film in a week and when they get back from a photo shoot with 10,000 photographs, one makes it to the magazine. My own ratio of photographs shot to photographs posted on my flickr page (http://www.flickr.com/groups/balubk/) is slightly more than 1%. Some of the shots are there because they are shots of some funny signs I saw on the road, so if you discount them, the ratio comes to less than 1%. For every photo you see on my flickr page, there nearly a hundred more on my hard disk.

Workarounds for not having data as code

2008-12-03T09:22:00.006-08:00

I was recently reading up on Spring framework for Java that takes care of setting objects at runtime among other things (this is called Dependency Injection). Let me briefly explain what DI means. If we have a Java class Foo and it has a reference of interface Bar called bar, then either Foo or someone else is responsible for setting bar. This can be done in the class itself by calling bar = BarImpl(), which means that the implementation of Foo is tied to BarImpl. Or, whoever is using Foo can call the setBar method and set the object. If the bar reference is set in Java code, then everytime a different implementation needs to be used, the code has to be recompiled. Worse, if we want to ship two different versions of the implementation for two different customers, we have to have to maintain two branches. Which is very messy.

Enter Spring. Spring allows us to write an XML file where we can specify that there exists an object of BarImpl class and there also exists an object of Foo class. We can further specify that the bar property of the Foo class is to be set with the BarImpl object just created. In the main body of code, we read in the call SpringFramework.readXMLFileAndGiveMeTheObject and we have a nice Foo object configured to have the correct implementation. To support different versions, all we need to do is ship the same code with different XML configuration files!

The reason a framework such as Spring is needed is because of the notion that Java code is executable and the XML configuration file is not. This distinction is important because for the Java code to be executable, it needs to be compiled first. If we start hardcoding implementation details, then the code base needs a fresh recompile every time a different implementation is chosen. With Lisp, half the problems go away. We can separate the configuration from the implementation just like in our Java implementation, but we don't have to resort to learning and using a separate framework for that. The configuration can be stored just like any other lisp file. It can be read just as easily with the read function built into Lisp. The compile at read time functionality of Lisp will take care of loading the appropriate classes from the "configuration file".

Over the last few months I have to come to appreciate Java and its usefulness with the vast number of supporting libraries. Everywhere I look, people are writing frameworks in Java using reflection and all kinds of other esoteric things to circumvent a lot of restrictions inherent in the Java language. In Spring, since the classes are loaded and set at runtime, the code is now one step closer to an interpreted language. Which goes to show that all languages are evolving towards Lisp.

test

2006-06-02T20:23:00.000-07:00

test post

FP

2005-02-01T15:20:00.000-08:00

Frist Psot