Google Voice on the iPhone

After being previously rejected from the App Store for difficult to understand reasons (http://www.tuaw.com/2009/12/25/is-google-voice-available-for-the-iphone/), Google has returned today with an alternative -- in the form of a web application. If you already have a Google Voice account, just visit http://m.google.com/voice from your iPhone, and you can now do outbound dialing through Google Voice.  It even looks mostly like a native application if you click the '+' button in the browser to add it to your home screen.  It first dials a local number, which it routes through to your desired callee -- as my friend Jason pointed out, if you do the AT&T A-List option, you can add that number to your list, and effectively have unlimited outbound dialing!

Pancake Bunny

Sun "Sales"

Due to various technical integration issues, this year I was put in the position of making a decision on an OS refresh for our company, which runs almost all Java-based software.  I've been a user of Solaris for many, many years in very large organizations, and have always been happy with the maturity, stability and quality toolset for diagnosing system issues.  In my current position in a smaller organization, I recommended Solaris 10 for our primary OS, with Red Hat Linux as an exception case for specialized applications which were Linux only.  In addition, we were shopping for a directory server, so we settled on Sun's Java System Directory Server Enterprise Edition (whew, what a mouthful - hitherto referred to as DSEE). I called the sales number to obtain a quote on obtaining licensing for DSEE, and a blend of Premium and Basic support levels for Solaris on some of our more critical systems.  I would have expected to receive a concise response within a day or two at the most, but instead, I was passed around various sales groups for weeks on end.  We randomly received a call from a Sun reseller which seemed kind of odd at the time, but even they didn't seem to understand Sun's licensing models for DSEE. I decided to start over.  This time I would call back, and not "confuse" the issue by requesting a quote on two items -- this time I would ask simply for a quote on how to purchase DSEE by itself.  Later, rinse, repeat.  I finally called back after a few more weeks of this nonsense, and forced the front line sales rep to give me his direct number.  Unfortunately for him, I said, he would need to be my proxy through this organization until I got what I needed. After many red-faced phone "conversations" over a period of several weeks.  I finally received the full story.  Sun does not sell directly to smaller accounts like ours.  Great!  That doesn't bother me one bit, just say that up front, and connect us with a reputable channel partner/reseller.  We finally got connected with a good rep at a reseller.  He understood the licensing, and got us a quote within a reasonable time frame.  We executed the purchase request -- and waited.  Ten days went by, along with multiple calls to the reseller to find out what was going on.  Apparently the reseller had submitted the order to Sun, but they just were not delivering on getting the licensing information to us. After all this frustration, I am sorry to say we gave up.   The executive management in my company (rightfully so) lost confidence in Sun as a software vendor.  Contrast this situation to Red Hat.  We called Red Hat, I spoke with a sales representative that day, he explained our options, and told us that our best bet was to purchase from a channel partner such as CDW, as they could offer us better pricing.  We placed our orders, and were done within a couple days.  It's a sad story of business and sales issues forcing our hand on a technical decision.  I can't understand how Sun can be so broken as a company to not deliver on such a simple thing as licenses for a basic software package and support services for a handful of machines. P.S. I should add that throughout this time, we encountered an issue with our software running on the Solaris stack.  That would normally not have been a show-stopper, we work through issues like this on a regular basis -- but the technical issue on Solaris, in addition to the business hurdles, just turned out to be too distracting for a company our size with larger issues to solve, such as developing our own systems.

Java DNS Lookups

DNS lookup performance is not something Java developers typically worry about, but in some cases, it can become a real gotcha.  Allow me to share my little tale of woe with you. If you didn't know by now, the default behavior of the Java runtime is to cache DNS lookups for the life of the JVM.  This may be OK for some applications, but if you have a long running process, and an administrator 'moves' a service via a DNS change, your application will never know.  This is almost never acceptable in my experience for typical deployment scenarios.  You can disable this behavior by setting the system property networkaddress.cache.ttl to a value indicating the number of seconds to cache successful DNS lookups. See the javadoc for java.net.InetAddress for more details. I ran into a situation at work recently where, in an ill-conceived effort to load balance access to a remote resource via round-robin DNS, the cache time was set to 0, effectively requiring a DNS lookup for every connection attempt.  This wasn't immediately horrible when used on the LAN near the DNS server.  The trouble began when we deployed applications across the WAN, and the remote ends of the WAN had no local DNS server.  Latency for a roundtrip from the remote servers to the datacenter housing the DNS service was approximately 80ms.  You might expect an added cost of just slightly more than 80ms per connection, but it was worse than that. Our first indication of trouble was a connection pool that would randomly fail a connection attempt.  The connection pooling mechanism had a connection timeout functionality that would fail the attempt if it took over a certain time, by default, 5 seconds.  Connection attempts typically would take around 150-300ms or so to be created, fairly typical -- but the occasional spikes were alarming.  We had a couple developers spend hours pouring over the home-grown connection pooling mechanism looking for threading issues or anything that could have caused the problem, in concert with forcing our TechOps team to spend time investigating potential network issues.  In the end, it turned out to be none of the usual suspects. An average lookup via InetAddress.getByName() was taking ~480ms -- significantly longer than expected.  A network trace revealed that DNS lookups were returning quite fast, but there were more queries than expected for each lookup.  It was then that the realization hit us.  The Java runtime was spending a lot of time looking for IPv6 addresses (AAAA records) that are not used within our network.  And it was even worse than that.  Here's how a typical lookup would go:
  1. AAAA lookup for db.mycorp.com (FAILED) ~80ms
  2. AAAA lookup for db.mycorp.com.localdomain.mycorp.com (FAILED) ~80ms
  3. AAAA lookup for db.mycorp.com.otherdomain.mycorp.com (FAILED) ~80ms
  4. AAAA lookup for db.mycorp.com.yetanotherdomain.mycorp.com (FAILED) ~80ms
  5. AAAA lookup for db.mycorp.com.mycorp.com (FAILED) ~80ms
  6. A lookup for db.mycorp.com (HIT) ~80ms
Becuase the host's /etc/resolv.conf specified a number of search domains to attempt, the host resolver spent over 400ms on every attempt looking up records that we would never care about.  This turns out to have an easy fix: set the system property java.net.preferIPv4Stack=true.  With that property set, the runtime will skip steps 1-5 above, and request the desired A record directly. We still have the issue of the random connection failure.  Because DNS lookups use UDP, packet loss results in a failed DNS lookup attempt.  The default behavior (for the Linux resolver at least) is to timeout after 5 seconds, and then re-attempt.  Because our connection pool timed out the connection attempt after 5 seconds, it never had a chance to even pass the DNS lookup which could have taken up to ~5480ms (5 second timeout, plus AAAA lookup overhead).  Setting a value of greater than 5 or less than 5 would have been fine, but 5 seconds was an unfortunate coincidence with the resolver timeout.  Another option would be to configure the resolver to timeout more quickly.  I'll leave that as a discussion between you and your network administrator as to what gives. :) At the end of the day, we set our ttl value to something more reasonable, shortened our connection timeout (forcing a reconnect attempt more quickly), and with the IPv4 preference, our problems have been resolved.