Keeping you in the loop

The email subscription form we introduced at Wilton Diaries a couple weeks ago has really taken off. In other words, we've got three subscribers, but the "cool" factor is way up there.

When you subscribe via email, you will get a message when the Blog gets updated. To subscribe, simply enter your email address and click the "Subscribe" button. A form will then pop-up asking you to verify that you are an acutal person and not an aggregator. You simply type the letters that show in the box and click the buttons you are instructed to. You will then be sent an email message with a URL that you must click on to finish your subscription. This last part is important as you don't want to go through all that work and not get the feed!

If that all sounds too complicated, feel free to continue checking the So What Co-operative every day.

Playing in the Sand

Just got back from a week in Kuwait upgrading our database down there from 9i to 10g and upgrading our ESRI from 9.0 to 9.1.

I must say upgrading Oracle9i to 10g on a Sun Solaris 10 OS is the most easiest and painless install I’ve ever done. I found out on our Dev box that trying to put 10g on Solaris 8 was like trying to put a square peg in a round hole. Luckily, I got our UNIX admin on board with upgrading all of our servers to Solaris 10 before I started my production upgrade festivities.

When I got down there we had one database serving up maps and vehicle tracking data. All the tracking data is OLTP oriented and the maps are nothing but a bunch of blobs. The user has the ability to see maps by themselves and vehicle information (text) by itself. The user also has the ability to see maps and the vehicle data at the same time.

The server has plenty of horsepower and space so I decided to break the database into two. I created another database and put the maps on it. I configured it for bulk stuff .One other thing I did was, we have a particular map that automatically loads up when a user first accesses the webpage, I threw that in to the keep pool. Performance is very nice. It’s so refreshing when you have a database configured correctly for the environment it supports.

Where am I deploying MySQL?

If cost were no object, I'd always deploy Oracle. I'm comfortable with Oracle technology and I think I have a pretty good idea how to implement and administer it.

In the world of corporate IT, however, budgets are king. Projects are measured by their Return on Investment (ROI) and the lower I can get that investment, the better return I can get for my investment. I have a real hard time spending $160K on an application that will occupy 40G of space.

In my opinion, I'd use MySQL for anything but the most mission critical applications. I'm not saying MySQL can't handle the most mission critical applications, but I'm not comfortable betting my business on MySQL at this point.

I think there are about three sweet spots for MySQL. The first is small to medium size OLTP databases (<100 GB) that are fronted by something like a java middle-tier. These applications typically control most of the business logic and authentication/authorization in the middle-tier (right or wrong) and use the database as a big storage bucket. These applications rely on the backend serving data as fast as it can and MySQL can serve data just as fast as the next guy.

Another area where MySQL excels in serving database driven content directly on the webserver. This type of application typically cranks out high numbers of queries and has very little updates to worry about.

Last, but not least, MySQL is suited for data marts ( < 1TB). Stuffing lots of historical data into denormalized relational tables is what "LOAD DATA LOCAL" is all about. These types of applications aren't needed 24x7 but require snappy response times when queried.

No, MySQL doesn't have some of the features that some of the big-box databases have. And it's got plenty of limitations. But when you want an 80% solution, I think it's the right choice. My company is sold on MySQL and as our confidence grows in the software, so will our installed base.

Quick and Dirty MySQL Backup

Until recently, the MySQL databases I work with contain data that can be retrieved from other sources. Most of the data is either batched in from flat files or another database. It would be inconvenient to reload a couple months worth of data, but since these databases are not mission critical, the business could operate without them for a couple days. Lately, we've been implementing some semi-critical systems that rely on a somewhat expedient recovery.

The requirements for the project were that the database must remain up during the backup and losing a day's worth of data was acceptable. All of my regular Oracle readers are cringing at the moment, but hey, that was the rules I was working with.

My first thought was to use mysqlhotcopy because it backed up the actual physical files. However, mysqlhotcopy only allows you to backup MyISAM tables and we extensively use InnoDB.

My next choice was mysqldump. mysqldump basically takes the entire database and dumps a text file containing DDL and DML that will re-create your database. Coming from an Oracle background, I knew there were shortcomings to dumping the entire database, but hopefully I could mitigate them.

The first hurdle was security. I specifically turn off unauthenticated root access on my databases, but I needed to be able to read all the tables to do a backup. I don't want to hard-code my root password or any password in a script as I don't have suicidal tendencies (diagnosed, anyway). So I created a user called backup that could only login from the server machine, but could login unauthenticated.

The next thing I had to figure out was how to get a consistent view of the data. I knew that my developers preferred InnoDB for it's Referential Integrity features and getting inconsistent data would be disasterous. Fortunately, one of the flags to mysql_dump is the --single-transaction which essentially takes a snapshot in time.

So I wrote a script around mysql_dump and --single-transaction and dumped my entire database to disk. Every now and again, I encountered an "Error 2013: Lost connection to MySQL server during query when dumping table `XYZ` at row: 12345". The row number changed each time, so I figured it had something to do with either activity in the database or memory. I could rerun the command and it usually finished the second or third time.

After the third day straight of my backup failing, I decided to research it a little more. mysql_dump has a flag called --quick which bypasses the cache and writes directly to disk. I put this flag in my backup script and the script started finishing more consistently.

The last hurdle was having enough space on disk to store my backups. Since the backup file is really a text file, I decided to pipe the output through gzip to reduce it's size.

Currently, my quick and dirty backup script is a wrapper around the following command:

mysqldump --all-databases --quick --single-transaction -u backup | gzip > mybackup.sql.gz

We're adopting MySQL at a blistering pace, so I'm sure I'll need to make changes in the future. For right now, though, it gets the job done.

Check this out

I usually employ a logon trigger for most of my Oracle databases so I can grab certain identifying information about the session. Then I save this information in another table for later analysis.

I have started testing 9iR2 on a 64-bit Linux box and have come across a certain peculiarity. v$session is defined as:

SQL> desc v$session
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
SADDR                                              RAW(4)
SID                                                NUMBER
...

I then create a table using the same type and try to insert a value:

SQL> create table jh1 (saddr raw(4));

Table created.

SQL> desc jh1
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
SADDR                                              RAW(4)

SQL> insert into jh1 select saddr from v$session;
insert into jh1 select saddr from v$session
                      *
ERROR at line 1:
ORA-01401: inserted value too large for column

Hmmmf. So I do a CTAS:

SQL> drop table jh1;

Table dropped.

SQL> create table jh1 as select saddr from v$session;

Table created.

SQL> desc jh1
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
SADDR                                              RAW(8)

...and look what size the column is!

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
PL/SQL Release 9.2.0.7.0 - Production
CORE    9.2.0.7.0       Production
TNS for Linux: Version 9.2.0.7.0 - Production
NLSRTL Version 9.2.0.7.0 - Production

Update: 2006/11/01 16:09:
From Support:

I checked my Windows (32bit) database and v$session.saddr is a RAW(4).

OK, that explains it.

The OS wars heat up

You may remember we talked about Oracle's Unbreakable Linux the other day.

Anybody try to get on Metalink yesterday? The Register is reporting on the poor response time yesterday. Maybe Linux will kill Oracle...

Here's an interesting take from Dave Dargo...

Firefox 2.0

I downloaded Firefox 2.0 this weekend to see what it was all about. I thought a couple pages of my regular sites loaded slowly, but that could just be my internet connection. Once of the first features I noticed was it's ability to detect a fraudulent website.

I received an email from a suspected ebay spoof. Sometimes I just click on the links to see how close they are to the real website. To my surprise, this message came up from Firefox:

Will Oracle Kill Linux?

That's right, not will Oracle Kill Red Hat, but will Oracle Kill Linux?

There seems to be some buzz that Oracle's Unbreakable Linux is positioning itself against Red Hat's Linux. Oracle will be offering support on a version of Linux that they have basically ripped off from Red Hat.

There's no doubt in my mind that Oracle won't kill Red Hat. Linux is used for more than just running Oracle software. I should switch my whole enterprise of umpteen hundred Red Hat computers so I can run 20 Oracle servers more efficiently? I don't think so. Can you see a SysAdmin calling in to Oracle for support and having to wait 6 days to talk to Sandeep in some far reaching corner of the globe? Can't see it myself. Besides, Oracle needs Red Hat to continue to develop the platform so they can rip it off again.

The big question in my mind is will Oracle kill Linux? To successfully deploy Unbreakable Linux, Oracle is going to have to snuggle up to the hardware vendors in order to get Unbreakable Linux pushed out on their hardware. I would imagine that is going to tick off some of the proprietary vendors.

Or better yet, will Linux kill Oracle? Will Oracle get so distracted from it's core business of infrastructure software that the products go downhill further?

Only time will answer these questions. Of course, I don't have a yacht and a billion dollars, so what do I know?

At this point, I'd just be happy with a filesystem that doesn't reboot my box every 5 days.

I can see you!

My apologies for not posting anything for awhile but I’ve had some personal issues that I needed to take care of . So….. enough of that I’m back now and hope to contribute on a regular basis.

Let me seeee… We left off with the wonderful world of Spatial and me ranting about ESRI. I want to continue with the Spatial stuff but I wanted to share with you guys a more intriguing (at least in my opinion) turn of events. Even though this is totally out of my job description, in my shop we have to do what it takes to get the job done even if that means doing *shutter* development work.

We recently had a complaint from one of our customers that in certain parts of a country that his equipment couldn’t send or receive a satellite signal due to terrain obstacles and he would like to be able to see on the webmap those areas that are “blacked out” so he can avoid them. Well, since we’re all about keeping the customer happy I was tasked to “Make it happen”.

The first step in the process is getting the elevation data of the desired location, once you have the elevation, you find out which satellite your equipment is using. Once you know the satellite, you find out what the Lat/Log and height (location and distance above the earth) of it is.

When we have all that information we start piecing the puzzle together. We have to create physical objects in the database like a shape file (our satellite) and a Raster image ( our map) and input all the data that we gathered up (elevation, lat/log, height) once all the values are given we use a tool to calculate the black out areas. The example above, I’ve used an observation tower as an example. The red areas are not visible from the observers view but the green areas are. Get it?


This website gives you in real time all satellite names, and locations using a nice little java thingy. http://science.nasa.gov/RealTime/JTrack/3D/JTrack3D.html

Taking the plunge

Within the last six months, I've gotten my enterprise off Oracle 8i and on to 9iR2. We were really behind the 8 ball (no pun intended) since the support window for 8i was quickly running out. We forged ahead and were able to get everything to 9i. Along the way, we upgraded some development systems to 10gR1 and put out a non-critical system in 10gR2. However, something was missing. We hadn't taken advantage of a great deal of the 9i features that didn't come straight out of the box (Optimizer enhancements, PL/SQL fixes, bug fixes, etc).

I didn't want the upgrade to 10gR2 to go the same way. I don't want a hard-and-fast deadline for 10g and I want to be able to take advantage of some of the bells and whistles that come with 10g to ease my management burden. My educational jumping off point is to be certified in 10g.

I'm a firm believer in certification for enhancing the individual's self-worth. That doesn't mean every certified DBA is worth something to a company, nor does it mean somebody certified is worth more than someone who is not certified. It simply means that I view the certification process a valid educational opportunity for the individual to benchmark his knowlege against a standard.

Now, it's not the full-blown-from-the-top certification path that new DBAs are going on. I'm certified in 7, 8, and 8i, so I'll be looking to upgrade from 8i to 10g through 1Z0-045. I know, kind of a cop-out, but I think this will give me an opportunity to try out some of the new stuff before I need it. I bought the book last week, so now I have to start studying.

Don't be surprised if you see interesting (to me, anyway) 10g thingies on the blog in the next few months.

Apex Rocks

Two days ago, I knew very little about Apex (HTMLDB). I knew how to set it up and install it on the database side, but as far as developing something with it, forget it. There are other people using HTMLDB at my company and they have been getting good results with it. Since I didn't know that much about it, I kind of let them have free reign over things.

I am still supporting some reports that I did as a favor for a user a couple years ago. I don't mind since it's a real simple process, but the report relies on me running a query and dumping the results to a CSV file so I can send it to the user.

It's that time a year where I have to go through this process again and this year I decided to hand over the power to the user. From what I heard from other developers, it was a simple tool, so I gave myslef an extra week. I knew I could bang out hte CSV file in about 2 hours if I had to, so I basically had the whole week to work on it.

I started playing with Apex in the morning and in about four hours I had a basic report. In another day, I added some calendar pickers to let them enter a date range and some other dodads to give the user flexibility in how they wanted to filter the report. The best part was the "Spread Sheet" link that automatically downloads the report to a CSV file. Jeff - exit stage left.

I deployed it and gave the user the link in just under two days.

Apex Rocks!

Passwords

One of the things about IT security that really irks me is passwords. As a user I need a password for system X and a different password for system Y. Not only that, but system X requires a password at least 6 characters long with at least one alpha character and one numeric character. System Y requires a password 8 characters long, with two numeric characters and I can never reuse the same password. To add insult to injury, System Y's password expires every 45 days and system X's password expires every 365 days. I just changed my password for system Y to something I know I'll never remember.

Now, I know what you are thinking: LDAP server. Centralize the authentication and authorization and you only need to supply a password once. That's all fine and dandy when I have control over the security, but not when system X is where I do my online banking and system Y is my brokerage account.

Things are changing in the financial world, and not for the better, IMHO. At some sites, I have to answer a personal question every time I login. Others, I have to choose a picture before I even get to enter my password. Others still, I need an RSA key along with another password. I think there should be a standard of authentication practices that your personal trading partners should have to adhere to. I've got so many passwords in my head, I can barely remember how to login to work. In the time I wrote this post, I've forgot system Y's password.

Truth and the DBA

I don’t tolerate lying at all.

I don’t even want you to spin the truth. Give me the whole truth and nothing but the truth and we’ll be fine. An article about conducting business in an ethical manner by Bud Bilanich at Trump University got my attention. It’s worth a good read.

When I first moved up the ranks from a team member to a team leader, I had somebody that worked for me that skated on the edge of the truth quite often.

“How’s project the upgrade project going?” I asked.

“Fine. I’m right on schedule”, she answered.

“Last time I did an upgrade, the JServ configuration gave me problems. How did that go with this upgrade?” I countered.

“No issues.” She replied.

OK, I guess they fixed that.

Two weeks before the big upgrade, I asked again if we were on schedule and she replied “Oh yeah, probably be done in a week.” So I sent a note to the users about the upgrade and how we’ll need people here to test on Sunday to make sure everything is fine. The users got their army ready for Sunday, upper management was notified since they had been breathing down our neck for getting this project done as well.

Monday before the conversion came and I ask for the new URL so I can look at the new software.

“Not quite done yet, definitely this afternoon.”

Hmm, something is sounding fishy here. I looked at the machine and the database wasn’t even up yet. I poked around some logs and saw that certain pieces were failing to come up for various reasons. Did a quick search on Metalink and saw a couple resolutions to the issues so I didn’t think they were too serious.

Tuesday and Wednesday I was out for training, but left explicit instructions that if progress wasn’t being made I was to be notified.

When I got back Thursday, I went to get a quick status.

“JServ doesn’t work, the Concurrent Managers keep dying, and Apache dies when you hit the login URL” was the reply.

Needless to say, the upgrade was cancelled. Upper management was steamed and since I was the project leader, it was my fault. That person no longer works for me.

Granted, it was my fault for not asking the right questions. However, if they had been truthful about their progress and struggles they would have garnered much more respect and I could portray an accurate picture of the progress to upper management. Their spin on the truth (or outright lies) caused my group to lose a lot of respect from the powers that be.

That’s one of the many reasons I manage the way I do today. As Tom Kyte puts it, “Trust, but verify.”

The world of Spatial part I.....

I want to apologize even before I begin because there’s so much information that I feel I need to share with others that my topics may jump around.

Notice how I didn’t say “The world of Oracle Spatial”? That’s because there is an alternative in the land of GIS (Geographic, Information, System), you don’t have to use Oracle’s spatial module. The big dog in GIS is ESRI and if you use ArcSDE (component of ESRI) it has its own way of doing spatial stuff. Before I get down and dirty with the technical aspects of running and maintaining a spatial database, I feel that it’s important that you (the reader) know upfront you do have a choice of how you can mange your spatial storage.

*Steps up on soapbox*

The ESRI company started out as bunch of engineers who wanted to develop geographic software. The thought was great but, engineers have this idiosyncrasy about delegating work, they think they can do it all and it reflects grossly in their product. The first thing you will become distinctly aware of is that Oracle is kind of an afterthought in the eyes of ESRI, SQL Server is all that is holy with ESRI and the reason for that is because most of it’s customers run SQL Server. Why anyone would want to run an enterprise system with terabytes of data on SQL Server is beyond me (yes, I am Oracle biased). The second thing is Bind variables, they are unheard of, if you have a spatial database and you run ESRI get use to thousands of literal statements plaguing your shared pool . I have done battle for two years with these people and they just don’t get it. There are days where I really want to hop on an airplane, fly to CA and cause bodily harm to the development staff. End of rant.

*Steps off soapbox*

As far as using Oracle spatial vs ESRI “spatial” there are pros and cons of each, it’s up to you to decide which is best for your environment and skill set. I look at using Oracle spatial as kind of like “more moving parts”. The less “stuff” I have to deal with in a database the better, especially if it’s big. I figure if I let ESRI handle ESRI there’s less that can go wrong. I hang out over on ESRI’s forums and the number of Oracle Spatial problems are limited but when there are, they’re usually pretty bad and the question(s) go unanswered. As far as performance gained with Oracle spatial, the jury is still out on that one with me because I’m waiting to see what ESRI’s new release of 9.2 is going to be like. Apparently, they’re going to finally take advantage of the SDO_GEORASTER parameter. Right now they’re only using the SDO_GEOMETRY parameter which handles shape files. It’s like they developed a product, packaged it up, shipped it out, and forgot to put the CD’s in the box. Yeah, if you do use Oracle spatial (now) you get rid of your F and S tables but what’s the sense if you can only use half of the modules ability? It boils down to if you want to use Oracle Spatial when ArcSDE 9.2 comes out you’re going to have to completely drop your rasters and bring them back in (at least that’s the way I see it). I can see it now, managers across the nation giving birth to small farm animals when they find out they’re going to have to drop terabytes of data to take full advantage of Oracle Spatial because of ESRI laziness.

The last thing I want to do is go over anyone’s head when I’m on a roll talking about this stuff so if I mention something that you don’t quite get or want me to elaborate on, please speak up and I’ll be more than happy to pull the reins in sit for a spell.

Job Opportunity

Just learned of a great DBA opportunity in the Jersey City area. Contact Evan Lerman from IJC Partners LLC at (212)626-6920. From Evan:

FINANCIAL EXPERIENCE A MUST ORACLE 9I AND 10G LOCATION JERSEY CITY PAYS UP TO 120K BASE

QUALIFICATIONS:

Minimum of three years experience working as a Database Administrator.

Familiarity with Oracle and Microsoft SQL Server with emphasis on
Oracle. Knowledge of relational database concepts and standards, best practices and procedures relating to database administration.

Experience in financial industry, insurance industry or law firm a plus. Must have excellent technical skills and knowledge of Unix and Windows operating systems. Strong interpersonal, analytical and troubleshooting skills with superior verbal/written skills are required.

JOB Description:
The Senior Database Administrator directs and controls the activities related to data planning and development, and the establishment of policies and procedures pertaining to its management, security, maintenance and utilization. Sets and monitors standards; ensures that database objects, program data access, procedures and facilities are used properly. Advises management on database concepts and functional capabilities. Position is also responsible for installation and ongoing maintenance of enterprise server operating systems and system management products, as well as coordination of installation and upgrades to enterprise servers. Position also provides on-going production system support and performs other duties, as assigned.

Duties & Skills
A. Business/Application Knowledge

• Understands the company's general business functions, and has a conceptual understanding of each unit's activities.
• Has general knowledge of assigned application systems.
• Comprehends the relationships between business activities and application systems. Is able to determine impact of database changes to the application systems, and vice versa.

B. Technical/Programming Skills

• Builds and maintains all Corporate database environments.
• Builds and maintains test database environments.
• Is responsible for recommending and planning the installation of new releases of database software.
• Ensures the integrity of all physical database objects and established database procedures.
• Creates storage groups, databases, tables and views, reviews SQL, develops and enforces database standards.
• Ensures work is thoroughly tested and smoothly implemented.
• Is responsible for database performance and capacity monitoring and tuning. Prepares regular capacity analyses for management review.
• Assists in determining storage procedures for on- and off-site storage of historical data.
• Assists in establishing backup, recovery and restart procedures.
• Assesses the need for additional hardware or software to assist in monitoring or performance of database applications.
• Communicates availability requirements for database accessibility.
• Coordinates schedules and procedures for the implementation or discontinuance of relational database applications.
• Provides technical support and basic training in the proper use of production databases to database users.
• Assists in troubleshooting application problems where database management is an integral element.
• Mentors junior staff on database management techniques.
• Installs upgrades and fixes to server operating systems (UNIX, NT, etc)
• Analyzes and recommends upgrades and/or new acquisitions of hardware to support new systems or growth of existing systems.
• Coordinates vendor installation of hardware.
• Installs and utilizes third party system management software to monitor overall server performance and capacity utilization.
• Designs and implements backup schedules for critical databases.

C. Analysis Skills

• Provides ongoing research and development activities to investigate new technologies and tools which might be used by Company personnel to more effectively and efficiently perform their jobs.
• Performs functional evaluations of candidate products.
• Prepares time and cost estimates for assigned projects.
• Understands the Company System Lifecycle Methodology, and project development lifecycle.
• Acts as methodology and process mentor for junior staff, as they prepare project deliverables.
• Contributes to database design reviews.
• Develops and maintains a security scheme for the database environments.
• Assists in disaster recovery planning, testing and execution as needed.
• Possesses strong understanding of the system deployment process and correlation with database administration responsibilities.
• Coordinates and conducts database design reviews.
• Possesses keen troubleshooting and creative problem solving skills.
• Possesses the ability to translate user needs and projections into system hardware and/or software requirements.

D. Basic Skills

• Adheres to Company standards and methodology.
• Adheres to company confidentiality and security requirements.
• Communicates effectively.
• Consistently demonstrates a high level of integrity and professionalism.

The Co-operative

Things are changing in the home offices of "So What?". Some of my guest bloggers expressed an interest in continuing to blog about IT goings on and I thought "Why Not?". We're going to concentrate more on IT stuff on So What and I'll leave the personal stuff over at Wilton Diaries. I present to you, the So What Co-operative.

Oracle Spatial & Wildebeests

Before I make an effort to Blog about Oracle Spatial, Rasters, and Shape files is there anyone who reads this Blog that would benefit from me sharing my experiences with it?

Don’t get me wrong, I’m not saying my time is valuable and I have better things to do with it. I’m just trying to get a feel for what interests you (the reader). I’m sure there are those of you who would much rather discuss the migration habits of the West African Wildebeest during the dry season or how histograms for join predicates only work if you stick your tongue out in the right place. Sorry Dave, you know I have to mess with you :)

So Often I read peoples blogs and it looks like they threw up a paragraph or two of gibberish just to take up virtual space and to make it look like there’s activity. I refuse to succumb to that. You take the time out of your day/evening to come and pay this place a visit the least thing that the contributors can do is write decent content that will make your time spent here either enjoyable or knowledge gained. Nuff Said?

What do you do all day?

The conversation started out "What do you do all day?"

I was talking to a fellow IT worker and was trying to explain my job function. I often get this question from non-computer people and I just respond "computers", but this required a more in-depth answer.

I started off with "Basically, I make sure all those databases that the company uses stay up and functional."

"Ah....But what does that mean?"

So I start explaining my day.

My day typically starts with resolving any non-critical problems that happen overnight. I don't have to worry about the critical problems, because they have already been resolved by the person on duty (which is me every other week). I might create a new schema for a user in Europe. Or I might try and find out why userX tried to login yesterday over 300 times using the wrong password. This type of stuff usually lasts from 15 minutes to an hour. In my group, we each have about the same amount of this type of work in the morning. In addition, we'll respond to these type of quickie tasks throughout the day, recording each one.

Then I spend about 15 minutes catching up on how my people are doing with their projects. Sometimes more, sometimes less, but I generally like to get a feel of how things are going before I start doing my heavy duty work.

Most of my day is spend doing what I call "project work". Project Work are those tasks that can't be done in less than a day. A project might be as short as a day, or may be as long as 18 months. An example of a project might be as complex as upgrading Oracle Applications to 11.5.10 or might be as simple as setting up a connection manager for a particular sub-net. I typically schedule my work so I can work on two projects at the same time (ie. while Oracle Applications is applying patch XYZ, I write code for my monitoring software).

Occasionally, I'll have to respond to a critical situation. I have monitoring software running all the time and when it encounters something that it thinks I should know about, the software sends us a message. If a process is running over X minutes, I get a message indicating that maybe I should investigate more. If the software encounters a condition that could potentially stop business, I get notified right away using a text message. If my backups fail at 02:00, I get notified. If the log_archive_dest gets over 90% full, I get notified. On average, I get about three after-hours messages a week when I'm on duty.

I don't worry about backups, they're automated. I don't worry about my alert.log, it's being monitored. I don't worry about the database being up, it's monitored.

The other person usually wakes up from their coma at that point and says "Oh."

Weathervanes indeed

You may recall Steve's entry about weathervane theft in New England. At these prices, I can understand why.

On top of the world

I’ve never climbed a mountain, but seeing a sunset at 14,000 feet is a breathtaking experience. We took the Mauna Kea (pronounced Mona Kaya) Sunset and Stargazing tour by Hawaii Forest and Trail one evening. They take you from your hotel (sea level, temperature 93) on a 90 minute ride to the summit of Mauna Kea (14,000 feet, 38 degrees) to watch the sunset. The view did not disappoint as an unobstructed sunset was one of the most incredible things I have ever seen.

Mauna Kea is home to some of the world’s most powerful observatories. The stargazing portion of the tour was a 90 minute talk on the constellations and how ancient Hawaiians used these stars to navigate. Then the guide pulled out an 8” telescope and let us see some of the stars up close. The view of Jupiter was striking and we saw details on the moon almost like it was coming from Google Maps.

I’ll let the pictures do the talking, but if you’re ever on the big island, you won’t be disappointed with this tour.