Maybe I'm starting to get it

Yes, another R12 post. It's all I've been doing lately.

When I first installed 11.5.3, I just went with the default two-tier architecture; Web & Forms on the middle-tier and Reports & CCM on the db host. After a while, I realized that was a mistake for my environment. I should have put everything but the db on the middle-tier. Live and learn, I guess.

R12 was my chance to partially correct that mistake. I had a grand plan to migrate my middle-tier services to a hot x86_64 box and leave my db on Solaris. I figured that would give me an easier path if I needed to scale out either tier.

Except Oracle doesn't have a supported upgrade path on multiple platforms yet.

So I bounced a couple ideas off a very helpful Oracle Support Tech and he prefaced each one with "This isn't supported, but you might try...". I finally settled on a plan and executed it.

First, I collapsed my 11.5.9 tiers on to a single Solaris box. I used the Metalink whitepaper and rapidclone to get everything on my db box.

Then, I upgraded to 12.0.0 and applied the 12.0.4 RUP. At that point, I had a working 12.0.4 system with all my middle tier services on Solaris.

Next, I did a fresh install of 12.0.0 on a Linux box telling the configuration that I was going to have two tiers and my db was going to be on Solaris. Then I deleted my network topology and reran autoconfig that populated the db piece of my topology.

The next step was to apply the 12.0.4 patch to the Linux box. I did this with adpatch, but used the "nodatabaseportion" option because I knew I had already applied the object changes to my db.

The final step was running autoconfig on the middle-tier.

Except for a few glitches here and there, it was a surprisingly straight-forward split. Either I'm starting to get this Oracle Apps stuff or Oracle is finally getting serious about their configuration management.

I'm In!

I went round and round with Oracle Support creating spin-off TARs and transferring it from one group to another. I finally found an article that said one of the 11.5.x versions was really executing fnd_web_sec.validate_login() under the covers.

Might as well give that a try, SINCE I'M STILL WAITING.

So I ran the function on my GUEST user and sure enough, it came back with a "Y" which indicated the GUEST user and password were correct.

I then tried the same for my SYSADMIN user and the function returned "N".

Hmm, that's interesting, let me try it again.

Sure enough, my old SYSADMIN password didn't work for some reason. I posted the results in the TAR and the analyst suggested I change my SYSADMIN password with FNDCPASS.

I changed it, and was in!

If this little tidbit of an upgrade is published somewhere, it sure wasn't obvious to me.

Of course, now I have to figure out why anything using the forms servlet won't come up. And how to use WebADI. And why my printers aren't defined.

But at least I'm in!

Fun & Games, Part III

The main TAR had been in "Review" status for 24 hours, so I duty managed it. I usually don't like to duty manage things because if you cry wolf enough, eventually nobody will listen.

The analyst updated the TAR and wanted the results from about a dozen URLs along with another half dozen log files.

Aha, trying to bury me, are you?

After an hour of trying the URLs (which 90% of them were from 11.5.9 and didn't work in R12), I assembled the output and packaged in a nice neat .tar file. As expected, it went back to "Review" status.

Yesterday I had something else to do, so I wasn't very diligent in following up. Which was OK, because there was nothing to follow up with. My TAR remained in "Review" status for the whole day.

This morning I had some free time, so I duty managed my TAR again. Within 20 minutes, the analyst handed the TAR off to another group by creating a "spinoff" and marked the original TAR as "Awaiting Internal Response".

It will end up being a 2 second fix, I just know it. A week later, and I'm still waiting for the slightest bit of help.

The Fun & Games Continued

Eventually my TAR from R12 Upgrade Fun & Games suggested that I rerun autoconfig for my DB Tier and I ran into another problem. Since it was not related to the original TAR, I opened a new one and put the original TAR in "Waiting for Customer" state until my new problem could be solved. Interestingly enough, the root cause of the new problem was that I had my $ORACLE_HOME in a different location than rapidWiz thought it should be, so rapidWiz put my newly created context file in the wrong location. I was trying to run adconfig.sh with an 11.5.9 context file, which kept giving me an error about version compatibility. Fair enough, I was back on track in about two days.

So I went back to my original TAR and posted that while I ran autoconfig on my db tier and bounced all my services, that still didn't fix my issue.

And I wait...

While waiting, I found Stephen Chan's blog that pointed me to Oracle Application Server Diagnostic Tools and Log Files in Applications R12 (ml note 454178.1). Definitely something to test, SINCE I'M JUST WAITING.

...to be continued... (hopefully)

R12 Upgrade Fun & Games

So I've been toying around with an Oracle R12 upgrade lately. The first attempt went miserably wrong with all sorts of issues when I applied RUP3.

Then I decided that upgrading my middle-tier to Linux x86_64 sounded like a good idea. Except it wasn't. You see, first off 11.5.9 isn't supported on x86_64, so you can't add a node to your existing 11.5.9 instance. And you can't install an R12.0 base on x86_64 and then upgrade your 11.5.9 db, because that too is not supported. I ended up collapsing my two-tier environment to a single tier so I can upgrade to R12 and then add an R12 node that runs x86_64 later on. That, I double checked, is supported.

I performed the 11.5.9 to R12.0.0 upgrade again on my test system and applied the RUP4. Surprisingly, I only encountered about four problems. Two of the problems were not really problems because I wasn't running those modules and two problems that I researched on Metalink and quickly resolved.

When I brought the whole thing up (on a single tier), all the middle-tier services came up quite fine. Yes, I was surprised too.

The first time I tried to login is when my problems started. The ominous "Login failed. Please verify your login information or contact the system administrator" message comes up. Hey, I am the system administrator, WTF I am supposed to do from here? No errors, no messages, no nothing.

No problem, I'll just go to my standard diagnostic tests that I used in 11.5.9 to see where my problem is. Wait, there's no FND_WEB.Ping anymore. Nor is there any isItWorking servlets.

Finally, I find http://hostname/OA_HTML/jsp/fnd/aolj_conn_test.jsp and run through the tests. Surprisingly, everything but the AOL/J connection test works.

"Aha, I'll provide that information to Oracle Support and they'll be able to tell me where to look right away.", I thought to myself.

"Please upgrade to the latest RUP" was their response.

Me: "Errr, I am on RUP4."

Them: "Please rerun autoconfig."

Me. "Errr, done, but that doesn't solve my problem. Besides, my AOL/J connection test doesn't work. What does AOL/J do and why would it fail with this error?"

Wait.

Wait.

Wait.

Me: "Um, this is kind of urgent. When I try to login to the forms server directly I get 'APP-FND-01542: This Applications Server is not authorized to access this database.'"


Wait.

Wait.

Me: "I found that if I change my s_appserverid_authentication to OFF in the context file I get a different error logging into the forms server. Now I'm getting 'Error: validateLogin returned false; failed to login as XXX, please try again'"

Wait.

Wait.

Me: "Is anyone looking at this TAR?"

Them: "Sorry for the delay, I am looking at your log files"

Wait.

Wait.

...to be continued.

Patch 4502962

Seems as though there's no version of the R12.AD.A minipack for Linux X86_64. Yet, since there's no binaries that get changed for this patch, you can run the 32bit version without a problem.

R12 Upgrade, Attempt #2

My first upgrade was a total bomb. The R12 portion went OK, only three minor problems that I corrected as the patching was taking place. Took about three days, but it finished with about 40 or so objects invalid (which I consider pretty good considering the umpteen thousand that are in the database).

Instead of just going with the 12.0.0, I decided to apply the RUP3. Might as well be up-to-date when you upgrade, right? While applying the RUP3 patch, it filled up my filesystem. I started deleteing some logfiles so I could continue, but couldn't keep up with adpatch. It left my db in a state of total chaos. Did I mention I didn't backup after my R12.0.0 upgrade was complete?

I noticed that during the R12.0.0 that I was using a decent amount of CPU on the middle-tier and my db tier was busy but not pegged. I swapped out my Solaris middle-tier for an X86_64 middle-tier to see if that speeds up the process.

One thing that came up during the initial install was adrunias.sh was trying to start Java from /tmp/RapidInstall/jre/Linux. The rapidwiz kept throwing an error of RW-50010 which was totally non-descript. I finally ran adrunias.sh with ksh -v to see what it was really doing and found it was looking for the jre in /tmp/RapidInstall/jre/Linux but the only directory was /tmp/RapidInstall/jre/Linux_x64. A soft link cured that problem and I am continuing on my way. Lets see where this takes us...

Upgrading to R12

Upgrading Oracle Applications is never an easy process. I like to test my upgrades both for completeness as well as performance so I can give my users a semi-accurate picture of how long things will take.

I'm doing a direct upgrade from 11.5.9.CU2 to R12 (probably CU3). I got to the point where the db objects are upgraded and received the following message:

There are now 126521 jobs remaining (current phase=A3):
 6 running, 951 ready to run and 125564 waiting

OK, then. When I upgraded from 11.5.3 to 11.5.9, the object upgrade portion took about 26 hours, but I was on a slower box (Sun 2x900 with 8G RAM) and Oracle 9.2.0.7. Now I'm on a Sun 4x1200 with 16G of RAM and Oracle 10.2.0.3. This is only my first attempt at upgrading and I'll look at the AWR stats after to see where my bottlenecks were during the upgrade.

Any guesses on how long this upgrade run will take?