Author Topic: Server 3.1 Released - Early impressions  (Read 6238 times)

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Server 3.1 Released - Early impressions
« on: March 17, 2014, 09:09:52 PM »
As with all updates of OS X or Server.app, make sure you are planning for problems.

1:  Confirm that you have a regression plan.
2:  Check all your backups.
3:  Don't try to "do this quick" as that usually means a blown weekend.
4:  Stop as many services as you can (basically everything except DNS and Open Directory)
5:  Reboot the server.
6:  Install the update.
7:  Reboot the server.
8:  Turn services on and test.

Remember, if you are running a mail server, shutdown external access to ensure that the data stores are not altered.

Ok, that being said, I've already seen my first "failed" upgrade.  It really was not a failure, but it was not as expected.

When running Server 3.1 for the first time, make sure you have Console open and you are watching the system.log (or tail -f /var/log/system.log).  Watch the progress of the upgrade.  If the progress bar gets to the end and never completes, look for the following in your system.log:

servermgrd[77950]: [77950] error in getAndLockContext: flock(servermgr_info) FATAL time out
servermgrd[77950]: [77950] process will force-quit to avoid deadlock
servermgrd[77950]: outstanding requests are: (
           {
           Command = getState;
           Module = "servermgr_info";
           Thread = 4543123456;
           Timestamp = "2014-03-18 00:07:43 +0000";
       },
           {
           Command = Idle0;
           Module = "servermgr_info";
           Thread = 4543123456;
           Timestamp = "2014-03-18 00:07:43 +0000";
       },
           {
           Command = Idle;
           Module = "servermgr_netboot";
           Timestamp = "2014-03-18 00:08:43 +0000";
       }
   )
servermgrd[77950]: blameOldestRequestForModule: couldn't find a thread to blame for servermgr_info
com.apple.launchd[1] (com.apple.servermgrd[77950]): Exited with code: 1

If you see the exit status 1, it means that the servermgrd process has completed and your progress bar will not complete.  Reboot the server and then launch Server.app and confirm that each service is working as expected.

I know 1.2 just came out but I will submit a 1.3 to Apple by the end of the week to address any Server 3.1 changes.

Francesco DellaPorta

  • Newbie
  • *
  • Posts: 14
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #1 on: March 18, 2014, 12:00:21 PM »
One impression and one question to share.

1. Server update
Strictly following the eight steps, turning all relevant services back on -- one by one with an eye on the system.log. All went smooth till the Profile Manager service, which yet came on and running. However with some symptoms:

a. A new local group account "appeared" in the group list. Its name, hold on...
"Deprecated Profile Manager Access Group"; Group ID: 1000; Account Name: deprecated_pm_access_61e93cd7. Its members are diradmin, localadmin, and all other network accounts. Weird!

b. A repeated set of log entries. Just like this... (server FQDN intentionally removed)
Mar 18 16:15:06 share kernel[0]: Sandbox: xscertd(339) deny file-read-metadata /private
Mar 18 16:15:06 --- last message repeated 69 times ---
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: DeviceInformation  for: <Device>
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: Restrictions  for: <Device>
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: CertificateList  for: <Device>
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: ProvisioningProfileList  for: <Device>
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: ProfileList  for: <Device>
Mar 18 16:15:06 mdmclient[81]: [Daemon:0] Processing server request: InstalledApplicationList  for: <Device>
Mar 18 16:15:07 mdmclient[81]: === __40-[CKClientDispatch _xpcConnectionDirect]_block_invoke: An error was received (Connection invalid).
Mar 18 16:15:07 mdmclient[81]: [Daemon:0] Processing server request: SecurityInfo  for: <Device>

2. Question
Quote
Remember, if you are running a mail server, shutdown external access to ensure that the data stores are not altered.
Given that, by external access it is meant mail client users trying to access a temporary unavailable mail server, what is intended with shutdown and how is that accomplished? 

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #2 on: March 18, 2014, 03:45:45 PM »
I hope your Profile Manager heals itself.  I've gotten three more servers complete and aside from the first one, the rest have gone as one would expect.  No surprises.  And yes, there is a ton of log information that makes you scratch your head.  And how about that.  I missed that deprecated group in DS Local.  I am sure we will never find documentation on the meaning of that.

So to your question.

When I do anything that is associated with mail, I tend to;

a: perform it on off hours to limit the number of users who might be using it
b: temporarily close firewall ports to prevent external connections (smtp, imap, http).  This ensures that no new mail comes in and that no clients are connected.
c: log off all LAN users if possible.  This reduces or eliminates the number of open mailboxes and ensures that if there is a problem
d: clone the server (if possible) or replicate all service data to recoverable formats

Then I will go through my procedure and apply the update(s).  Once the unit comes back up, I start up and confirm each service is performing as expected.  If so, I will then open the firewall back up, allowing the flow of mail.

For customers without an SMTP backup, I generally can complete the process before a sender server will expire and bounce a message.


Francesco DellaPorta

  • Newbie
  • *
  • Posts: 14
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #3 on: March 20, 2014, 12:08:24 PM »
Follow-up...

1. Server update
Yes, the Profile Manager did heal. However, it needed some manual intervention. Moreover, the deprecated group in DS Local is due to the new profile manager feature which allows device enrollment during Setup Assistant. Please let me detail.

a. During the server update all relevant data concerning profile manager have been successfully migrated. Check Logs under Profile Manager > Migration Log for a confirmation. Server 3.1 brings in a new feature which introduces the possibility to "Allow device enrollment during Setup Assistant". These settings need to be broadcasted to the managed devices, if so chosen. In the migration process a new profile for Everyone has been updated. Look for it under Profile Manager > Configuration Profiles > Default Configuration Profile with the name "Settings for Everyone". All of these changes are obviously almost invisible from the Server.app GUI.

Risking of being proven wrong, I would say that from the profile manager service point of view, the new local group Deprecated Profile Manager Access Group is the old local group Everyone.

b. The recurring set of system.log entries posted herein are about and around the mdmclient (Mobile Device Management client) command/process. Opening Profile Manager and issuing an Update Info (gear menu) request to the devices stops the recursive logging. Eventually it also solves the problem, if any.

I would recommend to add this step in the server update procedure: if you are running a profile manager service, then send update info to all managed devices.   

2. Mail server access shutdown re: Question
Many thanks for your answer. That is more than appreciated. Very well detailed.

So, if the firewall under question is... please let me name it... an AirPort device, then the external access may be temporarily shutdown by removing Mail from the list of available Public Services. Once that the Mail server is up and running, and ready to manage traffic, then it is time to add Mail back to the list of AirPort services.

If anything else, then please see the attachment AirPort_Network_Mail_2014-03-20 for the exact port numbers and types deployed by the Mail server running on OS X Server 3.1.

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #4 on: March 20, 2014, 09:18:38 PM »
I was poking around some more and comparing a 3.0.3 server to a 3.1 and it looks like the mystery of the Deprecated Profile Manager Access Group is starting the come clear.  This was the old SACL group com.apple.access_devicemanagement.  That no longer exists and you can not longer grant or revoke access to the Profile Manager service through Server.app.  You must log into Profile Manager and grant the access from there.

And the mail server stuff.  I am lucky enough not to support anyone using an Apple Airport as a firewall.  :)  But you are correct.  Removing the ports from the port forwarding sections is the same as disabling a firewall rule.  You are effectively preventing the traffic from coming inbound and potentially causing a data set variation that you can not recover from.


urban420

  • Newbie
  • *
  • Posts: 11
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #5 on: March 23, 2014, 01:19:09 PM »
I updated our server last week and everything seemed to go as it should, but I did notice the new group that was created - "Deprecated Profile Manager Access Group".

Being new to OS X Server and not yet using Profile Manager on our server I am a little confused by the info you guys have posted regarding this group that was created. Am I correct in assuming this group can safely be ignored, but you now need to use the Profile Manager portal to grant or revoke access to services?

Kind of funny how Apple just assumes there is no need to tell users about this change. This might have been something to include in the release notes?

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #6 on: March 23, 2014, 06:55:27 PM »
If you are not using Profile Manager this group does not impact you. 

Also, only the SACL for Profile Manager has been moved into Profile Manager.  All other Service Controls remain in Users and Groups.

And yes, better transparency with changes has always been requested.  Apple simply never seems to be very interested in providing details.  You should see the sorry excuse for release notes with the beta program under 10.9.x.  You have no idea what to test.  It is all self-discovery.

Francesco DellaPorta

  • Newbie
  • *
  • Posts: 14
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #7 on: March 27, 2014, 04:30:39 AM »
Apple released on Monday, Mar 24, 2014 the OS X Server v3.1.1 software update. It replaces and contains all improvements included in the Server 3.1 release.

Among other things, it fixes the issue describe earlier in this post: 1. Server update, b. running Profile Manager causes "recurring set of system.log entries".
Other symptoms of the issue are:
  • extra cpu activity on the server
  • extra cpu activity on managed devices
  • quick battery drain on managed iDevices
Measures
  • Sending Update Info requests to all managed devices did not solve the issue. It just postponed (of about 24 hours) the unnecessary recurring process.
  • Removing and then enrolling again all managed devices did not solve the issue. It just caused unnecessary administrative work
  • Wiping up the database by means of wipeDB command din not solve the issue. It just refreshed the profile manager settings and its database.
Solution
Update the Server.app with the latest OS X Server v3.1.1 software update.
 
Quote
Server v3.1.1 fixes an issue that could cause Profile Manager to be unresponsive or generate extra cpu activity after updating to Server 3.1.
Source: http://support.apple.com/kb/HT6172

Francesco DellaPorta

  • Newbie
  • *
  • Posts: 14
    • View Profile
Re: Server 3.1 Released - Early impressions
« Reply #8 on: May 21, 2014, 02:44:10 AM »
Apple released on Tuesday, May 20, 2014 the OS X Server v3.1.2 software update.

Quote
What’s New in Version 3.1.2
• Calendar Server improvements for imports, invites and group scheduling
• Improvements to Messages Server stability when using Chat Rooms
• Fixes for Profile Manager deploying profiles containing variables when code signing is enabled
• Improved Profile Manager reliability for sending Volume Purchase Program invitations
• Fixes to enable Profile Manager to manage Device Enrollment Program systems with long descriptive names
Source: Mac App Store Preview, https://itunes.apple.com/ch/app/os-x-server/id714547929
« Last Edit: May 21, 2014, 02:46:41 AM by francesco.dellaporta »