Author Topic: Getting Network Homes to work over Network  (Read 2030 times)

adi007

  • Newbie
  • *
  • Posts: 5
    • View Profile
Getting Network Homes to work over Network
« on: November 30, 2015, 02:52:08 AM »
Ok, i think i am going crazy here or i just need to change direction....

I have been in OSx game for over 12 years now. I manage many 10.6.8 OS X servers (as i refused to move) and over 120 computers; I am completely stumped now, so i bought all your books.... I want to move to 10.11 and Server 5

I am trying to do what is nice and easy in 10.6.8, create home share on a different server than where open directory is:

OD server is on odserver.example.com

Home Network file server is on fileserver.example.com

Everything works fine and as per your book, DNS, DHCP,OD etc. However, when i create an account and point home folder to the share on fileserver.example.com, i only get three folders created from the skeleton initially, and when i try to login i get the message that i cannot login this time.

If i create home share on local hard drive on odserver, everything is fine, but not any other way.

Any clues?

Thank you

Adi

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #1 on: November 30, 2015, 08:07:55 AM »
Thanks for reading and I hope I can help you out.  I think I can point you in the right direction.  But first some questions.

1A:  Are you working on an all 10.11 test environment?  In other words, are the odserver and fileserver both 10.11 with server 5?  Or...
1B:  Are you working with a 10.6.8 core and a 10.11 file server?  In other words, is the odserver 10.6.8 and the fileserver 10.11?

2:  How is the fileserver associated to the OD domain?  Replica?  Member?

3:  Which file sharing protocol are you using to provide network home folders?  AFP or SMB?

4:  What client version are you trying to login with?  10.6.8 or 10.11.1?

Next some observations.  Back in the 10.6.x days, it was possible to join servers to the domain and allow them to participate are network home folder systems without the overhead of OD replication.  The net home server would publish an auto mount record and WGM would see it allowing you to define home folders on remote servers.  With Yosemite and above, I've never been able to get this to work properly.  Instead, I've been creating replica servers for each remote net home server.  Ah, but it gets a little more funky.  Let's say you have nethome1 (OD Master) and nethome2 (replica).  Both are offering net homes.  Running Server.app on nethome1 will NOT offer the network home folder created on nethome2 as a home destination.  Examining LDAP will reveal the published mount records but Server.app simply does not show it.  You need to turn Server.app on nethome2 and set home folders on the local server.

Next, the protocol.  I know Apple wants to move to SMB but good gravy, it still stinks.  (Truth be told El Cap has improved it quite a bit but...)  So far I have kept all net home deployments on AFP and will continue to do so.  Applications that are known to support net homes (and even those that don't) are more predictable with AFP. 

Ah, and then the bane of us OS X admins... Cross version support.  This is one of those thorny topics that I avoided on purpose.  While you can mix a match OS X Server versions for certain tasks (old OD server with new file server) it all falls apart if you try to get too fancy.  If you are trying to make this work with a mix of OD and OS versions, I fear you will continue to believe you are crazy when you are really just swimming against the tide.  When possible we strongly encourage unified OS and Server versions.  I know it is not always possible, but there are certain limitations that will make you scream.

Now, with a 120 system deployment and 10.6.8, I will bet you are in education supporting units on a cart.  If this is the case, don't try to change course in the production environment between now and June.  This is your time to build a test network and validate the migration to El Cap.  I can say that it is successful.  (In many cases I prefer Yosemite deployments as they are the most stable net homes I can ever remember setting up, even better than Snow).  Do what you can to cobble together some test gear and setup a new OD master and at least one file server.  Bring over a handful of clients and test and test.

Let's start there.  Give me some more details on the environment and let's see if we can devise a plan to solve your challenges.

adi007

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #2 on: November 30, 2015, 03:59:26 PM »
Yes your books are an amazing, easy to understand and helpful insight into OS X El Capitan server environment.

So answers to your questions:

1A: Yes 10.11.1 test environment all the way. All servers are 10.11.1 and Server.app is v5, as well as my clients are 10.11.1
1B: In the test environment everything is 10.11.1

2: I have tried both ways, Replica of master and yesterday i reinstalled fileserver and just joined to the master via System Preferences and allowed File Sharing on fileserver via Server.app v5
Today i will reinstall both again and try again with your advice later in the day (a bit presumptions of me :)

3: AFP all the way, not sold on the SMB yet; i would try NFS, but not 100% sure

4: Clients will all be 10.11

Yes, your observation is correct. 10.6.8 is easy to setup with this situation and works well, no issues there, i am almost thinking of sticking with it and just serve homes to 10.11 machines, while maintaining Profile Manager for computer/user settings, on 10.11 Server.app v5! The situation with Server.app you describe is exactly what i was getting in my test results, What i also found is even if you create Replica of 10.11 OD master, you have to bind the client to Replica, otherwise you would not get the access to network homes that is offered through Replica File Sharing.

Do you think that is intentionally done by Apple engineers or could it be a bug in the Server.app system or is Apple pushing customers away from Network Homes?

I have always preferred AFP, since the first 10.something....

My frustration comes from Apple moving away from Server software and hardware, when they had a really good thing going with 10.6.8 server and then decide to get the blasted 10.7 and then 10.8 out. I have now decided it is time to let go of Xserve’s and move into 10.11 Server.app space as it looks promising and stable, but.....

Yes, we are a school and based in Australia, so my testing time is now until the end of December :) We have 120 iMacs across the college, 3 Xserve’s and 2 Mac Mini servers. For next year, the plan is to go to ESXi VM space with two black vases (Mac Pro cylinders :), so i am vigorously testing and recording my highs and lows haha

I can confidently say that running 10.11 in VM is a breeze and ESXi with vSphere 6 is working nicely.

I hope this info helps and speak soon,

Adi 

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #3 on: November 30, 2015, 08:34:47 PM »
Sorry for the slow reply.  Had a full day engagement today so everything else gets put aside.  Ah, nothing like deployments in financial services companies...  Can't download anything because the networks are locked tighter than a drum.

Ok, let's go point by point.

1:  Excellent.  Always better to work with a unified branch that trying to stitch version together.

2:  So 10.11.1 and Server 5.0.15 clearly changed how some things are done when compared with the beta cycle and the initial release.  It is possible you are running into one of those oddities.  I know network homes is part of what I test for the books as it is something I support in the field.  All things being equal, we should be able to get this to work.  More later.

3:  Agree.  SMB has burned me more times that I can count.  While Mac to Mac SMB is almost usable, the debacle of 10.7 through 10.9 nearly got me kicked out of a couple large enterprises.  At the same time the Xserves where being retired and data was being migrated to windows servers, Apple destroys the SMB client.  Not a good couple of years.

4:  Excellent.  Once again, keep everything consistent is easiest for testing.

Quote
..you have to bind the client to Replica, otherwise you would not get the access to network homes that is offered through Replica File Sharing.

Do you think that is intentionally done by Apple engineers or could it be a bug in the Server.app system or is Apple pushing customers away from Network Homes?

With Yosemite and above I recommend always binding to the replica.  In the old days, a machine bound to the domain would respect the entire OD tree and seek out a controller should the primary be lost.  This no longer seems to work.  In order to get the full tree you must bind to the replica.  I don't think Apple ever publicly revealed this tidbit but it really does make a difference.  So add binding to the replica as your standard process.

Now, the next statement is clearly my opinion.  But it is based on a number of observable facts.  I do believe that Apple is making the support for Network Homes more difficult in a number of ways and for a number of reasons.  For example, Apple truly believes that everyone should have her own machine.  The iPad exemplifies this.  There is no user account.  It is yours and you wan do with it what you please.  This mentality is being applied to OS X in a number of ways.  They include DEP and VPP.  The assumption of both of these programs is that the end user is the devices administrator.  There is no unified admin account.  There is no centralized authority that managements and maintains the device.  The end user does these tasks.  With DEP, hand a device to an end user and let them set it up.  If the MDM is setup properly, you deliver everything required via an on enrollment policy.  Then, if you believe that the entire OS X ecosystem exists in the App Store, then VPP once again removes management from the IT group.  You invite the user, assign apps, and the rest takes care of itself.

Now you and I live in the real world.  The world of Microsoft, Adobe, plugins, and edu-tainment software that is not available via the App Store.  We also live in the world were schools can no afford to have a device for every student and device sharing is a requirement.  Apple has established a long history of providing NetBoot and Network home technology.  Heck this stuff was possible with OS 9.  And when you consider the advances made since then (802.11ac vs 802.11b, 1000Base or higher vs 100Base, SSD drives vs IDE, etc) there is no reason NetHomes should not be well supported and highly successful when considering today's wireless networks and device technology.

Ah, but the other question is who is Apple encouraging to make software that is Nethome aware?  If Microsoft and Google are struggling to do it, you can pretty much be sure that all these mom and pops making apps in the App Store are not testing for it.  I fear that Apple's vision of the future is a Mac in every hand and a directory-less deployment.  No binding to domains.  Only enrollment to management.

Ok, you got me on my soapbox.  Back to your issue.

Quote
My frustration comes from Apple moving away from Server software and hardware, when they had a really good thing going with 10.6.8 server and then decide to get the blasted 10.7 and then 10.8 out. I have now decided it is time to let go of Xserve’s and move into 10.11 Server.app space as it looks promising and stable, but.....

I've been there and I've come to terms with it.  But yes... 10.7 was crap.  10.8 was better and we still have many customers on it.  10.9 was the skipped OS for many reasons.  10.10 has proven to be my favorite.  10.11 is growing on me but the fourth quarter of the year is always slow for customer upgrades.  January and February are the months to watch for.  By then I expect 10.11.3 at least and a few fixes to Server.app.

Quote
Yes, we are a school and based in Australia, so my testing time is now until the end of December :) We have 120 iMacs across the college, 3 Xserve’s and 2 Mac Mini servers. For next year, the plan is to go to ESXi VM space with two black vases (Mac Pro cylinders :), so i am vigorously testing and recording my highs and lows haha

I can confidently say that running 10.11 in VM is a breeze and ESXi with vSphere 6 is working nicely.

Aha!  Sorry.  I was assuming US school calendar.  Hello from the other side of the planet.  :)  Very nice on the ESXi setup.  I explored that with some spare 2009 Xserves a while back and was very please with the performance as well.  I never invested in good storage though so it was really just an experiment.  Here is another area Apple could improve.  Imaging if OS X Server could be run on non-Apple hardware!  Oh man, talk about the instant acceptance by enterprise.  Oh to dream.  Heck, I would even pay for that version!

Ok, no really back to the issue.

So, in my mind you should be able to do the following.  I don't have enough gear with me tonight (on the road) so I can not actually test this.  But, this is the junk that rolls around in my head so I have the implementation thought through.

1:  Build the master.  Ensure that DNS is setup for both the master and the fileserver.  Then create an OS Master on the master server.
2:  Build the replica.  Ensure DNS and time are in compliance and create a replica on the secondary server.
3:  Test by doing the following:
a:  Create a user on master and watch it replicate down to the replica - user can be a Services only account.  Let's call this user Chief Master
b:  Create a user on the replica and watch it replicate up to the master - user can be a Services only account.  Let's call this user Carbon Copy
4:  Create a network home share on the master using Server.app on Master - Do not add any ACEs as the POSIX should be enough.
4a:  Start file sharing on the master server
5:  Create a network home share on the replica using Server.app on replica - Do not add any ACEs as the POSIX should be enough.
5a:  Start file sharing on the replica server
6:  On the Master, edit Chief Master and set his home to the net home share available on the Master.  (You will note that the one from the replica is not visible anyway)  Check the net home share and you should see Chief Master's home folder created.
7:  On the Replica, edit Carbon Copy and set his home to the net home share available on the Replica. (You will note that the one from the master is not visible anyway)  Check the net home share and you should see Carbon Copy's home folder created.

8:  Don't start any other services.  Yes, I know, Profile Manager is needed to redirect those cache files.  And Profile Manager can simplify the binding process.  But for now, let's just focus on the core items.  You should have on Master:  DNS primary (if no DNS is available elsewhere), Open Directory Master, and File Sharing with a Network home share available over AFP.  On the Replica you should have:  DNS secondary (optional and if no DNS is available elsewhere), Open Directory Replica, and File Sharing with a Network home share available over AFP.

9:  Now it is time to focus on a client device.  Do the following:  Open System Preferences > Users & Groups.  When you go to bind, press the Directory Utility button and bind the workstation using that tool.  Yep, I know, it all should be doing the same thing... But I just don't trust the simplified System Preference method.  Bind to the REPLICA!
10:  Once the workstation is bound, confirm that you can see the user accounts.  Open Terminal and enter:
id cmaster
id ccopy
Replace the short names with the ones in your test environment.  You should be able to get basic account data for both.
11:  Logout or reboot.
12:  Try to login as each user.  What is the result

So summary stuff.   The creation of the user and the definition of the user as a network home folder user should create the user's home folder within the net home share.  You should not need to login to create it.  If you are getting a partial share I will suspect you have permission problems.  What type of storage are you creating the net home shares on?  If external are you enforcing permissions?  If you are ignoring permissions you will end up with a mess.  If you are enforcing an ACE on the parent you will end up with a mess.

Ok, homework time.  I will stop here as this can give you a start.  Once I am back to my lab I will build this exact scenario, adding any details I may have missed.

Quote
Yes your books are an amazing, easy to understand and helpful insight into OS X El Capitan server environment.

Awesome!  If you have some time, please write reviews in the iBooks Store.  I would appreciate that.  Also, if you catch any errors or mistakes, please let me know.  I will credit you with the correction. 

I will have new releases for Book 1 and Book 2 when 10.11.2 drops.  Book 1 adds a section for multihoming, corrections, and minor additions.  This round for Book 2 will be mostly corrections (dang my feeble editing).  And Book 3 should be out before the end of the year. 

Let's start here.  Look forward to your status report.

Reid

adi007

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #4 on: November 30, 2015, 09:45:24 PM »
Hi Reid,

Thank you very much for this extensive reply. Today is my early day and i will be back in lab tomorrow, so i will post success or more questions :)

For now i am using a thunderbolt LaCie drive as a test drive, but the storage for homes will live on iSCSI 10GB fibre storage unit. Yes, from memory i think the checkbox "ignore ownership on this volume" is checked, if that is what you mean.

I will play with it all again tomorrow and then report back!

Thank you again for quick replays and for not ignoring questions!! Very rare these days...

Adi

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #5 on: November 30, 2015, 11:02:01 PM »
Quote
Thank you again for quick replays and for not ignoring questions!! Very rare these days...

I am committed to my mission.  Your success with Server means more use.

Make sure the ignore ownership box is unchecked for any drive connected to server.   You want to be able to enforce permissions.  Not on the parent home share but on the subfolders.  Otherwise (in theory) any student would be able to access any other students work.

Let me know what you find.  As mentioned, I will replicate in my environment as soon as I am back to lab (Wednesday).

adi007

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #6 on: December 01, 2015, 09:20:40 PM »
Hi Reid,

So following your instructions, I have to say it all worked very nicely.

I have created OD Master and users created from there, can login to their accounts with out a problem. All folders are available and accessible.

Same thing with the Replica, I created the Replica and accounts created on there are accessible as well, without any problem.

Client is bound to the Replica server and all is working very well. So, so far so great! Your instructions have solved my prior issues and I could definitely see this work in our environment.


Thinking ahead now, do you think that with current Server.app we could have a way to create high availability of Homes on the network? You can also be honest with me and tell me that I am asking apple Server.app to do too much ☺ however we all love pushing the boundaries!

With this scenario that we have setup now, if OD Master fails, users from there will loose connection to their homes or if Replica fails, users from there will loose their connection to homes, until we reboot or in worse case scenario, rebuild and move the network home share from failed server to the new server.

This year has been nuts with AFP crashing (could be because all homes are on 10.6.8 and computers are all 10.10.5), and server reboots….

Scenario one, my ideal situation for network homes would be iSCSI attached storage to a file server that OD Master and OD replica could connect to and in case of any one of the OD servers crashes, there would be no loss of network homes.

Scenario two, could be what you propose and I could just break up Students homes on to Replica_1 and Staff/Admin homes on Replica_2, with failovers set up in VM ESXi.

Am I dreaming or could we get this to work?! What are your thoughts?


It’s refreshing to have a great discussion that actually leads to an awesome outcome! Thank you again

Adi

Reid Bundonis

  • Administrator
  • Full Member
  • *****
  • Posts: 107
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #7 on: December 01, 2015, 10:19:42 PM »
Awesome news!  I am very glad it all came together.  As mentioned, I am enjoying great success with both Yosemite and El Cap in net home deployments.  Only one production El Cap due to the US school year as school starts in September and El Cap was not released until end of Sept.  But this summer I upgraded 6 schools to Yosemite and so far so good.  Not one adjustment has been made in any of the districts with the exception of software updates.  Really solid performance and AFP has been humming along like a champ.  Once school still has not done a server reboot...  They asked me to come in before the holiday break and "oversee" the reboot.  Very happy with the deploys.

Quote
Thinking ahead now, do you think that with current Server.app we could have a way to create high availability of Homes on the network? You can also be honest with me and tell me that I am asking apple Server.app to do too much ☺ however we all love pushing the boundaries!

With this scenario that we have setup now, if OD Master fails, users from there will loose connection to their homes or if Replica fails, users from there will loose their connection to homes, until we reboot or in worse case scenario, rebuild and move the network home share from failed server to the new server.

I will admit I've struggled with this for a long time.  Remembering the days in which failover was part of OS X Server... This is a really challenging problem because of the way everything stitches together.  Let's look at it from a naming and path process. 

You create two servers, one named mac1 and the other mac2.  They get a DNS identity such as mac1.elcap.com and mac2.elcap.com.  These are independent systems with unique IP addresses. Next, you chain on some storage that is uniquely connected (direct attach) to each server.  On the storage you define a folder and then the student homes get populated.  Then, you create users and assign users an NFSHomeFolder attribute pointing to a DNS name and a file system path.  The name and the path is unique to the box and trickles down to an IP.  The storage is in isolation to the host. 

So, mac2 decides to go in the weeds for a while.  Ok, under the typical model above you could move the storage to mac1 and change all the NFSHomeFolder attributes for students once on mac2 to now point to mac1.  Or, you can change DNS to point mac2 to the same IP address as mac1, allowing no change in the user records.  Either method can resolve an outage in a relatively short time but it requires physical relocation of the data or possibly a flush of mDNSResponder to recognize the DNS change.  Possible.  Not automatic.

Ah, but what if it is storage that fails and not the host?  If you are not replicating the data or using some form of a shared file system, you are really in a bind.  At one of my schools I have a similar setup that you just did.  I have two servers (minis) each with storage (pegasus) and the load is divided between the two based on grade level divisions.  There is a period in the day where no computers are ever used (lunch/recess).  During that period and at the end of the day I rsync data between the two systems.  It is not live replication but it is always within 4 hours.  By doing so, I can ensure that I can drop an entire server and storage device and still be able to server the whole school.  My plan is to do the DNS swap allowing the remaining server to assume the ID of both servers.  Now that is my plan.  However, I also wrote some dscl scripts to rapidly alter student home folder paths to allow a quick flip from mac1 to mac1.  Luckily I've not needed to implement this.  And I have concerns about OD trying to replicate to itself.  But in theory this should work for the client devices.

Quote
Scenario one, my ideal situation for network homes would be iSCSI attached storage to a file server that OD Master and OD replica could connect to and in case of any one of the OD servers crashes, there would be no loss of network homes.

But you still have the DNS and NFSHomeFolder path challenge.  Even with common storage, if the Mary's home is on mac1 and John's is on mac2, the lost of the host remains the roadblock.  Now, there is likely some round robin DNS games you can play but knowing OS X and mDNSResponder, the machine will start resolving different addresses during a user session.  This is worth a shot in isolation.  The idea being that multiple servers will have the same host name but different IP addresses.  If you were to do this I would recommend multihoming the Ethernet and setting the primary address to fixed names and numbers master = 172.16.4.10 and replica = 172.16.4.11.  Then you would create nethome and point it to 172.16.4.12 and 13.  Then you would assign 12 to 10 and 13 to 11.  This way OD can communicate using 10 and 11 and master and replica while network homes can all be assigned to nethome at both 12 and 13.  In theory, clients will, by law of averages, split the response by the DNS system and each will seek nethome at a different IP address.  But since each student record has a path for nethome as the NFSHomeFolder path it will simply route to were it belongs.  Now, the one challenge here is what happens with a host is unresponsive.  Round robin does not come back to DNS if the destination is unreachable.  This is theory by the way.  I have not tried this (but now I may  ;))

Quote
Scenario two, could be what you propose and I could just break up Students homes on to Replica_1 and Staff/Admin homes on Replica_2, with failovers set up in VM ESXi.

That may work also, especially if you have a failover unit that assumes the role of the failed instance.  If you get that going it might be the best of all solutions since you are just discarding an instance for another one.

Quote
Am I dreaming or could we get this to work?! What are your thoughts?
It’s refreshing to have a great discussion that actually leads to an awesome outcome! Thank you again

It is good to dream and to push.  I think you have some options with the ESXi side that neither of us is seeing quite yet.  I can see some OD synchronization issue with spinning up a replacement instance but if you are much like the schools I work with, modification to OD after the school year starts is minimal. 

I say try it while you have the opportunity to explore.  And glad to help.  As mentioned, making Server a successful product has become my mission.  I truly believe in the product and the role it can play in organizations of nearly any size.

Reid

adi007

  • Newbie
  • *
  • Posts: 5
    • View Profile
Re: Getting Network Homes to work over Network
« Reply #8 on: December 01, 2015, 11:05:24 PM »
Hi Reid,

i will go and test more and keep you posted on the progress.

I too believe in the product and that the possibilities are there for it to be a great server software!

Thank you and speak soon, I am also looking forward to your third book!!

Adi